Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 3 Next »

Overview

The following components are available in the 1.0 release:

  • Canvas Extractor
  • Google Classroom Extractor
  • Schoology Extractor
  • LMS Data Store Loader

Please see LMS Toolkit for more information about the purpose of these tools.

The LMS Data Store Loader pushes CSV files, created by the extractors, into a SQL Server database. That database can be the same as an Ed-Fi ODS. However, all of the data are loaded into tables in the lms schema instead of the edfi schema.

Pre-Requisites

  • Python 3.9 or higher
  • In theory these tools should work from any operating system that supports Python 3.9. In practice, these tools have only been tested on Windows 10.

Running the Tools from Source Code

The source code repository has detailed information on each tool. To get started, clone the repository from https://github.com/Ed-Fi-Alliance-OSS/LMS-Toolkit and review the main readme file for instructions on how to configure and execute the extractors from the command line.

Running the Tools from Published Packages

All of the components are published on PyPi.org so that they can  be incorporated into other Python scripts. 

Install

The following commands install all fours tools into the active virtual environment; however, each tool is independent and you can install only the tools you need.

pip install edfi-canvas-extractor
pip install edfi-google-classroom-extractor
pip install edfi-schoology-extractor
pip install edfi-lms-ds-loader

To install the most current pre-release version, add the --pre  flag on each command.

Execution

We have developed sample Jupyter notebooks that demonstrate execution of each extractor paired with execution of the LMS Data Store Loader:

Operational Concerns

Logging

From Packages

When you incorporate the LMS Toolkit components as package dependencies in other Python scripts, then you need to pass the log-level to the main facade class and you need to define the logging format. For example:

import logging
import sys
from edfi_schoology_extractor.helpers.arg_parser import MainArguments as s_args
from edfi_schoology_extractor import extract_facade

# Setup global logging
logging.basicConfig(stream=sys.stdout, level=logging.INFO)

# Prepare parameters
arguments = s_args(
    client_key=KEY,
    client_secret=SECRET,
    output_directory=OUTPUT_DIRECTORY,
	# ----------- Here is the log level setting -----------
    log_level=LOG_LEVEL,
	# -----------------------------------------------------
    page_size=200,
    input_directory=None,
    sync_database_directory=SYNC_DATABASE_DIRECTORY
)

# Run the Schoology extractor
extract_facade.run(arguments)

From Source Code

When running from source code, each extractor logs output to the console; these log messages can be captured in a file by redirecting output to a file:

Redirect to file
poetry run python edfi-canvas-extractor > 2021-05-02-canvas.log

The above example assumes that all configuration has been placed into a .env  file or environment variables.

The log level defaults to INFO. You can lower the number of log messages by changing to WARNING, or get increased logging by changing to DEBUG. The log level can be set at the command line, in a .env file, or an environment variable (the exact environment variable name depends on the extractor; run the extractor with --help  for more information). 

Set level to DEBUG
poetry run python edfi-canvas-extractor --log-level DEBUG > 2021-05-02-canvas.log

Security

Upstream APIs

Each API has its own process for securing access. Please see the respective readme files for more information:

Data Storage

Given the LMS Toolkit deals with student data, both the filesystem and database (if uploading to SQL Server) are subject to all of the same access restrictions as the Ed-Fi ODS database.

Database Permissions

The LMS Data Store Loader tool manages its own database tables. Thus the first time you run the tool, the credentials used to  connect to SQL Server need to have the db_ddladmin  permission in order to create the necessary tables. Subsequent executions can use an account with more restrictive permissions, i.e. the db_datawriter role.

Scheduling

The API's provided by these three learning management systems are well defined a granular level. From a performance perspective, this means that the process of getting a complete set of data is very chatty and may take a long time to process. It is difficult to predict the exact impact, although generally the time will scale proportional to the number of course sections. Some of the API's also do not have any mechanism for restricting the date range or looking for changed data, resulting in each execution of the extractor re-pulling the entire data set.

If running on a daily basis, then we recommend running after normal school hours to minimize contention with network traffic to the source system. If running weekly, then it may be best to run over the weekend. 

It should be trivial to call these programs from Windows Task Scheduler, or Linux chron, or even a workflow engine such as Apache Airflow.

Contents

  • No labels