Overview

The following components are available in the 1.0 release:

• Canvas Extractor
• Google Classroom Extractor
• Schoology Extractor
• LMS Data Store Loader

Please see LMS Toolkit for more information about the purpose of these tools.

Pre-Requisites

• Python 3.9 or higher
• In theory these tools should work from any operating system that supports Python 3.9. In practice, these tools have only been tested on Windows 10.

Running the Tools from Source Code

The source code repository has detailed information on each tool. To get started, clone the repository from https://github.com/Ed-Fi-Alliance-OSS/LMS-Toolkit and review the main readme file for instructions on how to configure and execute the extractors from the command line.

Running the Tools from Published Packages

All of the components are published on PyPi.org so that they can  be incorporated into other Python scripts. 

Install

The following commands install all fours tools into the active virtual environment; however, each tool is independent and you can install only the tools you need.

pip install edfi-canvas-extractor
pip install edfi-google-classroom-extractor
pip install edfi-schoology-extractor
pip install edfi-lms-ds-loader

Execution

We have developed sample Jupyter notebooks that demonstrate execution of each extractor paired with execution of the LMS Data Store Loader:

• Canvas
• Google Classroom
• Schoology

Operational Concerns

Logging

TBD

Security

TBD

Scheduling

From Packages

When you incorporate the LMS Toolkit components as package dependencies in other Python scripts, then you need to pass the log-level to the main facade class and you need to define the logging format. For example:

import logging
import sys
from edfi_schoology_extractor.helpers.arg_parser import MainArguments as s_args
from edfi_schoology_extractor import extract_facade

# Setup global logging
logging.basicConfig(stream=sys.stdout, level=logging.INFO)

# Prepare parameters
arguments = s_args(
    client_key=KEY,
    client_secret=SECRET,
    output_directory=OUTPUT_DIRECTORY,
	# ----------- Here is the log level setting -----------
    log_level=LOG_LEVEL,
	# -----------------------------------------------------
    page_size=200,
    input_directory=None,
    sync_database_directory=SYNC_DATABASE_DIRECTORY
)

# Run the Schoology extractor
extract_facade.run(arguments)

From Source Code

When running from source code, each extractor logs output to the console; these log messages can be captured in a file by redirecting output to a file:

poetry run python edfi-canvas-extractor > 2021-05-02-canvas.log

The above example assumes that all configuration has been placed into a .env  file or environment variables.

The log level defaults to INFO. You can lower the number of log messages by changing to WARNING, or get increased logging by changing to DEBUG. The log level can be set at the command line, in a .env file, or an environment variable (the exact environment variable name depends on the extractor; run the extractor with --help  for more information). 

poetry run python edfi-canvas-extractor --log-level DEBUG > 2021-05-02-canvas.log

Security

Upstream APIs

Each API has its own process for securing access. Please see the respective readme files for more information:

• Canvas
• Google Classroom
• Schoology

Data Storage

Given the LMS Toolkit deals with student data, both the filesystem and database (if uploading to SQL Server) are subject to all of the same access restrictions as the Ed-Fi ODS database.

Database Permissions

The LMS Data Store Loader tool manages its own database tables. Thus the first time you run the tool, the credentials used to  connect to SQL Server need to have the db_ddladmin  permission in order to create the necessary tables. Subsequent executions can use an account with more restrictive permissions, i.e. the db_datawriter role.

Scheduling

The API's provided by these three learning management systems are well defined a granular level. From a performance perspective, this means that the process of getting a complete set of data is very chatty and may take a long time to process. It is difficult to predict the exact impact, although generally the time will scale proportional to the number of course sections. Some of the API's also do not have any mechanism for restricting the date range or looking for changed data, resulting in each execution of the extractor re-pulling the entire data set.

If running on a daily basis, then we recommend running after normal school hours to minimize contention with network traffic to the source system. If running weekly, then it may be best to run over the weekend. 

It should be trivial to call these programs from Windows Task Scheduler, or Linux chron, or even a workflow engine such as Apache Airflow.