The following components are available in the 1.0 release:
- Canvas Extractor
- Google Classroom Extractor
- Schoology Extractor
- LMS Data Store Loader
Please see LMS Toolkit for more information about the purpose of these tools.
- Python 3.9 or higher
- In theory these tools should work from any operating system that supports Python 3.9. In practice, these tools have only been tested on Windows 10.
The source code repository has detailed information on each tool. To get started, clone or download the repository and review the main readme file for instructions on how to configure and execute the extractors from the command line.
All of the components are published on PyPi.org so that they can be incorporated into other Python scripts as dependencies
The following commands install all fours tools into the active virtual environment; however, each tool is independent and you can install only the tools you need.
pip install edfi-canvas-extractor
pip install edfi-google-classroom-extractor
pip install edfi-schoology-extractor
pip install edfi-lms-ds-loader
We have developed sample Jupyter notebooks that demonstrate execution of each extractor paired with execution of the LMS Data Store Loader:
When you incorporate the LMS Toolkit components as package dependencies in other Python scripts, then you need to pass the log-level to the main facade class and you need to define the logging format. For example:
import logging
import sys
from edfi_schoology_extractor.helpers.arg_parser import MainArguments as s_args
from edfi_schoology_extractor import extract_facade
# Setup global logging
logging.basicConfig(stream=sys.stdout, level=logging.INFO)
# Prepare parameters
arguments = s_args(
client_key=KEY,
client_secret=SECRET,
output_directory=OUTPUT_DIRECTORY,
# ----------- Here is the log level setting -----------
log_level=LOG_LEVEL,
# -----------------------------------------------------
page_size=200,
input_directory=None,
sync_database_directory=SYNC_DATABASE_DIRECTORY
)
# Run the Schoology extractor
extract_facade.run(arguments)
When running from source code, each extractor logs output to the console; these log messages can be captured in a file by redirecting output to a file:
poetry run python edfi-canvas-extractor > 2021-05-02-canvas.log
The above example assumes that all configuration has been placed into a .env file or environment variables.
The log level defaults to INFO. You can lower the number of log messages by changing to WARNING, or get increased logging by changing to DEBUG. The log level can be set at the command line, in a .env file, or an environment variable (the exact environment variable name depends on the extractor; run the extractor with --help for more information).
poetry run python edfi-canvas-extractor --log-level DEBUG > 2021-05-02-canvas.log
Each API has its own process for securing access. Please see the respective readme files for more information:
Given the LMS Toolkit deals with student data, both the filesystem and database (if uploading to SQL Server) are subject to all of the same access restrictions as the Ed-Fi ODS database.
The LMS Data Store Loader tool manages its own database tables. Thus the first time you run the tool, the credentials used to connect to SQL Server need to have the db_ddladmin permission in order to create the necessary tables. Subsequent executions can use an account with more restrictive permissions, i.e. the db_datawriter role.
The API's provided by these three learning management systems are well defined a granular level. From a performance perspective, this means that the process of getting a complete set of data is very chatty and may take a long time to process. It is difficult to predict the exact impact, although generally the time will scale proportional to the number of course sections. Some of the API's also do not have any mechanism for restricting the date range or looking for changed data, resulting in each execution of the extractor re-pulling the entire data set.
If running on a daily basis, then we recommend running after normal school hours to minimize contention with network traffic to the source system. If running weekly, then it may be best to run over the weekend.
It should be trivial to call these programs from Windows Task Scheduler, or Linux chron, or even a workflow engine such as Apache Airflow.