Overview

The following components are available in the 1.0 release:

• Canvas Extractor
• Google Classroom Extractor
• Schoology Extractor
• LMS Data Store Loader

Please see LMS Toolkit for more information about the purpose of these tools.

Pre-Requisites

• Python 3.9 or higher
• In theory these tools should work from any operating system that supports Python 3.9. In practice, these tools have only been tested on Windows 10.

Running the Tools

The LMS Toolkit components can be installed into other Python scripts as dependencies, or they can run as stand-alone command line scripts from the source code.

pip install edfi-canvas-extractor
pip install edfi-google-classroom-extractor
pip install edfi-schoology-extractor
pip install edfi-lms-ds-loader

Using Extractor Output

The LMS Data Store Loader pushes the extractor-created CSV files into a SQL Server database, where the data are available for every use that one can imagine from a SQL database. However, the CSV files can also be consumed directly to perform many interesting analyses. We have a developed a set of Jupyter notebooks that demonstrate analytics tasks that can be performed in Python using the Pandas framework, reading raw CSV files. Sample output from these notebooks is visible directly in GitHub, without needing to run the code locally:

• Filesystem Tutorial / In Danger of Failing / Missing Assignment Submissions: how to use the LMS Toolkit scripts to understand and access output files created by the extractors. Also includes two analysis scenarios - looking for students who are in danger of failing, and looking for missing assignment submissions.
• Record Counts: simply accesses all of the extracted files and provides summary count of records downloaded.
• Student Logins: simple visualization showing frequency of student logins to the LMS.
• Student Submissions: shows the count of assignments submitted per student, by status.

Operational Concerns

Logging

import logging
import sys
from edfi_schoology_extractor.helpers.arg_parser import MainArguments as s_args
from edfi_schoology_extractor import extract_facade

# Setup global logging
logging.basicConfig(stream=sys.stdout, level=logging.INFO)

# Prepare parameters
arguments = s_args(
    client_key=KEY,
    client_secret=SECRET,
    output_directory=OUTPUT_DIRECTORY,
	# ----------- Here is the log level setting -----------
    log_level=LOG_LEVEL,
	# -----------------------------------------------------
    page_size=200,
    input_directory=None,
    sync_database_directory=SYNC_DATABASE_DIRECTORY
)

# Run the Schoology extractor
extract_facade.run(arguments)

poetry run python edfi-canvas-extractor > 2021-05-02-canvas.log

poetry run python edfi-canvas-extractor --log-level DEBUG > 2021-05-02-canvas.log

Security

Upstream APIs

Each API has its own process for securing access. Please see the respective readme files for more information:

• Canvas
• Google Classroom
• Schoology

Data Storage

Given the LMS Toolkit deals with student data, both the filesystem and database (if uploading to SQL Server) are subject to all of the same access restrictions as the Ed-Fi ODS database.

Database Permissions

The LMS Data Store Loader tool manages its own database tables. Thus the first time you run the tool, the credentials used to  connect to SQL Server need to have the db_ddladmin  permission in order to create the necessary tables. Subsequent executions can use an account with more restrictive permissions, i.e. the db_datawriter role.

Scheduling

The API's provided by these three learning management systems are well defined a granular level. From a performance perspective, this means that the process of getting a complete set of data is very chatty and may take a long time to process. It is difficult to predict the exact impact, although generally the time will scale proportional to the number of course sections. Some of the API's also do not have any mechanism for restricting the date range or looking for changed data, resulting in each execution of the extractor re-pulling the entire data set.

If running on a daily basis, then we recommend running after normal school hours to minimize contention with network traffic to the source system. If running weekly, then it may be best to run over the weekend. 

It should be trivial to call these programs from Windows Task Scheduler, or Linux chron, or even a workflow engine such as Apache Airflow.