Overview

The following components are available in the 1.0 release:

• Canvas Extractor
• Google Classroom Extractor
• Schoology Extractor
• LMS Data Store Loader

Please see LMS Toolkit for more information about the purpose of these tools.

Pre-Requisites

• Python 3.9 or higher 

Running the Tools

The LMS Toolkit components can be installed into other Python scripts as dependencies, or they can run as stand-alone command line scripts from the source code.

pip install edfi-canvas-extractor
pip install edfi-google-classroom-extractor
pip install edfi-schoology-extractor
pip install edfi-lms-ds-loader
pip install edfi-lms-harmonizer

Extractors

Whether you run the extractors by incorporating into an existing Python package, or by using the stand-alone command line utility from the source repository, there are a number of required and optional arguments. When running with the command line tool, simply provide the --help  option for the full set of options for each extractor.

# Two ways to get these three optional features:
poetry run python .\edfi_google_classroom_extractor -f activities, grades, assignments
poetry run python .\edfi_google_classroom_extractor -f activities grades assignments

# Retrieve only the "activities" data (in addition to the core data set).
# Note the use of the "long flag" intead of `-f`.
poetry run python .\edfi_google_classroom_extractor --feature activities

LMS Data Store Loader

Please note that the Data Store Loader controls deployment of its own database schema. For performance optimization it also creates, drops, and renames tables during the upload process. Therefore the user account running this tool must have permission to modify the schema.

The following table lists the arguments for calling the loader utility.

Argument	Required	Purpose
​DB server	​yes	The destination database server/host name​
DB port	no	Optional alternate port number (default: 1433)
DB name	yes	Name of the database to connect to on the host
Exceptions report directory	no	Optional directory for writing out CSV files with LMS records that could not be matched to SIS records
DB username	no	Optional database username (must either use username and password, or use integrated security)
DB password	no	Optional database password
Use integrated security	no	Optional flag to use integrated authentication when connecting to the database
Use encrypted connection	no	Enables an encrypted connection to the database
Trust server certificates	no	When encrypting the database connection, trust the server certificate.  primarily used for development, not intended for production use
Log Level	No	Valid options are: DEBUG, INFO (default), WARNING, ERROR, CRITICAL

LMS Harmonizer

The following table lists the arguments for calling the harmonizer utility.

Argument	Required	Purpose
​DB server	​yes	The destination database server/host name​
DB port	no	Optional alternate port number (default: 1433)
DB name	yes	Name of the database to connect to on the host
Exceptions report directory	no	Optional directory for writing out CSV files with LMS records that could not be matched to SIS records
DB username	no	Optional database username (must either use username and password, or use integrated security)
DB password	no	Optional database password
Use integrated security	no	Optional flag to use integrated authentication when connecting to the database
Use encrypted connection	no	Enables an encrypted connection to the database
Trust server certificates	no	When encrypting the database connection, trust the server certificate.  primarily used for development, not intended for production use
Log Level	No	Valid options are: DEBUG, INFO (default), WARNING, ERROR, CRITICAL

Analyzing Student Data Using Extractor Output

The LMS Data Store Loader pushes the extractor-created CSV files into a SQL Server database, where the data are available for use via standard SQL Server interfaces and tools. However, the CSV files can also be consumed directly to perform many interesting analyses. We have a developed a set of Jupyter notebooks that demonstrate analytics tasks that can be performed in Python using the Pandas framework, reading raw CSV files. Sample output from these notebooks is visible directly in GitHub, without needing to run the code locally:

• Filesystem Tutorial / In Danger of Failing / Missing Assignment Submissions: how to use the LMS Toolkit scripts to understand and access output files created by the extractors. Also includes two analysis scenarios - looking for students who are in danger of failing, and looking for missing assignment submissions.
• Record Counts: simply accesses all of the extracted files and provides summary count of records downloaded.
• Student Logins: simple visualization showing frequency of student logins to the LMS.
• Student Submissions: shows the count of assignments submitted per student, by status.

Operational Concerns

Logging

import logging
import sys
from edfi_schoology_extractor.helpers.arg_parser import MainArguments as s_args
from edfi_schoology_extractor import extract_facade

# Setup global logging
logging.basicConfig(stream=sys.stdout, level=logging.INFO)

# Prepare parameters
arguments = s_args(
    client_key=KEY,
    client_secret=SECRET,
    output_directory=OUTPUT_DIRECTORY,
	# ----------- Here is the log level setting -----------
    log_level=LOG_LEVEL,
	# -----------------------------------------------------
    page_size=200,
    input_directory=None,
    sync_database_directory=SYNC_DATABASE_DIRECTORY
)

# Run the Schoology extractor
extract_facade.run(arguments)

poetry run python edfi-canvas-extractor > 2021-05-02-canvas.log

poetry run python edfi-canvas-extractor --log-level DEBUG > 2021-05-02-canvas.log

Security

Upstream APIs

Each API has its own process for securing access. Please see the respective readme files for more information:

• Canvas
• Google Classroom
• Schoology

Data Storage

Given the LMS Toolkit deals with student data, both the filesystem and database (if uploading to SQL Server) are subject to all of the same access restrictions as the Ed-Fi ODS database.

Database Permissions

The LMS Data Store Loader tool manages its own database tables. Thus the first time you run the tool, the credentials used to  connect to SQL Server need to have the db_ddladmin  permission in order to create the necessary tables. Subsequent executions can use an account with more restrictive permissions, i.e. the db_datawriter role.

Scheduling

The API's provided by these three learning management systems are well defined at a granular level. From a performance perspective, this means that the process of getting a complete set of data is very chatty and may take a long time to process. It is difficult to predict the exact impact, although generally the time will scale proportional to the number of course sections. Some of the API's also do not have any mechanism for restricting the date range or looking for changed data, resulting in each execution of the extractor re-pulling the entire data set.

If running on a daily basis, then we recommend running after normal school hours to minimize contention with network traffic to the source system. If running weekly, then it may be best to run over the weekend. 

It should be trivial to call these programs from Windows Task Scheduler, Linux chron, or a workflow engine such as Apache Airflow.