Overview

The following components are available in the 1.0 release:

• Canvas Extractor
• Google Classroom Extractor
• Schoology Extractor
• LMS Data Store Loader

Please see LMS Toolkit for more information about the purpose of these tools.

Pre-Requisites

• Python 3.9 or higher 

Running the Tools

The LMS Toolkit components can be installed into other Python scripts as dependencies, or they can run as stand-alone command line scripts from the source code.

pip install edfi-canvas-extractor
pip install edfi-google-classroom-extractor
pip install edfi-schoology-extractor
pip install edfi-lms-ds-loader
pip install edfi-lms-harmonizer

Extractors

Whether you run the extractors by incorporating into an existing Python package, or by using the stand-alone command line utility from the source repository, there are a number of required and optional arguments. When running with the command line tool, simply provide the --help  option for the full set of options for each extractor.

# Two ways to get these three optional features:
poetry run python .\edfi_google_classroom_extractor -f activities, grades, assignments
poetry run python .\edfi_google_classroom_extractor -f activities grades assignments

# Retrieve only the "activities" data (in addition to the core data set).
# Note the use of the "long flag" intead of `-f`.
poetry run python .\edfi_google_classroom_extractor --feature activities

LMS Data Store Loader

Please note that the Data Store Loader controls deployment of its own database schema. For performance optimization it also creates, drops, and renames tables during the upload process. Therefore the user account running this tool must have permission to modify the schema.

The following table lists the arguments for calling the loader utility.

Argument	Required	Purpose
​DB server	​yes	The destination database server/host name​
DB port	no	Optional alternate port number (default: 1433)
DB name	yes	Name of the database to connect to on the host
Exceptions report directory	no	Optional directory for writing out CSV files with LMS records that could not be matched to SIS records
DB username	no	Optional database username (must either use username and password, or use integrated security)
DB password	no	Optional database password
Use integrated security	no	Optional flag to use integrated authentication when connecting to the database
Use encrypted connection	no	Enables an encrypted connection to the database
Trust server certificates	no	When encrypting the database connection, trust the server certificate.  primarily used for development, not intended for production use
Log Level	No	Valid options are: DEBUG, INFO (default), WARNING, ERROR, CRITICAL

LMS Harmonizer

Database Tables

In addition to the tables created by the LMS Data Store Loader, in the lms  schema, the LMS Harmonizer requires access to the Ed-Fi ODS database tables and to the lmsx  extension tables.  Currently the toolkit officially supports ODS/API Suite 3, version 5.2. It should work in other versions but has not been tested.

1. Install the lmsx  schema tables through one of two options:
   1. Initdev option for a fresh ODS database:
      1. Copy the EdFi.Ods.Extensions.LMSX folder from the LMS Toolkit source code into your Ed-Fi-ODS-Implementation/Application  folder.
      2. In the WebAPI project, add a reference to the LMSX project
      3. Run initdev
   2. Manual option for existing ODS databases:
      1. From source code, open folder extension\EdFi.Ods.Extensions.LMSX\Artifacts\MsSql\Structure\Ods 
      2. Run each of the scripts there, in numeric order. If using change queries, run the scripts in the Changes folder as well
2. The Harmonizer has several stored procedures and views, which currently are only installed manually.
   1. From source code, open folder extension\EdFi.Ods.Extensions.LMSX\LMS-Harmonizer.
   2. Run each of the scripts there, in numeric order.

Arguments

The following table lists the arguments for calling the harmonizer utility.

Argument	Required	Purpose
​DB server	​yes	The destination database server/host name​
DB port	no	Optional alternate port number (default: 1433)
DB name	yes	Name of the database to connect to on the host
Exceptions report directory	no	Optional directory for writing out CSV files with LMS records that could not be matched to SIS records
DB username	no	Optional database username (must either use username and password, or use integrated security)
DB password	no	Optional database password
Use integrated security	no	Optional flag to use integrated authentication when connecting to the database
Use encrypted connection	no	Enables an encrypted connection to the database
Trust server certificates	no	When encrypting the database connection, trust the server certificate.  primarily used for development, not intended for production use
Log Level	No	Valid options are: DEBUG, INFO (default), WARNING, ERROR, CRITICAL

Analyzing Student Data Using Extractor Output

The LMS Data Store Loader pushes the extractor-created CSV files into a SQL Server database, where the data are available for use via standard SQL Server interfaces and tools. However, the CSV files can also be consumed directly to perform many interesting analyses. We have a developed a set of Jupyter notebooks that demonstrate analytics tasks that can be performed in Python using the Pandas framework, reading raw CSV files. Sample output from these notebooks is visible directly in GitHub, without needing to run the code locally:

• Filesystem Tutorial / In Danger of Failing / Missing Assignment Submissions: how to use the LMS Toolkit scripts to understand and access output files created by the extractors. Also includes two analysis scenarios - looking for students who are in danger of failing, and looking for missing assignment submissions.
• Record Counts: simply accesses all of the extracted files and provides summary count of records downloaded.
• Student Logins: simple visualization showing frequency of student logins to the LMS.
• Student Submissions: shows the count of assignments submitted per student, by status.

Operational Concerns

Logging

import logging
import sys
from edfi_schoology_extractor.helpers.arg_parser import MainArguments as s_args
from edfi_schoology_extractor import extract_facade

# Setup global logging
logging.basicConfig(stream=sys.stdout, level=logging.INFO)

# Prepare parameters
arguments = s_args(
    client_key=KEY,
    client_secret=SECRET,
    output_directory=OUTPUT_DIRECTORY,
	# ----------- Here is the log level setting -----------
    log_level=LOG_LEVEL,
	# -----------------------------------------------------
    page_size=200,
    input_directory=None,
    sync_database_directory=SYNC_DATABASE_DIRECTORY
)

# Run the Schoology extractor
extract_facade.run(arguments)

poetry run python edfi-canvas-extractor > 2021-05-02-canvas.log

poetry run python edfi-canvas-extractor --log-level DEBUG > 2021-05-02-canvas.log

Security

Upstream APIs

Each API has its own process for securing access. Please see the respective readme files for more information:

• Canvas
• Google Classroom
• Schoology

Data Storage

Given the LMS Toolkit deals with student data, both the filesystem and database (if uploading to SQL Server) are subject to all of the same access restrictions as the Ed-Fi ODS database.

Database Permissions

As noted in the LMS Data Store Loader section above, in addition to read and write permissions (db_datareader and db_datawriter roles), the database user running that tool must have permission to alter SQL schema, which is typically granted through membership in the db_ddladmin role.

The LMS Harmonizer can be run under an account that only has read and write permissions.

Scheduling

The API's provided by these three learning management systems are well defined at a granular level. From a performance perspective, this means that the process of getting a complete set of data is very chatty and may take a long time to process. It is difficult to predict the exact impact, although generally the time will scale proportional to the number of course sections. Some of the API's also do not have any mechanism for restricting the date range or looking for changed data, resulting in each execution of the extractor re-pulling the entire data set.

If running on a daily basis, then we recommend running after normal school hours to minimize contention with network traffic to the source system. If running weekly, then it may be best to run over the weekend. 

It should be trivial to call these programs from Windows Task Scheduler, Linux chron, or a workflow engine such as Apache Airflow.