Table of Contents

Overview

The following components are available.

• Canvas Extractor
• Google Classroom Extractor
• Schoology Extractor
• LMS Data Store Loader
• LMS Extension for the ODS/API Suite 3, version 5.2 and version 5.3
• LMS Harmonizer

Please see LMS Toolkit for more information about the purpose of these tools. The following chart explains which tools you need based on your circumstances.

Image Added

Pre-Requisites

• Python 3.9 or higher 

Installation

The LMS Toolkit components can be installed into other Python scripts as dependencies, or they can run as stand-alone command line scripts from the source code.

pip install edfi-canvas-extractor
pip install edfi-google-classroom-extractor
pip install edfi-schoology-extractor
pip install edfi-lms-ds-loader
pip install edfi-lms-harmonizer

If you intend to push your LMS data into the Ed-Fi ODS, you will also need to install the LMS DataStore Loader and the ODS LMS extension tables.

Installing the LMS Data Store Loader Database Objects

The LMS Data Stored Loader creates and manages its own database objects. Therefore, on at least the first execution of the program, the database connection credentials must be for an account with membership in the db_ddladmin, in addition to membership in the db_datareader and db_datawriter roles in the destination database.

Installing the ODS LMS Extension for the ODS/API

In addition to the tables created by the LMS Data Store Loader, in the lms  schema, the LMS Harmonizer requires access to the Ed-Fi ODS database tables and to the lmsx  extension tables. Currently the toolkit supports ODS/API Suite 3, version 5.2 and version 5.3.

1. Install the lmsx  schema tables through one of two options:

   Deck
   id lms-object-install
   Card label Fresh Database Using Dynamic Plugin title Fresh Database Using Dynamic Plugin Create a new file called lmsx.ps1 in the Ed-Fi-ODS-Implementation/Plugin directory, using the contents shown below ( make sure this file is "lmsx.ps1" not "lms.ps1"). Please read the comments in the file to see how to change from ODS/API version 5.3 to version 5.2. Code Block language powershell collapse true # SPDX-License-Identifier: Apache-2.0 # Licensed to the Ed-Fi Alliance under one or more agreements. # The Ed-Fi Alliance licenses this file to you under the Apache License, Version 2.0. # See the LICENSE and NOTICES files in the project root for more information. #requires -modules "path-resolver" Import-Module (Get-RepositoryResolvedPath 'logistics\scripts\modules\packaging\nuget-helper.psm1') Import-Module (Get-RepositoryResolvedPath "logistics\scripts\modules\tools\ToolsHelper.psm1") $configuration = @{ PackageName = "EdFi.Ods.Extensions.LMSX.1.0.0" ## Uncomment the appropriate line below # For ODS/API Suite 3, version 5.2: #PackageVersion = "5.2.3" # For ODS/API Suite 3, version 5.3: PackageVersion = "5.3.3" PackageSource = "https://pkgs.dev.azure.com/ed-fi-alliance/Ed-Fi-Alliance-OSS/_packaging/EdFi/nuget/v3/index.json" } $pluginPaths = @() $parameters = @{ packageName = $configuration.packageName packageVersion = $configuration.packageVersion packageSource = $configuration.packageSource outputDirectory = "$PSScriptRoot" toolsPath = (Get-ToolsPath) } $pluginPaths += Get-NuGetPackage @parameters return $pluginPaths In your Ed-Fi-ODS-Implementation/Application/EdFi.Ods.WebApi directory, run the following commands ( if you already have any dynamic extensions, then increment the script number in the second command accordingly): Code Block language powershell cd Ed-Fi-ODS-Implementation/Application/EdFi.Ods.WebApi dotnet user-secrets set "Plugin:Folder" "../../Plugin" dotnet user-secrets set "Plugin:Scripts:0" "lmsx" cd ../../ Run initdev from the root of the Ed-Fi-ODS-Implementation directory. Code Block language powershell ./Initialize-PowershellForDevelopment.ps1 Initdev Card label Upgrade Existing Database Using Plugin title Upgrade Existing Database Using Plugin Download the correct version of the EdFi.Suite3.RestApi.Databases NuGet package from Ed-Fi on Azure Artifacts. If you are targeting ODS/API version 5.2, then download version 5.2.14406. If targeting version 5.3, then download 5.3.1146. If you have nuget.exe you can download and extract files with the following command. This will create directory EdFi.Suite3.RestApi.Databases.5.2.14406 in the current working directory. See link above for other available versions, matching the ODS/API releases: Code Block language powershell $url = "https://pkgs.dev.azure.com/ed-fi-alliance/Ed-Fi-Alliance-OSS/_packaging/EdFi%40Release/nuget/v3/index.json" nuget.exe install EdFi.Suite3.RestApi.Databases -version 5.2.14406 -source $url If you don't have nuget.exe, just download from the link above and treat it as a zip file. Unzip to an appropriate location. Locate the directory for your WebAPI website. Does it have a Plugin sub-directory? If not, create it. Copy the full exact path for use in the next step. Create a new file called lmsx.ps1 in the Ed-Fi-ODS-Implementation/Plugin directory, using the contents shown below ( make sure this file is "lmsx.ps1" not "lms.ps1"). Please read the comments in the file to see how to change from ODS/API version 5.3 to version 5.2. Code Block language powershell collapse true # SPDX-License-Identifier: Apache-2.0 # Licensed to the Ed-Fi Alliance under one or more agreements. # The Ed-Fi Alliance licenses this file to you under the Apache License, Version 2.0. # See the LICENSE and NOTICES files in the project root for more information. #requires -modules "path-resolver" Import-Module (Get-RepositoryResolvedPath 'logistics\scripts\modules\packaging\nuget-helper.psm1') Import-Module (Get-RepositoryResolvedPath "logistics\scripts\modules\tools\ToolsHelper.psm1") $configuration = @{ PackageName = "EdFi.Ods.Extensions.LMSX.1.0.0" ## Uncomment the appropriate line below # For ODS/API Suite 3, version 5.2: #PackageVersion = "5.2.3" # For ODS/API Suite 3, version 5.3: PackageVersion = "5.3.3" PackageSource = "https://pkgs.dev.azure.com/ed-fi-alliance/Ed-Fi-Alliance-OSS/_packaging/EdFi/nuget/v3/index.json" } $pluginPaths = @() $parameters = @{ packageName = $configuration.packageName packageVersion = $configuration.packageVersion packageSource = $configuration.packageSource outputDirectory = "$PSScriptRoot" toolsPath = (Get-ToolsPath) } $pluginPaths += Get-NuGetPackage @parameters return $pluginPaths In the new EdFi.Suite3.RestApi.Databases directory, edit configuration.json and add "lmsx" to the Plugin.Scripts array, as shown below. Paste the Plugin directory path into the Folder entry below: Code Block language js "Plugin": { "Folder": "d:/Ed-Fi/5.2/WebApi/Plugin", "Scripts": [ "lmsx" ] } If you use any directory other than the WebAPI project's Plugin directory at this step , then the plugin will not load when you run the application at the final step below. In that same file, adjust the database connection strings and database engine as appropriate for your installation. If you are not sure what they are, then look in the appsettings.json file in your WebAPI directory. Run the database deployment process in PowerShell while in the EdFi.Suite3.RestApi.Databases directory: Code Block language powershell Import-Module ./Deployment.psm1 Initialize-DeploymentEnvironment Open the appsettings.json file in your WebAPI directory, and add an "lmsx" entry under Scripts, just as done in step 3 above. Restart the web site in IIS.

   
   

2. The LMS Harmonizer has several stored procedures and views, which are managed by the application. On first execution of the program, the database connection credentials must be for an account with membership in the db_ddladmin, in addition to membership in the db_datareader and db_datawriter roles in the destination database.

Using the Toolkit Components

Prepare Your LMS

To extract and integrate data from your LMS, you first need to configure it for access.

See - LMS Toolkit User Guide - Preparing your LMS for Data Integration

Extract Data from the LMS

Extraction of data from the LMD is the next step in the pipeline.

See LMS Toolkit User Guide - Extract Data From the LMS

Load Data into the ODS

Once the data is extracted, you can load the data into a relational database, and from there into your Ed-Fi ODS.

See - LMS Toolkit User Guide - Load Data into the ODS

Directly Pushing Data to the ODS/API

With the LMSX extension installed, vendors can directly integrate with the ODS/API via new Assignment and AssignmentSubmission resources.

See - LMSX Extension to the Ed-Fi ODS/API

Analyzing Student Data Using Juypter Notebooks

The LMS Data Store Loader pushes the extractor-created CSV files into a SQL Server database, where the data are available for use via standard SQL Server interfaces and tools. However, the CSV files can also be consumed directly to perform many interesting analyses. We have a developed a set of Jupyter notebooks that demonstrate analytics tasks that can be performed in Python using the Pandas framework, reading raw CSV files. Sample output from these notebooks is visible directly in GitHub, without needing to run the code locally:

• Filesystem Tutorial / In Danger of Failing / Missing Assignment Submissions: how to use the LMS Toolkit scripts to understand and access output files created by the extractors. Also includes two analysis scenarios - looking for students who are in danger of failing, and looking for missing assignment submissions.
• Record Counts: simply accesses all of the extracted files and provides summary count of records downloaded.
• Student Logins: simple visualization showing frequency of student logins to the LMS.
• Student Submissions: shows the count of assignments submitted per student, by status.

Operational Concerns

Logging

import logging
import sys
from edfi_schoology_extractor.helpers.arg_parser import MainArguments as s_args
from edfi_schoology_extractor import extract_facade

# Setup global logging
logging.basicConfig(stream=sys.stdout, level=logging.INFO)

# Prepare parameters
arguments = s_args(
    client_key=KEY,
    client_secret=SECRET,
    output_directory=OUTPUT_DIRECTORY,
	# ----------- Here is the log level setting -----------
    log_level=LOG_LEVEL,
	# -----------------------------------------------------
    page_size=200,
    input_directory=None,
    sync_database_directory=SYNC_DATABASE_DIRECTORY
)

# Run the Schoology extractor
extract_facade.run(arguments)

poetry run python edfi_canvas_extractor > output.log

poetry run python edfi_canvas_extractor --log-level WARNING > output.log

poetry run python edfi_lms_harmonizer --exceptions-report-directory C:\my-directory

Security

Upstream APIs

Each API has its own process for securing access. Please see the respective readme files for more information:

• Canvas
• Google Classroom
• Schoology

Data Storage

Given the LMS Toolkit deals with student data, both the filesystem and database (if uploading to SQL Server) are subject to all of the same access restrictions as the Ed-Fi ODS database.

Database Permissions

As noted in the LMS Data Store Loader section above, in addition to read and write permissions (db_datareader and db_datawriter roles), the database user running that tool must have permission to alter SQL schema, which is typically granted through membership in the db_ddladmin role.

The LMS Harmonizer can be run under an account that only has read and write permissions.

Scheduling

The API's provided by these three learning management systems are well defined at a granular level. From a performance perspective, this means that the process of getting a complete set of data is very chatty and may take a long time to process. It is difficult to predict the exact impact, although generally the time will scale proportional to the number of course sections. Some of the API's also do not have any mechanism for restricting the date range or looking for changed data, resulting in each execution of the extractor re-pulling the entire data set.

If running on a daily basis, then we recommend running after normal school hours to minimize contention with network traffic to the source system. If running weekly, then it may be best to run over the weekend. 

It should be trivial to call these programs from Windows Task Scheduler, Linux chron, or a workflow engine such as Apache Airflow.