The Ed-Fi “Classic Dashboards” are no longer supported through the Ed-Fi Alliance. You can still receive support and maintenance through the Ed-Fi vendor community. Please look at any of the vendors’ dashboard solutions on the Registry of Ed-Fi Badges or the Ed-Fi Starter Kits if you are looking for a visualization solution to use with the Ed-Fi ODS. This documentation will remain available to assist existing Classic Dashboard implementers.
ETL Developers' Guide - Key Concepts
- Itzel Torres
- Ian Christopher (Deactivated)
The Ed-Fi Dashboards v3.0 ETL (Extract-Transform-Load) Application is a .NET executable responsible for extracting relevant data from the ODS (Extract), performing metric calculations (Transform), and writing the results of those calculations to relevant Dashboard tables (Load). The diagram below shows where the ETL fits in the Ed-Fi Dashboards system:
Note that the Dashboards ETL is not responsible for loading data into the ODS.
The ETL Application has four fundamental components: Readers, Translators, Writers, and the Core Infrastructure.
Readers
Readers extract the data. In order to perform metric calculations, the ETL Application must first query the data needed from the ODS database. This data is extracted using SQL scripts that query the database and are then mapped to objects (i.e., C# classes using Dapper ORM). The classes for these operations are referred to as Readers.
Translators
Translators transform the data. Once the required data for a metric is read from the ODS, the metric calculation can be made which is handled by C# classes referred to as Translators. The Translators are responsible for applying the business logic that makes up each metric calculation and are tested accordingly. The output of a Translator is typically an object class that represents a record that will be written to a Dashboard table.
Writers
Writers load the data. Once the metric calculations are made, the results are written to the appropriate Dashboard tables. The ETL Application leverages T4 Templates to link the output objects of the Translators to the appropriate target Dashboard tables. The operation uses the C# IBuffer interface class to improve writing performance. These operations are referred to as Writers.
Core Infrastructure (Common)
The ETL is a .NET application, and several aspects of the .NET Framework were leveraged to make execution, troubleshooting, customization, and extension easier and more performant than other methods (e.g., the SSIS that shipped with earlier versions of the Ed-Fi Dashboards). Operations for the core infrastructure like starting the application, reading application settings, message publishing/streaming (via an IBus interface), and logging are referred to as Common. Whether Reader, Translator, or Writer, each part of the application publishes messages to the bus to be streamed by other components who care about them.
The diagram below shows how all these components work together:
