Overview

Guidance Summary

• Overall Status: Current.
• Alliance Review: Alliance in agreement, some SME consultations remaining.
• Governance Review: TAG in agreement.
• Dissemination Status: Not common knowledge in the Ed-Fi Community. Tech Docs requires updates to adhere to this guidance. Some implementations do not adhere to this guidance.
• Implementation Status: Some Alliance products do not adhere to this guidance.

Background

In early to mid-2022, the Ed-Fi Alliance TAG (Technical Advisory Group) came to strong agreement that API-first should be the model for analytics, not direct SQL on the ODS. This means that data is moved out of Ed-Fi via the API and transformed into datasets meant for analytics downstream. This typically takes the form of dimension and fact tables in a columnar data warehouse but can also be represented via views on top of the data lake if the BI tool has a built-in caching system (Power BI). 

There were several reasons for this recommendation: 

• The ODS is a transactional database using row-based storage. This is great for creating, retrieving, updating, and deleting records, but is not great for analytics type SQL queries where something like an aggregation is done across an entire column. 
• The Ed-Fi Data Standard is in part, the Ed-Fi API. The ODS is not a part of the Data Standard. Rather, it is simply the storage mechanism used by the API and should not be seen as anything more than that.
• Historically, the Ed-Fi Alliance has developed the Analytics Middle Tier to be a set of database views that sit on top of the ODS. This had performance issues since the beginning that the Alliance tried to resolve by adding more indexes to the ODS structure. As a longer-term solution, the AMT team is moving to adopt an API-first strategy as well with their tooling.  

Details

ODS as Storage

Related to an Ed-Fi API first strategy is adopting a viewpoint that the ODS is a storage mechanism for the API and nothing more. The Ed-Fi Data Standard is the Ed-Fi Unifying Data Model and the Rest API specification. It is not the ODS. The purpose of the ODS is to store the data sent to the API so that it may be retrieved later on by future API calls. It is recommended that implementers of the Ed-Fi technology suite take a "set it and forget it" view of the ODS. They should be discouraged from connecting to the ODS directly and instead be encouraged to use the API.

The Ed-Fi tech team maintains a highly normalized database with the purpose of easing transactional updates coming in via the API. Historically, between Ed-Fi technology version updates,  it has been the case where the tech team updates the database structure of the ODS without the Data Standard version updating. Treating the Ed-Fi technology separate from the Ed-Fi Data Standard is important.

Meadowlark is a R&D project within the Ed-Fi Alliance that is looking at replacing the ODS with a document store. Again, this is an example of improving the Ed-Fi technology separate from improving the data standard.