Effective Dates in the Ed-Fi Data Model

This document provides information on the Ed-Fi Unifying Data Model's (UDM) approach to effective dates, and – where possible – gives practical guidance on using the data model in contexts were dates are of particular importance.

Table of Contents

Overview

The Ed-Fi data model is optimized to capture data as that data exists currently in source systems, This characteristic derives from the central Ed-Fi community use case of aggregating current operational data into a centralized location so that current, actionable insights and current reports (such as compliance or other reports) can be derived from that data.

As such, the Ed-Fi data model has limited support for the concept of "effective dates" -- the dates where a transaction occurred or where organizational recognition of some data (often a demographic fact about an individual) was judged to be true. These latter entities are often seen in data models by using fields that employ an "as of" formulation, as in "the student was judged to be of race 'American Indian or Alaska Native' as of October 1, 2019."

The Ed-Fi data model therefore has no universal feature or strategy to capture effective or "as of" dates across all data entities, because many effective or "as of" dates do not exist in source systems.

This does not mean that the Ed-Fi data model does not have dates that can be used to determine effective dates: there are many, many such dates in the model. But those dates tend to be dates that are also captured in their original source systems, because those dates have important value operationally in the systems in which they are collected.

Examples

To clarify, let's consider some data facts that include dates:

On August 18, Paula Mills enrolled in the school Grand Bend Elementary.
On October 4, Paula Mills was determined to be eligible for Title I services.
On October 12, Paula Mills name was changed to "Pauloa Mills"
On December 10, Pauloa Mills race was changed from "Asian" to "American Indian or Alaska Native" to "Asian"

Of these facts that involve dates, the first 2 dates can be captured in Ed-Fi and the latter 2 cannot. The reasons why are simple:

For facts #1 and #1, the date of an enrollment or program eligibility is a strongly governed date, and so is universally present in student information systems. As such, those systems can provide that date and those dates are included in the Ed-Fi data model.
For facts #3 and #4, these are certainly legitimate "event" dates, but community experience has shown that student information systems either do not generally capture these dates or that if they do, those dates are not easily surfaced in data exchange (e.g. the date is buried in an audit log and inaccessible via routine processes).

Underlying all of this is Ed-Fi's practical approach to data exchange. It is certainly possible to produce data models that capture all the possible dates that any organization could be interested in (including dates for facts #3 and #4 above), but that model can't populated by current operational data from actual systems. As a result, Ed-Fi does not include such dates. You can see in this Ed-Fi's focus on actualize-able data exchange rather than possible organizational data models.

The Period Entity

Ed-Fi does provide one optional pattern that can be employed to capture effective dates where they exist.

The Period entity captures a BeginDate and an EndDate and can be used to provide dates that the value of a data field was judged to be valid. If you look at the use of this pattern in the Ed-Fi Address entity, you can see that it is used to (per the definition there) to capture: "the time periods for which the address is valid."

It is important to note that all uses of the Period entity in the core Ed-Fi UDM are optional: it is this way because the community has evidence that some systems may capture this information (and be able to efficiently provide it) but some do not capture that information.

Guidance for Agencies

Given this, the Alliance provides the following recommendations.

When extending the data model to ask systems to exchange data (e.g. via an Ed-Fi API), limit requests for dates to those that are actually known to be present in source systems.

Asking for data that source systems do not possess simply results in localized and sometimes complex work around for systems that communicate via Ed-Fi (e.g. "our SIS doesn't capture dates for a student name, so I'll just put in the ADA student count date for the state" or forces the system to invent data (e.g. "our SIS doesn't don't capture the date that a student ethnicity was valid, so we'll just set the dates to the first and last date of the current term"). Neither is a desirable outcome for the systems or the communtiy.

Instead of asking for effective dates on data entities, use current collection dates.

If an agency is interested in an account of how many students were classified as "Hispanic or Latino" on October 1, rather than collect "as of" dates on race, use the state of the collection data on October 1 to capture that data (i.e. by saving a copy of the data at the end of the October 1).

The reason is identical to the reason behind the first recommendation: most SIS systems do not capture date changes for many data model entities, including race. To ask a system to supply historical values forces them to either create complex workarounds (at best) or to invent data they don't have in a "best guess" manner (more likely).

Some agencies are surprised to learn that they can successfully dispense with asking for "historical" data, which is often a consequence of a agencies use of "collection windows" or similar strategies. But the experience of the Ed-Fi community has demonstrated that this is not only possible, but is a far less complex (and likely expensive) process for the ecosystem.

Guidance on how to enable such an "real-time" ecosystem is beyond the scope of this document, but readily available within the Ed-Fi community.

Limit use of the Period pattern by either making its use optional or be certain that all submitting systems in the data exchange context have the dates desired.

The Period pattern is meant to enable optional collection of dates, recognizing that some systems may have dates while others do not. Agencies should resist making Period required, unless they are confident that all dependent, submitting systems have the data being requested.