Meadowlark - API Parity Gaps

Owned by Eric Jansson
Last updated: Mar 30, 2023 by Stephen Fuqua

What is "API Parity"?

The goal of Meadowlark is to able to "fool" a current API client: to be able to substitute the Ed-Fi API provided by Meadowlark with the API provided by the Ed-Fi ODS/API and have API clients continue to function/not realize that they were communicating with a different API. We refer to this as "API parity."

Broadly speaking, API parity was achieved, but there were some gaps. These gaps could be closed, but in each case the project chose not to do so. This document provides a list of the known gaps to API parity, and explains why the gap was left in place.

Will these gaps be closed?

Some may, but it is unlikely that all such gaps will be closed.  Ed-Fi is an both an effort to build open source data infrastructure AND an effort to provide blueprints for standardize data flows. In respect of the latter goal of standardization, it is highly useful to compare API differences across API implementations.

Rather than try to close all these gaps, the goal should be to clearly define what API features are required and which should be allowed to vary. Doing so will allow for the development of alternative API implementations, whether through the open-source effort of the Ed-Fi community or through efforts independent outside of that community work.

List of API Parity Gaps

(tick) Likely to Support in the Future

OAuth 2.0 Provider

DescriptionMeadowlark's current authentication is hard coded to two key/secret pairs and hard-coded claims.
DirectionMilestone 0.3.0 will include full OAuth 2.0 support including an "internal" OAuth server. See Meadowlark - Internal OAuth 2 Client Credential Provider

Extensions

DescriptionIn the ODS/API, extensions to the Ed-Fi Data Model can be compiled and integrated into the software, thus expanding the API surface.
Likely directionWe believe this will be relatively easy in Project Meadowlark, and it simply hasn't become a critical priority yet.

Etags

DescriptionThe ODS/API includes an eTag value in each resource body sent from a GET request, and it validates the eTag when used in a header on a PUT request.
Likely DirectionThis should be easy to add at the point when it becomes a priority.

(warning) Depends on Feedback

"link" objects in JSON

Description

In the ODS/API, the JSON is annotated by "link" elements that show the path to the element using a GET by the resource ID. These elements appear like this:

"gradingPeriodReference": {
      "gradingPeriodDescriptor": "uri://ed-fi.org/GradingPeriodDescriptor#First Six Weeks",
      "periodSequence": 1,
      "schoolId": 255901001,
      "schoolYear": 2022,
      "link": {
        "rel": "GradingPeriod",
        "href": "/ed-fi/gradingPeriods/0d4a8d72801240fd805ee118b2641b0f"
      }
    },

These elements do not appear in the GET elements provided by Meadowlark.

Likely direction

It is unlikely that these will be supported, and in general the direction is to continue to omit these from Ed-Fi API specifications.

  • The utility of these elements is doubtful: they seem to be an implementation feature/decision made by the ODS/API project and do not seem to be in wide use. The intention seems to be to deliver a HATEOS-type information to clients, but that model of interaction has generally not emerged as best practice in REST APIs.
  • Since Meadowlark takes a document-centric approach to collection and data management, annotating the documents would create additional complexity for any APIs of this kind; without compelling value for this feature, it was judged to be better to simply omit the feature.

 "discriminator" fields on abstract class EducationOrganization

Description

The ODS API provides for discriminators that inform the API client what specific subclass of a abstract class is being referenced. This is done via a "link" object that includes a "rel" field that indicates the class of the referent object. See below for an example of this on the /course API resource.

{
    "id": "16904b88d3c144b4a43af2924f4c4590",
    "educationOrganizationReference": {
      "educationOrganizationId": 255901001,
      "link": {
        "rel": "School",
        "href": "/ed-fi/schools/c81a158d7caf49f299ff3c22b503b334"
      }
    },
    "courseCode": "03100500",
    "courseDefinedByDescriptor": "uri://ed-fi.org/CourseDefinedByDescriptor#SEA",
    "courseDescription": "Algebra I",
Likely direction

This feature was added to the ODS API in the interest of simplifying data usage for outbound/pulling API clients, especially for cases in which there is a high priority on API simplicity, as for the roster/enrollment API. See ODS-2299 - Getting issue details... STATUS

However, those use cases are not the focus of the initial Meadowlark scope, so it is unclear if this should be addressed. We will likely await further feedback, and if this emerges as a need, possibly look at other implementation options for solving the same problem (e.g., might it be better to ask a client to maintain a cache of EdOrgs, and possibly add support that allows them to do that more easily?). To insert the capability to annotate JSON documents would add complexity that is not clearly justified.

API Case Sensitivity

Description

Field names in the ODS/API are insensitive to case. For example, lastSurName , LASTSURNAME , and lastsurName  are treated interchangeably. Meadowlark treats these as distinct field names. Since a Student is defined in the data model as having a lastSurname , then Meadowlark does not allow any variant when accepting incoming data, including on the GET /ed-fi/student?lastSurname=  query string parameter.

The Ed-Fi API specifications, under /wiki/spaces/EFAPIGUIDE/pages/24281163, state: "URIs, parameter names, and parameter values must not be case sensitive. The two URI’s to the left will produce the same results."

This is consistent with many API's, especially those developed on Microsoft technology. However, it violates the letter of the HTTP specification: "The scheme and host are case-insensitive and normally provided in lowercase; all other components are compared in a case-sensitive manner."

Furthermore, JSON-based document storage inherently can only be case-insensitive on keys if you store everything in a consistent case (e.g. all lowercase) and make your comparisons in that case. Since the returned object needs to be in the correct case, having case insensitivity on query parameter names would require either double storage in the query engine, or using the query engine only for lookup of records, and then retrieval of those records from the primary storage (e.g. MongoDB or PostgreSQL), thus requiring much more network overhead.

Furthermore, the Open API specification states that "Parameter names are case sensitive." Thus conformance to the Ed-Fi published API "swagger" specs implies case sensitivity.

Likely Direction​The written specifications for an Ed-Fi API need to be updated to reflect the research above - namely, that all path and query elements should be considered case sensitive. That the ODS/API will accept any case should be seen as leniency rather than compliance.

Change Queries

DescriptionThe Changed Record Queries feature (aka "Change Queries") provides a mechanism for synchronizing a downstream system by retrieving only the records that have changed since the last synchronization process.
​Likely DirectionHas not been evaluated in detail for Meadowlark.

Complex Authorization Models

DescriptionClients of the Ed-Fi ODS/API Suite 3 can be authorized in a matrix of education organizations, people, and other factors.
​Likely Direction

​Meadowlark is choosing to keep permissions as simple as possible, but if there is sufficient demand, it conceivably can be expanded to accommodate other permission mechanisms.

As an alternative (for example), an Assessment Rostering API could be built from the /wiki/spaces/EFDSDRAFT/pages/22773804 specification, which would provide a different security model from the core student data API.

Profiles

DescriptionProfiles support customization of which data fields are available to an API client.
​​Likely DirectionIt is not relevant with the current ownership authorization model, but cannot be ruled out in a future model.

"Get" Search Pattern 

DescriptionODS/API search fields are database column based, and different from the document fields they search on. See Meadowlark - Get Search Pattern Gaps
​​Likely DirectionTBD

(error) Will Not Support

  • Begin and End Dates on Descriptors
    The Descriptor API is an unofficial one. The Ed-Fi ODS/API surfaces two properties that are not used anywhere in the application code: effectiveBeginDate  and effectiveEndDate. These are not implemented in Meadowlark.
  • Cascading Key Changes
    cf How To: Enable Cascading Updates on ODS / API Resources

  • Over-posting: posting fields not part of the JSON schema
    The Ed-Fi ODS API allows for extraneous fields to be posted without error; such fields are simply ignored. In Meadowlark, these are schema violations and a 4xx error is returned.

  • Accepting Strings as Numbers and Booleans
    The Ed-Fi ODS/API casts numeric and boolean values from string equivalents. For example, "2022" is handled as the number 2022, and "true" is handled as the Boolean True. This behavior does not conform to the publish API specification, and it will not be carried forward into Meadowlark.

  • Composites as Currently Used
    The purpose of composites may be fulfilled with a different feature, which will not use the current XML approach.


Parking Lot

Stephen Fuqua needs to expand on these issues

  • Descriptor namespaces
  • year and instance routing
    • probably support via reverse proxy, but even then need to make it easy to have separate database names
  • PUT requests that try to change the natural key
  • String minLength validation
    • If nullable, and minLength = 1, then should empty string be allowed?
  • ODS-3696 - Getting issue details... STATUS
  • null decimal treated as zero. CourseTranscript.EarnedCredits is required but you can leave it out. Seems like that shouldn't be required then.
  • Compare handling of date/time fields
    • Does ODS/API allow submitting date without time? If so, does it infer midnight?
    • What formats do each allow?

Table of Contents