Participants

Expand
Denver Public Schools - Max Paulson Ed-Fi staff & contractors Adam Hopkins Brad Banister Robert Hunter Sayee Srinivasan Stephen Fuqua Steven Arnold Vinaya Mayya Edufied - Geoff McElhanon EdWise Group - Britto Augustine Resultant - Joshua Impson Simpat Tech - Ashish Patel Utah State Board of Education - Katrina Brinkley

Agenda

Review the roadmap
Demonstration of current work
Review the roadmap
Design Questions

Demonstration

Configuration Service

Claimset management

Authentication

Keycloak

show start to finish initial setup and client creation

Roadmap

Note

What is actually critical for release candidate?

etag, profile, multitenancy & routing
multiple authorization strategies
ownership based auth
can we do just relationship & namespace based authorization?

DS 5.2

call out that IIS / Powershell scripted route will not be ready

be clear about change queries

Design Questions

Roadmap

Planned Architecture

...

Basic timeline

Tech Congress 2025 - release candidate with “basic” feature set useable for pilot testing typical data exchange scenarios.

Info

Focused primarily on LEA and vendor-to-vendor scenarios.

State scenarios are critical for this project. The Alliance will be designing for state usage scenarios before Tech Congress 2025. The primary missing pieces on the release candidate time frame:

Supporting all authorization models
Running on MSSQL
Running on PostgreSQL or MSSQL without Kafka and search database
ODS-like schema for reporting

Summit 2025 - production-ready version 1.0

Info
Aiming for feature parity with the ODS/API from the perspective of API-based integrations, with one primary exception: no plan to support XML-based composites.

Feature List

Note

Reviewing the list below:

Which features listed “By Summit” should we prioritize to try to release sooner?
Are there any features we failed to list?

Tip

Meeting notes:

Data Standard support
- Prioritize 5.2 for Tech Congress; many states are moving that direction right now.
- Nice to have support for 4.0 as well, for pilot testing with vendors who have not updated to 5.x yet.
- 6.0 preview is not useful from a pilot testing standpoint, since no one has an integration yet.
- In 1.0, MSP’s would like to have support for multiple data standards at the same time: a single API deployment that can store both 5.2 and 4.0 data structures, with validation based on URL route. Allows for better cost optimization.
  - Parking lot: design discussion on mechanisms for supporting this. Initial proposal to prefer separate database instances for different Data Standard versions. Maybe tie DS version into the URL routing for instances?
- Extensions should be the next priority after authorization; would like to have by Tech Congress if possible.
Profiles
- Remove “dynamic” from the list to avoid confusion (same with “XML” on “XML Composites”).
- Agree: must have for 1.0 and not mandatory for Release Candidate at Tech Congress.
Level 2 validation
- Need to explain both types better when turning the table below into a formal document.
- For streaming data, mention observability as well.
ETags
- Some client integration may be looking for _etag in the body or the ETag header on POST requests.
- For release candidate, should at least stub out a hard-coded value to prevent breaking client integrations.
  - Design note: we plan to calculate ETags based on hashing the _lastModifiedDate. This is relatively trivial. We should simply consider ETags as a must-have feature for the spring release candidate.
ODS-like Views
- This discussion helped us recognize the value of talking about (at least) two stages of pilot testing: transactional vendor integration (via the API), and downstream reporting.
- ODS-like views are intended to support the second stage of pilot testing. States could start testing a 1.0 release without the ODS-like views. Can potentially defer until after 1.0.
Core database storage
- Correction, Education Analytics is today using Aurora for the ODS/API.
Change Queries
- Current usage scenarios:
  - API-to-API synchronization via API Publisher
  - Assessment rostering
  - Enable Data Union (EDU) data warehousing.
- We should consider this a must have solution, even if the streaming architecture can provide alternative synchronization patterns. Too many integrations would break without this.
Authorization
- Ownership-based authorization and combined strategies are used in multiple states. Definitely a must have for 1.0, but can live without them for the release candidate.

Data Standard and API Standard Compatibility

Feature

ODS/API Platform

Data Management Service

By TC

By Summit

Resources API - core Ed-Fi Data Standard

Status

colour	Green
title	done

Descriptor API - core Ed-Fi Data Standard

Status

colour	Green
title	done

Discovery API

Status

colour	Green
title	done

Data Standard version independence

Status

colour	Blue
title	MUST HAVE

MetaEd-based extensions

Status

colour	Blue
title	MUST HAVE

Dynamic Profiles

Status

colour	Blue
title	must have

Multiple data standards in same deployment

Status

colour	Yellow
title	Could Have

Note
Does anyone want this?

Composites

Status

colour	Red
title	WOn't Have

Data Integrity and Validation Features

Feature

ODS/API Platform

Data Management Service

By TC

By Summit

Level 0 and level 1 validation

Status

colour	Green
title	done

Descriptor validation

Status

colour	Green
title	done

Reference validation

Status

colour	Green
title	done

Cascading updates on key changes

Status

colour	Green
title	done

Level 2 validation via SQL scripts

Status

colour	Blue
title	MUST HAVE

Once a compatibility layer is available, will be able to run the same SQL scripts used today for Level 2 validation

Realtime level 2 validations via streaming data

Status

colour	Yellow
title	Could Have

Note
Does anyone want this? Might be more of a demonstration than a core feature of the system

API Client Features

Feature

ODS/API Platform

Data Management Service

By TC

By Summit

Concurrency management with ETags

Status

colour	Blue
title	MUST HAVE

LIMIT/OFFSET paging

Status

colour	Green
title	done

Cursor-based paging

(7.3)

Status

colour	Blue
title	MUST HAVE

Identification code-based queries

(7.3)

Status

colour	Blue
title	MUST HAVE

Identities API

Status

colour	Yellow
title	could have

Note
Need to review field usage and fitness-for-purpose before committing.

Unique ID System Integration

Status

colour	Yellow
title	could have

Note
Need to review field usage and fitness-for-purpose before committing.

Database Technologies

Feature

ODS/API Platform

Data Management Service

By TC

By Summit

Core database storage in PostgreSQL

Status

colour	Green
title	done

The database structure is very different than the ODS database

Core database storage in MSSQL

Status

colour	Blue
title	MUST HAVE

GET queries using search database

Status

colour	Green
title	done

Either OpenSearch or Elasticsearch

GET queries using relational database

Status

colour	Blue
title	MUST HAVE

Removes the requirement to run Kafka and OpenSearch or Elasticsearch

Reporting queries using ODS database schema

Status

colour	Blue
title	MUST HAVE

Note

This will likely be a compatibility layer to ease the transition for those who have built reporting solutions on the ODS database structure.

Is this required for state-based pilot testing? Can it be finalized after the 1.0 release?

Core database storage in managed PostgreSQL-compatible databases (e.g. Aurora, Cosmos DB)

Status

colour	Purple
title	SHOULD Have

Conceptually this should work, but we may need community help for testing these scenarios while the development team focuses on code-level features.

Realtime population of a data lake

Status

colour	Purple
title	SHOULD Have

Redis-based caching

Status

colour	Purple
title	SHOULD Have

Data Management Features

Feature

ODS/API Platform

Data Management Service

By TC

By Summit

Streaming data out via Kafka

Status

colour	Green
title	done

Change Queries API

Status

colour	Purple
title	SHOULD Have

Required for API Publisher synchronization; if not implemented must provide alternative.

Security Features

Feature

ODS/API Platform

Data Management Service

By TC

By Summit

API-driven client credential management

(Admin API)

Status

colour	Green
title	done

OAuth token authentication

Status

colour	Green
title	done

Integration with third party OAuth identity providers

Status

colour	Blue
title	MUST HAVE

At minimum, will support Keycloak, with clear path for supporting other providers

API-driven claimset management

(Admin API)

Status

colour	Blue
title	MUST HAVE

Namespace authorization

Status

colour	Blue
title	MUST HAVE

Relationship authorization

Status

colour	Blue
title	MUST HAVE

Ownership authorization

Status

colour	Blue
title	MUST HAVE

Combined authorization

Status

colour	Blue
title	MUST HAVE

Extensible authorization filtering

(7.3)

Status

colour	Blue
title	MUST HAVE

Deployment Management

Feature

ODS/API Platform

Data Management Service

By TC

By Summit

Docker images and sample Docker Compose settings

Status

colour	Green
title	done

Multitenancy routing and instance management

Status

colour	Blue
title	MUST HAVE

PowerShell installation scripts for Windows Server

Status

colour	Yellow
title	Could Have

Note
Does anyone want this?

Deployment orchestration via (Kubernetes, Terraform, Cloud Formation, ARM, etc.)

(though there are Exchange contributions)

Status

colour	Red
title	WOn't Have

Unless a community member contributes

Other Integrations

Feature

ODS/API Platform

Data Management Service

By TC

By Summit

SwaggerUI

Status

colour	Blue
title	must have

Admin Console

coming soon

Status

colour	Blue
title	must have

Demonstration

Tip

Meeting notes:

Ran out of time for the demonstration. The sequence diagram below shows the interactions for client credential management.

The demonstration was going to use this file which uses the REST Client extension for Visual Studio Code. Even without VS Code, the file should be easy to follow along with while replicating the API calls in another client such as Postman.

Working Client Management and Authentication

Expand

title	Mermaid diagram

Code Block

sequenceDiagram
    actor Sys Admin

    rect rgb(191, 223, 255)
    note right of Sys Admin: One time setup.

        Sys Admin->>Config Service: POST /connect/register
        Config Service->>Identity Provider: Create credentials
        note right of Identity Provider: Created with config role
        
        Config Service-->>Sys Admin: clientCredentials
    end

    Sys Admin->>Config Service: POST /v2/vendors
    Config Service->>Config Database: INSERT dbo.Vendor

    Sys Admin->>Config Service: POST /v2/applications

    Config Service->>Identity Provider: Create credentials
    note right of Identity Provider: Created with dms role
    Identity Provider-->>Config Service: clientCredentials
    Config Service->>Config Database: INSERT dbo.Application
    Config Database -->>Config Service: applicationId
    Config Service->>Config Database: INSERT dbo.ApplicationEducationOrganization
    Config Service->>Config Database: INSERT dbo.ApiClient

    Config Service-->>Sys Admin: clientCredentials

...

Design Questions

Tip

Meeting notes:

Ran out of time. Will keep these design questions for a future meeting.

What plans are there for error detection (i.e. dropped records) and correction?

A question that arose at the Ed-Fi Summit session. Suggestions?

Gliffy

imageAttachmentId	att571637780
macroId	9501f8b1-3f9b-48ff-941c-47ce84761610
baseUrl	https://edfi.atlassian.net/wiki
name	points of failure
diagramAttachmentId	att572456967
containerId	571375621
version	1
timestamp	1733252844753

How will year rollover be handled?

Perhaps not germane to the release candidate, but we should still ask: are there special considerations that the development team needs to be thinking about in advance?

Version	Old Version 1	New Version Current
Changes made by	Stephen Fuqua	Stephen Fuqua
Saved on	Dec 03, 2024	Dec 07, 2024

Page Comparison

Versions Compared

Key

Participants

Agenda

Demonstration

Roadmap

Design Questions

Roadmap

Planned Architecture

Basic timeline

Feature List

Data Standard and API Standard Compatibility

Data Integrity and Validation Features

API Client Features

Database Technologies

Data Management Features

Security Features

Deployment Management

Other Integrations

Demonstration

Design Questions

What plans are there for error detection (i.e. dropped records) and correction?

How will year rollover be handled?