Agenda

Review the roadmap
Demonstration of current work
Design Questions

Roadmap

Planned Architecture

Basic timeline

Tech Congress 2025 - release candidate with “basic” feature set useable for pilot testing typical data exchange scenarios.

Focused primarily on LEA and vendor-to-vendor scenarios.

State scenarios are critical for this project. The Alliance will be designing for state usage scenarios before Tech Congress 2025. The primary missing pieces on the release candidate time frame:

Supporting all authorization models
Running on MSSQL
Running on PostgreSQL or MSSQL without Kafka and search database
ODS-like schema for reporting

Summit 2025 - production-ready version 1.0

Aiming for feature parity with the ODS/API from the perspective of API-based integrations, with one primary exception: no plan to support XML-based composites.

Feature List

Reviewing the list below:

Which features listed “By Summit” should we prioritize to try to release sooner?
Are there any features we failed to list?

Data Standard and API Standard Compatibility

Feature	ODS/API Platform	Data Management Service	By TC	By Summit
Resources API - core Ed-Fi Data Standard		DONE
Descriptor API - core Ed-Fi Data Standard		DONE
Discovery API		DONE
Data Standard version independence		MUST HAVE
MetaEd-based extensions		MUST HAVE
Dynamic Profiles		MUST HAVE
Multiple data standards in same deployment		COULD HAVE Does anyone want this?
XML Composites		WON'T HAVE

Data Integrity and Validation Features

Feature	ODS/API Platform	Data Management Service	By TC	By Summit
Level 0 and level 1 validation		DONE
Descriptor validation		DONE
Reference validation		DONE
Cascading updates on key changes		DONE
Level 2 validation via SQL scripts		MUST HAVE Once a compatibility layer is available, will be able to run the same SQL scripts used today for Level 2 validation
Realtime level 2 validations via streaming data		COULD HAVE Does anyone want this? Might be more of a demonstration than a core feature of the system

API Client Features

Feature	ODS/API Platform	Data Management Service
Concurrency management with ETags		MUST HAVE
LIMIT/OFFSET paging		DONE
Cursor-based paging	(7.3)	MUST HAVE

Database Technologies

Feature	ODS/API Platform	Data Management Service	By TC	By Summit
Core database storage in PostgreSQL		DONE The database structure is very different than the ODS database
Core database storage in MSSQL		MUST HAVE
GET queries using search database		DONE Either OpenSearch or Elasticsearch
GET queries using relational database		MUST HAVE Removes the requirement to run Kafka and OpenSearch or Elasticsearch
Reporting queries using ODS database schema		MUST HAVE This will likely be a compatibility layer to ease the transition for those who have built reporting solutions on the ODS database structure. Is this required for state-based pilot testing? Can it be finalized after the 1.0 release?
Core database storage in managed PostgreSQL-compatible databases (e.g. Aurora, Cosmos DB)		SHOULD HAVE Conceptually this should work, but we may need community help for testing these scenarios while the development team focuses on code-level features.
Realtime population of a data lake		SHOULD HAVE

Data Management Features

Feature	ODS/API Platform	Data Management Service	By TC	By Summit
Streaming data out via Kafka		DONE
Change Queries API		SHOULD HAVE Required for API Publisher synchronization; if not implemented must provide alternative.

Security Features

Feature	ODS/API Platform	Data Management Service
API-driven client credential management	(Admin API)	DONE
OAuth token authentication		DONE
Integration with third party OAuth identity providers		MUST HAVE At minimum, will support Keycloak, with clear path for supporting other providers
API-driven claimset management	(Admin API)	MUST HAVE
Namespace authorization		MUST HAVE
Relationship authorization		MUST HAVE
Ownership authorization		MUST HAVE
Combined authorization		MUST HAVE
Extensible authorization filtering	(7.3)	MUST HAVE

Deployment Management

Feature	ODS/API Platform	Data Management Service
Docker images and sample Docker Compose settings		DONE
Multitenancy routing and instance management		MUST HAVE
PowerShell installation scripts for Windows Server		COULD HAVE Does anyone want this?
Deployment orchestration via (Kubernetes, Terraform, Cloud Formation, ARM, etc.)	(though there are Exchange contributions)	WON'T HAVE Unless a community member contributes

Other Integrations

Feature	ODS/API Platform	Data Management Service	By TC	By Summit
SwaggerUI		MUST HAVE
Admin Console	coming soon	MUST HAVE

Demonstration

Working Client Management and Authentication

Mermaid diagram

sequenceDiagram
    actor Sys Admin

    rect rgb(191, 223, 255)
    note right of Sys Admin: One time setup.

        Sys Admin->>Config Service: POST /connect/register
        Config Service->>Identity Provider: Create credentials
        note right of Identity Provider: Created with config role
        
        Config Service-->>Sys Admin: clientCredentials
    end

    Sys Admin->>Config Service: POST /v2/vendors
    Config Service->>Config Database: INSERT dbo.Vendor

    Sys Admin->>Config Service: POST /v2/applications

    Config Service->>Identity Provider: Create credentials
    note right of Identity Provider: Created with dms role
    Identity Provider-->>Config Service: clientCredentials
    Config Service->>Config Database: INSERT dbo.Application
    Config Database -->>Config Service: applicationId
    Config Service->>Config Database: INSERT dbo.ApplicationEducationOrganization
    Config Service->>Config Database: INSERT dbo.ApiClient

    Config Service-->>Sys Admin: clientCredentials

Design Questions

What plans are there for error detection (i.e. dropped records) and correction?

A question that arose at the Ed-Fi Summit session. Suggestions?

How will year rollover be handled?

Perhaps not germane to the release candidate, but we should still ask: are there special considerations that the development team needs to be thinking about in advance?

December 2024 - Project Tanager Workgroup