Versions Compared

Version Old Version 3 New Version Current
Changes made by

Stephen Fuqua

Stephen Fuqua

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Participants

Expand

  • Denver Public Schools - Max Paulson

  • Ed-Fi staff & contractors

    • Adam Hopkins

    • Brad Banister

    • Robert Hunter

    • Sayee Srinivasan

    • Stephen Fuqua

    • Steven Arnold

    • Vinaya Mayya

  • Edufied - Geoff McElhanon

  • EdWise Group - Britto Augustine

  • Resultant - Joshua Impson

  • Simpat Tech - Ashish Patel

  • Utah State Board of Education - Katrina Brinkley

Agenda

  • Review the roadmap

  • Demonstration of current work

  • Review the roadmap

  • Design Questions

...

Roadmap

Planned Architecture

...

Working Client Management and Authentication

...

Roadmap

Getting to a pilot-ready release candidate by Tech Congress was always going to be challenging. The team has been working hard, but we need to adjust the milestones and expectations for release candidate 1.

Goals for release candidate are broadly:

  1. Deployable and fully functional in the basic CRUD operations on all Resources and Descriptors.

  2. Proper error handling and edge case management.

  3. Performance on par with the ODS/API.

  4. Be useable for basic field testing and non-critical path use cases.

While we want to be looking ahead to supporting state agency use, at the 2024 Summit we agreed to prioritize service provider / LEA usage scenarios first. Given that,

...

Milestone

...

Functional Goals

...

Basic timeline

  1. Tech Congress 2025 - release candidate with “basic” feature set useable for pilot testing typical data exchange scenarios.

Info

Focused primarily on LEA and vendor-to-vendor scenarios.

State scenarios are critical for this project. The Alliance will be designing for state usage scenarios before Tech Congress 2025. The primary missing pieces on the release candidate time frame:

  • Supporting all authorization models

  • Running on MSSQL

  • Running on PostgreSQL or MSSQL without Kafka and search database

  • ODS-like schema for reporting

  1. Summit 2025 - production-ready version 1.0

Info

Aiming for feature parity with the ODS/API from the perspective of API-based integrations, with one primary exception: no plan to support XML-based composites.

Feature List

Note

Reviewing the list below:

  1. Which features listed “By Summit” should we prioritize to try to release sooner?

  2. Are there any features we failed to list?

Tip

Meeting notes:

  • Data Standard support

    • Prioritize 5.2 for Tech Congress; many states are moving that direction right now.

    • Nice to have support for 4.0 as well, for pilot testing with vendors who have not updated to 5.x yet.

    • 6.0 preview is not useful from a pilot testing standpoint, since no one has an integration yet.

    • In 1.0, MSP’s would like to have support for multiple data standards at the same time: a single API deployment that can store both 5.2 and 4.0 data structures, with validation based on URL route. Allows for better cost optimization.

      • (warning) Parking lot: design discussion on mechanisms for supporting this. Initial proposal to prefer separate database instances for different Data Standard versions. Maybe tie DS version into the URL routing for instances?

    • Extensions should be the next priority after authorization; would like to have by Tech Congress if possible.

  • Profiles

    • Remove “dynamic” from the list to avoid confusion (same with “XML” on “XML Composites”).

    • Agree: must have for 1.0 and not mandatory for Release Candidate at Tech Congress.

  • Level 2 validation

    • Need to explain both types better when turning the table below into a formal document.

    • For streaming data, mention observability as well.

  • ETags

    • Some client integration may be looking for _etag in the body or the ETag header on POST requests.

    • For release candidate, should at least stub out a hard-coded value to prevent breaking client integrations.

      • (warning) Design note: we plan to calculate ETags based on hashing the _lastModifiedDate. This is relatively trivial. We should simply consider ETags as a must-have feature for the spring release candidate.

  • ODS-like Views

    • This discussion helped us recognize the value of talking about (at least) two stages of pilot testing: transactional vendor integration (via the API), and downstream reporting.

    • ODS-like views are intended to support the second stage of pilot testing. States could start testing a 1.0 release without the ODS-like views. Can potentially defer until after 1.0.

  • Core database storage

    • Correction, Education Analytics is today using Aurora for the ODS/API.

  • Change Queries

    • Current usage scenarios:

      • API-to-API synchronization via API Publisher

      • Assessment rostering

      • Enable Data Union (EDU) data warehousing.

    • We should consider this a must have solution, even if the streaming architecture can provide alternative synchronization patterns. Too many integrations would break without this.

  • Authorization

    • Ownership-based authorization and combined strategies are used in multiple states. Definitely a must have for 1.0, but can live without them for the release candidate.

Data Standard and API Standard Compatibility

0.2

Status
colourGreen
titleDONE

  • 📢 Milestone 0.2.0 has been reached!

  • .NET application with PostgreSQL storage

  • Level 0 and Level 1 document validation

  • Reference and descriptor validation

  • Error message like ODS/API 7.2

  • Docker and Kubernetes

0.3

Feature

ODS/API Platform

Data Management Service

By TC

By Summit

Resources API - core Ed-Fi Data Standard

(tick)

Status
colourGreen
titledone

Compliant Discovery API, Descriptor API, and Resource API definition (except GET by query): able to run bulk upload, smoke test. Includes JSON validation based on API schema file. Fake OAuth (1).

(tick)

Descriptor API - core Ed-Fi Data Standard

(tick)

Status
colourGreen
titledone

(tick)

Discovery API

(tick)

Status
colourGreen
titledone

(tick)

Data Standard version independence

(tick)

Status
colourBlue
titleMUST HAVE

(tick)

MetaEd-based extensions

(tick)

Status
colourBlue
titleMUST HAVE

(question)

(tick)

Dynamic Profiles

(tick)

Status
colourBlue
titlemust have

(question)

(tick)

Multiple data standards in same deployment

(error)

Status
colourYellow
titleCould Have

Note

Does anyone want this?

(question)

Composites

(tick)

Status
colourRed
titleWOn't Have

(error)

Data Integrity and Validation Features

Feature

ODS/API Platform

Data Management Service

By TC

By Summit

Level 0 and level 1 validation

(tick)

Status
colourGreen
titledone

(tick)

Descriptor validation

(tick)

Status
colourGreen
titledone

(tick)

Reference validation

(tick)

Status
colourGreen
titledone

(tick)

Cascading updates on key changes

(tick)

Status
colourGreen
titledone

(tick)

Level 2 validation via SQL scripts

(tick)

Status
colourBlue
titleMUST HAVE

Once a compatibility layer is available, will be able to run the same SQL scripts used today for Level 2 validation

(question)

Realtime level 2 validations via streaming data

(error)

Status
colourYellow
titleCould Have

Note

Does anyone want this? Might be more of a demonstration than a core feature of the system

(question)

API Client Features

Feature

ODS/API Platform

Data Management Service

By TC

By Summit

Concurrency management with ETags

(tick)

Status
colourBlue
titleMUST HAVE

(question)

(tick)

LIMIT/OFFSET paging

(tick)

Status
colourGreen
titledone

(tick)

Cursor-based paging

(tick) (7.3)

Status
colourBlue
titleMUST HAVE

(question)

Identification code-based queries

(tick) (7.3)

Status
colourBlue
titleMUST HAVE

(question)

Identities API

(tick)

Status
colourYellow
titlecould have

Note

Need to review field usage and fitness-for-purpose before committing.

(question)

Unique ID System Integration

(tick)

Status
colourYellow
titlecould have

Note

Need to review field usage and fitness-for-purpose before committing.

(question)

Database Technologies

0.4

Status
colourBlue
titlein progress

Token authorization

Feature

ODS/API Platform

Data Management Service

By TC

By Summit

Core database storage in PostgreSQL

(tick)

Status
colourGreen
titledone

The database structure is very different than the ODS database

(tick)

Core database storage in MSSQL

(tick)

Status
colourBlue
titleMUST HAVE

(tick)

GET queries using search database

(error)

Status
colourGreen
title

DONE

  • Streaming data out via Kafka

  • GET by query using OpenSearch

  • Cascading updates on allowed key changes

  • Abandoned direct Kubernetes support as too costly

  • by 10/1 (Ed-Fi Summit)

done

Either OpenSearch or Elasticsearch

(tick)

GET queries using relational database

(tick)

Status
colourBlue
titleMUST HAVE

Removes the requirement to run Kafka and OpenSearch or Elasticsearch

(tick)

Reporting queries using ODS database schema

(tick)

Status
colourBlue
titleMUST HAVE

Note

This will likely be a compatibility layer to ease the transition for those who have built reporting solutions on the ODS database structure.

Is this required for state-based pilot testing? Can it be finalized after the 1.0 release?

(question)

Core database storage in managed PostgreSQL-compatible databases (e.g. Aurora, Cosmos DB)

(error)

Status
colourPurple
titleSHOULD Have

Conceptually this should work, but we may need community help for testing these scenarios while the development team focuses on code-level features.

(question)

Realtime population of a data lake

(error)

Status
colourPurple
titleSHOULD Have

(question)

Redis-based caching

(tick)

Status
colourPurple
titleSHOULD Have

(question)

Data Management Features

Feature

ODS/API Platform

Data Management Service

By TC

By Summit

Streaming data out via Kafka

(error)

Status
colourGreen
title

DONEClient credentials management

done

(tick)

Change Queries API

(tick)

Status
colourPurple
titleSHOULD Have

Required for API Publisher synchronization; if not implemented must provide alternative.

(question)

Security Features

Namespace authorization

Feature

ODS/API Platform

Data Management Service

By TC

By Summit

API-driven client credential management

(tick) (Admin API)

Status
colourGreen
title

DONE

done

(tick)

OAuth token authentication

(tick)

Status
colour

Yellow

Green
title

nextBONUS: Elasticsearch/Kibana instead of OpenSearch/OpenSearch Dashboard

done

(tick)

Integration with third party OAuth identity providers

(error)

Status
colour

Green

Blue
title

DONE

0.5

Design spikes around ed-org authorization

MUST HAVE

At minimum, will support Keycloak, with clear path for supporting other providers

(tick)

API-driven claimset management

(tick) (Admin API)

Status
colourBlue
titleMUST HAVE

(tick)

Namespace authorization

(tick)

Status
colour

Green

Blue
title

DONEConcurrency management with eTags

MUST HAVE

(tick)

Relationship authorization

(tick)

Status
colour

Red

Blue
title

DEFER?Extensions

MUST HAVE

(tick)

Ownership authorization

(tick)

Status
colour

Red

Blue
title

DEFER?Multiple data standards

MUST HAVE

(question)

(tick)

Combined authorization

(tick)

Status
colour

Red

Blue
title

DEFER? (DS 5.2)

0.6

Dynamic profiles

MUST HAVE

(question)

(tick)

Extensible authorization filtering

(tick) (7.3)

Status
colourBlue
titleMUST HAVE

(question)

Deployment Management

Feature

ODS/API Platform

Data Management Service

By TC

By Summit

Docker images and sample Docker Compose settings

(tick)

Status
colour

Red

Green
titledone

DEFER?

(tick)

Multitenancy

and routing

routing and instance management

(tick)

Status
colourBlue
titleMUST HAVE

(question)

(tick)

PowerShell installation scripts for Windows Server

(tick)

Status
colour

Red

Yellow
title

DEFER?

0.7

Education organization authorization

Could Have

Note

Does anyone want this?

(question)

Deployment orchestration via (Kubernetes, Terraform, Cloud Formation, ARM, etc.)

(error)

(though there are Exchange contributions)

Status
colourRed
titleWOn't Have

Unless a community member contributes

(error)

Other Integrations

...

Feature

ODS/API Platform

Data Management Service

By TC

By Summit

SwaggerUI

(tick)

Status
colour

Purple

Blue
title

Move upClaimset customization

must have

(tick)

Admin Console

(warning) coming soon

Status
colour

Red

Blue
title

DEFER?

Items for future scope beyond this include: Change Queries, using PostgreSQL for queries instead of OpenSearch / Elasticsearch, real-time level 2 validation using Kafka, identity API, etc.

must have

(question)

(tick)

Demonstration

Tip

Meeting notes:

Ran out of time for the demonstration. The sequence diagram below shows the interactions for client credential management.

The demonstration was going to use this file which uses the REST Client extension for Visual Studio Code. Even without VS Code, the file should be easy to follow along with while replicating the API calls in another client such as Postman.

Working Client Management and Authentication

Expand
titleMermaid diagram
Code Block
sequenceDiagram
    actor Sys Admin

    rect rgb(191, 223, 255)
    note right of Sys Admin: One time setup.

        Sys Admin->>Config Service: POST /connect/register
        Config Service->>Identity Provider: Create credentials
        note right of Identity Provider: Created with config role
        
        Config Service-->>Sys Admin: clientCredentials
    end

    Sys Admin->>Config Service: POST /v2/vendors
    Config Service->>Config Database: INSERT dbo.Vendor

    Sys Admin->>Config Service: POST /v2/applications

    Config Service->>Identity Provider: Create credentials
    note right of Identity Provider: Created with dms role
    Identity Provider-->>Config Service: clientCredentials
    Config Service->>Config Database: INSERT dbo.Application
    Config Database -->>Config Service: applicationId
    Config Service->>Config Database: INSERT dbo.ApplicationEducationOrganization
    Config Service->>Config Database: INSERT dbo.ApiClient

    Config Service-->>Sys Admin: clientCredentials

...

Design Questions

Tip

Meeting notes:

Ran out of time. Will keep these design questions for a future meeting.

What plans are there for error detection (i.e. dropped records) and correction?

...