Versions Compared

Old Version 3

changes.mady.by.user Vinaya Mayya

Saved on

compared with

New Version Current

changes.mady.by.user Stephen Fuqua

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • Demonstration of current work

  • Review the roadmap

  • Design Questions

Attendees: …Max Paulson (Denver Public Schools), JF Guertin (EdGraph), Joshua Impson (Resultant), Erik Joranlien (Education Analytics), Doug Loyo (Edufied), Geoff McElhanon (Edufied), Adam Hopkins (Simpat Tech on behalf of Ed-Fi), Brad Banister (Doubleline Partners on behalf of Ed-Fi), Vinaya Mayya (Ed-Fi Alliance), Stephen Fuqua (Ed-Fi Alliance).

Demonstration

image-20240828-161611.png

Meeting notes:

Roadmap

Milestone 0.4

...

Milestone

...

Functional Goals

...

0.1

Status
colourGreen
titledone

...

Compliant Discovery API, Descriptor API, and Resource API definition (except GET by query): able to run bulk upload, smoke test. Includes JSON validation based on API schema file. Fake OAuth (1).

...

  • Working demonstration of:

    • POST multiple resources

    • See data flowing into Kafka and OpenSearch

    • GET ALL and GET by query examples, pulling from OpenSearch

  • We think this will work with Elasticsearch, but have not tested yet.

  • Recommendation: consider adding advanced query capabilities found in Elasticsearch/OpenSearch.

Roadmap

Milestone 0.4

Milestone

Functional Goals

0.1

Status
colourGreen
titleDONE

📢 Milestone 0.2.0 has

done

Compliant Discovery API, Descriptor API, and Resource API definition (except GET by query): able to run bulk upload, smoke test. Includes JSON validation based on API schema file. Fake OAuth (1).

0.2

Status
colourGreen
titleDONE

  • 📢 Milestone 0.2.0 has been reached!

  • .NET application with PostgreSQL storage

  • Level 0 and Level 1 document validation

  • Reference and descriptor validation

  • Error message like ODS/API 7.2

  • Docker and Kubernetes

0.3

Status
colourBlue
titlein progress

  • Streaming data out via Kafka

  • GET by query using OpenSearch

  • Cascading updates on allowed key changes

  • Abandoned direct Kubernetes support as too costly

  • by 10/1 (Ed-Fi Summit)

0.4

Status
colourRed
titleneed to accelerate

  • Token authorization

  • Client credentials management

  • Namespace authorization

  • Maybe by mid-October?

0.5

  • Design spikes around ed-org authorization

  • Concurrency management with eTags

  • Extensions

  • Multiple data standards

  • Maybe by mid-December?

0.6

  • Dynamic profiles

  • Multitenancy and routing

  • Maybe by mid-February?

0.7

  • Education organization authorization

  • Claimset customization

  • By Tech Congress?

Meeting notes:

Architecture

Storage of Metadata

This is a sample message in Kafka, created from Debezium:

...

  • Kubernetes: there is strong interest, but it does not make sense for the Ed-Fi development team to continue learning Kubernetes.

  • JF may pursue more about Kubernetes setup.

  • Recommendation: place production deployment utilities in a separate repository.

Architecture

Storage of Metadata

This is a sample message in Kafka, created from Debezium:

...

  • _lastModifiedDate, _etag (and _lineage in the future?)

  • (question) Store as metadata or add to the document itself? What should downstream consumers receive?

  • Already sending “API Standard” rather than “Data Standard”. Example: “SchoolYearType” instead of “SchoolYear”.

Meeting notes:

  • Today, anyone using the Ed-Fi API to populate a data lake is getting the metadata directly in the retrieved documents.

  • With the structure above, they will need to extract the edfidoc when saving out to a data lake.

  • Conclusion: least amount of rework if we put _lastModifiedDateinto the edfidoc.

Tenancy Routing

GitHub Discussion for asynchronous conversation: Multitenancy / routing / instance management

...

  • Allow ODS instances to reside on separate database servers.

  • Support various database segmentation strategies: ODS per year, ODS per district, ODS per district per year, and ODS per tenant. (Reference: Segmentation strategies in ODS API)

  • Enable API administrators to choose between implicit routes (segmentation information not visible in the routes) and explicit routes (Reference: Context-Based Routing).

  • Provide isolated configuration for client and security setup by tenants, making it easier to onboard new tenants or remove existing ones.

Meeting notes:

  • The ODS/API approach is working right now. Follow the same requirements and keep the interface as similar as possible to ease the transition to the Data Management Service.

Authentication

Given that client credentials will ultimately be in a 3rd party OAuth provider… do we want to give API clients the 3rd party OAuth token URL, or put it behind a proxy? If behind a proxy, three possible options:

  • Data Management Service

  • DMS Configuration Service

  • or API gateway / load balancer

...

Meeting notes:

  • Many client applications aren’t using the Discovery API, but people are beginning to learn about it.

  • If proxying to solve client integration problems, then it needs to be the same base address as the API (not the Configuration Service base address).

  • Between now and Tech Congress, this is probably a low priority unless and until something pushes us to see it otherwise.

    • Prefer solving at API gateway / load balancer.

Logging

Meeting notes:

  • Did not review in great detail.

  • Being able to search for the TraceId is the most important thing.

    • Need to include it in the Kafka messages.

  • We should try to make structured logging easily configurable.

  • We can return the telemetry vs. system log discussion in the future.

Showing output from docker logs:

...