Summit 2024 - Project Tanager Workgroup
- Stephen Fuqua
Attendance
No attendance taken in the packed session.
Presentation
Additional Notes
Several important questions came up around the OpenSearch integration, which will need to be addressed:
What kind of latency can we expect with the flow of data from PostgreSQL to Kafka to OpenSearch?
What plans are there for error detection (i.e. dropped records) and correction?
Can OpenSearch be used to support AI workloads over Ed-Fi data?
The Alliance has not looked into this. Potentially useful articles on the topic:
OpenSearch AI and machine learning: Recapping 2023 and what’s next · OpenSearch
OpenSearch adds a new generative AI assistant toolkit · OpenSearch
Power neural search with AI/ML connectors in Amazon OpenSearch Service | AWS Big Data Blog
We also anticipate that the new Ed-Fi Data Management Service will be able to work with Elasticsearch instead of OpenSearch (they are close cousins).
Any concern about eventual consistency?
The only people who raised their hands are from states, which is understandable and not surprising. This is why we plan to have a simpler model running purely from PostgreSQL or MSSQL, without OpenSearch/Elasticsearch.
Other topics:
How will year rollover be handled?
Suggestion: for the spring release candidate, focus on LEA use cases rather than SEA, working closely with the managed service provider community.