Project Meadowlark SIG - June 23, 2022

Attendees

Support

Ann Su - Ed-Fi Governance Support

The meeting was held on 2022-06-23  2:00pm - 3:15pm CT via WebEx

Meeting

Agenda/Notes        

• Introduction

  • SIG members comprise of MSP, LEA, SEA

• Brief recap of Project Meadowlark, its aims, and 2022 milestones

  • R&D effort to understand what it would look like to develop modular Ed-Fi stacks going towards more optimization at cloud-hosting (see diagram in PPT)

  • Prior release at Tech Congress: 0.1.0

  • Current: 0.2.0 milestone

    • Introduced Fastify, MongoDB, and PostgreSQL

    • Demo code

  • Risk: must avoid confusing vendors through API fragmentation

  • Upcoming: 0.3.0 milestone

    • Fixing API model problems

    • Remove DynamoDB code?

  • Upcoming: 0.4.0 milestone

    • Introduce authentication

  • Current Priorities

• Feedback on the research questions and priorities

  • Document store instead of relational model

  • Perceived complexity of the architecture

• Discussion: envisioning a production-ready system

  • Minimum viable product

    • Pilot Projects volunteers

      • Install & have vendor data flow thru in a few months for school year 22-23

        • SEA - State of Vermont (VT)

        • LEA - ??

        • MSP - EdWire and Instructure in 23-24

  • Requirements the dev team might overlook

    • Instructure - API service available on legacy ODS to pilot or to test

      • Stephen Fuqua (SF)- Large effort, need to do further investigation

      • VT - question re: analytics middle tier

      • EA - does a move to Docker invalidate the advantages of serverless?

        • SF: trying to provide both options

      • Microsoft - can it provide a better API for change queries?

  • VT - How define API

    • SF - Define by Meta-ed files

    • Brad Banister (BB) - different from ODS API; running first few MetaEd, drive API shape to apply validation code; able to see the references. No code generation required.

    • Moving to NoSQL data store

  • VT - you should do a presentation on the DB model differences, because it's hard wrapping my head around it

  • Instructure - are you using the Serverless framework?

    • SF - yes, and will soon have a new team member who can help us do more automation around its use.

  • Sayee

    • SEA use case focus on reporting

    • Data lake structure

  • Marcos

    • In a Google implementation, can use Google Pub-Sub instead of Kafka and drop the data out into a data lake or Big Query

  • Unicon - one question as you are thinking through priorities. Seems like there is opportunity to lay the foundation for the cloud centric architecture to support expanded use cases like multi-tenancy, single store, reusable models and more.  Perhaps those goals could be factored in as you establish the next set of priorities.

    • Marcos - Similar to what @Linda said. Straight from Eric's mouth: Meadowlark is a R&D project that is not for individual DIY LEAs. It is for managed service providers and SEAs. Now my opinion is for LEAs we want to prioritize talking about running the existing API on containers, managed databases, etc etc

  • Instructure

    • Materialized view - perhaps we can forward (multicast) HTTP requests to Meadowlark to get production data into materialized view

  • SF

    • Ownership Authorization or Ed-Org Authorization

      • EA - how claim sets and access to specific resources would work here

      • EdWise - Ed-Org auth is a need for SEAs, so they can use that for exchanging appropriate data within the org hierarchy; LEA may send the data, but the ESC who is above the LEA may need access to read the data

      • VT - ownership pattern described should work; access pattern; state want to offer to lea to integrate with vendors

      • Instructure - At Instructure we generate a separate ODS for each LEA

      • Marcos - Kind of wish that Ed-Fi API in the diagram was just replaced with Kafka. Hmm…

  • Marcos - Keeping the API is causing a more complex architecture to keep vendor code as is. That is a high cost.

  • Lake Washington - don’t know where we fit in, not sure what we can contribute

  • Instructure - Apache spark? Maybe an easy option for querying from Kafka.

  • JVT - My 2 cents, I'd rather see a ORM used that supports a lot of db options, than you focus on single api/schema architectures. Organizations have licensing and knowledge constraints that drive adoption. Is it possible to use the Meadowlark JavaScript code against the existing relational database structure?

  • Unicon- so if the data model were fixed, it could have an alternate (non REST API) async binding using something like a kafka topic?

  • Unicon - like where this is going so far - I'll check to see if Unicon might be able to help as test pilots

• Next steps:

  • Meeting again to discuss

    • Consider proposed model or a simpler model would work

  • Anyone want to pitch in on developing one or more components?

Action Items

Next meeting: TBD