These pages relate to the all things pub/sub, i.e. change data capture, streaming, and materialized views.

• Meadowlark - Durable Change Data Capture
• Meadowlark - Materialized Views
• Meadowlark - Streaming to Filesystem

Overview

Meadowlark will use Debezium for Change Data Capture from the source system databases (MongoDB, PostgreSQL), loading data into a streaming platform (initially Kafka). From there, different stream processing techniques can be used to read events from the stream, apply business logic, and write new events or write to other data stores.

There will be two broad categories of messages landing in the stream:

• "Instructional" events that tell another system about an action that needs to be taken, and
• "Document" events that copy the documents received to the stream for distribution to other data sources.

Initial Use Cases

1. Populating OpenSearch for API queries
2. Cascading Updates
3. Blob storage
4. Materialized views to support domain-specific API's; ex: assessment roster API