Introduction

Project Meadowlark is a proof-of-concept implementation of the Ed-Fi API Specification, currently supporting Data Standard 3.3b, built on managed services provided by AWS. This document describes the system architecture, including: managed service infrastructure and flow; frameworks used in programming the solution; and notes about potential future direction.

→ More information on  Meadowlark

Cloud Managed Services

The big three cloud providers (Amazon, Google, Microsoft) all provide similar managed services that could have been used to build this application. The choice of Amazon Web Services (AWS) is not an endorsement of Amazon per se. Rather, the development team needed to commit to one service in order to remain focused on delivering a usable proof-of-concept without over-engineering up-front. Further development of Meadowlark into a product would require additional effort to ensure that the core software can easily be used on any cloud platform that provides similar managed service capabilities.

→ More information on cloud parity

Infrastructure

The following diagram illustrates the managed service infrastructure utilized by Meadowlark.

What does each of these services provide?

• API Gateway is  front-end web server that acts as a proxy back to the separate Lambda functions. With the help of the API Gateway, client applications need know only a single base URL, and the different resource endpoints can opaquely point back to different back-end services.
• Lambda Functions are small, purpose-built, serverless runtime hosts for application code. In the Meadowlark solution, there are ten different Lambda Functions that handle inbound requests from the API Gateway. For simplicity, only a single icon represents all ten in the diagram above.
• DynamoDB is a high-performance NoSQL database for key-value storage. One of the powerful features of DynamoDB is its Change Data Capture Streaming: each change to an item stored in the database creates an event on a stream. Another Lambda function detects this event to provide post-processing.
• OpenSearch is a NoSQL database based on ElasticSearch, providing high-performance indexing and querying capabilities. All of the "GET by query" (aka "GET by example") client requests are served by this powerful search engine.
• CloudWatch provides advanced collection and monitoring capabilities for logs, including detailed logging written in to the Meadowlark Lambda functions.

Database Choices

In traditional application development, including the Ed-Fi ODS/API Platform, all Create-Read-Update-Delete (CRUD) operations are serviced by a single database server. Project Meadowlark has opted to adopt the strategy of choosing a database that is fit-to-purpose. DynamoDB is incredibly fast and highly scalable for online transaction processing (OLTP), allowing the web API layer to respond to the client very quickly. As a key-value store, a DynamoDB table contains only a small number of columns, and the raw JSON payload received in POST and PUT requests are stored directly in a column. This improves both the write speed and the speed of retrieving a single object from the database. 

A key difference between this document storage approach, compared to relational database modeling, comes in the form of searchability. DynamoDB has the ability to add "secondary indexes" that can help find individual items by some global criteria. But these are limited and very different than the indexes found in a relational database, which can be tuned to identify items based on any column. In other words, when storing an entire document, Dynamo DB is a poor choice for trying to search by query terms (e.g. "get all students with last name Doe"). 

But this is where OpenSearch shines. Based on ElasticSearch, OpenSearch is also a NoSQL document store. The key difference is that it indexes everything in the document, and has a powerful search engine across the indexes. 

As a managed service, there is no need to worry about backups

Data Flow