Current status update

Design for datastore instance management

OpenSearch

Gather input to prioritize features and refine the application design, ensuring alignment with early adopter needs.

Provide an opportunity for workgroup members to actively contribute by participating in hands-on testing, helping to identify bugs, usability issues, and areas for improvement. Your feedback will directly shape the final product.

Collaborate to ensure the development achieves a stable release candidate and reaches a 1.0 release by Q1 next year.

Data isolation, scalability, and operational manageability across all layers—Database, Kafka, and OpenSearch.

Flexible API routing for tenants and DMS datastore instances (e.g., district/year), consistent with the behavior of the ODS API.

Physical separation of instance data at the DMS database level to support privacy and compliance.

Scalable database deployment, allowing databases to be distributed across multiple servers to support horizontal scaling in large environments.

Shared Kafka and OpenSearch clusters that can be horizontally scaled while maintaining strong instance-level data isolation.

Row-level data segregation (low isolation)

Schema-level data segregation (medium isolation)

Database-level segregation (high isolation)

Hybrid Approach ✅

Cluster-per-instance segregation (high isolation, high resource consumption)

Topic-per-instance segregation (medium isolation)✅

• Database Considerations (connector count)
  PostgreSQL: Debezium uses replication slots, which are scoped to a single database. Each database needs a separate connector.
  To reduce connectors, use schema-per-instance or row-level segregation.
  SQL Server: Debezium uses CDC tables, allowing changes from multiple databases on the same server. Connector count depends on the number of SQL Server instances.

• OpenSearch Considerations:

  Instance-specific topics may need to be consolidated before ingestion.
  Requires transformations to retain tenant/instance identifiers for logical segregation (e.g., filtered aliases).

Shared topic with tenant/instance filtering (technically possible, but least desirable)

Cluster-level segregation (high isolation, least feasible)

Index-per-instance segregation (will not scale on current indexing scheme)

Shared indexes with filtered aliases (low isolation)

Deep Pagination Limitations
The total-count and limit-offset pagination approach defined in the Ed-Fi API Specification performs well in OpenSearch only for datasets with fewer than 10,000 records.

Lack of Native Support for Data Isolation
No built-in mechanisms for physical data separation in multitenant environments. This increases the risk of data leakage if filtered aliases are not implemented and enforced correctly.

Current Index Structure
The current index-per-resource implementation results in a large number of small indexes, which is considered an anti-pattern in OpenSearch. Indexing requires a redesign to support large-scale deployments effectively.

Making the schema name mandatory in database segmentation is not ideal, as it increases the complexity of administering data stores for new tenants. It introduces unnecessary overhead in identifying the correct schema name, and all SQL scripts would need to be customized per tenant. (Feedback from EA)

Schema-level segmentation for PostgreSQL is being considered to optimize CDC connector configuration. The development team will investigate alternative options to reduce orchestration complexity.

When a tenant requests a reset of their data store, row-level segmentation makes it difficult to purge tenant-specific data. (Feedback from EA)

Row-level segmentation is commonly used in SaaS platforms due to its operational efficiency and cost savings. Consider supporting row-level segmentation as an optional strategy. (Feedback from EdGraph)

However, Ed-Fi prefers not to enable row-level segmentation due to:

• The need for strict application-level filtering.

• Increased risk of data leakage if not implemented correctly.

• Complexity in purging data for individual tenants.