Versions Compared

Version Old Version 1 New Version Current
Changes made by

Jason Hoekstra

Gary Clarke

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Revised per decisions in EDFI-394 comments

Prepared by Certica Solutions on 7/31/2020, Reviewed by Ed-Fi 8/14/2020

Epic for Work: 

Jira Legacy
serverEd-Fi Issue Tracker
serverIde04b01cb-fd08-30cd-a7d6-c8f664ef7691
keyEDFI-372

Table of Contents

Table of Contents

...

The outcome of this analysis is to move forward with API version as a significant entity, accepting some limitations in the multiple connection release, while using a design that supports future enhancements to increase the flexibility of Data Import when working with different versions of the ODS API.

Note: The earlier recommendation of using a natural key for the ApiVersions table is reversed in favor of any identity column. This minimizes impact to the data model when implementing future enhancements. 

Future Enhancements:

  1. Enhance the versioning concept to understand variations on standard Ed-Fi versions resulting from extensions.
  2. Enhance the import feature to make it easier to permit data maps targeting one version to be used with another compatible version (for example, to enable a 3.2 shared template to be imported and used with a 3.4 version).

Bootstrap Data Applicability

...

The existing configuration screen includes ODS API server connection details, template sharing service details, enable user management, and enable product improvement. We agree to move the ODS API server connection management to a separate screen but want to ensure that users are still presented with the remaining configuration options at first setup.

Note: It was also discussed that the existing behavior requires setup of the template sharing service (except potentially when running on a development environment). However, this does not seem to be the case.

The existing behavior is as follows:

  • Redirect to configuration screen after registering first user.
  • Redirect to configuration screen on attempt to access ODS API if no ODS API configured.
  • Redirect to configuration screen on attempt to access Sharing API if no Sharing API configuration.
  • Validate Sharing API key/secret has been supplied when saving configuration settings (not required for local development environment). 

Proposed approach:

  • The existing behavior will remain unchanged. 
  • The configuration screen will include a link to navigate to a separate screen to manage ODS API server connection details (“API Connections”).
  • The “API Connections” page will only be accessible from the Configuration page (i.e., there is no direct access menu item).

...

Deletion Behavior

Agent Deletion

  • No change to Agent deletion behavior – they will continue to be archived (logically deleted).
  • When an agent is archived, it is disassociated from the associated API connection and bootstrap data associations.

Data Map Deletion

  • No change to Map deletion behavior – they are physically deleted, and associated agent references are removed.

...

  • API connection will be physically deleted.
  • An API connection cannot be deleted if any agents reference it (must delete the agents first or change their connections).
  • Associated references to bootstrap data are removed when an API connection is deleted.

User Interface Changes

UI Feature

Modifications

Activity

  • Add API connection name to the activity list.
  • Add API connection selector to allow optional filtering of results.

Data Browser -> Assessments / Schools

  • Add API connection selector antecedent for browsing assessment or school data.

Agents

  • Add API connection name to the agent list.
  • Add API connection selector to the agent form.
  • Add section (similar to Data Maps) to allow addition of Bootstraps to the Agent.

Maps

  • Add API version selector to obtain resource metadata required for mapping.
  • Add API connection selector in modal to preserve descriptor search functionality.

Logs -> Files tab

  • Add API connection name to the file list.
  • Add API connection selector to allow optional filtering of results.

Lookups

  • Add API connection selector in modal to preserve descriptor search functionality.
Bootstrap
Data
Datas
  • Add
associated API versions to the bootstrap items list
  • API version selector antecedent for populating available resources.

Configuration

  • Replace global connection setup with link to manage API Connections.

API Connections

  • New screen supporting management of multiple API connections.
  • Live connection verification is required to save the connection. Saving the connection will set up API version and associated resources in database if it does not already exist.
  • Connection name must be unique.

Export

Add API version selector antecedent – data maps and bootstraps are filtered down to the version.
  • Require selected data maps have same API version.
  • Limit selection of bootstrap data definitions to those matching the data map(s) API version

Import

  • Use payload version number and import only if this data import installation knows about the version.

Template Sharing

  • Follows same behavior as export and import.
  • Revised ODS API Version optional filter to use values of Technology Suite 2 and Technology Suite 3 instead of 2.x and 3.x (verify compatibility with 5.x).


Data Model Changes

Table

Change

Justification

ApiServers

  • Add Name column with unique constraint.
  • Replace ApiVersion column with ApiVersionId column with FK.

To help differentiate between API connections, each is associated with a name.

Agents

  • Add ApiServerId column. Nullable to support disassociated on archive of agent.

An agent executes against an API connection.

IngestionLogs

  • Add ApiVersionId with FK to ApiVersions.
  • Replace AgentId column with AgentName.
  • Add ApiServerName column.

Denormalize columns to preserve history.

ApiVersions

  • New table, primary key ApiVersionId (identity column), attribute Version.

A master table is established to support ApiVersion as a first-class citizen (e.g. to populate dropdown list of versions in UI) and FK relationships associated with it.

Resources

  • Add ApiVersionId column with FK to ApiVersions.

Supports the persistence of resource metadata per API version.

DataMaps

  • Add ApiVersionId column with FK to ApiVersions.

Maps are version specific.

BootstrapDatas

  • Add ApiVersionId column with FK to ApiVersions.
  • Remove ProcessedDate column.
  • Remove ProcessingOrder column.

Bootstrap datas are version specific.

Migrate connection specific bootstrap information to a new table.

Processing order is now managed by agent instead of globally.

BootstrapDataAgents

  • New junction table.

Associate a bootstrap data item with an agent.

BootstrapDataApiConnections

BootstrapDataApiServers

  • New junction table.

Migrate ProcessedDate from BootstrapDatas to track if bootstrap data needs to be posted

to the API.

Alternate Design

Another design approach for adding multiple connection support is for Data Import to operate within a current connection context. This approach was originally taken by Certica for their enhancements to Data Flow.

Both design approaches address the requirement for multiple connections although how that requirement is satisfied is significantly different. 

...

to

...

the

...

Characteristic

This Design

Alternate Design

Big Picture

A user should be able to focus on tasks related to describing data import requirements holistically without focusing on specific ODS

API

instances that it will be processed against.

A user works in a tenant-specific context to create data import configuration for the current connection.

Data Map Management

A data map is defined in the context of a specified Ed-Fi version and can be applied to compatible agents.

Data maps are either marked as shared, or must be exported to another tenant, which creates a copy for that tenant’s context.

Optimized For

Standardized file formats likely to be shared by tenants.

Customers with varied or individualized file import requirements.

Concerns

Some unexpected potential tenant-specific dependencies may limit the applicability of the broad concept (e.g. descriptor search and ODS extensibility).

User experience concerns with having to keep selecting a connection as they navigate through Data Import.

Updating a data map or bootstrap data requires updating each tenant.

Forces user to think in terms of everything being connection specific.

Implementation Effort

Lots of changes to existing behavior of Data Import expand the work effort, especially to address new UI concepts.

Limited changes to existing behavior of Data Import result in a small work effort to implement

.