/
Data Import Multi-Connection Design

Data Import Multi-Connection Design

Prepared by Certica Solutions on 7/31/2020, Reviewed by Ed-Fi 8/14/2020

Epic for Work:  EDFI-372 - Getting issue details... STATUS

Table of Contents

Introduction

Before multiple-connection awareness, a single global connection was implicitly referenced throughout the solution. Our efforts so far to implement support for multiple API connection have revealed design considerations that were not clear when we initially started on this initiative. Additional dialog has led to a refinement of the design so that concerns and challenges are addressed. This document describes the outcome of this effort.

Behavior Changes

This section describes functionality that is not clear from the perspective of the data model and UI changes, or calls out significant behavior decisions for multiple connection support that are motivators for the proposed data model and UI changes.

API Versions

With the introduction of support for multiple connections, there is a requirement to store distinct resource metadata per version of API connections encountered.

We explored whether to introduce API version as a strong identifier in Data Import or leave it purely as a visual indicator. We also considered whether to model this with its own master table, and whether to identity the primary key of the table with an identity column or a natural key. Furthermore, we reviewed the impact of associating API resource metadata with the API version or the API connection. 

The outcome of this analysis is to move forward with API version as a significant entity, accepting some limitations in the multiple connection release, while using a design that supports future enhancements to increase the flexibility of Data Import when working with different versions of the ODS API.

Note: The earlier recommendation of using a natural key for the ApiVersions table is reversed in favor of any identity column. This minimizes impact to the data model when implementing future enhancements. 

Future Enhancements:

  1. Enhance the versioning concept to understand variations on standard Ed-Fi versions resulting from extensions.
  2. Enhance the import feature to make it easier to permit data maps targeting one version to be used with another compatible version (for example, to enable a 3.2 shared template to be imported and used with a 3.4 version).

Bootstrap Data Applicability

How bootstrap data fits into a multiple connection design has been subject to some discussion. Two distinct contexts must be considered:

  1. Is the bootstrap data applicable to the execution of the agent?
    • We considered that it can always be treated as applicable regardless of API version. If it does not send successfully, then we handle that as an error preventing execution of the agent. However, this potentially creates an unresolvable situation in which bootstrap data design for one API version is incompatible when posting to a different API version.
    • A logical solution to this problem is to associate bootstrap data with API versions. However, we already tried something like that (for API connections instead of API versions) and it resulted in an uncompelling user experience.
    • A better option is to manage applicable bootstrap data in the context of defining the agent. This mirrors the approach already used for associating data maps with an agent.
    • This is preferred because it avoids trying to manage a compatibility matrix of bootstrap data items and API versions. Instead, applicability is determined solely within the context of where it will be applied.
  2. Does the bootstrap data need to be sent to the ODS API before the execution of the data maps against the file?
    • The existing semantics to automatically send every bootstrap data item as the first step of executing an agent if the bootstrap data had not previously been sent to the connection or the bootstrap data was changed since it was sent.
    • This is easily extended to the concept of multiple connections by tracking the processed data per API connection. Even better, by deciding to associate bootstrap data items to agents, we eliminate the requirement to process bootstrap data that is not related to the responsibilities of the agent.

Access to Configuration Screen

The existing configuration screen includes ODS API server connection details, template sharing service details, enable user management, and enable product improvement. We agree to move the ODS API server connection management to a separate screen but want to ensure that users are still presented with the remaining configuration options at first setup.

The existing behavior is as follows:

  • Redirect to configuration screen after registering first user.
  • Redirect to configuration screen on attempt to access ODS API if no ODS API configured.
  • Redirect to configuration screen on attempt to access Sharing API if no Sharing API configuration.
  • Validate Sharing API key/secret has been supplied when saving configuration settings (not required for local development environment). 

Proposed approach:

  • The existing behavior will remain unchanged. 
  • The configuration screen will include a link to navigate to a separate screen to manage ODS API server connection details (“API Connections”).
  • The “API Connections” page will only be accessible from the Configuration page (i.e., there is no direct access menu item).

Menu Changes

There is an annoying issue with the nav bar wrapping on smaller resolutions and causing page content to be hidden:

An alternate menu layout was introduced by Certica on the first PR and received favorable feedback in the Headspring review. While this does not technically need to be addressed in the context of multi-connection support, it is bothersome enough that we would like to apply the new design in the context of preparing a second PR.

The new design introduces a Configure dropdown:

Also, the “Hello <email>” on the far right is converted to a dropdown containing “Manage” and “Log Off”.

Collectively, these changes address the nav bar wrapping issue. As a bonus, with Bootstrap Data now being assigned to agents, it is also preferred to include it in the “Configure” menu rather than the “Admin” menu where it currently resides. 

Deletion Behavior

Agent Deletion

  • Agent will continue to be archived (logically deleted).
  • When an agent is archived, it is disassociated from the associated API connection and bootstrap data associations.

Data Map Deletion

  • No change to Map deletion behavior – they are physically deleted, and associated agent references are removed.

Bootstrap Data Deletion

  • Bootstrap data is physically deleted.
  • Deletion will now also require removal of associated API connection and agent references.

API Connection Deletion

  • API connection will be physically deleted.
  • An API connection cannot be deleted if any agents reference it (must delete the agents first or change their connections).
  • Associated references to bootstrap data are removed when an API connection is deleted.

User Interface Changes

UI Feature

Modifications

Activity

  • Add API connection name to the activity list.
  • Add API connection selector to allow optional filtering of results.

Data Browser -> Assessments / Schools

  • Add API connection selector antecedent for browsing assessment or school data.

Agents

  • Add API connection name to the agent list.
  • Add API connection selector to the agent form.
  • Add section (similar to Data Maps) to allow addition of Bootstraps to the Agent.

Maps

  • Add API version selector to obtain resource metadata required for mapping.
  • Add API connection selector in modal to preserve descriptor search functionality.

Logs -> Files tab

  • Add API connection name to the file list.
  • Add API connection selector to allow optional filtering of results.

Lookups

  • Add API connection selector in modal to preserve descriptor search functionality.
Bootstrap Datas
  • Add API version selector antecedent for populating available resources.

Configuration

  • Replace global connection setup with link to manage API Connections.

API Connections

  • New screen supporting management of multiple API connections.
  • Live connection verification is required to save the connection. Saving the connection will set up API version and associated resources in database if it does not already exist.
  • Connection name must be unique.

Export

  • Require selected data maps have same API version.
  • Limit selection of bootstrap data definitions to those matching the data map(s) API version

Import

  • Use payload version number and import only if this data import installation knows about the version.

Template Sharing

  • Follows same behavior as export and import.
  • Revised ODS API Version optional filter to use values of Technology Suite 2 and Technology Suite 3 instead of 2.x and 3.x (verify compatibility with 5.x).


Data Model Changes

Table

Change

Justification

ApiServers

  • Add Name column with unique constraint.
  • Replace ApiVersion column with ApiVersionId column with FK.

To help differentiate between API connections, each is associated with a name.

Agents

  • Add ApiServerId column. Nullable to support disassociated on archive of agent.

An agent executes against an API connection.

IngestionLogs

  • Add ApiVersionId with FK to ApiVersions.
  • Replace AgentId column with AgentName.
  • Add ApiServerName column.

Denormalize columns to preserve history.

ApiVersions

  • New table, primary key ApiVersionId (identity column), attribute Version.

A master table is established to support ApiVersion as a first-class citizen (e.g. to populate dropdown list of versions in UI) and FK relationships associated with it.

Resources

  • Add ApiVersionId column with FK to ApiVersions.

Supports the persistence of resource metadata per API version.

DataMaps

  • Add ApiVersionId column with FK to ApiVersions.

Maps are version specific.

BootstrapDatas

  • Add ApiVersionId column with FK to ApiVersions.
  • Remove ProcessedDate column.
  • Remove ProcessingOrder column.

Bootstrap datas are version specific.

Migrate connection specific bootstrap information to a new table.

Processing order is now managed by agent instead of globally.

BootstrapDataAgents

  • New junction table.

Associate a bootstrap data item with an agent.

BootstrapDataApiServers

  • New junction table.

Migrate ProcessedDate from BootstrapDatas to track if bootstrap data needs to be posted to the API.

Related content

Operational Data Store Version Compatibility Approach
Operational Data Store Version Compatibility Approach
More like this
Data Import SIG Report-Out
Data Import SIG Report-Out
More like this
Ed-Fi 104 - Introduction to Data Management
Ed-Fi 104 - Introduction to Data Management
More like this
_Ed-Fi Tools Home
_Ed-Fi Tools Home
More like this
Data Mapping Template - ODS / API v2.0
Data Mapping Template - ODS / API v2.0
More like this
What's New in v2.1
What's New in v2.1
More like this