Data Import 2019
Jason Hoekstra
Ian Christopher (Deactivated)
Background
The Data Import solution is a project to simplify the loading of CSV data to the Ed-Fi ODS / API. The solution loads data for domains where vendor integration to the Ed-Fi APIs is inchoate or non-existent, for example, from legacy data sources such as transportation systems. It does so by providing methods to extract and transform information from spreadsheet-based data (CSV files) for loading to the Ed-Fi ODS / API.
Data Import is a C# .NET solution with a web administration panel in ASP.NET to view data and job status, and server components as .NET command-line applications to process data. Data Import is designed to run on-premises or within cloud-hosted environments such as Microsoft Azure, Amazon Web Services, or Google Cloud. This is designed to match the Ed-Fi ODS / API operating model of choice by education-serving entities. The Data Import solution is intended to be used by system IT administrators and technical data analysts, in service of Local Education Agency (LEA) and State Education Agency (SEA) needs where directly integrated API solutions do not exist.
The Data Import solution will be based on code from the MSDF-sponsored Data Flow proof-of-concept project as published on the Ed-Fi Exchange.
Goals
The goal of the Data Import SIG is to guide feature prioritization for the initial release of Data Import v1.0. SIG members will provide input and guidance through the objectives and activities below.
Participation
The Ed-Fi Alliance will convene a special interest group (SIG) with the following objectives in mind:
- The Alliance will convene a group of active and participating members around the usage of a technology solution to import non-API data sources in the Ed-Fi ODS / API.
- Members will be expected to install, configure, map local data sources, import, test, quality check, and report results with pre-release versions of the Data Import solution.
- Members will be expected to participate in SIG conference calls and electronic exchanges to share experiences and help shape future development priorities (scheduled to begin the week of February 18, 2019).
- Member attendees of the 2019 Ed-Fi Tech Congress will be invited to attend an in-person convening for this SIG.
Meetings
The Ed-Fi Alliance will convene a series of four WebEx meetings for the Data Import SIG. Each meeting will be recorded and posted within this section once completed. This section will be updated as meeting details become available.
| Meeting Date | Agenda | WebEx |
|---|---|---|
| Monday, February 25, 2019 at 1pm CT | Field Experiences with Data Flow | Data Import SIG - February 25, 2019 |
| Monday, March 11, 2019 at 1pm CT | Data Import Use Case Review | Data Import SIG - March 11, 2019 |
| Thursday, April 11, 2019 at 4:15pm ET | Data Import Tech Preview Demonstration | Ed-Fi Tech Congress 2019, Banyan Room Bootcamp Session 5: Hands-On with Data Import Tool |
| Thursday, April 25, 2019 at 1pm CT | Extract-Transform-Load Workflow Review | Data Import SIG - April 25, 2019 |
Members
| Organization | Participants |
|---|---|
Boston Public Schools (BPS) (Massachusetts) |
|
| Center for Educational Leadership and Technology (CELT) |
|
Certica Solutions |
|
| Eduphoric |
|
| EdWire |
|
| Global Grid For Learning (formerly Edutone Corporation) |
|
| InnovateEDU |
|
| Indiana Student Information to Empower (INSITE) |
|
North East Florida Education Consortium (NEFEC) |
|
| Rio Rancho Public Schools (RRPS) (New Mexico) |
|
| UPD Consulting |
|
| Ed-Fi Alliance |
|
Development Priorities
We have gathered extensive feedback from the Data Flow proof-of-concept from numerous field pilot implementations. From this feedback, we have identified the following high-priority requests to address in pre-release versions of Data Import. We intend to have a pre-release version of Data Import available for testing ahead of the 2019 Ed-Fi Tech Congress on April 10-12, 2019.
The following are example questions to be addressed within the SIG, which largely reflect feedback we've received to date:
- What is the impact of a major defect in the User Interface which prevents the mapping of complex objects such as student assessment? What improvements could be made to the User Experience for easier use and improved productivity for ETL-related tasks? (There is a method that requires manual JSON mapping, but that is an advanced skill.)
- Would Data Import expansion to the Ed-Fi ODS / API for Technical Suite 3 benefit active implementations of this product line?
- Would expanding the supported API endpoints into other domains (such as special education or transportation) benefit active implementations? The Data Flow proof-of-concept originally targeted the assessment domain. However, the technology may be able to accommodate other data domains as identified by the SIG.
- Would a feature to share ETL maps be useful? This would enable implementation sites with similar product and data requirements to reuse these assets to aid and bootstrap others in the Ed-Fi community. This feature would share only ETL metadata and not actual data itself. We have a feature description along with screen mock-ups based on prior field feedback that will be shared with the SIG to gather additional input to decide on the future of its development.
Other development priorities will emerge within the SIG and will be scheduled as the group identifies the next round of priorities.
How to Join
To participate, please contact Jason.Hoekstra@ed-fi.org with a short summary (2-3 paragraph description) of the needs and use cases for non-API data ingestion for your project(s). Examples of information that would be useful are: identified data sources, relevant file formats (CSV, Excel, etc.), Ed-Fi environment version (v2.5, v3.1, etc.), plus any other details that pertain to the description above.