TAG Meeting 2021-01-21 - Subgroup on Data Import Roadmap

Participants

Background

Last year Ed-Fi Alliance began working with Certica/Instructure who offered community contributions into Data Import
Data Flow - precursor to Data Import source code
v1.2 update: new agent type, different level workflow. Template sharing in PowerShell (e.g. iReady), preprocessing ability using Powershell, security enhancements
Q: with net core change, should also consider powershell core since they are not necessarily mutually exclusive https://powershell.org/2019/02/tips-for-writing-cross-platform-powershell-code/
A: A few different models to be looked at and support for Python and PowerShell on Linux base platform
PowerShell enhancements in v1.2 planned for release in March. Python support in consideration for v1.3 along with .NET Core updates.
V1.2 presentation by Instructure (appended PPT)

Design documents discussion

There were some comments on the pre-processor language including request for Powershell Core and Python to be supported
Most discussion concerned the questions around pre-processing and how to secure it (or if it could be made sufficiently secure) and if there should be some higher-level abstraction for such preprocessing (i.e. a DSL type language that encapsulates particular options). There were some comments that it would generally be hard to get away from scripting, and that scripting is a function to which there is reasonable community access.
There were some comments on possible Web UIs as well, which are seen in other environments where data must be pre-prepared prior to ingestion.
There were also some observations/ comments about the difficulty of making anything based on scripting available to the wider Ed-Fi community, given that the target audience is a data analyst.
There were comments suggesting development of a CLI with a Web interface that calls the CLI, allowing for more development / consumption operations for users of the toolset. Generally there were suggestions of greater decoupling of components.