...

• Provide support for non-CSV files (e.g. tab-delimited, fixed-width, or XML)
• Enable reshaping of the row grain of the file (one row becomes multiple or vice-versa)
• Enable exclusion of rows (e.g. student assessment rows that indicate the student was not tested)
• Enable invocation of the ODS API from preprocessor scripts (dynamically selecting the API used by the executing agent)
• Improve the management and sharing capabilities of PowerShell scripts used by preprocessors
  • Maintain scripts in the database instead of the file system and provide a UI for managing
  • Include script dependencies in template sharing and import/export toolstemplates

Related epic: 

...

• Custom Record Processing
  • Provided via an optional selection of a Row Processor (PS1 file) on the agent screen.
  • This feature enables scripting of modifications to field values within a row.
  • More information
• Custom File Generation
  • Provided via a "File System / PowerShell" agent type choice on the agent screen, which enables the selection of a Generator (PS1 file).
  • This feature enables the scripting of file generation on a predefined schedule.
  • More information

...

Do we need both Custom Record Processing and Custom File Processing? NEED INPUT (working assumption is that we  retain)

There is an overlap between the capabilities of the existing Custom Record Processing and the new Custom File Processing feature:

...

Where will ODS API invocation support be implemented? NEED INPUT (working assumption is that we support both)

There are three possibilities:

• API invocation is supported by Custom Record Processing (on Agent screen)
  • This works well if we are confident that any logic requiring use of the API happens after the file has been converted to tabular data by the Custom File Processor.
  • However, it is of course only applicable if we retain the Custom Record Processing capability.
• API invocation is supported by Custom File Processing (on Data Map screen)
  • Building a Data Map requires discovery of the source file columns. When a Custom File Processor is used and that script requires the use of the API, then we must prompt for an API Connection for executing the script (per Multi-Connection enhancements).
  • If we use this approach, it may be preferred that ODS API support for the preprocessor is explicitly enabled as a configuration option for each applicable Preprocessor script.
• API invocation is supported by Custom Record Processing and Custom File Processing
  • This favors flexibility for undiscovered use cases.

User Interface Changes

Menu

Image Removed

Manage Scripts

Image Removed

Add Script

Image Removed

Add Data Map

Image Removed

Database Changes (TODO)

...

Security concerns? NEED INPUT

Providing the capability for an application to dynamically execute user-provided PowerShell scripts adds a potentially significant attack vector to the web app and transform load console app.

While Data Import already supports the execution of PowerShell scripts, those scripts are read from the file system. With these enhancements, scripts can be managed via the UI.

We need to put some thoughts into ensuring effective security measures are considered, whether this is constraints on script capabilities or documenting risks and recommendations for restricting access to the server. We can also consider an option to disable the use of scripts.

User Interface Changes

Menu

Image Added

Manage Preprocessors

Image Added

Add Preprocessor

Image Added800

Add Data Map

Image Added

Export Template

Image Added

• Custom File Processor script associated with selected Data Maps should be automatically included (similar to Lookups).
• If Custom Row Processors exist, an additional step in the wizard will be shown after selecting Data Maps called to provide an optional selection of Custom Row Processors.
• Custom File Generators are not supported in templates.
• No UI changes are required for importing templates.

Database Changes (TODO)

Questions:

• Should we name the new table "Scripts" instead of "Preprocessors"? Role-specific foreign key column names could more naturally be called "FileProcessorScriptId" versus "FileProcessorPreprocessorId" (notice that I opted to drop "Preprocessor" entirely in the changes described above).