| Table of Contents |
|---|
Purpose
The primary goals for introducing preprocessor enhancements to Data Import are to provide support for a wider range of file import use cases and improve the experience of working with Powershell scripts within Data Import.
...
- Custom Record Processing
- Provided via an optional selection of a Row Processor (PS1 file) on the agent screen.
- This feature enables scripting of modifications to field values within a row.
- More information
- Custom File Generation
- Provided via a "File System / PowerShell" agent type choice on the agent screen, which enables the selection of a Generator (PS1 file).
- This feature enables the scripting of file generation on a predefined schedule.
- More information
With the proposed enhancements, a third preprocessing capability will be supported:
...
Characteristics of the sandbox:
...
- The runspace is initialized with no available cmdlets.
- Cmdlets designed specifically for Data Import use are added to the runspace
- Invoke-OdsApiRequest
- New-ArrayList
- New-StringBuilder
- ...
- Cmdlets explicitly whitelisted by the administrator are added to the runspace (probably will be defined in a config file)
- High-risk cmdlets should be avoided (New-Object, Invoke-WebRequest, Invoke-Expression).
Importing modules:
- Via configuration, a list of modules to be imported into the constrained runspace is defined.
- Cmdlets designed specifically for Data Import are packaged as a PowerShell binary module and included by default.
- PowerShell binary modules and PowerShell script modules can be imported into the constrained runspace by configuration.
Breaking change:
- Cmdlets used by existing Custom Row Processor and Custom File Generator scripts must be added to the whitelist.
User Interface Changes
Menu
...
Manage Preprocessors
Add Preprocessor
...
Add Data Map
Export Template
...
- Custom File Processor script associated with selected Data Maps should be automatically included (similar to Lookups).
If Custom Row Processors exist, an additional step in the wizard will be shown after selecting Data Maps called to provide an optional selection of Custom Row Processors.- NOTE: This requirement was struck because agents are not shareable items, and the added functionality of supporting the export of Custom Row Processors seems unnecessary.
- Custom File Generators are not supported in templates.
- No UI changes are required for importing templates.
...
| Table | Change | Justification |
|---|---|---|
| Scripts | New Table:
| Supports storage of PowerShell scripts. EF Core note: Implement ScriptType as an enum. |
| DataMaps | New column:
| Optionally associate a data map with a Custom File Processor. |
| Agents | Removed columns:
New columns:
| Replace Custom Row Processor and Custom File Generator filenames with database references. This will require a specialized migration:
|
...