PowerShell scripts executed in a constrained runspace.
The runspace is initialized with no available cmdlets.
Cmdlets designed specifically for Data Import use are added to the runspace
- Invoke-OdsApiRequest
- New-ArrayList
- New-StringBuilder
- ...
Cmdlets explicitly whitelisted by the administrator are added to the runspace (probably will be defined in a config file)
- High-risk cmdlets should be avoided (New-Object, Invoke-WebRequest).

Importing modules:

Via configuration, a list of modules to be imported into the constrained runspace is defined.
Cmdlets designed specifically for Data Import are packaged as a PowerShell binary module and included by default.
PowerShell binary modules and PowerShell script modules can be imported into the constrained runspace by configuration.

User Interface Changes

...

Database Changes

Table

Change

Justification

PreprocessorsScripts

New Table:

Supports storage of PowerShell scripts.

EF Core note: Implement PreprocessorType ScriptType as an enum.

DataMaps

New column:

Optionally associate a data map with a Custom File Processor.

Agents

Removed columns:

New columns:

Replace Custom Row Processor and Custom File Generator filenames with database references.

This will require a specialized migration:

Create a preprocessor Script record with Name of the current value.
Reference it from the new column, and drop the old column.
On web application startup, for any Preprocessor Script records with a null ScriptContent value, get the file from the file system (by Name and known path per PreprocessorTypeScriptType) and populate contents into Script Content.

...