Versions Compared

Old Version 15

changes.mady.by.user Gary Clarke

Saved on

compared with

New Version Current

changes.mady.by.user Gary Clarke

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

Purpose

The primary goals for introducing preprocessor enhancements to Data Import are to provide support for a wider range of file import use cases and improve the experience of working with Powershell scripts within Data Import.

...

  • Custom Record Processing
    • Provided via an optional selection of a Row Processor (PS1 file) on the agent screen.
    • This feature enables scripting of modifications to field values within a row.
    • More information
  • Custom File Generation
    • Provided via a "File System / PowerShell" agent type choice on the agent screen, which enables the selection of a Generator (PS1 file).
    • This feature enables the scripting of file generation on a predefined schedule.
    • More information

With the proposed enhancements, a third preprocessing capability will be supported:

...

Characteristics of the sandbox:

  • PowerShell scripts executed in a constrained runspace.
  • The runspace is initialized with no available cmdlets.
  • Cmdlets designed specifically for Data Import use are added to the runspace
    • Invoke-OdsApiRequest
    • New-ArrayList
    • New-StringBuilder
    • ...
  • Cmdlets explicitly whitelisted by the administrator are added to the runspace (probably will be defined in a config file)
    • High-risk cmdlets should be avoided (New-Object, Invoke-WebRequest, Invoke-Expression).

Importing modules:

  • Via configuration, a list of modules to be imported into the constrained runspace is defined.
  • Cmdlets designed specifically for Data Import are packaged as a PowerShell binary module and included by default.
  • PowerShell binary modules and PowerShell script modules can be imported into the constrained runspace by configuration.

(warning) Breaking change:

  • Cmdlets used by existing Custom Row Processor and Custom File Generator scripts must be added to the whitelist.

User Interface Changes

Menu

...

Manage Preprocessors

Add Preprocessor

...

Image Added

Add Data Map

Export Template

...

  • Custom File Processor script associated with selected Data Maps should be automatically included (similar to Lookups).
  • If Custom Row Processors exist, an additional step in the wizard will be shown after selecting Data Maps called to provide an optional selection of Custom Row Processors.
    • NOTE: This requirement was struck because agents are not shareable items, and the added functionality of supporting the export of Custom Row Processors seems unnecessary.
  • Custom File Generators are not supported in templates.
  • No UI changes are required for importing templates.

...

TableChangeJustification
Scripts

New Table:

  • Id (PK, int)
  • Name (nvarchar(255) not null)
  • ScriptType (nvarchar(20), not null)
  • ScriptContent (nvarchar(max) null)
  • RequiresOdsApi RequireOdsApiAccess (bit, not null)

Supports storage of PowerShell scripts.

EF Core note: Implement ScriptType as an enum.

DataMaps

New column:

  • FileProcessorScriptId (FK, int, null)
Optionally associate a data map with a Custom File Processor.
Agents

Removed columns:

  • RowProcessor (nvarchar(max), null)
  • FileGenerator (nvarchar(max), null)

New columns:

  • RowProcessorScriptId (FK, int, null)
  • FileGeneratorScriptId (FK, int, null)

Replace Custom Row Processor and Custom File Generator filenames with database references.

This will require a specialized migration:

  • Create a Script record with Name of the current value.
  • Reference it from the new column, and drop the old column.
  • On web application startup, for any Script records with a null ScriptContent value, get the file from the file system (by Name and known path per ScriptType) and populate contents into ScriptContent.

...