Versions Compared

Version Old Version 4 New Version 5
Changes made by

Jason Hoekstra

Jason Hoekstra

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: title change

...

Info
titleVersion 1.3+ Only

Docker deployment is only available starting with Data Import 1.3.0. See Docker Hub for all available versions

Table of Contents

Data Import Docker Composition

Below is a minimal Docker Compose file for Data Import and accompanying .env  file. It includes images for the Data Import application, backing PostgreSQL database, and PgBouncer connection pool.

...

Code Block
title.env
linenumberstrue
collapsetrue
POSTGRES_USER=<default postgres database user>
POSTGRES_PASSWORD=<password for default postgres user>
PGBOUNCER_LISTEN_PORT=<port for pg bouncer to listen to>

ENCRYPTION_KEY=<base64-encoded 256-bit key>
TIME_ZONE=<US/Central>

## Additional Configurable Settings
# NOTE: These appsettings need to be added to the environment section of the dataimport container within the docker compose file in order to take effect

# Uncomment below to skip certificate validation for FTPS agents
# APPSETTINGS__ALLOWTESTCERTIFICATES: "true"


# Uncomment below allow arbitrary PowerShell code to run in preprocessors
# APPSETTINGS__USEPOWERSHELLWITHNORESTRICTIONS: "true"


# Uncomment and update to change logged in timeout rate
# APPSETTINGS__LOGINTIMEOUTINMINUTES: <amount of time>


### File Settings
# Uncomment the below to override upload / script path
## Must be a valid directory on the container.
## To use a location on the host machine, map a volume to the container and update this to match
# APPSETTINGS__SHARENAME: <path>

# Uncomment below to use Azure for file storage
# APPSETTINGS__FILEMODE: "Azure"
# CONNECTIONSTRINGS__STORAGECONNECTION: <azure storage connection string>


### External Preprocessor Settings
# Uncomment below to enable experimental "External Process" preprocessors
# EXTERNALPREPROCESSORS__ENABLED: "false"
# Uncomment to override process timeout (default 5000)
# EXTERNALPREPROCESSORS__TIMEOUTINMILLISECONDS: <amount>

Using Other Databases

The above configuration is designed with a dedicated  PostgreSQL server and connection pool as containers. If you wish to connect to a different postgreSQL instance or otherwise modify that setup, simply change the POSTGRES_ variables to match your environment. If you are not using the postgres  or pgbouncer  services, make sure to remove them from your orchestration.

Connecting to a MSSQL Database Server

To connect to a Microsoft SQL Server instance you must have a valid SQL username and password. It possible to leverage a Docker container or an instance installed on the host or elsewhere, but if using the latter options, consider the host/container networking relationship.

  • Remove POSTGRES_ environment settings
  • Change APPSETTINGS__DATABASEENGINE  to SqlServer 
  • Update CONNECTIONSTRINGS__DEFAULTCONNECTION to a valid connection string.

Upgrading Data Import

From A Previous Non-Docker Deployment

Warning
titleMigrating Databases

Migrating the Data Import database from outside of Docker into a Docker container is not supported, as the ODS/API connection information will not be functional and schemas may differ between MSSQL Server and PostgreSQL. If this is a necessity, you may perform such a migration yourself manually and correct any errors that occur.

Alternatively, you may export relevant Data Maps, Bootstraps, and Preprocessors to files and re-import them on the new installation, or re-load from Template Sharing and adjust them as needed. In this case, you will need to re-establish ODS/API connections and configure Agents to match your previous installation. Consider completing the Docker installation and configuration before uninstalling the previous version to verify the setups match.

...

  1. If you are using the same database, Make a backup of the Data Import database, for safety.
    The installer is capable of automatically upgrading the content of the existing database, so the uninstall/install process can use the same database.
  2. Make a backup of the Data Import configuration files for any values you may have set here. Note especially your EncryptionKey  value which appears the same in both files. Copy this especially as it will be re-used in the new Data Import installation. The files to check differ for versions less than 1.3.0:
    1. 1.2.0: The web application "Web.config" and the Transform/Load application's "DataImport.Server.TransformLoad.exe.config" 
    2. 1.3.0+: The web application "appsettings.json" and the Transform/Load application's "appsettings.json" 
  3. Stop the previous Data Import application and website from Internet Information Server
  4. Run the Docker Installation
    1. Update configuration values to match those copied above
    2. Verify the website is running correctly in Docker
  5. Manually delete the previous Data Import application, website, and app pools from Internet Information Server.

From An Existing Docker Deployment

To upgrade from an existing Docker deployment:

...