Overview

This page provides information for the Data Import DI-1270 hotfix.  During the upgrade between Data Import 1.4.0 and Data Import 2.0.0 for multi-threading and performance updates, logging strategies where updated away from log4net to Serilog and Microsoft.Extensions.Logging for architectural and performance reasons.  In Data Import 2.0.0, the Ingestion logging is using Microsoft.Extensions.Logging and the log messages are queued at the end of each job run to help accommodate multi-threading.  In field usage, it's been found that ingestion logging is consuming the majority of system RAM for large data loads, which has impacted the dependability of these data loads. 

This hotfix exposes a configuration setting for Ingestion logging levels, which will allow the implementer filter messages to the database as desired amongst existing levels (Debug, Information, Notice, Warning, Error, Critical), in attempt to reduce down the number of build-up of log messages during large data loads.  The field experiences from this hotfix will be considered into future releases into Data Import based on results.

Related Tickets

EDFI-2159 - Getting issue details... STATUS

DI-1270 - Getting issue details... STATUS

DI-1275 - Getting issue details... STATUS

Installation Instructions

Below are installation instructions for the Data Import DI-1270 hotfix.  The scope of the hotfix only applies to Data Import's TransformLoad command line service.

1. Back up the existing TransformLoad service and directory, in case of the need to rollback.

2. Download the package as listed in the right-sidebar on this page.
   

3. Unzip the package in your preferred path.
   
4. Locate the "appsetting.json" file from the unzip package and set the variable MinimumLevelIngestionLog in the following section:
   
5. The new variable called MinimumLevelIngestionLog can be set with one of the following values (it is not case-sensitive):  "Information" or "Error", as only in use by Data Import today.  "Information" contains every detail about successful and errored attempts to the ODS/API and "Error" contains only failed rows to the ODS/API.
6. Replace the other variables to match with your environment, such as "ConnectionString", "EncryptionKey" and so on.
7. Copy all the content of this package and replace it into your TransformLoad application.
   
8. Check the result.
   Now, you can check the results when you upload and transform csv files on the Ingestion Log page in the Data Import Admin.