Bulk Loading T-ODS Using Console Bulk Loader

Introduction

Bulk Temporal Data will now be accepted by the Ed-Fi Console Bulk Loader.  Bulk Operations now support new parameters used for the creation of snapshot metadata.  Ed-Fi Bulk XML representing an ODS snapshot can be loaded using the Bulk Loader.  When accompanied by temporal metadata, the Bulk Loader targets the new temporal staging ODS (see Databases section below), triggers a snapshot of that data, then populates the T-ODS. 

Bulk Loading Temporal Data

To run the Bulk Loader with temporal features, first ensure you have checked out a version of the codebase containing these features:

• Check out the following Github feature branches:
  • Ed-Fi-ODS/f-TODS-Segment2
  • Ed-Fi-ODS-Implementation/f-TODS-Segment2
  • Ed-Fi-Common/f-TODS-Segment2
• Open PowerShell and navigate to the Ed-Fi-ODS-Implementation path

• Run the command .\Initialize-PowershellForDevelopment.ps1 to initialize PowerShell for development.

• Run the command initdev

• Open EdFi.Ods.sln project (C:\Git\Ed-Fi-ODS-Implementation\Application\)

  • Select Build > Build Solution

• Set the startup project

  • In the solution explorer, right click on the EdFi.Ods.Tods > Entry Points > BulkLoad > EdFi.Ods.BulkLoad.Console 

  • Set this as the startup project

• At Command Prompt navigate to ...\Ed-Fi-ODS-Implementation\Application\EdFi.Ods.BulkLoad.Console\bin\Debug\EdFi.Ods.BulkLoad.Console.exe
• Supply bulk load and additional temporal snapshot parameters and hit enter

Snapshot Mode 

The current Ed-Fi Bulk Load pipeline will often require a workaround using incremental loading of bulk files.  This is due to load failures arising as a result of missing dependent data. The T-Bulk console supports incremental loads using Incremental mode. This allows a user to load the bulk files incrementally, domain by domain, in order to ensure dependencies are in place in the temporal staging database as data is loaded. Alternatively, users might choose to load a full set of bulk files repeatedly until all dependency issues are resolved and the temporal staging database accurately reflects the load that was submitted.  When the temporal staging database is finally loaded and ready for snapshotting, the user will execute the snapshot using SnapshotIncremental mode. 

Temporal Command-Line Parameters

The following parameters have been added in support of loading temporal data.  If no temporal parameters are supplied the Console initiates a standard (non-temporal) bulk load operation.

/y

/b

/e

/o

/n

/l

/s

/p

Databases

When initiating a temporal bulk load, data is loaded and staged in a database preloaded with Types but otherwise empty, called "EdFi_TemporalOds_Staging1" this database is created at build time.  The temporal bulk process clears out this database at the completion of the snapshot.

The landing place for the temporal data is the T-ODS associated with the target ODS database indicated using the the “Connection String” (/c) parameter provided. This database must conform to the same database schema that was used to generate the Bulk Loader application.

Usage 

Bulk Load Populated Template Data as Temporal Snapshot

1. Set up a manifest file per the instructions on /wiki/spaces/ODSAPI24/pages/21693249
2. Open Command window and navigate to Ed-Fi-ODS-Implementation\Application\EdFi.Ods.BulkLoad.Console\bin\Debug\EdFi.Ods.BulkLoad.Console.exe
3. Enter parameters similar to the example below and hit enter 
   1. Note that parameters after the Manifest location /m are temporal parameters
      
      

\Ed-Fi-ODS-Implementation\Application\EdFi.Ods.BulkLoad.Console\bin\Debug\EdFi.Ods.BulkLoad.Console.exe /c "Database=EdFi_Ods_Populated_Template; 
Data Source=(local); Persist Security Info=True; Trusted_Connection=True" /f "C:\Temp\BulkLoad\PopulatedTemplateBulkLoad\Data" 
/m "C:\Temp\BulkLoad\PopulatedTemplateBulkLoad\Manifest.xml" /s "snap1" /n "Snapshot 1" /y "2017-02-01" /b "2017-02-01"