A newer version of the Ed-Fi ODS / API is now available. See the Ed-Fi Technology Version Index for a link to the latest version.
Bulk Load Console
- Ian Christopher (Deactivated)
- Vinaya Mayya
Deprecated
Bulk Load Console is deprecated and may be removed in a future release. Plan on switching to the Ed-Fi Bulk Load Client utility. The Bulk Load Console will continue to work as described in this document until removed.
The Bulk Load Console (hereafter called the Loader) is a command-line tool that can be used by platform hosts to initially populate an ODS database with XML data conforming to the Ed-Fi Data Standard.
The Loader is built using code generation techniques that take into account the ODS database schema as well as the Ed-Fi XSD Schema definitions and metadata. The Loader can also be used during database initialization to create the Minimal ODS Template database by populating Ed-Fi Descriptors and education organizations into an empty ODS database. The Loader project is called EdFi.Ods.BulkLoad.Console. It is part of the Ed-Fi-Ods solution. More details on how to use the Bulk Load Console can be found at How To: Load the ODS Using Sample XML Data.
Command-Line Parameters
The Loader accepts the following command-line parameters:
| Parameter | Required/ Optional | Description | Example Parameter Value |
| Database Name | Required | The name of the ODS database to receive the bulk data |
|
| Connection String | Required | The SQL connection string specifying the target database server and user credentials |
|
| Source Folder | Required | The path to a folder containing the interchange files referenced by the manifest |
|
| Manifest | Required | The path to a manifest XML file describing the interchange files to load |
|
/t | API Token | Optional | An API authorization token, used as the security context for a bulk load. When the argument is omitted, no security is applied | 7ba37ccaa1bf4f0eb50d8f748ec2ebd0 |
/r | bulkOperationId | Optional | This option is used to release a destination database locked during a bulk operation, when a bulk operation was terminated before completion. | 2 |
| help | Used Alone | Displays help information |
|
Database
The target ODS database must already exist and be accessible using the “Connection String” (/c
) parameter provided. This database must conform to the same database schema that was used to generate the Loader application.
Manifest File
The manifest file is a simple XML document containing a list of the interchange XML files to load. Because files of the same type are loaded in parallel, it is important that files loaded by the same manifest have distinct XML IDs. Large interchange files may show errors when loading due to the highly parallelized nature of the Loader application. Usually, this is resolved by simply re-running the loader with the same settings.
A manifest file takes the following form:
<Interchanges> <Interchange> <Filename>Standards.xml</Filename> <Type>standards</Type> </Interchange> <Interchange> <Filename>EducationOrganization.xml</Filename> <Type>educationorganization</Type> </Interchange> </Interchanges>
The manifest file elements are described below:
- The
Interchanges
element must exist at the root, and may only be present one time. - The
Interchange
tag may exist more than once, and contains a singleFilename
andType
element. - The
Filename
element contains the name of the interchange file to be loaded. This may be a long filename. Note that the file must exist in the directory provided by the “Source Folder” (/f
) command line parameter at runtime. - The
Type
element contains the case-insensitive class name for the resource to be loaded.
Load Order
The load order of data is extremely important when creating a populated ODS database from a minimal one. The load order reflected in the “InterchangeOrderMetadata.xml” file (EdFi.Ods.CodeGen/App_Packages/Ed-Fi/Metadata) contains the correct load order information for the ODS database provided in the Ed-Fi repository. The following load order applies:
- Descriptors
- Standards
- EducationOrganization
- EducationOrgCalendar
- MasterSchedule
- StaffAssociation
- Student
- Parent
- AssessmentMetadata
- StudentCohort
- StudentAssessment
- StudentIntervention
- StudentTranscript
- StudentProgram
- StudentEnrollment
- StudentDiscipline
- PostSecondaryEvent
- StudentAttendance
- StudentGrade
- StudentGradebook
Troubleshooting
This section outlines a few troubleshooting tips related to the Loader.
- The Loader expects valid Ed-Fi Data Standard XML files. If the sample files referenced above load correctly, consider validating the files prior to running the Loader.
- The Loader sometimes has problems loading extremely large datasets (e.g., multiple files of multiple gigabytes). An indication that file size is an issue is when the bulk load operation "completes" but reports thousands of errors. The workaround is to load each file individually by wrapping each file in its own manifest. This approach generally works — but if you continue to receive errors that you suspect to be caused by a large dataset, consider splitting the XML data into smaller files and loading each file in its own manifest.
- The Loader process is an upsert. So, if the Loader reports errors (e.g., when loading an extremely large dataset), simply re-running the loader will add any records that failed due to transient issues.