This version of the Ed-Fi Dashboards is no longer supported. See the Ed-Fi Technology Version Index for a link to the latest version.
ETL Package Execution Order
- Itzel Torres
- Ian Christopher
Overview
This article describes the order in which the Dashboard ETL packages need to be executed in the Ed-Fi Dashboard.
At a high level, the order starts with loading data from XML Interchange files, default Ed-Fi Core Types and Descriptors, and ends with the packages that populate the Ed-Fi Dashboards metrics. The load order dependencies are based on the entities populated and the hierarchy defined by the entity-relationship model. These entities and relationships are defined by the education industry and their interaction is modeled by Ed-Fi data domains to represent all its business rules.
Since most database operations are kept consistent by the principle of atomicity (which means, either all the elements are successfully executed or not), all the packages in the Dashboard ETL follow this principle individually, though there is some modularity between packages.
As result, a package execution order is established to ensure that before the execution of a package, the preceding package did its job properly and its output works as partial or total input of the package about to be executed, forming a dependency between them. Following this order of package execution will ensure that valid data is populated in the corresponding tables successfully.
A conceptual view of the sequence follows:
The Package Type column defines the stage or phase in the process the package belongs to. It is important to note the packages are classified according to the databases they interact with or the type of data they process. There are four categories matching the stages or phases:
- Types. Type packages load data from spreadsheets to populate Ed-Fi Enumeration values, Ed-Fi Descriptor values and Metric Metadata (i.e., Dashboard Types).
- Loader. Loader packages load data from the different XML Interchange files to the Operational Data Store (ODS).
- Dashboard. Dashboard packages load and aggregate data from the ODS to the Dashboard Data Mart (DDM) database.
- DW. Loads data from the DDM to the Dashboard Data Warehouse database (DW).
The following table shows the load order for the Dashboard ETL packages delivered with the Ed-Fi Dashboards:
Order | Package | Package Type |
|---|---|---|
1 | Import Ed-Fi Core Types.dtsx | Types |
2 | Import Ed-Fi Descriptors.dtsx | Types |
3 | Import Dashboard Types.dtsx | Types |
4 | StudentParent | Loader |
5 | EducationOrganization | Loader |
6 | AssessmentMetadata | Loader |
7 | EducationOrgCalendar | Loader |
8 | PostSecondaryEvent | Loader |
9 | StudentProgram | Loader |
10 | MasterSchedule | Loader |
11 | FactAutomateBOY | DW |
12 | StudentEnrollment | Loader |
13 | StaffAssociation | Loader |
14 | StudentDiscipline | Loader |
15 | StaffInformation | Dashboard |
16 | StudentCohort | Loader |
17 | StudentIntervention | Loader |
18 | StudentGrade | Loader |
19 | StudentAttendance | Loader |
20 | StudentGradebook | Loader |
21 | StudentTranscript | Loader |
22 | StudentInformation | Dashboard |
23 | StudentAssessment | Loader |
24 | StudentParentInformation | Dashboard |
25 | StudentSchoolInformation | Dashboard |
26 | StudentIndicator | Dashboard |
27 | TeacherSection | Dashboard |
28 | StaffCohort | Dashboard |
29 | StudentRecordCourseHistory | Dashboard |
30 | StudentDisciplineMetrics | Dashboard |
31 | SchoolInformation | Dashboard |
32 | StudentAttendanceMetrics | Dashboard |
33 | SchoolStaffMetrics | Dashboard |
34 | LocalEducationAgencyInformation | Dashboard |
35 | SchoolDisciplineMetrics | Dashboard |
36 | StudentObjectiveAssessmentMetrics | Dashboard |
37 | StudentGrowthMetrics | Dashboard |
38 | StudentAcademicChallengeMetrics | Dashboard |
39 | StudentCollegeCareerReadinessMetrics | Dashboard |
40 | SchoolMetricAssessmentRate | Dashboard |
41 | StudentBenchmarkAssessment | Dashboard |
42 | StudentRecordAssessmentHistory | Dashboard |
43 | SchoolGoals | Dashboard |
44 | LocalEducationAgencyGoals | Dashboard |
45 | MetricThreshold | Dashboard |
46 | StudentCourseGradeMetrics | Dashboard |
47 | StudentRecord | Dashboard |
48 | FactLocalAssessmentBenchmark | DW |
49 | StudentAssessmentMetrics | Dashboard |
50 | SchoolBenchmarkAssessment | Dashboard |
51 | SchoolAttendanceMetrics | Dashboard |
52 | SchoolAssessmentMetrics | Dashboard |
53 | SchoolGrowthMetrics | Dashboard |
54 | StudentContextContainerMetrics | Dashboard |
55 | SchoolCollegeCareerReadinessMetrics | Dashboard |
56 | SchoolAcademicChallengeMetrics | Dashboard |
57 | LocalEducationAgencyStaffMetrics | Dashboard |
58 | LocalEducationAgencyCollegeCareerReadinessMetrics | Dashboard |
59 | SchoolFootnoteContainerMetrics | Dashboard |
60 | SchoolCourseGradeMetrics | Dashboard |
61 | SchoolContextContainerMetrics | Dashboard |
62 | LocalEducationAgencyCourseGradeMetrics | Dashboard |
63 | LocalEducationAgencyAssessmentMetric | Dashboard |
64 | LocalEducationAgencyAcademicChallengeMetrics | Dashboard |
65 | LocalEducationAgencyGrowthMetrics | Dashboard |
66 | LocalEducationAgencyBenchmarkAssessment | Dashboard |
67 | LocalEducationAgencyEngagementMetrics | Dashboard |
68 | StudentAggregateMetrics | Dashboard |
69 | LocalEducationAgencyContextContainerMetrics | Dashboard |
70 | FactMetricInstance | DW |
71 | FactAttendance | DW |
72 | FactDiscipline | DW |
73 | FactCourseGrades | DW |
74 | FactTeacherAttendance | DW |
75 | FactCampusBenchmarkPerformance | DW |
76 | SchoolLongitudinalMetrics | Dashboard |