ETL Developers' Guide - Installation

This documentation covers the steps required for loading the Dashboard v3.0 for implementations using the Ed-Fi ODS v3.1. These instructions assume that you have already followed the UI Developers Guide - Installation.

The steps can be summarized as follows:

Detail about each step follows.

Step 1. Restore Sample Database

• Glendale Sample Data Set. A robust sample set with around 47,000 students. This data set is used by the Alliance for dashboard development, unit testing, and integration testing.

To restore the sample data set:

• Open a PowerShell window and navigate to C:\Ed-Fi\Database\ (presuming the installation path used in the Getting Started documentation was the recommended C:\Ed-Fi\).
• If the SQL Server is not set up as the default instance (i.e., MSSQLSERVER), then run the command $env:db_server = ".\MyNamedInstance" to set up the named SQL Server instance.
• By default, the script will restore the Glendale data set.
  • To restore the Glendale sample data set, execute $env:db_agency = "Glendale" from the PowerShell command window.
• Run the command .\build rebuild 3.0  to restore the following:
  • The ODS v3.2 database  with the necessary views to support the ETL process.
  • The Application database with the necessary data to support the ETL process.
  • Blank copies of the Dashboard, Dashboard DW, and ETLLog databases.

Step 2. Configure the ETL app.config Settings

Locate the app.config file in C:\Ed-Fi\Etl\src\EdFi.Runtime\ and verify the following parameters are configured correctly. The most common configuration values to verify are the database connection strings and the OdsVersion value.

Database Connection Strings

 <connectionStrings>
    <add name="Dashboard" connectionString="Data Source=(local);Initial Catalog=EdFi_Dashboard;Integrated Security=SSPI;Application Name=ETLRuntime" providerName="System.Data.SqlClient"/>
    <add name="Warehouse" connectionString="Data Source=(local);Initial Catalog=EdFi_DashboardDW;Integrated Security=SSPI;Application Name=ETLRuntime" providerName="System.Data.SqlClient"/>
    <add name="Ods" connectionString="Data Source=(local);Initial Catalog=EdFi;Integrated Security=SSPI;Application Name=ETLRuntime" providerName="System.Data.SqlClient"/>
    <add name="Application" connectionString="Data Source=(local);Initial Catalog=EdFi_Application;Integrated Security=SSPI;Application Name=ETLRuntime" providerName="System.Data.SqlClient"/>
    <add name="EtlLogDb" connectionString="Data Source=(local);Initial Catalog=EtlLogDb;Integrated Security=SSPI;Application Name=ETLRuntime" providerName="System.Data.SqlClient"/>
  </connectionStrings>

Application Configuration Settings

Metric Calculation Configuration Settings

Advanced Configuration Settings

App Settings with the As-Shipped Values

  <appSettings>
    <add key="DailyAttendanceCalculationSource" value="Both"/>
    <add key="ReportingPeriodLastXDays" value="20"/>
    <add key="ReportingPeriodLastYDays" value="40"/>
    <add key="APScoreMasteryLevel" value="3"/>
    <add key="IBScoreMasteryLevel" value="4"/>
    <add key="IncludeWithdrawnStudents" value="true"/>
    <add key="MinimumLogLevel" value="Information"/>
    <add key="EnableConsoleLogging" value="true"/>
    <add key="DryRun" value="false"/>
    <add key="DaysNeededForLateEnrollment" value="14"/>
    <add key="CalculateDailyHistoricalAttendance" value="true"/>
    <add key="RunOnlyYearEndSnapshot" value="false"/>
    <add key="RebuildHistoricalCalculations" value="true"/>
    <add key="ExcludeExtensionList" value=""/>
    <add key="SqlQueryCommandTimeout" value="1200"/>
    <add key="OdsVersion" value="3.1"/>
    <add key="CacheAllQueries" value="false"/>
    <add key="SessionTraceFlags" value=""/>

    <!-- SQL insert properties -->
    <add key="BulkCopyBufferSizeMin" value="4096"/>
    <add key="BulkCopyBufferSizeMax" value="1000000"/>
    <add key="BulkCopyBatchSize" value="4096"/>
    <add key="UseFileBuffer" value="false"/>
    <add key="FileBufferLocation" value="C:\Windows\Temp"/>

    <!--Linked Server Configuration -->
    <add key="DashboardLink" value=""/>
    <add key="OdsLink" value=""/>

  </appSettings>

Step 3. Configure Application Database

The following should be set up to ensure a successful ETL execution:

• Grading scales for all grade levels should be configured. They can be set up in the UI when logged in as an Admin user under the Metric Settings tab.
• For Algebra metric calculations, the dashboard.CourseLevelMapping table needs to be populated with Course Identity Codes that map to Algebra courses. For more information, see Algebra Course Level Mapping in the Metric Metadata Configuration section of this documentation.

Step 4. Compile and Run the ETL Solution

• Open a PowerShell window and navigate to C:\Ed-Fi\Etl\.
• Run the command .\build compile to compile the solution.
• Navigate to C:\Ed-Fi\Etl\src\EdFi.Runtime\bin\Release\ and execute EdFi.Runtime.exe. 
  • The ETL runtime may take several hours to complete depending on the size of the ODS. This process can be monitored by looking at row counts in the metric.MetricInstance table in the Edfi_Dashboard DB or errors and warnings logged to the dbo.LogEntries view in the EtlLogDb.

Step 5. Build and Launch the Dashboards Web Application 

Return to Step 7 in UI Developers' Guide - Installation and repeat the steps to view the metrics in the Ed-Fi Dashboard.

Troubleshooting

This section contains troubleshooting tips for a few situations:

• The EdFi.Runtime.exe exits immediately. Confirm that the database connection strings are configured correctly. Query the dbo.LogEntries view in EtlLogDb database for any exception info.
• Need to re-run the EdFi.Runtime.exe after a successful completion. Simply execute EdFi.Runtime.exe again.

Next Steps & Further Information

This section contains links to helpful information found in other sections of this documentation:

• If you'd like to customize the metrics for your environment, see the technical article Managing Metrics Using the Metric Configuration Tool.  
• If you'd like to understand the ETL Application processing in more detail, see the technical article Overview of the ETL Application Runtime Process.