A newer version of the Ed-Fi ODS / API is now available. See the Ed-Fi Technology Version Index for a link to the latest version.
Performance Test Analysis on ODS / API v3.4
- Ian Christopher (Deactivated)
- Vinaya Mayya
Summary
This article describes the methodology and result of performance testing on the Ed-Fi ODS / API v3.4.0.
In brief, performance testing did not uncover any significant concerns in performance relative to previous 3.x versions.
Article Contents:
Test Methodology
Volume Tests
Volume testing of v3.4.0 occurred in May 2020, using the Locust-based 3.x performance testing framework (an Exchange contribution available in GitHub). This volume test covers the resources and HTTP verbs described in the Ed-Fi SIS vendor certification process. It runs for 30 minutes, spawning 30 clients that run simultaneously to perform tens of thousands of operations.
Data Set
Northridge-inspired v3 data set, which contains 21,628 students.
Azure Test Lab Virtual Machines
The test lab environment used a three-server setup: one each for the database, web applications, and the Locust performance testing. VM "Sizes" listed here, such as "DS11_v2", are Microsoft-defined names for the specs of an Azure VM. Key specs are listed beside these size names. These sizes were chosen as their specs are comparable to those of the Vendor Certification VMs but have SSD disks to more closely match a production environment.
Database VM
Image: Free SQL Server License: SQL Server 2017 Developer and PostgreSQL 11.8 on Windows Server 2016
Size: DS11_v2 (Standard, 2 vcpus, 14 GB ram, 6400 max iops, 127GB local SSD)
Web VM
Image: Windows Server 2016 Datacenter
Size: B2ms (Standard, 2 vcpus, 8 GB ram, 4800 max iops, 127GB local SSD)
Test Runner VM
Image: Windows Server 2016 Datacenter
Size: B2ms (Standard, 2 vcpus, 8 GB ram, 4800 max iops, 127GB local SSD)
Test Results
Experimentation Summary
These tests included the out-of-the-box installation of 3.2.0, 3.3.0 and 3.4.0
| Version | Execution Date | Variation | # of Requests | Mean Response Time in ms | Max Response Time in ms |
|---|---|---|---|---|---|
| 3.2.0 | 05/01/2020 | SQL Server ODS | 124,683 | 105 | 2660 |
| 3.3.0 | 05/01/2020 | SQL Server ODS | 147,923 | 90 | 1647 |
3.4.0 | 05/08/2020 | SQL Server ODS | 130,160 | 129 | 1884 |
| 3.4.0 | 05/21/2020 | PostgreSQL ODS | 117,741 | 191 | 2809 |
Key Take-Aways
- No two executions of the same code/configuration will result in the exact same mean response time — there is a degree of randomness in the Locust-based clients. Thus the difference between 130 ms, 135 ms, and 136 ms is not significant.
- Performance PostgreSQL backed ODS / API were shown to be comparable to that of the ODS / API backed by SQL Server ODS.
Server Stats
Overall the web and database server statistics do not show any serious concerns.
Database Server Stats
