Summary
This article describes the methodology and result of performance testing on the Ed-Fi ODS / API v5.2.
In brief, performance testing did not uncover any significant concerns in performance relative to previous suite 3 versions.
Article Contents:
| Table of Contents | ||||
|---|---|---|---|---|
|
Test Methodology
Volume Tests
Volume testing of v5.2 occurred in July 2021, using the Locust-based suite 3 performance testing framework (an Exchange contribution available in GitHub). This volume test covers the resources and HTTP verbs described in the Ed-Fi SIS vendor certification process. It runs for 30 minutes, spawning 30 clients that run simultaneously to perform tens of thousands of operations.
Data Set
Populated Template data set, which contains approximately 1000 students.
Azure Test Lab Virtual Machines
The test lab environment used a three-server setup: one each for the database, web applications, and the Locust performance testing. VM "Sizes" listed here, such as "DS11_v2", are Microsoft-defined names for the specs of an Azure VM. Key specs are listed beside these size names. These sizes were chosen as their specs are comparable to those of the Vendor Certification VMs but have SSD disks to more closely match a production environment.
Database VM
Image: Free SQL Server License: SQL Server 2017 Developer on Windows Server 2016
Size: DS11_v2 (Standard, 2 vcpus, 14 GB ram, 6400 max iops, 127GB local SSD)
Web VM
Image: Windows Server 2016 Datacenter
Size: B2ms (Standard, 2 vcpus, 8 GB ram, 4800 max iops, 127GB local SSD)
Test Runner VM
Image: Windows Server 2016 Datacenter
Size: B2ms (Standard, 2 vcpus, 8 GB ram, 4800 max iops, 127GB local SSD)
Test Results
Experimentation Summary
These tests included the out-of-the-box installation of 3.3.0, 5.0.1, and 5.2.
| Version | Execution Date | # of Requests | Mean Response Time in ms | Max Response Time in ms |
|---|---|---|---|---|
| 3.3.0 |
| 64,687 | 58 | 9797 |
5.0.1 |
| 63,943 | 65 | 9848 |
5.2 (Change Queries Enabled) |
| 63,814 | 73 | 9885 |
Key Take-Aways
- No two executions of the same code/configuration will result in the exact same mean response time — there is a degree of randomness in the Locust-based clients. Thus the difference between 130 58 ms, 135 65 ms, and 136 73 ms is not significant.
- More thorough analysis with larger Northridge dataset could reveal more details.
- A higher max response time was recorded during this test run on all versions of the API tested. On analyzing this further, volume distribution shows that 99% of the requests had similar response times as previous runs and shows a consistent behavior across releases. This leads to conclude that the max response time outliers are possibly due to environmental changes overtime.
The table below shows volume distribution for some recourses across API versions. The 50% column here represents the time under which 50% of the requests were completed.
| Version | Method | # requests | 50% | 66% | 75% | 80% | 90% | 95% | 98% | 99% | 100% |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 3.3.0 | POST /data/v3/ed-fi/parents | 535 | 62 | 78 | 78 | 93 | 110 | 130 | 170 | 230 | 9700 |
| 3.3.0 | POST /data/v3/ed-fi/staffs | 2051 | 78 | 93 | 93 | 110 | 130 | 160 | 220 | 550 | 9000 |
| 3.3.0 | POST /data/v3/ed-fi/students | 4729 | 46 | 62 | 62 | 78 | 93 | 110 | 170 | 1800 | 9700 |
| 5.0.1 | POST /data/v3/ed-fi/parents | 547 | 62 | 78 | 78 | 93 | 110 | 130 | 190 | 230 | 9000 |
| 5.0.1 | POST /data/v3/ed-fi/staffs | 2078 | 78 | 93 | 93 | 110 | 130 | 160 | 200 | 270 | 9700 |
| 5.0.1 | POST /data/v3/ed-fi/students | 4809 | 46 | 62 | 62 | 63 | 93 | 110 | 160 | 230 | 9800 |
| 5.2 | POST /data/v3/ed-fi/parents | 557 | 62 | 62 | 78 | 78 | 110 | 130 | 190 | 230 | 8300 |
| 5.2 | POST /data/v3/ed-fi/staffs | 2061 | 62 | 78 | 78 | 93 | 120 | 160 | 270 | 530 | 9300 |
| 5.2 | POST /data/v3/ed-fi/students | 5599 | 46 | 62 | 62 | 63 | 78 | 94 | 130 | 1060 | 9800 |
Server Stats
Overall the web and database server statistics do not show any serious concerns.
Web Server Stats
Database Server Stats
