ODS / API Load Testing Utility Cookbook

Introduction

In January of 2015, an initial load testing project was completed with the objective of understanding the performance characteristics of the Ed-Fi ODS / API under load. The initial study was limited in scope and simply measured performance of the API by simulating transactions to create new entities. In June 2015, this effort was expanded to include the full spectrum of API capabilities. This included Creating, Reading, Updating, and Deleting entities (i.e., performing the transactional CRUD operations), and uploading large files for bulk consumption by the back-end import services.

The Ed-Fi Alliance has published the Load Testing Utility used in the load test so that licensees can test their specific deployment environments. This document lists the load testing tool components and describes their use.

The components are:

• Load Testing Utility. This component applies a configurable, variable level of load to the ODS / API covering the transactional CRUD operations.

• File Uploader. This component submits files for processing by Bulk Services.

• Console Bulk Loader. This component is similar to the File Uploader, but bypasses the API and back-end processing. Instead, this component directly inserts a file into the database (which requires a direct connection to the database). This component is mentioned here as a utility that is useful in testing, but usage of this tool is covered in the documentation for the Console Bulk Loader.

• PowerShell Scripts and Batch Files. These scripts and batch files are provided to automate parts of the testing, and to allow users of the Load Testing Utility to run multiple tests simultaneously.

Audience

This document is written for a technical operator who will be using these utilities to perform load testing on their instance of an Ed-Fi ODS / API system. Business personnel and technical management stakeholders may benefit from the high-level detail.

Load Testing Utility Overview

The Load Testing Utility applies load to an Ed-Fi ODS / API instance and records metrics regarding request type, round-trip time to execute, and the result of the request.

The Load Testing Utility can be configured along essentially 5 axes. One axis is the CRUD operation mixture. Another axis is the Model Mixture, meaning stuff like "Create 4 discipline incidents for each Student." Another configurable axis is the number of threads that will be applying load, each thread simulating an additional client. In addition, there are parameters that specify the details about the test, for example the OAuth key/secret, the API URLs, and so forth. Finally, there are Load-Tester-specific configurations. Each configuration is outlined below.

Operation Mixture Parameters

• Create Ratio. Decimal indicating the ratio of create operations to perform.

• Read Ratio. Decimal indicating the ratio of read operations to perform.

• Update Ratio. Decimal indicating the ratio of update operations to perform.

• Delete Ratio. Decimal indicating the ratio of delete operations to perform.

Model Mixture Parameters

The purpose of the model mixture is to communicate an order of operation when creating entities so that referential requirements for subsequent models are created first. In addition, there are other instructions that are included such as Family, Order, and Exclude instructions (discussed below). The model mixture is configurable via an XML file, which allows for a detailed vocabulary. (Note that the command-line parameter is named OrderFamilyCsvFileName, which is a vestigial structure from when the configuration was in CSV format).

The XML file has a root entity of <entities> and can only contain child entities of the <entity> element. What follows are the attributes of the entity element.

• Family. Consists of a set of values [Core, People, Enrollment, Operations] and execute in that order. The purpose of this categorization is to easily set up scenarios that simulate these activities in real implementations.

• Order. This load order starts over for each Family.

• Weight. Typically 1 for a one-to-one. These values are ratios to each other, so a 1 and a 2 would evaluate to ratios of 1/3 and 2/3.

• Name. The name of the entity. This name must match the SDK aggregate.

• [ExcludeFromCreate]. Optional. Set to true if you wish to ensure that this entity is never created. Used when the data set is pre-populated with Ed-Fi Descriptors and Ed Orgs for an ODS / API instance.

• [ExcludeFromRead]. Optional. Set to true if you wish to ensure that this entity does not participate in any read operations.

• [ExcludeFromUpdate]. Optional. Set to true if you wish to ensure that this entity does not participate in any update operations.

• [ExcludeFromDelete]. Optional. Set to true if you wish to ensure that this entity does not participate in any delete operations. This is ideal for root-level entities in EdOrg, EdOrgCalendar, Student, Parent, etc.

• [ExcludeFromAllButInitialCreate]. Optional. Set to true if you wish to create these entities in the warm up phase of creation, but exclude the entity from all subsequent operations later in the testing cycle. This is ideal for root-level entities in EdOrg, EdOrgCalendar, Student, Parent, and so forth.

Thread Parameters

• NumberOfThreads. This is the number of threads that you want to create for applying load to the ODS / API. Each thread will be applying constant load to the server in either a synchronous or asynchronous pattern.

• NumberOfWorkGenerationThreads. The entire application is a producer / consumer pattern. The Number of Threads pattern is the number of threads applying load to the ODS / API, but on the other end of the spectrum are workers that are generating the work that needs to be done, and populating the queue of work to be done.
• ExecuteSynchronously. In synchronous mode, the thread waits for a response from the server before sending another request. In asynchronous mode, the thread continues to launch each request as fast as it can, not waiting for the response.

ODS Parameters

These are all fairly standard for any tool that needs to speak to the ODS / API. They are listed briefly here for reference, we won’t go into detail about each argument.

• Token.

• OAuthUrl.

• ApiUrl.

• Key.

• Secret.

• CurrentSchoolYear.

Load Testing-Specific Parameters

• IsParent. When running multiple load tester utilities, the IsParent flag establishes itself as the master. All client load testers should be started first with the IsChild flag set to true. Finally the Parent runs with the IsParent flag, and that action sets off the run.

• IsChild. When running multiple load tester utilities, running many different clients with the IsChild mode mades them all wait for the parent to start so that they can begin their work.

• ParentRunId. This is the GUID that identifies the test run. All clients should use the same ParentRunId so that all of the events line up in the load testing database event table.

• ClientId. This is the GUID that uniquely identifies each client. This should be unique. If not specified, then each client will create its own random GUID.

• TimeToRunInMinutes. If not specified, the load testing will run until manually stopped. Entering this argument will stop the test once the amount of time has expired.

• LimitNumberOfInFlightRequests. (Experimental.) This is the argument where you can specify the number of requests that should be in flight at any one time. Be careful with this argument. If set too low, no matter how high your threads are, you will be throttled. If set too high, you may get into trouble with weird events where things don’t run.

Understanding the Load Testing Utility Reports

The Load Testing Utility records a ton of information about the test that is running. The method by which reporting data is persisted to a non-volatile storage medium is well-abstracted to allow for a variety of methods. The as-shipped configuration uses a SQL Server Database and Entity Framework. Care was taken to ensure that while the process is saving data to the database it is not taking away execution cycles for applying load. What follows are details about the information that the Load Testing Utility captures.

Tables

• RunInfoes.

• Events (1 Run Info to many Events).

RunInfoes Table

• RunId uniqueidentifier not null PK.

• CreatePercent decimal (18, 2) not null.

• UpdatePercent decimal(18,2) not null.

• DeletePercent decimal (18,2) not null.

• NumberOfThreads int not null.

• TimeToRunMinutes int not null. For calculations use the Max and Min DateTime of the event startdatetime column. Important that this is the parameter that was sent in.

• NumberOfWorkGenerationThreads int not null.

• RunStart datetime. For calculations use the Max and Min DateTime of the event startdatetime column.

• RunEnd datetime. For calculations use the Max and Min DateTime of the event startdatetime column.

Events Table

• RunId uniqueidentifier not null PK. FK to RunInfoes.RunId.

• Id uniqueidentifier not null PK.

• StartDateTime datetime not null. The date and time when the request began.

• EndDateTime datetime not null. The date and time when the request ended.

• EventType not null. Either [Create-0, Read-1, Update-2, Delete - 3].

• SdkModelType nvarchar(max) null. The aggregate name.

• EventDisposition int not null. Either [1-Success, 2-Error].

• HttpStatusCode nvarchar(max) null. The HttpStatus code returned from the request object. See the HTTP RFC for all return codes.

• Error nvarchar(max) null. If an error was encountered, this will contain any content that is returned with the error as well.

Running the Load Testing Utility

The Load Testing Utility runs in three different phases. The first phase is the warm up. During the warm up phase, some housecleaning is done to synch up Ed-Fi Descriptors and types with this ODS / API as well as create a buffer of entities that will be the critical mass required in order to keep the process running. Remember that we are executing CRUD operations. Before the utility can read, update or delete, it must first have created a sufficient number of items. It is important when configuring the operation mixture that the number of creates is sufficiently higher than the number of deletes.

The second phase is the burn in. The burn in phase is when all of the threads kick into high gear and pummel the ODS / API with CRUD operations. The algorithm uses a bucketed distribution mechanism, which is a fancy way of saying that, provided you run on a long enough time and regardless of how long you run, the numbers will always fall within a general ratio. So if you have 2c 1r .5u .3 d, then that adds up to 3.8. So 2 times out of 3.8 we will be creating, 1 time of out 3.8 we will be reading, .5 out of 3.8 we will be updating, and .3 out of 3.8 we will be deleting.

Once we know what operation we need to execute, the next step is determining which entity type we should create. The algorithm strives to have the most equally dispersed and diverse usage of the types in the model. Keep in mind that some types have been specifically disqualified from a particular operation or set of operations in the configuration.

Now we know what type of operation to execute, and what type of aggregate we should execute on. We use that information in order to retrieve the correct type of factory from IoC, and then have the factory create our Unit of Work, which gets enqueued onto the UnitOfWorkManager.

Finally, once the application has been running for its allotted time (or the operator sends the kill switch) we enter the final phase, the burn down. The following takes place during the burn down phase:

• The Work Generators stop working to fill up the queue.

• The Load Generators finish whatever they were doing, but do not pick up any more work.

• The Application pauses to wait for any outstanding requests to be returned and to be persisted to the data store.

Load Testing for Operators

This section provides a brief walkthrough for Load Testing Utility users. It should be noted that the ideal planned solution involved a much improved user experience, with starting and stopping a horde of clients from one simple-to-use UI. Because of budgetary and time constraints some of these nice-to-have features were removed from the initial project scope.

In single-client mode, all testing is done from one computer. Starting the utility by omitting IsParent or IsChild will do the trick. While testing it was observed that in order to really saturate one IIS box, more than one client machine was needed. Whether or not more than one client is required has a lot to do with the unique network infrastructure and topology of the environment being tested. If you need to run more than one client, it is supported. The process for running this configuration is a bit convoluted, but it can be done. Essentially, you will need to remote into all client boxes, start up the all of the clients first, and then start up a single parent that will then command the clients to begin.

Prerequisites for Load Testing an ODS / API Instance

Prior to running the Load Testing Utility, you need to set up a sandbox for the utility to work against. This sandbox can be easily set up by using the Admin tool, and creating a Minimal sandbox. This is the recommended approach as it will correctly configure security and set up a key and secret. If unable to use the Admin tool, the following requirements are necessary for a functioning sandbox:

• The ODS database must be populated with descriptors.
• At least one Education Organization must be defined.
• An API client, application, and vendor must be defined in EdFi_Admin in order to furnish the Utility with a key/secret pair.
• Said API client must be given a claims set with sufficient permissions. The as-shipped "Ed-Fi Sandbox" claim set defined in the development branch has correct permissions.
• Said API client must be associated with at least one Education Organization.

Running the Load Testing Utility in Single-Client Mode

Running the Load Testing Utility without specifying the IsChild or IsParent will cause the utility to run in Single-Client Mode. In this mode, the warm up phase begins right away, then we enter the burn in phase until a stop command is issued or the configured amount of time has elapsed.

C:\LoadGenerationV2\LoadGeneration2.exe --CreatePercent 1 --ReadPercent 2 --UpdatePercent 0.5 --DeletePercent 0.3 --NumberOfThreads 1 --NumberOfWorkGenerationThreads 1 --OAuthUrl http://localhost:54746/ --ApiUrl http://localhost:54746/api/v2.0/ --Key RvcohKz9zHI4 --Secret E1iEFusaNf81xzCxwHfbolkC --CurrentSchoolYear 2016 --OrderFamilyCsvFileName C:\LoadGenerationV2\LoadTestingEntityMetaData.xml --TimeToRunInMinutes 15

Running the Load Testing Utility in Multiple-Client Mode

You should always have all clients running in client mode before you run the parent. This is a two-step process.

Step 1. Start all Clients in Client Mode by passing the IsChild flag. You will know the client is ready when it prints out “Waiting for Parent Process to begin”.

C:\LoadGenerationV2\LoadGeneration2.exe --CreatePercent 1 --ReadPercent 2 --UpdatePercent 0.5 --DeletePercent 0.3 --NumberOfThreads 1 --NumberOfWorkGenerationThreads 1 --OAuthUrl http://localhost:54746/ --ApiUrl http://localhost:54746/api/v2.0/ --Key RvcohKz9zHI4 --Secret E1iEFusaNf81xzCxwHfbolkC --CurrentSchoolYear 2016 --OrderFamilyCsvFileName C:\LoadGenerationV2\LoadTestingEntityMetaData.xml --TimeToRunInMinutes 15 --IsChild

Step 2. Once all Clients have started, then Start the parent process by passing the IsParent flag in the list of arguments. Once that starts, it should signal to all of the children to begin running.

C:\LoadGenerationV2\LoadGeneration2.exe --CreatePercent 1 --ReadPercent 2 --UpdatePercent 0.5 --DeletePercent 0.3 --NumberOfThreads 1 --NumberOfWorkGenerationThreads 1 --OAuthUrl http://localhost:54746/ --ApiUrl http://localhost:54746/api/v2.0/ --Key RvcohKz9zHI4 --Secret E1iEFusaNf81xzCxwHfbolkC --CurrentSchoolYear 2016 --OrderFamilyCsvFileName C:\LoadGenerationV2\LoadTestingEntityMetaData.xml --TimeToRunInMinutes 15 --IsParent

File Uploader Overview

The File Uploader is a console application that submits files to the API. The ODS / API has a back-end process that will import bulk XML files into the ODS. A transactional API is used to create operations, upload files for that operation, commit the operation to start back-end processing. Then, an API is used in order to query for status of the operation as it is imported. Finally, once the import is complete, the API can be used to query for any execution errors that may have been encountered as a result of execution.

The File Uploader simplifies this whole process because the caller needs only provide the ODS Information and either the path to a file or a folder, and it will handle the rest for you. It will even download any execution errors to a folder of your choosing, one for each file uploaded. The File Uploader was developed for a project with the Michigan Department of Education. It will eventually be pulled into the Ed-Fi Alliance code base. What follows are the arguments required to execute the File Uploader:

• Secret.

• Key.

• AdminUrl.

• ApiUrl.

• File. The file to upload. If you choose to specify a file, then Folder is omitted.

• Folder. The folder to upload. All files in the folder will be uploaded. If you choose to specify a file, then Folder is omitted.

• SchoolYear.

• ProcessingErrorsOutputFolderPath. This is where any processing errors will be downloaded. All are downloaded, one for each file uploaded (if there were errors).

Testing PowerShell Scripts

Generally speaking, testing processes should be automated to support repeatable findings, and to reduce the chance of human error. PowerShell scripts were developed by the Ed-Fi Alliance to facilitate automated testing, and the scripts are included in the Load Testing Utility distribution. What follows is a breakdown of those scripts.

There are two main script files, one for Console Bulk Loader (load.ps1) and one for File Uploader (FileUploader.ps1). They both follow the same program flow: record number of records to an output file, perform some work, record number of records and time to execute to output file. Repeat until all work has been performed. Once the test is concluded, calculate speed in terms of number of records inserted over the time-to-execute period, and use that information to compute a number of records per second.