Participants

Agenda

1. Bulk data exchange over API (see materials linked in deck for meeting TAG Meeting 2022-08-18)

2. Further TAG review and input on release cycle planning

Materials

API Evolution

Bulk API - 

• Bulk Data Proxy Service

• past work

  • SIG: Bulk Data Exchange Over REST API

  • Design work: https://techdocs.ed-fi.org/download/attachments/45777870/Batch%20Modalities%20of%20Data%20Exchange%20-%20Final.pdf?api=v2

Notes

API Evolution plan

There were no red flags raised on a go-ahead on this plan. There were some key points raised:

• the managed provider perspective is missing - how will this impact their roadmaps and services? Will they be able to coordinate on fewer versions or be forced to stay "up to date" and therefore deploy many?

• the pivot to an API version is a big change for the ecosystem, but is needed. How do we make this work?

• what about use cases where the API may need to be running for multiple years

  • state use cases? Our sense is that 3 years will cover most of these

  • archival use cases? Maybe the ODS needs to be "emptied" into a data lake or other structure

Bulk API

There was a great discussion about this point. Some cases of where the problem has occurred were shared - cases where a sync can take hours or possibly even longer than a day. For example, this can happen when a district turns on the API mid-year. This can also occur in areas where there is a lot of data, which seemed to be limited to Attendance and Gradebook domains (and for attendance, mainly when there is positive attendance), and possibly Finance data (as Finance data seems to be done through large scale "re-syncs" (it was not clear why or if this is the way that finance data has to work).

Key discussion points

• It wasn't clear that the field and API clients had yet exhausted the design possibilities for solving the issue, for example by synchronizing data in parallel threads/processes. One vendor shared that they felt that their technical solution was working OK to address performance issues. There were also some benchmarks shared (100K records in 9 minutes, I believe) that suggested that there were more technical options available.

• It was also raised that the presence of a bulk API spec could result in more vendor work, as vendors would effectively be required to support multiple implementations

• The possibility of developing a "custom" solution on  the Ed-Fi platform that a) has the same behavior as the current API, but b) is more performant was also raised. This could be done in limited domains,