TAG Meeting 2020-08-10 - Subgroup on Data Out
Eric Jansson
Participants
- Jean-Francois G
- Andrew R
- John Watson, San Diego County OE
- Patrick Yoho, Innovate Edu
- David Clements, Ed-Fi ALliance
- Vinaya Mayya, Ed-Fi Alliance
Notes
The main use case explored in the meeting was ODS-to-ODS replication. That use case has emerged as a higher priority since the Strategies for Data Out from the ODS/API SIG meet in 2019. Field evidence likewise points to data usage coming not from analytics or BI tools pointed at the ODS, but from analytics that are using a data warehouse or datamart downstream of the ODS.
Much discussion also reinforced that local data analytics needs seem to be shaping the design and schema of local data warehouses or datamarts. Given these, it seemed unclear that recommendations of the former Data Out SIG were indeed the right directions:
- A GraphQL interface on the ODS doesn't make sense if downstream data marts are actually driving analytics
- The Analytics Middle Tier is a fairly light interface on the ODS database, and so less work to maintain;but likewise, it may not be the right investment if the ODS is not powering analytics
- Ditto for composite APIs, but composite APIs as paroviding very narrow data out for operational use cases (e..g roster) may still make sense)
It was acknowledged that perhaps GraphQL and AMT were technologies to hydrate downstream datamarts. However, it was also noted that that is not how those technologies were envisioned, as a primary use case.
One field issue was raised on the call in this area: the lack of access to change queries on Suite 2 ODS API has resulted in significant latency that makes large ODS to ODS transfer. This is documented in https://tracker.ed-fi.org/browse/ODS-4261 However, given the movement of the overall community to Suite 3 platform (where this is not a problem) the TAG won't focus on this issue. It is important, but resolution is likely more a business prioritization decision, whereas the TAG focuses on technical architecture for the overall community.