August 28, 2024, 11:00 am - 11:45 am central. Contact @Stephen Fuqua for the meeting invitation.

Agenda

• Demonstration of current work

• Review the roadmap

• Design Questions

Demonstration

Open

Roadmap

Milestone 0.4

Architecture

Storage of Metadata

This is a sample message in Kafka, created from Debezium:

Open

• _lastModifiedDate, _etag (and _lineage in the future?)

• Store as metadata or add to the document itself? What should downstream consumers receive?

• Already sending “API Standard” rather than “Data Standard”. Example: “SchoolYearType” instead of “SchoolYear”.

Tenancy Routing

GitHub Discussion for asynchronous conversation: Multitenancy / routing / instance management

There are the requirements gathered from the multitenancy / routing / instance management workgroup:

• Allow ODS instances to reside on separate database servers.

• Support various database segmentation strategies: ODS per year, ODS per district, ODS per district per year, and ODS per tenant. (Reference: Segmentation strategies in ODS API)

• Enable API administrators to choose between implicit routes (segmentation information not visible in the routes) and explicit routes (Reference: Context-Based Routing).

• Provide isolated configuration for client and security setup by tenants, making it easier to onboard new tenants or remove existing ones.

Authentication

Given that client credentials will ultimately be in a 3rd party OAuth provider… do we want to give API clients the 3rd party OAuth token URL, or put it behind a proxy? If behind a proxy, three possible options:

• Data Management Service

• DMS Configuration Service

• or API gateway / load balancer

Logging

Showing output from docker logs:

Initial Startup

• First several lines are from a Bash script

• Next block of lines, before the first one with a date/time stamp, is obviously from initial database setup.

• any concern about these lines not having date/time stamp or a log level? If so, likely we can remedy that with a small user story. But if there is no concern, then it is not worth spending any time on it.

Happy Path Flow

• Most of this is telemetry information. Should this be going into the regular console log, or should we move that off to another log sink? (possibly using Open Telemetry).

• We have not been very careful yet with our logging messages and we should do a little cleanup to make these more useful. For example:

  INF Request starting HTTP/1.1 GET http://localhost:8080/data/ed-fi/schoolYearTypes - null null INF {"method":"GET","path":"/data/ed-fi/schoolYearTypes","traceId":"0HN67FTPTI90R:00000001","clientId":{"Value":"localhost:8080","HasValue":true,"Host":"localhost","Port":8080}}

  • We ended up logging the GET request twice

  • We used structured logging in the second entry, but nowhere else.

  • Sidenote: the traceId that you see here was auto-generated by ASP.NET. However, we have already built in support for using a correlation / trace ID header if it is present.

  • I don’t know what that clientId value is about. Certainly not showing anything useful. I would have expected clientId to be something we pull from an auth token claim, but there was no auth token in this case.

  • structured logs or unstructured? If we write our logging code correctly, we may be able to control this purely through Serilog settings, letting you choose structured or unstructured.

  • The date/time and the level should probably be in the structured message.

• Most of these debug messages show entry into a function. With the TraceId present, you can follow the performance for an individual request.

  • The function entered is not consistently separated from TraceId with “space dash space”.

  • Wouldn’t a structured message be easier to use here?

Client Error

Performed a POST operation with an invalid payload.

• The debug messages include the raw input value with no data masking. In Meadowlark, we kept the structure while masking the values. Shall we do the same in the DMS?

• Note that the client error is not treated as an application error (with ERR log level).

Application Error

Stopped the PostgreSQL container. Client received the following response:

Console log:

• We deliberately left the stack trace in place.

  • Not “structured” today

• bug - Some of the DBG messages are mistakenly missing the TraceId in their output.

Application Log - Circuit Breaker

We’ve implemented the circuit breaker pattern, with configurable settings. Sample log after “a bunch” of calls to the DMS while PostgreSQL is down:

• Note the WRN, FTL, and ERR logs.

• Probably better to try to avoid the final stack trace, on the circuit breaker, as that does not add useful information.

• However, a search on 0HN67FTPTI917 reveals that the actual error that finally cause the circuit to open has not been logged.

• Should the FTL message have simply been an ERR? After all, the application is still running, and the database connection may restore itself after a time.

Restart PostgreSQL

• We shouldn’t be logging the query result when the circuit successfully re-closes.