Purpose

This document describes the logging policy in the Meadowlark API source code. In general, this policy seeks to balance the goals of providing sufficient information for an administrator to understand the health of the system and understand user interaction with the system with the equally important goals of protecting sensitive data and avoiding excessive log storage size.

Logging Principles

• Use structured logging for integration into log-monitoring applications (LogStash, Splunk, CloudWatch, etc.).
• Do not log sensitive data.
• Use an appropriate log level.
• Include a correlation / trace ID wherever possible, with the ID being unique to each HTTP request.
• Provide enough information to help someone understand what is going on in the system, and where, but
• Be careful not to make the log entries too large, thus becoming a storage problem.
• Logs will be written to the console, at minimum.
• If any transformation or business logic is necessary for writing an info or debug message, use the utility isDebugEnabled  and isInfoEnabled  functions first before executing that logic.

Log Levels 

Summary

Meadowlark will utilize the following levels when logging messages. These levels help the reader to understand if any remedial action is needed, and they allow the administrator to tune the amount of data being logged.

Examples

These examples are general guidelines and not 100% exhaustive.

Error

• Unhandled null reference
• Database connection / transaction failure after exhausting retry attempts

Warning

• A database connection / transaction failure occurred, but was recovered with an automatic retry

Informational

• Received an HTTP request
  • URL
  • clientId
  • traceId
  • verb
  • contentType
  • no payload
• Responded to an HTTP request
  • URL
  • response code
  • clientId
  • duration from time of receipt of HTTP request to response (milliseconds)
  • no payload
• Process startup and shutdown
• Database created

Debug

• Received an HTTP request → add anonymized payload
  • Replace potentially sensitive string and numeric data with null  before logging.
  • Could hard code restrictions to "known-to-be-sensitive" attributes, for example attributes on Student, Parent, and Staff.
  • However, that could fall short with a change to the data model.
  • Therefore, it will be safest to replace all string and numeric data.
  • One potential exception: descriptors.
    • descriptor values will never contain sensitive data;
    • since the other string and numeric values are anonymized, the descriptor value itself does not provide a side channel to sensitive information;
    • there is value to having this when debugging failed HTTP requests.
• Responded  to an HTTP request → add payload

  • Will require anonymization of the natural key fields when reporting a referential integrity problem

    Potential scenario:

    • Entity1 has natural key {personName, personId}.
    • Entity2 has a reference to Entity1
    • Post Entity2 with a {personName, personId} that do not exist. Then the response message will have is missing identity {\"personName\": \"the actual value\", \"personId\": ... }
• Entered a function
• About to connect to a service or run through an interesting algorithm
• Received information back from a service
  • Metadata only