How To: Enable logging API request and response content

Owned by Vinaya Mayya
Last updated: Jun 04, 2024 by Matthew Jackson
7 min read

This example will show how the content of the requests and responses for the web API can be logged. Because the content of API requests and responses may contain sensitive data, this information can only be logged to a table in the ODS database that handled the API call. Furthermore, the duration for which this logging can be enabled is limited because it has the potential to generate a large amount of data.

The steps to enable web API request/response logging can be summarized:

Step 1. Create a table for storing the log in the ODS database

Because the content of an API transaction may contain sensitive data, the only supported storage for logging the content of the transaction is the ODS database used to process it. This ensures that the same level of security provided for the data at rest in the ODS database app also applies to the data stored in the log entries required table can be created by running the following SQL script on the ODS database:

Using MS SQL Server

CREATE TABLE dbo.RequestResponseContentLog (
    Id int IDENTITY (1, 1) NOT NULL,
    Date datetime NOT NULL,
	Thread varchar(255) NOT NULL,
	CorrelationId varchar(255) NOT NULL,
    ApiClientId varchar(255) NULL,
    Level varchar(50) NULL,
	RequestUrl varchar(255) NULL,
	RequestMethod varchar(10) NULL,
	ProfilesHeader varchar(255) NULL,
	RequestBody nvarchar(max) NULL,
	ResponseBody nvarchar(max) NULL,
    ResponseMessage varchar(255) NULL,
    Exception varchar(2000) NULL
)

Using PostgreSQL

CREATE TABLE edfi.RequestResponseContentLog (
Id SERIAL PRIMARY KEY NOT NULL,
Date TIMESTAMP NOT NULL,
Thread varchar (255) NOT NULL,
CorrelationId varchar (255) NOT NULL,
ApiClientId varchar (255) NULL,
Level varchar (50) NULL,
RequestUrl varchar (255) NULL,
RequestMethod varchar (10) NULL,
ProfilesHeader varchar (255) NULL,
RequestBody varchar NULL,
ResponseBody varchar NULL,
ResponseMessage varchar (255) NULL,
Exception varchar (2000) NULL
)

Step 2. Update the EdFi.Ods.WebApi log4net configuration

The log4net configuration for EdFi.Ods.WebApi must be updated to include the settings needed for request/response content logging.

Add the following elements inside the root <log4net> element of the active log4net configuration file for the web API, typically either log4net.development.config or log4net.config.

By default, only requests that result in an exception are logged. To log the request/response content of all calls to the web API, change <threshold value="ERROR" /> to <threshold value="INFO" /> in both of the places, it appears in the log4net configuration.

Using MS SQL Server

  <logger name="RequestResponseContentLogger" additivity="false">
	<level value="ERROR" />
	<appender-ref ref="RequestResponseContentAppender" />
  </logger>
  <appender name="RequestResponseContentAppender" type="log4net.Appender.AdoNetAppender">
    <bufferSize value="0" />
    <threshold value="ERROR" />
    <connectionType value="Microsoft.Data.SqlClient.SqlConnection, Microsoft.Data.SqlClient" />
    <commandText value="INSERT INTO RequestResponseContentLog ([Date],[Thread],[CorrelationId],[ApiClientId],[Level],[RequestUrl],[RequestMethod],[ProfilesHeader],[RequestBody],[ResponseBody],[ResponseMessage],[Exception]) VALUES (@log_date, @thread, @correlation_id, @api_client_id, @log_level, @request_url, @request_method, @profiles_header, @request_body, @response_body, @response_message, @exception)" />
    <parameter>
        <parameterName value="@log_date" />
        <dbType value="DateTime" />
        <layout type="log4net.Layout.RawTimeStampLayout" />
    </parameter>
    <parameter>
        <parameterName value="@thread" />
        <dbType value="String" />
        <size value="255" />
        <layout type="log4net.Layout.PatternLayout">
            <conversionPattern value="%thread" />
        </layout>
    </parameter>
    <parameter>
        <parameterName value="@correlation_id" />
        <dbType value="String" />
        <size value="255" />
        <layout type="log4net.Layout.PatternLayout">
            <conversionPattern value="%property{CorrelationId}" />
        </layout>
    </parameter>
    <parameter>
	    <parameterName value="@api_client_id" />
	    <dbType value="String" />
	    <size value="255" />
	    <layout type="log4net.Layout.PatternLayout">
		    <conversionPattern value="%property{ApiClientId}" />
	    </layout>
    </parameter>
    <parameter>
        <parameterName value="@log_level" />
        <dbType value="String" />
        <size value="50" />
        <layout type="log4net.Layout.PatternLayout">
            <conversionPattern value="%level" />
        </layout>
    </parameter>
    <parameter>
	    <parameterName value="@request_url" />
	    <dbType value="String" />
	    <size value="255" />
	    <layout type="log4net.Layout.PatternLayout">
		    <conversionPattern value="%property{RequestUrlWithQueryString}" />
	    </layout>
    </parameter>
    <parameter>
	    <parameterName value="@request_method" />
	    <dbType value="String" />
	    <size value="10" />
	    <layout type="log4net.Layout.PatternLayout">
		    <conversionPattern value="%property{RequestMethod}" />
	    </layout>
    </parameter>
    <parameter>
	    <parameterName value="@profiles_header" />
	    <dbType value="String" />
	    <size value="255" />
	    <layout type="log4net.Layout.PatternLayout">
		    <conversionPattern value="%property{ProfilesHeader}" />
	    </layout>
    </parameter>
    <parameter>
        <parameterName value="@request_body" />
        <dbType value="String" />
        <size value="-1" />
        <layout type="log4net.Layout.PatternLayout">
            <conversionPattern value="%property{RequestBody}" />
        </layout>
    </parameter>
    <parameter>
	    <parameterName value="@response_body" />
	    <dbType value="String" />
	    <size value="-1" />
	    <layout type="log4net.Layout.PatternLayout">
		    <conversionPattern value="%property{ResponseBody}" />
	    </layout>
    </parameter>
    <parameter>
        <parameterName value="@response_message" />
        <dbType value="String" />
        <size value="255" />
        <layout type="log4net.Layout.PatternLayout">
            <conversionPattern value="%message" />
        </layout>
    </parameter>
    <parameter>
        <parameterName value="@exception" />
        <dbType value="String" />
        <size value="2000" />
        <layout type="log4net.Layout.ExceptionLayout" />
    </parameter>
  </appender>

Using PostgreSQL

  <logger name="RequestResponseContentLogger" additivity="false">
	<level value="ERROR" />
	<appender-ref ref="RequestResponseContentAppender" />
  </logger>
  <appender name="RequestResponseContentAppender" type="log4net.Appender.AdoNetAppender">
    <bufferSize value="0" />
    <threshold value="ERROR" />
     <connectionType value="Npgsql.NpgsqlConnection, Npgsql" />
    <commandText value="INSERT INTO edfi.RequestResponseContentLog (date,thread,correlationid,apiclientid,level,requesturl,requestmethod,profilesheader,requestbody,responsebody,responsemessage,exception) VALUES (@log_date, @thread, @correlation_id, @api_client_id, @log_level, @request_url, @request_method, @profiles_header, @request_body, @response_body, @response_message, @exception)" />
    <parameter>
        <parameterName value="@log_date" />
        <dbType value="DateTime" />
        <layout type="log4net.Layout.RawTimeStampLayout" />
    </parameter>
    <parameter>
        <parameterName value="@thread" />
        <dbType value="String" />
        <size value="255" />
        <layout type="log4net.Layout.PatternLayout">
            <conversionPattern value="%thread" />
        </layout>
    </parameter>
    <parameter>
        <parameterName value="@correlation_id" />
        <dbType value="String" />
        <size value="255" />
        <layout type="log4net.Layout.PatternLayout">
            <conversionPattern value="%property{CorrelationId}" />
        </layout>
    </parameter>
    <parameter>
	    <parameterName value="@api_client_id" />
	    <dbType value="String" />
	    <size value="255" />
	    <layout type="log4net.Layout.PatternLayout">
		    <conversionPattern value="%property{ApiClientId}" />
	    </layout>
    </parameter>
    <parameter>
        <parameterName value="@log_level" />
        <dbType value="String" />
        <size value="50" />
        <layout type="log4net.Layout.PatternLayout">
            <conversionPattern value="%level" />
        </layout>
    </parameter>
    <parameter>
	    <parameterName value="@request_url" />
	    <dbType value="String" />
	    <size value="255" />
	    <layout type="log4net.Layout.PatternLayout">
		    <conversionPattern value="%property{RequestUrlWithQueryString}" />
	    </layout>
    </parameter>
    <parameter>
	    <parameterName value="@request_method" />
	    <dbType value="String" />
	    <size value="10" />
	    <layout type="log4net.Layout.PatternLayout">
		    <conversionPattern value="%property{RequestMethod}" />
	    </layout>
    </parameter>
    <parameter>
	    <parameterName value="@profiles_header" />
	    <dbType value="String" />
	    <size value="255" />
	    <layout type="log4net.Layout.PatternLayout">
		    <conversionPattern value="%property{ProfilesHeader}" />
	    </layout>
    </parameter>
    <parameter>
        <parameterName value="@request_body" />
        <dbType value="String" />
        <size value="-1" />
        <layout type="log4net.Layout.PatternLayout">
            <conversionPattern value="%property{RequestBody}" />
        </layout>
    </parameter>
    <parameter>
	    <parameterName value="@response_body" />
	    <dbType value="String" />
	    <size value="-1" />
	    <layout type="log4net.Layout.PatternLayout">
		    <conversionPattern value="%property{ResponseBody}" />
	    </layout>
    </parameter>
    <parameter>
        <parameterName value="@response_message" />
        <dbType value="String" />
        <size value="255" />
        <layout type="log4net.Layout.PatternLayout">
            <conversionPattern value="%message" />
        </layout>
    </parameter>
    <parameter>
        <parameterName value="@exception" />
        <dbType value="String" />
        <size value="2000" />
        <layout type="log4net.Layout.ExceptionLayout" />
    </parameter>
  </appender>

Step 3. Set the logging duration

Update the following setting in the appsettings.json for EdFi.Ods.WebApi to reflect the number of minutes request/response content should be logged. The measurement of this duration begins when the first API request is received after the application startup. 

"LogRequestResponseContentForMinutes": 0