JSON Configuration Transformation

Owned by Jim McKay (Deactivated)
Last updated: Sep 10, 2020 by Stephen Fuqua
7 min read

Problem

Microsoft has changed how configuration works for net core applications (https://docs.microsoft.com/en-us/aspnet/core/fundamentals/configuration/?view=aspnetcore-3.1) . In the past we would use a XML transformation to adjust our settings on the fly and/or at deployment time. With the migration to net core, we have to address how we want to work with deployments and our local environments.

Background

Web Applications (Web Api and/or MVC)

Microsoft suggests the use of CreateDefaultBuilder. When invoked the class provides the default configuration in the following order:

  1. ChainedConfigurationProvider : Adds an existing IConfiguration as a source. In the default configuration case, adds the host configuration and setting it as the first source for the app configuration.
  2. appsettings.json using the JSON configuration provider.
  3. appsettings.Environment.json using the JSON configuration provider.
  4. App secrets when the app runs in the Development environment.
  5. Environment variables using the Environment Variables configuration provider.
  6. Command-line arguments using the Command-line configuration provider

This gives us the ability to derive settings via inheritance. For example you can force a settings change using the command line without modifying the appsettings.json file. This pattern also allows for customized environments that are developer specific. For example a developer wants to use SQL server 2019 on a Linux container, and they have mapped the port to 1433 instead of port 1432, and/or a developer has a named instance version of SQL server.

Console Applications

Console applications can also benefit from this pattern, however this is done by including the providers for configuration as packages. For example:

<ItemGroup>
    <PackageReference Include="Microsoft.Extensions.Configuration" Version="3.1.3" />
    <PackageReference Include="Microsoft.Extensions.Configuration.FileExtensions" Version="3.1.3" />
    <PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="3.1.3" />
</ItemGroup>

Recommendation

Though we can use environment variables, we are choosing to exclude that usage for the ODS/API as a default implementation; however, environment variables are available and can be utilized as a deployment mechanism. As a solution for development and we will favor the usage a specific locals settings file as follows:

This pattern should be used for all of the ODS/API tools.

appsettings.json

In this main file, we will have the settings setup for initdev and this file will be checked in. This file will be added to the .gitignore file so that changes are not accidentally checked in. Test assemblies also will have their own specific appsettings.json file.

appsettings.local.json

This file is for customization for development overrides. The file also will be to the .gitignore file so that it is not accidentally checked in. Power shell scripts will need to be updated to address the overrides used by the TestHarness etc. We will include a checked in appsettings.local.json.example that users can copy for usage. Documentation will be updated to include how to use this solution.

Connection Strings

The appsettings.json file has a predefined connection strings section. We have a couple of options to address the use case of multiple database support:

Option 1: ConnectionStringsByEngine

Define a new section named ConnectionStringsByEngine that would be a dictionary of connection strings keyed off the database engine. For example:

{
  "ConnectionStringsByEngine": {
    "sqlServer": {
      "EdFi_Admin": "Server=(local); Database=EdFi_Admin; Trusted_Connection=True; Application Name=EdFi.Ods.WebApi;",
      "EdFi_Security": "Server=(local); Database=EdFi_Security; Trusted_Connection=True; Persist Security Info=True; Application Name=EdFi.Ods.WebApi;",
      "EdFi_Ods": "Server=(local); Database=EdFi_{0}; Trusted_Connection=True; Application Name=EdFi.Ods.WebApi;"
    },
    "postgreSql": {
      "EdFi_Admin": "Host=localhost; Port=5432; Username=postgres; Database=EdFi_Admin; Application Name=EdFi.Ods.WebApi;",
      "EdFi_Security": "Host=localhost; Port=5432; Username=postgres; Database=EdFi_Security; Application Name=EdFi.Ods.WebApi;",
      "EdFi_Ods": "Host=localhost; Port=5432; Username=postgres; Database=EdFi_{0}; Application Name=EdFi.Ods.WebApi;"
    }
  }
}

We would need to modify the ConfigConnectionStringsProvider to build the dictionary based on DatabaseEngine.

The benefit of this solution, this makes extension more flexible without any future changes to the connection strings provider.

Option 2. Separate Sections

Define two sections for the connections strings. SqlServerConnectionStrings, and PostgreSqlConnectionStrings. For example:

{
  "SqlServerConnectionStrings": {
    "EdFi_Admin": "Server=(local); Database=EdFi_Admin; Trusted_Connection=True; Application Name=EdFi.Ods.WebApi;",
    "EdFi_Security": "Server=(local); Database=EdFi_Security; Trusted_Connection=True; Persist Security Info=True; Application Name=EdFi.Ods.WebApi;",
    "EdFi_Ods": "Server=(local); Database=EdFi_{0}; Trusted_Connection=True; Application Name=EdFi.Ods.WebApi;"
  },
  "PostgreSqlConnectionStrings": {
    "EdFi_Admin": "Host=localhost; Port=5432; Username=postgres; Database=EdFi_Admin; Application Name=EdFi.Ods.WebApi;",
    "EdFi_Security": "Host=localhost; Port=5432; Username=postgres; Database=EdFi_Security; Application Name=EdFi.Ods.WebApi;",
    "EdFi_Ods": "Host=localhost; Port=5432; Username=postgres; Database=EdFi_{0}; Application Name=EdFi.Ods.WebApi;"
  }
}

The disadvantage to this solution is that we would need to modify ConfigConnectionStringsProvider whenever there is an extension to database engines.

Discussion/Decision 8/18/2020

After a discussion, we have decided to do the following:

  • appsettings.json is required, and will have connection string keys with empty values, along with a production deployment setup.
    • For Octopus, we can poke the file, with values we would like address via replacements.
Sample appsettings.json 
{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft": "Warning"
    }
  },
  "AllowedHosts": "*",
  "Urls": "https://localhost:54254;http://localhost:5000",
  "ConnectionStrings": {
    "EdFi_Ods_Empty": "",
    "EdFi_Admin": "",
    "EdFi_Security": "",
    "EdFi_Ods": ""
  }
}
  • Connection strings will be generated via initdev process and stored in appsettings.local.json
    • Will not go with option 1 or 2 but instead will use the standard ConnectionStrings section, but the generated connection string will be based off of engine.
  • Engine will be written to appsettings.local.json
  • Introduce an appsettings.edfi.json that will be checked that has the "developer setup". This file will contain enabled features and extensions for the developer experience.
Example NetCore App Program Example 
namespace EdFi.Ods.WebApi.NetCore
{
    public class Program
    {
        public static async Task Main(string[] args)
        {
            var _logger = LogManager.GetLogger(typeof(Program));
            _logger.Debug("Loading configuration files");

            var executableAbsoluteDirectory = Path.GetDirectoryName(typeof(Program).Assembly.Location);

            ConfigureLogging(executableAbsoluteDirectory);

            var host = Host.CreateDefaultBuilder(args)
                .ConfigureAppConfiguration(
                    (hostBuilderContext, configBuilder) =>
                    {
                        _logger.Debug($"Content RootPath = {hostBuilderContext.HostingEnvironment.ContentRootPath}");

                        configBuilder.SetBasePath(hostBuilderContext.HostingEnvironment.ContentRootPath)
                            .AddJsonFile(
                                Path.Combine(executableAbsoluteDirectory, "appsettings.json"), false,true)
                            .AddJsonFile(
                                Path.Combine(
                                    executableAbsoluteDirectory,
                                    $"appsettings.{hostBuilderContext.HostingEnvironment.EnvironmentName}.json"),true, true)
                            .AddJsonFile(
                                Path.Combine(executableAbsoluteDirectory, "appsettings.edfi.json"),true, true)
                            .AddJsonFile(
                                Path.Combine(executableAbsoluteDirectory, "appsettings.local.json"), true, true)
                            .AddEnvironmentVariables();
                    })
                .UseServiceProviderFactory(new AutofacServiceProviderFactory())
                .ConfigureWebHostDefaults(webBuilder => { webBuilder.UseStartup<Startup>(); }).Build();

            await host.RunAsync();

            static void ConfigureLogging(string executableAbsoluteDirectory)
            {
                string configPath = Path.Combine(executableAbsoluteDirectory, "log4net.config");

                XmlConfigurator.Configure(
                    LogManager.GetRepository(typeof(Program).GetTypeInfo().Assembly), new FileInfo(configPath));
            }
        }
    }
}

Deployment Automation

Deployment automation processes will need to create the correct settings config file during deployment, instead of using the environment file(s) checked into source control. The deployment PowerShell scripts* will need to be updated to support generating the environment-specific config file (* or other replacement for PowerShell, for example Python).

In PowerShell, reading a JSON file into a proper object is more difficult than expected. It is easier to create a file from scratch, rather than opening and modifying a template. Example:

function New-ProductionAppSettingsFile {
    [CmdletBinding()]
    param (
        [hashtable]
        [Parameter(Mandatory=$true)]
        $Config
    )
 
    $connectionString = New-ConnectionString -ConnectionInfo $Config.DbConnectionInfo
 
    $prodSettings = @{
        SqlServerConnectionStrings = @{
                EdFi_Admin = $connectionString
				# Etc.
        }
    }
 
    $prodSettingsFile = Join-Path -Path $Config.PackageDirectory -ChildPath "appsettings.Production.json"
 
    $prodSettings | ConvertTo-Json | Out-File -FilePath $prodSettingsFile -NoNewline -Encoding UTF8
}

Function New-ConnectionString  comes from the EdFI.Installer.AppCommon package of PowerShell scripts: Configuration.psm1.