.NET Core and AWS Secrets

.NET Core with runtime secrets has been a bit of a journey within the AWS ecosystem over the past 5 years. At SPS< our journey of onboarding and shifting our entire infrastructure from on-premise into AWS Cloud started in 2016. Along the way, we have sought to continuously improve secret management as our systems and technology are ever-evolving. And I’m here to say, secret management and usage that you can be proud of is much easier and much simpler using .NET Core and AWS Secret Manager than ever.

History with CredStash

If you’re not a history or story kind of person, feel free to jump past this. I always find the journey as interesting as the solution.

At SPS Commerce circa 2016, our core secret management architecture in the cloud revolved around the usage of the CredStash pattern. It combined AWS Key Management Service for encryption with storage in an AWS Dynamo DB table. Consumers would retrieve the KMS decryption key and encrypted value from the table and then have to perform the decryption themselves on the client. Permission would be controlled by both access to the table record and the KMS key via IAM Permissions. It wasn’t a horrible pattern. At the time audit records were difficult in a limited CloudTrail world, and the most frustrating aspect was dealing with encryption on clients between different versions of the decryption algorithm. This often bubbled up issues where one client could decrypt a secret but not another or even worse, no client could decrypt the secret that they encrypted. You were additionally heavily dependent on processing and code to manage the decryption yourself. Often there were available libraries to help, but nothing standardized. It was okay until AWS SSM Parameter Store offered encrypted values natively.

History with AWS Parameter Store

AWS SSM Parameter Store seemed like AWS’s best-kept secret at the beginning of 2017. Even working with the AWS Solutions Architects, they were a bit surprised and this nestled little independent and simple service to store secrets inside AWS Systems Manager. It still offered encryption via AWS KMS Keys but handled the encryption and decryption on the server-side. As officially part of the AWS ecosystem, it was accessible and standardized across the AWS CLI and SDK. It just worked, and it worked awesomely (still does work well). You stored values based on hierarchical keys, making the pulling of secrets in a single bulk request inside your application runtime at the start, very easy and simple.

Parameter Store integration additionally existed into Cloud Formation templates and conveniently it showed up in other integrations throughout the AWS ecosystem, making the transition to Parameter Store for secrets a no-brainer at the time. Not long, and AWS released their own .NET Core Provider for easily extending configuration from the AWS SSM Parameter Store. It is really easy to use, by just loading all the parameters (secret or not) into the .NET Core Configuration Provider based on path.

public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
    WebHost.CreateDefaultBuilder(args)
        .ConfigureAppConfiguration(builder =>
        {
            builder.AddSystemsManager("/myapplication/path");
        })
        .UseStartup();

This worked well for a long time across many language ecosystems. However, as our patterns matured and the capabilities in other Secret Management utilities got better, so did the need for the next iteration of secret consumption become necessary for our teams.

Limitations of the AWS Parameter Store

Some of the growing needs and concerns that we began to face as a rapidly growing and expanding enterprise:

  • Parameter Limits - Parameter Store has a 10,000 hard limit of the maximum number of items that you can use in a single AWS account. You can store more, but not in standard mode. You must transition to Advanced parameter types, which incurs additional cost for usage. The standard mode is also limited to 4 KB, which may be a concern depending on what your storing. It took us a while to reach 10,000, but as our organization was working through our shift to a multi-account AWS architecture we definitely ran up against some of the limits and threshold breaching for API requests. More concerning as we scaled (literally as our applications scaled) it became more and more of a cascading problem since runtime secrets would fail due to rate-limiting (1,000 requests / second max). This failure would result in containers failure to start, which would just retry, compounding the problem.
  • Auditing - AWS CloudTrail does capture API calls to AWS Systems Manager, but lacks certain contextual information that would be helpful related to encryption for an audit. Newer AWS Services also offer the ability to use Resource Policies to define who has access to your secret (so the whole world cannot just add their own access via IAM Policies unless they control the resource themselves).
  • Cross Account Capabilities - AWS Parameter Store ARNs are not accessible in a multi-account architecture via a direct API call, without assuming a role in the respective AWS Account. Using Resource Policies you can expose an AWS resource like AWS Secret Manager, to another AWS Account very easily. Find out more about our AWS Cross-Account Resource Access Pattern Journey.

AWS SSM Parameters were really just key-value storage in a hierarchy. Not being a first-class or native primary secret management system it lacks a lot of other necessary features. These include the ability to automatically rotate secrets based on automation, set expiration dates, central auditing, and built-in replication. That brings us to the AWS Secret Manager!

Getting Started with AWS Secret Manager

AWS Secrets Manager actually wasn’t released very long after the capability for secret encryption with the Parameter Store, as it hit prime time on April 4, 2018: Introducing AWS Secrets Manager. That being said, with AWS Parameter Store pattern working well for us for a while there was little need and to be honest little company appetite in moving “yet again” to the next thing (given the recent transition from CredStash to Parameter Store). Additionally, the pricing for AWS Secrets was considerably more in comparison. In mid-2020 the limitations of our usage with Parameter Store noted above really started to peak and a transition was required.

AWS Secret Manager really does hit most of the pain points we had. By default, it allows for 5,000 requests per second as an API threshold, has integrated auditing, and supports resource policies enabling cross-account AWS account usage. Not to mention the numerous additions around RDS Integration, sizing, expiration, complex object types, etc.

As we dive into the AWS Secret Manager, I’m going to assume you have working knowledge on the AWS side of how to create, manage and work with Secrets in the AWS Console. If you don’t, definitely dive into this fantastic article by Andrew Lock: Secure secrets storage for ASP.NET Core with AWS Secrets Manager. In fact, this article is what we based our customized internal enhancements for .NET Core integration on. Go read it first and come back so you have the context…. I’ll wait… I promise!

… ok good, your back. By now you’ve learned that we can easily integrate with AWS Secrets Manager on application start using the .NET Configuration Provider much the same way we did with the Parameter Store, except in this case using a library provided by Kralizek called the AWSSecretsManagerConfigurationExtensions.

public static IWebHost BuildWebHost(string[] args) =>
    WebHost.CreateDefaultBuilder(args)
            .ConfigureAppConfiguration((hostingContext, config) =>  
            {
                config.AddSecretsManager();
            })
            .UseStartup()
            .Build();

I love this library since it’s simple, targeted, does one thing well, and offers some pretty decent configuration flexibility. However, as we prepared a similar library to this code-base for wider organizational rollout internally with .NET Core integration, there were a few optimizations that seemed to make sense at the Enterprise usage level and additional pattern considerations that might be a benefit to you.

Naming Conventions

At SPS Commerce we have developed a series of “Improvement Proposals”, which is standard indexed documentation and processes used by the entire tech organization. This allows us a process to propose, discuss and follow certain conventions across all our teams. One of those conventions was created by our internal Cloud Operations team for the naming of our secrets. Abiding by these conventions internally helps us reason about secret access and provides a methodology that can make our code simpler as well.

While AWS Secret Manager does not explicitly work with a path-hierarchy like AWS Parameter Store, AWS does still suggest naming your secrets based on a conventional path that helps imply some type of namespace. This is what our naming looks like:

//
c7ad0e07-2fe0-41fa-b7f6-a4beb9ba7989/dev/MyTestSecret
c7ad0e07-2fe0-41fa-b7f6-a4beb9ba7989/test/MyTestSecret
c7ad0e07-2fe0-41fa-b7f6-a4beb9ba7989/prod/MyTestSecret

This convention easily provides us the capability to separate secrets by environment (pretty standard for secret audit requirements and best practices), while keeping the secrets associated to a single service grouped together with our internal and unique service identifier (this correlates to an internal service registry). Additionally, since the name is part of the AWS ARN, this helps me with IAM to provide access policies to my service where I can use wildcards.

// ARN: 
arn:aws:secretsmanager:us-east-1:782645404356:secret:c7ad0e07-2fe0-41fa-b7f6-a4beb9ba7989/prod/MyTestSecret

// Wildcard to all Secrets for My Service
arn:aws:secretsmanager:us-east-1:782645404356:secret:c7ad0e07-2fe0-41fa-b7f6-a4beb9ba7989/prod/*

In this case, using AWSSecretsManagerConfigurationExtensions library you would be able to use the KeyGenerator configuration option when adding Secrets Manager to parse the name of the key to add to the configuration.

builder.AddSecretsManager(configurator: options =>
{
     // c7ad0e07-2fe0-41fa-b7f6-a4beb9ba7989/prod/MyTestSecret -> "MyTestSecret"
     options.KeyGenerator = (entry, key) => key.Split('/').ToList().Last().ToString();
});

As we roll this out to the organization though, we wanted to make this convention the default and configuration options required. In our updated configuration we enhanced the available options with the Environment and ServiceId allowing us to add inline code to automatically extract the keys in the provider:

builder.AddSecretsManager(configurator: options =>
{
    options.Environment = "dev";
    options.ServiceId = "c7ad0e07-2fe0-41fa-b7f6-a4beb9ba7989";
});

You would obviously want to drive the “Environment” value based on an Environment variable or other source.

Bulk Loading & Complex Types

Unlike the AWS Parameter Store, working with the AWS SDK for Secret Manager, there is no API that returns more than a single secret value at a time. We have no option to request secrets by the nice conventional path that we created. There are some relevant usage patterns to consider based on this and the other attributes of a Secret that are fundamentally different than Parameter Store:

  • Single Secret Request at a Time - Since we can only make a request for one secret at a time, we should prefer to minimize the number of secrets we need. Unlike Parameter Store, a single Secret can store up to 64 Kb of data each.
  • Cost Per Secret - Since we pay-per-secret, it is also economically helpful to store more key values in a single secret, different entirely from the architecture of key-values in the Parameter Store.
  • JSON - Secrets can be stored as plaintext, but they can also be stored and natively understood as complex and nested types in JSON in the AWS Console. Theoretically, we can store a lot of secrets in here with up to the 64 Kb size restriction.

With these key differences, the lifecycle management of a “Secret” is much more interesting and manageable. Moving forward our design suggests that we might only need and want a single Secret per deployable unit or service. We will want to deploy the Secret via AWS Cloud Formation and manage the resource and its policies. The value and complex structure stored within the secret is the dynamic aspect that I might interact with through automation or AWS Console to update the contents.

We might structure our secrets in JSON like shown below. I try to keep it to a single root level object with primitive types. This makes it easy to reason about translation to a .NET Configuration Provider and makes it accessible natively with key-values in the AWS Console when viewing/updating as well.

{
   "SecretOne": "secret-value-1",
   "SecretTwo": 10,
   "NotASecret": "its in here too!"
}

Additionally, you may want to store configuration data for the service in the secret as well. Similar concept to storing configuration data unencrypted in the Parameter Store, putting other data in the Secret is just easy and free at this point. It might be the most obvious location for you outside your application to do so.

This is helpful when sharing configuration across development environments as well. While many prefer to use local .NET User Secrets (which is an awesome feature), if I am already integrated with AWS Secret Manager, and am required to authorize for that storage and other AWS Resources already, I find it very compelling to just load secrets directly for development purposes without having to set up any local .NET User Secrets. I know Andrew Lock was not a fan in his article, but I find it eliminates one additional pain point when getting started with a service locally for the first time, and all you need is IAM Permissions to get going (still secured through team IAM permission practices and audits). If development requires personalized secrets, I can appreciate where that pattern is not an advantage.

While a single Secret per service is nice, in reality, it’s probably not that clean for you. You may have other Secrets that need to be pulled with Database credentials or are on a separate rotation or expiration policy. Generally, that means you likely have 2-3 Secrets to pull in a typical micro or tiny service (but far less than 30 individual plaintext Secrets).

With bulk loading not an option and the usage of complex types with JSON for the values, we will need to find a pattern to properly deserialize the JSON complex keys into our .NET Configuration Provider. The good news is by default the AWSSecretsManagerConfigurationExtensions library does support JSON values in Secrets and loads the properties effectively based on the path recursively through the parsed JSON properties.

Concrete AppSettings with “Secret” Attributes

Another core .NET development pattern we use a lot for configuration is to map the .NET Configuration Provider results directly into a concrete object that is injected as a singleton and available for dependency injection across any service in the app domain that requires it. That makes reasoning about configuration and using intelli-sense really easy and simple. We wanted to find a way to intersect this design practice with the targeted loading of specific Secrets by name (instead of just loading all Secrets available under the AWS Permissions for that Role/App) and handle complex JSON deserialization into it.

Kralizek discusses the ability to filter secrets before they are retrieved as often some applications are not able to describe or retrieve a list of all available secrets for security purposes. It is always nice to use some declarative alternatives (that we explore below) instead of just listing out the “acceptedARNs”.

We’ll start by defining what our “AppSettings.cs” file:

public class AppSettings
{
    public string OtherSetting { get; set; }

    [Secret]
    public ComplexSecret MyTestSecret { get; set; }
}

public class ComplexSecret
{
    public string SecretOne { get; set; }

    public int SecretTwo { get; set; }

    public string NotASecret { get; set; }
}

You’ll notice we create a plain old class object (POCO) for AppSettings, and associate our earlier “MyTestSecret” with a complex object that represents the JSON structure specified previously. We denote the field as a Secret so we can conventionally use the ServiceId and Environment (previously configured in the AddSecretsManager extension method) with the Property Name to form the name of our Secret that we can pull intentionally instead of looking for all secrets. This pattern just helps us invert the idea of supplying a concrete list of ARNs for our secret into conventionally determining that list based on the declarative usage of the SecretAttribute in the settings class. The name used to pull the Secret (full ARN is not necessary for the same account) is:

c7ad0e07-2fe0-41fa-b7f6-a4beb9ba7989/dev/MyTestSecret

The SecretAttribute is pretty straightforward:

public class SecretAttribute : Attribute
{

    public SecretAttribute()
    {
    }

    public SecretAttribute(string Arn)
    {
        this.Arn = Arn;
    }

    public string Arn { get; }
}

With those POCO’s in place, we additionally will have to tell the Secret Manager extension method about our AppSettings so that it can use reflection to glean the information about the secret and property name. We add an overloaded extension method that accepts the settings as a generic type.

builder.AddSecretsManager(configurator: options =>
{
     options.Environment = "dev";
     options.ServiceId = "c7ad0e07-2fe0-41fa-b7f6-a4beb9ba7989";
});

The new extension method then takes the type and creates a map of the secret keys to load (just like “acceptedARNs”):

public static IConfigurationBuilder AddSecretsManager(this IConfigurationBuilder configurationBuilder, Action configurator = null)
    where T : class, new()
{
    var options = new SecretsManagerConfigurationProviderOptions();
    options.SecretKeys = new SecretsManagerSecretMapLoader(typeof(T)).LoadMap();
    configurator?.Invoke(options);

    var source = new SecretsManagerConfigurationSource(options);
    configurationBuilder.Add(source);

    return configurationBuilder;
}

The SecretsManagerSecretMapLoader LoadMap method is pretty straightforward using reflection to load the list of secrets:

public List LoadMap()
{
    var secretProps = new List();
    var properties = _type.GetProperties();
    foreach (var prop in properties)
    {
        var attr = prop
            .GetCustomAttributes(typeof(SecretAttribute), true)
            .FirstOrDefault() as SecretAttribute;
        if (attr != null)
        {
            secretProps.Add(new SecretMap
            {
                Name = prop.Name,
                Arn = attr.Arn
            });
        }
    }

    return secretProps;
}

Wow, we are cranking through a lot of code now, and getting pretty close to the desired pattern and architecture. That being said, we are still missing the glue and updates of how we bring together all the changes we made into the SecretsManagerConfigurationProvider that will actually query the Secret in AWS.

private async Task> FetchConfigurationAsync(CancellationToken cancellationToken)
{
    var configuration = new HashSet<(string, string)>();
    foreach (var secret in Options.SecretKeys)
    {
        var secretArn = secret.Arn ?? secret.Name;
        try
        {
            // determine ARN to use based on configuration (or just name)
            if (string.IsNullOrWhiteSpace(secret.Arn))
            {
                // if custom prefix provided, just use that hardcoded
                if (!String.IsNullOrWhiteSpace(Options.SecretPrefix))
                {
                    secretArn = $"{Options.SecretPrefix}/{ secret.Name }";
                }
                else
                {
                    // add ARN account id if provided
                    var accountPrefix = string.Empty;
                    if (!String.IsNullOrWhiteSpace(Options.AccountId))
                    {
                        accountPrefix = 
                            $"arn:aws:secretsmanager:{ Options.Region.SystemName }" +
                                    ":{ Options.AccountId }:secret:";
                    }

                    // name prefix for service id / test
                    var namePrefix = string.Empty;
                    if (!String.IsNullOrWhiteSpace(Options.ServiceId))
                    {
                        namePrefix += Options.ServiceId;
                    }

                    if (!String.IsNullOrWhiteSpace(Options.Environment))
                    {
                        if (!String.IsNullOrWhiteSpace(namePrefix))
                        {
                            namePrefix += "/";
                        }

                        namePrefix += Options.Environment;
                    }

                    if (!String.IsNullOrWhiteSpace(namePrefix))
                    {
                        namePrefix += "/";
                    }

                    secretArn = $"{accountPrefix}{namePrefix}{secret.Name}";
                }
            }

            // get the secret
            var secretValue = await Client.GetSecretValueAsync(
                new GetSecretValueRequest { SecretId = secretArn }, cancellationToken)
                    .ConfigureAwait(false);
            var secretString = secretValue.SecretString;
            if (secretString != null)
            {
                if (IsJson(secretString))
                {
                    var obj = JToken.Parse(secretString);
                    var values = ExtractValues(obj, secret.Name);
                    foreach (var value in values)
                    {
                        configuration.Add((value.key, value.value));
                    }
                }
                else
                {
                    configuration.Add((secret.Name, secretString));
                }
            }
        }
        catch (ResourceNotFoundException e)
        {
            throw new MissingSecretValueException(
                $"Error retrieving secret value (Secret: {secretArn})", secretArn, e);
        }
    }

    return configuration;
}

You can see the inversion here, where we entirely drive the secret request loop based on the provided SecretKeys generated from the AppSettings reflection map loading. The routine to “FetchAllSecretsAsync” is now unnecessary entirely and has been removed.

If you were following along closely you’ll notice we also have the ability to specify a full Secret AWS ARN within the SecretAttribute that is consumed as well for retrieval. This capability has the added advantage of being an escape hatch when you need to load a secret that doesn’t fit the naming convention discussed, or for whatever reason is outside of the standard ServiceId and Environment convention used by default for all:

public class AppSettings
{
    [Secret(Arn="arn:aws:secretsmanager:us-east-1:782645404356:secret:WeirdSecretName")]
    public ComplexSecret MyTestSecret { get; set; }
}

This flexibility also enables us to use Secrets from an entirely different AWS Account and Region if configured properly for IAM permissions.

Cross Account Secret Configuration

Cross Account access patterns in AWS are actually pretty interesting…. errr I guess for some. Either way, it is always nice in a pinch to have Resource Policy capability to allow for this when needed. It is not ideal and does require full ARNs to be used for the request but is an incredible capability in the right scenario. We already discussed above how you can configure full ARNs on your resources using the declarative attribute. If you need further information about Resource Policies or cross-account access patterns check out a previous post on “AWS Cross-Account Resource Access Patterns”. This includes using OIDC Providers in some cases to avoid cross-account access by assuming the IAM Role inside the destination resource account (this really helps simplify your application code and configuration).

Secret Value Refresh Interval

AWS Secret Manager has support for auto-rotating of your secrets. You can create lambda functions with custom code that derives or generates new secrets and update the value and version of your Secret resource. We have not had the opportunity to make use of the automated secret rotation yet, but manual secret rotation or updating secret values are a common occurrence. In this scenario, it usually involves an update to the Secret value, followed by a restart of your application domain so that it can pull the latest version of the Secret at startup. In many cases, this is recycling the container instances your running. If your service is small, a container restart and rescheduling are pretty straightforward, but a bit annoying to have to worry about it. On a larger fleet that can take some time for the container orchestrater to reschedule all the workloads and cycle through them, this can be a very undesirable situation to be in.

While working with the AWSSecretsManagerConfigurationExtensions library I noticed a seemingly undocumented option for “PollingInterval”. It does exactly as you would expect, in that it re-executes the Secret value retrieval and stores it in the .NET Configuration Provider in a background thread on the provided interval (i.e. every 5 minutes). This is a pretty powerful little feature that makes it easier for your application to refresh its secret value on your schedule.

builder.AddSecretsManager(configurator: options =>
{
    // refresh secrets every 5 min
    options.PollingInterval = new TimeSpan(0, 5, 0);
});

However, there are definite implications on changing configuration values after start. The core difference is that you have to intentionally decide to use an instance of the settings rather than the original, especially if it was a singleton. To do that we inject our configuration as an IOptionsSnapshot (instead of IOptions or even AppSettings directly):

public class MyClass
{
    private readonly AppSettings _settings;

    private MyClass(IOptionsSnapshot settings)
    {
        _settings = settings.Value;
        var mySecret = _settings.MyTestSecret.SecretOne;
    }
}

Additionally, if the secret being consumed is cached inside a connection pool or other resource you’ll need to be aware and handle that change. Fittingly enough, Andrew Lock also provides a great breakdown and guidance to be aware of with Dependency Injection of Scoped services that may consume IOptionsSnapshot in his article “The dangers and gotchas of using scoped services in IConfigureOptions”.

Conclusion

That is all for now. Hopefully, you are leaving with an increase in incremental knowledge on using secrets in AWS for .NET development compared to what you started with. If everything you build and work on is in the AWS Cloud, then AWS Secret Manager makes a lot of sense. However, if you find yourself traversing other ecosystems it might be a better strategy to find a more cloud agnostic provider, such as HashiCorp Vault. For us, AWS Secret Manager provided very tight integration with the AWS ecosystem that was mostly what we needed, at the right cost. That being said, there are times when continuing to use secrets encrypted via AWS Parameter Store will also make sense when needed outside of application runtimes, such as integration with infrastructure as code (i.e. CloudFormation Parameters).

Definitely check out Kralizek’s AWSSecretsManagerConfigurationExtensions .NET Configuration Provider as you dive-in more to AWS Secret consumption with .NET Core.

-Travis Gosselin

Reference: http://blog.travisgosselin.com/net-core-and-aws-secrets/

Travis Gosselin @travisgosselin