Cloudformation Registry Resource Type - Do not show sensitive values in logs

Hi Simon,

Thank you for your exceptional patience with regards to this query, we have been conducting extensive research into this topic to be able to offer you the best answer possible which I am happy to be sharing with you now. To confirm my understanding, there seems to be concerns with regards to how one can safely handle sensitive information, such as API credentials, in resource types developed by CloudFormation Registry. Please do correct me if I am mistaken.

The Short: To summarize the answer to this query, it is typically not advisable to pass sensitive information directly into a CloudFormation template, and it behooves one to be mindful of not logging sensitive information in code, inadvertently or otherwise. A potential solution to this predicament is to rather allow the resource type's handler itself to retrieve the sensitive information directly from AWS Secrets Manager, thereby circumventing the risk of inadvertently exposing sensitive information in an API event.

The Long: Now that we have an overview of the situation, let us dive deeper into the technical details. First, we shall take a moment to reproduce the behavior as observed to serve as our control test. We start by initializing the scaffolding of a new resource type project by means of 'cfn init' [1]. For the purpose of this demo, we shall name the resource type "Sandbox::Test::Worker".

In the generated resource type schema file [2], named 'sandbox-test-worker.json' we define a new property named "SensitiveInfo" of type String. We shall be passing our sensitive information into this property for the test. In the handler code, named 'handler.py', we now add a simple command to log the ProgressEvent entering the create_handler() function. It will look like this "LOG.debug(progress)".

Remember to explicitly set the logging level appropriately, else when provisioning our private resource type we may find ourselves in the situation where the expected LogGroups are not created in CloudWatch. In 'handler.py', add the following:

LOG = logging.getLogger(__name__)
LOG.setLevel(logging.DEBUG) # <-----------

Now that we have configured our handler code and schema, we validate [3], generate [4], and submit [5] our private resource type to Cloudformation. For convenience, these commands can be joined into one as in the following example:

`=== Generate and submit resource type ===
'cfn generate && cfn submit --set-default --region us-east-1'`

Once our resource type has been successfully registered in Cloudformation, we can now create a stack which provisions our resource type. In the following sample template we will notice that we are passing our sensitive information into the 'SensitiveInfo' property:

=== Sample Template containing ===
Resources:
  MyPrivateResource:
    Type: Sandbox::Test::Worker
    Properties:
      SensitiveInfo: foryoureyesonly

Upon stack creation, we notice the following event being logged in CloudWatch:

=== Logged create_handler ===
ProgressEvent(status=<OperationStatus.SUCCESS: 'SUCCESS'>, errorCode=None, message='', result=None, callbackContext=None, callbackDelaySeconds=0, resourceModel=ResourceModel(SensitiveInfo='foryoureyesonly', DisplayName=None, Tags=None), resourceModels=None, nextToken=None)

As expected, after we had inspected the logs in CloudWatch, we can observe the sensitive information "foryoureyesonly" has been exposed.

Now that we have demonstrated and confirmed the concern with regards to exposing sensitive information in a resource type's logs, let's take a moment to discuss best practices in terms of data protection in the context of Cloudformation.

The first noteworthy point in the documentation on Data protection in AWS CloudFormation states that it is strongly recommended that one never puts confidential or sensitive information, such as customers' email addresses, into tags or free-form fields such as a Name field. This includes when you work with AWS CloudFormation or other AWS services using the console, API, AWS CLI, or AWS SDKs. [6]

Whilst Cloudformation does indeed offer encryption at rest as well as in transit, the words of caution above urge us to rather avoid passing sensitive information directly into templates, and the possibility remains that said information is exposed to unauthorized persons who incidentally happen to have access to the log files.

The following documentation further reiterates this sentiment in a section that recommends that instead of embedding credentials into one's template, it is recommended to rather make use of dynamic references [8][9].

Dynamic references provide a compact, powerful way for one to reference external values that are stored and managed in other services, such as the AWS Systems Manager Parameter Store or AWS Secrets Manager. When one uses a dynamic reference, CloudFormation retrieves the value of the specified reference when necessary during stack and change set operations, and passes the value to the appropriate resource. However, CloudFormation never stores the actual reference value [8].

=== Considerations when using dynamic references ===
As an additional remark, it is strongly recommended to not include dynamic references, or any sensitive data, in resource properties that are part of a resource's primary identifier.

When a dynamic reference parameter is included in a property that forms a primary resource identifier, CloudFormation may use the actual plaintext value in the primary resource identifier. This resource ID may appear in any derived outputs or destinations. [10]

SOLUTION

This brings us to our solution to address the concern of exposing sensitive information in logs. In the next section we shall discuss how we can have our resource type handlers themselves dynamically retrieve sensitive information directly from AWS Secrets Manager, thus effectively eliminating the risk of inadvertently exposing them in logs.

We begin by first creating a new secret [11] in AWS Secrets Manager which will securely house our sensitive information, such as API credentials for example. For the sake of understanding the examples in this report, we shall assume that the secret name is 'topsykrettz', and contains one secret value as follows "ApiKey":"foryoureyesonly".

Next, we modify our resource type's schema to: 2.1 include a property called "SecretName" of type String.

e.g

"properties": { "SecretName": { "type": "string" },

2.2 Add the "secretsmanager:GetSecretValue" permission to the handlers

e.g

"handlers": {
  "create": {
    "permissions": [
      "secretsmanager:GetSecretValue"
    ]
  },

We then modify the handler code to retrieve the secret from Secrets Manager:

When viewing the Secret in Secrets Manager console, the bottom of the page includes a section called "Sample Code". We shall use and modify this code as the basis for our solution.

3.1 Import dependencies

At the top of 'handler.py', add the following:
  import base64
  import logging
  from botocore.exceptions import ClientError

3.2 Copy Sample getSecret() function code from Secrets Manager console and modify accordingly. We shall use the following example in our solution:

def get_secret(secret_name: str, session: Optional[SessionProxy]):
      secret_ = None

      # Create a Secrets Manager client
      client = session.client(
          service_name='secretsmanager'
      )
      LOG.info('Retrieving value of secret: ' + secret_name)

      # In this sample we only handle the specific exceptions for the 'GetSecretValue' API.
      # See https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_GetSecretValue.html
      # We rethrow the exception by default.

      try:
          get_secret_value_response = client.get_secret_value(
              SecretId=secret_name
          )
      except ClientError as e:
          if e.response['Error']['Code'] == 'DecryptionFailureException':
              # Secrets Manager can't decrypt the protected secret text using the provided KMS key.
              # Deal with the exception here, and/or rethrow at your discretion.
              LOG.exception(e)
              raise e
          elif e.response['Error']['Code'] == 'InternalServiceErrorException':
              # An error occurred on the server side.
              # Deal with the exception here, and/or rethrow at your discretion.
              LOG.exception(e)
              raise e
          elif e.response['Error']['Code'] == 'InvalidParameterException':
              # You provided an invalid value for a parameter.
              # Deal with the exception here, and/or rethrow at your discretion.
              LOG.exception(e)
              raise e
          elif e.response['Error']['Code'] == 'InvalidRequestException':
              # You provided a parameter value that is not valid for the current state of the resource.
              # Deal with the exception here, and/or rethrow at your discretion.
              LOG.exception(e)
              raise e
          elif e.response['Error']['Code'] == 'ResourceNotFoundException':
              # We can't find the resource that you asked for.
              # Deal with the exception here, and/or rethrow at your discretion.
              LOG.exception(e)
              raise e
          else:
              # For everything else...
              LOG.exception(e)
              raise e
      else:
          # Decrypts secret using the associated KMS key.
          # Depending on whether the secret is a string or binary, one of these fields will be populated.
          LOG.info('Decrypting secret ' + str(secret_name))
          if 'SecretString' in get_secret_value_response:
              secret = get_secret_value_response['SecretString']
              secret_ = secret
          else:
              decoded_binary_secret = base64.b64decode(get_secret_value_response['SecretBinary'])
              secret_ = decoded_binary_secret

      if secret_ is None:
          LOG.warning('Secret returned a null value!!')

      # Your code goes here.
      return secret_

3.3 Call getSecret() from handler function.

In this proof-of-concept, we shall be retrieving the secret in the create_handler() function only. An example of which is as follows:

secret_value = str(get_secret(progress.resourceModel.SecretName, session))
LOG.warning(
    'Here is the secret, but we should NOT be logging this!! -> ' + secret_value)

In the snippet above, we can notice that we are passing the name of the secret to be retrieved which will be defined in our template which we shall discuss shortly. Additionally, one may notice that we are also passing the Optional[SessionProxy] object from the parameters of the create_handler(). As this code will ultimately be called by Cloudformation's Lambda-backed resource handlers, this is important to ensure the SDK client [12] can correctly retrieve and decrypt our secret value.

=== WARNING ===

Finally, we notice that in this snippet we are explicitly logging the decrypted secret value. This is purely for demonstration purposes for us to confirm that our code is correctly decrypting the secret as expected. Kindly remember to NOT do this in production.

3.4 Submit new version of resource type Now that our code is modified appropriately, we need to generate and submit a new version of our resource type to Cloudformation. To do this, we once again run the 'cfn submit' command.

=== Generate and submit resource type ===
'cfn generate && cfn submit --set-default --region us-east-1'

Test 4.1 Once our new version is registered, we need to test our resource type by provisioning it in a stack. Kindly find the following example of a template:

Parameters:
SecretName:
Type: String
Default: topsykrettz

Resources:
MyPrivateResource:
Type: Sandbox::Test::Worker
Properties:
SecretName: !Ref SecretName

In this template, we are passing in the name of our Secret, 'topsykrettz', which had created in the beginning of this walkthrough. The name of the Secret is then passed into our 'SecretName' property which we had defined in Step 2.1.

4.2 Create stack
We are now ready to test our resource type. We do so by creating a stack with the abovementioned template.

4.3 Inspect logs
Once the stack has successfully been created, we inspect the logs in CloudWatch once more. This time, we are expecting to not see any occurrence of our secret value, "foryoureyesonly", being exposed in the logs. Except for the single warning where we explicitly log the value for confirmation!

CONCLUSION

In the above report we had addressed concerns of Cloudformation Registry resource types inadvertently exposing sensitive information in its respective CloudWatch logs. We had researched and discussed various best practices, cautions and recommendations regarding the handling of sensitive data in Cloudformation. And finally, we and dive deep into a solution to circumvent the risk of exposing secrets by configuring our resource type's handler code to retrieve and decrypt the sensitive information directly from AWS Secrets Manager.

=== References ===

[1] - cfn init https://docs.aws.amazon.com/cloudformation-cli/latest/userguide/resource-type-cli-init.html

[2] - Resource type schema https://docs.aws.amazon.com/cloudformation-cli/latest/userguide/resource-type-schema.html

[3] - cfn validate https://docs.aws.amazon.com/cloudformation-cli/latest/userguide/resource-type-cli-validate.html

[4] - cfn generate https://docs.aws.amazon.com/cloudformation-cli/latest/userguide/resource-type-cli-generate.html

[5] - cfn submit https://docs.aws.amazon.com/cloudformation-cli/latest/userguide/resource-type-cli-submit.html

[6] - Data protection in AWS CloudFormation https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/security-data-protection.html

[7] - Do not embed credentials in your templates https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/security-best-practices.html#creds

[8] - Do not embed credentials in your templates https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/security-best-practices.html#creds

[9] - Using dynamic references to specify template values https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/dynamic-references.html

[10] - Considerations when using dynamic references https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/dynamic-references.html#dynamic-references-considerations

[11] - Create a secret https://docs.aws.amazon.com/secretsmanager/latest/userguide/create_secret.html

[12] - Boto3 - SecretsManager https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager.html

I am using AWS Secret and I am referencing it using dynamic reference. As you have mentioned this is a native way to introduce secrets to the CF template and to pass them on into the resources. This should also be the way to introduce them into custom resource type.

The problem I see is that Cloudformation's default behavior should NOT be to log the payload that comes into the custom resource, as it should be aware that the given payload might contain a resolved secret (I am not sure why is Cloudformation logging the payload in the first place. Lambdas do not log the payload by default. Why do it here?)

Suggesting retrieval of the secret in the code is ridiculous to me, as this would mean that I need to give the custom resource permission to retrieve any secret (this is what you did in the example). This I think is not a good practice. Imagine I am developing a private type for a customer. It would be unacceptable for them to give the resource (whose internals they did not study nor they need to) full access to reading secrets in the account. Yes, you can theoretically play with the role and set up tighter permissions with some custom scripting after submitting the module and the role, but to that with multiple customer when each of them has different approaches to handling security is ridiculous.

The problem is that retrieving the secret programmatically(and giving resource permissions to do so) is not something you should be doing in the resource in the first place. This for me is a breach of separation of concerns because I believe it is a concern of Cloudformation to resolve the secrets (in the end that is why dynamic references were introduced).

TLDR: Cloudformation should not log the incoming payload by default.