Skip to content

How do I mask sensitive data in API Gateway execution logs without turning off data tracing?

6 minute read
0

I want to mask sensitive data, such as API keys, tokens, and personally identifiable information (PII), in my Amazon API Gateway execution logs. I don't want to turn off data tracing.

Short description

If you turn on data tracing for API Gateway in Amazon CloudWatch Logs, then the service logs full request and response payloads. These payloads include sensitive values such as API keys, authorization tokens, and PII. To mask this data and keep troubleshooting visibility, use CloudWatch Logs data protection policies. These policies detect and mask sensitive data in log events with AWS managed data identifiers or custom regex patterns.

CloudWatch Logs data protection uses data identifiers to detect sensitive data patterns in log events. When the service finds a match, it masks the data with asterisks in the log output. There are two types of data identifiers:

  • Managed data identifiers are pre-built patterns that AWS provides for common sensitive data types. Use them to detect credit card numbers, Social Security numbers, email addresses, IP addresses, and names.
  • Custom data identifiers are regex patterns that you define for organization-specific sensitive data, such as internal tokens, custom API keys, and proprietary identifiers.

Resolution

Note: Masking doesn't delete the original data. It hides the sensitive values when someone views the logs. Make sure that your users have the logs:Unmask AWS Identity and Access Management (IAM) permission to view the original data for debugging.

Prerequisites:

  • Verify that you have an API Gateway REST API that has data tracing turned on.
  • Confirm that you use a CloudWatch Logs execution log group for your API. The format is API-Gateway-Execution-Logs_api-id/stage-name.
  • Make sure that you have the IAM permissions logs:PutDataProtectionPolicy to create data protection policies.

Create a data protection policy

Create an account-wide data protection policy or create a data protection policy for a single log group.

Create an account-wide data protection policy

To apply consistent data protection across all CloudWatch log groups in your AWS account, complete the following steps:

  1. Open the CloudWatch console.
  2. In the navigation pane, choose Settings.
  3. Choose the Logs tab.
  4. In the Data protection account policy section, choose Configure.
  5. Under Managed data identifiers, select the types of data to audit and mask. For example, choose EmailAddress, CreditCardNumber, and IpAddress.
  6. Choose Activate data protection.

Create a data protection policy for a single log group

To target a specific API Gateway log group, complete the following steps:

  1. Open the CloudWatch console.
  2. In the navigation pane, under Logs, choose Log Management.
  3. Select the API Gateway execution log group. For example, API-Gateway-Execution-Logs_api-id/stage-name.
  4. Choose the Data protection tab.
  5. Choose Create policy.
  6. Under Managed data identifiers, select the types of data to audit and mask.
  7. Choose Activate data protection.
    Note: If both an account-level and a log-group-level data protection policy exist, then both policies apply to that log group. The service audits and masks all managed data identifiers that either policy specifies.

Add a custom data identifier for organization-specific data

AWS managed data identifiers cover common sensitive data types, but they don't cover all organization-specific patterns. For example, managed identifiers don't cover the x-api-key header value.

To mask custom sensitive data, complete the following steps:

  1. Open the CloudWatch console.

  2. In the navigation pane, under Logs, choose Log Management.

  3. Select the API Gateway execution log group. For example, API-Gateway-Execution-Logs_api-id/stage-name.

  4. Choose the Data protection tab.

  5. Choose Create policy.

  6. Choose Add custom data identifier.

  7. Enter a regex pattern that matches your sensitive data. For example, to match an API key value in the x-api-key header, use the following pattern:

    (x-api-key[^a-zA-Z0-9]{1,10}[a-zA-Z0-9]{20,})
  8. Enter a descriptive name for the custom data identifier. For example, enter APIGateway-XApiKey.

  9. Complete the activation process.
    Note: Adjust the regex pattern to match your specific API key format and how it appears in your log events.

Verify masking in log events

After you activate the data protection policy, confirm that the service masks sensitive data in your log events.

Before you apply the data protection policy

If you turned on data tracing and you haven't applied a data protection policy, then the log events show the full request, including sensitive values in plaintext.

Example log event shows the x-api-key value in plaintext:

(a1b2c3d4-5678-90ab-cdef-EXAMPLE11111) Endpoint request body after transformations:
{"resource":"/upload","path":"/upload","httpMethod":"POST","headers":{
"Accept":"/",
"User-Agent":"PostmanRuntime/7.51.1",
"x-api-key":"EXAMPLEKEY1234567890EXAMPLEKEY12345",
"X-Forwarded-For":"203.0.113.1",
"X-Forwarded-Port":"443",
"X-Forwarded-Proto":"https"} ... [TRUNCATED]

In the preceding output, the x-api-key value is fully visible.

After you turn on the custom data identifier policy

After you apply the data protection policy with the custom data identifier for x-api-key, the same API request produces the following log event:

(a1b2c3d4-5678-90ab-cdef-EXAMPLE22222) Endpoint request body after transformations:
{"resource":"/upload","path":"/upload","httpMethod":"POST","headers":{
"Accept":"/",
"User-Agent":"PostmanRuntime/7.51.1",
"***************************************",
"X-Forwarded-For":"203.0.113.1",
"X-Forwarded-Port":"443",
"X-Forwarded-Proto":"https"} ... [TRUNCATED]

In the preceding output, the service replaces the x-api-key header and its value with asterisks. All other request details stay visible for troubleshooting.

(Optional) Unmask protected data

To view the original unmasked data for debugging, the IAM user or role must have the logs:Unmask permission.

Add the following IAM policy statement to the user or role that needs to view unmasked data:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "logs:Unmask",
      "Resource": "arn:aws:logs:REGION:ACCOUNT_ID:log-group:LOG_GROUP_NAME:"
    }
  ]
}

Note: Replace REGION with your AWS Region, ACCOUNT_ID with your account ID, and LOG_GROUP_NAME with your log group name.

After you grant this permission, open the CloudWatch Logs console. Then, choose Show unmasked data from the Display dropdown list to view the original values.

Review considerations and best practices

Review the following considerations and best practices:

  • Configure data protection policies to send audit findings for compliance tracking. You can send findings to an Amazon Simple Storage Service (Amazon S3) bucket, a CloudWatch Logs log group, or an Amazon Data Firehose delivery stream.
  • CloudWatch Logs data protection incurs additional charges based on the volume of data that the service scans. For more information, see Amazon CloudWatch pricing.

Note: Data protection applies only to log events that the service ingests after you activate the policy. The service doesn't retroactively mask existing log events. Data protection policies add minimal latency to log ingestion and don't affect API Gateway performance.

Related information

Help protect sensitive log data with masking

Types of data that you can protect

Create a data protection policy for a single log group

Set up CloudWatch logging for REST APIs in API Gateway

AWS OFFICIALUpdated 10 days ago