Troubleshooting and monitoring Amazon Bedrock Guardrails usage with Amazon CloudWatch

Introduction

Organizations use gen AI applications to create new content, increase productivity, and gain valuable insights for decision-making while reducing costs through task automation. To make sure that you correctly use these applications, AWS developed Amazon Bedrock Guardrails. Amazon Bedrock Guardrails detects and filters input prompts sent to the foundation model and the model's output. After you configure guardrails, use Amazon CloudWatch to monitor the guardrail efficiency and observe filtered content through dashboards, alarms, and logs.

Amazon Bedrock Guardrails provides an extra layer of security on gen AI applications. You can use this extra security to block or count topics or words based on specific use cases or responsible AI policies. To customize Amazon Bedrock Guardrails, you can take the following actions:

• Filter content, words, and sensitive information
• Deny topics
• Add contextual grounding and automated reasoning checks

Admin users can use guardrails to review their health and performance, observe topics that users prompt, verify that topics are correctly filtered, and detect potential false positives. It’s a best practice to monitor your guardrails for the following reasons:

• Performance: If the overall latency of your gen AI application increases, then review your configuration to see if the guardrail is the cause.
• Cost: To see how frequently you use the guardrail, review the number of texts to see if guardrail usage increases your costs. Then, you can adjust the usage as needed.
• Security: Review and detect unusual jailbreak attempts where users produce harmful content with the large language model (LLM). You can also observe the number of sensitive information retrieval attempts. If these attempts continue to increase, then a user might be trying to fetch unauthorized data.
• Employee and user protection: Review the guardrails for negative topics that are prompted to the LLM, and respond to these concerns.
• Guardrail quality: Check if your guardrail correctly works, if the metrics are increasing, and why unwanted behavior of your gen AI application occurs without detection by Amazon Bedrock Guardrails.

To help monitor Amazon Bedrock Guardrails, you can use the following CloudWatch metrics:

Metric	Usage
InvocationLatency	This metric helps you observe the latency that the guardrail adds to a prompt request.
InvocationThrottles	If the gen AI app doesn’t provide responses to users, then this metric can provide more information about the number of times that the system throttled invocations.
TextUnitCount	This metric indicates the number of text units that Amazon Bedrock Guardrails processes. You can use it, and you can use this metric to track the usage and costs of the guardrail.
InvocationsIntervened	This metric counts when the guardrail intervenes, and can be broken down with the GuardrailPolicyType dimension. Use this metric to catch jailbreak or data leakage attempts or discussions of sensitive topics with the gen AI application.

Using CloudWatch with Amazon Bedrock Guardrails

In this article, we use an example of a customer in the finance industry who built a chatbot for their employees. The customer uses a guardrail to filter harmful categories, such as hate, sexual content, insults, violence, misconduct, and profanity. The guardrail also filters for specific words, such as topics related to financial advice. The customer wants to mask Personally Identifiable Information (PII) data, such as credit card numbers and social security numbers.

Figure 1: High-level architecture of a gen AI application with Amazon Bedrock Guardrails and CloudWatch.

Prerequisites

Before you begin, make sure that you complete the following tasks:

• Have an active AWS account with the appropriate permissions
• Have a running AI application
• Configure Amazon Bedrock Guardrails
• Have familiarity with CloudWatch
• Install Python and Boto3 SDK for logging examples

Monitoring with CloudWatch dashboards

CloudWatch offers out-of-the-box metrics that you can see on CloudWatch dashboards. The metrics are ready to use, and you can set up the dashboard in a few minutes.

After you configure your gen AI application with Amazon Bedrock Guardrails and invoke your guardrail, create your CloudWatch dashboard. Complete the following steps:

1. Open the CloudWatch console.
2. In the navigation pane, under Dashboard, choose Create dashboard.
3. Enter a name for your dashboard, and then choose Create dashboard.
4. A modal appears where you can select your widget type: select Bedrock/Guardrails.
   Note: You must invoke the guardrail before this step.
5. Select the metrics that you want to display.

Figure 2: Example of a CloudWatch dashboard.

Setting up CloudWatch alarms

You can configure CloudWatch alarms to notify you of specific issues, such as the following:

• Unusually high usage of a particular topic or category
• A user writes their credit card or social security number
• Invocation latency is higher than usual

To configure a CloudWatch alarm, complete the following steps:

1. Open the CloudWatch console.
2. In the navigation pane, under Alarms, choose Create alarm.
3. Select the metric that you want to create an alarm for, such as InvocationsIntervened.
4. Configure the conditions that you want to trigger the alarm.
5. Configure the actions that you want to occur, such as notify an Amazon Simple Notification Service (Amazon SNS) topic.
6. Choose Save.
7. Add the alarm to the dashboard.

Troubleshooting with CloudWatch Logs

To troubleshoot a specific invocation, use CloudWatch Logs. The following example uses the Python Boto3 SDK to troubleshoot the invocation.

Use the Boto3 invoke_model API, and fetch the returned data from the Amazon Bedrock invocation:

client = boto3.client('bedrock-runtime', region_name='string')

response = client.invoke_model(
    body=b'bytes'|file,
    contentType='string',
    accept='string',
    modelId='string',
    trace='ENABLED'|'DISABLED'|'ENABLED_FULL',
    guardrailIdentifier='string',
    guardrailVersion='string',
    performanceConfigLatency='standard'|'optimized'
)

response_body = json.loads(response['body'].read())

After you reformat the response, use the Boto3 put_log_events API to send the log event to your Log Group and Log Stream in CloudWatch Logs:

client = boto3.client('logs', region_name='string')

response = client.put_log_events(
    logGroupName='string',
    logStreamName='string',
    logEvents=[
        {
            'timestamp': 123,
            'message': 'string'
        },
    ],
    sequenceToken='string',
    entity={
        'keyAttributes': {   
            'string': 'string'
        },
        'attributes': {
'string': 'string'
        }
    }
)

Note: You must create the Log Group and Log Stream before this step.

Then, you can view the CloudWatch Logs in the CloudWatch console.

Figure 3: Example of Amazon Bedrock Guardrails logs in the CloudWatch Console.

You can also query the logs to filter specific invocations.

Example:

fields @timestamp, @message, @logStream, @log
| filter category = "financial_advice"
| sort @timestamp desc
| limit 10000

Figure 4: Example of logs filtered by the financial_advice category.

You can also automatically log Amazon Bedrock invocation logs, model input data, and model output data to CloudWatch Logs or Amazon S3. With Amazon S3, you can load to data into OpenSearch, or use Amazon EventBridge targets to process it.

How AWS Support can help

There are several ways that AWS Support can help you set up Amazon Bedrock Guardrails. During the setup of your solution, AWS Support can open a support case for you if you don’t see metrics on your CloudWatch dashboard. For your support case, a Support Engineer helps troubleshoot and resolve your issues.

If you’re an AWS Enterprise Support customer, then a Technical Account Manager (TAM) can proactively help you configure your solution. This support includes determining the following configurations:

• What metric is the most relevant for your solution
• The appropriate monitoring thresholds for security-related metrics
• How to configure your CloudWatch alarms for critical guardrail metrics
• How to optimize dashboard configurations to monitor guardrail performance

After you configure your guardrail, if you detect unusual behavior, then you can open a support case with AWS. A Support Engineer can also guide you on how to detect and respond to potential jailbreak attempts, and how to develop a guardrail strategy for PII data. AWS can even help you create custom queries for log analysis.

Conclusion

When you combine Amazon Bedrock Guardrails with CloudWatch, you have a comprehensive solution to monitor and troubleshoot gen AI applications. Through the three main components of CloudWatch (dashboards, alarms, and logs), organizations can effectively maintain robust security controls and make sure that their gen AI applications operate as intended. When teams use these tools, they can quickly identify and address potential issues, optimize their guardrail configurations, and maintain compliance posture with their responsible AI policies.

To get the most out of your AWS environment, contact your TAM. They can help you with general guidance, best practices, troubleshooting, and operational support on AWS. To learn more about our plans and offerings, see AWS Support. To learn more about the suggested solution, contact your TAM or AWS account team.

About the author

Thomas Jeffery
Thomas has been working in various roles in the IT industry. From working at telecommunication manufacturers to testing connectivity on Amazon devices, he has now fully embraced the Cloud technologies with a strong interest in Generative AI.