Skip to content

CloudWatch Logs Missing from Invocation Handler for All Sessions Except the First — Bedrock AgentCore Runtime

0

I am experiencing a consistent and reproducible issue with CloudWatch log delivery in Amazon Bedrock AgentCore Runtime where application logs generated inside the invocation handler are only captured for the first session after a deployment, and are absent for all subsequent sessions.

Environment:

  • Deployment: Amazon Bedrock AgentCore Runtime via bedrock-agentcore-starter-toolkit
  • Agent framework: Strands Agents with a BedrockAgentCoreApp entrypoint
  • Base image: ghcr.io/astral-sh/uv:python3.11-bookworm-slim (ARM64)
  • PYTHONUNBUFFERED=1 is set in the Dockerfile
  • Log group: /aws/bedrock-agentcore/runtimes/<agent_name>-<id>-DEFAULT
  • CloudWatch log delivery is enabled

Observed Behavior:

My agent code contains print statements and Python logging calls both at the module level (outside the entrypoint function) and inside the @app.entrypoint decorated handler.

For the first session created after calling runtime.launch(), both the module-level logs and the invocation handler logs appear correctly in CloudWatch. Multiple consecutive invocations within the same session all log correctly.

However, when I create a new session (by using a different session_id), a new log stream is created in the same log group, which correctly contains all the module-level startup logs — confirming that a new microVM is provisioning and its stdout/stderr is being captured. But none of the logs from inside the @app.entrypoint handler appear in that stream, despite the invocation completing successfully and returning the correct response.

This behavior is 100% reproducible: every first session logs everything, every subsequent new session logs only startup code.

What I Have Ruled Out:

  • PYTHONUNBUFFERED=1 is set — buffering is not the cause
  • The new log stream is created, so CloudWatch delivery is connected to the new microVM
  • The invocation itself succeeds and returns the correct response
  • The issue is not payload or framework related — the handler executes correctly
  • Both stdout and stderr have been tested; neither captures invocation-level logs for non-first sessions
  • sys.stdout.flush() and sys.stderr.flush() calls inside the handler make no difference

Key Question: Since the new log stream is created and captures startup logs, CloudWatch delivery is clearly working for the new microVM. Why would stdout/stderr output from within the invocation handler be captured for the first session but not for any subsequent sessions, despite identical code and configuration?

Please advise on whether this is a known limitation, a bug in the log delivery mechanism, or whether there is a supported way to capture application-level logs from within the invocation handler across all sessions.

Thank you.

2 Answers
4

I think as your startup logs appear in the new log stream but invocation logs do not (despite a successful 200 OK response), this is likely not a permissions or buffering issue, but rather a problem with how the AgentCore Runtime manages Python’s standard streams during session transitions.

Potential Root Causes

1. Handler Context Isolation:

The @app.entrypoint decorator may be executing the handler in a spawned process or a thread where sys.stdout and sys.stderr are not correctly mapped to the CloudWatch-linked file descriptors of the new MicroVM. If the runtime "hijacks" these streams during the first session, it might fail to re-bind them for subsequent sessions.

2. The uv / Python 3.11 Slim Environment:

While you have PYTHONUNBUFFERED=1 set, the ARM64 slim image combined with the uv package manager might be interacting with the Firecracker (MicroVM) logging driver in a way that drops the stream if the handler doesn't explicitly flush to the underlying system file descriptor.

3. Logging Hierarchy:

If you are using the standard logging module, the BedrockAgentCoreApp might be resetting the root logger configuration after the first initialization.

Troubleshooting to confirm it’s a bug:

  • Test with Low-Level Writes: Inside your handler, try writing directly to the file descriptor to bypass Python's internal buffering entirely:
import os
os.write(1, b"Direct FD1 Write: Handler is executing\n")

If this appears in CloudWatch but print() does not, I think the issue is within Python’s stream wrapping.

EXPERT
answered 2 months ago
1

When you encounter missing CloudWatch Logs from your invocation handler for sessions after the first one, there are several troubleshooting steps to consider:

First, verify you're checking the correct log group. AgentCore stores logs under your agent's application logs or under your own custom log group when CloudWatch Logs is used as the destination. Make sure you're looking at the right location for all session logs.

If logs appear to be missing or empty, consider running your agent container locally to verify that logging is working as expected in your code. This can help isolate whether the issue is with your application logic or the AgentCore integration.

You may also want to enable verbose logging in your configuration to capture more detailed information about what's happening during invocations across different sessions.

Since AgentCore provides structured application logs that include fields like timestamp, session_id, trace_id, request_payload, and response_payload, you should be able to correlate logs across sessions. These logs are designed to help you gain visibility into agent runtime invocations and session-level resource consumption.

If you're still experiencing issues after these steps, it's worth noting that AgentCore can output logs to multiple destinations including CloudWatch Logs, Amazon S3, or Firehose stream. You might consider testing with an alternative destination to see if the behavior persists.

Additionally, ensure that your execution role has the proper permissions to write logs to CloudWatch, as permission issues can sometimes cause intermittent log delivery problems that might manifest differently across sessions.

For comprehensive troubleshooting, set up monitoring using CloudWatch metrics and alarms, enable comprehensive logging, and use structured error handling to better understand what's happening across all your sessions.
Sources
AgentCore generated runtime observability data - Amazon Bedrock AgentCore
Troubleshoot AgentCore Runtime - Amazon Bedrock AgentCore

answered 2 months ago
EXPERT
reviewed a month ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.