CloudWatch Trace Map Nodes and Trails are missing after X-Ray UpdateTraceDestination
I'm implementing an observability solution for the service. Service components include API Gateway, AWS Lambdas, and microservices running on EKS.
First, I enabled X-Ray tracing for API Gateway and Lambdas. After that, I was able to see the nodes in CloudWatch->APM->Trace Map. Everything worked as expected, I was able to select a node and watch its traces.
After that, I set up an exporter in EKS to publish microservices traces to https://xray.us-east-1.amazonaws.com/v1/traces and got an error:
Permanent error: rpc error: code = InvalidArgument desc = error exporting items, request to https://xray.us-east-1.amazonaws.com/v1/traces responded with HTTP Status Code 400, Message=The OTLP API is supported with CloudWatch Logs as a Trace Segment Destination. Please enable the CloudWatch Logs destination for your traces using the UpdateTraceSegmentDestination API
To make this work, I updated the X-Ray trace segment destination to CloudWatchLogs with
aws xray update-trace-segment-destination --destination CloudWatchLogs
command. And after this change, API Gateway and Lambda Nodes have disappeared from CloudWatch->APM->Trace Map. At the same time, some of the EKS microservices traces (but not all!) appeared at Trace Map. Also, all spans for all service components became available in CloudWatch->APM->Transaction Search. From Transaction Search, I am able to query and filter spans and get traces, but it is not convenient.
Have I missed something? What is the logic of appearing nodes in the Trace Map? Why do some spans appear there and others do not? Is it possible to fix this?
- Topics
- Management & Governance
- Language
- English
asked a year ago434 views
- Newest
- Most votes
- Most comments
When you updated the X-Ray trace segment destination to CloudWatchLogs using the UpdateTraceSegmentDestination API, you switched from the legacy X-Ray storage model to the newer CloudWatch Observability (CWOT) model. This transition explains the changes you're seeing in your trace visualization.
Here's what happened:
-
Initially, your API Gateway and Lambda traces were being sent to the legacy X-Ray service and data store, which made them visible in the CloudWatch Trace Map.
-
After running
aws xray update-trace-segment-destination --destination CloudWatchLogs, all traces are now being sent to the/aws/spanslog group in CloudWatch Logs instead of the X-Ray segment store. -
This change affects how traces are processed and displayed:
- Traces are now available in CloudWatch Transaction Search (which you've confirmed)
- Only some of your EKS microservices appear in the Trace Map
The reason only some components appear in the Trace Map is likely related to how they're instrumented. Components using OpenTelemetry with proper service name definitions will appear in the Trace Map, while others might not have the correct metadata for visualization.
To fix this issue:
- Ensure all your services have proper instrumentation with consistent service naming
- Verify that transaction search is properly enabled
- Check that your resource policies allow X-Ray to send traces to CloudWatch Logs
- For Lambda functions, ensure they're configured with "Active" tracing mode rather than "PassThrough" if you want them to appear in the trace map
The transition to CloudWatch Logs as the trace destination is the recommended approach for modern observability, but it requires ensuring all components are properly configured to work with this model.
Sources
AWS XRay 0 traces retrieved but can be found by traceid | AWS re:Post
Traces sent to X-Ray - Amazon CloudWatch Logs
Enable transaction search - Amazon CloudWatch
Streamlining trace sampling behavior for AWS Lambda functions with AWS X-Ray | AWS Compute Blog
answered a year ago
Relevant content
asked a year ago
asked 2 years ago
- AWS OFFICIALUpdated 10 months ago
