How do I resolve the trace quota error in X-Ray?

Short description

The following scenarios might cause trace documents to exceed the allowed limit in AWS X-Ray:

• You sent an increased number of batched segments for a trace where the trace document size exceeds 500 KB size.
• You sent non-batched segments for a trace where trace document size exceeds 100 KB size.
• You added custom segments, metadata, and annotations that increased the trace document size.

The upper limit of an X-Ray trace document size changes dynamically as per the number of segments that you send together. This is due to the limit exceeded trace feature. For a high number of segments that you send together in a batch that's attached to a trace, the upper limit is 500 KB. For individual segments that you send with a time gap that's attached to a trace, the upper limit is 100 KB.

The faster you send a trace (the more segments that you batch together and send), the more the compression efficiency increases. The slower you send a trace (send segments individually with a time gap), the more the trace splits into multiple revisions. Also, the slower you send a trace, the more it consumes storage capacity for cache in the backend. Traces that last longer produce more duplicates and results in X-Ray collecting less data.

Resolution

Note: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, then make sure that you’re using the most recent AWS CLI version.

View the trace in the X-Ray console, or run the following command to confirm that the trace document exceeds the size limit:

aws xray batch-get-traces --trace-ids <EXAMPLE_TRACE_ID>

Note: Replace EXAMPLE_TRACE_ID with your trace's ID.

X-Ray collects the following information for a larger trace:

{
    "Id": "<EXAMPLE_TRACE_ID>",
    "Duration": 187.859,
    "LimitExceeded": true,
    "Segments": [
         ...
     ]
     ....
 }

Note: If the LimitExceeded parameter is true, then you exceeded the trace or segment quota.

Increased number of batched segments for a trace where trace document size exceeds 500 KB size

This scenario occurs when you integrate Step Functions with X-Ray. When you integrate Step Functions with X-Ray, you can't customize what X-Ray does or doesn't trace. When you pass the trace ID through all the AWS Lambda functions, segment data gets added to the trace ID.

If you're using Step Functions, then complete the following steps:

1. Go to Step Functions, and then turn off active tracing.
2. Pass the trace ID across Lambda functions only for critical workflows, and don't exceed the trace upper limit.
3. If Lambda is receiving the trace header from upstream services, then remove the PutTraceSegments permission from the Lambda execution role. The upper limit is dynamically adjusted.

If you aren't using Step Functions, then complete the following steps:

1. Debug your code to check if you're passing the same trace ID for different requests.
2. Break the trace. To do this, don't pass the trace ID in the invocation to downstream services.
3. Create a new trace ID from the breaking point.

Note: If you create a trace ID for every application, then your trace count increases. For easier tracing, keep the critical application workflows as part of one trace.

For more information, see AWS X-Ray and Step Functions.

Non-batched segments for a trace where trace document size exceeds 100 KB size

For this scenario, use the solution from the preceding section.

Break traces for the new upper limit of 100 KB. Or, batch more segments in the application to increase the upper limit to 500 KB. Batching segments is supported only with open-telemetry SDK. If you're using X-Ray SDK, then change the way that the application sends segments.

Added custom segments, metadata, and annotations that increased the document size

To reduce the trace document size, don't add extra custom segments to the same trace ID. Add custom segments only for necessary workflows. Also, to reduce the trace document size, reduce the metadata and annotations in the traces.