Skip to content

xray traces between ECS->SNS->SQS-lambda not showing in the same xray graph

0

Hi team,

I’m refactoring an AWS application, and I need some help with tracing. In the original version of the app, I have a Node.js application running inside an ECS Fargate cluster that sends a message to an SQS queue. Then, I have a Lambda function polling from the queue. The Node.js app, when sending a message, specifies the MessageSystemAttributes with trace information, as shown in this code snippet:

const sendMessage = async () => {
  const sampled = segment.notTraced ? '0' : '1';
  await sqsClient.send(
    new SendMessageCommand({
      QueueUrl: config.scanResultQueue,
      DelaySeconds: 0,
      MessageBody: JSON.stringify(body),
      MessageSystemAttributes: { 
        AWSTraceHeader: { 
          DataType: "String", 
          StringValue: `Root=${segment.trace_id};Parent=${segment.id};Sampled=${sampled}` 
        } 
      }
    })
  );
};

In this original setup, when I view the trace in X-Ray, I can see two linked traces: one for the ECS service and one for the Lambda function in the same trace graph.

In the new version of the application, I introduced an SNS topic. Now, my ECS Node.js application publishes messages to SNS, and an SQS queue is subscribed to that SNS topic. The Lambda function still polls from the queue. When I publish to SNS, I send the trace header in the MessageAttributes as follows:

const publihMessageTopic = async () => {
  const sampled = segment.notTraced ? '0' : '1';
  const messageBody = JSON.stringify(body);
  const traceHeader = `Root=${segment.trace_id};Parent=${segment.id};Sampled=${sampled}`;
  
  await snsClient.send(
    new PublishCommand({
      TopicArn: config.scanResultTopic,
      Message: messageBody,
      MessageAttributes: {
        AWSTraceHeader: { 
          DataType: "String", 
          StringValue: traceHeader 
        }
      }
    })
  );
};

However, in the new application, when I view the trace in X-Ray, I do not see the linked traces between ECS and Lambda. I’ve confirmed that X-Ray tracing is enabled for ECS, Lambda, and SNS, so I’m not sure why the trace linkage is missing.

Is there something I'm missing or any configuration I need to update in the new setup to ensure that the traces are linked properly in X-Ray?

is this still true? https://stackoverflow.com/questions/71217941/aws-x-ray-tracing-from-lambda-to-sns-to-sqs-to-another-lambda-python3-boto3

Thanks in advance!

1 Answer
0

Below are some important points to consider:

  1. SNS and SQS support: Both SNS and SQS support trace context propagation, which means they can pass the tracing header to downstream services.

  2. Linked traces: In your original setup (ECS -> SQS -> Lambda), you saw linked traces because both ECS and Lambda emit segment data. In the new setup (ECS -> SNS -> SQS -> Lambda), the introduction of SNS and SQS in the middle might be breaking the direct linkage in the trace map.

  3. Trace continuity: Despite not appearing in the trace map, the trace context is still being propagated through SNS and SQS. This means that if you examine the individual traces, you should still see the correct trace IDs being passed along. However to ensure the propagation of trace IDs:

a. When publishing to SNS, include the trace header in both the MessageAttributes and the Message body:

const publishMessageTopic = async () => {
  const sampled = segment.notTraced ? '0' : '1';
  const traceHeader = `Root=${segment.trace_id};Parent=${segment.id};Sampled=${sampled}`;
  
  const messageBody = JSON.stringify({
    ...body,
    AWSTraceHeader: traceHeader
  });

  await snsClient.send(
    new PublishCommand({
      TopicArn: config.scanResultTopic,
      Message: messageBody,
      MessageAttributes: {
        AWSTraceHeader: { 
          DataType: "String", 
          StringValue: traceHeader 
        }
      }
    })
  );
};

b. Set up a subscription filter policy on your SNS to SQS subscription to include all messages:

{
  "AWSTraceHeader": [{"exists": true}]
}

c. In your Lambda function, extract the trace header from the message body and set it manually:

const AWS = require('aws-sdk');
const AWSXRay = require('aws-xray-sdk-core');

exports.handler = async (event) => {
  for (const record of event.Records) {
    const body = JSON.parse(record.body);
    if (body.AWSTraceHeader) {
      AWSXRay.setSegment(new AWSXRay.Segment('lambda', body.AWSTraceHeader));
    }
    // Process the message...
  }
};

Try following the above updated code.

[+] https://docs.aws.amazon.com/xray/latest/devguide/xray-concepts.html#xray-concepts-traces

As a workaround, you can also use Step Functions: If possible, consider using AWS Step Functions to orchestrate your workflow. Step Functions maintains trace context across services, including SNS and SQS.

To address your specific question about the Stack Overflow post: There have been multiple updates with SNS supporting XRay now and can be referred at:

[+] https://aws.amazon.com/about-aws/whats-new/2023/02/amazon-sns-x-ray-active-tracing-visualize-analyze-debug-application-performance/

To improve your tracing visibility:

  1. Ensure that active tracing is enabled for your SNS topic.
  2. Check that your IAM permissions are correctly set up to allow SNS to send trace data to X-Ray.
  3. When analyzing traces, you may need to manually correlate the trace IDs between your ECS and Lambda traces, as they won't be visually linked in the X-Ray console.

Additionally please check the below resources which will help achieve the use-case:

[+] https://serverlessland.com/patterns/sns-sqs-lambda-xray [+] https://serverlessland.com/patterns/sns-sqs-lambda-xray

While this situation isn't ideal for visualization, rest assured that the trace context is still being propagated through your entire system. The limitation is in how X-Ray represents these traces in its console, not in the actual tracing mechanism itself.
Sources
Amazon S3 and AWS X-Ray - AWS X-Ray
AWS Lambda and AWS X-Ray - AWS X-Ray
Amazon SNS and AWS X-Ray - AWS X-Ray

answered a year ago
AWS
SUPPORT ENGINEER
revised a year ago
AWS
SUPPORT ENGINEER
revised a year ago
  • but i don't see the lambda part in my x-ray graph, i see only client -> ECS service, but i'm expecting to see client -> ECS service -> myLmabdaFunction->..

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.