AWS Lambda not using X-Ray Sampling rules

Hello,

AWS Lambda uses its own sampling strategy today if your traces start at the Lambda function. For this use case, the requests are sampled at 1 request per second and %5 anything above that. The sampling rate for AWS Lambda can be adjusted by submitting a support case. We are working with AWS Lambda team for service sampling rule support but don't have any ETA at the moment. Please stay tuned.

Best,
Haotian

I am not seeing the sampling rate of 1 request per second and %5 anything above that. Instead, every request (which is an sqs message) is getting traced. So 100 messages in SQS leads to 100 traces in X-Ray. I tried creating a new sampling rule targeted at my service and it had no impact. In case this was something strange with the Java X-Ray SDK I also tried tracing with a basic python lambda and it also traced every request (see below).

def handle_request(event, context):
    for record in event['Records']:
        payload=record["body"];
        print(str(payload))

Any thoughts?

Lambda samples all requests in low volume case. The sampling logic within lambda will not be affected by the X-Ray SDK code. We are working with Lambda team to get better customer experience on this. Unfortunately there is no workaround for SQS triggered lambda functions.

Thanks,
Haotian

What is considered a "low volume" case? By putting more data into our SQS queue I was able to see some sampling, but nothing near the 5% after 1/s. The more data in SQS and the faster lambda was executed, the fewer traces that were sampled. For example, when I drop 10,000 messages as fast as I could into SQS, ~20% of the messages were sampled. However, if I placed 10 messages a second into SQS, which more closely resembles our use case, ~85% of the messages had traces. Creating my own sampling rules had no affect on these values.

From your response below, are you saying that the reason we are seeing sampling that does not match the rules is because lambda is invoked by an SQS event?

Based on how lambda function is triggered by SQS messages on this AWS blog https://aws.amazon.com/blogs/aws/aws-lambda-adds-amazon-simple-queue-service-to-supported-event-sources/, depending on how fast you are sending messages and how fast for consumer function to drain the queue, lambda will try to drain messages as fast as possible by applying concurrent invocation with batching APIs, as described in the blog. So it is non-deterministic of how many function invocations could happen.

Whichever event triggers a lambda function it samples on the invocation request, which could contain batching events due to reasons discussed above. So the sampling rate is not meaningful if calculated by number of messages or events like DynamoDB streams or Kinesis records.

Lambda uses its own sampling strategy that will not be affected by changing X-Ray SDK code or creating sampling rules on X-Ray console. It is a populate request for lambda sampling to be easily configurable and we are working closely with Lambda team on this.

What are you trying to achieve here? Are you estimating costs or you are trying to reach some certain rate for investigation? If you could explain a little bit more on your use case we can bring lambda team to further help.

Thanks,
Haotian

Edited by: haotianw@AWS on Jan 16, 2019 4:49 PM

Cost estimate/management is what I am trying to do. I thought from a previous response you had mentioned configuring the sampling rules, but from this response it is clear that configuring the sampling rules will not affect how Lambda samples, nor will using the X-RAY SDK.

Unfortunately, if we cannot configure the sampling rate of Lambda it does not seem cost effective for us to use X-Ray right now. I'm glad to hear you are working on making the Lambda sampling rate easily configurable and I'll be excited to hear about any updates.

Thanks,

Bill

Any change on this in the past year? I've been seeing similar behavior with my SQS-driven Lambda functions being sampled at a high rate (30 to 50%), and my attempts to set a sampling rate in X-Ray console appear futile.

Edited by: RossM on Apr 14, 2020 11:41 PM

Unfortunately X-Ray still does not support sampling rules configuration in Lambda environments. This thread will be updated once this feature is available.