Why are my X-Ray sampling rules being ignored for my Node.js app running in EC2?

Question

I have a Node.js app running in EC2 and have successfully launched the X-Ray collector using user data. The X-Ray service is getting the traces from the application (instrumented using the javascript xray-sdk), but I see no trend information in the sampling rules I have set up in the AWS console (including the default rule).

My code to initialize the sdk is pretty simple:

```
const AWS = AWSXRay.captureAWS(require('aws-sdk'));
AWS.config.update({ region: process.env.DEFAULT_AWS_REGION || 'us-west-2' });

...

app.use(AWSXRay.express.closeSegment());

...

app.use(AWSXRay.express.closeSegment());
```
I have also added code to use the AWS sdk to query for and log the sampling rules to see if they are visible to the app, and they are. I have not called `AWSXRay.middleware.disableCentralizedSampling()`, so I don't know why the sampling rules are not being used.

Is there some other configuration I need to set up to allow the rules to be used?

Answer

Centralized sampling rules are stored in X-Ray service. X-Ray sdk in user application has to take time to poll it and this poll action is lazy initialization, triggered by the first segment. So, the first segment generated in user application will fall back to a local rule (1/sec + 5%), which means the first segment is always sampled.

X-Ray centralized sampling has some limits, it does not work well in such cases:
1. Instrumented application's life is short. Centralized sampling rules is not ready to take effect. 
2. If user's fleet scale is large, poll centralized sampling rules and sync up status with X-Ray service would take more time.

The alternative solution is using local sampling rules.

Answer

Tested in my test environment and it is working fine for nodejs application on EC2 with xray Daemon. Below is steps performed: --- - Installed nodejs on EC2 Amazon Linux. ``` # yum groupinstall 'Development Tools' # curl -sL https://rpm.nodesource.com/setup_16.x | sudo -E bash - # yum install -y nodejs ``` - Installed xray deamon[2](https://docs.aws.amazon.com/xray/latest/devguide/xray-daemon.html) ``` # wget https://s3.us-west-2.amazonaws.com/aws-xray-assets.us-west-2/xray-daemon/aws-xray-daemon-3.x.rpm # yum install aws-xray-daemon-3.x.rpm ``` - Added needed permission for xray to instance IAM role [3](https://docs.aws.amazon.com/xray/latest/devguide/security_iam_id-based-policy-examples.html#xray-permissions-managedpolicies) ``` AWSXrayReadOnlyAccess AWSXRayDaemonWriteAccess Sampling), you can see data in Trend graph --- Please try on your above in your test environment.

Why are my X-Ray sampling rules being ignored for my Node.js app running in EC2?

Relevant content