Why are my X-Ray sampling rules being ignored for my Node.js app running in EC2?

Tested in my test environment and it is working fine for nodejs application on EC2 with xray Daemon. Below is steps performed:

---

• Installed nodejs on EC2 Amazon Linux.

# yum groupinstall 'Development Tools'
# curl -sL https://rpm.nodesource.com/setup_16.x | sudo -E bash - 
# yum install -y nodejs 

• Installed xray deamon2

# wget https://s3.us-west-2.amazonaws.com/aws-xray-assets.us-west-2/xray-daemon/aws-xray-daemon-3.x.rpm
# yum install aws-xray-daemon-3.x.rpm

• Added needed permission for xray to instance IAM role 3

AWSXrayReadOnlyAccess
AWSXRayDaemonWriteAccess   <<read permissions to get sampling rules
AWSXrayFullAccess

• Deployed the sameple application4 5

# wget https://github.com/aws-samples/aws-xray-sdk-node-sample/archive/refs/heads/master.zip
# unzip master.zip
# cd aws-xray-sdk-node-sample-master/
# npm install aws-xray-sdk
# npm install aws-xray-sdk --save
# npm install express
# node --inspect index.js

• Generate multiple requests (in my case modified port to 80 instead of 3000)

# curl -v http://<INSTANCE_IP>/http-request/

• Checked on EC2 instance xray data been successfully sent.

$ tail  -f  /var/log/xray/xray.log
[Info] Successfully sent batch of 1 segments (0.003 seconds)
[Info] Successfully sent batch of 1 segments (0.005 seconds)
[Info] Successfully sent batch of 2 segments (0.005 seconds)
[Info] Successfully sent batch of 1 segments (0.121 seconds)
[Info] Successfully sent batch of 1 segments (0.015 seconds) 

• Monitor Sampling rules (AWS X-Ray> Sampling), you can see data in Trend graph

---

Please try on your above in your test environment.

Centralized sampling rules are stored in X-Ray service. X-Ray sdk in user application has to take time to poll it and this poll action is lazy initialization, triggered by the first segment. So, the first segment generated in user application will fall back to a local rule (1/sec + 5%), which means the first segment is always sampled.

X-Ray centralized sampling has some limits, it does not work well in such cases:

1. Instrumented application's life is short. Centralized sampling rules is not ready to take effect.
2. If user's fleet scale is large, poll centralized sampling rules and sync up status with X-Ray service would take more time.

The alternative solution is using local sampling rules.