AWS XRay Demon Connection Issue

0

I created an linux basdd EC2 instance and installed the XRay Demon using the user data script as mentioned here: https://docs.aws.amazon.com/xray/latest/devguide/xray-daemon-ec2.html

I am running the django application with following configuration

    "AWS_XRAY_DAEMON_ADDRESS": "18.199.236.72:2000",
    "AUTO_INSTRUMENT": True,
    "AWS_XRAY_CONTEXT_MISSING": "LOG_ERROR",
    "AWS_XRAY_TRACING_NAME": "ExploApp",
    "PATCH_MODULES": ["requests"],
    "PLUGINS": ("ElasticBeanstalkPlugin",),
    "SAMPLING": True,
    "SAMPLING_RULES": {
        "version": 1,
        "rules": [
            {
                "description": "Exclude health checks",
                "service_name": "*",
                "http_method": "GET",
                "url_path": "/health/",
                "fixed_target": 0,
                "rate": 0.0
            }
        ],
        "default": {
            "fixed_target": 1,
            "rate": 0.1
        }
    },
    "DYNAMIC_NAMING": None,
    "STREAMING_THRESHOLD": None,
}
MIDDLEWARE = [
    'aws_xray_sdk.ext.django.middleware.XRayMiddleware',
... ]
INSTALLED_APPS = [
    'aws_xray_sdk.ext.django',
... ]

Django application is hosted on django based beanstalk environment.

The app is not able to connect to the demon running on different Ec2 instance. I have attached the xray full access perimission to both beanstank and ec2 instance. Here is the error

 File "/usr/local/lib/python3.9/site-packages/botocore/endpoint.py", line 281, in _do_get_response
    http_response = self._send(request)
  File "/usr/local/lib/python3.9/site-packages/botocore/endpoint.py", line 377, in _send
    return self.http_session.send(request)
  File "/usr/local/lib/python3.9/site-packages/botocore/httpsession.py", line 494, in send
    raise EndpointConnectionError(endpoint_url=request.url, error=e)
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://18.199.236.72:2000/GetSamplingRules"
Encountered an issue while polling sampling rules.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 169, in _new_conn
    conn = connection.create_connection(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/botocore/httpsession.py", line 465, in send
    urllib_response = conn.urlopen(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 507, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.9/site-packages/urllib3/packages/six.py", line 735, in reraise
    raise value
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.9/http/client.py", line 1285, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.9/site-packages/botocore/awsrequest.py", line 94, in _send_request
    rval = super()._send_request(
  File "/usr/local/lib/python3.9/http/client.py", line 1331, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1280, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.9/site-packages/botocore/awsrequest.py", line 123, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.9/site-packages/botocore/awsrequest.py", line 218, in send
    return super().send(str)
  File "/usr/local/lib/python3.9/http/client.py", line 980, in send
    self.connect()
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 200, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 181, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPConnection object at 0x7f820b24c820>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/aws_xray_sdk/core/sampling/rule_poller.py", line 48, in _refresh_cache
    new_rules = self._connector.fetch_sampling_rules()
  File "/usr/local/lib/python3.9/site-packages/aws_xray_sdk/core/sampling/connector.py", line 41, in wrapper
    result = func(self, *args, **kargs)
  File "/usr/local/lib/python3.9/site-packages/aws_xray_sdk/core/sampling/connector.py", line 55, in fetch_sampling_rules
    resp = self._xray_client.get_sampling_rules()
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 947, in _make_api_call
    http, parsed_response = self._make_request(
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 970, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/usr/local/lib/python3.9/site-packages/botocore/endpoint.py", line 119, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/local/lib/python3.9/site-packages/botocore/endpoint.py", line 202, in _send_request
    while self._needs_retry(
  File "/usr/local/lib/python3.9/site-packages/botocore/endpoint.py", line 354, in _needs_retry
    responses = self._event_emitter.emit(
  File "/usr/local/lib/python3.9/site-packages/botocore/hooks.py", line 412, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/hooks.py", line 256, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/hooks.py", line 239, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/retryhandler.py", line 207, in __call__
    if self._checker(**checker_kwargs):
  File "/usr/local/lib/python3.9/site-packages/botocore/retryhandler.py", line 284, in __call__
    should_retry = self._should_retry(
  File "/usr/local/lib/python3.9/site-packages/botocore/retryhandler.py", line 320, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/usr/local/lib/python3.9/site-packages/botocore/retryhandler.py", line 363, in __call__
    checker_response = checker(
  File "/usr/local/lib/python3.9/site-packages/botocore/retryhandler.py", line 247, in __call__
    return self._check_caught_exception(
  File "/usr/local/lib/python3.9/site-packages/botocore/retryhandler.py", line 416, in _check_caught_exception
    raise caught_exception
  File "/usr/local/lib/python3.9/site-packages/botocore/endpoint.py", line 281, in _do_get_response
    http_response = self._send(request)
  File "/usr/local/lib/python3.9/site-packages/botocore/endpoint.py", line 377, in _send
    return self.http_session.send(request)
  File "/usr/local/lib/python3.9/site-packages/botocore/httpsession.py", line 494, in send
    raise EndpointConnectionError(endpoint_url=request.url, error=e)
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://18.199.236.72:2000/GetSamplingRules"
asked 14 days ago41 views
1 Answer
0

It appears that your Django application is unable to connect to the X-Ray daemon running on a separate EC2 instance. This connection issue is likely due to network configuration or security group settings. Here are some steps you can take to troubleshoot and resolve the issue:

  1. Ensure that the security group for the EC2 instance running the X-Ray daemon allows inbound traffic on port 2000 from the Elastic Beanstalk environment where your Django application is hosted.

  2. Verify that the X-Ray daemon is actually running on the EC2 instance. You can check this by SSH-ing into the instance and running a command like ps aux | grep xray to see if the process is active.

  3. Make sure the IP address you're using for the X-Ray daemon (18.199.236.72 in your configuration) is correct and hasn't changed. EC2 instances can have their public IP addresses change when stopped and started unless you're using an Elastic IP.

  4. Instead of specifying the public IP address of the EC2 instance, consider using its private IP address if both the Elastic Beanstalk environment and the EC2 instance are in the same VPC.

  5. Check if there are any VPC or network ACL rules that might be blocking the connection between your Elastic Beanstalk environment and the EC2 instance.

  6. As an alternative approach, you might want to consider running the X-Ray daemon on the same Elastic Beanstalk environment as your Django application. This can be done by including the daemon installation in your Elastic Beanstalk configuration files.

  7. If you continue to have issues, you may want to temporarily set up CloudWatch Logs for the X-Ray daemon on the EC2 instance to get more detailed information about any connection problems it might be experiencing.

Remember to restart your Django application after making any changes to the configuration. If the problem persists, you may need to review your overall architecture and consider alternative setups, such as running the X-Ray daemon as a sidecar container if you're using ECS, or leveraging AWS managed services that integrate with X-Ray more seamlessly.
Sources
Getting started with the Scorekeep sample application - AWS X-Ray
AWS X-Ray Pricing

profile picture
answered 14 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions