Error in Greengrass component when using boto3 iot client

0

Please note this a port of a question from the old forums: https://forums.aws.amazon.com/thread.jspa?threadID=348553&tstart=0

Using greengrass v2:

We are using the python boto3 'iot' client as part of our greengrass components, to get some of the core iot thing attributes. I have attached sample code as "test_boto3.py" below.

When we attempt to create the iot client, the application crashes with many errors - "crash_logfile.txt", which is attached, shows the complete output of the Greengrass logs for that component.

The relevant component recipe and logfiles are attached in the linked post, I'm unsure how to upload them to this post.

asked 2 years ago597 views
5 Answers
0

Hi,

We have seen multiple reports where components are using an incorrect port to connect to the Nucleus credential provider and are actively looking to root cause the issue.

You could help us by telling us a little bit about your device and any events that led up to the device not behaving properly.

Is it a low end device such as a Raspberry Pi? We are curious about it's ability keep accurate system time - is the time accurate immediately after a system reboot or does it require NTP to synchronize time first? Were there any device reboots leading up to the issue?

Thanks, -joe

AWS
answered 2 years ago
  • Hi Joe,

    Following up here on behalf of my colleague.

    The issue arose when we were carrying out a manual deployment to fix a broken component. We're using an Nvidia production device from the Jetson family. The time is accurate on reboot.

    The device had not been rebooted prior to this issue. We have since rebooted the device which does not solve the issue.

  • Hi Joe,

    Any follow up on this issue or a suggested workaround ?

    The device is using the htpdate.service to synchronise time.

0

Hi Janice,

Thanks for the follow-up, yes, the issue in that case lay with DNS. Adding a fallback DNS was able to solve our issue for us.

We are seeing a new issue now with a different greengrass core device, this time (or so I believe) the component crashes when it tries to connect to the Token Exchange Service component.

I have attached anonymised logs below.

Component Dependency Defined in Recipe:

  "ComponentDependencies": {
    "aws.greengrass.TokenExchangeService": {
      "VersionRequirement": ">=2.0.0 <3.0.0",
      "DependencyType": "HARD"
    }
  },

Greengrass Cli "component list" command showing the Token Exchange Service as running:

Component Name: aws.greengrass.TokenExchangeService
    Version: 2.0.3
    State: RUNNING
    Configuration: {"port":0.0}

Component Logs:

shell-runner-start. {scriptName=services.com.edge.ComponentName.lifecycle.Run, serviceName=com.edge.ComponentName, currentState=STARTING, command=["python3 -u /greengrass/v2/packages/artifacts/com.edge.ComponentName/3.2.2/p..."]}
INFO:root:Main loop running. 
Traceback (most recent call last):. 
File "/home/ggc_user/.local/lib/python3.6/site-packages/urllib3/connection.py", line 175, in _new_conn. 
(self._dns_host, self.port), self.timeout, **extra_kw. 
File "/home/ggc_user/.local/lib/python3.6/site-packages/urllib3/util/connection.py", line 95, in create_connection. 
raise err. 
File "/home/ggc_user/.local/lib/python3.6/site-packages/urllib3/util/connection.py", line 85, in create_connection. 
sock.connect(sa). 
ConnectionRefusedError: [Errno 111] Connection refused. 

During handling of the above exception, another exception occurred:. 

Traceback (most recent call last):. 
File "/home/ggc_user/.local/lib/python3.6/site-packages/botocore/httpsession.py", line 412, in send. 
chunked=self._chunked(request.headers),. 
File "/home/ggc_user/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 786, in urlopen. 
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]. 
File "/home/ggc_user/.local/lib/python3.6/site-packages/urllib3/util/retry.py", line 525, in increment. 
raise six.reraise(type(error), error, _stacktrace). 
File "/home/ggc_user/.local/lib/python3.6/site-packages/urllib3/packages/six.py", line 770, in reraise. 
raise value. 
File "/home/ggc_user/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 710, in urlopen. 
chunked=chunked,. 
File "/home/ggc_user/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 398, in _make_request. 
conn.request(method, url, **httplib_request_kw). 
File "/home/ggc_user/.local/lib/python3.6/site-packages/urllib3/connection.py", line 239, in request. 
super(HTTPConnection, self).request(method, url, body=body, headers=headers). 
File "/usr/lib/python3.6/http/client.py", line 1281, in request. 
self._send_request(method, url, body, headers, encode_chunked). 
File "/home/ggc_user/.local/lib/python3.6/site-packages/botocore/awsrequest.py", line 93, in _send_request. 
method, url, body, headers, *args, **kwargs). 
File "/usr/lib/python3.6/http/client.py", line 1327, in _send_request. 
self.endheaders(body, encode_chunked=encode_chunked). 
File "/usr/lib/python3.6/http/client.py", line 1276, in endheaders. 
self._send_output(message_body, encode_chunked=encode_chunked). 
File "/home/ggc_user/.local/lib/python3.6/site-packages/botocore/awsrequest.py", line 120, in _send_output. 
self.send(msg). 
File "/home/ggc_user/.local/lib/python3.6/site-packages/botocore/awsrequest.py", line 204, in send. 
return super(AWSConnection, self).send(str). 
File "/usr/lib/python3.6/http/client.py", line 980, in send. 
self.connect(). 
File "/home/ggc_user/.local/lib/python3.6/site-packages/urllib3/connection.py", line 205, in connect. 
conn = self._new_conn(). 
File "/home/ggc_user/.local/lib/python3.6/site-packages/urllib3/connection.py", line 187, in _new_conn. 
self, "Failed to establish a new connection: %s" % e. 
urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPConnection object at 0x7f8db65ef0>: Failed to establish a new connection: [Errno 111] Connection refused. 

During handling of the above exception, another exception occurred:. 

Traceback (most recent call last):. 
File "/home/ggc_user/.local/lib/python3.6/site-packages/botocore/utils.py", line 2476, in _get_response. 
response = self._session.send(request.prepare()). 
File "/home/ggc_user/.local/lib/python3.6/site-packages/botocore/httpsession.py", line 432, in send. 
raise EndpointConnectionError(endpoint_url=request.url, error=e). 
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://localhost:38875/2016-11-01/credentialprovider/". 

During handling of the above exception, another exception occurred:. 

Traceback (most recent call last):. 
File "/home/ggc_user/.local/lib/python3.6/site-packages/botocore/credentials.py", line 1879, in fetch_creds. 
full_uri, headers=headers). 
File "/home/ggc_user/.local/lib/python3.6/site-packages/botocore/utils.py", line 2426, in retrieve_full_uri. 
return self._retrieve_credentials(full_url, headers). 
File "/home/ggc_user/.local/lib/python3.6/site-packages/botocore/utils.py", line 2463, in _retrieve_credentials. 
full_url, headers, self.TIMEOUT_SECONDS). 
File "/home/ggc_user/.local/lib/python3.6/site-packages/botocore/utils.py", line 2494, in _get_response. 
raise MetadataRetrievalError(error_msg=error_msg). 
botocore.exceptions.MetadataRetrievalError: Error retrieving metadata: Received error when attempting to retrieve ECS metadata: Could not connect to the endpoint URL: "http://localhost:38875/2016-11-01/credentialprovider/". 

During handling of the above exception, another exception occurred:. 

Traceback (most recent call last):. 
File "/greengrass/v2/packages/artifacts/com.edge.ComponentName/3.2.2/component_script.py", line 299, in <module>. 
get_thing_details(). 
File "/greengrass/v2/packages/artifacts/com.edge.ComponentName/3.2.2/component_script.py", line 227, in get_thing_details. 
iot_client = boto3.client('iot'). 
File "/home/ggc_user/.local/lib/python3.6/site-packages/boto3/__init__.py", line 93, in client. 
return _get_default_session().client(*args, **kwargs). 
File "/home/ggc_user/.local/lib/python3.6/site-packages/boto3/session.py", line 275, in client. 
aws_session_token=aws_session_token, config=config). 
File "/home/ggc_user/.local/lib/python3.6/site-packages/botocore/session.py", line 855, in create_client. 
credentials = self.get_credentials(). 
File "/home/ggc_user/.local/lib/python3.6/site-packages/botocore/session.py", line 458, in get_credentials. 
'credential_provider').load_credentials(). 
File "/home/ggc_user/.local/lib/python3.6/site-packages/botocore/credentials.py", line 1985, in load_credentials. 
creds = provider.load(). 
File "/home/ggc_user/.local/lib/python3.6/site-packages/botocore/credentials.py", line 1849, in load. 
return self._retrieve_or_fail(). 
File "/home/ggc_user/.local/lib/python3.6/site-packages/botocore/credentials.py", line 1858, in _retrieve_or_fail. 
creds = fetcher(). 
File "/home/ggc_user/.local/lib/python3.6/site-packages/botocore/credentials.py", line 1884, in fetch_creds. 
error_msg=str(e)). 
botocore.exceptions.CredentialRetrievalError: Error when retrieving credentials from container-role: Error retrieving metadata: Received error when attempting to retrieve ECS metadata: Could not connect to the endpoint URL: "http://localhost:38875/2016-11-01/credentialprovider/". 

It seems to be an issue with connecting to the token exchange service. Do you have any guidance on this ?

answered 2 years ago
  • Can you please provide the Greengrass log at the time when the connection was unsuccessful? Please also tell us the exact version of Greengrass Nucleus in use.

0

Hello,

Picking up here from the discussion on aws forums.

Thank you for the log file. It looks like it is a DNS issue. Please check that the configured credential endpoint is correct and that it is able to be looked up with DNS.

Let us know if this does not resolve the problem.

Best, Janice

answered 2 years ago
0

Component Versions: aws.greengrass.Cli 2.5.3 aws.greengrass.Nucleus 2.5.0

I have attached the logs via pastebin, as the character limit and lack of attachment options don't allow me to post it here. https://pastebin.com/p3gd55A6

answered 2 years ago
0

I was wondering if there was an update on this subject or someone found a solution. From time to time this error occurs to my Greengrass devices as well and for me it seems that 'randomly' (I don't really know why yet) the token exchange service changes it's port and the token exchangeURL environment variable provided to the component on boot doesn't work anymore. Rebooting the greengrass service always solves the issue because the components get restarted with a new Token exchange service URL environment variable

Thanks in advance

Hacor

Hacor
answered 14 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions