aws.greengrass.Cloudwatch - Broken when network disconnects

Question

Hello,  
  
I am currently trying the 'aws.greengrass.Cloudwatch' component in order to save various metrics to Cloudwatch. The Metrics are getting written as expected, and I am able to view them using my AWS Console.  
  
However, when the device is disconnected from the Internet for few minutes, the 'aws.greengrass.Cloudwatch' ceases to work and exits unsuccessfully. After three attempts, this causes the component to go to the 'BROKEN' state, causing no more metrics to be uploaded, thus rendering the whole deployment non-functional.  
  
A minimum working example on how to reproduce this issue follows:  
  
Cloudwatch=2.0.7  
Nucleus=2.0.4  
  
Consider the following component written in Python. Let us call it 'Thermometer', this component has 'aws.greengrass.Cloudwatch' as one of its dependencies, and is configured as suggested by  . This component publishes a message with the Temperature info to the default Cloudwatch topic. The code can be found below:  
  
```python  
def _break_cloudwatch():  
    def _publish_one_message(metric_name, value, unit):  
        request = PublishToTopicRequest(topic='cloudwatch/metric/put')  
        publish_message = PublishMessage()  
        publish_message.binary_message = BinaryMessage()  
        publish_message.binary_message.message = bytes(json.dumps({  
            "request": {  
                "namespace": "Greengrass",  
                "metricData": {"metricName": f"{metric_name}",  
                               "value": float(value),  
                               "unit": f"{unit}"}}}),  
            'utf-8')  
  
        request.publish_message = publish_message  
        operation = client.new_publish_to_topic()  
        operation.activate(request)  
        return operation.get_response()  
  
    while True:  
        client = ipc.connect()  
        try:  
            _publish_one_message('Temp', 30, 'Count')  
        except Exception as e:  
            logger.warning(f'caught {e}')  
        time.sleep(5)  
```  
  
Now all we need to do it so disconnect the device from the Internet. After a while, the Cloudwatch component will fail three times in a row, causing it to enter a BROKEN state, which hinders the proper functionality of the deployment as a whole. A log from the Cloudwatch component can be found below:  
```  
2021-10-22T11:09:43.903Z \[WARN] (pool-2-thread-222) aws.greengrass.Cloudwatch: ipc_client.py:163,deprecated arg port=8000 will be ignored. {serviceInstance=0, serviceName=aws.greengrass.Cloudwatch, currentState=RUNNING}  
2021-10-22T11:09:57.846Z \[ERROR] (pool-2-thread-291) aws.greengrass.Cloudwatch: client.py:21,Error was encountered publishing to cloudwatch:. {serviceInstance=0, serviceName=aws.greengrass.Cloudwatch, currentState=RUNNING}  
2021-10-22T11:10:11.803Z \[ERROR] (pool-2-thread-291) aws.greengrass.Cloudwatch: client.py:21,Error was encountered publishing to cloudwatch:. {serviceInstance=0, serviceName=aws.greengrass.Cloudwatch, currentState=RUNNING}  
2021-10-22T11:10:21.055Z \[ERROR] (pool-2-thread-291) aws.greengrass.Cloudwatch: client.py:21,Error was encountered publishing to cloudwatch:. {serviceInstance=0, serviceName=aws.greengrass.Cloudwatch, currentState=RUNNING}  
2021-10-22T11:10:28.772Z \[ERROR] (pool-2-thread-291) aws.greengrass.Cloudwatch: client.py:21,Error was encountered publishing to cloudwatch:. {serviceInstance=0, serviceName=aws.greengrass.Cloudwatch, currentState=RUNNING}  
2021-10-22T11:10:38.629Z \[ERROR] (pool-2-thread-291) aws.greengrass.Cloudwatch: client.py:21,Error was encountered publishing to cloudwatch:. {serviceInstance=0, serviceName=aws.greengrass.Cloudwatch, currentState=RUNNING}  
2021-10-22T11:10:55.368Z \[ERROR] (pool-2-thread-291) aws.greengrass.Cloudwatch: client.py:21,Error was encountered publishing to cloudwatch:. {serviceInstance=0, serviceName=aws.greengrass.Cloudwatch, currentState=RUNNING}  
2021-10-22T11:11:06.687Z \[ERROR] (pool-2-thread-291) aws.greengrass.Cloudwatch: client.py:21,Error was encountered publishing to cloudwatch:. {serviceInstance=0, serviceName=aws.greengrass.Cloudwatch, currentState=RUNNING}  
2021-10-22T11:11:08.172Z \[ERROR] (pool-2-thread-291) aws.greengrass.Cloudwatch: client.py:21,Error was encountered publishing to cloudwatch:. {serviceInstance=0, serviceName=aws.greengrass.Cloudwatch, currentState=RUNNING}  
2021-10-22T11:11:12.734Z \[INFO] (pool-2-thread-28) aws.greengrass.Cloudwatch: shell-runner-start. {scriptName=services.aws.greengrass.Cloudwatch.lifecycle.shutdown.script, serviceInstance=0, serviceName=aws.greengrass.Cloudwatch, currentState=BROKEN, command=\["/greengrass/v2/packages/artifacts/aws.greengrass.LambdaLauncher/2.0.8/lambda-l..."]}  
2021-10-22T11:11:12.780Z \[INFO] (pool-2-thread-204) aws.greengrass.Cloudwatch: lambda_runtime.py:370,Caught signal 15. Stopping runtime.. {serviceInstance=0, serviceName=aws.greengrass.Cloudwatch, currentState=BROKEN}  
2021-10-22T11:11:20.708Z \[ERROR] (pool-2-thread-291) aws.greengrass.Cloudwatch: client.py:21,Error was encountered publishing to cloudwatch:. {serviceInstance=0, serviceName=aws.greengrass.Cloudwatch, currentState=BROKEN}  
```   
As we can see, the component exits unsuccessfully, and enters BROKEN state after around 3 minutes.  
  
Is there something that I could have missed concerning the configuration of the deployment? Also, is there perhaps another way of publishing the metrics to AWS Cloundwatch which avoids breaking the 'Cloudwatch' component?  
  
Thank you,  
Simon  
  
Edited by: SimonManour on Oct 22, 2021 4:29 AM

AWS-User-zhaoylin · Answer

Hi SimonManour,  
  
we noticed a support ticket was opened for you for this issue. We have replied and will be tracking there.   
To also update this thread, we have identified the root cause is that in no internet scenario put metric call, awssdk retries several times and sleeps in between retries resulting in lambda timeout for Cloudwatch which is 3 seconds. As a workaround you should be able to override the timeout by doing a new deployment increasing the timeoutInSeconds config parameter to 10-15 seconds here (https://us-west-2.console.aws.amazon.com/iot/home?region=us-west-2#/greengrass/v2/components/public/aws.greengrass.Cloudwatch/versions/2.0.7). This is a bug as there should not be any blocking code inside the function handler and we will be working on that.  
  
Thank you,  
zhaoylin

Simon · Answer

To elaborate a bit more, the following error message gets posted inside the main 'greengrass.log'  
  
```  
2021-10-22T11:11:12.733Z \[ERROR] (pool-1-thread-3) com.aws.greengrass.lambdamanager.WorkManager: work-item-timeout. lambda work item timed out. {workItem=76bbb577-fe91-4fd4-a08c-35ea2cb7ab80, arn=arn:aws:lambda:eu-west-1:aws:function:aws-greengrass-cloudwatch:5}  
2021-10-22T11:11:12.733Z \[ERROR] (pool-1-thread-3) com.aws.greengrass.lambdamanager.UserLambdaService: service-errored. {serviceInstance=0, serviceName=aws.greengrass.Cloudwatch, currentState=RUNNING}  
com.aws.greengrass.lambdamanager.WorkItemTimeoutException: The work item with invocation id 76bbb577-fe91-4fd4-a08c-35ea2cb7ab80 has timed out, worker lambda arn: arn:aws:lambda:eu-west-1:aws:function:aws-greengrass-cloudwatch:5  
	at com.aws.greengrass.lambdamanager.WorkManager.lambda$getNextWork$12(WorkManager.java:332)  
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)  
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)  
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)  
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)  
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)  
	at java.base/java.lang.Thread.run(Thread.java:829)  
  
2021-10-22T11:11:12.733Z \[INFO] (pool-1-thread-3) com.aws.greengrass.lambdamanager.UserLambdaService: service-report-state. {serviceInstance=0, serviceName=aws.greengrass.Cloudwatch, currentState=RUNNING, newState=ERRORED}  
2021-10-22T11:11:12.734Z \[INFO] (aws.greengrass.Cloudwatch-lifecycle) com.aws.greengrass.lambdamanager.UserLambdaService: service-set-state. {serviceInstance=0, serviceName=aws.greengrass.Cloudwatch, currentState=RUNNING, newState=BROKEN}  
  
```

Simon · Answer

Hello,  
thank you for the provided information. So far, I can confirm that by updating Cloudwatch component to version 3.0.0 and using the configuration you have recommended, the service has indeed started to work as expected. The component does not get BROKEN when the network is disconnected, and properly re-establishes the connection as soon as the network is reachable again and stays RUNNING even after a prolonged period of network downtime.  
  
Have a nice day,  
Simon

aws.greengrass.Cloudwatch - Broken when network disconnects

Add your answer

Relevant content