Greengrass v2 - Issues with MQTT reconnecting AWS_ERROR_MQTT5_OPERATION_FAILED_DUE_TO_OFFLINE_QUEUE_POLICY(5156)

0

We have a greengrass core device installed in the field and it will loose internet occasionally. When I create new deployments it sometimes the core device does not respond. the last few messages in the log show:

2023-05-31T13:29:31.620Z [INFO] (AwsEventLoop 4) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Connection resumed. {clientId=IoTThingName#2, sessionPresent=true}
2023-05-31T13:32:11.294Z [INFO] (pool-2-thread-590) com.aws.greengrass.tes.CredentialRequestHandler: Received IAM credentials that will be cached until 2023-05-31T14:27:11Z. {iotCredentialsPath=/role-aliases/RoleAlias/credentials}

I can connect to the device remotely so I know there is an internet connection. This is the rest of the logs prior:

2023-05-31T13:27:06.906Z [WARN] (AwsEventLoop 2) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Connection interrupted. {clientId=IoTThingName, error=Remote endpoint did not respond to a PINGREQ before timeout exceeded}
2023-05-31T13:27:13.798Z [ERROR] (AwsEventLoop 2) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName, error=A query to dns failed to resolve.}
2023-05-31T13:27:18.137Z [WARN] (AwsEventLoop 4) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Connection interrupted. {clientId=IoTThingName#2, error=Remote endpoint did not respond to a PINGREQ before timeout exceeded}
2023-05-31T13:27:21.286Z [ERROR] (AwsEventLoop 2) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName, error=A query to dns failed to resolve.}
2023-05-31T13:27:21.286Z [ERROR] (AwsEventLoop 4) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName#2, error=A query to dns failed to resolve.}
2023-05-31T13:27:26.516Z [ERROR] (AwsEventLoop 4) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName#2, error=A query to dns failed to resolve.}
2023-05-31T13:27:26.517Z [ERROR] (AwsEventLoop 2) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName, error=A query to dns failed to resolve.}
2023-05-31T13:27:29.176Z [ERROR] (AwsEventLoop 4) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName#2, error=A query to dns failed to resolve.}
2023-05-31T13:27:33.297Z [ERROR] (AwsEventLoop 2) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName, error=A query to dns failed to resolve.}
2023-05-31T13:27:34.189Z [ERROR] (AwsEventLoop 4) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName#2, error=A query to dns failed to resolve.}
2023-05-31T13:27:53.414Z [ERROR] (AwsEventLoop 2) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName, error=A query to dns failed to resolve.}
2023-05-31T13:27:53.414Z [ERROR] (AwsEventLoop 4) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName#2, error=A query to dns failed to resolve.}
2023-05-31T13:28:05.830Z [ERROR] (AwsEventLoop 4) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName#2, error=A query to dns failed to resolve.}
2023-05-31T13:28:05.830Z [ERROR] (AwsEventLoop 2) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName, error=A query to dns failed to resolve.}
2023-05-31T13:28:17.158Z [ERROR] (AwsEventLoop 4) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName#2, error=A query to dns failed to resolve.}
2023-05-31T13:28:53.204Z [INFO] (AwsEventLoop 2) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Connection resumed. {clientId=IoTThingName, sessionPresent=true}
2023-05-31T13:28:53.206Z [INFO] (AwsEventLoop 2) com.aws.greengrass.status.FleetStatusService: fss-status-update-published. Status update published to FSS. {trigger=RECONNECT, serviceName=FleetStatusService, currentState=RUNNING}
2023-05-31T13:28:53.208Z [ERROR] (AwsEventLoop 4) com.aws.greengrass.mqttclient.MqttClient: Failed to publish the message via Spooler and will retry. {}
java.util.concurrent.CompletionException: software.amazon.awssdk.crt.CrtRuntimeException: Mqtt5 operation failed due to a disconnection event in conjunction with the client's offline queue retention policy. AWS_ERROR_MQTT5_OPERATION_FAILED_DUE_TO_OFFLINE_QUEUE_POLICY(5156)
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
        at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
        at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:632)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
Caused by: software.amazon.awssdk.crt.CrtRuntimeException: Mqtt5 operation failed due to a disconnection event in conjunction with the client's offline queue retention policy. AWS_ERROR_MQTT5_OPERATION_FAILED_DUE_TO_OFFLINE_QUEUE_POLICY(5156)

2023-05-31T13:28:53.211Z [ERROR] (AwsEventLoop 4) com.aws.greengrass.mqttclient.MqttClient: Failed to publish the message via Spooler and will retry. {}
java.util.concurrent.CompletionException: software.amazon.awssdk.crt.CrtRuntimeException: Mqtt5 operation failed due to a disconnection event in conjunction with the client's offline queue retention policy. AWS_ERROR_MQTT5_OPERATION_FAILED_DUE_TO_OFFLINE_QUEUE_POLICY(5156)
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
        at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
        at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:632)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
Caused by: software.amazon.awssdk.crt.CrtRuntimeException: Mqtt5 operation failed due to a disconnection event in conjunction with the client's offline queue retention policy. AWS_ERROR_MQTT5_OPERATION_FAILED_DUE_TO_OFFLINE_QUEUE_POLICY(5156)

2023-05-31T13:28:53.212Z [ERROR] (AwsEventLoop 4) com.aws.greengrass.mqttclient.MqttClient: Failed to publish the message via Spooler and will retry. {}
java.util.concurrent.CompletionException: software.amazon.awssdk.crt.CrtRuntimeException: Mqtt5 operation failed due to a disconnection event in conjunction with the client's offline queue retention policy. AWS_ERROR_MQTT5_OPERATION_FAILED_DUE_TO_OFFLINE_QUEUE_POLICY(5156)
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
        at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
        at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:632)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
Caused by: software.amazon.awssdk.crt.CrtRuntimeException: Mqtt5 operation failed due to a disconnection event in conjunction with the client's offline queue retention policy. AWS_ERROR_MQTT5_OPERATION_FAILED_DUE_TO_OFFLINE_QUEUE_POLICY(5156)

2023-05-31T13:28:53.424Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.425Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.425Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.425Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.425Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.425Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.425Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.425Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.425Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.426Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.426Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.426Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.426Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.426Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.426Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.426Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.426Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.427Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.427Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.427Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.427Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.427Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.427Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.427Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}

2023-05-31T13:29:31.620Z [INFO] (AwsEventLoop 4) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Connection resumed. {clientId=IoTThingName#2, sessionPresent=true}
2023-05-31T13:32:11.294Z [INFO] (pool-2-thread-590) com.aws.greengrass.tes.CredentialRequestHandler: Received IAM credentials that will be cached until 2023-05-31T14:27:11Z. {iotCredentialsPath=/role-aliases/RoleAlias/credentials}

If I manually restart the device the deployment is completed on reboot - What is causing this error?

  • HI Phil. We have the same problem. Did you figure out what the issue was? Thanks

已提問 1 年前檢視次數 683 次
1 個回答
0

Hello,

"A query to dns failed to resolve." means that your DNS resolver did not respond with the IP address(es) for AWS, so it was unable to connect to IoT Core. This would happen when you do not have internet, or your DNS provider is not working. Being able to log into the device using some other mechanism proves you have internet, but it does not necessarily prove that your DNS is working.

DNS was eventually able to resolve and get connected again to AWS.

The reason why MQTT initially disconnected was "Remote endpoint did not respond to a PINGREQ before timeout exceeded" which would typically mean that your internet connection did go out, at least briefly.

Cheers,

Michael

AWS
專家
已回答 9 個月前

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南