Greengrass v2 - Issues with MQTT reconnecting AWS_ERROR_MQTT5_OPERATION_FAILED_DUE_TO_OFFLINE_QUEUE_POLICY(5156)

0

We have a greengrass core device installed in the field and it will loose internet occasionally. When I create new deployments it sometimes the core device does not respond. the last few messages in the log show:

2023-05-31T13:29:31.620Z [INFO] (AwsEventLoop 4) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Connection resumed. {clientId=IoTThingName#2, sessionPresent=true}
2023-05-31T13:32:11.294Z [INFO] (pool-2-thread-590) com.aws.greengrass.tes.CredentialRequestHandler: Received IAM credentials that will be cached until 2023-05-31T14:27:11Z. {iotCredentialsPath=/role-aliases/RoleAlias/credentials}

I can connect to the device remotely so I know there is an internet connection. This is the rest of the logs prior:

2023-05-31T13:27:06.906Z [WARN] (AwsEventLoop 2) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Connection interrupted. {clientId=IoTThingName, error=Remote endpoint did not respond to a PINGREQ before timeout exceeded}
2023-05-31T13:27:13.798Z [ERROR] (AwsEventLoop 2) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName, error=A query to dns failed to resolve.}
2023-05-31T13:27:18.137Z [WARN] (AwsEventLoop 4) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Connection interrupted. {clientId=IoTThingName#2, error=Remote endpoint did not respond to a PINGREQ before timeout exceeded}
2023-05-31T13:27:21.286Z [ERROR] (AwsEventLoop 2) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName, error=A query to dns failed to resolve.}
2023-05-31T13:27:21.286Z [ERROR] (AwsEventLoop 4) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName#2, error=A query to dns failed to resolve.}
2023-05-31T13:27:26.516Z [ERROR] (AwsEventLoop 4) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName#2, error=A query to dns failed to resolve.}
2023-05-31T13:27:26.517Z [ERROR] (AwsEventLoop 2) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName, error=A query to dns failed to resolve.}
2023-05-31T13:27:29.176Z [ERROR] (AwsEventLoop 4) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName#2, error=A query to dns failed to resolve.}
2023-05-31T13:27:33.297Z [ERROR] (AwsEventLoop 2) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName, error=A query to dns failed to resolve.}
2023-05-31T13:27:34.189Z [ERROR] (AwsEventLoop 4) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName#2, error=A query to dns failed to resolve.}
2023-05-31T13:27:53.414Z [ERROR] (AwsEventLoop 2) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName, error=A query to dns failed to resolve.}
2023-05-31T13:27:53.414Z [ERROR] (AwsEventLoop 4) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName#2, error=A query to dns failed to resolve.}
2023-05-31T13:28:05.830Z [ERROR] (AwsEventLoop 4) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName#2, error=A query to dns failed to resolve.}
2023-05-31T13:28:05.830Z [ERROR] (AwsEventLoop 2) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName, error=A query to dns failed to resolve.}
2023-05-31T13:28:17.158Z [ERROR] (AwsEventLoop 4) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName#2, error=A query to dns failed to resolve.}
2023-05-31T13:28:53.204Z [INFO] (AwsEventLoop 2) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Connection resumed. {clientId=IoTThingName, sessionPresent=true}
2023-05-31T13:28:53.206Z [INFO] (AwsEventLoop 2) com.aws.greengrass.status.FleetStatusService: fss-status-update-published. Status update published to FSS. {trigger=RECONNECT, serviceName=FleetStatusService, currentState=RUNNING}
2023-05-31T13:28:53.208Z [ERROR] (AwsEventLoop 4) com.aws.greengrass.mqttclient.MqttClient: Failed to publish the message via Spooler and will retry. {}
java.util.concurrent.CompletionException: software.amazon.awssdk.crt.CrtRuntimeException: Mqtt5 operation failed due to a disconnection event in conjunction with the client's offline queue retention policy. AWS_ERROR_MQTT5_OPERATION_FAILED_DUE_TO_OFFLINE_QUEUE_POLICY(5156)
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
        at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
        at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:632)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
Caused by: software.amazon.awssdk.crt.CrtRuntimeException: Mqtt5 operation failed due to a disconnection event in conjunction with the client's offline queue retention policy. AWS_ERROR_MQTT5_OPERATION_FAILED_DUE_TO_OFFLINE_QUEUE_POLICY(5156)

2023-05-31T13:28:53.211Z [ERROR] (AwsEventLoop 4) com.aws.greengrass.mqttclient.MqttClient: Failed to publish the message via Spooler and will retry. {}
java.util.concurrent.CompletionException: software.amazon.awssdk.crt.CrtRuntimeException: Mqtt5 operation failed due to a disconnection event in conjunction with the client's offline queue retention policy. AWS_ERROR_MQTT5_OPERATION_FAILED_DUE_TO_OFFLINE_QUEUE_POLICY(5156)
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
        at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
        at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:632)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
Caused by: software.amazon.awssdk.crt.CrtRuntimeException: Mqtt5 operation failed due to a disconnection event in conjunction with the client's offline queue retention policy. AWS_ERROR_MQTT5_OPERATION_FAILED_DUE_TO_OFFLINE_QUEUE_POLICY(5156)

2023-05-31T13:28:53.212Z [ERROR] (AwsEventLoop 4) com.aws.greengrass.mqttclient.MqttClient: Failed to publish the message via Spooler and will retry. {}
java.util.concurrent.CompletionException: software.amazon.awssdk.crt.CrtRuntimeException: Mqtt5 operation failed due to a disconnection event in conjunction with the client's offline queue retention policy. AWS_ERROR_MQTT5_OPERATION_FAILED_DUE_TO_OFFLINE_QUEUE_POLICY(5156)
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
        at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
        at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:632)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
Caused by: software.amazon.awssdk.crt.CrtRuntimeException: Mqtt5 operation failed due to a disconnection event in conjunction with the client's offline queue retention policy. AWS_ERROR_MQTT5_OPERATION_FAILED_DUE_TO_OFFLINE_QUEUE_POLICY(5156)

2023-05-31T13:28:53.424Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.425Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.425Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.425Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.425Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.425Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.425Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.425Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.425Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.426Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.426Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.426Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.426Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.426Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.426Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.426Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.426Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.427Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.427Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.427Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.427Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.427Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.427Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.427Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}

2023-05-31T13:29:31.620Z [INFO] (AwsEventLoop 4) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Connection resumed. {clientId=IoTThingName#2, sessionPresent=true}
2023-05-31T13:32:11.294Z [INFO] (pool-2-thread-590) com.aws.greengrass.tes.CredentialRequestHandler: Received IAM credentials that will be cached until 2023-05-31T14:27:11Z. {iotCredentialsPath=/role-aliases/RoleAlias/credentials}

If I manually restart the device the deployment is completed on reboot - What is causing this error?

  • HI Phil. We have the same problem. Did you figure out what the issue was? Thanks

質問済み 1年前683ビュー
1回答
0

Hello,

"A query to dns failed to resolve." means that your DNS resolver did not respond with the IP address(es) for AWS, so it was unable to connect to IoT Core. This would happen when you do not have internet, or your DNS provider is not working. Being able to log into the device using some other mechanism proves you have internet, but it does not necessarily prove that your DNS is working.

DNS was eventually able to resolve and get connected again to AWS.

The reason why MQTT initially disconnected was "Remote endpoint did not respond to a PINGREQ before timeout exceeded" which would typically mean that your internet connection did go out, at least briefly.

Cheers,

Michael

AWS
エキスパート
回答済み 9ヶ月前

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ