By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Greengrass v2 - Issues with MQTT reconnecting AWS_ERROR_MQTT5_OPERATION_FAILED_DUE_TO_OFFLINE_QUEUE_POLICY(5156)

0

We have a greengrass core device installed in the field and it will loose internet occasionally. When I create new deployments it sometimes the core device does not respond. the last few messages in the log show:

2023-05-31T13:29:31.620Z [INFO] (AwsEventLoop 4) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Connection resumed. {clientId=IoTThingName#2, sessionPresent=true}
2023-05-31T13:32:11.294Z [INFO] (pool-2-thread-590) com.aws.greengrass.tes.CredentialRequestHandler: Received IAM credentials that will be cached until 2023-05-31T14:27:11Z. {iotCredentialsPath=/role-aliases/RoleAlias/credentials}

I can connect to the device remotely so I know there is an internet connection. This is the rest of the logs prior:

2023-05-31T13:27:06.906Z [WARN] (AwsEventLoop 2) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Connection interrupted. {clientId=IoTThingName, error=Remote endpoint did not respond to a PINGREQ before timeout exceeded}
2023-05-31T13:27:13.798Z [ERROR] (AwsEventLoop 2) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName, error=A query to dns failed to resolve.}
2023-05-31T13:27:18.137Z [WARN] (AwsEventLoop 4) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Connection interrupted. {clientId=IoTThingName#2, error=Remote endpoint did not respond to a PINGREQ before timeout exceeded}
2023-05-31T13:27:21.286Z [ERROR] (AwsEventLoop 2) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName, error=A query to dns failed to resolve.}
2023-05-31T13:27:21.286Z [ERROR] (AwsEventLoop 4) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName#2, error=A query to dns failed to resolve.}
2023-05-31T13:27:26.516Z [ERROR] (AwsEventLoop 4) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName#2, error=A query to dns failed to resolve.}
2023-05-31T13:27:26.517Z [ERROR] (AwsEventLoop 2) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName, error=A query to dns failed to resolve.}
2023-05-31T13:27:29.176Z [ERROR] (AwsEventLoop 4) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName#2, error=A query to dns failed to resolve.}
2023-05-31T13:27:33.297Z [ERROR] (AwsEventLoop 2) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName, error=A query to dns failed to resolve.}
2023-05-31T13:27:34.189Z [ERROR] (AwsEventLoop 4) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName#2, error=A query to dns failed to resolve.}
2023-05-31T13:27:53.414Z [ERROR] (AwsEventLoop 2) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName, error=A query to dns failed to resolve.}
2023-05-31T13:27:53.414Z [ERROR] (AwsEventLoop 4) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName#2, error=A query to dns failed to resolve.}
2023-05-31T13:28:05.830Z [ERROR] (AwsEventLoop 4) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName#2, error=A query to dns failed to resolve.}
2023-05-31T13:28:05.830Z [ERROR] (AwsEventLoop 2) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName, error=A query to dns failed to resolve.}
2023-05-31T13:28:17.158Z [ERROR] (AwsEventLoop 4) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Failed to connect to AWS IoT Core. {clientId=IoTThingName#2, error=A query to dns failed to resolve.}
2023-05-31T13:28:53.204Z [INFO] (AwsEventLoop 2) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Connection resumed. {clientId=IoTThingName, sessionPresent=true}
2023-05-31T13:28:53.206Z [INFO] (AwsEventLoop 2) com.aws.greengrass.status.FleetStatusService: fss-status-update-published. Status update published to FSS. {trigger=RECONNECT, serviceName=FleetStatusService, currentState=RUNNING}
2023-05-31T13:28:53.208Z [ERROR] (AwsEventLoop 4) com.aws.greengrass.mqttclient.MqttClient: Failed to publish the message via Spooler and will retry. {}
java.util.concurrent.CompletionException: software.amazon.awssdk.crt.CrtRuntimeException: Mqtt5 operation failed due to a disconnection event in conjunction with the client's offline queue retention policy. AWS_ERROR_MQTT5_OPERATION_FAILED_DUE_TO_OFFLINE_QUEUE_POLICY(5156)
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
        at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
        at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:632)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
Caused by: software.amazon.awssdk.crt.CrtRuntimeException: Mqtt5 operation failed due to a disconnection event in conjunction with the client's offline queue retention policy. AWS_ERROR_MQTT5_OPERATION_FAILED_DUE_TO_OFFLINE_QUEUE_POLICY(5156)

2023-05-31T13:28:53.211Z [ERROR] (AwsEventLoop 4) com.aws.greengrass.mqttclient.MqttClient: Failed to publish the message via Spooler and will retry. {}
java.util.concurrent.CompletionException: software.amazon.awssdk.crt.CrtRuntimeException: Mqtt5 operation failed due to a disconnection event in conjunction with the client's offline queue retention policy. AWS_ERROR_MQTT5_OPERATION_FAILED_DUE_TO_OFFLINE_QUEUE_POLICY(5156)
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
        at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
        at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:632)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
Caused by: software.amazon.awssdk.crt.CrtRuntimeException: Mqtt5 operation failed due to a disconnection event in conjunction with the client's offline queue retention policy. AWS_ERROR_MQTT5_OPERATION_FAILED_DUE_TO_OFFLINE_QUEUE_POLICY(5156)

2023-05-31T13:28:53.212Z [ERROR] (AwsEventLoop 4) com.aws.greengrass.mqttclient.MqttClient: Failed to publish the message via Spooler and will retry. {}
java.util.concurrent.CompletionException: software.amazon.awssdk.crt.CrtRuntimeException: Mqtt5 operation failed due to a disconnection event in conjunction with the client's offline queue retention policy. AWS_ERROR_MQTT5_OPERATION_FAILED_DUE_TO_OFFLINE_QUEUE_POLICY(5156)
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
        at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
        at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:632)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
Caused by: software.amazon.awssdk.crt.CrtRuntimeException: Mqtt5 operation failed due to a disconnection event in conjunction with the client's offline queue retention policy. AWS_ERROR_MQTT5_OPERATION_FAILED_DUE_TO_OFFLINE_QUEUE_POLICY(5156)

2023-05-31T13:28:53.424Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.425Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.425Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.425Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.425Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.425Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.425Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.425Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.425Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.426Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.426Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.426Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.426Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.426Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.426Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.426Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.426Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.427Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.427Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.427Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.427Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.427Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.427Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}
2023-05-31T13:28:53.427Z [INFO] (AwsEventLoop 2) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=IoTThingName}

2023-05-31T13:29:31.620Z [INFO] (AwsEventLoop 4) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Connection resumed. {clientId=IoTThingName#2, sessionPresent=true}
2023-05-31T13:32:11.294Z [INFO] (pool-2-thread-590) com.aws.greengrass.tes.CredentialRequestHandler: Received IAM credentials that will be cached until 2023-05-31T14:27:11Z. {iotCredentialsPath=/role-aliases/RoleAlias/credentials}

If I manually restart the device the deployment is completed on reboot - What is causing this error?

  • Guy P
    10 months ago

    HI Phil. We have the same problem. Did you figure out what the issue was? Thanks

Topics
Internet of Things (IoT)
Tags
AWS IoT GreengrassAWS IoT Core
Language
English
Phil Begg
asked a year ago644 views
1 Answer
0

Hello,

"A query to dns failed to resolve." means that your DNS resolver did not respond with the IP address(es) for AWS, so it was unable to connect to IoT Core. This would happen when you do not have internet, or your DNS provider is not working. Being able to log into the device using some other mechanism proves you have internet, but it does not necessarily prove that your DNS is working.

DNS was eventually able to resolve and get connected again to AWS.

The reason why MQTT initially disconnected was "Remote endpoint did not respond to a PINGREQ before timeout exceeded" which would typically mean that your internet connection did go out, at least briefly.

Cheers,

Michael

AWS
EXPERT
MichaelDombrowski-AWS
answered 9 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content