greengrass v2 client discovery cached ca publisher pem and endpoint stops working within a minute-ish

Question

I am using the greengrass v2 python sdk, with:
```
from awscrt import io, mqtt
from awsiot import mqtt_connection_builder, greengrass_discovery
from awscrt import mqtt
.....
```
Given that i need my client device to publish to the core device even when off line, the only way I can get it working is if I cache to local storage the pem file and the local endpoint that the discovery code retrieves.  This works fine, I can continue to publish messages when I disable network connection (while it's running), so that's good.  I can even stop the publisher.py file and run it again (while still offline) and it will work, pulling the cached pem and endpoint.

However, my problem lies in that  if I stop/restart the publisher multiple times (it loads from disk cache) a few times in a row, it works and then all of a sudden after a few times I get this error:
```
2024-11-23 03:30:04,554 - INFO - Loaded Greengrass info from local files...
2024-11-23 03:30:04,554 - INFO - Attempting to connect to Greengrass Core at 172.28.0.3:8883 using cached credentials and endpoint...
2024-11-23 03:30:04,620 - ERROR - Failed to connect using cached credentials: AWS_IO_TLS_ERROR_NEGOTIATION_FAILURE: TLS (SSL) negotiation failed
```
Obviously if I enable network access, start it again, it will use discovery to get the latest and store in cache, then the whole scenario happens again.

I ran through tests using the open ssl cli and found that the pem was valid, and it was not expired, so I don't understand why this happens.  i wanted to be able to stop/restart my client while off line for an undermined amount of time if the network goes down, but this only works for what seems like a minute or something.

Any ideas/suggestions?

re:Post Agent · Answer

The issue you're experiencing with the Greengrass V2 client discovery and cached credentials stopping working after multiple restarts is likely related to the core device's certificate rotation or endpoint changes.

AWS IoT Greengrass core devices periodically rotate their certificates for security reasons. When this happens, the cached credentials on the client device become invalid, leading to the TLS negotiation failure you're seeing.

To address this issue, consider the following approaches:

1. Implement a retry mechanism with exponential backoff in your client code. When you encounter the TLS negotiation failure, attempt to reconnect using the cached credentials a few times. If it continues to fail, trigger a new discovery process to fetch updated credentials.

2. Store multiple sets of discovered endpoints and certificates. When one set fails, try the others before initiating a new discovery.

3. Implement a local certificate authority (CA) on your core device. This would allow you to issue client certificates that remain valid for longer periods, reducing the frequency of credential invalidation.

4. Increase the rotation period for the core device's certificates if possible. This can be done through the AWS IoT Greengrass configuration, but be aware that longer rotation periods may have security implications.

5. Implement a mechanism to periodically refresh the cached credentials even when the client is offline. This could involve running a background process that attempts to update the cached information at regular intervals.

6. Ensure that the core device's IP address is stable and included in the MQTT broker certificate's subject alternative name (SAN) field. You can use the IP detector component to keep this information up-to-date.

Remember that while these solutions can help mitigate the issue, they may not completely eliminate it, especially in prolonged offline scenarios. The balance between security (frequent credential rotation) and offline functionality is a common challenge in IoT deployments, and the best approach often depends on your specific use case and security requirements.  
**Sources**  
[Set up and troubleshoot AWS IoT Greengrass with client devices - AWS Prescriptive Guidance](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/set-up-and-troubleshoot-aws-iot-greengrass-with-client-devices.html)  
[Test client device communications - AWS IoT Greengrass](https://docs.aws.amazon.com/greengrass/v2/developerguide/test-client-device-communications.html)  
[Troubleshooting client devices - AWS IoT Greengrass](https://docs.aws.amazon.com/greengrass/v2/developerguide/troubleshooting-client-devices.html)  
[Troubleshooting AWS IoT Greengrass - AWS IoT Greengrass](https://docs.aws.amazon.com/greengrass/v1/developerguide/gg-troubleshooting.html)

Saranya · Answer

Hi,

By default, the core device will trust the client device certificates (cache) for one minute only. So, when the core device is offline, it relies on the cached certificates of the client devices for upto a minute as it won't be able to verify the client device identity with the AWS IoT cloud.

If you'd like to extend this trust duration and allow client devices to connect when the core device if offline, you should update the [client device auth component's configuration](https://docs.aws.amazon.com/greengrass/v2/developerguide/client-device-auth-component.html#client-device-auth-component-configuration) for `clientDeviceTrustDurationMinutes` under `security`.

greengrass v2 client discovery cached ca publisher pem and endpoint stops working within a minute-ish

Add your answer

Relevant content