AWS IoT Secure Tunnel Connection Drops

Question

We are using the AWS Secure Tunneling component to remotely access our devices, which is great when it works.

However, the tunnel frequently disconnects, which is very disruptive to our work. As far as we can tell, there is no interruption in the internet connection, as we have MQTT messages being sent back frequently from the devices, which aren't interrupted.

We are accessing via the 'Create a secure tunnel' option in the AWS IoT dashboard on the browser.

When these disconnections happen, the 'Generate new token' option does not solve it, nor does closing/deleting and starting a new tunnel - we often must resort to uninstalling and re-installing the Secure Tunneling component to the deployment.

My questions are:
 - Is there any known reason that would cause a disconnection (other than internet connection loss)? 
 - Is there a way to debug such an event ?

Answer

Hello,

You mentioned that the tunnel frequently disconnects. Do you have any data around the frequency? 
The secure tunnel sessions have a default 1 hour timeout. This can be extended up to 12 hours by changing the timeout setting. Disconnections may simply be due to sessions expiring.

Something else to check is network interruptions - Any brief internet interruption could cause the tunnel to drop, even if other protocols like MQTT recover quickly. Tracing routes and monitoring for packet loss could help identify this.

Is there a way to debug such an event? - Yes, here are some troubleshooting steps:

* Check CloudWatch Logs for tunnel connection errors or logs from the IoT Device SDK/Secure Tunneling module
* Use network tracing utilities like Wireshark to inspect tunnel traffic and identify disruptions
* Try keeping tunnel timeout high (12 hrs) to see if disconnects map to timeouts
* Monitor endpoint resource usage for any constraints related to tunnel upkeep
* Create an allow list of AWS IoT IPs and domains in local firewall policies

CloudWatch metrics around tunnel state changes and packet loss could also help narrow down the root cause. But the timeout theory seems plausible if reconnects resolve it.

Answer

I think in many cases the timeout connection is set quite long, so it's not the timeout.

Monitor endpoint resource usage for any constraints related to tunnel upkeep
 - What do you mean by endpoint, which endpoint is this ? Are you referring to the destination device the secure tunnel is connecting to, that theres some memory/compute issue keeping the proxy up ? or to connections to AWS cloud endpoints ?

Answer

Hello, to find out whether or not the connection issue is related to the Secure Tunneling service, either paste your tunnel ID here or if you want to keep it private, reach out to customer support and have them forward the info to AWS > IoT > Tunneling. If there is an issue with the service, we can help extract the server-side logs for you. If that does not help get a handle on the issue, then chances are that the issue is stemming from the device component.

FYI a new version of the component is in the works. We will continue trying to iron out the bugs in the current version, but there may be architectural limitations that cloud require a rewrite of significant portions of the code, so we are diverting attention to that instead. Please keep a lookout for more details in a few months time.

AWS IoT Secure Tunnel Connection Drops

Relevant content