Debugging the SecureTunneling Greengrass component exit 1


We are trying to deploy the secure tunneling component to our core devices. The component installs and subscirbes to right topic succesfully as can be seen from these logs in aws.greengrass.SecureTunneling.log:

{"thread":"Copier","level":"INFO","eventType":"stdout","message":"[INFO ] 2023-09-11 22:58:48.439 [main] SecureTunnelingExecutor - Starting secure tunneling.","contexts":{"scriptName":"","serviceName":"aws.greengrass.SecureTunneling","currentState":"RUNNING"},"loggerName":"aws.greengrass.SecureTunneling","timestamp":1694473128440,"cause":null}
{"thread":"Copier","level":"INFO","eventType":"stdout","message":"[INFO ] 2023-09-11 22:58:51.054 [AwsEventLoop 1] SecureTunnelingTask - Successfully subscribed to topic: $aws/things/<redacted_thing_name>/tunnels/notify","contexts":{"scriptName":"","serviceName":"aws.greengrass.SecureTunneling","currentState":"RUNNING"},"loggerName":"aws.greengrass.SecureTunneling","timestamp":1694473131061,"cause":null}

However once a new tunnel is opened is crashes immediatly with no debuging information other than the exit code:

{"thread":"Copier","level":"INFO","eventType":"stdout","message":"[INFO ] 2023-09-11 23:07:09.735 [AwsEventLoop 1] SubscribeResponseHandler - Received new tunnel notification message.","contexts":{"scriptName":"","serviceName":"aws.greengrass.SecureTunneling","currentState":"RUNNING"},"loggerName":"aws.greengrass.SecureTunneling","timestamp":1694473629736,"cause":null}
{"thread":"Copier","level":"INFO","eventType":"stdout","message":"[INFO ] 2023-09-11 23:07:09.860 [pool-3-thread-1] SubscribeResponseHandler - Secure tunnel process completed with exit code: 1","contexts":{"scriptName":"","serviceName":"aws.greengrass.SecureTunneling","currentState":"RUNNING"},"loggerName":"aws.greengrass.SecureTunneling","timestamp":1694473629864,"cause":null}

This had previously been working ~8 months ago. We have insured the correct policies are in place as listed here What are possible cuases for the tunnel process to exit 1? How can these be resolved or what are other methods to enable more verbose logging?.

Version info:

Component Name: aws.greengrass.Nucleus
    Version: 2.10.1
Component Name: aws.greengrass.SecureTunneling
    Version: 1.0.15
    State: RUNNING
    Configuration: {"accessControl":{"aws.greengrass.ipc.mqttproxy":{"aws.greengrass.SecureTunneling:mqttproxy:1":{"operations":["aws.greengrass#SubscribeToIoTCore"],"policyDescription":"Access to tunnel notification pubsub topic","resources":["$aws/things/+/tunnels/notify"]}}},"OS_DIST_INFO":"raspberrypi"}
asked 9 months ago251 views
1 Answer

What specific OS are you using? If possible, please try updating your OS to a more recent major version that implements a newer Linux kernel. There could be some backward incompatibilities with the libc version the component is built against vs. the version support by your OS.

answered 9 months ago
  • Thanks for the reply. We are on an older OS:

    Description:	Debian GNU/Linux 9.13 (stretch)
    Release:	9.13

    Running on armv7 with glibc version 2.28-10. Unfortunate updating the OS is not an option in the immediate future although it is on our road map. But as mentioned on the question this had been working previously which leads me to believe it is not a libc version issue. Is there a way to get more information on what the issue is? Looking at the documentation for that component I do not see a log level option.

  • Hey, the fact the component version is 1.0.15 leads me to believe that what you have currently may have been released recently (about 3-4 months ago). You might want to try out an older version especially if that was what has previously been installed on your device ~8 months prior. Try reinstalling from version 1.0.10 and work backwards from there. Also as a quick sanity check, try unzipping the jar file in /greengrass/v2/packages/artifacts or wherever the Greengrass artifact gets deployed to. From there, run ./aws-iot-device-client --help and see if you get an error.

    Unfortunately, our team is also in a similar situation in that we have a long-term fix planned to prevent these kinds of issues from happening again, but we are unable to work on it right now.

  • Also one more thing, the log level of the component cannot be changed as of right now. But even if you could it would be unlikely to give you any more information for this specific. This is also something we want to fix long-term since there are many other reasons to have configurable log levels

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions