By using AWS re:Post, you agree to the Terms of Use

How to debug CLI crash when running Lightsail container commands?

0

I only use a couple of commands in my workflow but very often both will randomly throw a cryptic error in the middle of a deploy on CI:

  • aws lightsail get-container-service-deployments --region us-west-2 --service-name my_service --output json
  • aws lightsail get-container-images --region us-west-2 --output json --service-name my_service

My CI stack is Ubuntu Server 18.04 which runs a Github action-runner service.

Signal received: -1695824320, errno: 32575

Stack trace:

/usr/local/aws-cli/v2/2.3.0/dist/_awscrt.cpython-38-x86_64-linux-gnu.so(aws_backtrace_print+0x4d) [0x7f3f9483edbd]
/usr/local/aws-cli/v2/2.3.0/dist/_awscrt.cpython-38-x86_64-linux-gnu.so(+0x68513) [0x7f3f947b5513]
/lib/x86_64-linux-gnu/libc.so.6(+0x3f040) [0x7f3f9af46040]
/usr/local/aws-cli/v2/2.3.0/dist/libpython3.8.so.1.0(+0x1f9ed0) [0x7f3f9ab65ed0]
/usr/local/aws-cli/v2/2.3.0/dist/libpython3.8.so.1.0(+0xbb58b) [0x7f3f9aa2758b]
/usr/local/aws-cli/v2/2.3.0/dist/libpython3.8.so.1.0(+0x1fa930) [0x7f3f9ab66930]
/usr/local/aws-cli/v2/2.3.0/dist/libpython3.8.so.1.0(PyGC_Collect+0x81) [0x7f3f9ab67aa1]
/usr/local/aws-cli/v2/2.3.0/dist/libpython3.8.so.1.0(Py_FinalizeEx+0xe2) [0x7f3f9ab3ff02]
/usr/local/aws-cli/v2/2.3.0/dist/libpython3.8.so.1.0(Py_Exit+0x8) [0x7f3f9ab40818]
/usr/local/aws-cli/v2/2.3.0/dist/libpython3.8.so.1.0(+0x1d8c8b) [0x7f3f9ab44c8b]
aws(+0x378b) [0x55705f60f78b]
aws(+0x3b1f) [0x55705f60fb1f]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f3f9af28bf7]
aws(+0x24fa) [0x55705f60e4fa]

Any ideas what to do about this?

edit Jan 1, 2022

I turned on the --debug flag. There is no debug output. The command seems to immediately crash. I find if I run these command multiple times, they randomly crash with and sometimes it prints "Segmentation fault" which looks to likely to be a memory access/management issue :(

Something else worth saying is this issue only presents itself on Ubuntu distros. I ran out of options to try and reinstalled other OS like ArchLinux and did not experience the same problem. Unfortunately some of the other software I use isn not well supported under ArchLinux and I have to come back to Ubuntu :(

2 Answers
0

One of the first things you can do when the AWS CLI produces results that you don't expect, is get more detail by running the command again and including the --debug option at the end of the command line. This causes the AWS CLI to report details about every step it takes to process your command, send the request to the AWS servers, receive the response, and process the response into the output you see.

answered 9 months ago
  • I did try the --debug flag in my deploy automation script. The problem is I need to capture the JSON output and process it. Getting the cli to output its debug information means debug output will be mixed with the JSON I'm trying to parse; so the whole process won't work. I haven't worked out how to both turn on debug output and still be able to process the command output.

    Scratch that. --debug flag sends its output to stderr. I must have been confused by something else.

0

Hi! Along with what other users have suggested: using the debug option at the end of the command line commands can help.

As for help troubleshooting the lightsail deploy, the following additional API calls may help:

If there's enough progress, you can also look at the system status check or the instance status check for more information: https://lightsail.aws.amazon.com/ls/docs/en_us/articles/amazon-lightsail-viewing-instance-health-metrics#viewing-instance-metrics-console

You may also want to try bumping up the lightsail server to one with more memory, CPU, space, etc depending on what your error is and see if that helps.

answered 9 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions