How to debug CLI crash when running Lightsail container commands?

Question

I only use a couple of commands in my workflow but very often both will randomly throw a cryptic error in the middle of a deploy on CI:

- `aws lightsail get-container-service-deployments --region us-west-2 --service-name my_service --output json`
- `aws lightsail get-container-images --region us-west-2 --output json --service-name my_service`

My CI stack is Ubuntu Server 18.04 which runs a Github action-runner service.

```
Signal received: -1695824320, errno: 32575

Stack trace:

/usr/local/aws-cli/v2/2.3.0/dist/_awscrt.cpython-38-x86_64-linux-gnu.so(aws_backtrace_print+0x4d) [0x7f3f9483edbd]
/usr/local/aws-cli/v2/2.3.0/dist/_awscrt.cpython-38-x86_64-linux-gnu.so(+0x68513) [0x7f3f947b5513]
/lib/x86_64-linux-gnu/libc.so.6(+0x3f040) [0x7f3f9af46040]
/usr/local/aws-cli/v2/2.3.0/dist/libpython3.8.so.1.0(+0x1f9ed0) [0x7f3f9ab65ed0]
/usr/local/aws-cli/v2/2.3.0/dist/libpython3.8.so.1.0(+0xbb58b) [0x7f3f9aa2758b]
/usr/local/aws-cli/v2/2.3.0/dist/libpython3.8.so.1.0(+0x1fa930) [0x7f3f9ab66930]
/usr/local/aws-cli/v2/2.3.0/dist/libpython3.8.so.1.0(PyGC_Collect+0x81) [0x7f3f9ab67aa1]
/usr/local/aws-cli/v2/2.3.0/dist/libpython3.8.so.1.0(Py_FinalizeEx+0xe2) [0x7f3f9ab3ff02]
/usr/local/aws-cli/v2/2.3.0/dist/libpython3.8.so.1.0(Py_Exit+0x8) [0x7f3f9ab40818]
/usr/local/aws-cli/v2/2.3.0/dist/libpython3.8.so.1.0(+0x1d8c8b) [0x7f3f9ab44c8b]
aws(+0x378b) [0x55705f60f78b]
aws(+0x3b1f) [0x55705f60fb1f]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f3f9af28bf7]
aws(+0x24fa) [0x55705f60e4fa]
```

Any ideas what to do about this?

**edit Jan 1, 2022**

I turned on the `--debug` flag. **There is no debug output**. The command seems to immediately crash. I find if I run these command multiple times, they randomly crash with and sometimes it prints "Segmentation fault" which looks to likely to be a memory access/management issue :(

Something else worth saying is this issue only presents itself on Ubuntu distros. I ran out of options to try and reinstalled other OS like ArchLinux and did not experience the same problem. Unfortunately some of the other software I use isn not well supported under ArchLinux and I have to come back to Ubuntu :(

Answer

Hi!  Along with what other users have suggested: using the debug option at the end of the command line commands can help.

As for help troubleshooting the lightsail deploy, the following additional API calls may help:
*  lightsail get-container-log: https://docs.aws.amazon.com/cli/latest/reference/lightsail/get-container-log.html
* lightsail get-cloud-formation-stack-records (I don't think this will help as your error most likely isn't with the CFT, but with the container): https://docs.aws.amazon.com/cli/latest/reference/lightsail/get-cloud-formation-stack-records.html

If there's enough progress, you can also look at the system status check or the instance status check for more information: https://lightsail.aws.amazon.com/ls/docs/en_us/articles/amazon-lightsail-viewing-instance-health-metrics#viewing-instance-metrics-console

You may also want to try bumping up the lightsail server to one with more memory, CPU, space, etc depending on what your error is and see if that helps.

Answer

One of the first things you can do when the AWS CLI produces results that you don't expect, is get more detail by running the command again and including the --debug option at the end of the command line. This causes the AWS CLI to report details about every step it takes to process your command, send the request to the AWS servers, receive the response, and process the response into the output you see.

How to debug CLI crash when running Lightsail container commands?

Relevant content