SSM Agent update failure

Question

Hi all. We're just starting with SSM and hoping to use this quite extensively moving forward. I have a small handful of devices online using it but I just noticed that one of my devices has gone offline. Luckily it is not in the field yet so I was able to access it locally. What appears to have happened was SSM Agent was attempting to update but failed and never came back. I'm not so concerned that it didn't update and more concerned that it didn't start again. Going through the ssm agent logs I came across this line

```
"standardError": "E: Could not get lock /var/lib/dpkg/lock-frontend - open (11: Resource temporarily unavailable)
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?
WARNING: Could not install the python3-apt, this may cause the patching operation to fail.
failed to run commands: exit status 1
```

The SSM update log itself ends like this

```
2022-01-20 21:47:09 DEBUG UpdateInstanceInformation Response{

}
2022-01-20 21:47:09 INFO initiating cleanup of other versions in amazon-ssm-agent and amazon-ssm-agent-updater folder
2022-01-20 21:47:09 INFO removing artifacts in the folder: /var/lib/amazon/ssm/update/amazon-ssm-agent
2022-01-20 21:47:09 INFO removed files and folders: 3.1.821.0
2022-01-20 21:47:09 INFO removing artifacts in the folder: /var/lib/amazon/ssm/update/amazon-ssm-agent-updater
2022-01-20 21:47:09 INFO removed files and folders: 3.1.715.0
2022-01-20 21:47:09 INFO initiating cleanup of files in update download folder
2022-01-20 21:47:09 INFO Successfully downloaded manifest
Successfully downloaded updater version 3.1.821.0
Updating amazon-ssm-agent from 3.1.715.0 to 3.1.821.0
Successfully downloaded https://s3.us-east-2.amazonaws.com/amazon-ssm-us-east-2/amazon-ssm-agent/3.1.715.0/amazon-ssm-agent-ubuntu-amd64.tar.gz
Successfully downloaded https://s3.us-east-2.amazonaws.com/amazon-ssm-us-east-2/amazon-ssm-agent/3.1.821.0/amazon-ssm-agent-ubuntu-amd64.tar.gz
Initiating amazon-ssm-agent update to 3.1.821.0
failed to install amazon-ssm-agent 3.1.821.0, ErrorMessage=The execution of command returned Exit Status: 125
 exit status 125
Initiating rollback amazon-ssm-agent to 3.1.715.0
failed to uninstall amazon-ssm-agent 3.1.821.0, ErrorMessage=The execution of command returned Exit Status: 2
 exit status 2
Failed to update amazon-ssm-agent to 3.1.821.0
```
Then the error log gives me this

```
2022-01-20 16:29:54 ERROR [Submit @ processor.go.140] [ssm-agent-worker] [MessagingDeliveryService] [Association] [associationId=5752f0d0-1f57-492e-83f7-740484b81d73] Document Submission failed: Job with id 5752f0d0-1f57-492e-83f7-740484b81d73 already exists
2022-01-20 21:47:02 ERROR [AppendError @ context.go.129] failed to install amazon-ssm-agent 3.1.821.0, ErrorMessage=The execution of command returned Exit Status: 125
 exit status 125
```

Could anyone help me out here? I cannot have these fail like this when they go out into the wild.

The OS is Ubuntu Server 18 on this box, 20 on others we have.

Answer

Hello, 
 
I understand that you have been experiencing issues with one of your instances and that after its SSM agent attempted to update, the instance is no longer available. 
 
I noticed that in the below error message indicates that the package manager was already being used: 
Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend)
 
This would just prevent the agent from being updated, but it’s not clear why it caused the agent to fail. This is most likely an issue with the operating system and not the CloudWatch agent. 
 
To fix the issue, you may consider attempting to manually uninstall and then reinstall the agent as outlined in this documentation: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/download-cloudwatch-agent-commandline.html
 
If your problem continues to persist after implementing these steps, I strongly encourage you to contact an AWS support engineer who will be able to assist you.

SSM Agent update failure

Relevant content