How do I troubleshoot a failed Patch Manager (Linux) operation?

Short description

To troubleshoot your failed Patch Manager operation, look for related error messages in the AWS Management Console or the API's response. Then, run the AWSSupport-TroubleshootPatchManagerLinux runbook, or manually troubleshoot the error message that you see.

For more information, see Troubleshooting Patch Manager.

Resolution

Identify related error messages

The following message is an example of an error message that you find in the console:

"failed to run commands: exit status 1"

The console truncates its output to 24,000 characters. To review the full output that's stored on the managed node, go to the following location:

/var/lib/amazon/ssm/example-instance-id/document/orchestration/example-command-id/awsrunShellScript/PatchLinux/stdout

Note: Replace example-instance-id with your Systems Manager instance ID and example-command-id with your command ID.

If the previous location doesn't contain the output of Run Command, a capability of AWS Systems Manager, then look for the output in Amazon Simple Storage Service (Amazon S3) or Amazon CloudWatch. Users can configure operations to send Run Command outputs to Amazon S3 or CloudWatch.

Use the AWSSupport-TroubleshootPatchManagerLinux runbook

Use the AWSSupport-TroubleshootPatchManagerLinux runbook to troubleshoot patching failures for Linux-based managed nodes. For more information, see AWSSupport-TroubleshootPatchManagerLinux.

Manually troubleshoot Patch Manager errors

Complete the following troubleshooting actions based on your error message to resolve your Patch Manager issue.

"Cannot acquire lock. Another patching operation is in progress"

You receive the following error when multiple patching operations run on the same managed node at the same time:

"[ERROR]: Cannot acquire lock on /var/log/amazon/ssm/patch-baseline-concurrent.lock. Another patching operation is in progress."

Important: It's a best practice to initiate manual Patch now operations only when there isn't a scheduled patching operation in progress.

To make sure that you didn't schedule multiple patching operations to run at the same time on the same managed node, complete the following checks:

• Check the patch policy configurations for Quick Setup, a capability of AWS Systems Manager, and verify that they don't overlap with other patching schedules.
• Review your maintenance window associations and verify that only one window targets each managed node with patching tasks at a time.
• Be sure not to initiate manual Patch now operations when a scheduled patching is in progress.

"Permission denied"

If you mount /var/lib/amazon with noexec permissions, then you receive the following error message:

"/var/lib/amazon/ssm/example-instance-id/document/orchestration/example-command-id/PatchLinux/_script.sh: Permission denied failed to run commands: exit status 126"

To resolve this issue, configure exclusive partitions to /var/log/amazon and /var/lib/amazon, and mount them with exec permissions.

"Unable to download payload"

If the managed node doesn't have the necessary permissions to access the specified S3 bucket, then you receive the following error message:

"Unable to download payload: https://s3.doc-example-bucket.region.amazonaws.com/aws-ssm-region/patchbaselineoperations/linux/payloads/patch-baseline-operations-1.23.tar.gz.failed to run commands: exit status 156"

To resolve this issue, update your network configuration and make sure that you can reach a Regional endpoint. For more information, see AWS Systems Manager Agent (SSM Agent) communications with AWS managed S3 buckets.

"No such file or directory"

If there isn't any available disk space on the /var directory, then you receive the following error message:

"IOError: [Errno 2] No such file or directory: 'patch-baseline-operations-1.23.tar.gz' Unable to extract tar file: /var/log/amazon/ssm/patch-baseline-operations/patch-baseline-operations-1.75.tar.gz. failed to run commands: exit status 155 Unable to load and extract the content of payload, abort. failed to run commands: exit status 152"

To resolve this issue, make more disk space available under the /var directory.

"Another process has acquired yum lock"

If AWS-RunPatchBaseline runs on a managed node that already runs yum and another process locked the database, then you receive the following error message:

"11/22/3333 44:55:66 root [INFO]: another process has acquired yum lock, waiting 2 s and retry."

To resolve this issue, complete the following checks:

• Make sure that no State Manager association, maintenance window tasks, or other configurations that run AWS-RunPatchBaseline on a schedule target the same managed node simultaneously.
• Make sure that no manual yum operations run at the same time.

"An unsupported package manager and python version combination was found"

For Red Hat Enterprise Linux (RHEL), Debian Server, Raspberry Pi, or Ubuntu Server instances, make sure that you install a supported version of Python 3. If you don't install a supported version, then you receive the following error message:

"An unsupported package manager and python version combination was found. Dnf requires Python 2 or Python 3 to be installed."

To resolve this issue, install Python 3 versions 3.0 through 3.9 on the necessary server.

"UnsupportedOperatingSystem"

If your operating system (OS) doesn't support Patch Manager, then you receive the following error message:

"An error occurred (UnsupportedOperatingSystem) when calling the GetDeployablePatchSnapshotForInstance operation: patch_common.exceptions. PatchManagerError: ('Unsupported Operating System', 146)"

To resolve this issue, use an OS that supports Patch Manager.