How do I troubleshoot a missing patch after I used Patch Manager to run a patching operation on EC2 Linux instances?

Short description

After you run a Patch Manager operation, patches might be missing for the following reasons:

• You're using an outdated version of Systems Manager Agent (SSM Agent).
• You ran a Scan operation instead of an Install operation.
• Reboots are pending.
• Your Amazon EC2 Linux instance role doesn't have the necessary AWS Identity and Access Management (IAM) permissions.
• Patch Manager doesn't support the version of Python that you use.
• Your operating system (OS) released a patch after your last patching operation.
• The package isn't available in Patch Manager.
• Your patch baseline configuration is incorrect.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Use the latest version of SSM Agent

Check that you're using the latest version of SSM Agent. If the agent is outdated, then update it.

Check the Patch Manager operation type

If you set the Operation parameter to only Scan in your AWS-RunPatchBaseline SSM Command document, then your patch didn't install.

To check what option you set for the Operation parameter, run the following command to view the Patch Manager logs on your instance:

sudo cat /var/log/amazon/ssm/patch-configuration/patch-states-configuration.json

If the value of the Operation line is Scan, then you ran only a Scan operation. To apply missing patches, run Patch now, and choose Scan and install under Patching operation.

To retrieve a list of recent patch operations, run the following list-commands AWS CLI command:

aws ssm list-commands --filters "Key=DocumentName,Values=AWS-RunPatchBaseline"

In the Parameters section of the command output, check the value of the Operation key to confirm the type of operations that you ran.

Check for pending reboots

After you run Patch Manager with the NoReboot option, you must manually reboot instances that have an InstalledPendingReboot status.

To check your reboot configuration, run the following command to view your SSM Agent Patch Manager logs:

sudo cat /var/log/amazon/ssm/amazon-ssm-agent.log | grep "Reboot Option"

If the output shows Reboot Option: NoReboot, then run the following command to manually reboot the instance and complete the patch installation:

sudo reboot

When you reboot your instance, its compliance status automatically updates the next time that Patch Manager completes a scheduled scan. To immediately update the compliance status of your instance, run Patch now and choose the Scan operation.

To view the patch status of your instance, run the following describe-instance-patches AWS CLI command:

aws ssm describe-instance-patches --instance-id example-instance-id

Note: Replace example-instance-id with your instance ID.

Check your IAM permissions for baseline overrides

When you use Quick Setup to create a patch policy, AWS automatically creates an Amazon Simple Storage Service (Amazon S3) bucket. Systems Manager stores your policy's baseline_overrides.json configuration file in the Amazon S3 bucket.

To check whether your role has the necessary permissions, run the following command to view your SSM Agent logs:

sudo cat /var/log/amazon/ssm/errors.log | grep -i "access denied"

If the output contains "Access denied" errors for configuration file downloads, then your role doesn't have the required permissions.

To resolve this issue, see Issue: Invoke-PatchBaselineOperation : Access Denied" error or "Unable to download file from S3" error for baseline_overrides.json.

Check that Patch Manager supports your version of Python

If your instance uses an unsupported version of Python, then Patch Manager operations fail. For most versions of Linux, Patch Manager supports Python versions 2.6–3.12. For Ubuntu Server and Debian Server, Patch Manager requires Python 3.0–3.12. Patch Manager doesn't support Python 2.x on Ubuntu Server and Debian Server or Python 3.13 and later.

To check the version of Python on your instance, run the following command:

python3 --version

If Patch Manager doesn't support your version of Python, then run the following command to review your SSM logs for Python errors:

sudo cat /var/log/amazon/ssm/errors.log | grep -i "ModuleNotFoundError"

To manually patch your OS, run one of the following commands for the system that you use.

Red Hat Enterprise Linux (RHEL) systems:

sudo yum update

Debian systems:

sudo apt update

-or-

sudo apt upgrade

Important: When you use a package manager to apply patches, Patch Manager compliance reports include the patches only after the next Scan operation completes. After you manually patch, run Patch now and choose the Scan operation to update the compliance status.

To make sure that your instance doesn't use an unsupported Python version, use a utility, such as pyenv, to manage multiple Python versions inside user home directories. Don't install or update Python at the system level as the root user.

Check for issues with Ubuntu

If you use Ubuntu, then check for the following issues.

Patching fails on Ubuntu 24.04 LTS

Patch Manager supports Ubuntu Server 24.04 LTS. If patching fails on Ubuntu 24.04 LTS, then check your SSM Agent logs for errors. You can use the AWS-RunShellScript SSM document to run standard Ubuntu update commands. For instructions, see Walkthrough: Use the AWS CLI with Run Command.

You receive python3-apt package errors

If you receive a "No module named 'apt_pkg'" error message on an Ubuntu instance, then your python3-apt package is corrupted. To resolve this error, reinstall the python3-apt package.

Important: When you reinstall python3-apt, services or tools that depend on Python might experience disruptions. It's a best practice to create an Amazon Machine Image (AMI) of the instance where you receive the error message.

To reinstall the python3-apt package, run the following command:

sudo apt-get install --reinstall python3-apt

Check when an OS released a patch

If vulnerability scans show that an OS released missing patches after your last patching operation, then Patch Manager installs them during your next scheduled maintenance window.

To check when your OS released a missing patch, check the vendor's security bulletin. For RHEL systems, see Security advisories on the Red Hat Customer Portal website.

To compare patch release dates with your patching operation timestamps, run the following command to review your SSM Agent Patch Manager logs:

sudo cat /var/log/amazon/ssm/amazon-ssm-agent.log | grep "Operation type"

If your OS released a missing patch after your last patching operation, run Patch now and choose the Scan and install operation or wait for the next scheduled maintenance window.

Check that the package is available in Patch Manager

To check the packages that are available in Patch Manager, run the following command to view your configuration file:

sudo cat /var/log/amazon/ssm/patch-configuration/patch-states-configuration.json | grep -i example-package-name

Note: Replace example-package-name with the name of the package as it appears in your package manager, such as openssl or curl.

If the package doesn't appear in the output, then the package isn't available because the package manager of your OS didn't install it.

To update packages that aren't available in Patch Manager, run one of the following commands for the programming language that you use.

Note: In the following commands, replace example-package-name with the name of the package.

Python packages that you install from pip:

pip install --user --upgrade example-package-name

Important: Don't run the pip install command with sudo or as root. You might overwrite the version of Python that Patch Manager uses.

Node.js packages that you install from npm:

npm update example-package-name

Check your custom patch baseline configuration

If you use a custom patch baseline, then verify that you configured it to approve new patches. The default patch baseline includes an auto-approval delay after your OS releases a patch. Patches that your OS releases during your delay window don't install during patching operations.

To check the patch baseline that Patch Manager uses, run the following command to view your SSM Agent Patch Manager logs:

sudo cat /var/log/amazon/ssm/amazon-ssm-agent.log | grep "baselineId"

To install patches that your OS releases during a delay window, create a custom patch baseline with a shorter approval delay or specific approval rules.

Check your diagnostics and logs to further troubleshoot issues

If you continue to experience issues, then review your logs and diagnostics for other errors.

To view your Systems Manager logs, separately run the following two commands:

sudo tar -czvf /tmp/aws-ssm-logs.tar.gz /var/log/amazon/ssm /etc/amazon/ssm /var/lib/amazon/ssm/

sudo chmod a+r /tmp/aws-ssm-logs.tar.gz

To view your diagnostic information, run the following ssm-cli command:

ssm-cli get-diagnostics --output table

To view the full command output from the patch operation that failed, run the following command:

sudo cat /var/lib/amazon/ssm/example-instance-id/document/orchestration/example-command-id/awsrunShellScript/PatchLinux/stdout

Note: Replace example-instance-id and example-command-id with the actual values from your environment.

To check for available common vulnerabilities and exposures (CVE) updates through the package manager, run one of the following commands for the system that you use.

RHEL systems:

sudo yum updateinfo list cves

Debian systems:

sudo apt list --upgradable

To check the status of your patch compliance, run the following describe-instance-patch-states AWS CLI command:

aws ssm describe-instance-patch-states --instance-ids example-instance-id

Note: Replace example-instance-id with your EC2 instance ID.

In the output, check the MissingCount and FailedCount values. If either value is greater than 0, then your patches failed to install or are missing. To identify patches with pending reboots, check the InstalledPendingRebootCount field. To identify non-compliant patches and their severity, check the CriticalNonCompliantCount, SecurityNonCompliantCount, and OtherNonCompliantCount fields. For more information, see What is compliance in Patch Manager?

Related information

How patches are installed

Predefined and custom patch baselines

Patch Manager prerequisites