How do I troubleshoot patching issues related to Amazon EC2 Windows instance when I use the Patch Manager AWS-RunPatchBaseline document?

Short description

You can encounter the following issues when you use the Patch Manager, a capability of AWS Systems Manager, AWS-RunPatchBaseline document:

• The node can't reach Amazon Simple Storage Service (Amazon S3) endpoints
• The token request isn't valid
• You can't search for Windows Updates
• Updates fail to install after download
• A patch is missing

Resolution

The node can't reach Amazon S3 endpoints

You might see the following example error messages when Amazon S3 can't connect to the remote server:

"Invoke-PatchBaselineOperation : The underlying connection was closed: An unexpected error occurred on a send."

-or-

"An error occurred when executing PatchBaselineOperations: Unable to connect to the remote server"

The preceding errors occur because the node can't connect to S3 endpoints. Or, the AWS Identity and Access Management (IAM) profile that's attached to the instance is missing the minimum required permissions for S3 connectivity.

To resolve these errors, complete the following steps:

1.    Check that the IAM permissions for the instance profile are attached to the node. For the minimum permissions, see AWS Systems Manager Agent (SSM Agent) communications with AWS managed S3 buckets.

2.    To check the connection to the Amazon S3 endpoint, run the following command in PowerShell:
Note: Replace <region_id> with your AWS Region ID.

tnc s3.<region_id>.amazonaws.com -port 443

3.    If the preceding test fails, then check how the instance is connected to Systems Manager endpoints:

• If the instance is in a public subnet, then make sure that the RouteTable that's attached to the instance uses an internet gateway to route to 0.0.0.0/0.
• If the instance is in a private subnet, then either use a NAT gateway to connect, create an S3 gateway VPC endpoint, or create an interface VPC endpoint.

The token request isn't valid

You might see the following example error messages because the token request isn't valid:

"Invoke-PatchBaselineOperation failed with: The security token included in the request is invalid:

-or-

"Invoke-PatchBaselineOperation: Instance Id i-00fe5918abcdef12 doesn't match the credentials"

The preceding errors occur when SSM Agent uses credentials that are different from the credentials that you expect it to use. For example, SSM Agent is using a shared credentials file instead of an instance profile.

Note: For SSM Agent version 3.1.1927.0 or earlier, SSM Agent looks for credentials in the order that's prescribed for the default credential provider chain. This also applies to Amazon Elastic Container Service (Amazon ECS) container instances.

Starting with SSM Agent version 3.2.183.0, SSM Agent stores a set of temporary credentials at /var/lib/amazon/ssm/credentials (Linux and macOS) or %PROGRAMFILES%\Amazon\SSM\credentials (Windows Server). The temporary credentials have the permissions that you specify for the IAM role that you selected. The IAM role has the permissions from the Default Host Management Configuration or the instance profile that's attached to your managed node.

To resolve these errors, update SSM Agent to version 3.2.183.0 or later so that it can fetch credentials from a file in the SSM folder.

To use an earlier SSM Agent version, find the shared credentials, delete them, and then test the patch again.

To locate the credentials, complete the following steps:

1.    Press the Windows key, and then enter environment variables.

2.    Find the variables named AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

3.    Delete the credentials, and then test the patch again.

Shared credential locations

The following are examples of shared credential locations:

SystemsProfile

• C:\Windows\System32\config\systemprofile\.aws\
• C:\Windows\System32\config\systemprofile\AppData\Local\AWSToolkit\RegisteredAccounts.json

User Profile

• %USERPROFILE%\.aws\
• %USERPROFILE%\AppData\Local\AWSToolkit\RegisteredAccounts.json

You can't search for Windows updates

You might see the following example error messages when you can't search for Windows updates:

"Invoke-PatchBaselineOperation : Exception Details: An error occurred when attempting to search Windows Update. Error Message: Exception from HRESULT: 0x80240437"

-or-

"Invoke-PatchBaselineOperation : Exception Details: An error occurred when attempting to search Windows Update. Error Message: Exception from HRESULT: 0x80072EE2"

The preceding errors are related to either the Windows update components, a lack of connectivity to the Windows Update Catalog, or Windows Server Update Services (WSUS).

To resolve these errors, complete the following steps:

1.    Check the connection to Windows Update. Determine whether the instance has a direct internet connection to the Microsoft patch source through a proxy or using WSUS.

2.    You can use a group policy or registry keys to configure WSUS servers. To check the WSUS configuration, run the following commands:

Check the group policy:

gpresult /H %USERPROFILE%\Desktop\report.html

Check the registry keys that might be manually set:

Get-ItemProperty -Path 'HKLM:\Software\Policies\Microsoft\Windows\WindowsUpdate\AU' | Select-Object AUOptions, NoAutoUpdate,  UseWUServer

Get-ItemProperty -Path 'HKLM:\Software\Policies\Microsoft\Windows\WindowsUpdate' | Select-Object DisableWindowsUpdateAccess,  WUServer, WUStatusServer

To check the proxy configurations, run the following commands:

Note: You must get the proxy configurations under the context of the system user account S-1-5-18 where you perform the patching operations. To get the configurations, use Run Command, a capability of AWS Systems Manager, with the AWS-RunPowerShellScript document.

Download PSExec on the Microsoft website, and run the following command to launch a new PowerShell prompt under the system user context:

psexec -i -s C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe

To see the AmazonSSMAgent registry settings, run the following command:

Get-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\AmazonSSMAgent'

To see the system environment variables (http_proxy, https_proxy, no_proxy), run the following commands:

Get-Item -Path Env:http_proxy  
Get-Item -Path Env:https_proxy  
Get-Item -Path Env:no_proxy

To see Internet Explorer settings (HTTP, secure, exceptions), run the following command:

Get-ItemProperty -Path 'HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings' | Select-Object ProxyServer,  ProxyEnable

To see WinINet proxy, run the following command:

[System.Net.WebRequest]::DefaultWebProxy

To see WinHTTP proxy settings (http=, https=, bypass-list=), run the following command:

netsh winhttp show proxy

If the proxy is set up only for SSM Agent, then Patch manager doesn't work. This is because the Windows update client uses system-wide proxy settings to scan and download the updates. To set the proxy system wide, run the following command:

netsh winhttp set proxy proxy-server="hostname:port" bypass-list="169.254.169.254"

If there's a connection, then reset the Windows Update components and retry the patching.

Updates fail to install after download

To resolve an update that fails to install after download, complete the following steps:

1.    Review the patch installation logs at C:\Windows\Logs\CBS, and match the timestamp of the command that's running. Then, review the logs for error messages.

2.    If the patch requires a reboot but fails to install after rebooting, then use an Amazon EC2 screenshot to see the message on the login screen.

Note: Patches might be rolled back with the following message on the boot screen: "We couldn't complete the updates. Undoing changes. Don't turn off your computer"

3.    Find the HResult code in the Component-Based Servicing (CBS) log file, and then search for the code in the Windows Update error documentation.

4.    Reset the Windows Update components, and then retry the installation.

A patch is missing

To troubleshoot missing patches, see How do I troubleshoot a missing KB patch after a successful patching operation on EC2 Windows instances through Patch Manager?

Related information

How patches are installed