How do I troubleshoot and deep dive windows patching / updates installation failures on EC2 Windows Instances?

Firstly, Let us understand the key elements of the Windows Server update process:

1. Essential Windows Update Services: The following services manage the Windows update process:

i. Windows Update Service (wuauserv) - Manages Microsoft Update and WSUS server installations.

ii. Windows Installer Service (msiserver) - Handles software component installation, modification, and removal.

iii. Background Intelligent Transfer Service (BITS) - Manages asynchronous file transfers between client and server, optimizing download efficiency.

iv. Cryptographic Services (CryptSvc) - Verifies update authenticity and integrity.

v. Windows Modules Installer (TrustedInstaller) - Protects critical system files and manages update installation.

2. Critical Windows Update Locations:

System Directories:

• %SystemRoot%\SoftwareDistribution - Stores temporary update files and downloads
• %SystemRoot%\System32 - Contains system executables and libraries for updates
• %SystemRoot%\winsxs - Stores multiple versions of system components for compatibility

Registry Keys: The Registry stores update configurations, version data, and feature settings.

• HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate

• HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU

Please see - Configuring Automatic Updates by editing the registry

3. Windows Update Installation Process:

i. Detection - Windows Update service scans the system and identifies required updates from Microsoft Update or WSUS.

ii. Download - Windows Update service downloads files to %SystemRoot%\SoftwareDistribution.

iii. Verification - Cryptographic Services validates Update integrity and File authenticity using digital signatures.

iv. Pre-Installation - System validates prerequisites, dependencies, and compatibility.

v. Installation - Windows Installer modifies system files, registry settings, Configures components Reboot Management i.e. Handles required system restarts automatically or prompts for manual reboot.

vi. Post-Installation - System updates configurations, rebuilds caches, and cleans temporary files.

4. Windows Update Logs and Reports:

i. System Logs: Under %SystemRoot%\Logs: WindowsUpdate.log and CBS.log

ii. Event Viewer: Open Event Viewer >Navigate to "Windows Logs" > "System". In the "Source" column, look for "Microsoft-Windows-WindowsUpdateClient".

iii. WSUS Reports: For WSUS-managed environments, monitor: Track compliance, deployment, and system status.

Please see: Windows Update log files

5. Windows Update Release Types:

For detailed information about: Update categories, Release cycles and Deployment schedules, Please see: Update release cycle for Windows clients

Phase 1: Let's go ahead with troubleshooting Windows Updates:

Windows update process consists of three distinct phases, each requiring different troubleshooting approaches:

• Update Detection - Searches for applicable updates, Validates system requirements and Download.
• Retrieves update files - Verifies file integrity and Installation.
• Applies updates - Configures system changes.

Automated Troubleshooting :

1. Use AWS Systems Manager Automation to diagnose update issues: Primary Tool : AWSSupport-TroubleshootWindowsUpdate

• Identifies common update problems
• Provides specific recommendations
• Works on EC2 Windows instances

Documentation - AWS Systems Manager Automation Guide

Manual Troubleshooting :

2. Verify Update Source : Identify Update Provider whether its Microsoft Update (online), Windows Server Update Services (WSUS).

• This helps to isolate the issue whether its persisting at OS level (Windows updates components corruption) or connectivity issues with the updates server.
• You can try to install the updates online (check for updates) or install the stand alone package by downloading from Microsoft catalog.
• If manual installation is success, Means its connectivity issue with the Updates/WSUS server. Here, You need to verify the updates services status and if any internet connectivity issues.
• In case, Online update (check for updates) is downloading the KBs, However failing during installation stage, Then verify the Windows update logs and CBS logs to understand OS level corruptions.

Review Microsoft documentation below for logs and common errors and troubleshooting at this stage:

Windows Update log files

Fix Windows Update corruptions and installation failures

WSUS Client Agent Troubleshooting

WSUS Connection Issues

If its failing manually with stand alone package, Proceed with below.

3. Verify installed Updates to understand current status:

• View Installed Updates via PowerShell :

Get-Hotfix | Select HotfixId, InstalledOn | Sort InstalledOn -descending

• Match above results with the release history based on the OS version as mentioned here Windows Server release history

4. Reset Windows Update Components : Run these commands in an elevated PowerShell:

• Stop Services:

net stop wuauserv

net stop cryptSvc

net stop bits

net stop msiserver

• Rename Directories

ren C:\Windows\SoftwareDistribution SoftwareDistribution.old

ren C:\Windows\System32\catroot2 catroot2.old

• Restart Services

net start wuauserv

net start cryptSvc

net start bits

net start msiserver

Additional Resources : Reset Windows Update components manually

Note:

Whilst latest KBs does not create error while installing (in most cases), it would be the last 'Cumulative Security Update' or the 'Service Stack Update' which is running with corruptions will cause the future KB's not to get installed, You can uninstall the last CU and try to install latest CU. The Objective is to remediate this KB by refreshing the Windows Update folders and attempting a standalone DISM based installation of fresh KB.

5. Manual Update Installation using DISM:

i. Download Update : Get standalone KB file from Microsoft Update Catalog

ii. Save to C:\Windows\Temp.

iii. Extract and Install.

• Extract MSU file via PowerShell: expand -f:* "C:\Windows\Temp\KBFileName.MSU" "C:\Windows\Temp"

• Install using DISM: dism /online /add-package /packagepath:"C:\windows\temp\nameofthecabefile.cab"

Example Command - dism /online /add-package /packagepath:"Windows10.0-KB123450-x64.cab"

Please see - DISM Operating System Package Servicing Command-Line Options

Phase 2: Download Troubleshooting :

Common causes for download failures - Software Distribution issues, Network connectivity problems and Corrupted download cache.

Troubleshooting Steps:

i. Check Windows Update logs for specific errors.

ii. Reference error codes in below section: Common error codes, Reasons and Action Plan.

Note: Detailed analysis and solutions are provided in the following section.

Phase 3: Installation Troubleshooting:

CBS Log Analysis - The Component-Based Servicing (CBS) log provides installation details in this format.

| Message Portion | Description | | Time / Date Stamp | The time that the message was logged | | Message Severity | May be either Info, Warning, or Error. | | Message Source | Component that logged the message. For CBS.log, this is either CBS or CSI. | | Operation Type | Startup, Session, Appl, Exec, Plan, WatchList, etc | | Main Text | The main text of the message. |

“…, Error…” messages in the CBS.log can indicate errors during a servicing event.

Log Analysis Guide:

i. Locate CBS Logs - Check primary CBS.log (%SystemRoot%\Logs\CBS.log).

If file is small with no errors, check CBSpersist_*.log.

Note: CBS.log rotates at 50MB size

ii. Search for Errors : Search term: " , Error " (Include comma and spaces), Failed, Corrupt.

iii. Error Analysis : Locate error messages, Identify failure points and Document error codes.

iv. Root Cause Investigation -Scroll up from errors to find: Missing manifests, File corruption, Unresolved dependencies, OS integrity issues and Component failures.

Common error codes, Reasons and Action Plan:

Covering the error codes :

0x8024402F 0x80242006 0x80070BC9 0x80200053 0x80072EFD or 0x80072EFE or 0x80D02002 0X8007000D 0x8024A10A 0x80240020 0x80242014 0x80246017 0x8024000B 0x8024000E 0x8024D009 0x80244007 0x80070422 0x800f0821 0x800f0825 0x800F0920 0x800f081f 0x800f0831 0x80070005 0x80070570 0x80070003 0x80070020 0x80073701 0x8007371b 0x80072EFE 0x80072F8F 0x80072EE2 0x80240022 0x8024401B 0x80244022 0x80070490 0x800f0922 0x80070bc9 0x800706be

Please see Windows Update common errors and mitigation

In case this needs further deep dive, Do reach out to AWS Support.

Thank you!