How do I troubleshoot an Amazon EC2 instance that's stuck in a reboot loop after I installed Windows updates?

Resolution

Note: The following resolution requires you to stop and start your instance.

Configure your instance for a stop and start

Note: When you stop and start an instance, the instance's public IP address changes. It's a best practice to use an Elastic IP address to route external traffic to your instance instead of a public IP address. If you use Amazon Route 53, then you might need to update the Route 53 DNS records when the public IP address changes. A stop and start is different from an instance reboot. For more information, see How EC2 instance stop and start works.

Before you stop and start your instance, take the following actions:

• If your instance uses an instance store, then save the volume data on the instance store to persistent storage. For example, you can save the data to Amazon Elastic Block Store (Amazon EBS) volumes or Amazon Simple Storage Service (Amazon S3) buckets.
  Important: Amazon EC2 deletes instance store data when you stop the instance.
• Create a snapshot of your Amazon EBS volume. If you experience issues, then you can use the snapshot to restore your instance.
• Temporarily remove the instance from its Amazon EC2 Auto Scaling group so that you don't accidentally terminate the instance when you stop it.
  Note: EC2 Auto Scaling might terminate stopped instances in an Auto Scaling group based on your scale-in protection settings. Instances that you launch with Amazon EMR, AWS CloudFormation, or AWS Elastic Beanstalk might be in an Auto Scaling group.
• Set the instance shutdown behavior to Stop to make sure that the instance doesn't terminate when you stop it.

Create a rescue instance and clear the pending changes

Complete the following steps:

1. Open the Amazon EC2 console.
2. In the navigation pane, choose Instances, and then choose the instance that's in the reboot loop.
3. Stop the instance.
4. Detach the root EBS volume from the instance. Note: The device name for root volumes on Windows instances is /dev/sda1.
5. Launch a new rescue instance in same Availability Zone as the original instance. To avoid disk signature issues, use a different AMI for the rescue instance than the original instance's AMI. For example, if the original instance uses a Windows Server 2012 R2 AMI, then launch the rescue instance with a Windows Server 2016 AMI.
   Note: You can also use an existing instance that's in the same Availability Zone as the rescue instance.
6. Attach the root volume to the rescue instance as a secondary volume. Use the default secondary volume name that Amazon EC2 uses when you attach the volume to the instance.
7. Use Remote Desktop Protocol (RDP) to connect to the rescue instance.
8. Make the volume available to use. If you encounter issues, then see Why doesn't the Amazon EBS volume that's attached to my Amazon EC2 Windows instance appear in my OS or in Disk Management?
   Important: Note the drive letter of the secondary volume.
9. Open Command Prompt or Windows PowerShell as an Administrator, and then run the following command to clear pending changes in the operating system (OS):

   DISM /image:E:\ /cleanup-image /revertpendingactions

   Note: Replace E: with the secondary volume's drive letter.
10. Unmount and detach the secondary volume from the rescue instance.
11. Attach the volume back to original instance as a /dev/sda1 root volume.
12. Start the original instance.

Troubleshoot issues

If you cleared the pending updates, but still encounter issues, then take the following actions.

Manually remove the packages

Complete the following steps:

1. Detach the root volume from the original instance.

2. Attach the root volume to the rescue instance as a D: drive.

3. Choose the Start menu, and then open Disk Management.

4. Open the context (right-click) menu for the volume, and then select Online.

5. Run the following commands to fix any corrupted system files:

   sfc /SCANNOW /OFFBOOTDIR=D:\ /OFFWINDIR=D:\windows 

6. Run the following command to check the status of the installed updates:

   Dism /Image:D: /Get-packages 

7. In the preceding command's output, check for packages that have Uninstall and Install set to Pending. Run the following command to remove those packages:

   DISM /image:D:\ /Remove-Package /PackageName:PackageNameExample 

   Note: Replace PackageNameExample with the package name.

8. Run the following command to repair the Windows image:

   DISM /image:D:\ /Cleanup-Image /RestoreHealth 

9. Unmount and detach the secondary volume from the rescue instance.

10. Attach the volume back to original instance as a /dev/sda1 root volume.

11. Start the original instance.

Delete the Pending.xml file

Complete the following steps:

1. Detach the root volume from the original instance.
2. Attach the root volume to the rescue instance as a secondary volume.
3. Choose the Start menu, and then open Disk Management.
4. Open the context (right-click) menu for the volume, and then select Online. Note the volume's drive letter.
5. Choose the Start menu, and then open Registry Editor.
6. Choose HKEY_LOCAL_MACHINE, and then choose File.
7. Choose Load Hive....
8. Navigate to D:\Windows\System32\Config\SYSTEM, and then give the loaded hive file a unique name.
9. Navigate to the following registry keys, and then change the Start value to 4:
   Loaded Hive name\ControlSet001\Services\TrustedInstaller\Start
   Loaded Hive name\ControlSet002\Services\TrustedInstaller\Start
   Note: Replace Loaded Hive name with the loaded hive name. This configuration deactivates the registry keys.
   By default, you don't have permission to modify the Start value. If you don't have access, then open the context (right-click) menu for the TrustedInstaller registry key. Choose Permission. For Permissions, choose Full Control.
10. Open the context (right-click) menu for the registry hive file, and then choose Unload Hive.
11. Unmount and detach the secondary volume from the rescue instance.
12. Attach the volume back to original instance as a /dev/sda1 root volume.
13. Start the original instance.
14. Log in to the instance to verify that you can access the instance.
15. Navigate to C:\Windows\Winsxs\Pending.xml.
16. Save a copy of the Pending.xml file as a backup.
17. Open the context (right-click) menu for the Pending.xml file, and then select Properties.
18. Choose Security, and then choose Advanced.
19. Choose Ownership, and then choose Change.
20. Choose Select Administrator, and then choose Apply.
21. Delete the Pending.xml file to block the pending changes that caused the reboot loop.