SMS Patching Fails for ALL Windows Server 2019 EC2 Instances

0

I just starting using SMS to manage Windows 2019 Server EC2 instance patching (security updates). I noticed that by default, AWS prevents Windows OS to automatically run Windows Update. I followed the instructions for SMS Quick Setup and the Patching of my servers are failing with the following error message: (I have been searching ALL day for a resolution to this. Modifying registry settings, running DSIM commands, etc. Nothing helps. Seems like some type of certificate issue but I can't resolve it). Has anyone else had issues with getting SMS to patch AWS Windows Server 2019 EC2 instances?

Invoke-PatchBaselineOperation : Exception Details: An error occurred when attempting to search Windows Update. Exception Level 1: Error Message: A certificate chain processed, but terminated in a root certificate which is not trusted by the trust provider. (Exception from HRESULT: 0x800B0109) Stack Trace: at WUApiLib.IUpdateSearcher.Search(String criteria) at Amazon.Patch.Baseline.Operations.PatchNow.Implementations.WindowsUpdateAgent.SearchForUpdates(String searchCriteria) At C:\ProgramData\Amazon\SSM\InstanceData\i-03638bdca902ef8fd\document\orchestration\86ed2eda-065a-49d3-b084-69bfc89c14 3d\PatchWindows_script.ps1:233 char:13

  • $response = Invoke-PatchBaselineOperation -Operation Scan -SnapshotId ...
  •         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    
    • CategoryInfo : OperationStopped: (Amazon.Patch.Ba...UpdateOperation:FindWindowsUpdateOperation) [Invoke -PatchBaselineOperation], Exception

    • FullyQualifiedErrorId : Exception Level 1: Error Message: Exception Details: An error occurred when attempting to search Windows Update. Exception Level 1: Error Message: A certificate chain processed, but terminated in a root certificate which is not trusted by the t rust provider. (Exception from HRESULT: 0x800B0109) Stack Trace: at WUApiLib.IUpdateSearcher.Search(String criteria) at Amazon.Patch.Baseline.Operations.PatchNow.Implementations.WindowsUpdateAgent.SearchForUpdates(String searc hCriteria)

      Stack Trace: at Amazon.Patch.Baseline.Operations.PatchNow.Implementations.WindowsUpdateAgent.SearchForUpdates( String searchCriteria) at Amazon.Patch.Baseline.Operations.PatchNow.Implementations.WindowsUpdateOperation.SearchAndProcessResult(Lis t1 kbGuids) at Amazon.Patch.Baseline.Operations.PatchNow.Implementations.WindowsUpdateOperation.SearchByGuidsPaginated(Lis t1 kbGuids, Int32 maxPageSize) at Amazon.Patch.Baseline.Operations.PatchNow.Implementations.WindowsUpdateOperation.FilterWindowsUpdateSearch( List`1 filteringMethods) at Amazon.Patch.Baseline.Operations.PatchNow.Implementations.FindWindowsUpdateOperation.DoWindowsUpdateOperati on() at Amazon.Patch.Baseline.Operations.PatchNow.Implementations.WindowsUpdateOperation.DoBeginProcessing() ,Amazon.Patch.Baseline.Operations.PowerShellCmdlets.InvokePatchBaselineOperation

failed to run commands: exit status 4294967295

3 Answers
0
Accepted Answer

Hello! SSM uses the Windows Update APIs to install updates. This means that any issues reported by Windows Update are sent back to SSM. Manually using Windows Update would produce the same result, as you are currently experiencing.

The way Windows Update works, it downloads updates in the .CAB format. These updates are signed with a certificate. On a very high level, the certificate is evaluated to make sure it's not expired and that it has been emitted by a CA (certification authority).

This is part of the certificate chaining process. Under normal conditions, the CA will be from Microsoft. The Windows OS must trust the CA used, so the root CA certificate should be present under the Trusted Root and Trusted Publishers certificate store, which you can see when running mmc (Microsoft Management Console) and adding the Certificate snap in tool.

A common issue for the 0x800B0109 message happens when you use WSUS. WSUS is a service that can be installed on Windows Servers to allow downloading and installing updates. It's useful if you have multiple machines and you want to save on some internet bandwidth, or if you want granularity when controlling the installation of patches. WSUS needs a self signed certificate tied to the WSUS server, and this certificate needs to be deployed via Group Policy. Therefore, here are some questions that might help pinpoint the issue:

  1. Are the instances joined to a domain?

  2. Are you using WSUS or any mechanism that would change the source of the Windows Update download location? If so, make sure that your WSUS certificate is not expired and that you have a group policy in place that installs this certificate on every machine that wants to install updates.

  3. Do you have any group policies in place to configure Windows Update? You can see this by opening the Local Group Policy Editor (if not joined to a domain) or by going to the Group Policy management option on a machine that is joined to the domain and has this tool installed. The group policy is located under Computer Configuration > Administrative Templates > Windows components > Windows Update. You can also run the command "rsop.msc" in any of the affected instances to view all group policies applied to the instance.

  4. Do the instances have direct access to the internet, via an Internet Gateway or NAT Gateway? Ensure that the security group and Network ACL used by the instances have outbound rules to allow the traffic.

If you are not using WSUS or the instances are not joined to a domain, this suggests that the OS doesn't trust the Microsoft certificate, which is less common. You might want to run Get-WindowsUpdateLog as explained in the Microsoft documentation https://docs.microsoft.com/en-us/powershell/module/windowsupdate/get-windowsupdatelog?view=windowsserver2022-ps to troubleshoot. There are multiple tutorials that explain how to reset all Windows Update components, which you might have tried already. Due to the AWS Shared Responsibility Model https://aws.amazon.com/compliance/shared-responsibility-model/ this is a Windows OS issue and AWS has no visibility inside the running OS for your own security and privacy. However, I trust that the above information leads you into the right direction. Let me know if you have any questions. Have a great day!

AWS
SUPPORT ENGINEER
answered 2 years ago
  • Francisco! Thank you for this detailed explanation. Here is my response:

    1. All these servers were provisioned from an AWS Windows 2019 DC AMI. Nothing was changed regarding certificates or installation of 3rd party software. In fact, AWS disables any windows automatic updates by default.
    2. These servers are NOT part of a Windows AD domain structure
    3. I am not using a WSUS server.
    4. All these servers have internet access via a web proxy that our central IT department manages for us.
    5. Out of 6 AWS EC2 instances, only 2 are able to run the Windows Update Service. I also have that working using AWS SMS. I compared the Certs from the working servers to the non-working servers. All Microsoft Trusted Certs, Intermediate, etc match. I'm baffled.
    6. I will check the Group Policies next.

    Thanks again for the insigths Francisco

0

Hello Kevin, here are some additional things you might try:

  1. Check that all instances have configured the Proxy you are using. Particularly, check the instances that are working and check their proxy configuration, as well as the Windows route table with the route -n command. Some proxies have their own certificate, particularly to access HTTPS content. If the proxy has its own certificate, ensure that the instances trust it. You can also check the proxy logs. You can confirm the proxy configuration at the instance level with the netsh winhttp show proxy command.
  2. Consider disabling the proxy and have the instance connect to the internet directly for testing purposes (with an Internet Gateway if launched on a public subnet, or via a NAT Gateway if launched on a private subnet).
  3. Ensure that they have the same networking configuration at the AWS level (security group, network ACL, route table). If the instances are on different subnets or even VPCs/Regions confirm that the instances are able to the Proxy.
  4. After running the Get-WindowsUpdateLog PowerShell command, find out any individual patch shown in the logs and attempt to download and install manually from the Microsoft Update Catalog https://www.catalog.update.microsoft.com/

Regards,

AWS
SUPPORT ENGINEER
answered 2 years ago
  • Will do. Great recommendations. I will try that. I will also try to spin up a new Windows 2019 EC2 instance and see if that also has any issues.

  • So, been spending way too much time on this issue. So I decided to launch a brand new Windows 2019 instance from the EC2 Console. Once it was running I connected to it and immediately tried to execute the Windows Update again. Failed. I understand your comments about the proxy. The same proxy is being used for all the EC2 instances on my private VPC. 2 of them work, the other 4 (and including the new instance) do not. I will try to contact my internal proxy support team to see if they can help. The next test I will do is to launch a new Windows Server in a Public VPC and see if the windows update runs there. If so, then I would tend to think this has something to do with the proxy server.

  • It was a misconfigured proxy server!!! Windows update is working again.

  • Kevin, I'm glad you were able to solve this! Since some machines were working with the proxy, it would be useful to have an idea what type of proxy issue was causing this, as this would help other customers. Have a great day!

0

Questions that might help find an answer:

  1. what version of SSM agent is running?
  2. Is it based on an AWS AMI?
  3. Is AWS CLI loaded?
  4. Do manual windows updates work?
  5. Have you run patch baseline with success?

With so many variable it can be hard to troubleshoot but looks like either windows cert or aws cert is missing.
For AWS cert latest agent and CLI should resolve.
For Windows cert this would cause manual update to fail but windows guide should show how to resolve that.

profile picture
answered 2 years ago
  • Follow up:

    1. I am running latest version of SSM agent
    2. Yes, they are all Windows Server 2019 AWS AMI
    3. Is the AWS CLI required to be installed on ALL the EC2 instances?
    4. Manual windows updates do not work.
    5. Nom patch baseline fails too I am leaning toward a certificate chain issue with Microsoft. I am getting loads of CAPI2 error messages. I did find one server in my group that is working properly, so I checked all the Certificates there and exported them. Then I imported the Certs into the server that will not run Window update. No luck. Strange thing is that the servers have not been modified since AWS launched them 3 months ago. This is a tough one! Thanks for the reply Robin

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions