Skip to content

ACM Amazon-issued certs: new trusted chain missing on some computers (Starfield Services Root ... G2)

0

Context: AWS Security Blog: ACM will no longer cross sign certificates with Starfield Class 2 starting August 2024 (27 JUN 2024)

In short, clients connecting to servers with Amazon-issued certificates must now trust the Amazon Root CA 1 or Starfield Services Root Certificate Authority - G2, i.e. they must have either in their trust store.


Our issue: our organization just got a renewed certificate from ACM, subject to the change announced above.

Shortly after, we started receiving reports that some of our clients are facing SSL/TLS issues. We don't have a clear environment where we could reproduce this, but some information from clients showed they do not have Starfield Services Root Certificate Authority - G2 in their Windows (10 & 11) trust store.

Browsers have no issues (they all use their own embedded trust stores nowadays), but things are breaking for many people running our Windows application, when it tries to connect to our AWS servers with the renewed cert.

Is there any reason why many Windows 10 & 11 computers would be missing this CA, but not other computers? According to the article, that G2 CA should have been in most OSs for a decade. Most Windows machines we looked at did have this CA already (from Windows Update?).

asked a year ago3.9K views
2 Answers
1

Hello.

I think it is usually updated by something called "Microsoft Trusted Root Certificate Program".
Therefore, the certificate is distributed by Windows Update.
https://learn.microsoft.com/en-us/security/trusted-root/release-notes

EXPERT
answered a year ago
EXPERT
reviewed a year ago
  • Hi Riku, Indeed! And all the certificates at play are listed on the Microsoft "included" list: https://ccadb.my.salesforce-sites.com/microsoft/IncludedCACertificateReportForMSFT With those deployed by Windows Update, it's surprising to hear about Windows machines missing it. Many of our affected users triggered Windows Update today and it fixed the issue for them.

    On the other hand, the "Amazon Root CA 1"/2/3/4 CAs are also in that Microsoft "included" list but I have yet to see a machine with them installed.

  • On the other hand, the "Amazon Root CA 1"/2/3/4 CAs are also in that Microsoft "included" list but I have yet to see a machine with them installed.

    I also verified that my PC does not contain any certificates. I'm guessing that "Starfield Services Root Certificate Authority - G2" is probably included, and because of this, it is determined that "Amazon Root CA 1" is unnecessary.
    I think the explanation in the following part of the document you are looking at is relevant. https://aws.amazon.com/jp/blogs/security/acm-will-no-longer-cross-sign-certificates-with-starfield-class-2-starting-august-2024/

    Starting August 2024, the last certificate in an AWS issued certificate chain will be one of Amazon Root CAs 1 to 4 where the trust anchor is Starfield Services G2. Currently, the last certificate in the chain that is returned by ACM is the cross-signed Starfield Services G2 root where the trust anchor could be Starfield Class 2, as shown in Figure 1 that follows. In other words, I think that "Amazon Root CA 1" is included under "Starfield Services Root Certificate Authority - G2".

0

(Not the best form to self-answer, but after digging we found out we simply didn't know how the Windows certificate trust store operates.)


Definition: root / CA certificates may be referred to as trusted roots and packaged in Certificate Trust List (CTLs) by Microsoft.

Windows downloads most⁽¹⁾ CAs at runtime on the fly / on-demand, when a compatible Windows library⁽²⁾ starts any TLS session: during the TLS handshake, if the certificate chain is not trusted based on the current local trusted roots, Windows downloads the matching root certificate from CTLs on Windows Update servers – ctldl.windowsupdate.com in this case. CTLs that are indeed from the Microsoft Trusted Root Program (updated lists).

_

It turns out the Windows application mentioned in the question does not use the Windows APIs⁽²⁾ that auto-downloads missing CAs such as the Starfield Services Root Certificate Authority - G2 in this case. Definitely an application bug.

Again, most browsers are unlikely to be affected as they all use their own trust stores now (including Edge), so their behaviour on Windows won't match desktop applications.

_

References: I have not been able to find official sources clearly detailing the whole mechanism, or which Windows libraries / APIs must be used to trigger it⁽²⁾. But clearly, the trust store is not periodically updated by the Windows Update client/feature. Clues:

  • The description of the Windows registry key HKEY_LOCAL_MACHINE\Software\Policies\Microsoft\SystemCertificates\AuthRoot → DisableRootAutoUpdate alludes to it: "when you are presented with a certificate issued by an untrusted root authority, your computer will [not] contact the Windows Update website to see if Microsoft has added the CA to its list of trusted authorities." This is the most official reference I've found describing it done on the fly.
  • learn.microsoft.com / Windows Server / Certificates and trust in Windows details the servers used and certificates origin. It also note that "Windows Server features automatic daily update functionality that includes downloads of latest CTLs." I cannot confirm if this feature actually exists on servers, but I'd strongly suggest it doesn't on Windows desktops.
  • Searching for the registry key yields many third-parties describing this behaviour, such as https://support.sophos.com/support/s/article/KBA-000008163

____________

  • ⁽¹⁾ I presume Windows installs and updates some key CAs allowing it to trust their own servers e.g. to download missing CAs and do updates.
  • ⁽²⁾ I didn't search for the exact library calls that must be used, I am not a C/++/# programmer. "winhttp" is allegedly okay, and someone mentioned "the Microsoft s_client crypto library" on Reddit. curl.exe also seems to do it.
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.