By using AWS re:Post, you agree to the Terms of Use
AWS re:Postre:Post

How do I troubleshoot the error "InvalidIdentityToken - OpenIDConnect provider's HTTPS certificate doesn't match configured thumbprint" when I'm using the Amazon EKS IAM role to access the service account?

5 minute read
0

The thumbprint for my Amazon Elastic Kubernetes Service (Amazon EKS) cluster has changed, causing the Application Load Balancer controller to fail updates. -or- My Amazon EKS pods are in failed state with the following error: "WebIdentityErr: failed to retrieve credentials\r\ncaused by: InvalidIdentityToken: OpenIDConnect provider's HTTPS certificate doesn't match configured thumbprint\r\n".

Resolution

All Amazon EKS service accounts use the OpenID Connect (OIDC) to authenticate. When you create an AWS Identity and Access Management (IAM) OIDC provider for your Amazon EKS cluster, the thumbprint generated uses the root certificate. The Amazon root Certificate Authority (CA) has a validation period of around 25 years. You get the "HTTPS certificate doesn't match configured thumbprint" error under either of the following conditions:

  • The thumbprint used in the OIDC provider is expired.
  • The thumbprint doesn't match the CA.

To troubleshoot this issue and obtain a thumbprint, you must install and configure the OpenSSL command line tool.

To obtain a thumbprint for the OIDC provider, do the following:

1.    Find the URL for the OIDC identity provider (IdP) by doing the following:

  1. Open the Amazon EKS console.
  2. In the navigation pane, Choose Clusters.
  3. Select the cluster that you want to check.
  4. Select the Configuration tab.
  5. Note the OICD provider URL under the Details section.
    Example: https://oidc.eks.us-east-1.amazonaws.com/id/1111222233334444555566667777888F/

Include /.well-known/openid-configuration at the end of the OICD provider URL to form the URL for the IdP's configuration document.

Example: https://oidc.eks.us-east-1.amazonaws.com/id/1111222233334444555566667777888F/.well-known/openid-configuration

Access this URL in a web browser and make a note of the value of jwks_uri from the output. The browser output looks similar to the following:

{"issuer":"https://oidc.eks.us-east-1.amazonaws.com/id/1111222233334444555566667777888F","jwks_uri":"https://oidc.eks.us-east-1.amazonaws.com/id/1111222233334444555566667777888F/keys","authorization_endpoint":"urn:kubernetes:programmatic_authorization","response_types_supported":["id_token"],"subject_types_supported":["public"],"claims_supported":["sub","iss"],"id_token_signing_alg_values_supported":["RS256"]}

2.    Use the OpenSSL command line tool to run the following command to display all the certificates used:

Note: Be sure to replace oidc.eks.us-east-2.amazonaws.com with the domain name returned in Step 1.

openssl s_client -connect oidc.eks.us-east-2.amazonaws.com:443 -showcerts

The output looks similar to the following:

[root@ip-172-31-1-202 ~]# openssl s_client -connect oidc.eks.us-east-2.amazonaws.com:443 -showcerts
CONNECTED(00000003)
depth=4 C = US, O = "Starfield Technologies, Inc.", OU = Starfield Class 2 Certification Authority
verify return:1
depth=3 C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2
verify return:1
depth=2 C = US, O = Amazon, CN = Amazon Root CA 1
verify return:1
depth=1 C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
verify return:1
depth=0 CN = *.execute-api.us-east-2.amazonaws.com
verify return:1
---
Certificate chain
 0 s:/CN=*.execute-api.us-east-2.amazonaws.com
   i:/C=US/O=Amazon/OU=Server CA 1B/CN=Amazon
-----BEGIN CERTIFICATE-----
CERTIFICATE Redacted
-----END CERTIFICATE-----
 1 s:/C=US/O=Amazon/OU=Server CA 1B/CN=Amazon
   i:/C=US/O=Amazon/CN=Amazon Root CA 1
-----BEGIN CERTIFICATE-----
CERTIFICATE Redacted
-----END CERTIFICATE-----
 2 s:/C=US/O=Amazon/CN=Amazon Root CA 1
   i:/C=US/ST=Arizona/L=Scottsdale/O=Starfield Technologies, Inc./CN=Starfield Services Root Certificate Authority - G2
-----BEGIN CERTIFICATE-----
CERTIFICATE Redacted
-----END CERTIFICATE-----
 3 s:/C=US/ST=Arizona/L=Scottsdale/O=Starfield Technologies, Inc./CN=Starfield Services Root Certificate Authority - G2
   i:/C=US/O=Starfield Technologies, Inc./OU=Starfield Class 2 Certification Authority
-----BEGIN CERTIFICATE-----
MIIEdTCCA12gAwIBAgIJAKcOSkw0grd/MA0GCSqGSIb3DQEBCwUAMGgxCzAJBgNV
BAYTAlVTMSUwIwYDVQQKExxTdGFyZmllbGQgVGVjaG5vbG9naWVzLCBJbmMuMTIw
MAYDVQQLEylTdGFyZmllbGQgQ2xhc3MgMiBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0
eTAeFw0wOTA5MDIwMDAwMDBaFw0zNDA2MjgxNzM5MTZaMIGYMQswCQYDVQQGEwJV
UzEQMA4GA1UECBMHQXJpem9uYTETMBEGA1UEBxMKU2NvdHRzZGFsZTElMCMGA1UE
ChMcU3RhcmZpZWxkIFRlY2hub2xvZ2llcywgSW5jLjE7MDkGA1UEAxMyU3RhcmZp
ZWxkIFNlcnZpY2VzIFJvb3QgQ2VydGlmaWNhdGUgQXV0aG9yaXR5IC0gRzIwggEi
MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDVDDrEKvlO4vW+GZdfjohTsR8/
y8+fIBNtKTrID30892t2OGPZNmCom15cAICyL1l/9of5JUOG52kbUpqQ4XHj2C0N
Tm/2yEnZtvMaVq4rtnQU68/7JuMauh2WLmo7WJSJR1b/JaCTcFOD2oR0FMNnngRo
Ot+OQFodSk7PQ5E751bWAHDLUu57fa4657wx+UX2wmDPE1kCK4DMNEffud6QZW0C
zyyRpqbn3oUYSXxmTqM6bam17jQuug0DuDPfR+uxa40l2ZvOgdFFRjKWcIfeAg5J
Q4W2bHO7ZOphQazJ1FTfhy/HIrImzJ9ZVGif/L4qL8RVHHVAYBeFAlU5i38FAgMB
AAGjgfAwge0wDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAYYwHQYDVR0O
BBYEFJxfAN+qAdcwKziIorhtSpzyEZGDMB8GA1UdIwQYMBaAFL9ft9HO3R+G9FtV
rNzXEMIOqYjnME8GCCsGAQUFBwEBBEMwQTAcBggrBgEFBQcwAYYQaHR0cDovL28u
c3MyLnVzLzAhBggrBgEFBQcwAoYVaHR0cDovL3guc3MyLnVzL3guY2VyMCYGA1Ud
HwQfMB0wG6AZoBeGFWh0dHA6Ly9zLnNzMi51cy9yLmNybDARBgNVHSAECjAIMAYG
BFUdIAAwDQYJKoZIhvcNAQELBQADggEBACMd44pXyn3pF3lM8R5V/cxTbj5HD9/G
VfKyBDbtgB9TxF00KGu+x1X8Z+rLP3+QsjPNG1gQggL4+C/1E2DUBc7xgQjB3ad1
l08YuW3e95ORCLp+QCztweq7dp4zBncdDQh/U90bZKuCJ/Fp1U1ervShw3WnWEQt
8jxwmKy6abaVd38PMV4s/KCHOkdp8Hlf9BRUpJVeEXgSYCfOn8J3/yNTd126/+pZ
59vPr5KW7ySaNRB6nJHGDn2Z9j8Z3/VyVOEVqQdZe4O/Ui5GjLIAZHYcSNPYeehu
VsyuLAOQ1xk4meTKCRlb/weWsKh/NEnfVqn3sF/tM+2MR7cEXAMPLE=
-----END CERTIFICATE-----
---
Server certificate
subject=/CN=*.execute-api.us-east-2.amazonaws.com
issuer=/C=US/O=Amazon/OU=Server CA 1B/CN=Amazon
---

If you see more than one certificate in the output, then look for the last certificate displayed at the end of the output. The last certificate is the root CA in the certificate authority chain.

3.    Create a certificate file (example: certificate.crt), and the copy the contents of the last certificate to this file. Then, run the following command:

openssl x509 -in certificate.crt -text

The output looks similar to the following:

[root@ip-172-31-1-202 ~]# openssl x509 -in certificate.crt -text
Certificate:    Data:
        Version: 3 (0x2)
        Serial Number:
            a7:0e:4a:4c:34:82:b7:7f
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=Starfield Technologies, Inc., OU=Starfield Class 2 Certification Authority
        Validity
            Not Before: Sep  2 00:00:00 2009 GMT
            Not After : Jun 28 17:39:16 2034 GMT

You can check the validity of the certificate from the values in the Not Before and Not After fields. From the output, you can see that the validity of Amazon CA is around 25 years.

4.    If the output indicates that the certificate is expired, then you must renew the certificate with your OIDC provider. After you renew the certificate, run the following command using the OpenSSL command line tool to get the latest thumbprint:

openssl x509 -in certificate.crt -fingerprint -noout

The output looks similar to the following:

SHA1 Fingerprint=9E:99:A4:8A:99:60:B1:49:26:BB:7F:3B:02:E2:2D:A2:B0:AB:72:80

Delete the colons (:) from this string to get the final thumbprint:

9E99A48A9960B14926BB7F3B02E22DA2B0AB7280

-or-

Run the following command to get the latest thumbprint after excluding the semicolons:

$ openssl x509 -in certificate.crt -fingerprint -noout | sed s/://g

5.    If the current thumbprint has expired, then use the latest thumbprint from step 4 to replace it. You can do so from the IAM console or using the AWS Command Line Interface (AWS CLI).

To replace the thumbprint using the console, do the following:

  1. Open the IAM console.
  2. In the navigation pane, choose Identity providers.
  3. Choose the identity provider that you want to update.
  4. In the Thumbprints section, choose Manage.
  5. Choose Add thumbprint, and enter the new value.
  6. Choose Save changes.

-or-

Run a command similar to the following using the AWS CLI:

aws iam update-open-id-connect-provider-thumbprint --open-id-connect-provider-arn arn:aws:iam::111122223333:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/5ECB2797CB1324A37FC79E3C46851CED --thumbprint-list 9E99A48A9960B14926BB7F3B02E22DA2B0AB7280

Note: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent version of the AWS CLI.

Related information

Obtaining the root CA thumbprint for an OpenID Connect Identity Provider

Creating OpenID Connect (OIDC) identity providers

Create an IAM OIDC provider for your cluster

Topics
Containers
Tags
Amazon Elastic Kubernetes Service
Language
English
AWS OFFICIAL
AWS OFFICIALUpdated 2 years ago
No comments

Relevant content