nginx-ingress-controller can not start after EKS upgrade from 1.33 to 1.34

Question

Hi , 
In EKS 1.33,  the nginx-ingress-controller  deployment can deployment and run successful, but after I upgrade the EKS to 1.34, the nginx-ingress-controller can not start with below error,  could someone help to to check this issue? Thanks!
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.13.0
  Build:         4cbb78a9dc4f1888af802b70ddf980272e01268b
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.27.1

-------------------------------------------------------------------------------

W1009 13:57:57.354240       7 client_config.go:667] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1009 13:57:57.354522       7 main.go:205] "Creating API client" host="https://172.20.0.1:443"
I1009 13:57:57.354744       7 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I1009 13:57:57.354776       7 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I1009 13:57:57.354786       7 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I1009 13:57:57.354794       7 envvar.go:172] "Feature gate default state" feature="InOrderInformers" enabled=true
I1009 13:57:57.354802       7 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I1009 13:57:57.355182       7 main.go:225] "Trying to discover Kubernetes version"
E1009 13:57:57.365432       7 main.go:234] "Unexpected error discovering Kubernetes version" err="the server has asked for the client to provide credentials" attempt=0
E1009 13:57:58.434252       7 main.go:234] "Unexpected error discovering Kubernetes version" err="the server has asked for the client to provide credentials" attempt=1
E1009 13:58:00.044215       7 main.go:234] "Unexpected error discovering Kubernetes version" err="the server has asked for the client to provide credentials" attempt=2
E1009 13:58:02.507971       7 main.go:234] "Unexpected error discovering Kubernetes version" err="the server has asked for the client to provide credentials" attempt=3
E1009 13:58:06.084419       7 main.go:234] "Unexpected error discovering Kubernetes version" err="the server has asked for the client to provide credentials" attempt=4
E1009 13:58:11.189731       7 main.go:234] "Unexpected error discovering Kubernetes version" err="the server has asked for the client to provide credentials" attempt=5
E1009 13:58:18.966222       7 main.go:234] "Unexpected error discovering Kubernetes version" err="the server has asked for the client to provide credentials" attempt=6
E1009 13:58:30.817004       7 main.go:234] "Unexpected error discovering Kubernetes version" err="the server has asked for the client to provide credentials" attempt=7

Ling Zhang · Accepted Answer

Issue was resolved without any change, it works automatically.

re:Post Agent · Answer

The error you're experiencing with nginx-ingress-controller after upgrading from EKS 1.33 to 1.34 is related to authentication issues. The error message "the server has asked for the client to provide credentials" indicates that the controller can't authenticate with the Kubernetes API server.

This issue is likely related to service account authentication changes or IAM role configuration problems after the upgrade. When you upgrade an EKS cluster, sometimes the authentication mechanisms need to be updated as well.

Here are some steps to resolve this issue:

1. Check if the service account for your nginx-ingress-controller has the proper permissions and is correctly configured.

2. Verify that the IAM role associated with the service account has the necessary permissions and is properly attached.

3. Make sure the service account has automountServiceAccountToken set to true, which allows it to access the Kubernetes API.

4. If you're using IAM Roles for Service Accounts (IRSA), ensure that the proper annotations are present on the service account.

5. Check if there were any changes to RBAC policies or ClusterRoles that might affect the nginx-ingress-controller's ability to communicate with the API server.

This issue is similar to other credential-related problems that can occur after EKS upgrades, where components that previously worked fine suddenly lose their ability to authenticate with the cluster. Upgrading the nginx-ingress-controller to a version that's fully compatible with EKS 1.34 might also help resolve the issue.  
**Sources**  
[Failed build model due to NoCredentialProviders: no valid providers in chain. Deprecated. For verbose messaging see aws.Config.CredentialsChainVerboseErrors | AWS re:Post](https://repost.aws/questions/QUpBkLqWiUTL2mFBjzYcJ3mg/failed-build-model-due-to-nocredentialproviders-no-valid-providers-in-chain-deprecated-for-verbose-messaging-see-aws-config-credentialschainverboseerrors)

nginx-ingress-controller can not start after EKS upgrade from 1.33 to 1.34

Hi , In EKS 1.33, the nginx-ingress-controller deployment can deployment and run successful, but after I upgrade the EKS to 1.34, the nginx-ingress-controller can not start with below error, could someone help to to check this issue? Thanks!

Add your answer

Relevant content