ALB load balancer controller start up errors

Question

When the ALB controller pods come up i see below errors. looks like target group binding was deleted abnormally and i am stuck with it. How to get ride of this?

```
{"level":"error","ts":"2024-02-05T21:44:29Z","msg":"Reconciler error","controller":"targetGroupBinding","controllerGroup":"elbv2.k8s.aws","controllerKind":"TargetGroupBinding","TargetGroupBinding":{"name":"ngd-dashboard-tgb","namespace":"ingress-nginx"},"namespace":"ingress-nginx","name":"ngd-dashboard-tgb","reconcileID":"5841bf6b-3d92-471c-bd22-2d9c917963b8","error":"WebIdentityErr: failed to retrieve credentials
caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity
	status code: 403, request id: 7258cf29-b0ac-46e4-a186-135d85996a26"}
```

Answer

The error indicates an issue with the trust relationship between the IAM role used by the AWS Load Balancer Controller pod and the OIDC provider for your EKS cluster. A few things to check:

Verify the IAM role trusts the correct OIDC provider for your EKS cluster and allows the sts:AssumeRoleWithWebIdentity action. It should reference the pod's service account ARN and sts.amazonaws.com audience.

Confirm the OIDC provider ID matches what was used when creating the EKS cluster. You can get this from the EKS console.

Check the AWS Load Balancer Controller deployment is using the correct service account name and namespace.

Make sure any mutating webhooks are not modifying the pod's service account details.

To troubleshoot further, you can describe the IAM role trust policy and also get the mutating webhook configurations:

aws iam get-role --role-name ROLE_NAME
kubectl get mutatingwebhookconfiguration

ALB load balancer controller start up errors

Relevant content