Gateway API crds installation and AWS gateway API controller error

Question

I followed AWS VPC lattice gateway API controller docs and installed gateway API CRDs (experimental-install.yaml), post that I installed AWS gateway API controller. But the controller pods has `ImagePullBackOff` error.

```
  Normal   Scheduled  3m39s                 default-scheduler  Successfully assigned aws-application-networking-system/gateway-api-controller-64cc58464b-45q58 to ip-10-0-18-135.ap-south-1.compute.internal
  Normal   Pulling    3m37s                 kubelet            Pulling image "public.ecr.aws/aws-application-networking-k8s/aws-gateway-controller:v2.0.1"
  Normal   Pulled     3m31s                 kubelet            Successfully pulled image "public.ecr.aws/aws-application-networking-k8s/aws-gateway-controller:v2.0.1" in 6.011s (6.011s including waiting). Image size: 26034532 bytes.
  Normal   Created    3m31s                 kubelet            Created container: manager
  Normal   Started    3m30s                 kubelet            Started container manager
  Normal   Pulling    35s (x5 over 3m38s)   kubelet            Pulling image "gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0"
  Warning  Failed     35s (x5 over 3m37s)   kubelet            Failed to pull image "gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0": rpc error: code = NotFound desc = failed to pull and unpack image "gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0": failed to resolve reference "gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0": gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0: not found
  Warning  Failed     35s (x5 over 3m37s)   kubelet            Error: ErrImagePull
  Normal   BackOff    21s (x13 over 3m30s)  kubelet            Back-off pulling image "gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0"
  Warning  Failed     21s (x13 over 3m30s)  kubelet            Error: ImagePullBackOff
```
The above logs clearly states that image `gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0` not present in `gcr.io`. But nowhere in official docs or issue has been reported such a incident. Am I missing something here ?

Similarly the CRDs(experimental-install.yaml) installation shows the following error. My question regarding this is whether cert-manager needs to be installed before deploying CRDs ?

```
Warning: tls: failed to find any PEM data in certificate input
secret/webhook-cert created
service/gateway-api-controller-metrics-service created
service/webhook-service created
deployment.apps/gateway-api-controller created
mutatingwebhookconfiguration.admissionregistration.k8s.io/aws-appnet-gwc-mutating-webhook created
The CustomResourceDefinition "tlsroutes.gateway.networking.k8s.io" is invalid: status.storedVersions[0]: Invalid value: "v1": missing from spec.versions; v1 was previously a storage version, and must remain in spec.versions until a storage migration ensures no data remains persisted in v1 and removes v1 from status.storedVersions
```

re:Post Agent · Answer

Based on the errors you're encountering, there are two distinct issues:

**ImagePullBackOff Error:**
The `ImagePullBackOff` error occurs when a container fails to pull the required image from a container registry. In your case, the image `gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0` cannot be found. This could be due to several reasons:

1. **Network connectivity issues** - Your cluster nodes may not have proper internet access to reach gcr.io
2. **Incorrect image name or tag** - The image may have been moved, deprecated, or the tag may no longer exist at that location
3. **Missing credentials** - If the registry requires authentication, your cluster may lack the necessary credentials
4. **Insufficient permissions** - The service account may not have the required permissions to pull from the registry

To troubleshoot this, you should:
- Verify network connectivity from your nodes to gcr.io
- Check if the image exists at the specified location and tag
- Ensure your nodes have the necessary credentials configured if the registry requires authentication
- Consider checking if there's an updated version of the AWS Gateway API Controller that references a different or more current image

**CRD Installation Error:**
The error regarding `tlsroutes.gateway.networking.k8s.io` indicates a version conflict with the Custom Resource Definition. The error states that v1 was previously a storage version but is missing from the current spec.versions. This suggests you may have had a previous version of the Gateway API CRDs installed.

Regarding cert-manager: While cert-manager is commonly used with webhook configurations, the specific error you're seeing is related to CRD versioning rather than certificate management. The TLS warning about PEM data suggests there may be certificate-related issues, but the primary error is the CRD version mismatch.

To resolve the CRD issue, you should completely remove any existing Gateway API CRDs before installing the new ones to avoid version conflicts.  
**Sources**  
[Troubleshoot Kubernetes Pods in Amazon EKS | AWS re:Post](https://repost.aws/knowledge-center/eks-troubleshoot-kubernetes-pods)

Gateway API crds installation and AWS gateway API controller error

Add your answer

Relevant content