timout when updating/reading crds

Cluster information:

Kubernetes version: 1.18
AWS EKS

We have a small (3-node) AWS cluster that we’ve been working on for a few months.

It’s been running ArgoCD 1.7.10, and we recently tried to upgrade that to 1.8.7.
That upgrade failed - the command didn’t return, it just hung.

Since then attempting to apply CRDs has just caused a timeout.

To try to work this out We’ve extracted the application CRD from https://raw.githubusercontent.com/argoproj/argo-cd/v1.8.7/manifests/install.yaml

and trying to apply that goes like this:

$ kubectl -n argocd apply -f application-crd.yaml
Error from server (Timeout): error when creating "application-crd.yaml": the server was unable to return a response in the time allotted, but may still be processing the request (post customresourcedefinitions.apiextensions.k8s.io)

We can’t find anything in any logs and we suspect that the problem lies in etcd which is invisible to us but cannot be sure.

We have tried applying other CRD's, some more basic to prove the point and some just as complex. We have trimmed down the argoo CRD yaml and removed the schema and other parts to see which parts are failing but there appears to be no rhyme or reason to what is causing the problem. Sometimes it fails and sometimes it succeeeds.

We have also tried the sample CRD from here:
https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/

And seen it both work and fail.

In short, our kube is behaving erratically and we suspect a bug or a fundamental problem that we cannot diagnose.

We are stuck and this is going to be used in production but we are losing confidence in our AWS EKS kube. Where do we go from here?

Edited by: kophones77 on Mar 25, 2021 2:31 AM