timout when updating/reading crds


Cluster information:

Kubernetes version: 1.18

We have a small (3-node) AWS cluster that we’ve been working on for a few months.

It’s been running ArgoCD 1.7.10, and we recently tried to upgrade that to 1.8.7.
That upgrade failed - the command didn’t return, it just hung.

Since then attempting to apply CRDs has just caused a timeout.

To try to work this out We’ve extracted the application CRD from https://raw.githubusercontent.com/argoproj/argo-cd/v1.8.7/manifests/install.yaml

and trying to apply that goes like this:

$ kubectl -n argocd apply -f application-crd.yaml
Error from server (Timeout): error when creating "application-crd.yaml": the server was unable to return a response in the time allotted, but may still be processing the request (post customresourcedefinitions.apiextensions.k8s.io)

We can’t find anything in any logs and we suspect that the problem lies in etcd which is invisible to us but cannot be sure.

We have tried applying other CRD's, some more basic to prove the point and some just as complex. We have trimmed down the argoo CRD yaml and removed the schema and other parts to see which parts are failing but there appears to be no rhyme or reason to what is causing the problem. Sometimes it fails and sometimes it succeeeds.

We have also tried the sample CRD from here:

And seen it both work and fail.

In short, our kube is behaving erratically and we suspect a bug or a fundamental problem that we cannot diagnose.

We are stuck and this is going to be used in production but we are losing confidence in our AWS EKS kube. Where do we go from here?

Edited by: kophones77 on Mar 25, 2021 2:31 AM

asked 3 years ago305 views
1 Answer

It turns out that there were some validating webhooks that did not have backends and causing timeouts to API and strange behaviour.

A lack of errors and no real debugging options made this a tough one to fix.

answered 3 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions