Amazon VPC ENI plugin upgrade breaks readiness checks

Question

When upgrading the Amazon VPC ENI plugin from v1.11.2-eksbuild.1 to v1.12.2-eksbuild.1 two things get broken and need to be set again.

```
kubectl set env daemonset aws-node -n kube-system ENABLE_POD_ENI=true

kubectl patch daemonset aws-node -n kube-system \
  -p '{"spec": {"template": {"spec": {"initContainers": [{"env":[{"name":"DISABLE_TCP_EARLY_DEMUX","value":"true"}],"name":"aws-vpc-cni-init"}]}}}}'

```
fixes the problem, but in the meantime it leaves readiness checks (and potentially other things) broken.

Why are those setting changes undone in this upgrade?  Previous upgrades were fine.  This is a blocker for doing this in production, we can't afford the downtime.  Why would you deliberately disable the whole purpose of the plugin?

The alternative of running those on a loop during the whole process is not very clever at all.  Doing it manually is not feasible, even though we're fairly small with only about 20 clusters.  We use terraform to manage resources.

Answer

I am assuming here that you are using managed add-on on EKS cluster. To preserve your add-on configuration while updating your add-on, you can select "PRESERVE" in conflict resolution method, which will make sure that If conflict exists, it will preserve conflicting fields on the cluster.

In terraform, to implement the same, you can refer [terraform document](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_addon) related to add-on, where in example you can see option of setting "PRESERVE" in resolve_conflicts_on_update.

You can read [AWS blog](https://aws.amazon.com/blogs/containers/amazon-eks-add-ons-preserve-customer-edits/) to know more about preserving edits.

Amazon VPC ENI plugin upgrade breaks readiness checks

Relevant content