Iam attempting to create a self managed node groups to launch EC2 instances using Amazon Linux 2023 EKS optimized AMI. However. I am encountering an issue where the node groups are not joining the cluster, which is resulting in a 'DEGRADE' error for CoreDNS.

When I use the same Terraform code and eks module to create an EKS cluster with managed node groups, it works perfectly, with no issues related to node joining or CoreDNS.

This appears to be a bug. Is there a workaround to resolve this problem by modifying the Terraform code? Any suggestions or advice would be greatly appreciated.

Error: waiting for EKS Add-On (ecp-ppp-prod:coredns) create: timeout while waiting for state to become 'ACTIVE' (last state: 'DEGRADED', timeout: 20m0s) │ │ with module.eks.aws_eks_addon.this["coredns"], │ on .terraform/modules/eks/main.tf line 498, in resource "aws_eks_addon" "this": │ 498: resource "aws_eks_addon" "this" { │

Here is the Terraform modules and Reproduction Code

`module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.13"

  cluster_name                   = local.name
  cluster_version                = local.cluster_version 

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets  
  control_plane_subnet_ids = module.vpc.intra_subnets

  cluster_endpoint_private_access = true
  cluster_endpoint_public_access  = true

  enable_irsa = true

  enable_cluster_creator_admin_permissions = true

  # This will set the cluster authentication use API and CONFIG MAP, EKS will automatically create an access entry for the IAM role(s) used by managed nodegroup(s)
  authentication_mode = "API_AND_CONFIG_MAP" 


  # EKS Addons
  cluster_addons = {
    coredns    = {
      most_recent = true
    }

    eks-pod-identity-agent = {
      most_recent = true
    }

    kube-proxy = {
      most_recent = true
    }
    vpc-cni = {
      # Specify the VPC CNI addon should be deployed before compute to ensure
      # the addon is configured before data plane compute resources are created
      # See README for further details
      before_compute = true
      most_recent    = true # To ensure access to the latest settings provided
      configuration_values = jsonencode({
        env = {
          # Reference docs https://docs.aws.amazon.com/eks/latest/userguide/cni-increase-ip-addresses.html
          ENABLE_PREFIX_DELEGATION = "true"
          WARM_PREFIX_TARGET       = "1"
        }
      })
    }
  } 
 

  self_managed_node_groups = {
    # AL2023 node group utilizing new user data format which utilizes nodeadm
    # to join nodes to the cluster (instead of /etc/eks/bootstrap.sh)
    al2023_nodeadm = {

      name = "cis-self-mng"
      use_name_prefix = true
      launch_template_description     = "Self managed node group example launch template"

      # ebs_optimized     = true
      enable_monitoring = true

      subnet_ids = module.vpc.public_subnets

      min_size         = 1
      max_size         = 3
      desired_capacity = 1

      instance_type = "m6i.large"

      enable_bootstrap_user_data = true
      is_eks_managed_node_group = false

      ami_id = data.aws_ami.image_cis_eks.id 
 

      launch_template_name            = "amazon-eks-al2023-node-1.30"
      launch_template_use_name_prefix = true 
      launch_template_description     = "amazon-eks-al2023-node-1.30"

   
      // The following variables are necessary if you decide to use the module outside of the parent EKS module context.
      // Without it, the security groups of the nodes are empty and thus won't join the cluster.
      vpc_security_group_ids = [
        module.eks.cluster_primary_security_group_id,
        module.eks.cluster_security_group_id,
      ]
 
      # AL2023 node group utilizing new user data format which utilizes nodeadm
      # to join nodes to the cluster (instead of /etc/eks/bootstrap.sh)
      cloudinit_pre_nodeadm = [
        {
          content_type = "application/node.eks.aws"
          content      = <<-EOT
            ---
            apiVersion: node.eks.aws/v1alpha1
            kind: NodeConfig
            spec:
              featureGates: 
                InstanceIdNodeName: true
              cluster: ecp-ppp-prod
              apiServerEndpoint: https://aaa.us-east-1.eks.amazonaws.com
              certificateAuthority: aaa
              cidr: 1aa.bb.0.0/16
              kubelet:
                config:
                  shutdownGracePeriod: 30s
                  featureGates:
                    DisableKubeletCloudCredentialProviders: true 
              config: |
              [plugins."io.containerd.grpc.v1.cri".containerd]
              discard_unpacked_layers = false            
          EOT
        }
      ]
    } 
  } 

  tags = local.tags 
} `

And i even tried with self managed module however getting the same issue.

module "self_managed_node_group" {
  source = "terraform-aws-modules/eks/aws//modules/self-managed-node-group"
  version = "20.13.1"

  name                = "cis-self-mng"
  cluster_name        = "aaa-ppp-prod"
  cluster_version     = "1.30"
  cluster_endpoint    = "https://aa.gr7.us-east-1.eks.amazonaws.com"
  cluster_auth_base64 = "bb"
  cluster_ip_family    = "ipv4"
  cluster_service_cidr = "aa.bb.0.10"

  subnet_ids = module.vpc.private_subnets


  ami_id   = data.aws_ami.image_cis_eks.id

  user_data_template_path = "${path.module}/modules/user_data/templates/al2023_custom.tpl"

  cloudinit_pre_nodeadm = [{
    content      = <<-EOT
      ---
      apiVersion: node.eks.aws/v1alpha1
      kind: NodeConfig
      spec:
        kubelet:
          config:
            shutdownGracePeriod: 30s
            featureGates:
              DisableKubeletCloudCredentialProviders: true
    EOT
    content_type = "application/node.eks.aws"
  }]

  cloudinit_post_nodeadm = [{
    content      = <<-EOT
      echo "All done"
    EOT
    content_type = "text/x-shellscript; charset=\"us-ascii\""
  }]

  // The following variables are necessary if you decide to use the module outside of the parent EKS module context.
  // Without it, the security groups of the nodes are empty and thus won't join the cluster.
  vpc_security_group_ids = [
    module.eks.cluster_primary_security_group_id,
    module.eks.cluster_security_group_id,
  ]

  min_size     = 1
  max_size     = 4
  desired_size = 1

  launch_template_name   = "cis-self-mng"
  instance_type          = "m5.2xlarge"

  tags = {
    Environment = "ppp-prod"
    Terraform   = "true"
  }
}

This is my VPC supporting TF module

################################################################################

VPC supportings

################################################################################

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0.0"

  name = local.name
  cidr = local.vpc_cidr

  azs             = ["us-east-1a", "us-east-1b", "us-east-1c"]
  private_subnets = ["10.aa.aa.0/24", "10.aa.aa.0/24", "10.aa.bb.0/24"]
  public_subnets  = ["10.aa.16.aa/26", "10.cc.dd.128/26", "10.dd.ee.1ff![![![![Enter image description here](/media/postImages/original/IMLiInYSrVSD6DomEd9MK5-w)
Enter image description here](/media/postImages/original/IM_8p8_BYQTTy_oxui2_zRjA)
Enter image description here](/media/postImages/original/IM2cOdzz2HQtaLXw1Gn4v5bg)
Enter image description here](/media/postImages/original/IMLi54e6AaRO6xuY-S9vY9MQ)
/26"]

  enable_nat_gateway     = true
  create_igw             = true

  single_nat_gateway     = false
  one_nat_gateway_per_az = false

  enable_dns_hostnames   = true
  enable_dns_support     = true

  enable_flow_log                      = true
  create_flow_log_cloudwatch_iam_role  = true
  create_flow_log_cloudwatch_log_group = true


  public_subnet_tags = {
    "kubernetes.io/role/elb"                        = 1
    "kubernetes.io/cluster/${var.environment_name}" = "owned"
  }

  private_subnet_tags = {
    "kubernetes.io/role/internal-elb" = 1
    # Tags subnets for Karpenter auto-discovery
    # "karpenter.sh/discovery" = local.name

    "kubernetes.io/cluster/${var.environment_name}" = "owned"

  }

  tags = local.tags

}