How do I use custom user data with AL2023 Amazon EKS nodes?

Short description

AL2023 uses the nodeadm node initialization process that uses a YAML configuration schema. For self-managed node groups or managed node groups with a custom launch template, you must provide additional cluster metadata when you create a new node group. The minimum metadata requirements that you must include in the YAML configuration schema are the cluster name, API server endpoint, certificate authority, and service CIDR.

Example configuration with the minimum required parameters:

---
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  cluster:
    name: EKS-CLUSTER-NAME
    apiServerEndpoint: API-SERVER-ENDPOINT
    certificateAuthority: CERTIFICATE-AUTHORITY
    cidr: SERVICE-IPv4-RANGE

For more information, see the Before upgrading to AL2023 section in the Amazon EKS-optimized Amazon Linux 2023 AMIs now available. 

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

You must use an Amazon EKS AMI that runs on AL2023. To view the AMI ID, run the following get-parameter AWS CLI command:

aws ssm get-parameter --name /aws/service/eks/optimized-ami/kubernetes-version/ami-type/recommended/image_id \
    --region region-code --query "Parameter.Value" --output text

Note: Replace kubernetes-version with a supported Amazon EKS version and region-code with your AWS Region. Also, replace ami-type with amazon-linux-2023/x86_64/standard for x86 based instances or amazon-linux-2023/arm64/standard for ARM instances.

Add the minimum required parameters to user data in the following format:

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="BOUNDARY"

--BOUNDARY
Content-Type: application/node.eks.aws

---
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec: ...

--BOUNDARY--

Note: You can also embed the parameters within a mime multi-part document. For more information, see Mime multi-part archive on the cloud-init website.

Use the DescribeCluster API to find the ClusterDetails and view the minimum required parameters. For example, when you use NodeConfigSpec, you can pass the config and flag kubelet options to kubelet. For more information, see CredentialProviderConfig and kubelet on the Kubernetes website.

Example user data configuration:

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="BOUNDARY"

--BOUNDARY
Content-Type: application/node.eks.aws

---
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  cluster:
    apiServerEndpoint: API_SERVER_ENDPOINT
    certificateAuthority: CERTIFICATE
    cidr: SERVICE_IPv4_RANGE
    name: CLUSTER_NAME
  kubelet:
    config:
      maxPods: 17 
    flags:
    - "--node-labels=key=value" 

--BOUNDARY
Content-Type: text/x-shellscript;

#!/bin/bash
  set -o xtrace
  yum install htop -y

--BOUNDARY--

Note: When you enter user data in the launch template, make sure not to include extra spaces or indentations in the code. Nodes can't join the cluster when there's a spacing issue because nodeadm fails to start.