Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

NodeCreationFailure:Instances failed to join the kubernetes cluster

0

Dear Support for around 10 days I am trying to create EKS cluster and it is not working. I tried at least 100s of variations still it did not work. in my organization we are planning to migrate from Oracle cloud to AWS for that i am doing POC. i am not using internet gateway as it is not allowed. so POC is my java spring boot application will be deployed in AWS EKS and it will send email through SES and sms through pinpoint. what I am trying i will use vpc endpoint for s3,dynamodb,codebuild ecr ,pinpoint ,ses etc. but its not working. i tried security group ,s3, port connectivilty ingress outgresss, and thousands of things. but its not working. can you guide me what is wrong in my script. even AI is not helping.

$ cat setup4x.sh #!/bin/bash set -e # Exit on command failure

Configuration

AWS_REGION="us-east-1" AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text) VPC_NAME="pocemail-vpc" CLUSTER_NAME="pocemail-cluster" NODEGROUP_NAME="pocemail-workers" ENDPOINT_SG_NAME="pocemail-endpoint-sg" echo "AWS Account: $AWS_ACCOUNT_ID | Region: $AWS_REGION"

1. IAM Roles

echo "Configuring IAM roles..."

EKS Cluster Role

EKS_CLUSTER_ROLE="pocemail-eksclusterrole" if aws iam get-role --role-name "$EKS_CLUSTER_ROLE" >/dev/null 2>&1; then echo "$EKS_CLUSTER_ROLE already exists." else aws iam create-role --role-name "$EKS_CLUSTER_ROLE" --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"eks.amazonaws.com"},"Action":"sts:AssumeRole"}]}' aws iam attach-role-policy --role-name "$EKS_CLUSTER_ROLE" --policy-arn arn:aws:iam::aws:policy/AmazonEKSClusterPolicy fi

EKS Node Role

EKS_NODE_ROLE="pocemail-eksnoderole" if aws iam get-role --role-name "$EKS_NODE_ROLE" >/dev/null 2>&1; then echo "$EKS_NODE_ROLE already exists." else aws iam create-role --role-name "$EKS_NODE_ROLE" --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"ec2.amazonaws.com"},"Action":"sts:AssumeRole"}]}' aws iam attach-role-policy --role-name "$EKS_NODE_ROLE" --policy-arn arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy aws iam attach-role-policy --role-name "$EKS_NODE_ROLE" --policy-arn arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy aws iam attach-role-policy --role-name "$EKS_NODE_ROLE" --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly fi sleep 10 # Wait for IAM propagation

2. VPC & Networking

echo "Setting up VPC..."

VPC_ID=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16 --query 'Vpc.VpcId' --output text) aws ec2 create-tags --resources "$VPC_ID" --tags Key=Name,Value="$VPC_NAME" aws ec2 modify-vpc-attribute --vpc-id "$VPC_ID" --enable-dns-support aws ec2 modify-vpc-attribute --vpc-id "$VPC_ID" --enable-dns-hostnames

DHCP Options with AmazonProvidedDNS

DHCP_OPTIONS_ID=$(aws ec2 create-dhcp-options --dhcp-configuration "Key=domain-name-servers,Values=AmazonProvidedDNS" --query 'DhcpOptions.DhcpOptionsId' --output text) aws ec2 associate-dhcp-options --dhcp-options-id "$DHCP_OPTIONS_ID" --vpc-id "$VPC_ID"

Subnets (two private subnets for high availability)

SUBNET1=$(aws ec2 create-subnet --vpc-id "$VPC_ID" --cidr-block 10.0.1.0/24 --availability-zone "us-east-1f" --query 'Subnet.SubnetId' --output text) aws ec2 create-tags --resources "$SUBNET1" --tags Key=Name,Value=pocemail-subnet-1 "Key=kubernetes.io/role/internal-elb,Value=1" "Key=kubernetes.io/cluster/$CLUSTER_NAME,Value=shared"

SUBNET2=$(aws ec2 create-subnet --vpc-id "$VPC_ID" --cidr-block 10.0.2.0/24 --availability-zone "us-east-1d" --query 'Subnet.SubnetId' --output text) aws ec2 create-tags --resources "$SUBNET2" --tags Key=Name,Value=pocemail-subnet-2 "Key=kubernetes.io/role/internal-elb,Value=1" "Key=kubernetes.io/cluster/$CLUSTER_NAME,Value=shared"

Route Table

RTB_ID=$(aws ec2 create-route-table --vpc-id "$VPC_ID" --query 'RouteTable.RouteTableId' --output text) aws ec2 associate-route-table --route-table-id "$RTB_ID" --subnet-id "$SUBNET1" aws ec2 associate-route-table --route-table-id "$RTB_ID" --subnet-id "$SUBNET2"

Create S3 Gateway Endpoint

echo "Creating S3 Gateway Endpoint..." aws ec2 create-vpc-endpoint
--vpc-id "$VPC_ID"
--service-name "com.amazonaws.$AWS_REGION.s3"
--route-table-ids "$RTB_ID"
--vpc-endpoint-type Gateway
--region "$AWS_REGION"

Security Group for Endpoints and Nodes

ENDPOINT_SG_ID=$(aws ec2 create-security-group --group-name "$ENDPOINT_SG_NAME" --description "SG for VPC Endpoints and Nodes" --vpc-id "$VPC_ID" --query 'GroupId' --output text) aws ec2 authorize-security-group-ingress --group-id "$ENDPOINT_SG_ID" --protocol tcp --port 443 --cidr 10.0.0.0/16 # For EKS API aws ec2 authorize-security-group-ingress --group-id "$ENDPOINT_SG_ID" --protocol tcp --port 10250 --cidr 10.0.0.0/16 # For Kubelet aws ec2 authorize-security-group-ingress --group-id "$ENDPOINT_SG_ID" --protocol tcp --port 53 --cidr 10.0.0.0/16 # For DNS (TCP) aws ec2 authorize-security-group-ingress --group-id "$ENDPOINT_SG_ID" --protocol udp --port 53 --cidr 10.0.0.0/16 # For DNS (UDP)

VPC Endpoints for EKS and ECR

echo "Creating EKS VPC Endpoint..." aws ec2 create-vpc-endpoint --vpc-id "$VPC_ID" --service-name "com.amazonaws.$AWS_REGION.eks" --vpc-endpoint-type Interface --subnet-ids "$SUBNET1" "$SUBNET2" --security-group-ids "$ENDPOINT_SG_ID" --private-dns-enabled --query 'VpcEndpoint.VpcEndpointId' --output text

echo "Creating ECR API VPC Endpoint..." aws ec2 create-vpc-endpoint --vpc-id "$VPC_ID" --service-name "com.amazonaws.$AWS_REGION.ecr.api" --vpc-endpoint-type Interface --subnet-ids "$SUBNET1" "$SUBNET2" --security-group-ids "$ENDPOINT_SG_ID" --private-dns-enabled --query 'VpcEndpoint.VpcEndpointId' --output text

echo "Creating ECR DKR VPC Endpoint..." aws ec2 create-vpc-endpoint --vpc-id "$VPC_ID" --service-name "com.amazonaws.$AWS_REGION.ecr.dkr" --vpc-endpoint-type Interface --subnet-ids "$SUBNET1" "$SUBNET2" --security-group-ids "$ENDPOINT_SG_ID" --private-dns-enabled --query 'VpcEndpoint.VpcEndpointId' --output text

3. EKS Cluster & Node Group

echo "Creating EKS cluster: $CLUSTER_NAME..." aws eks create-cluster --name "$CLUSTER_NAME" --role-arn "arn:aws:iam::$AWS_ACCOUNT_ID:role/$EKS_CLUSTER_ROLE" --resources-vpc-config "subnetIds=$SUBNET1,$SUBNET2,securityGroupIds=$ENDPOINT_SG_ID,endpointPublicAccess=false,endpointPrivateAccess=true" --region "$AWS_REGION" aws eks wait cluster-active --name "$CLUSTER_NAME" --region "$AWS_REGION" || { echo "Cluster creation failed."; exit 1; }

echo "Creating node group: $NODEGROUP_NAME..." aws eks create-nodegroup --cluster-name "$CLUSTER_NAME" --nodegroup-name "$NODEGROUP_NAME" --subnets "$SUBNET1" "$SUBNET2" --node-role "arn:aws:iam::$AWS_ACCOUNT_ID:role/$EKS_NODE_ROLE" --scaling-config minSize=1,maxSize=3,desiredSize=2 --instance-types t3.medium --region "$AWS_REGION" aws eks wait nodegroup-active --cluster-name "$CLUSTER_NAME" --nodegroup-name "$NODEGROUP_NAME" --region "$AWS_REGION" || { echo "Node group failed. Check issues:" aws eks describe-nodegroup --cluster-name "$CLUSTER_NAME" --nodegroup-name "$NODEGROUP_NAME" --query 'nodegroup.health.issues' exit 1 }

echo "Setup complete! Cluster and nodes are active."

AND ERROR IS

Waiter NodegroupActive failed: Waiter encountered a terminal failure state: For expression "nodegroup.status" we matched expected path: "CREATE_FAILED" Node group failed. Check issues: [ { "code": "NodeCreationFailure", "message": "Instances failed to join the kubernetes cluster", "resourceIds": [ "i-088b128cab9aa8e47", "i-091e9310a591e80bb" ] } ]

There are many more version of this script and i can provide if it can help. but I need your guidance. i am struggling for 10 days.

Topics
ComputeNetworking & Content DeliveryContainers
Tags
Amazon VPCAmazon EKS AnywhereKubernetesAmazon EKS DistroSecurity Group
Language
English

asked 2 years ago227 views

3 Answers
0

Based on the error message and your setup script, it appears that the nodes are unable to join the EKS cluster. This is likely due to networking issues, specifically related to the VPC endpoints and security group configurations. Here are some suggestions to troubleshoot and potentially resolve the issue:

VPC Endpoints: Ensure you have created VPC endpoints for EC2, ECR (API and DKR), and EKS. Your script creates endpoints for EKS and ECR, but not for EC2. #Add the EC2 endpoint:

aws ec2 create-vpc-endpoint --vpc-id "$VPC_ID" --service-name "com.amazonaws.$AWS_REGION.ec2" --vpc-endpoint-type Interface --subnet-ids "$SUBNET1" "$SUBNET2" --security-group-ids "$ENDPOINT_SG_ID" --private-dns-enabled

Security Group: Ensure the security group allows all necessary traffic between nodes and the control plane. Add these rules to your security group:

aws ec2 authorize-security-group-ingress --group-id "$ENDPOINT_SG_ID" --protocol all --source-group "$ENDPOINT_SG_ID"
aws ec2 authorize-security-group-egress --group-id "$ENDPOINT_SG_ID" --protocol all --destination-cidr-block 0.0.0.0/0

Route Table: Ensure your route table has a route to the S3 gateway endpoint. Add this after creating the S3 endpoint:

S3_PREFIX_LIST=$(aws ec2 describe-prefix-lists --filters "Name=prefix-list-name,Values=com.amazonaws.$AWS_REGION.s3" --query "PrefixLists[0].PrefixListId" --output text)
aws ec2 create-route --route-table-id "$RTB_ID" --destination-prefix-list-id "$S3_PREFIX_LIST" --gateway-id "$S3_ENDPOINT_ID"

DNS Resolution: Ensure DNS resolution is working correctly within your VPC. Add a VPC endpoint for Amazon DNS:

aws ec2 create-vpc-endpoint --vpc-id "$VPC_ID" --service-name "com.amazonaws.$AWS_REGION.aps-workspaces" --vpc-endpoint-type Interface --subnet-ids "$SUBNET1" "$SUBNET2" --security-group-ids "$ENDPOINT_SG_ID" --private-dns-enabled

IAM Roles: Ensure the node IAM role has the necessary permissions. Add these policies:

aws iam attach-role-policy --role-name "$EKS_NODE_ROLE" --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
aws iam attach-role-policy --role-name "$EKS_NODE_ROLE" --policy-arn arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy

EKS Cluster Endpoint: Ensure the EKS cluster endpoint is accessible from within the VPC. Modify your cluster to use both public and private endpoints temporarily:

aws eks update-cluster-config --name "$CLUSTER_NAME" --resources-vpc-config endpointPublicAccess=true,endpointPrivateAccess=true

After making these changes, try creating the node group again. If issues persist, you can check the node logs by connecting to the instances using Systems Manager Session Manager (since you don't have direct internet access) and examining the /var/log/cloud-init-output.log file.

AWS

answered 2 years ago

  • Rohit Jha
    2 years ago

    Thank you so much, actually its not working. I have put the output of script and code below. it would be really really helpful if you can give more suggestions. i tried for many days.

0

its still not working. below are the output of script.

$ ./setup9x.sh
AWS Account: 739275462423 | Region: us-east-1
Configuring IAM roles...
poc1email-eksclusterrole already exists.
poc1email-eksnoderole already exists.
Setting up VPC...
{
    "AssociationId": "rtbassoc-0a42c9213c626ac4c",
    "AssociationState": {
        "State": "associated"
    }
}
{
    "AssociationId": "rtbassoc-0045226d3df4b66c5",
    "AssociationState": {
        "State": "associated"
    }
}
Creating S3 Gateway Endpoint...
{
    "VpcEndpoint": {
        "VpcEndpointId": "vpce-05e3849d123aab4e1",
        "VpcEndpointType": "Gateway",
        "VpcId": "vpc-04282dc026389197f",
        "ServiceName": "com.amazonaws.us-east-1.s3",
        "State": "available",
        "PolicyDocument": "{\"Version\":\"2008-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":\"*\",\"Action\":\"*\",\"Resource\":\"*\"}]}",
        "RouteTableIds": [
            "rtb-02b14117b5dde11fa"
        ],
        "SubnetIds": [],
        "Groups": [],
        "PrivateDnsEnabled": false,
        "RequesterManaged": false,
        "NetworkInterfaceIds": [],
        "DnsEntries": [],
        "CreationTimestamp": "2025-02-24T06:44:44+00:00",
        "OwnerId": "739275462423"
    }
}
Creating security groups...
{
    "Return": true,
    "SecurityGroupRules": [
        {
            "SecurityGroupRuleId": "sgr-0c7e896aca9c50e9d",
            "GroupId": "sg-07fef715e2af9fef0",
            "GroupOwnerId": "739275462423",
            "IsEgress": false,
            "IpProtocol": "tcp",
            "FromPort": 443,
            "ToPort": 443,
            "ReferencedGroupInfo": {
                "GroupId": "sg-01087bbabde0abb3d",
                "UserId": "739275462423"
            }
        }
    ]
}
Configuring security group rules...
{
    "Return": true,
    "SecurityGroupRules": [
        {
            "SecurityGroupRuleId": "sgr-04f9538cf346acdd9",
            "GroupId": "sg-0a90cddc18af6f344",
            "GroupOwnerId": "739275462423",
            "IsEgress": false,
            "IpProtocol": "tcp",
            "FromPort": 443,
            "ToPort": 443,
            "ReferencedGroupInfo": {
                "GroupId": "sg-01087bbabde0abb3d",
                "UserId": "739275462423"
            }
        }
    ]
}
{
    "Return": true,
    "SecurityGroupRules": [
        {
            "SecurityGroupRuleId": "sgr-0efbf9add67ca37f3",
            "GroupId": "sg-01087bbabde0abb3d",
            "GroupOwnerId": "739275462423",
            "IsEgress": false,
            "IpProtocol": "tcp",
            "FromPort": 10250,
            "ToPort": 10250,
            "ReferencedGroupInfo": {
                "GroupId": "sg-0a90cddc18af6f344",
                "UserId": "739275462423"
            }
        }
    ]
}
{
    "Return": true,
    "SecurityGroupRules": [
        {
            "SecurityGroupRuleId": "sgr-05cf0a9744581901a",
            "GroupId": "sg-01087bbabde0abb3d",
            "GroupOwnerId": "739275462423",
            "IsEgress": false,
            "IpProtocol": "-1",
            "FromPort": -1,
            "ToPort": -1,
            "ReferencedGroupInfo": {
                "GroupId": "sg-01087bbabde0abb3d",
                "UserId": "739275462423"
            }
        }
    ]
}
{
    "Return": true,
    "SecurityGroupRules": [
        {
            "SecurityGroupRuleId": "sgr-0bc3af90d803f4a1f",
            "GroupId": "sg-01087bbabde0abb3d",
            "GroupOwnerId": "739275462423",
            "IsEgress": false,
            "IpProtocol": "tcp",
            "FromPort": 53,
            "ToPort": 53,
            "CidrIpv4": "10.0.0.0/16"
        }
    ]
}
{
    "Return": true,
    "SecurityGroupRules": [
        {
            "SecurityGroupRuleId": "sgr-0bb12eed7b6334713",
            "GroupId": "sg-01087bbabde0abb3d",
            "GroupOwnerId": "739275462423",
            "IsEgress": false,
            "IpProtocol": "udp",
            "FromPort": 53,
            "ToPort": 53,
            "CidrIpv4": "10.0.0.0/16"
        }
    ]
}
Creating VPC Endpoints...
{
    "VpcEndpoint": {
        "VpcEndpointId": "vpce-0b91f25fb143ebd2b",
        "VpcEndpointType": "Interface",
        "VpcId": "vpc-04282dc026389197f",
        "ServiceName": "com.amazonaws.us-east-1.eks",
        "State": "pending",
        "RouteTableIds": [],
        "SubnetIds": [
            "subnet-0744daa2d3f1ad997",
            "subnet-0901b07f0143fa854"
        ],
        "Groups": [
            {
                "GroupId": "sg-07fef715e2af9fef0",
                "GroupName": "poc1email-endpoint-sg"
            }
        ],
        "IpAddressType": "ipv4",
        "DnsOptions": {
            "DnsRecordIpType": "ipv4"
        },
        "PrivateDnsEnabled": true,
        "RequesterManaged": false,
        "NetworkInterfaceIds": [
            "eni-0e4f252d73440f43d",
            "eni-0c88306181bd8c739"
        ],
        "DnsEntries": [
            {
                "DnsName": "vpce-0b91f25fb143ebd2b-58c8y43w.eks.us-east-1.vpce.amazonaws.com",
                "HostedZoneId": "Z7HUB22UULQXV"
            },
            {
                "DnsName": "vpce-0b91f25fb143ebd2b-58c8y43w-us-east-1f.eks.us-east-1.vpce.amazonaws.com",
                "HostedZoneId": "Z7HUB22UULQXV"
            },
            {
                "DnsName": "vpce-0b91f25fb143ebd2b-58c8y43w-us-east-1d.eks.us-east-1.vpce.amazonaws.com",
                "HostedZoneId": "Z7HUB22UULQXV"
            },
            {
                "DnsName": "eks.us-east-1.amazonaws.com",
                "HostedZoneId": "ZONEIDPENDING"
            },
            {
                "DnsName": "eks.us-east-1.api.aws",
                "HostedZoneId": "ZONEIDPENDING"
            }
        ],
        "CreationTimestamp": "2025-02-24T06:46:43.435000+00:00",
        "OwnerId": "739275462423"
    }
}
{
    "VpcEndpoint": {
        "VpcEndpointId": "vpce-0be19b30c63f9fb9e",
        "VpcEndpointType": "Interface",
        "VpcId": "vpc-04282dc026389197f",
        "ServiceName": "com.amazonaws.us-east-1.ecr.api",
        "State": "pending",
        "RouteTableIds": [],
        "SubnetIds": [
            "subnet-0744daa2d3f1ad997",
            "subnet-0901b07f0143fa854"
        ],
        "Groups": [
            {
                "GroupId": "sg-07fef715e2af9fef0",
                "GroupName": "poc1email-endpoint-sg"
            }
        ],
        "IpAddressType": "ipv4",
        "DnsOptions": {
            "DnsRecordIpType": "ipv4"
        },
        "PrivateDnsEnabled": true,
        "RequesterManaged": false,
        "NetworkInterfaceIds": [
            "eni-07de11d91c3f901ca",
            "eni-07e7b9da996771797"
        ],
        "DnsEntries": [
            {
                "DnsName": "vpce-0be19b30c63f9fb9e-50qspj55.api.ecr.us-east-1.vpce.amazonaws.com",
                "HostedZoneId": "Z7HUB22UULQXV"
            },
            {
                "DnsName": "vpce-0be19b30c63f9fb9e-50qspj55-us-east-1d.api.ecr.us-east-1.vpce.amazonaws.com",
                "HostedZoneId": "Z7HUB22UULQXV"
            },
            {
                "DnsName": "vpce-0be19b30c63f9fb9e-50qspj55-us-east-1f.api.ecr.us-east-1.vpce.amazonaws.com",
                "HostedZoneId": "Z7HUB22UULQXV"
            },
            {
                "DnsName": "api.ecr.us-east-1.amazonaws.com",
                "HostedZoneId": "ZONEIDPENDING"
            },
            {
                "DnsName": "ecr.us-east-1.api.aws",
                "HostedZoneId": "ZONEIDPENDING"
            }
        ],
        "CreationTimestamp": "2025-02-24T06:46:47.199000+00:00",
        "OwnerId": "739275462423"
    }
}
{
    "VpcEndpoint": {
        "VpcEndpointId": "vpce-0ad1eb21c2d3f77e7",
        "VpcEndpointType": "Interface",
        "VpcId": "vpc-04282dc026389197f",
        "ServiceName": "com.amazonaws.us-east-1.ecr.dkr",
        "State": "pending",
        "RouteTableIds": [],
        "SubnetIds": [
            "subnet-0744daa2d3f1ad997",
            "subnet-0901b07f0143fa854"
        ],
        "Groups": [
            {
                "GroupId": "sg-07fef715e2af9fef0",
                "GroupName": "poc1email-endpoint-sg"
            }
        ],
        "IpAddressType": "ipv4",
        "DnsOptions": {
            "DnsRecordIpType": "ipv4"
        },
        "PrivateDnsEnabled": true,
        "RequesterManaged": false,
        "NetworkInterfaceIds": [
            "eni-0930c2cdadada0357",
            "eni-04548a0315a1cc488"
        ],
        "DnsEntries": [
            {
                "DnsName": "vpce-0ad1eb21c2d3f77e7-nfxsa5nr.dkr.ecr.us-east-1.vpce.amazonaws.com",
                "HostedZoneId": "Z7HUB22UULQXV"
            },
            {
                "DnsName": "vpce-0ad1eb21c2d3f77e7-nfxsa5nr-us-east-1f.dkr.ecr.us-east-1.vpce.amazonaws.com",
                "HostedZoneId": "Z7HUB22UULQXV"
            },
            {
                "DnsName": "vpce-0ad1eb21c2d3f77e7-nfxsa5nr-us-east-1d.dkr.ecr.us-east-1.vpce.amazonaws.com",
                "HostedZoneId": "Z7HUB22UULQXV"
            },
            {
                "DnsName": "dkr.ecr.us-east-1.amazonaws.com",
                "HostedZoneId": "ZONEIDPENDING"
            },
            {
                "DnsName": "*.dkr.ecr.us-east-1.amazonaws.com",
                "HostedZoneId": "ZONEIDPENDING"
            },
            {
                "DnsName": "dkr-ecr.us-east-1.on.aws",
                "HostedZoneId": "ZONEIDPENDING"
            },
            {
                "DnsName": "*.dkr-ecr.us-east-1.on.aws",
                "HostedZoneId": "ZONEIDPENDING"
            }
        ],
        "CreationTimestamp": "2025-02-24T06:46:51.113000+00:00",
        "OwnerId": "739275462423"
    }
}
Creating EKS cluster: poc1email-cluster...
{
    "cluster": {
        "name": "poc1email-cluster",
        "arn": "arn:aws:eks:us-east-1:739275462423:cluster/poc1email-cluster",
        "createdAt": "2025-02-24T12:16:56.214000+05:30",
        "version": "1.31",
        "roleArn": "arn:aws:iam::739275462423:role/poc1email-eksclusterrole",
        "resourcesVpcConfig": {
            "subnetIds": [
                "subnet-0901b07f0143fa854",
                "subnet-0744daa2d3f1ad997"
            ],
            "securityGroupIds": [
                "sg-0a90cddc18af6f344"
            ],
            "vpcId": "vpc-04282dc026389197f",
            "endpointPublicAccess": false,
            "endpointPrivateAccess": true,
            "publicAccessCidrs": []
        },
        "kubernetesNetworkConfig": {
            "serviceIpv4Cidr": "172.20.0.0/16",
            "ipFamily": "ipv4"
        },
        "logging": {
            "clusterLogging": [
                {
                    "types": [
                        "api",
                        "audit",
                        "authenticator",
                        "controllerManager",
                        "scheduler"
                    ],
                    "enabled": false
                }
            ]
        },
        "status": "CREATING",
        "certificateAuthority": {},
        "platformVersion": "eks.19",
        "tags": {},
        "accessConfig": {
            "bootstrapClusterCreatorAdminPermissions": true,
            "authenticationMode": "CONFIG_MAP"
        }
    }
}
Creating node group: poc1email-workers...
{
    "nodegroup": {
        "nodegroupName": "poc1email-workers",
        "nodegroupArn": "arn:aws:eks:us-east-1:739275462423:nodegroup/poc1email-cluster/poc1email-workers/30ca9b5e-78fa-48ab-915e-8d9f1de55deb",
        "clusterName": "poc1email-cluster",
        "version": "1.31",
        "releaseVersion": "1.31.5-20250212",
        "createdAt": "2025-02-24T12:25:08.281000+05:30",
        "modifiedAt": "2025-02-24T12:25:08.281000+05:30",
        "status": "CREATING",
        "capacityType": "ON_DEMAND",
        "scalingConfig": {
            "minSize": 1,
            "maxSize": 3,
            "desiredSize": 2
        },
        "instanceTypes": [
            "t3.medium"
        ],
        "subnets": [
            "subnet-0901b07f0143fa854",
            "subnet-0744daa2d3f1ad997"
        ],
        "amiType": "AL2023_x86_64_STANDARD",
        "nodeRole": "arn:aws:iam::739275462423:role/poc1email-eksnoderole",
        "diskSize": 20,
        "health": {
            "issues": []
        },
        "updateConfig": {
            "maxUnavailable": 1
        },
        "tags": {}
    }
}

Waiter NodegroupActive failed: Waiter encountered a terminal failure state: For expression "nodegroup.status" we matched expected path: "CREATE_FAILED"
Node group failed. Check issues:
[
    {
        "code": "NodeCreationFailure",
        "message": "Instances failed to join the kubernetes cluster",
        "resourceIds": [
            "i-0448d51ae742bf302",
            "i-0462b218409e01b05"
        ]

and below is the script itself.

sctipt is in another answer as it is not allowing because of word limit.

answered 2 years ago

0

This is the script which is not working.

#!/bin/bash
set -e  # Exit on command failure

# Configuration
AWS_REGION="us-east-1"
AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text)
VPC_NAME="poc1email-vpc"
CLUSTER_NAME="poc1email-cluster"
NODEGROUP_NAME="poc1email-workers"
CLUSTER_SG_NAME="poc1email-cluster-sg"
NODE_SG_NAME="poc1email-node-sg"
ENDPOINT_SG_NAME="poc1email-endpoint-sg"
echo "AWS Account: $AWS_ACCOUNT_ID | Region: $AWS_REGION"

# 1. IAM Roles
echo "Configuring IAM roles..."

# EKS Cluster Role
EKS_CLUSTER_ROLE="poc1email-eksclusterrole"
if aws iam get-role --role-name "$EKS_CLUSTER_ROLE" >/dev/null 2>&1; then
    echo "$EKS_CLUSTER_ROLE already exists."
else
    aws iam create-role --role-name "$EKS_CLUSTER_ROLE" --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"eks.amazonaws.com"},"Action":"sts:AssumeRole"}]}'
    aws iam attach-role-policy --role-name "$EKS_CLUSTER_ROLE" --policy-arn arn:aws:iam::aws:policy/AmazonEKSClusterPolicy
fi

# EKS Node Role
EKS_NODE_ROLE="poc1email-eksnoderole"
if aws iam get-role --role-name "$EKS_NODE_ROLE" >/dev/null 2>&1; then
    echo "$EKS_NODE_ROLE already exists."
else
    aws iam create-role --role-name "$EKS_NODE_ROLE" --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"ec2.amazonaws.com"},"Action":"sts:AssumeRole"}]}'
    aws iam attach-role-policy --role-name "$EKS_NODE_ROLE" --policy-arn arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
    aws iam attach-role-policy --role-name "$EKS_NODE_ROLE" --policy-arn arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
    aws iam attach-role-policy --role-name "$EKS_NODE_ROLE" --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
fi
sleep 10  # Wait for IAM propagation

# 2. VPC & Networking
echo "Setting up VPC..."

VPC_ID=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16 --query 'Vpc.VpcId' --output text)
aws ec2 create-tags --resources "$VPC_ID" --tags Key=Name,Value="$VPC_NAME"
aws ec2 modify-vpc-attribute --vpc-id "$VPC_ID" --enable-dns-support
aws ec2 modify-vpc-attribute --vpc-id "$VPC_ID" --enable-dns-hostnames

# DHCP Options with AmazonProvidedDNS
DHCP_OPTIONS_ID=$(aws ec2 create-dhcp-options --dhcp-configuration "Key=domain-name-servers,Values=AmazonProvidedDNS" --query 'DhcpOptions.DhcpOptionsId' --output text)
aws ec2 associate-dhcp-options --dhcp-options-id "$DHCP_OPTIONS_ID" --vpc-id "$VPC_ID"

# Subnets (two private subnets for HA)
SUBNET1=$(aws ec2 create-subnet --vpc-id "$VPC_ID" --cidr-block 10.0.1.0/24 --availability-zone "us-east-1f" --query 'Subnet.SubnetId' --output text)
aws ec2 create-tags --resources "$SUBNET1" --tags Key=Name,Value=poc1email-subnet-1 "Key=kubernetes.io/role/internal-elb,Value=1" "Key=kubernetes.io/cluster/$CLUSTER_NAME,Value=shared"

SUBNET2=$(aws ec2 create-subnet --vpc-id "$VPC_ID" --cidr-block 10.0.2.0/24 --availability-zone "us-east-1d" --query 'Subnet.SubnetId' --output text)
aws ec2 create-tags --resources "$SUBNET2" --tags Key=Name,Value=poc1email-subnet-2 "Key=kubernetes.io/role/internal-elb,Value=1" "Key=kubernetes.io/cluster/$CLUSTER_NAME,Value=shared"

# Route Table
RTB_ID=$(aws ec2 create-route-table --vpc-id "$VPC_ID" --query 'RouteTable.RouteTableId' --output text)
aws ec2 associate-route-table --route-table-id "$RTB_ID" --subnet-id "$SUBNET1"
aws ec2 associate-route-table --route-table-id "$RTB_ID" --subnet-id "$SUBNET2"

# Create S3 Gateway Endpoint
echo "Creating S3 Gateway Endpoint..."
aws ec2 create-vpc-endpoint \
  --vpc-id "$VPC_ID" \
  --service-name "com.amazonaws.$AWS_REGION.s3" \
  --route-table-ids "$RTB_ID" \
  --vpc-endpoint-type Gateway \
  --region "$AWS_REGION"

# Security Groups
echo "Creating security groups..."

# Cluster Security Group
CLUSTER_SG_ID=$(aws ec2 create-security-group --group-name "$CLUSTER_SG_NAME" --description "EKS Cluster SG" --vpc-id "$VPC_ID" --query 'GroupId' --output text)

# Node Security Group
NODE_SG_ID=$(aws ec2 create-security-group --group-name "$NODE_SG_NAME" --description "EKS Node SG" --vpc-id "$VPC_ID" --query 'GroupId' --output text)

# Endpoint Security Group
ENDPOINT_SG_ID=$(aws ec2 create-security-group --group-name "$ENDPOINT_SG_NAME" --description "VPC Endpoint SG" --vpc-id "$VPC_ID" --query 'GroupId' --output text)
aws ec2 authorize-security-group-ingress --group-id "$ENDPOINT_SG_ID" --protocol tcp --port 443 --source-group "$NODE_SG_ID"  # Allow nodes to access endpoints

# Security Group Rules
echo "Configuring security group rules..."

# Cluster SG: Allow inbound 443 from Node SG
aws ec2 authorize-security-group-ingress --group-id "$CLUSTER_SG_ID" --protocol tcp --port 443 --source-group "$NODE_SG_ID"

# Node SG: Allow inbound 10250 from Cluster SG
aws ec2 authorize-security-group-ingress --group-id "$NODE_SG_ID" --protocol tcp --port 10250 --source-group "$CLUSTER_SG_ID"

# Node SG: Allow all traffic from itself for Kubernetes networking (NEW)
aws ec2 authorize-security-group-ingress --group-id "$NODE_SG_ID" --protocol all --port -1 --source-group "$NODE_SG_ID"

# Node SG: Allow DNS (TCP/UDP 53) within VPC
aws ec2 authorize-security-group-ingress --group-id "$NODE_SG_ID" --protocol tcp --port 53 --cidr 10.0.0.0/16
aws ec2 authorize-security-group-ingress --group-id "$NODE_SG_ID" --protocol udp --port 53 --cidr 10.0.0.0/16

# VPC Endpoints
echo "Creating VPC Endpoints..."
aws ec2 create-vpc-endpoint --vpc-id "$VPC_ID" --service-name "com.amazonaws.$AWS_REGION.eks" --vpc-endpoint-type Interface --subnet-ids "$SUBNET1" "$SUBNET2" --security-group-ids "$ENDPOINT_SG_ID" --private-dns-enabled
aws ec2 create-vpc-endpoint --vpc-id "$VPC_ID" --service-name "com.amazonaws.$AWS_REGION.ecr.api" --vpc-endpoint-type Interface --subnet-ids "$SUBNET1" "$SUBNET2" --security-group-ids "$ENDPOINT_SG_ID" --private-dns-enabled
aws ec2 create-vpc-endpoint --vpc-id "$VPC_ID" --service-name "com.amazonaws.$AWS_REGION.ecr.dkr" --vpc-endpoint-type Interface --subnet-ids "$SUBNET1" "$SUBNET2" --security-group-ids "$ENDPOINT_SG_ID" --private-dns-enabled

# 3. EKS Cluster & Node Group
echo "Creating EKS cluster: $CLUSTER_NAME..."
aws eks create-cluster --name "$CLUSTER_NAME" --role-arn "arn:aws:iam::$AWS_ACCOUNT_ID:role/$EKS_CLUSTER_ROLE" --resources-vpc-config "subnetIds=$SUBNET1,$SUBNET2,securityGroupIds=$CLUSTER_SG_ID,endpointPublicAccess=false,endpointPrivateAccess=true" --region "$AWS_REGION"
aws eks wait cluster-active --name "$CLUSTER_NAME" --region "$AWS_REGION" || { echo "Cluster creation failed."; exit 1; }

echo "Creating node group: $NODEGROUP_NAME..."
aws eks create-nodegroup --cluster-name "$CLUSTER_NAME" --nodegroup-name "$NODEGROUP_NAME" --subnets "$SUBNET1" "$SUBNET2" --node-role "arn:aws:iam::$AWS_ACCOUNT_ID:role/$EKS_NODE_ROLE" --scaling-config minSize=1,maxSize=3,desiredSize=2 --instance-types t3.medium --region "$AWS_REGION"
aws eks wait nodegroup-active --cluster-name "$CLUSTER_NAME" --nodegroup-name "$NODEGROUP_NAME" --region "$AWS_REGION" || {
    echo "Node group failed. Check issues:"
    aws eks describe-nodegroup --cluster-name "$CLUSTER_NAME" --nodegroup-name "$NODEGROUP_NAME" --query 'nodegroup.health.issues'
    exit 1
}

echo "Setup complete! Cluster and nodes are active."

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content