Introduction

Because cybersecurity threats are on the rise, organizations are becoming increasingly aware of the need to adhere to regulations and security standards. Security standards, such as PCI Security Standards Council and Health Insurance Portability and Accountability Act (HIPAA), require that entities build and maintain a secure network and system environments. Implementing controls, such as prohibiting public access to resources that host sensitive data and prohibiting instances from having direct internet access, might help achieve the requirements of such standards.

AWS Solutions Architects work closely with highly regulated organizations to understand and design solutions for their specific security requirements. Examples include assisting a healthcare organization to design a container platform that meets the requirements of HIPAA, and guiding a financial services organization to implement Kubernetes clusters that meet the requirements of PCI DSS.

Organizations that build containers in a strictly regulated industry environment or any organization that seeks to improve their security posture can tighten their security by running Amazon EKS on Fargate in a private only environment. In this environment, your API server doesn’t have public access to the internet. You must run all of your kubectl commands from within your Amazon VPC or a connected network. kubectl is a command line tool that Kubernetes provides for communicating with a Kubernetes cluster's control plane through the Kubernetes API.

In this article, you will learn how to build a Fargate based Amazon EKS cluster in Amazon VPC with a private subnet configuration, and access this cluster through a peered VPC. For additional requirements and consideration, see Deploy private clusters with limited internet access. You can use this use case-specific template to create a cluster that’s preconfigured to run your utmost security-sensitive workloads.

Solution overview

This article explains how to build a private cluster for your security-sensitive workloads using Amazon EKS on Fargate and VPC endpoints for private connectivity to AWS services. The article discusses the following components:

• Serverless compute: With Fargate, your teams can focus on building applications without managing servers. Kubernetes deployments often involve manual provisioning, scaling, patching, and maintenance of servers. AWS manages these tasks in Amazon EKS with Fargate to reduce administrative overhead. When Fargate creates a Fargate Pod, it attaches an elastic network interface in the isolated subnet to the Pod.
• Authentication: Amazon EKS provides ways to grant AWS Identity and Access Management (IAM) permissions to workloads that run in Amazon EKS clusters. To implement the solution in this article, use IAM roles for service accounts (IRSAs) mappings for allowing communication between the Kubernetes Pods and AWS services. IRSA includes the AWS Load Balancer Controller to manage the Elastic Load Balancing (ELB) services in the Kubernetes cluster, Kubernetes external DNS to manage DNS records automatically and facilitate external service discovery, and cert-manager to streamline the management of SSL/TLS certificates. Also, an OpenID Connect (OIDC) endpoint promotes seamless and secure communication.
• Networking modes: With cluster endpoint access control, you can configure the endpoint to be reachable from the public internet or through your VPC. The solution in this article suggests that you deactivate the public endpoint and activate only the private endpoint. Deactivating the public endpoint is mostly ideal for the financial services industry or any industry that intends to prevent unwanted connectivity from the internet. With this setting, you can configure the cluster endpoint to be reachable only through your VPC.
• Secret encryption: You can encrypt Kubernetes secrets with AWS Key Management Service (AWS KMS) keys that you create. Or, you can import the keys that another system generated to AWS KMS and use these keys with the cluster. The KMS keys that you create are customer managed keys. Encryption of your Kubernetes secrets is a security best practice for applications that store sensitive data.

This article also shares a high-level walkthrough through an eksctl quick-start template to build the cluster. The following architecture illustrates a private Amazon EKS cluster that’s accessed by a peered VPC.

Based on your use cases and the type of workloads that you want to deploy to the cluster, you might need to install additional add-ons, such as the following:

• Amazon Elastic Block Store (Amazon EBS) Container Storage Interface (CSI) driver for persistent storage
• ExternalDNS to manage DNS routing of your applications automatically with Amazon Route 53
• AWS Load Balancer Controller to manage Elastic Load Balancing for the Kubernetes cluster
• Fargate logging for observability

For an example use case, see Enabling mTLS with ALB in Amazon EKS.

Prerequisites

To work on this tutorial, you must meet the following prerequisites:

• You have an active AWS account.
• You installed and configured the latest version of the AWS Command Line Interface (AWS CLI) version 2. To check your version, run the following command: aws --version.

Solution walkthrough

Step 1: Prepare your workstation

Launch an Amazon Elastic Compute Cloud (Amazon EC2) instance with an IAM instance profile in your Amazon VPC. You need this instance to run the eksctl commands and access your API server with kubectl commands and API calls from within the VPC. A default VPC comes with a public subnet in each Availability Zone, an internet gateway, and settings to turn on DNS resolution.

1. On your development machine, copy and paste the following text to create an IAM trust policy that allows Amazon EC2 to work on your behalf:

cat << EOF > EC2-Trust.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Service": "ec2.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}
EOF

2. Create an IAM role. This example uses the name EC2-Instance-Profile for the IAM role.

aws iam create-role --role-name EC2-Instance-Profile --assume-role-policy-document file://EC2-Trust.json

Replace EC2-Instance-Profile with the name of the IAM role.

3. Grant the required permissions to the role to manage other AWS services:

aws iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/AdministratorAccess --role-name EC2-Instance-Profile

4. Create an IAM instance profile named, and then add the role to this profile:

EC2InstanceProfile=$(aws iam create-instance-profile --instance-profile-name EC2-Instance-Profile --query 'InstanceProfile.Arn' --output text)
aws iam add-role-to-instance-profile --instance-profile-name EC2-Instance-Profile --role-name EC2-Instance-Profile

Replace EC2-Instance-Profile with the name of your IAM instance profile.

5. Launch an EC2 instance in the default VPC where you intend to create the cluster with the instance profile that you created in the previous step:

InstanceId=$(aws ec2 run-instances \
    --image-id ami-09f85f3aaae282910 \
    --instance-type t2.medium \
    --count 1 \
    --associate-public-ip-address \
    --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=k8sclusterprovisioner}]' \
    --iam-instance-profile Arn=$EC2InstanceProfile \
    --region us-east-2 --query 'Instances[0].[InstanceId]' --output text)
echo $InstanceId

6. After the EC2 instance is in the running state, connect to the instance with AWS Systems Manager Session Manager.

Complete the following steps:

1. Install the latest version of kubectl on the EC2 instance. To check your version, run the following command: kubectl version --client.

2. Install the latest version of eksctl on the EC2 instance. To check your version, run the following command: eksctl info.

3. Install and configure the latest version of the AWS CLI version 2 on the EC2 instance. To check your version, run the following command: aws --version.

4. Install docker on the EC2 instance.

Step 2: Create a private VPC

Use an AWS CloudFormation template to create a VPC that the Amazon EKS cluster can use. This VPC has three private subnets that are deployed in different Availability Zones in the AWS Region. Resources that are deployed to the subnets can't access the internet. Also, the internet can’t access the resources in the subnets. The template uses AWS PrivateLink to create VPC endpoints for several AWS services that nodes typically need access to. If your nodes need outbound internet access, then you can add a public NAT gateway in the Availability Zone of each subnet after you create the VPC. The template creates a security group that denies all inbound traffic, except from resources deployed in the subnets. The security group allows all outbound traffic. The subnets are tagged so that Kubernetes can deploy internal load balancers in these subnets.

To deploy the environment, complete the following steps in your deployment machine:

1. Open the CloudFormation console.

2. On the Region selector, choose the AWS Region where you want to create the stack.

3. On the Stacks page, choose Create stack, and then choose With new resources (standard).

4. On the Create stack page, for Prepare template, select Choose an existing template. Then, under Specify template, choose Amazon S3 URL.

5. For Amazon S3 URL, enter a URL to the template file in an Amazon Simple Storage Service (Amazon S3) bucket. Example: https://s3.us-west-2.amazonaws.com/amazon-eks/cloudformation/2020-10-29/amazon-eks-fully-private-vpc.yaml

6. Choose Next.

7. On the Specify stack details page, for Stack name, enter a name for your stack. Example: amazon-eks-fully-private-vpc.

8. In the Parameters section, enter the values for the parameters that were defined in the template.

9. Choose Next.

10. (Optional) On the Configure stack options page, change the default stack options. For more information, see Configure stack options.

11. Choose Next.

12. On the Review and create page, review the details of your stack. To change any of the values before launching the stack, choose Edit on the section that has the setting that you want to change.

13. If your template contains IAM resources, then select the acknowledgment check box to acknowledge that CloudFormation might create IAM resources with custom names. For more information, see Acknowledging IAM resources in AWS CloudFormation templates.

14. (Optional) You can create a change set to preview the configuration of the stack before creating it. On the Review and create page, choose Create change set and follow the directions. For more information, see Preview the configuration of your stack.

15. To launch your stack, choose Submit.

After you create the stack, note the VpcId for the VPC that the stack created. You need this information when you create your cluster and nodes. Then, note the SubnetIds for the subnets that the stack created. You need the SubnetIDs for least two of the subnets when you create your cluster and nodes.

It takes a few minutes for the CloudFormation stack to be created. To check on the stack's deployment status, run the following command in your EC2 instance.

aws cloudformation describe-stacks --stack-name amazon-eks-fully-private-vpc --query Stacks\[\].StackStatus --output text

Don't continue to the next step until the output of the command is CREATE_COMPLETE.

Then, from the Amazon VPC console, confirm that the VPC endpoints were successfully created, as shown in the following figure:

Step 3: Prepare the cluster environment

From the EC2 instance, note the ID of your cluster VPC and private subnets. You need this information for subsequent steps.

vpc_id=$( aws cloudformation describe-stack-resources --stack-name amazon-eks-fully-private-vpc --query "StackResources[?LogicalResourceId=='VPC'].PhysicalResourceId" --output text)
PrivateSubnet01=$( aws cloudformation describe-stack-resources --stack-name amazon-eks-fully-private-vpc --query "StackResources[?LogicalResourceId=='PrivateSubnet01'].PhysicalResourceId" --output text)
PrivateSubnet02=$( aws cloudformation describe-stack-resources --stack-name amazon-eks-fully-private-vpc --query "StackResources[?LogicalResourceId=='PrivateSubnet02'].PhysicalResourceId" --output text)
PrivateSubnet03=$( aws cloudformation describe-stack-resources --stack-name amazon-eks-fully-private-vpc --query "StackResources[?LogicalResourceId=='PrivateSubnet03'].PhysicalResourceId" --output text)
ControlPlaneSecurityGroup=$( aws cloudformation describe-stack-resources --stack-name amazon-eks-fully-private-vpc --query "StackResources[?LogicalResourceId=='ControlPlaneSecurityGroup'].PhysicalResourceId" --output text)
echo $vpc_id $PrivateSubnet01 $PrivateSubnet02 $PrivateSubnet03 $ControlPlaneSecurityGroup

1. Create a VPC peering connection between the default VPC and the private-only VPC. Create and accept a VPC peering connection between your VPCs:

default_vpc=$(aws ec2 describe-vpcs --filter "Name=isDefault, Values=true" --query 'Vpcs[0].VpcId' --output text)
VpcPeeringConnectionId=$(aws ec2 create-vpc-peering-connection --vpc-id $vpc_id --peer-vpc-id $default_vpc --output text --query 'VpcPeeringConnection.VpcPeeringConnectionId')

aws ec2 accept-vpc-peering-connection --vpc-peering-connection-id $VpcPeeringConnectionId

2. Create a route to the default VPC in the private only VPC:

vpc_id_RouteTableId=$(aws ec2 describe-route-tables --filter "Name=vpc-id, Values=$vpc_id" --query 'RouteTables[0].RouteTableId' --output text)

aws ec2 create-route --route-table-id $vpc_id_RouteTableId --destination-cidr-block 172.31.0.0/16 --vpc-peering-connection-id $VpcPeeringConnectionId

3. Create a route to the private only VPC in the default VPC:

default_vpc_RouteTableId=$(aws ec2 describe-route-tables --filter "Name=vpc-id, Values=$default_vpc" --query 'RouteTables[0].RouteTableId' --output text)

aws ec2 create-route --route-table-id $default_vpc_RouteTableId --destination-cidr-block 192.168.0.0/16 --vpc-peering-connection-id $VpcPeeringConnectionId

4. Create an inbound rule for the control plane security group to allow communication from both VPCs:

aws ec2 authorize-security-group-ingress \
    --group-id $ControlPlaneSecurityGroup \
    --ip-permissions IpProtocol=-1,FromPort=-1,ToPort=-1,IpRanges="[{CidrIp=172.31.0.0/16}]" IpProtocol=-1,FromPort=-1,ToPort=-1,IpRanges="[{CidrIp=192.168.0.0/16}]"

5. Create AWS KMS customer managed keys that the Amazon EKS cluster can use to encrypt your Kubernetes secrets. Use the customer managed key when you run the commands to create the cluster:

export AWS_DEFAULT_REGION="us-east-2"
aws kms create-alias --alias-name alias/fgsecurityquickstart --target-key-id $(aws kms create-key --query KeyMetadata.Arn --output text)
export MASTER_ARN=$(aws kms describe-key --key-id alias/fgsecurityquickstart --query KeyMetadata.Arn --output text)

Now, the EC2 instance can access the resources in the private VPC.

Step 4: Configure the cluster

This section shows how to configure the Amazon EKS cluster in the existing VPC that you recently created with CloudFormation to meet the specific demands of security-sensitive applications in a private only environment. Create a cluster configuration file to define the settings for IAM roles for service accounts, and your AWS KMS key to turn on secret encryption in your cluster. Turn on all the available control plane log types that correspond to the available components of the Kubernetes control plane, and then configure log retention for 60 days. To run multiple workloads in respective namespaces, use Fargate Profiles to create several Linux node pools for the respective workloads so that you can provision the required compute resources.

Create a configuration file (example: cluster-config.yaml), and then enter the following text into an open terminal in your EC2 instance.

Note: Replace the metadata according to your use case.

cat << EOF > cluster-config.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: fg-security-quickstart
  region: us-east-2
  version: "1.30"
  tags:
    # Add more cloud tags if needed for billing
    environment: fargate-security-quickstart

iam:
  withOIDC: true  
  serviceAccounts:
  - metadata:
      name: aws-load-balancer-controller
      namespace: kube-system
    wellKnownPolicies:
      awsLoadBalancerController: true    
  - metadata:
      name: external-dns
      namespace: kube-system
    wellKnownPolicies:
      externalDNS: true
  - metadata:
      name: cert-manager
      namespace: cert-manager
    wellKnownPolicies:
      certManager: true

vpc:
    id: "${vpc_id}"
    securityGroup: "${ControlPlaneSecurityGroup}" # this is the ControlPlaneSecurityGroup
    # only allow private access
    clusterEndpoints:
      publicAccess: false
      privateAccess: true    
    subnets:
      private:
        us-east-2a: { id: $PrivateSubnet01 }
        us-east-2b: { id: $PrivateSubnet02 }
        us-east-2c: { id: $PrivateSubnet03 }


fargateProfiles:
  - name: fp-default
    selectors:
      - namespace: default

  - name: fp-nginx-ingress
    selectors:
      - namespace: ingress-nginx

  - name: fp-mtls
    selectors:
      - namespace: mtls

  - name: fp-kube-system
    selectors:
      - namespace: kube-system

  - name: fp-cert-manager
    selectors:
      - namespace: cert-manager
      
cloudWatch:
 clusterLogging:
 # enable all types of cluster control plane logs 
   enableTypes: ["*"]
    # Sets the number of days to retain the logs for (see [CloudWatch docs](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutRetentionPolicy.html#API_PutRetentionPolicy_RequestSyntax)).
    # By default, log data is stored in CloudWatch Logs indefinitely.
   logRetentionInDays: 60
   
secretsEncryption:
  # ARN of the KMS key
  keyARN: "$MASTER_ARN"   
EOF

Step 5: Create the cluster

The cluster creation process takes several minutes to complete. If you want to monitor the status of cluster creation, then use the CloudFormation console.

To create the Amazon EKS cluster, use the .yaml file that you created in the previous section:

eksctl create cluster -f cluster-config.yaml

This command creates an Amazon EKS cluster with some preconfigured service accounts.

The output looks like the following:

2024-03-11 11:56:22 [✔]  EKS cluster "fg-security-quickstart" in "us-east-2" region is ready

You can also open the Amazon EKS console, choose the cluster, and then choose the Compute tab. As shown in the following figure, you can see that there are no EC2 nodes in the cluster. This decreases the possibility of logging into the nodes to install or configure packages.

If you choose the Networking tab, then you can see that the API server endpoint access is Private.

To verify the state of your cluster, run the following command:

kubectl get pods -A -o wide

The output looks like the following:

NAMESPACE     NAME                       READY   STATUS    RESTARTS   AGE   IP               NODE                                                   NOMINATED NODE   READINESS GATES
kube-system   coredns-78f8b4b9dd-fkfqf   1/1     Running   0          16m   192.168.87.225   fargate-ip-192-168-87-225.us-east-2.compute.internal   <none>           <none>
kube-system   coredns-78f8b4b9dd-vgtrw   1/1     Running   0          16m   192.168.24.41    fargate-ip-192-168-24-41.us-east-2.compute.internal    <none>           <none>

Step 6: Deploy a sample application

The next step is to deploy security sensitive workloads to your Kubernetes cluster. To avoid the ErrImagePull error in a private cluster, make sure that your cluster pulls images from a container registry that's in your VPC. You can create an Amazon Elastic Container Registry (Amazon ECR) in your VPC and copy container images to it for your nodes to pull from. For more information, see Copy a container image from one repository to another repository. For this sample application, you can copy a publicly available NGINX image into a private Amazon ECR that Pods in the Amazon EKS cluster can access.

Complete the following steps to copy the NGINX image into a private Amazon ECR:

1. Create an Amazon ECR repository:

repositoryUri=$(aws ecr create-repository --region us-east-2 --repository-name docker-2048 --query 'repository.repositoryUri' --output text) 

2. Pull the container image from the Amazon ECR Public Gallery:

docker pull public.ecr.aws/docker/library/nginx:alpine

3. Tag the image that you pulled with your registry, repository, and tag:

docker tag public.ecr.aws/docker/library/nginx:alpine $repositoryUri:alpine

4. Authenticate into your registry. Replace the Region in the following command with the Region of your choice.

ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' --output text)\
aws ecr get-login-password --region us-east-2 | docker login --username AWS --password-stdin $ACCOUNT_ID.dkr.ecr.us-east-2.amazonaws.com

The output looks like the following:

Login succeeded

5. Push the image to your repository:

docker push $repositoryUri:alpine

To deploy a sample application, complete the following steps: 1. The following sample application is a basic NGINX web application. To create a sample workload, copy and paste the following code block into your terminal:

cat << EOF > mtls.yaml
---
apiVersion: v1
kind: Namespace
metadata:
  name: mtls
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: mtls
  name: deployment-2048
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: app-2048
  replicas: 2
  template:
    metadata:
      labels:
        app.kubernetes.io/name: app-2048
    spec:
      containers:
      - image: $repositoryUri:alpine
        imagePullPolicy: Always
        name: app-2048
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  namespace: mtls
  name: service-2048
spec:
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
  type: NodePort
  selector:
    app.kubernetes.io/name: app-2048
EOF

2. Create the mtls namespace, application deployment, and service objects in the cluster:

kubectl create -f mtls.yaml -n mtls

3. After approximately 60 seconds, verify that the application is running on Fargate:

kubectl get pods -n mtls -o wide

The output looks like the following:

NAME                               READY   STATUS    RESTARTS   AGE   IP                NODE                                                    NOMINATED NODE   READINESS GATES
deployment-2048-74d5d8944b-xs22l   1/1     Running   0          80s   192.168.126.96    fargate-ip-192-168-126-96.us-east-2.compute.internal    <none>           <none>
deployment-2048-74d5d8944b-zh65m   1/1     Running   0          80s   192.168.186.188   fargate-ip-192-168-186-188.us-east-2.compute.internal   <none>           <none>

4. Create a test Pod in the same namespace as the application:

kubectl run tmp --image=${repositoryUri}:alpine -n mtls

5. Wait for approximately 60 seconds for the Pod to be in the running state. Verify the HTTP status code of the service that you use to manage the application within the Amazon EKS cluster. A successful HTTP request returns a 200 HTTP code.

kubectl exec -it tmp -n mtls -- curl -s -o /dev/null -w "%{http_code}" http://service-2048

Cleanup

To avoid incurring future charges, delete the resources that you created for this tutorial.

• Delete all the Kubernetes objects provisioned in the namespace:

kubectl delete namespace mtls

• Delete the Amazon EKS cluster:

eksctl delete cluster -f ./cluster-config.yaml

• Delete the Amazon ECR repository:

aws ecr delete-repository --region us-east-2 --repository-name docker-2048 --force

• Delete VPC peering:

aws ec2 delete-vpc-peering-connection --vpc-peering-connection-id $VpcPeeringConnectionId

• Delete the CloudFormation stack:

aws cloudformation delete-stack --stack-name amazon-eks-fully-private-vpc

• To terminate the EC2 instance, and then delete the IAM role that you attached to the instance, run the following commands from your development machine:

aws ec2 terminate-instances --instance-ids $InstanceId --region us-east-2

aws iam remove-role-from-instance-profile \
    --instance-profile-name EC2-Instance-Profile \
    --role-name EC2-Instance-Profile

aws iam remove-role-from-instance-profile \
    --instance-profile-name EC2-Instance-Profile \
    --role-name EC2-Instance-Profile

aws iam delete-instance-profile --instance-profile-name EC2-Instance-Profile

aws iam detach-role-policy \
    --role-name EC2-Instance-Profile \
    --policy-arn arn:aws:iam::aws:policy/AdministratorAccess

aws iam delete-role --role-name EC2-Instance-Profile

Conclusion

In this article, you learned how to successfully set up an Amazon EKS cluster with Fargate nodes that are in a fully private environment. This infrastructure can create the foundation that you need for secure workload deployments with reduced risks and less administrative efforts in cluster management. With only a private endpoint turned on, all traffic to your cluster API server is from within your cluster's VPC or a connected network with AWS Virtual Private Network,  AWS Direct Connect, or a VPC peering connection. There is no public access to your API server from the internet. You can run the kubectl commands only from within your VPC or a connected network.

AWS Support engineers and Technical Account Managers (TAMs) can help you with general guidance, best practices, troubleshooting, and operational support on AWS and Amazon Elastic Kubernetes Services Security Immersion Workshop. To learn more about our plans and offerings, see AWS Support. To participate in the Amazon EKS Security Immersion Workshop that's hosted by AWS, contact your TAM.

About the author

Olawale Olaleye

Olawale is a Senior Containers Specialist Solutions Architect at AWS. He has several years of experience in architecting, building, and managing enterprise-scale solutions. He is passionate about helping customers efficiently architect container technologies on AWS. Connect with him on LinkedIn at /in/olawale-olaleye/.