How do I use Topology Aware Hints in Amazon EKS?

5 minute read

I want to use Topology Aware Hints (TAH) in my Amazon Elastic Kubernetes Service (Amazon EKS) cluster.

Resolution

Note: TAH might not be suitable for clusters that have Amazon Elastic Compute Cloud (Amazon EC2) Spot Instances, Horizontal Pod Autoscaling, or auto scaling turned on. When you use these cluster configurations, you can't achieve an allocation that's proportional to the CPU cores that Amazon EKS allocates to nodes. You exceed the allowed overhead threshold. Also, if there are pod assignment constraints that prohibit endpoint redistribution, then kube-proxy doesn't use TAH.

Set up TAH in your cluster

Prerequisites:

Make sure that your Amazon EKS cluster version is 1.24 or later.
Set up an Amazon EKS cluster and a managed node group with three nodes. Each node must have the same CPU capacity and you must distribute the nodes across three Availability Zones.

To use TAH in Amazon EKS, complete the following steps:

Create a new namespace:

apiVersion: v1
kind: Namespace
metadata:
  name: "example-namespace"
  labels:
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/warn: restricted

Note: Replace example-namespace with your namespace name.

To create a sample deployment, use the BusyBox image:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-deployment-name
  namespace: example-namespace
spec:
  replicas: 3
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      labels:
        app: demo
    spec:
      dnsPolicy: Default
      enableServiceLinks: false
      automountServiceAccountToken: false
      securityContext:
        seccompProfile:
          type: RuntimeDefault
        runAsNonRoot: true
        runAsUser: 1000
        runAsGroup: 1000
      containers:
        - name: busybox
          image: public.ecr.aws/docker/library/busybox:latest
          command: ["/bin/sh"]
          args:
            - "-c"
            - |
              echo "<html><body><h1>PodName: $MY_POD_NAME  NodeName: $MY_NODE_NAME podIP:$MY_POD_IP</h1></body></html>" > /tmp/index.html;
              while true; do
                printf 'HTTP/1.1 200 OK\n\n%s\n' $(cat /tmp/index.html) | nc -l -p 8080
              done
          ports:
            - containerPort: 8080
          env:
          - name: MY_NODE_NAME
            valueFrom:
             fieldRef:
              fieldPath: spec.nodeName
          - name: MY_POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          - name: MY_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          resources:
            limits:
              memory: "128Mi"
              cpu: "500m"
            requests:
              memory: "64Mi"
              cpu: "250m"
          securityContext:
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
          volumeMounts:
          - name: tmp
            mountPath: /tmp
      volumes:
        - name: tmp
          emptyDir: {}

Note: Replace example-deployment-name with your deployment name and example-namespace with your namespace name.

Expose the deployment as a ClusterIP service type, and then add service.kubernetes.io/topology-mode: auto as an annotation:

apiVersion: v1
kind: Service
metadata:
  name: example-service-name
  namespace: example-namespace
  annotations:
   service.kubernetes.io/topology-mode: auto
spec:
  selector:
    app: demo
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080

Note: Replace example-service-name with your service name and example-namespace with your namespace name. For version 1.26 or earlier, use the service.kubernetes.io/topology-aware-hints: auto annotation instead.

To check if the TAHs populate in the endpoint, run the following command:

kubectl get 'endpointslices.discovery.k8s.io' -l kubernetes.io/service-name=example-service-name -n example-namespace -o yaml

Note: Replace example-namespace with your namespace name and example-service-name with your service name.
Example output:

endpoints:
- addresses:
  - 10.0.21.125
  conditions:
    ready: true
    serving: true
    terminating: false
  hints:
    forZones:
    - name: eu-west-1b
  nodeName: ip-10-0-17-215.eu-west-1.compute.internal
  targetRef:
    kind: Pod
    name: example-deployment-name-5875bbbb7c-m2j8t
    namespace: example-namespace
    uid: 4e789648-965e-4caa-91db-bd27d240ea59
  zone: eu-west-1b

To check if the traffic routes to a pod in the same Availability Zone, run the following command to deploy a test pod:

kubectl run tmp-shell --rm -i --tty --image nicolaka/netshoot --overrides='{"spec": { "nodeSelector": {"kubernetes.io/hostname":"example-node-name"}}}'

Note: Replace example-node-name with your node name.

Run the following command to find the pod and node that your test pod connects to:

curl example-service-name.example-namespace:80

Note: Replace example-namespace with your namespace name and example-service-name with your service name.
Example output:

PodName: 7b7b9bf455-c27z9  HTTP/1.1 200 OK
NodeName: ip-10-0-9-45.eu-west-1.compute.internal
HTTP/1.1 200 OK
podIP: example-10.0.11.140

Use PodName and NodeName from the preceding output to check if traffic aligns with the same Availability Zone where you deployed your test pod.

Scale the deployment to four replicas, and then run the following command to inspect the EndpointSlices:

kubectl -n example-namespace scale deployments example-deployment-name --replicas=4

Note: Replace example-namespace with your namespace name and example-deployment-name with your deployment name.

A deployment that you scaled to four replicas results in at least one Availability Zone that has a 50% ratio of endpoints. Also, you exceed the overhead threshold of 20% and kube-proxy doesn't use TAHs.

Troubleshoot TAH issues in your cluster

When you use TAH in Amazon EKS, you might see the following error:

"Skipping topology aware endpoint filtering since node is missing label"

You might experience this issue because your node is missing labels or you used a custom domain name and didn't identify the node IP address.

To resolve this issue, complete the following steps:

To verify that the labels are present on the nodes, run the following command:

kubectl get nodes --show-labels |grep "topology.kubernetes.io/zone"

If Amazon EKS labeled the nodes, then check the kube-proxy logs to confirm that Amazon EKS correctly identified the node IP address:

kubectl logs -n kube-system kube-proxy-pod-name | grep -i retrieved

Note: Replace kube-proxy-pod-name with your kube-proxy pod name.

If you use a custom domain name, then Amazon EKS might not correctly identify your node IP address and you receive the following error:

"I1215 12:24:22.082120 1 server_others.go:138] "Detected node IP" address="127.0.0.1" "

The --hostname-override must be equal to the PrivateDnsName that an EC2 DescribeInstances call returns.

To modify a virtual private cloud (VPC) DNS and Dynamic Host Configuration Protocol (DHCP) options, use a custom domain. The following example uses a modified hostname:

kubectl edit ds -n kube-system kube-proxy

spec:
  template:
    spec:
      containers:
        - name: kube-proxy
          command:
            - kube-proxy
            - --hostname-override=$(NODE_NAME)
            - --v=6
            - --config=/var/lib/kube-proxy-config/config
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName

For more information, see bootstrap.sh on the GitHub website.

Related information

Topology Aware Routing on the Kubernetes website

Exploring the effect of Topology Aware Hints on network traffic in Amazon Elastic Kubernetes Service

Topics: Containers
Tags: Amazon Elastic Kubernetes Service
Language: English

AWS OFFICIALUpdated 7 months ago

No comments

Relevant content

Does EKS support topology Aware Hints?
Accepted Answer
rePost-User-7467190
asked 4 years ago
How to deploy statefulset in EKS with more than one availability zone ?
rePost-User-8613500
asked 2 years ago
TopologySpreadConstraints in EKS auto mode does not seem to work
Accepted Answer
LucD
asked a year ago
AWS DevOps Agent - Topology Shows 0 Resources Found
Octavian
asked 6 months ago
How to provision GPU nodes in EKS Auto mode type cluster
Gourav Gupta
asked 2 years ago
How do I create and troubleshoot topology aware volume provisioning that uses an EBS CSI driver in Amazon EKS?
AWS OFFICIALUpdated 3 years ago
How do I troubleshoot cluster scaling with Karpenter autoscaling in Amazon EKS?
AWS OFFICIALUpdated 9 months ago
FAQs: Cluster networking architecture in Amazon EKS
AWS OFFICIALUpdated 2 years ago
How do I troubleshoot cluster creation errors in Amazon EKS?
AWS OFFICIALUpdated 2 years ago
How to troubleshoot common scaling issues in EKS Auto-Mode
EXPERT
ghanem
published 9 months ago