How do I configure ipvs kube-proxy mode in Amazon EKS?

4 minute read
0

I want to change the Kubernetes network proxy mode from the default 'iptables' to 'ipvs' in Amazon Elastic Kubernetes Service (Amazon EKS).

Resolution

Prerequisites:

  • Install kubectl.
  • Install and configure the latest version of the AWS Command Line Interface (AWS CLI).
  • Install eksctl from the eksctl website.

Note: If you receive errors when you run AWS CLI commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

For an existing Amazon EKS cluster

To configure ipvs kube-proxy mode for an existing EKS cluster, complete the following steps:

  1. Make sure that ipvs is turned on for your worker node:

    sudo ipvsadm -L

    When the proxy mode is set to the default iptables, the output looks similar to the following example:

    IP Virtual Server version 1.2.1 (size=4096)
    Prot LocalAddress:Port Scheduler Flags
      -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
  2. To make sure that you have the necessary IPVS kernel modules, run the following command:

    sudo lsmod | egrep -i "ip_vs|ip_vs_rr|ip_vs_wrr|ip_vs_sh|nf_conntrack"
  3. If IPVS modules are missing in the output, then run the following command to install the missing kernel modules:

    sudo modprobe ip_vs 
    sudo modprobe ip_vs_rr
    sudo modprobe ip_vs_wrr 
    sudo modprobe ip_vs_sh
    sudo modprobe nf_conntrack  
  4. To determine if your kube-proxy is a managed Amazon EKS add-on or self-managed add-on, run the following command:

    aws eks list-addons --cluster-name my-cluster | grep proxy

    Note: Replace my-cluster with your cluster name. A managed Amazon EKS add-on returns kube-proxy as an output.

  5. Based on your add-on, configure the kube-proxy add-on with ipvs mode and the round robin option to equally distribute traffic to the backing servers.
    Managed Amazon EKS add-on

    aws eks update-addon --cluster-name my-cluster --addon-name kube-proxy \
        --addon-version v1.24.17-eksbuild.4 \
        --configuration-values '{"ipvs": {"scheduler": "rr"}, "mode": "ipvs"}' \
        --resolve-conflicts OVERWRITE

    Self-managed add-on
    Backup the kube-proxy config configmap:

    kubectl get cm kube-proxy-config -n kube-system -o yaml > kube-proxy-config-old.yml

    Edit the kube-proxy-config configmap:

    kubectl edit cm kube-proxy-config -n kube-system

    In the config, change the mode parameter from iptables to ipvs, and then change scheduler to rr for round robin.

    ...
        ipvs:
          excludeCIDRs: null
          minSyncPeriod: 0s
          scheduler: "rr"          # add rr
          syncPeriod: 30s
        kind: KubeProxyConfiguration
        metricsBindAddress: 0.0.0.0:10249
        mode: "ipvs"         # change from iptables
    ... 

    To apply the configuration changes, reload your cluster worker nodes. Use eksctl to scale in and scale out the worker nodes:

    # get node group names
    eksctl get nodegroup --cluster=my-cluster
    
    # scale-in
    eksctl scale nodegroup --cluster=my-cluster --nodes=0 --name=my-nodegroup-name --nodes-min=0 --nodes-max=3 --wait
    
    # scale-out
    eksctl scale nodegroup --cluster=my-cluster --nodes=2 --name=my-nodegroup-name --nodes-min=2 --nodes-max=3 --wait

    Note: Replace my-cluster and my-nodegroup-name with your parameters. When you scale out, replace the node counts based on your cluster needs.

  6. To verify that ipvs mode is configured, run the following command:

    sudo ipvsadm -L

    Example output:

    IP Virtual Server version 1.2.1 (size=4096)
    Prot LocalAddress:Port Scheduler Flags
      -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
    TCP  ip-10-100-0-1.eu-west-1.comp rr
      -> ip-192-168-118-22.eu-west-1. Masq    1      5          0
      -> ip-192-168-187-76.eu-west-1. Masq    1      6          0
    TCP  ip-10-100-0-10.eu-west-1.com rr
      -> ip-192-168-168-152.eu-west-1 Masq    1      0          0         
      -> ip-192-168-183-81.eu-west-1. Masq    1      0          0         
    UDP  ip-10-100-0-10.eu-west-1.com rr
      -> ip-192-168-168-152.eu-west-1 Masq    1      0          0         
      -> ip-192-168-183-81.eu-west-1. Masq    1      0          0 

    The TCP and UDP entries are for Kubernetes and CoreDNS services in the cluster.

  7. To make sure that there are no iptables entries for kube-svc* in the cluster, run the following command:

    sudo iptables-save | grep -i kube-svc

    If there are no iptables in the cluster, then the preceding command doesn't generate an output.

For a new Amazon EKS cluster

To configure ipvs kube-proxy mode for a new EKS cluster, complete the following steps:

  1. When you create node groups, bootstrap the worker nodes user data to install the IPVS dependencies:

    ...
    #!/bin/bash
    echo "Running custom user data script"
    yum install -y ipvsadm
    ipvsadm -l
    modprobe ip_vs 
    modprobe ip_vs_rr
    modprobe ip_vs_wrr 
    modprobe ip_vs_sh
    modprobe ip_vs_lc
    modprobe nf_conntrack
    ...
  2. To create a managed Amazon EKS add-on for kube-proxy with the IPVS parameter, run the following command:

    aws eks create-addon --cluster-name my-cluster --addon-name kube-proxy \
        --addon-version v1.29.0-minimal-eksbuild.1 \
        --configuration-values '{"ipvs": {"scheduler": "rr"}, "mode": "ipvs"}' \
        --resolve-conflicts OVERWRITE

    Note: Replace v1.29.0-minimal-eksbuild.1 with the latest available kube-proxy version that's compatible with your Amazon EKS cluster version. For more information, see Updating the Kubernetes kube-proxy self-managed add-on.

Related information

Proxy modes on the Kubernetes website

Configuration for using NodeLocal DNSCache in Kubernetes clusters on the Kubernetes website

AWS OFFICIAL
AWS OFFICIALUpdated a month ago