Run Virtual Machine workloads with KubeVirt on Amazon EKS Hybrid Nodes

Amazon EKS Hybrid Nodes provides centralized Kubernetes management across cloud, on-premises and edge environments. The solution leverages existing infrastructure to accelerate modernization without requiring additional hardware investment. It also enables you to fully harness the scalability, availability, and managed benefits of Amazon EKS while maintaining consistent operations and tooling across all environments. Alongside container workloads, you can utilize KubeVirt to bring VM workloads to your Amazon EKS Hybrid Nodes running on baremetal infrastructure.

KubeVirt is an open-source VM management add-on for Kubernetes, and it is currently an incubating project under CNCF. It allows you to run, deploy and manage VMs with Kernel-based Virtual Machine (KVM) as its backend hypervisor, while utilizing Kubernetes as the underlying orchestration platform. In this case, the VM instance is wrapped up in a Kubernetes Pod and this operating model is also known as Container-Native Virtualization. With KubeVirt, you can now manage both container and VM workloads within Amazon EKS, streamlining operations and accelerating application delivery through a unified platform.

In this post, I'll show you how to install KubeVirt onto an Amazon EKS cluster with Hybrid Nodes. I'll also walk you through the process of deploying a Windows VM onto the EKS hybrid cluster.

Disclaimer: Although many organizations have adopted KubeVirt in their production Kubernetes environments, AWS currently does not provide support for KubeVirt deployments on Amazon EKS.

 

Pre-requisites

• An Amazon EKS cluster with Hybrid Nodes (to deploy one, follow this blog)
• Hybrid private connectivity between your on-prem environment and AWS (e.g. VPN or DX)
• Install and configure CNI (Calico/Cilium) and an on-prem Load Balancer controller (e.g. MetalLB etc) (see this post for a comprehensive guide)
• Install a CSI driver (such as OpenEBS) to provide Persistent Volumes (PVs) for VM storage
• AWS CLI version 2.22.8 or later, or v1.36.13 or later with appropriate credentials
• Install Virtctl, the official command-line utility for Kubevirt
• To access graphical console for Windows, you'll need a GUI with Virtual Machine Viewer (virt-viewer)

For this demo, I have deployed a EKS control-plane on v1.31.5 with 2x Hybrid Nodes running on Ubuntu 24.04. I have installed Cilium v1.16.7 for cluster networking, along with MetalLB v0.14.9 configured in L2 mode to provide on-prem load-balancing solution. Additionally, I'm using Synology CSI to provision PVs for VM disks in my lab environment.

 

Walkthrough

• Install and verify KVM on the hybrid nodes
• Install KubeVirt onto the EKS hybrid cluster
• Install Containerized Data Importer (CDI) for importing ISOs and VM disk images
• Prepare PVs for VM disks and upload ISO images
• Launch a Windows VM using KubeVirt
• Clean Up

 

Install KVM on hybrid nodes

To begin, install KVM as the virtualization engine on our hybrid nodes. Make sure all validations are passed before move to the next steps. Note: if your EKS hybrid nodes are also deployed as VMs then you'll need to enable nested virtuzalition (refer to examples for KVM and vSphere). However, for production environment you should only run KVM on the baremetal nodes to avoid performance or reliability issues.

$ sudo apt-get install qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils

$ sudo virt-host-validate qemu
  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking if device /dev/kvm exists                                   : PASS
  QEMU: Checking if device /dev/kvm is accessible                            : PASS
  QEMU: Checking if device /dev/vhost-net exists                             : PASS
  QEMU: Checking if device /dev/net/tun exists                               : PASS
  QEMU: Checking for cgroup 'cpu' controller support                         : PASS
  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS
  QEMU: Checking for cgroup 'cpuset' controller support                      : PASS
  QEMU: Checking for cgroup 'memory' controller support                      : PASS
  QEMU: Checking for cgroup 'devices' controller support                     : PASS
  QEMU: Checking for cgroup 'blkio' controller support                       : PASS

 

Install KubeVirt

To install KubeVirt, first download the latest installation manifest as per the official guide.

export VERSION=$(curl -s https://api.github.com/repos/kubevirt/containerized-data-importer/releases/latest | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/')
wget https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-operator.yaml
wget https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-cr.yaml

Starting from v1.3.0, by default KubeVirt will schedule its infrastructure Pods only on control-plane nodes. However, since Amazon EKS is a managed Kubernetes solution, its control plane nodes are not available to customers. As such, we'll need to update the KubeVirt custom resource (CR) deployment file to explicitly enable scheduling on our hybrid worker nodes.

$ cat kubevirt-cr.yaml 
---
apiVersion: kubevirt.io/v1
kind: KubeVirt
metadata:
  name: kubevirt
  namespace: kubevirt
spec:
[...]
  infra:  ### add this section to enable KubeVirt deployment on our EKS hybrid nodes
    nodePlacement:
      nodeSelector:
        eks.amazonaws.com/compute-type: hybrid

Go ahead and deploy KubeVirt, and wait until all KubeVirt components are up.

$ kubectl apply -f kubevirt-operator.yaml
$ kubectl apply -f kubevirt-cr.yaml

$ kubectl -n kubevirt wait kv kubevirt --for condition=Available
kubevirt.kubevirt.io/kubevirt condition met

 

Install CDI

CDI is a persistent storage management add-on for Kubernetes. It allows you to populate PVCs with VM or ISO images for VM deployment.

Install the latest CDI release, and verify all Pods are running correctly.

$ export VERSION=$(curl -s https://api.github.com/repos/kubevirt/containerized-data-importer/releases/latest | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/')
$ kubectl create -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-operator.yaml
$ kubectl create -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-cr.yaml

$ kubectl get pods -n cdi
NAME                               READY   STATUS    RESTARTS   AGE
cdi-apiserver-68bc795dc5-plq5m     1/1     Running   0          6d4h
cdi-deployment-58767657f4-kz9d2    1/1     Running   0          6d4h
cdi-operator-6b548cdffd-fgthz      1/1     Running   0          6d4h
cdi-uploadproxy-5bff9584d7-9nkl5   1/1     Running   0          6d4h

For this demo, we'll deploy a VM from scratch and install Windows 2022 via an ISO image. To do so, we'll need to expose the CDI Upload Proxy service for uploading the Windows ISO. We will achieve this by deploying a Load Balancer service for cdi-uploadproxy using MetalLB.

$ cat cdi-proxy-lb.yaml 
apiVersion: v1
kind: Service
metadata:
  name: cdi-uploadproxy-lb
  namespace: cdi
  labels:
    cdi.kubevirt.io: "cdi-uploadproxy"
spec:
  type: LoadBalancer
  loadBalancerClass: metallb.io/metallb  
  ports:
    - port: 443
      targetPort: 8443
  selector:
    cdi.kubevirt.io: cdi-uploadproxy

Deploy the Load Balancer service for the cdi-uploadproxy, and take a note of the external IP (192.168.200.204 in my example).

$ kubectl apply -f cdi-proxy-lb.yaml 

$ kubectl get svc -n cdi
NAME                     TYPE           CLUSTER-IP       EXTERNAL-IP       PORT(S)         AGE
[...]
cdi-uploadproxy-lb       LoadBalancer   172.16.105.141   192.168.200.204   443:31636/TCP   6d3h

 

Prepare PVs for VM disks and ISO images

First, we'll prepare a 10G PV for uploading the Windows ISO image. I'm creating a PVC via the Synology CSI to dynamically provision the PV.

$ cat win-iso_pvc.yaml 
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: win-iso
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: synostorage-iscsi
  resources:
    requests:
      storage: 10Gi

kubectl apply -f win-iso_pvc.yaml 

$ kubectl get pvc
NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS        VOLUMEATTRIBUTESCLASS   AGE
win-iso        Bound    pvc-77b13328-6a4e-4ec7-8c7b-ffecf2dd73f9   10Gi       RWO            synostorage-iscsi   <unset>                 6d3h

Next, we'll upload the Windows installation ISO to the win-iso PV via the CDI upload proxy (use the LB external IP as noted from above).

$ virtctl image-upload pvc win-iso --no-create  --uploadproxy-url=https://192.168.200.204  --image-path=/home/sc/demo/iso/win2k22.iso --insecure
Using existing PVC default/win-iso
Waiting for PVC win-iso upload pod to be ready...
Pod now ready
Uploading data to https://192.168.200.204

4.70 GiB / 4.70 GiB [---------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 102.80 MiB p/s 47s

Uploading data completed successfully, waiting for processing to complete, you can hit ctrl-c without interrupting the progress
Processing completed successfully
Uploading /home/sc/demo/iso/win2k22.iso completed successfully

We'll now provision another 80G PV as for the VM disk.

$ cat win-disk_pvc.yaml 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: win-disk
spec:
  storageClassName: synostorage-iscsi
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 80Gi

By now you should have 2x PVCs (one for the ISO, and one for the VM disk) and make sure both status are showing as "Bound".

$ kubectl get pvc
NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS        VOLUMEATTRIBUTESCLASS   AGE
win-disk       Bound    pvc-33e0f9db-5fd8-4f54-84f0-38e074f51ecd   80Gi       RWO            synostorage-iscsi   <unset>                 45h
win-iso        Bound    pvc-77b13328-6a4e-4ec7-8c7b-ffecf2dd73f9   10Gi       RWO            synostorage-iscsi   <unset>                 6d3h

 

Launch a Windows VM

We'll now prepare a KubeVirt VM deployment file. For this example, I have defined the following configurations for the virtual machine.

• CPU: 2 Cores
• Memory: 4G
• CDROM: mapped to PVC win-iso (bootOrder: 1)
• Disk0: mapped to PVC win-disk (bootOrder: 2)
• Network Interface: e1000, bridge mode

$ cat win-testvm.yaml 
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  labels:
    kubevirt.io/os: windows
  name: win-testvm
spec:
  runStrategy: Always
  template:
    metadata:
      creationTimestamp: null
    spec:
      domain:
        cpu:
          cores: 2
        resources:
          requests:
            memory: 4Gi
        devices:
          disks:
          - disk:
              bus: sata
            bootOrder: 2  
            name: disk0
          - cdrom:
              bus: sata
            bootOrder: 1  
            name: winiso
          interfaces:
          - bridge: {}
            model: e1000
            name: default
        features:
          acpi: {}
          apic: {}
          smm: {}      
        firmware:
          bootloader:
            efi:
          uuid: 5d307ca9-b3ef-1234-5678-000000000000
      networks:
      - name: default
        pod: {}
      volumes:
      - name: disk0
        persistentVolumeClaim:
          claimName: win-disk
      - name: winiso
        persistentVolumeClaim:
          claimName: win-iso

Finally we are ready to launch the VM! Create the VM resource and use Virtctl to power it up.

$ kubectl apply -f win-testvm.yaml
$ virtctl start vm win-testvm.yaml

Once the VM is up and running, you can connect to the graphic console using virt-viewer.

$ kubectl get vm
NAME         AGE   STATUS    READY
win-testvm   45h   Running   True

$ virtctl vnc win-testvm

You should see the Windows installation screen, as we have set the VM to boot from the ISO first.

Proceed with the installation, once completed you should see the VM has been automatically assigned an IP address via DHCP.

This IP actually belongs to the underlying KubeVirt launch Pod, since we have set the network interface to "bridge" mode.

$ kubectl get pods  -o wide
NAME                             READY   STATUS    RESTARTS   AGE   IP              NODE                   NOMINATED NODE   READINESS GATES
virt-launcher-win-testvm-hgtvv   2/2     Running   0          45h   192.168.32.61   mi-045b220b7e7d8668f   <none>           1/1

If your Pod CIDR range are routable to the external network, you should now be able to directly connect to the VM via RDP.

 

Clean Up

Once you have completed the lab, follow these steps to remove the resources provisioned during this demo.

$ kubectl delete -f win-testvm.yaml
$ kubectl delete -f win-disk_pvc.yaml
$ kubectl delete -f win-iso_pvc.yaml

 

Conclusion

This post provides a comprehensive guide for installing KubeVirt on an Amazon EKS cluster with Hybrid Nodes, followed by a step-by-step demonstration of deploying a Windows virtual machine instance.

This hybrid computing solution provides the following benefits:

• Unified control-plane for managing both Container and VM workloads
• Consistent operational experience across hybrid environments
• Leverages existing infrastructure without requiring additional hardware investment
• Addresses data locality and compliance requirements

To learn more, please refer to the following resources:

• A deep dive into Amazon EKS Hybrid Nodes
• Amazon EKS User Guide
• KubeVirt User Guide