Run stateful workloads on-premises with OpenEBS and EKS Hybrid Nodes

Introduction

Running stateful workloads on-premises while maintaining cloud-native practices can be challenging for organizations with hybrid infrastructure requirements. In this post, we'll demonstrate how to leverage OpenEBS with Amazon EKS Hybrid Nodes to manage persistent storage for your containerized applications in an on-premises environment. This solution enables teams to run stateful workloads with the same operational consistency they experience in the cloud while meeting data locality and compliance requirements and other specific use cases.

OpenEBS is a fully open source storage stack used for easily and effectively running stateful applications on Kubernetes. OpenEBS offers Local Volumes for distributed applications that handle their own replication (like Cassandra and MongoDB), and Replicated Volumes for applications that need built-in data redundancy and enterprise storage features across nodes or availability zones (like Jira and GitLab).

In this post, we will use the OpenEBS Local Persistent Volumes (PV) backed by Hostpath to setup a storage for a persistent workload as an example. The OpenEBS Dynamic Local PV provisioner will create Kubernetes Local Persistent Volumes using a unique Hostpath (directory) on the node to persist data.

Prerequisites

• An existing Amazon EKS cluster with Hybrid Nodes. For this post we have a previously deployed an EKS Hybrid Nodes with Kubernetes version 1.32 cluster on Ubuntu 22.04.5 LTS.
• Helm version v3.7 or later.
• Install and configure latest versions of AWS Command Line Interface (AWS CLI), kubectl
• Networking connectivity between nodes on-premise. Note: For replicated Storage, firewall settings should not restrict connection to the nodes. Worker nodes must satisfy the OpenEBS requirements listed here.

Walkthrough

In this post we walk you through the following steps:

• Verify and Install Open-iSCSI on our nodes
• Install OpenEBS 4.2.0 on our cluster through Helm
• Demo deployment
• Cleaning up

Verify Hybrid Nodes

Validate that your nodes are connected and in a Ready state by running the following kubectl command:

kubectl get nodes -o wide
NAME                   STATUS   ROLES    AGE     VERSION               INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
mi-11111111111111111   Ready    <none>   2d20h   v1.32.1-eks-5d632ec   10.80.146.203   <none>        Ubuntu 22.04.5 LTS   5.15.0-134-generic   containerd://1.7.24
mi-22222222222222222   Ready    <none>   2d20h   v1.32.1-eks-5d632ec   10.80.146.201   <none>        Ubuntu 22.04.5 LTS   5.15.0-134-generic   containerd://1.7.24

Install Open-iSCSI

We need to set up Open-iSCSI on each worker node in our data plane. Since our data plane is already established, we SSH into each worker node and add that. For a production environment we would customize an image-builder pipeline to add the open-iSCSI package.

SSH into each hybrid node and verify or install open-iSCSI:

Verify if it is currently installed:

sudo systemctl status iscsid

If it is not already installed, proceed to install it with the following commands:

sudo apt-get update
sudo apt-get install -y open-iscsi
sudo systemctl enable --now iscsid
sudo systemctl status iscsid

Install OpenEBS

helm repo add openebs https://openebs.github.io/openebs
helm repo update
helm install openebs --namespace openebs openebs/openebs --create-namespace

Verify the installation:

helm ls -n openebs
NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
openebs openebs         1               2025-03-21 16:31:13.897492 +0000 UTC    deployed        openebs-4.2.0   4.2.0    

Check the default storage class created

kubectl get sc | grep openebs
mayastor-etcd-localpv    openebs.io/local          Delete          WaitForFirstConsumer   false                  64m
mayastor-loki-localpv    openebs.io/local          Delete          WaitForFirstConsumer   false                  64m
openebs-hostpath         openebs.io/local          Delete          WaitForFirstConsumer   false                  64m
openebs-single-replica   io.openebs.csi-mayastor   Delete          Immediate              true                   64m

Check for the OpenEBS pods:

kubectl get pods --field-selector=status.phase=Running -n openebs
NAME                                              READY   STATUS             RESTARTS        AGE
openebs-localpv-provisioner-699ddcb856-xq9lz      1/1     Running            0               19m
openebs-loki-0                                    1/1     Running            0               19m
openebs-lvm-localpv-controller-86b4d6dcff-dlm66   5/5     Running            0               19m
openebs-lvm-localpv-node-6m7bz                    2/2     Running            0               19m
openebs-lvm-localpv-node-l67j4                    2/2     Running            0               19m
openebs-nats-0                                    3/3     Running            0               19m
openebs-nats-1                                    3/3     Running            0               19m
openebs-nats-2                                    3/3     Running            0               19m
openebs-obs-callhome-6868fd6bb9-cqfvf             2/2     Running            0               19m
openebs-promtail-fh2f6                            1/1     Running            0               19m
openebs-promtail-n5l8g                            1/1     Running            0               19m
openebs-zfs-localpv-controller-5b7846bf9-cljvz    5/5     Running            0               19m
openebs-zfs-localpv-node-586bw                    2/2     Running            0               19m
openebs-zfs-localpv-node-hzkrx                    2/2     Running            0               19m

Deploy Sample workloads

In this section we will deploy a simple stateful application in your hybrid nodes. Create a directory called persistent-workload for your manifests. The next step is to create a PersistentVolumeClaim. Pods uses PersistentVolumeClaims to request for storage. We will request Hostpath Local PV from the OpenEBS Dynamic Local PV provisioner.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: local-hostpath-pvc
spec:
  storageClassName: openebs-hostpath
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5G

The Deployment runs a container that writes timestamps to the persistent volume. The Pod uses Local PV

1. Create a file named deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: inflate-stateful
spec:
  replicas: 1
  selector:
    matchLabels:
      app: inflate-stateful
  template:
    metadata:
      labels:
        app: inflate-stateful
    spec:
      terminationGracePeriodSeconds: 0
      nodeSelector:
        eks.amazonaws.com/compute-type: hybrid
      containers:
        - name: bash
          image: public.ecr.aws/docker/library/bash:4.4
          command: ["/usr/local/bin/bash"]
          args: ["-c", "while true; do echo $(date -u) >> /data/out.txt; sleep 60; done"]
          resources:
            limits:
              cpu: "0.5"
              memory: "100Mi"
            requests:
              cpu: "0.5"
              memory: "50Mi"        
          volumeMounts:
          - mountPath: /data
            name: local-storage
      volumes:
        - name: local-storage
          persistentVolumeClaim:
            claimName: local-hostpath-pvc

1. Run the commands below to deploy the stateful workload:

% kubectl apply -f persistent-workload 
deployment.apps/inflate-stateful created
persistentvolumeclaim/local-hostpath-pvc created

1. Verify the deployment:

% kubectl get po,pv,pvc -n hybrid-nodes-demo
NAME                                    READY   STATUS    RESTARTS   AGE
pod/inflate-stateful-76f8fd99f8-q7wz9   1/1     Running   0          80s
NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                  STORAGECLASS            VOLUMEATTRIBUTESCLASS   REASON   AGE
persistentvolume/pvc-4a0f8502-b18b-429d-bf64-4489432779a5   5G         RWO            Delete           Bound    local-hostpath-pvc                     openebs-hostpath        <unset>                          75s
NAME                                       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       VOLUMEATTRIBUTESCLASS   AGE
persistentvolumeclaim/local-hostpath-pvc   Bound    pvc-4a0f8502-b18b-429d-bf64-4489432779a5   5G         RWO            openebs-hostpath   <unset>                 80s

1. When the pod is created, the Hostpath Local PV will create the PVCs in the /var/openebs/local on the hybrid nodes or any other directory you specified in the storage class. Verify that the data is being written to the volume:

% kubectl exec -it inflate-stateful-76f8fd99f8-q7wz9 -n hybrid-nodes-demo -- cat /data/out.txt
Fri Mar 21 17:22:39 UTC 2025
Fri Mar 21 17:23:39 UTC 2025
Fri Mar 21 17:24:39 UTC 2025

1. On the hybrid node where the pod is running, you can also verify the content of the file:

When the pod is created, the Hostpath Local PV will create the PVCs in the /var/openebs/local on the hybrid nodes or any other directory you specified in the storage class.You can also verify the content of the file directly from the hybrid node where the pod is running:

eksahybrid@hybrid-node:~$ ls -lrt /var/openebs/local
total 4
drwxrwxrwx 2 root root 4096 Mar 21 17:22 pvc-4a0f8502-b18b-429d-bf64-4489432779a5
eksahybrid@hybrid-node:~$ ls -lrt /var/openebs/local/pvc-4a0f8502-b18b-429d-bf64-4489432779a5/
total 4
-rw-r--r-- 1 root root 87 Mar 21 17:24 out.txt
eksahybrid@hybrid-node:~$ cat /var/openebs/local/pvc-4a0f8502-b18b-429d-bf64-4489432779a5/out.txt 
Fri Mar 21 17:22:39 UTC 2025
Fri Mar 21 17:23:39 UTC 2025
Fri Mar 21 17:24:39 UTC 2025

Clean Up

To avoid incurring unnecessary resource usage and to maintain a clean environment, follow these steps to remove all resources created during this demonstration:

First, delete the deployed workload and its associated resources:

kubectl delete -f persistent-workload 

Verify all resources are cleaned up:

kubectl get pods 
kubectl get pvc 
kubectl get pv

Conclusion

In this post, we demonstrated how to implement OpenEBS with Amazon EKS Hybrid Nodes to manage persistent storage for your on-premises containerized workloads. This solution provides:

• Cloud-native storage capabilities for your on-premises environment
• Simplified storage management using Kubernetes-native abstractions
• Consistent operational experience across hybrid deployments
• Enhanced data resilience and availability for stateful applications

By following this implementation pattern, you can confidently run stateful workloads on-premises while maintaining the operational benefits of Amazon EKS. As organizations continue to adopt hybrid architectures, solutions like OpenEBS with EKS Hybrid Nodes provide the flexibility needed to meet diverse infrastructure requirements while ensuring consistent container orchestration practices.

For more information about EKS Hybrid Nodes, visit the official documentation.