How to massively increase EKS pod limit

0

I am trying to move an existing Kubernetes cluster to EKS but I ran into a bit of a snag.

By default a t2.small instance for example has a pod limit of 11 (eleven). That is ridiculously low. On a node with equivalent CPU and RAM I'm currently running 110 pods.

This is mainly because my pods are tiny, plenty and mostly idle. I am running review apps for feature branches during software development, so there is one installation for each feature branch and each installation has a couple of pods, one per service. Most of the time those pods do nothing, except for a couple of minutes per day when someone reviews the particular feature branch.

I need to increase the pod limit per instance massively for this to make sense. A possible alternative is of course to deploy a couple of EC2 instances and install k3s on them but I would prefer to have this all on EKS.

AndreKR
asked 2 years ago8009 views
3 Answers
2

First of all, generally speaking you should not run 110 pods on t2.small. You should seriously consider a different way of launching pods. If you don't need the pod, delete them. Or have some other ways to schedule the jobs.

As to why you encounter the limit is due to the fact that by default each pod get its IPs from ENIs of the node. And there is a limit how many ENIs and IPs the node can have.

One way to work around the issue is to use custom networking and CNI add-on. You can refer here. You should be able to at least get ~110 pods. However, you should seriously consider your scheduling strategy.

Jason_S
answered 2 years ago
  • Not sure if I get the reasons why many pods are bad. A pod is just a process, with minimal overhead. If the executable uses maybe 5 MB RAM and is mostly idle, why not squeeze them all on a small instance. Anyway, I will look into "prefix delegation", that seems to be the magic sauce that will make it work.

1

Let's assume 110 pods need to run in EKS, you need to review how many IPs can a node have. I have the same situation and I use m5.4xlarge that can host 234 max pods/ node (cost effective I can use also with two ec2 instances m5.2xlarge it will give me 116 total pods per nodes)
Other option you can consider multi containers pod to reduce the number of node note: the instance type will depends on your application

profile picture
answered 2 years ago
  • Well, sure, but m5.4xlarge is 50 times the price of t3.small and still 2.5 times the price per pod. Makes sense when your pods actually need the performance but for a bunch of idle pods it seems like a waste. I will look into the solution that Jason_S provided.

  • of course it will depends on what the resources the pods need that will drive your choice. my choice is memory optimized because the pod consumes memory. The number of pods are driven from xlarge 2xlage 4xlarge

0

You can increase the number of available IP addresses for your Amazon EKS nodes by configuring your cluster to assign IP address prefixes to nodes. This approach significantly expands the number of IP addresses that nodes can assign to Pods, thereby overcoming limitations in scaling Kubernetes workloads.

To achieve this, you need to enable the prefix delegation mode in your Amazon EKS cluster. This involves configuring your cluster to assign IP address prefixes to network interfaces, rather than individual secondary IP addresses to nodes.

For detailed steps and guidelines, refer to the official AWS documentation on increasing IP addresses for Amazon EKS nodes: https://docs.aws.amazon.com/eks/latest/userguide/cni-increase-ip-addresses.html

Navya
answered 21 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions