- Newest
- Most votes
- Most comments
First of all, generally speaking you should not run 110 pods on t2.small. You should seriously consider a different way of launching pods. If you don't need the pod, delete them. Or have some other ways to schedule the jobs.
As to why you encounter the limit is due to the fact that by default each pod get its IPs from ENIs of the node. And there is a limit how many ENIs and IPs the node can have.
One way to work around the issue is to use custom networking and CNI add-on. You can refer here. You should be able to at least get ~110 pods. However, you should seriously consider your scheduling strategy.
Let's assume 110 pods need to run in EKS, you need to review how many IPs can a node have.
I have the same situation and I use m5.4xlarge that can host 234 max pods/ node (cost effective I can use also with two ec2 instances m5.2xlarge it will give me 116 total pods per nodes)
Other option you can consider multi containers pod to reduce the number of node
note: the instance type will depends on your application
Well, sure, but m5.4xlarge is 50 times the price of t3.small and still 2.5 times the price per pod. Makes sense when your pods actually need the performance but for a bunch of idle pods it seems like a waste. I will look into the solution that Jason_S provided.
of course it will depends on what the resources the pods need that will drive your choice. my choice is memory optimized because the pod consumes memory. The number of pods are driven from xlarge 2xlage 4xlarge
You can increase the number of available IP addresses for your Amazon EKS nodes by configuring your cluster to assign IP address prefixes to nodes. This approach significantly expands the number of IP addresses that nodes can assign to Pods, thereby overcoming limitations in scaling Kubernetes workloads.
To achieve this, you need to enable the prefix delegation mode in your Amazon EKS cluster. This involves configuring your cluster to assign IP address prefixes to network interfaces, rather than individual secondary IP addresses to nodes.
For detailed steps and guidelines, refer to the official AWS documentation on increasing IP addresses for Amazon EKS nodes: https://docs.aws.amazon.com/eks/latest/userguide/cni-increase-ip-addresses.html
Relevant content
- asked 3 months ago
- asked a year ago
- asked 2 years ago
AWS OFFICIALUpdated a year ago- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 2 years ago
Not sure if I get the reasons why many pods are bad. A pod is just a process, with minimal overhead. If the executable uses maybe 5 MB RAM and is mostly idle, why not squeeze them all on a small instance. Anyway, I will look into "prefix delegation", that seems to be the magic sauce that will make it work.