Pods are not spinning due to exceeding the node's file descriptor limit

From my understanding, you're encountering issues related to the file descriptor and thread exhaustion, which is commonly associated with resource limits at the system level, especially for services like Zookeeper that handle numerous network connections and I/O operations.

Please follow the below steps to troubleshoot and hopefully resolve the issue:

1. Determine the Number of Threads and Open File Descriptors

• Check Threads for Each Process: You can use the following command inside your Zookeeper pod to determine how many threads each process is running:

  ps -eLf | grep java

  This will show you the number of threads each process is using.

• Check Open File Descriptors: To find out how many files are open by a process, you can use:

  lsof -p <process_id> | wc -l

  Or directly check the number of open files by using:

  ls /proc/<process_id>/fd | wc -l

  Replace <process_id> with the Zookeeper process ID (which you can get from ps).

2. Zookeeper Configuration for File Descriptors

Zookeeper requires a high number of file descriptors and threads due to its connection handling. You can adjust the Zookeeper configuration:

• Increase Zookeeper file descriptor limit: Modify the zoo.cfg or the environment variables in the Zookeeper pod's configuration to explicitly set the ulimit values. Add this to the pod spec or Dockerfile to make sure the ulimit is set higher within the container runtime.

  ulimit -n 65536  # For file descriptors
  ulimit -u 4096   # For max number of processes/threads

3. Check Java and Zookeeper Tuning Parameters

• Java Heap Size: Check the JVM options for the Zookeeper container to ensure that you're not running out of memory, which might also prevent the creation of new threads. If you're using the default heap size, it might be too low. Consider increasing it by setting the appropriate -Xmx and -Xms JVM parameters in the pod spec or container runtime.
• Zookeeper thread limits: Ensure that Zookeeper’s own thread pools (like NIOWorkers) have sufficient thread capacity. You can adjust the configuration for Zookeeper’s worker threads in the zoo.cfg:

  workerThreads=128  # Increase this number based on your load

4. Node Resource Monitoring

Since you observed that the issue worsens when two pods are running on the same node, it might indicate that you’re hitting resource limits (e.g., CPU, memory, or file descriptors) at the node level. Some actions you can take:

• Node resource pressure: Use kubectl top node or CloudWatch metrics to check if the nodes are running low on resources (CPU, memory).
• Pod resource limits: Make sure your Zookeeper pods have appropriate resource requests and limits set in their Kubernetes manifests. If the pods are underspecified, they could be starved of resources when running on the same node.

5. Separate Pods on Different Nodes

Since the issue decreases when the pods run on separate nodes, consider enforcing pod anti-affinity to ensure that no two Zookeeper pods run on the same node. You can achieve this with a PodAntiAffinity rule in the Zookeeper deployment:

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchExpressions:
        - key: app
          operator: In
          values:
          - zookeeper
      topologyKey: "kubernetes.io/hostname"

6. Investigate Node-Level File Descriptor Limits

The ulimit -n 1048576 suggests a high file descriptor limit, but the second command (cat /proc/sys/fs/file-nr) shows that the system currently has a total of around 3 million maximum file descriptors. Ensure that the nodes themselves are not hitting file descriptor exhaustion:

• Increase system-wide file descriptor limit: On the EKS nodes, check or increase the file descriptor limit by updating /etc/sysctl.conf:

  fs.file-max = 10000000

7. Consider JVM Thread Pool Exhaustion

Java applications like Zookeeper rely on thread pools that can exhaust if the workload is too high. If you suspect thread exhaustion, you can:

• Enable GC and thread monitoring: Use Java options like -XX:+PrintGCDetails and -XX:+PrintConcurrentLocks to monitor thread contention and garbage collection.
• Tune thread pools: Zookeeper’s internal thread pools, such as the NIO worker threads, may need tuning for your load. You can adjust these in the zoo.cfg or Zookeeper’s advanced settings.

Next Steps:

1. Investigate thread and file descriptor usage.
2. Tune Zookeeper and JVM parameters.
3. Ensure pods are not contending for resources on the same node.
4. Increase node-level file descriptor limits if necessary.

I hope these steps help you resolve the issue

The issue you're experiencing with Zookeeper pods not spinning up due to exceeding the node's file descriptor limit is a complex problem that involves several factors. Let's break down the situation and provide some suggestions:

1. File descriptor limits: While the node-level limit (1048576) seems high, it's important to note that this limit is shared across all processes on the node. Each pod and its containers also have their own limits, which might be lower than the node-level limit.

2. Resource allocation: Since you're using Karpenter for node management, ensure that you've properly configured resource requests and limits for your Zookeeper pods. This helps Karpenter make better decisions about pod placement and prevents resource contention.

3. Pod Security Policies: Check if there are any Pod Security Policies in place that might be restricting the file descriptor limits for your pods. You may need to adjust these policies to allow higher limits for your Zookeeper pods.

4. Container-level limits: You can set container-level file descriptor limits using the ulimit command in your Dockerfile or by specifying them in your pod specification.

5. Monitoring and metrics: Implement monitoring for file descriptor usage at both the node and pod level. This will help you identify when you're approaching limits and allow you to take proactive measures.

6. Pod anti-affinity: To reduce the likelihood of multiple Zookeeper pods running on the same node, consider implementing pod anti-affinity rules in your deployment specifications.

7. Kernel parameters: You might need to adjust kernel parameters related to file descriptors, such as fs.file-max and fs.nr_open, on your worker nodes.

8. Java-specific settings: For Java applications like Zookeeper, you may need to adjust JVM settings related to thread creation and management.

To resolve the issue:

1. Increase the file descriptor limits for your Zookeeper containers by adding the following to your pod spec:

securityContext:
sysctls:
- name: fs.file-max
value: "65536"

2. Implement pod anti-affinity to spread Zookeeper pods across different nodes:

affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- zookeeper
topologyKey: "kubernetes.io/hostname"

3. Monitor file descriptor usage and adjust limits as necessary.

4. Review and optimize your Java application's thread usage to reduce the number of open file descriptors.

By implementing these suggestions, you should be able to mitigate the file descriptor limit issue and improve the stability of your Zookeeper pods on EKS.

Sources
Pod security - Architecting for HIPAA Security and Compliance on Amazon EKS
Pod Security - Amazon EKS