EKS Upgrade issue

The issues you're experiencing when migrating between different operating systems for your EKS nodes are related to fundamental differences in how these operating systems manage resources, particularly memory.

Root Cause Analysis:

1. Bottlerocket OOMKilled (Exit Code 137): This is likely due to cgroup differences. Bottlerocket uses cgroup v2 by default, while Amazon Linux 2 uses cgroup v1. Cgroup v2 has different memory accounting and enforcement mechanisms, which can surface hidden memory issues in applications that weren't apparent under cgroup v1. When your container crosses its memory limit under cgroup v2, it gets terminated with OOMKilled (exit 137).

2. Amazon Linux 2023 Error (Exit Code 1): AL2023 also uses cgroup v2 and introduces several other changes:

• A new node initialization process called nodeadm with a YAML configuration schema
• Requires IMDSv2 (Instance Metadata Service Version 2)
• Sets default hop count for managed node groups to 1 (restricting container access to node metadata)
• Requires Amazon VPC CNI add-on version 1.16.2 or later

The exit code 1 suggests a general application failure rather than a memory issue. This could be due to different kernel/seccomp defaults or changes in metadata/IAM access patterns.

3. Instance Type Behavior: The fact that pods run successfully on t3.medium but fail on larger instances like t3.large and t3a.xlarge is counterintuitive but could be related to how memory limits are enforced relative to the total available memory. On smaller instances, your application might be staying within its proportional memory limits.

Potential Solutions:

1. For Bottlerocket OOMKilled issues:

• Increase memory limits for your pods
• Optimize your application's memory usage
• If using Java, add JVM flags like -XX:MaxRAMPercentage=75 -XX:InitialRAMPercentage=50
• For Node.js, set NODE_OPTIONS=--max-old-space-size=<MiB> (about 70-75% of your limit)
• For Python applications, set MALLOC_ARENA_MAX=2 to control glibc memory arenas

2. For AL2023 Exit Code 1 issues:

• Check application logs for "operation not permitted" errors (indicating seccomp/kernel issues)
• Look for AWS SDK credential/IMDS errors
• Consider implementing IRSA (IAM Roles for Service Accounts) if your pods rely on node role access

3. Migration Strategy:

• Consider a blue/green upgrade strategy when moving between these operating systems
• Test thoroughly in a non-production environment first
• Ensure your applications are compatible with cgroup v2 before migrating

When migrating between different operating systems for EKS nodes, it's important to account for these differences in system behavior and resource management.
Sources
EKS Upgrade Issue | AWS re:Post
Upgrade from Amazon Linux 2 to Amazon Linux 2023 - Amazon EKS