- Newest
- Most votes
- Most comments
Hello.
I searched through GitHub issues, but I couldn't find anyone reporting the same bug as you.
It's difficult to determine at this point whether the issue lies with the EC2 OS, so how about trying to update the node group using the previous AMI?
The previous version, according to the release notes below, was "amazon-eks-node-al2023-x86_64-standard-1.34-v20260618," so I recommend testing with that version.
If this version doesn't work, I recommend gradually trying older AMIs, such as "amazon-eks-node-al2023-x86_64-standard-1.34-v20260610," to see if it works.
https://github.com/awslabs/amazon-eks-ami/releases
Based on the available information, this specific kernel panic in the ENA driver's ena_reset_reason_info_free function during secondary ENI detachment does not appear to be documented as a known issue in the current AWS knowledge base. However, there are a few relevant points to consider:
Regarding Your Specific Issue: The kernel panic you're experiencing appears to be a race condition in the ENA driver's device removal path, specifically in the reset-reason cleanup code. The fact that it's intermittent and occurs during routine ENI detachment operations (PCIe hotplug events) suggests a timing-sensitive bug in the driver's shutdown sequence.
Available Information: Amazon Linux 2023 version 2023.10.20260316 includes updates to core packages and kernel components, with some releases being recalled due to availability issues. However, there's no specific mention of ENA driver fixes related to the reset-reason path in the available release notes.
Recommendations:
-
For immediate support: Since this is causing production impact with spontaneous reboots, I recommend adding detailed correspondence to your existing support case through the AWS Support Center (https://console.aws.amazon.com/support/). Include the console output logs, kernel version details, and reproduction patterns you've documented.
-
Kernel version consideration: Given that you mentioned another cluster on the identical kernel performs more ENI detaches without issues, this suggests the race condition may be influenced by additional factors like timing, workload patterns, or specific hardware configurations.
-
Temporary mitigation options:
- Consider testing with kernel 6.1 (which is also available for AL2023) as an interim measure, as it represents an earlier, more mature kernel line
- Review your ENI attachment/detachment frequency and patterns to see if reducing the rate of operations helps avoid triggering the race condition
- Monitor for any AL2023 kernel updates that may address this issue
- GitHub issue: Your approach of opening an issue on the amzn-drivers repository is appropriate for driver-specific bugs and may yield more targeted information from the driver development team.
Since this appears to be a driver-level bug requiring code analysis and potentially a patch, working through your support case and the GitHub issue simultaneously will give you the best path to resolution.
Sources
Amazon Linux 2023 version 2023.10.20260316 release notes - Amazon Linux 2023
Relevant content
asked a year ago
- AWS OFFICIALUpdated 3 years ago
Thank you for the recommendation. We have already pinned our cluster to v20260618 (kernel 6.12.90) and are monitoring, no reboots so far. For reference, AWS's own EKS AMI release notes for v20260625 already acknowledge this issue: "This release contains ENA driver (2.17.0g) in Amazon Linux 2023 AMIs which might cause kernel panic and unexpected node reboot." https://github.com/awslabs/amazon-eks-ami/releases/tag/v20260625 The root cause is therefore the ENA driver 2.17.0g introduced in v20260625 (kernel 6.12.92); v20260618 (kernel 6.12.90) predates it, which is why the pin resolves it. We captured the panic from the EC2 serial console — a NULL-pointer dereference on secondary-ENI detach: ena_reset_reason_info_free -> __ena_shutoff -> ena_remove Kernel panic - not syncing: Oops: Fatal exception
Could you please advise:
Thank you.
If it has been logged as a GitHub issue, I imagine it will be addressed at some point. Since I am not an AWS employee, I do not know the schedule or roadmap for when it might be addressed.
If no restarts are currently occurring, I recommend continuing to use v20260618 for the time being and switching to the latest version once the issue is resolved.