EKS cluster update from 1.27 to 1.28 Worker Node Group doesn't not join the cluster

I've tried to update my EKS cluster with two worker nodes from 1.27 to 1.28. Since it should be updating worker nodes first before the cluster update, one out of two worker nodes (cpu node) could be updated successfully to Kubernetes version 1.28. However, the gpu node could not be update was failed giving the error;

"NodeCreationFailure - Couldn't proceed with upgrade process as new nodes are not joining node group"

During the inspection process I also noticed that previous cluster updates were carried out without updating the worker nodes in first place. The node group versions were kept at version 1.25 until now.

Below are the steps already taken so far according to troubleshoot options provided by AWS post.

1. VPC and Networking Checks:

• IP Address Availability: Confirmed that the Virtual Private Cloud (VPC) has an adequate number of IP addresses available.
• Subnet Inspection: Verified that the subnets linked to the node group have a sufficient number of free IP addresses.
• Security Group Configuration: Ensured that the security groups are set up to permit the necessary traffic between the nodes and the control plane.

2. IAM Roles and Policies Verification:

• Node IAM Role Permissions: Checked and confirmed that the node IAM role includes all required permissions for joining the cluster.
• Policy Attachments: Verified that the AmazonEKSWorkerNodePolicy, AmazonEKS_CNI_Policy, and AmazonEC2ContainerRegistryReadOnly policies are correctly attached to the node role.

3. CloudWatch Logs Examination:

• Error Message Review: Analyzed the CloudWatch logs for any specific error messages that occurred during the node creation process.
• Issue Identification: Discovered an issue within the logs; there's an issue during unmounting process of the EFS volume in the GPU node because it is not empty

4. AWS Support Runbook Utilization:

• Runbook Execution: Implemented the AWSSupport-TroubleshootEKSWorkerNode runbook to conduct a detailed analysis of the EC2 worker nodes and the EKS cluster.

5. Kubernetes Version Compatibility Check:

• Compatibility Assurance: Ensured that the Kubernetes version targeted for the upgrade is compatible with our existing setup, including any custom configurations or extensions in use.

Cluster Check

1. Checking if the cluster Security Group is allowing traffic from the worker node:

• Passed

2. Checking DHCP options of the cluster VPC:

• Passed

3. Checking cluster IAM role arn: <Account> for the required permissions:

• Passed

4. Checking control plane Elastic Network Interfaces(ENIs) in the cluster VPC:

• Passed

6. Checking cluster VPC for required DNS attributes: -Passed

Worker Node Check

1. Checking if the EC2 instance family is supported:

• Passed

2. Checking the worker node network configuration:

• No issue

3. Checking the IAM instance Profile of the worker node:

• Passed

4. Checking the worker node <name> tags:

• Passed

5. Checking the AMI version for EC2 instance <name>:

• [WARNING]: Worker node AMI version 1.25 must be at the same cluster version 1.28 or one minor version behind

6. Checking worker node Elastic Network Interfaces(ENIs) and Private IP addresses to check if CNI is running:

• Passed

7. Checking the outbound SG rules for worker node:

• Passed

8. Checking if the worker node is running in AWS Outposts subnet , Checking basic NACL rules, Checking STS regional endpoint availability, Checking if Instance Metadata http endpoint is enabled on the worker node, Checking if SSM agent is running and reachable on worker node

• Passed

6. Consideration of Force Upgrade: I have carried out the force upgrade method as well a the last option which failed as well.

"Error Upgrading nodegroup Stack: waiter state transitioned to failure"

Is there any other checks I have missed since I still can't update the gpu node. Please Support