EMR 6.15 shows IMDS Error on reading file from S3 bucket

Hello,

I have upgraded the EMR from 6.14 to 6.15, and started seeing errors on the existing core node: org.apache.hadoop.fs.s3a.auth.NoAwsCredentialsException: IAMInstanceCredentialsProvider: Failed to load credentials from IMDS. and FileDownloadException: Failed to download file path: s3a://abc.log.parquet, range: 0-123414, partition values: [empty row], isDataPresent: false, eTag: null

On EC2 machines, IMDSv2 is optional. There is no such error on the primary node. I have also tried to get a response using the curl command for IMDSv1 and v2. I got responses for both of the commands. There is a version upgrade for Hadoop (3.3.6) from older (3.3.3) for EMR-6.15. Also, I'm not sure if AWS-SDK java would make any impact on this error.

I'm using EMR on EC2.

I'm looking to get help in debugging and solving the issue.