Why does SignatureDoesNotMatch occur for S3 operations in EKS but not the other environments with the exact same credentials?

I am currently utilizing two-inline authenticated S3 calls in a Python application that is containerized. The application creates 2 S3 clients with the Boto3 library with different credentials. In a single invocation of a Python application, first S3 client is instantiated and used to retrieve a file from Bucket A, the second client is then instantiated and used to list files in Bucket B. Both buckets are in the same region. The Docker container is configured with separate environment variables for each client which are instantiated at run time.

Success Scenario 1 - Local Docker runtime: A works and B works The behavior I am seeing is that when I run the Docker container locally, the call to retrieve a file from Bucket A and the call to list files in Bucket B works fine. Both signatures are calculated properly and both calls succeed.

Failed Scenario 2 - EKS Runtime: A works, B doesn't However, when I run this container in an EKS environment, the call to retrieve a file from Bucket A works, and the call to list files from Bucket B fails with a 403 and a "SignatureDoesNotMatch - The request signature we calculated does not match the signature you provided. Check your key and signing method."

Same code base, same container, same configuration. The only difference is the runtime environment.

I've enabled boto3.set_stream_logger('') and DEBUG level logging and have confirmed that the only difference between the two executions (other than the signatures and dates of course) is the 'User-Agent' header:

• In the failed attempt the User-Agent header is b'Boto3/1.28.85 md/Botocore#1.31.85 ua/2.0 os/linux#5.10.178-162.673.amzn2.x86_64 md/arch#x86_64 lang/python#3.10.11 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.85
• In the successful attempt, the User-Agent header is b'Boto3/1.28.85 md/Botocore#1.31.85 ua/2.0 os/linux#6.4.16-linuxkit md/arch#x86_64 lang/python#3.10.11 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.85',

Also via those details logs, the debug log trail is exactly the same up to the point where the response is retrieved for Bucket B.

• Failed case: 2024-01-03 18:39:25,933 urllib3.connectionpool [DEBUG] https://redacted.amazonaws.com:443 "GET /?list-type=2&prefix=redacted%2Fredacted%2Fredacted%2Fredacted%2F&encoding-type=url HTTP/1.1" 403 None
• Success case: 2024-01-03 18:28:18,129 urllib3.connectionpool [DEBUG] https://redacted.s3.amazonaws.com:443 "GET /?list-type=2&prefix=redacted%2Fredacted%2Fredacted%2Fredacted%2F&encoding-type=url HTTP/1.1" 200 None

I've ruled out the following common occurrences of a signature failure error:

• Not a server time issue: I've ruled out that this is server time sync issue since, in my case, the call to Bucket A using the same approach works fine in both scenarios while the call to Bucket B fails. If the server were out of sync, the call to Bucket A would fail.
• Not a "different credentials" issue: I've confirmed the credentials are the same in both the local and EKS environment. I've also confirmed that we don't have the pesky newline issue with EKS secrets by echoing the variables directly from the container in the pod.
• Not a credential format issue: Both runtimes are utilizing the same credentials (no issues with escape characters needed for +'s or /'s)
• Not a default credentials issue: I've confirmed are no default AWS_XXXXX environment variables in the containers in either runtime environment
• Not a library version / code difference issue: Since the application is containerized and dependencies are built into the image, I've ruled this out. Both applications are running the same version of boto3 (1.28.85) and botocore (1.31.85).

Note: If I separate out the Bucket A and Bucket B processes into two separate containers, the both execute successfully. So this issue seems to have to do with having two sets of inline authenticated S3 clients operating simultaneously.