S3 connection randomly times out

0

I'm storing and retrieving objects on S3 using AWS SDK for PHP. Most of the time it works fine, but occasionally the following errors pop up in a seemingly random way: "Error retrieving credentials from the instance profile metadata server. (cURL error 28: Connection timed out after 1001 milliseconds (see http://curl.haxx.se/libcurl/c/libcurl-errors.html))" Our connection timeout parameter is set at 30 seconds so I'm trying to find out what else can be going on.

  • This seems more like an issue with the EC2 instance. Can you please provide more details on your architecture and the code snippet where you are accessing the objects in S3 to better understand the issue?

  • We use two EC2 instances and a load balancer. It makes sense that it may be an EC2 problem - I tried to run the same code with the same objects on my local VM, and everything is retrieved perfectly. Here's the (simplified) connection code.

    $S3connectParams = array( 'region' => 'us-east-1', 'version' => '2006-03-01', 'http' => array ( 'timeout => 30, 'connect_timeout' => 30 ) ); $client = new S3Client($S3connectParams); $result = $client->GetObject(array("Bucket" => $ourBucket, "Key" => $S3objectKey));

  • Thank you for providing the details.

julia_g
asked 2 years ago747 views
1 Answer
0

Hello @julias_g,

Considering the above details, I am assuming that there are no hard coded credentials in your application and it is instead getting the necessary credentials from the IAM role attached to the instance.

Please feel free to correct me if I am wrong.

The PHP application first looks for credentials in environment variables and then checks the shared credentials profile. If neither of these are found, it gets the credentials from the instance's IAM role.
[+] https://docs.aws.amazon.com/sdk-for-php/v3/developer-guide/guide_credentials.html#default-credential-chain

To get the credentials from the IAM role attached to the instance, the SDKs are designed to make a query to the instance's metadata service which is a pool of instance related details.
[+] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html

It is important to note that these credentials retrieved from the Instance's IAM role are temporary. If you are using IMDS version 2, the credentials are refreshed every 6 hours by default. I believe your application is facing issues connecting while the credentials in the metadata are being rotated. As a result, the applications fails to reach the metadata service and times out. More details here:
[+] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html

Full disclosure, my expertise with PHP is very limited. However, I dived deep into our documentation to see if I can find anything that might help you. I found the below:\

While using IAM role credentials is the preferred method for providing credentials to an application running on an Amazon EC2 instance, the roundtrip from the application to the instance metadata server on each request can introduce latency. In these situations, you might find that utilizing a caching layer on top of your IAM role credentials can eliminate the introduced latency.

The easiest way to add a cache to your IAM role credentials is to specify a credentials cache using the credentials.cache option in a client's factory method or in a service builder configuration file. The credentials.cache configuration setting should be set to an object that implements Guzzle's Guzzle\Cache\CacheAdapterInterface (see Guzzle cache adapters). This interface provides an abstraction layer over various cache backends, including Doctrine Cache, Zend Framework 2 cache, etc

Every request that uses this cache adapter first checks if the credentials are in the cache. If the credentials are found in the cache, the client then ensures that the credentials are not expired. In the event that cached credentials become expired, the client automatically refreshes the credentials on the next request and populates the cache with the updated credentials.

More details on the same here:
[+] https://docs.aws.amazon.com/aws-sdk-php/v2/guide/credentials.html#environment-credentials

I hope the above information proves useful.

In case you feel I missed out to address something more to your concern, please do not hesitate to let me know.

Take Care and Stay Safe!

profile pictureAWS
SUPPORT ENGINEER
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions