By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

When using aws s3 cp to copy files over to s3, fails due to "Could not connect to the endpoint URL", but inconsistent

0

Hi, I'm using aws cli and boto3 in Linux ubunto 20.4 locally and encountered similar problems. Once in a while, the upload started and I get an upload fail......Could not connect to the endpoint URL......during the uploading

I've been uploading files with various sizes (100s MB to several GB). and followed the self-check in the post

The ping stat is as following

--- s3.amazonaws.com ping statistics ---
491 packets transmitted, 435 received, 11.4053% packet loss, time 491274ms
rtt min/avg/max/mdev = 221.863/234.481/316.978/11.746 ms

However, I was monitoring the internet at the same time. There was no disconnection and my other computer under the same network also seems to be fine. It only happens to AWS.

Is this something related to stability and anything I can check to improve the situation. TIA

Topics
StorageManagement & GovernanceCompute
Tags
Amazon Simple Storage ServiceAWS Command Line InterfaceLinux
Language
English
rePost-User-1377988
asked a year ago697 views
2 Answers
0

Would it be possible for you to share the actual command or code that you executed?
If sometimes it succeeds, but sometimes it fails, it could be a problem around the network.

profile picture
EXPERT
Riku_Kobayashi
answered a year ago
  • rePost-User-1377988
    a year ago

    sure, the actual code is just aws s3 cp <local-file-dir> <s3-dir>

  • Riku_Kobayashi EXPERT
    a year ago

    Thanks for sharing. There did not seem to be anything wrong with the commands themselves. If you add "--debug" to the command options, you can check the debug log and possibly find something that will help you resolve the error.

    aws s3 cp <local-file-dir> <s3-dir> --debug
0

The internet is a big place; there are likely many networks between your computer and AWS. It's possible that there is a problem with an AWS service (S3 in this case) but it's also possible that there is packet loss on a network between you and AWS that is causing the problem. You may not see packet loss to other destinations because the path to there is different; you may also be crossing a network that is rate limiting traffic to or from AWS. It's very difficult to tell because all of those networks are controlled by different entities.

If this is an ongoing issue and it is detrimental to your usage of AWS, please create a support case so that the team can investigate.

profile pictureAWS
EXPERT
Brettski-AWS
answered a year ago
  • rePost-User-1377988
    a year ago

    thx. Is there a way I can pin point down which steps in that whole process just roughly?

  • Brettski-AWS EXPERT
    a year ago

    You might try running a tool like MTR - it provides (somewhat) real-time statistics on the path between you and a destination - think of it as "traceroute on steroids". It's still not perfect because packet loss is not always related to path detection/testing. But it's something.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content