By using AWS re:Post, you agree to the Terms of Use
/"aws cli cp" command gives inconsistent results/

"aws cli cp" command gives inconsistent results

0

I am using the following command to download files from S3 to my local server "aws s3 cp s3://bucket-name/dir-name/ . --recursive --debug" Sometimes I will get the files downloaded successfully. If I run the same command a few times, sometimes I will get error.

With --debug flag, this is the output: GET / encoding-type=url&list-type=2&prefix=2022-04-18%2F host:glue-bucket-interactions-staging.s3.ca-central-1.amazonaws.com x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20220423T041824Z

host;x-amz-content-sha256;x-amz-date e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2022-04-22 22:18:24,445 - MainThread - botocore.auth - DEBUG - StringToSign: AWS4-HMAC-SHA256 20220423T041824Z 20220423/ca-central-1/s3/aws4_request 184d4f7de08e4ea90234c5717ce78cfd7c31c01cfe854a3d11fa94381f9ab1c3 2022-04-22 22:18:24,445 - MainThread - botocore.auth - DEBUG - Signature: 87d083e547678eebd416afb6691a541b05fa930456c5bb57bb3f9a650cf8c276 2022-04-22 22:18:24,445 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://glue-bucket-interactions-staging.s3.ca-central-1.amazonaws.com/?list-type=2&prefix=2022-04-18%2F&encoding-type=url, headers={'User-Agent': b'aws-cli/2.5.4 Python/3.9.11 Linux/3.10.0-1160.45.1.el7.x86_64 exe/x86_64.rhel.7 prompt/off command/s3.cp', 'X-Amz-Date': b'20220423T041824Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAVLZENK7ICR7W3PXG/20220423/ca-central-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=87d083e547678eebd416afb6691a541b05fa930456c5bb57bb3f9a650cf8c276'}> 2022-04-22 22:18:24,445 - MainThread - botocore.httpsession - DEBUG - Certificate path: /usr/local/aws-cli/v2/2.5.4/dist/awscli/botocore/cacert.pem 2022-04-22 22:18:24,446 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (3): glue-bucket-interactions-staging.s3.ca-central-1.amazonaws.com:443 2022-04-22 22:18:24,448 - MainThread - botocore.endpoint - DEBUG - Exception received when sending HTTP request. Traceback (most recent call last): File "urllib3/connection.py", line 174, in _new_conn File "urllib3/util/connection.py", line 95, in create_connection File "urllib3/util/connection.py", line 85, in create_connection ConnectionRefusedError: Errno 111 Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "awscli/botocore/httpsession.py", line 358, in send File "urllib3/connectionpool.py", line 785, in urlopen File "urllib3/util/retry.py", line 525, in increment File "urllib3/packages/six.py", line 770, in reraise File "urllib3/connectionpool.py", line 703, in urlopen File "urllib3/connectionpool.py", line 386, in _make_request File "urllib3/connectionpool.py", line 1040, in _validate_conn File "urllib3/connection.py", line 358, in connect File "urllib3/connection.py", line 186, in _new_conn urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPSConnection object at 0x7fad81a9d7c0>: Failed to establish a new connection: Errno 111 Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "awscli/botocore/endpoint.py", line 199, in _do_get_response File "awscli/botocore/endpoint.py", line 271, in _send File "awscli/botocore/httpsession.py", line 387, in send botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://glue-bucket-interactions-staging.s3.ca-central-1.amazonaws.com/?list-type=2&prefix=2022-04-18%2F&encoding-type=url" 2022-04-22 22:18:24,448 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7fad81af65e0>> 2022-04-22 22:18:24,448 - MainThread - botocore.retries.standard - DEBUG - Max attempts of 3 reached. 2022-04-22 22:18:24,448 - MainThread - botocore.retries.standard - DEBUG - Not retrying request. 2022-04-22 22:18:24,449 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7fad81af6670>> 2022-04-22 22:18:24,449 - MainThread - awscli.customizations.s3.results - DEBUG - Exception caught during command execution: Could not connect to the endpoint URL: "https://glue-bucket-interactions-staging.s3.ca-central-1.amazonaws.com/?list-type=2&prefix=2022-04-18%2F&encoding-type=url" Traceback (most recent call last): File "urllib3/connection.py", line 174, in _new_conn File "urllib3/util/connection.py", line 95, in create_connection File "urllib3/util/connection.py", line 85, in create_connection ConnectionRefusedError: Errno 111 Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "awscli/botocore/httpsession.py", line 358, in send File "urllib3/connectionpool.py", line 785, in urlopen File "urllib3/util/retry.py", line 525, in increment File "urllib3/packages/six.py", line 770, in reraise File "urllib3/connectionpool.py", line 703, in urlopen File "urllib3/connectionpool.py", line 386, in _make_request File "urllib3/connectionpool.py", line 1040, in _validate_conn File "urllib3/connection.py", line 358, in connect File "urllib3/connection.py", line 186, in _new_conn urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPSConnection object at 0x7fad81a9d7c0>: Failed to establish a new connection: Errno 111 Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "awscli/customizations/s3/s3handler.py", line 149, in call File "awscli/customizations/s3/fileinfobuilder.py", line 31, in call File "awscli/customizations/s3/filegenerator.py", line 142, in call File "awscli/customizations/s3/filegenerator.py", line 322, in list_objects File "awscli/customizations/s3/utils.py", line 412, in list_objects File "awscli/botocore/paginate.py", line 252, in iter File "awscli/botocore/paginate.py", line 329, in _make_request File "awscli/botocore/client.py", line 304, in _api_call File "awscli/botocore/client.py", line 620, in _make_api_call File "awscli/botocore/client.py", line 640, in _make_request File "awscli/botocore/endpoint.py", line 101, in make_request File "awscli/botocore/endpoint.py", line 155, in _send_request File "awscli/botocore/endpoint.py", line 199, in _do_get_response File "awscli/botocore/endpoint.py", line 271, in _send File "awscli/botocore/httpsession.py", line 387, in send botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://glue-bucket-interactions-staging.s3.ca-central-1.amazonaws.com/?list-type=2&prefix=2022-04-18%2F&encoding-type=url" fatal error: Could not connect to the endpoint URL: "https://glue-bucket-interactions-staging.s3.ca-central-1.amazonaws.com/?list-type=2&prefix=2022-04-18%2F&encoding-type=url" 2022-04-22 22:18:24,450 - Thread-1 - awscli.customizations.s3.results - DEBUG - Shutdown request received in result processing thread, shutting down result thread.

1 Answers
0

Not sure how many requests you have been sending to that bucket but can you check to see if you are hitting the S3 API quota?

Also you seem to have exposed your bucket name in the debugging information.

answered a month ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions