"aws cli cp" command gives inconsistent results

I am using the following command to download files from S3 to my local server "aws s3 cp s3://bucket-name/dir-name/ . --recursive --debug" Sometimes I will get the files downloaded successfully. If I run the same command a few times, sometimes I will get error.

With --debug flag, this is the output: GET / encoding-type=url&list-type=2&prefix=2022-04-18%2F host:glue-bucket-interactions-staging.s3.ca-central-1.amazonaws.com x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20220423T041824Z

host;x-amz-content-sha256;x-amz-date e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2022-04-22 22:18:24,445 - MainThread - botocore.auth - DEBUG - StringToSign: AWS4-HMAC-SHA256 20220423T041824Z 20220423/ca-central-1/s3/aws4_request 184d4f7de08e4ea90234c5717ce78cfd7c31c01cfe854a3d11fa94381f9ab1c3 2022-04-22 22:18:24,445 - MainThread - botocore.auth - DEBUG - Signature: 87d083e547678eebd416afb6691a541b05fa930456c5bb57bb3f9a650cf8c276 2022-04-22 22:18:24,445 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://glue-bucket-interactions-staging.s3.ca-central-1.amazonaws.com/?list-type=2&prefix=2022-04-18%2F&encoding-type=url, headers={'User-Agent': b'aws-cli/2.5.4 Python/3.9.11 Linux/3.10.0-1160.45.1.el7.x86_64 exe/x86_64.rhel.7 prompt/off command/s3.cp', 'X-Amz-Date': b'20220423T041824Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAVLZENK7ICR7W3PXG/20220423/ca-central-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=87d083e547678eebd416afb6691a541b05fa930456c5bb57bb3f9a650cf8c276'}> 2022-04-22 22:18:24,445 - MainThread - botocore.httpsession - DEBUG - Certificate path: /usr/local/aws-cli/v2/2.5.4/dist/awscli/botocore/cacert.pem 2022-04-22 22:18:24,446 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (3): glue-bucket-interactions-staging.s3.ca-central-1.amazonaws.com:443 2022-04-22 22:18:24,448 - MainThread - botocore.endpoint - DEBUG - Exception received when sending HTTP request. Traceback (most recent call last): File "urllib3/connection.py", line 174, in _new_conn File "urllib3/util/connection.py", line 95, in create_connection File "urllib3/util/connection.py", line 85, in create_connection ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "awscli/botocore/httpsession.py", line 358, in send File "urllib3/connectionpool.py", line 785, in urlopen File "urllib3/util/retry.py", line 525, in increment File "urllib3/packages/six.py", line 770, in reraise File "urllib3/connectionpool.py", line 703, in urlopen File "urllib3/connectionpool.py", line 386, in _make_request File "urllib3/connectionpool.py", line 1040, in _validate_conn File "urllib3/connection.py", line 358, in connect File "urllib3/connection.py", line 186, in _new_conn urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPSConnection object at 0x7fad81a9d7c0>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "awscli/botocore/endpoint.py", line 199, in _do_get_response File "awscli/botocore/endpoint.py", line 271, in _send File "awscli/botocore/httpsession.py", line 387, in send botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://glue-bucket-interactions-staging.s3.ca-central-1.amazonaws.com/?list-type=2&prefix=2022-04-18%2F&encoding-type=url" 2022-04-22 22:18:24,448 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7fad81af65e0>> 2022-04-22 22:18:24,448 - MainThread - botocore.retries.standard - DEBUG - Max attempts of 3 reached. 2022-04-22 22:18:24,448 - MainThread - botocore.retries.standard - DEBUG - Not retrying request. 2022-04-22 22:18:24,449 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7fad81af6670>> 2022-04-22 22:18:24,449 - MainThread - awscli.customizations.s3.results - DEBUG - Exception caught during command execution: Could not connect to the endpoint URL: "https://glue-bucket-interactions-staging.s3.ca-central-1.amazonaws.com/?list-type=2&prefix=2022-04-18%2F&encoding-type=url" Traceback (most recent call last): File "urllib3/connection.py", line 174, in _new_conn File "urllib3/util/connection.py", line 95, in create_connection File "urllib3/util/connection.py", line 85, in create_connection ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "awscli/botocore/httpsession.py", line 358, in send File "urllib3/connectionpool.py", line 785, in urlopen File "urllib3/util/retry.py", line 525, in increment File "urllib3/packages/six.py", line 770, in reraise File "urllib3/connectionpool.py", line 703, in urlopen File "urllib3/connectionpool.py", line 386, in _make_request File "urllib3/connectionpool.py", line 1040, in _validate_conn File "urllib3/connection.py", line 358, in connect File "urllib3/connection.py", line 186, in _new_conn urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPSConnection object at 0x7fad81a9d7c0>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "awscli/customizations/s3/s3handler.py", line 149, in call File "awscli/customizations/s3/fileinfobuilder.py", line 31, in call File "awscli/customizations/s3/filegenerator.py", line 142, in call File "awscli/customizations/s3/filegenerator.py", line 322, in list_objects File "awscli/customizations/s3/utils.py", line 412, in list_objects File "awscli/botocore/paginate.py", line 252, in iter File "awscli/botocore/paginate.py", line 329, in _make_request File "awscli/botocore/client.py", line 304, in _api_call File "awscli/botocore/client.py", line 620, in _make_api_call File "awscli/botocore/client.py", line 640, in _make_request File "awscli/botocore/endpoint.py", line 101, in make_request File "awscli/botocore/endpoint.py", line 155, in _send_request File "awscli/botocore/endpoint.py", line 199, in _do_get_response File "awscli/botocore/endpoint.py", line 271, in _send File "awscli/botocore/httpsession.py", line 387, in send botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://glue-bucket-interactions-staging.s3.ca-central-1.amazonaws.com/?list-type=2&prefix=2022-04-18%2F&encoding-type=url" fatal error: Could not connect to the endpoint URL: "https://glue-bucket-interactions-staging.s3.ca-central-1.amazonaws.com/?list-type=2&prefix=2022-04-18%2F&encoding-type=url" 2022-04-22 22:18:24,450 - Thread-1 - awscli.customizations.s3.results - DEBUG - Shutdown request received in result processing thread, shutting down result thread.