'fatal error: Could not connect to the endpoint URL ' when using Slurm

0

Configured according to the following settings: Enter image description here

aws s3 cp s3://allen-brain-observatory/visual-coding-neuropixels/raw-data /public_bme/data/LTN/Neuropixels/visual-coding-neuropixels/raw_data/ --recursive

This command work well in local and login nodes of Slurm. However, when I submit it as a job, it returned 'fatal error: Could not connect to the endpoint URL ' Enter image description here

aws s3 cp s3://allen-brain-observatory/visual-coding-neuropixels/raw-data /public_bme/data/LTN/Neuropixels/visual-coding-neuropixels/raw_data/ --recursive --debug

It shows some details but I still can't solve the problem: Traceback (most recent call last): File "awscli/botocore/endpoint.py", line 199, in _do_get_response File "awscli/botocore/endpoint.py", line 271, in _send File "awscli/botocore/httpsession.py", line 477, in send botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://allen-brain-observatory.s3.eu-north-1.amazonaws.com/?list-type=2&prefix=visual-coding-neuropixels%2Fraw-data%2F&encoding-type=url" 2023-06-12 16:36:39,241 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7fd886e0ab90>> 2023-06-12 16:36:39,241 - MainThread - botocore.retries.standard - DEBUG - Max attempts of 3 reached. 2023-06-12 16:36:39,241 - MainThread - botocore.retries.standard - DEBUG - Not retrying request. 2023-06-12 16:36:39,241 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7fd886defe90>> 2023-06-12 16:36:39,241 - MainThread - awscli.customizations.s3.results - DEBUG - Exception caught during command execution: Could not connect to the endpoint URL: "https://allen-brain-observatory.s3.eu-north-1.amazonaws.com/?list-type=2&prefix=visual-coding-neuropixels%2Fraw-data%2F&encoding-type=url" Traceback (most recent call last): File "urllib3/connection.py", line 174, in _new_conn File "urllib3/util/connection.py", line 72, in create_connection File "socket.py", line 962, in getaddrinfo socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "awscli/botocore/httpsession.py", line 448, in send File "urllib3/connectionpool.py", line 787, in urlopen File "urllib3/util/retry.py", line 525, in increment File "urllib3/packages/six.py", line 770, in reraise File "urllib3/connectionpool.py", line 703, in urlopen File "urllib3/connectionpool.py", line 386, in _make_request File "urllib3/connectionpool.py", line 1042, in _validate_conn File "urllib3/connection.py", line 363, in connect File "urllib3/connection.py", line 186, in _new_conn urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPSConnection object at 0x7fd886ca4610>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "awscli/customizations/s3/s3handler.py", line 149, in call File "awscli/customizations/s3/fileinfobuilder.py", line 31, in call File "awscli/customizations/s3/filegenerator.py", line 142, in call File "awscli/customizations/s3/filegenerator.py", line 322, in list_objects File "awscli/customizations/s3/utils.py", line 412, in list_objects File "awscli/botocore/paginate.py", line 252, in iter File "awscli/botocore/paginate.py", line 329, in _make_request File "awscli/botocore/client.py", line 341, in _api_call File "awscli/botocore/client.py", line 683, in _make_api_call File "awscli/botocore/client.py", line 703, in _make_request File "awscli/botocore/endpoint.py", line 101, in make_request File "awscli/botocore/endpoint.py", line 155, in _send_request File "awscli/botocore/endpoint.py", line 199, in _do_get_response File "awscli/botocore/endpoint.py", line 271, in _send File "awscli/botocore/httpsession.py", line 477, in send botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://allen-brain-observatory.s3.eu-north-1.amazonaws.com/?list-type=2&prefix=visual-coding-neuropixels%2Fraw-data%2F&encoding-type=url" fatal error: Could not connect to the endpoint URL: "https://allen-brain-observatory.s3.eu-north-1.amazonaws.com/?list-type=2&prefix=visual-coding-neuropixels%2Fraw-data%2F&encoding-type=url" 2023-06-12 16:36:39,244 - Thread-1 - awscli.customizations.s3.results - DEBUG - Shutdown request received in result processing thread, shutting down result thread.

Does anyone know what I am doing wrong or what else I can try?

1 Answer
0

The AWS CLI can be optionally configured with "--debug".
This may be used to identify the cause of the error.
https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-troubleshooting.html#general-debug

aws s3 cp s3://allen-brain-observatory/visual-coding-neuropixels/raw-data /public_bme/data/LTN/Neuropixels/visual-coding-neuropixels/raw_data/ --recursive --debug
profile picture
EXPERT
answered a year ago
  • Thx. I update the question after using "--debug",but still not know how to fix the connection error.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions