Configured according to the following settings:
aws s3 cp s3://allen-brain-observatory/visual-coding-neuropixels/raw-data /public_bme/data/LTN/Neuropixels/visual-coding-neuropixels/raw_data/ --recursive
This command work well in local and login nodes of Slurm.
However, when I submit it as a job, it returned 'fatal error: Could not connect to the endpoint URL '
aws s3 cp s3://allen-brain-observatory/visual-coding-neuropixels/raw-data /public_bme/data/LTN/Neuropixels/visual-coding-neuropixels/raw_data/ --recursive --debug
It shows some details but I still can't solve the problem:
Traceback (most recent call last):
File "awscli/botocore/endpoint.py", line 199, in _do_get_response
File "awscli/botocore/endpoint.py", line 271, in _send
File "awscli/botocore/httpsession.py", line 477, in send
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://allen-brain-observatory.s3.eu-north-1.amazonaws.com/?list-type=2&prefix=visual-coding-neuropixels%2Fraw-data%2F&encoding-type=url"
2023-06-12 16:36:39,241 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7fd886e0ab90>>
2023-06-12 16:36:39,241 - MainThread - botocore.retries.standard - DEBUG - Max attempts of 3 reached.
2023-06-12 16:36:39,241 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2023-06-12 16:36:39,241 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7fd886defe90>>
2023-06-12 16:36:39,241 - MainThread - awscli.customizations.s3.results - DEBUG - Exception caught during command execution: Could not connect to the endpoint URL: "https://allen-brain-observatory.s3.eu-north-1.amazonaws.com/?list-type=2&prefix=visual-coding-neuropixels%2Fraw-data%2F&encoding-type=url"
Traceback (most recent call last):
File "urllib3/connection.py", line 174, in _new_conn
File "urllib3/util/connection.py", line 72, in create_connection
File "socket.py", line 962, in getaddrinfo
socket.gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "awscli/botocore/httpsession.py", line 448, in send
File "urllib3/connectionpool.py", line 787, in urlopen
File "urllib3/util/retry.py", line 525, in increment
File "urllib3/packages/six.py", line 770, in reraise
File "urllib3/connectionpool.py", line 703, in urlopen
File "urllib3/connectionpool.py", line 386, in _make_request
File "urllib3/connectionpool.py", line 1042, in _validate_conn
File "urllib3/connection.py", line 363, in connect
File "urllib3/connection.py", line 186, in _new_conn
urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPSConnection object at 0x7fd886ca4610>: Failed to establish a new connection: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "awscli/customizations/s3/s3handler.py", line 149, in call
File "awscli/customizations/s3/fileinfobuilder.py", line 31, in call
File "awscli/customizations/s3/filegenerator.py", line 142, in call
File "awscli/customizations/s3/filegenerator.py", line 322, in list_objects
File "awscli/customizations/s3/utils.py", line 412, in list_objects
File "awscli/botocore/paginate.py", line 252, in iter
File "awscli/botocore/paginate.py", line 329, in _make_request
File "awscli/botocore/client.py", line 341, in _api_call
File "awscli/botocore/client.py", line 683, in _make_api_call
File "awscli/botocore/client.py", line 703, in _make_request
File "awscli/botocore/endpoint.py", line 101, in make_request
File "awscli/botocore/endpoint.py", line 155, in _send_request
File "awscli/botocore/endpoint.py", line 199, in _do_get_response
File "awscli/botocore/endpoint.py", line 271, in _send
File "awscli/botocore/httpsession.py", line 477, in send
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://allen-brain-observatory.s3.eu-north-1.amazonaws.com/?list-type=2&prefix=visual-coding-neuropixels%2Fraw-data%2F&encoding-type=url"
fatal error: Could not connect to the endpoint URL: "https://allen-brain-observatory.s3.eu-north-1.amazonaws.com/?list-type=2&prefix=visual-coding-neuropixels%2Fraw-data%2F&encoding-type=url"
2023-06-12 16:36:39,244 - Thread-1 - awscli.customizations.s3.results - DEBUG - Shutdown request received in result processing thread, shutting down result thread.
Does anyone know what I am doing wrong or what else I can try?
Thx. I update the question after using "--debug",but still not know how to fix the connection error.