Using curl POST request to import file from S3 bucket to Git

Hi I'm new to AWS, and I've been trying to upload a repository to an S3 bucket and use it to import this repository to GitLab, using their documentation. The bucket isn't publicly accessible, and I've created an access point and token for my user. The code suggested is:

>curl --request POST "https://gitlab.com/api/v4/projects/remote-import-s3"
 --header "PRIVATE-TOKEN:**MYTOKEN**" --header 'Content-Type:application/json'
 --data "name":"**NAME**","path":"https://gitlab.com/PATH","region":"**REGION NAME**",
"bucket_name":"**BUCKET NAME**","file_key":"**FILE KEY**","access_key_id":"********","secret_access_key":"********"

I also filled in the details needed, which I removed from the code above. When running this code in cmd I get the error: {"error":"region is missing, bucket_name is missing, file_key is missing, access_key_id is missing, secret_access_key is missing, path is missing"}.

What am I missing? Should I configure the bucket in a certain way? Allow some public access?

Topics

Storage Management & Governance

Tags

Amazon Simple Storage Service AWS Config Storage AWS Account Management

Language

English

Eran

asked 10 months ago673 views

2 Answers

Newest
Most votes
Most comments

Are these answers helpful? Upvote the correct answer to help the community benefit from your knowledge.

Hi, Thanks for helping out!

I've tried:

--data '{  
    "name":"**NAME**",
    "path":"https://gitlab.com/PATH",
    "region":"**REGION NAME**",
    "bucket_name":"**BUCKET NAME**",
    "file_key":"**FILE KEY**",
    "access_key_id":"********",
    "secret_access_key":"********" 
}'

but got an error, then I tried " double quotes and followed the errors until i finally got to run the command:

curl -v --request --url "https://gitlab.com/api/v4/projects/remote-import-s3" --header "PRIVATE-TOKEN: ******" 
--header "Content-Type: application/json" --data '{ "name": "****", "path": "https://gitlab.com/******", 
"region": "eu-north-1", "bucket_name": "*****", "file_key": "******", "access_key_id": "******", 
"secret_access_key": "*********" }'

Now I have this error:

*   Trying 172.65.251.78:443...
* Connected to gitlab.com (172.65.251.78) port 443 (#0)
* schannel: disabled automatic use of client certificate
* ALPN: offers http/1.1
* ALPN: server accepted http/1.1
* using HTTP/1.1
> --url /api/v4/projects/remote-import-s3 HTTP/1.1
> Host: gitlab.com
> User-Agent: curl/8.0.1
> Accept: */*
> PRIVATE-TOKEN: ********
> Content-Type: application/json
> Content-Length: 2
>
* schannel: server closed the connection
< HTTP/1.1 400 Bad Request
< Server: cloudflare
< Date: Tue, 18 Jul 2023 14:20:51 GMT
< Content-Type: text/html
< Content-Length: 155
< Connection: close
< CF-RAY: -
<
<html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<hr><center>cloudflare</center>
</body>
</html>
* Closing connection 0
* schannel: shutting down SSL/TLS connection with gitlab.com port 443
* Closing connection -1
curl: (3) URL using bad/illegal format or missing URL
* Closing connection -1
curl: (3) URL using bad/illegal format or missing URL
* Closing connection -1
curl: (3) URL using bad/illegal format or missing URL
* Hostname gitlab.com was found in DNS cache
*   Trying 172.65.251.78:443...
* Connected to gitlab.com (172.65.251.78) port 443 (#1)
* ALPN: offers http/1.1
* ALPN: server accepted http/1.1
* using HTTP/1.1
> --url /envitechsoftware/envitech_web/maintain-view, HTTP/1.1
> Host: gitlab.com
> User-Agent: curl/8.0.1
> Accept: */*
> PRIVATE-TOKEN: *******
> Content-Type: application/json
> Content-Length: 2
>
* schannel: server closed the connection
< HTTP/1.1 400 Bad Request
< Server: cloudflare
< Date: Tue, 18 Jul 2023 14:20:51 GMT
< Content-Type: text/html
< Content-Length: 155
< Connection: close
< CF-RAY: -
<
<html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<hr><center>cloudflare</center>
</body>
</html>
* Closing connection 1
* schannel: shutting down SSL/TLS connection with gitlab.com port 443
* Closing connection -1
curl: (3) URL using bad/illegal format or missing URL
* Closing connection -1
curl: (3) URL using bad/illegal format or missing URL
* Closing connection -1
curl: (3) URL using bad/illegal format or missing URL
* Closing connection -1
curl: (3) URL using bad/illegal format or missing URL
* Closing connection -1
curl: (3) URL using bad/illegal format or missing URL
* Closing connection -1
curl: (3) URL using bad/illegal format or missing URL
* Closing connection -1
curl: (3) URL using bad/illegal format or missing URL
* Closing connection -1
curl: (3) URL using bad/illegal format or missing URL
* Closing connection -1
curl: (3) URL using bad/illegal format or missing URL
* Could not resolve host: ******
* Closing connection 2
curl: (6) Could not resolve host: *******
curl: (3) unmatched close brace/bracket in URL position 1:
}'
 ^

The 2 rows mentioning "Could not resolve host: *******" refers to the content of "secret_access_key".

I'm sorry it took a while to responding, i was sick and away, I appreciate the help!

Eran

answered 9 months ago

kentrad EXPERT
9 months ago
Any special characters in the Secret Key that is throwing off the shell interpreter?
Eran
9 months ago
no. I managed to run the command through Git Bash instead of CMD and it worked, but it couldn't find the project. are there any special permissions I need to create? I'm a root user, opened a S3 bucket, added an access key, and uploaded the project. am I missing something?
kentrad EXPERT
9 months ago
Not sure at this point what the issue is. But, please, please, do not use AWS root user access keys for this this operation. Create an IAM user with a least-privilege policy for the S3 bucket and use the IAM access keys. Please delete the root user access keys ASAP.

It looks like you are missing the opening and closing brackets {} for --data.

--data '{ 
    "name":"**NAME**",
    "path":"https://gitlab.com/PATH",
    "region":"**REGION NAME**",
    "bucket_name":"**BUCKET NAME**",
    "file_key":"**FILE KEY**",
    "access_key_id":"********",
    "secret_access_key":"********" 
}'

https://gitlab.com/gitlab-org/gitlab/-/merge_requests/77259

EXPERT

kentrad

answered 10 months ago

Eran
9 months ago
Hi! I had to comment as a separate answer because of the code. Please follow below. Thanks!

Relevant content

How to use lambda function to copy a newly uploaded file from one S3 bucket to another S3 bucket
zozz
asked a year ago
Migrate repository from CodeCommit to Gitlab
rePost-User-3897276
asked a year ago
Using CloudFront to upload files to S3 bucket using presign URL and custom domain
Roi
asked 7 months ago
Importing dataset files to encrypted s3 bucket
AWS-User-6477897
asked 2 years ago
How can I use a CloudFormation resource import to create an Amazon S3 notification configuration for Lambda on an existing S3 bucket?
AWS OFFICIALUpdated 2 years ago
My Amazon S3 bucket has default encryption using a custom AWS KMS key. How can I allow users to download from and upload to the bucket?
AWS OFFICIALUpdated 2 years ago
Why do I get Access Denied errors when I use a Lambda function to upload files to an Amazon S3 bucket in another AWS account?
AWS OFFICIALUpdated a year ago
How can I use a Lambda function to copy files from one Amazon S3 bucket to another?
AWS OFFICIALUpdated 6 months ago
Support Automation Workflow (SAW) Runbook: Upload EC2 Rescue log bundle from the target instance to the specified Amazon S3 bucket
EXPERT
Maya Karalic
published a year ago
Using PowerCLI to import and export VMs with VMware Cloud on AWS
EXPERT
Patrick Kremer
published 7 months ago