Accessing S3 private bucket using HTTPS with an ESP32

Question

I have an ESP32 with some features already included like MQTT communication to AWS IoT (I followed [this example](https://aws.amazon.com/de/blogs/compute/building-an-aws-iot-core-device-using-aws-serverless-and-an-esp32/)). I now want to communicate with S3 using HTTPS requests. I am aware of the FreeRTOS AWS IoT SDK but since I am using the arduino framework within VScode I would rather do this task on my own than integrating the libraries as I do not even know if this would be working. I am also aware, that I could use API Gateway or maybe something like CloudFront. But I would like to understand the basics first and try to implement the communication on my own.

So far I worked my self through the documentations of being able to access my private S3 bucket and download/GET the testfile "firmware.bin", which is inside the bucket. The `http_header` I am trying to send has been built following these AWS documentations:

[sig-v4-header-based-auth][1]

[GetObject][2]

[create-string-to-sign][3]

This is my canonical request:

GET
    /firmware.bin
    
    content-type:application/octet-stream
    host:test_bucket.s3.amazonaws.com
    x-amz-date:20230308T151538Z
    
    host;x-amz-date
    e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

This is my string to sign (hash has been altered):

AWS4-HMAC-SHA25
    20230308T151538Z
    20230308/eu-central-1/s3/aws4_request
    08f718a3ffa27f18fad718eab178f3564f1a1c2f389540cc36f17e329304bc45

This is my final header (content has been adapted regarding AWS ACCESS key and signature):

GET /firmware.bin HTTP/1.1
    Host: test_bucket.s3.eu-central-1.amazonaws.com
    Date: Wed, 08 Mar 2023 15:15:38 GMT
    Authorization: AWS4-HMAC-SHA25 Credentials=AKIAZOKIXQT4EXAMPLE/20230308/eu-central-1/s3/aws4_request,SignedHeaders=host;x-amz-date,Signature=0d4e215b75606f5e8ff45f8b87cac62d7c9a2e58b6cead7fdc34886a7417154f

Here is a short generalized code snippet I am using the [WiFiClientSecure](https://github.com/espressif/arduino-esp32/tree/master/libraries/WiFiClientSecure) library from arduino:

```
String bucket_url = "https://" + bucket_name + "." + service + "." + region + "." + host + "/" + object;
Serial.println("Bucket URL: ");
Serial.println(bucket_url);

WiFiClientSecure ota_client;
ota_client.setCACert(AWS_CERT_CA);
ota_client.connect(bucket_url.c_str(), 443);
ota_client.print(http_header);

// Wait for the response
while (ota_client.connected() && !ota_client.available());

// Read and print the response
while (ota_client.available()) 
 {
          String line = ota_client.readStringUntil('\r');
          Serial.print(line);
 }
```
The certificate I am using is the root certificate, which I generated during the creation of my AWS IoT Thing from the link at the beginning of my post. I do not know if this is the correct certificate to be used to access S3.

The errors I get are:

[ 33426][E][WiFiGeneric.cpp:1438] hostByName(): DNS Failed for https://test_bucket.s3.eu-central-1.amazonaws.com/firmware.bin
    [ 33430][E][WiFiClientSecure.cpp:135] connect(): start_ssl_client: -1

So I believe at this point something with the certificate but also with the URL is wrong. I checked the firmware.bin URL in AWS S3 bucket and it is exactly the same as the one I use for the connection. Now I hope someone can help me with a few things:

1. Is the certificate I am using the correct one - it was generated when I created a "Thing" in AWS IoT? If not how can I generate a proper one to connect to S3?
 2. Does my general signing approach look correct or can you see any issues?
 3. To create hashs, I use my AWS SECRET ACCESS key, is this correct? It is not mentioned in the third link I provided.
 4. Is there something wrong in my general approach to connect to S3 via HTTPS?

Any help would be great!

[1]: https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html
  [2]: https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html
  [3]: https://docs.aws.amazon.com/general/latest/gr/create-signed-request.html#create-string-to-sign

Answer

I just noticed, that I had to adapt the bucket_url and remove the "https://" and the "/firmware.bin" at the end and I finally got a response:

Bucket URL: 
test_bucket.s3.eu-central-1.amazonaws.com
HTTP/1.1 400 Bad Request
x-amz-request-id: 1JTPQC3C4ET8JTXB
x-amz-id-2: qVT0RKD/ScBqX51/TdjtT8B9w1/A+01pkYFalSr1yJ9/jJc4Pan5OV+Q226wVtPQHN/X+qSBjtk=
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Wed, 08 Mar 2023 17:38:02 GMT
Server: AmazonS3
Connection: close

20f
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidArgument</Code><Message>Unsupported Authorization Type</Message><ArgumentName>Authorization</ArgumentName><ArgumentValue>AWS4-HMAC-SHA25 Credentials=AKIAZOKIXQT4DEXAMPLE/20230308/eu-central-1/s3/aws4_request,SignedHeaders=host;x-amz-date,Signature=0cea32460ea4345096d06dc8a35346aadb83d4ceeb1e712cb8185bda197dd</ArgumentValue><RequestId>1JTPQC3C4ET8JTXB</RequestId><HostId>qVT0RKD/ScBqX51/TdU8KlI0O/A+01pkYFalSr1yJ9/jJc4Pan5OV+Q226wVtPQHN/X+qSBjtk=</HostId></Error>
0

What is meant by "Unsupported Authorization"?

Has it maybe something to do with the timestamp? In my code I use GMT but my region is eu-central-1 and we actually have GMT + 1?

Does the error message also indicate, that using my AWS_CERT_CA root certificate is correct?

Any hints are welcome.

Answer

Hi randy. I don't know what's wrong. In general, it's discouraged to use AWS CLI credentials on an IoT device. You already have the X.509 certificate and private key credentials on the device. These should be the only credentials you need. You can use the AWS IoT credential provider to use your X.509 certificate to get temporary SigV4 credentials: https://docs.aws.amazon.com/iot/latest/developerguide/authorizing-direct-aws.html

Most commonly. AWS IoT devices will get and put objects to S3 using either SigV4 credentials obtained that way, or by using pre-signed URLs sent to the device over MQTT: https://aws.amazon.com/blogs/iot/securely-ingesting-large-sized-payloads-from-iot-devices-to-the-aws-cloud/

The AWS IoT embedded C SDK has very lean and portable libraries for [SigV4](https://github.com/aws/aws-iot-device-sdk-embedded-C#aws-sigv4) and [HTTP](https://github.com/aws/aws-iot-device-sdk-embedded-C#corehttp).  You can also find examples of S3 upload and download using HTTP and pre-signed URLs or SigV4 credentials obtained from the IoT credential provider: https://github.com/aws/aws-iot-device-sdk-embedded-C/tree/main/demos/http

I recommend you consult these examples to understand where your implementation is breaking down. However, it may just be easier to use these libraries and raise your level of abstraction.

Answer

So, I have gotten some steps forward, but I am still struggling. At the moment it looks like the signature is somehow invalid. Here my current status:
1. Canonical request (new lines generated by \n):

GET
/firmware.bin

content-type:application/octet-stream
host:esp-data-exchange.s3.eu-central-1.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20230310T113003Z

content-type;host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

2. SHA256-Hash of the canonical request:
b41cef3a93d40fb0be07c818531db0f25fee816c9e49a1bb8fce449471f59558
3. String to sign:
Each line ends with an \n except for the last. This string2sign is used in the next step.

AWS4-HMAC-SHA256
20230310T113003Z
20230310/eu-central-1/s3/aws4_request
b41cef3a93d40fb0be07c818531db0f25fee816c9e49a1bb8fce449471f59558

4. Signature calculation:
Note that "payload" means that the payload is a string, but not that quotation marks are included to the payload! Well, I can not provide my secret key here, so I provide you with the first HMAC-SHA256 hash generated from:
HMAC-SHA256("AWS4" + SECRET_ACCESS_KEY, "20230310") = 43d0ce257da1febfc3e750bb7c6eeab529041a7b69cc420a97039ce7e4e97a87
The next steps towards the final signature are (each generated hash is used as the key in the next line):

HMAC-SHA256(43d0ce257da1febfc3e750bb7c6eeab529041a7b69cc420a97039ce7e4e97a87, "eu-central-1") 
HMAC-SHA256(ed0a58777ae43da3ffab603384945c9a92ed662173746459ecaaafb3b44af906, "s3")
HMAC-SHA256(a71f73bb68dc425032d40df024aaa841441b3634443dec12d5f940cec223d20a, "aws4_request")
HMAC-SHA256(24d4d6bdf0d42a70f4321152c90efb9d62e40bac0942da9c790c744440e7580f, string2sign)

Calculated signature (the result is the same I get with using this online calculator) :
8733e2abe92c25651be2f59bc3e7573884a2a9a07515a97531a98a0b6177dacc
5. Authentication header:
Again, the ACCESS_KEY is not my real one.
AWS4-HMAC-SHA256 Credential=AKIAZOKIXQTEXAMPLE/20230310/eu-central-1/s3/aws4_request,SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date,Signature=39b03577c18a8d9f63bfd89b497252e57ca31365d4b3dd73d367f4047aaed4ef
6. Final HTTP Request Header:

GET /firmware.bin HTTP/1.1
Content-Type: application/octet-stream
Host: esp-data-exchange.s3.eu-central-1.amazonaws.com
Authorization: AWS4-HMAC-SHA256 Credential=AKIAZOKIXQTEXAMPLE/20230310/eu-central-1/s3/aws4_request,SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date,Signature=39b03577c18a8d9f63bfd89b497252e57ca31365d4b3dd73d367f4047aaed4ef
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20230310T113003Z

The response I get is stating the following error:

HTTP/1.1 403 Forbidden
x-amz-request-id: 2XQQKP6D0QHZK98E
x-amz-id-2: Ci5wgxD5+OZsge9+phzObrCUE/zCPGfx2eDIB7kHAwjc/XUFjBefsr4E3E9GaLw9F6RZC3Tz5gs=
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Fri, 10 Mar 2023 11:30:07 GMT
Server: AmazonS3

683
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>AKIAZOKIXQTEXAMPLE</AWSAccessKeyId>
<StringToSign>AWS4-HMAC-SHA256
20230310T113003Z
20230310/eu-central-1/s3/aws4_request
b41cef3a93d40fb0be07c818531db0f25fee816c9e49a1bb8fce449471f59558</StringToSign><SignatureProvided>39b03577c18a8d9f63bfd89b497252e57ca31365d4b3dd73d367f4047aaed4ef</SignatureProvided>
<StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 32 33 30 33 31 30 54 31 31 33 30 30 33 5a 0a 32 30 32 33 30 33 31 30 2f 65 75 2d 63 65 6e 74 72 61 6c 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 62 34 31 63 65 66 33 61 39 33 64 34 30 66 62 30 62 65 30 37 63 38 31 38 35 33 31 64 62 30 66 32 35 66 65 65 38 31 36 63 39 65 34 39 61 31 62 62 38 66 63 65 34 34 39 34 37 
31 66 35 39 35 35 38</StringToSignBytes>
<CanonicalRequest>GET
/firmware.bin

content-type:application/octet-stream
host:esp-data-exchange.s3.eu-central-1.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20230310T113003Z

content-type;host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855</CanonicalRequest>
<CanonicalRequestBytes>47 45 54 0a 2f 66 69 72 6d 77 61 72 65 2e 62 69 6e 0a 0a 63 6f 6e 74 65 6e 74 2d 74 79 70 
65 3a 61 70 70 6c 69 63 61 74 69 6f 6e 2f 6f 63 74 65 74 2d 73 74 72 65 61 6d 0a 68 6f 73 74 3a 65 73 70 2d 64 61 74 61 2d 65 78 63 68 61 6e 67 65 2e 73 33 2e 65 75 2d 63 65 6e 74 72 61 6c 2d 31 2e 6
37b
1 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 32 30 32 33 30 33 31 30 54 31 31 33 30 30 33 5a 0a 0a 63 6f 
6e 74 65 6e 74 2d 74 79 70 65 3b 68 6f 73 74 3b 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3b 78 2d 61 6d 7a 2d 64 61 74 65 0a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35</CanonicalRequestBytes>
<RequestId>2XQQKP6D0QHZK98E</RequestId>
<HostId>Ci5wgxD5+OZsge9+phzObrCUE/zCPGfx2eDIB7kHAwjc/XUFjBefsr4E3E9GaLw9F6RZC3Tz5gs=</HostId></Error>
0

What I noticed in the response are:

The content type of the response is not the same compared to my request.
The canonical request and string2sign seem to be identical between response and request.

Any idea how I could go on? At the moment I am clueless

Accessing S3 private bucket using HTTPS with an ESP32

Relevant content