Making S3 API calls through and Nginx reverse proxy

Question

It seems the S3 is using a special signing method to authenticate API calls (like PutObject). This information is send on headers, but these headers appear to be getting lost or altered when passed through an nginx reverse proxy. My test code works without the proxy, but when the endpoint URL is supplied, I get the following error:

`api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method`

Here are some example request headers from the Go SDK - the credentials come from an STS call, but they work outside of the proxy. The proxy endpoint is provided to the client via a ResolveEndpoint function:

```
SDK 2024/03/29 20:03:35 DEBUG Request
PUT /archive_24c00b22-1b96-47bf-bb20-9e72bea75c62.gz?x-id=PutObject HTTP/1.1
Host: s3.flare.dev.vaultara.com
User-Agent: aws-sdk-go-v2/1.26.0 os/macos lang/go#1.20.4 md/GOOS#darwin md/GOARCH#arm64 api/s3#1.53.0 ft/s3-transfer
Content-Length: 6162208
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 32dc6d8a-3379-4722-8a72-983eb42b8b5f
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=ASIA5C37OGSDFWFCIEVW/20240330/us-west-2/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-security-token;x-amz-server-side-encryption, Signature=489b96d389658c823e2f3ff26dc3f5facb6625ad8aaef1306506347199bb6a5e
Content-Type: application/octet-stream
Expect: 100-continue
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20240330T000335Z
X-Amz-Security-Token: IQoJb3JpZ2luX2VjEOD//////////wEaCXVzLXdlc3QtMiJHMEUCIQDINYn3NYLW3/eUnzH0p2eyGcqp4ehxl+KjYpvxg2B2HQIgEaLAiBdSVvKlLknC5WU7hGDkXvrqvIbGjmzqRzvEGY4qugII+f//////////ARAFGgw4OTk1MjYwNDY4NTQiDHeSSLAGVaiVyg+r2iqOAgVTVlpBV9LFOYDrWaBp18YKa+6f3TasO+rPIezdA22hhMn+pj9o7ctN25Bpk+rsoxwlHzhYwxf/reSWan3wm1owqFFZ254NQA7099eRq/1U1zsfbH2hMq+x0wwbfYC/SaIBNCih1/QLrew7DCVsCo2LDbFHyJLd4hq50Z3DMMkH7U7YKnjiwK/0CkCh/5o7hACio6DxSCXB8TlLiMt4Iby1DZVa77U8cjMczwcfamQfhtEUkkViDhMqK2b8MymmY6+gpCDJWEKxVjHx5hr0lbCRv9TZ8TrCnV1BJOyMQRtT75Bx3olXggJbUdbywTsavcAGE2ArBuM4rrGTcGOjg6oHTnD50YU+cGPkivfjFTDXrZ2wBjqdAWWOLiddRPdt455h2tlGxkhjyhNpdQ81u0Jvm7nUS5O7J70H5i7ygNriEiJpew6g3YU1WaTLu8TNrkSpeb/cDrDDgbTcvP7Kd4D75BYpDAtiAhuV4vKgwC8tYXubaHMwcUCvDOfhZc9fuiYZNnxN3aZ7b+1KS1+2jtMZSEma3EiYgXlJ8E8WP/XeFy0UKGv7UyP07Zd9y+NTaOYBLN4=
X-Amz-Server-Side-Encryption: aws:kms
```

The proxy config looks like this. I'm assuming that all other headers for the request are being passed through, as `proxy_pass_request_headers` defaults to `on`:

```
    proxy_set_header    X-Real-IP           $remote_addr;
    proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;
    proxy_set_header    CF-Connecting-IP    $proxy_add_x_forwarded_for;
    proxy_set_header    X-Forwarded-Host    $host:$server_port;
    proxy_set_header    X-Forwarded-Proto   $scheme;
    proxy_set_header    Upgrade             $http_upgrade;
    proxy_set_header    Connection "upgrade";

server {
        listen              443 ssl;
        server_name         "s3.dev.example.com";
        location / {
            proxy_pass           "https://s3.us-west-2.amazonaws.com";
        }
        proxy_set_header        Host    s3.us-west-2.amazonaws.com;
    }
```

Any ideas about what could be missing? Could Cloudtrail logs shed some light on what needs to change here?

Thanks in advance for any help.

Answer

The `SignatureDoesNotMatch` error is posibly caused by the Nginx reverse proxy modifying or removing the headers required for AWS Signature Version 4 authentication when forwarding the requests to S3. This causes a mismatch between the signature calculated by the client and the signature expected by S3.

The solution involves either disabling Nginx's header normalization, explicitly passing the required headers through the proxy without modification, or using AWS's proxy support for Signature V4 by setting the `X-Amz-Credential` and `X-Amz-Signature` headers and configuring the proxy to forward them correctly.

Making S3 API calls through and Nginx reverse proxy

Relevant content