s3 buckets used by ssm in eu-north-1 are not working

Question

I enabled ssm on my instances in eu-north-1, after a while I start seeing access deny cloudtrails coming from this instances where the event name is "HeadObject" and the request parameters are "bucketName": "amazon-ssm-packages-eu-north-1", "key": "active-birdwatcher-fallback", and "Host": "amazon-ssm-packages-eu-north-1.s3.eu-north-1.amazonaws.com"

I followed the recommendations at https://docs.aws.amazon.com/systems-manager/latest/userguide/ssm-agent-technical-details.html of adding a policy to my instance profile to allow s3:GetObject to all ssm buckets, but the problem persists.

I tried to query this public buckets by hand and I was unable to, but if I change the region to us-west-1 I can access the bucket:

```
~ aws s3 cp s3://amazon-ssm-packages-eu-north-1/active-birdwatcher-fallback .
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden
~ aws s3 cp s3://amazon-ssm-packages-us-west-1/active-birdwatcher-fallback .
fatal error: An error occurred (404) when calling the HeadObject operation: Key "active-birdwatcher-fallback" does not exist
```

so it looks like the bucket is not correctly setup by AWS in eu-north-1 or is something else going on?

Ankush_s786 · Answer

For the HEAD action to be used to retrieve metadata from the AWS controlled S3 buckets, the roles used must have appropriate READ access to the object. Relevant read object permission is required but if it does not exist, the error Amazon S3 returns depends on whether you also have the s3:ListBucket permission. The errors can be [1]:
 - If you have the s3:ListBucket permission on the bucket, Amazon S3 returns an HTTP status code 404 ("no such key") error.
 - If you don’t have the s3:ListBucket permission, Amazon S3 returns an HTTP status code 403 ("access denied") error.

[1] https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html

You have mentioned that you already have s3:GetObject permission to all SSM buckets. In order to mitigate the issue, could you please add "s3:ListBucket" permission to instance profile and try again. Please let us know in case the issue still persist.

s3 buckets used by ssm in eu-north-1 are not working

Relevant content