Unexpected 403 response (Unable to initialize s3 client for bucket myBucket)

Question

I have two roles (ECS execution role and an EC2 instance role) that I have configured to read myBucket. I have confirmed that both roles have read access via [the policy simulator](https://policysim.aws.amazon.com/home/index.jsp). Everything is within a single account.

However, my deploy tasks (from updating my cluster service) fail due to a 403 error, which puzzles me given the above.
```
Unable to initialize s3 client for bucket myBucket, error: AccessDenied: Access Denied status code: 403, request id: , host id: 
```

CloudTrail seems to be suggesting that my EC2 instance role is the one initiating the rejected request, so to double check I opened a shell on the instance and successfully read the bucket contents via aws cli.

I've done a *lot* of role and policy tweaking trying to solve this, but here's what I have for the moment:

EC2 instance role contains:
* AmazonS3ReadOnlyAccess
* AmazonEC2ContainerServiceforEC2Role

ECS execution task role contains:
* AmazonECSTaskExecutionRolePolicy
* Custom s3 read-only policy:
```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::myBucket/*",
                "arn:aws:s3:::myBucket"
            ]
        }
    ]
}
```
S3 bucket policies:
* Public access is blocked
* Bucket is encrypted, but not KMS encrypted (it uses the default SSE-S3)
* Bucket policy:
```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::370058150712:role/ec2InstanceRole"
            },
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::myBucket/*",
                "arn:aws:s3:::myBucket"
            ]
        }
    ]
}
```

Note: Because of the CloudTrail log, earlier today I changed the Principal from my ECS execution role to my EC2 instance role (as seen above) with no effect.

I even tried:
* Opening up a VPC endpoint
* Opening up an access point from my VPC to my bucket

Can anyone see what is wrong? 
And can anyone explain why the deploy task (from updating my cluster service) disagrees with the policy simulator?

Answer

Assuming you have attached correct permissions on relevant role and even S3 bucket allows access, the best way to figure out the issue is to configure S3 server access logs:
[+] https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable-server-access-logging.html

However, in addition to enabling the S3 server access logs, try to debug your workflow and stage at which this "Access Denied" error occur. This will help out in isolating the API action that is getting rejected.

With this API action, check the Request ID and Extended Request ID for the respective API in S3 server access logs. Also verify these areas where action can be denied:
- Identity based policy (permission present and checked)
- Resource based policy (bucket in same account so checked)
- Any permission boundary on IAM role
- VPC endpoint policy
- SCP on account

If nothing helps, gather all the details and contact AWS Support.

Answer

Did you ever figure out the solution to this problem? I am currently running into the same exact issue. The funny thing is that I only see this issue when trying to use the S3 Environment file from EC2 container instance and it works perfectly fine from Fargate.
I even looked into the EC2 container instance ecs-agent settings and the one that may block communication with the internal endpoints is disabled ` ECS_AWSVPC_BLOCK_IMDS=false` so i dont know what else could be causing this. I have my ECS container instance role and the task execution role both setup to be able to access the bucket but it doesnt work.

Answer

I figured it out... for some reason the EC2 container instances require GetBucketLocation in addition to ListBucket, but Fargate doesnt require GetBucketLocation.

Unexpected 403 response (Unable to initialize s3 client for bucket myBucket)

Relevant content