Intermittent AccessDenied for s3:CreateBucket using same STS session/AccessKeyId in ECS tasks

Based on the information provided, there are several potential causes for the intermittent AccessDenied errors you're experiencing with s3:CreateBucket operations using the same STS session credentials.

1. Regarding STS session propagation delays or consistency issues: Yes, this could be a factor. When temporary credentials are issued through STS, there can be a brief propagation delay before they're fully recognized across all AWS regions and services. Even though you're operating in us-east-1, S3 is a global service with regional endpoints, which could introduce some eventual consistency behaviors.

2. MFA requirements could be a potential cause: Check if there are any policies that require recent MFA authentication for certain S3 operations. Some policies can be configured to deny access to S3 resources if MFA was authenticated more than a specific time ago (e.g., 30 minutes). If your STS session is longer than this window, operations might start failing until re-authentication.

3. IP address restrictions: If your ECS tasks are running in different subnets or if there are IP-based restrictions in your policies, this could cause intermittent failures. Some policies might deny access based on source IP conditions, which could affect your ECS tasks if they're running across different network configurations.

4. Additional checks to consider:

• Examine the exact timestamps of failures and compare them to the STS session creation time to see if failures occur after a specific duration
• Check if there are any bucket policies on existing buckets that might conflict with your create bucket operations
• Verify if there are any resource-based policies with explicit denies that might be affecting your operations
• Look for any conditions in your policies that might be time-based or dependent on request context
• Check if your IAM role has the s3:GetBucketLocation permission, which is often required alongside s3:CreateBucket

5. Temporary credential expiration: While you mentioned a 1-hour maximum session duration, verify that the actual STS tokens aren't expiring earlier than expected. The error message specifically mentions "no identity-based policy allows the action," which suggests the credentials might be valid but the permissions are somehow different at different times.

If the issue persists after these checks, you might want to implement a retry mechanism with exponential backoff for your S3 bucket creation operations, which seems to be working in your case since the operations eventually succeed.
Sources
KBHS-00719: Error 'AccessDenied' | AWS re:Post
Resolve access denied errors in S3 for IP-restricted IAM roles | AWS re:Post
Amazon S3: S3 Bucket access, but production bucket denied without recent MFA - AWS Identity and Access Management
How do I troubleshoot when User: arn is no authorized to perform Create * or Delete * or Update * on resource(s) because no identity- policy allows the actions | AWS re:Post

Hi There, could you paste the s3 bucket policy here?

In your example, you are using the same bucket name. When you delete a S3 bucket, it's possible you cannot use the same bucket name for a period of time.

When you delete a general purpose bucket, the bucket might not be instantly removed. Instead, Amazon S3 queues the bucket for deletion. Because Amazon S3 is distributed across AWS Regions, the deletion process takes time to fully propagate and achieve consistency throughout the system.

Refer: https://docs.aws.amazon.com/AmazonS3/latest/userguide/delete-bucket.html