AWS API GW Access issue

0

Problem:

I created a public API Gateway with IAM-based authorization and integrated it with DynamoDB. The API Gateway does not have a resource-based policy.

During testing, I can successfully invoke the API from my local machine using the following curl command:

curl "$url" \
	--user "$AWS_ACCESS_KEY_ID":"$AWS_SECRET_ACCESS_KEY" \
	-H "x-amz-security-token: $AWS_SESSION_TOKEN" \
	--aws-sigv4 "aws:amz:us-east-1:execute-api"

However, when I try invoking it from an ECS container, I immediately get a Forbidden error:

Server
< Date: Thu, 01 May 2025 14:21:29 GMT
< Content-Type: application/json
< Content-Length: 23
< Connection: keep-alive
< x-amzn-RequestId: 024a5447-86f0-420c-9882-e48d0987d14d
< x-amzn-ErrorType: ForbiddenException
< x-amz-apigw-id: J5F4iEo_joEFWKg=
< 
* Connection #0 to host jdjpu2kn9d.execute-api.eu-west-1.amazonaws.com left intact
{"message":"Forbidden"}

Details:

  • The ECS task is running in a private subnet with egress network connectivity allowed.
  • WAF is not in use.
  • I am using the same AWS ACCESS, SECRET, and TOKEN credentials for testing both locally and in the ECS container.
  • The task IAM role has the necessary permissions for invoking the API.

What might be causing this issue?

3 Answers
0

The issue you're experiencing is likely related to the IAM authentication and authorization process for your API Gateway. Here are a few potential causes and solutions to consider:

  1. IAM Authentication Configuration: Ensure that you have properly enabled IAM authentication (AWS_IAM) for the specific methods in your API Gateway. If this is not set correctly, the API will treat all incoming requests as anonymous, regardless of the credentials provided.

  2. IAM Permissions: Verify that the IAM role associated with your ECS task has the necessary permissions to invoke the API. The role should have a policy that explicitly allows the "execute-api:Invoke" action on your API resource.

  3. Signature Mismatch: The AWS SigV4 signing process is sensitive to time differences and the exact format of the request. Ensure that the time on your ECS container is accurately synchronized. Also, make sure you're using the latest version of the AWS SDK in your container, as it handles the signing process automatically and correctly.

  4. Network Configuration: Although you mentioned that egress network connectivity is allowed, double-check that your ECS task can reach the API Gateway endpoint. This includes verifying DNS resolution and any necessary network routes or NAT gateways.

  5. Temporary Credentials: If you're using temporary credentials (session token), ensure they haven't expired. Temporary credentials have a limited lifetime, and if they expire, requests will be denied.

  6. Cross-Region Access: If your API Gateway and ECS task are in different AWS regions, make sure you're specifying the correct region in your API invocation code.

  7. API Gateway Stage Variables: If you're using stage variables in your API Gateway setup, ensure they are properly configured and accessible from your ECS environment.

To troubleshoot further:

  • Review the CloudWatch logs for your API Gateway to see if there are any additional error messages or details about the forbidden access.
  • Temporarily enable detailed CloudTrail logging for API Gateway calls to get more information about the authentication process.
  • Consider using AWS X-Ray to trace the request and identify where exactly it's being blocked.

If none of these solutions resolve the issue, you may want to create a minimal test case that reproduces the problem and contact AWS support for further assistance.
Sources
Control access to HTTP APIs with IAM authorization in API Gateway - Amazon API Gateway
Control access for invoking an API - Amazon API Gateway

profile picture
answered 17 days ago
0

There are a few potential causes for the "Forbidden" error when invoking the API from an ECS container:

  1. VPC Endpoint Configuration: If your API Gateway is not publicly accessible and you're using a VPC endpoint to access it from your private subnet, ensure that:

    • You have created a VPC endpoint for API Gateway (com.amazonaws.region.execute-api)
    • The VPC endpoint's security group allows inbound traffic from your ECS container's security group
    • The VPC endpoint is associated with the subnet where your ECS task is running
  2. IAM Role Trust Relationship: Verify that the IAM role associated with your ECS task has a proper trust relationship. It should allow the ECS service to assume the role:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "Service": "ecs-tasks.amazonaws.com"
          },
          "Action": "sts:AssumeRole"
        }
      ]
    }
  3. API Gateway Resource Policy: Although you mentioned that the API Gateway doesn't have a resource-based policy, it might be worth adding one to explicitly allow access from your VPC or ECS tasks:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": "*",
          "Action": "execute-api:Invoke",
          "Resource": "arn:aws:execute-api:region:account-id:api-id/*",
          "Condition": {
            "StringEquals": {
              "aws:SourceVpc": "vpc-123456"
            }
          }
        }
      ]
    }
  4. Network ACLs: Check if there are any Network ACLs associated with your VPC's subnets that might be blocking outbound traffic to the API Gateway.

  5. DNS Resolution: Ensure that DNS resolution and DNS hostnames are enabled for your VPC.

  6. Credentials Handling: Verify that the ECS task is correctly retrieving and using the IAM role credentials. Instead of passing AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN explicitly, try using the AWS SDK's default credential provider chain, which will automatically use the task's IAM role.

  7. API Gateway Stage Variables: If you're using stage variables in your API Gateway setup, ensure they are correctly configured for the stage you're accessing.

  8. Time Synchronization: Ensure that the time on your ECS container is correctly synchronized. AWS API requests are time-sensitive, and significant clock skew can cause authentication failures.

  9. Region Mismatch: Double-check that the region specified in your API invocation matches the region where your API Gateway is deployed. In your curl command, you're using "us-east-1", but the error message shows "eu-west-1".

To troubleshoot further:

  • Enable CloudWatch Logs for your API Gateway and examine the logs for more detailed error messages.
  • Use AWS X-Ray to trace the API requests and identify where they're failing.
  • Try invoking the API from an EC2 instance in the same VPC to isolate whether the issue is specific to ECS or related to your VPC configuration.

If none of these solutions resolve the issue, you may need to examine the network flow logs and API Gateway access logs for more detailed diagnostics.

AWS
answered 17 days ago
0

Hi,

as you mention "public API Gateway with IAM-based authorization " and working with curl using IAM user/role.

Problem : ECS:- 403 Forbidden Error: -

[ Remove hardcode credentials from ECS, ----SDK can handle the permission.

The curl command uses --aws-sigv4 "aws:amz:us-east-1:execute-api", but your API Gateway is in eu-west-1 (as seen in the error URL). This region mismatch invalidates the signature

]

ECS Tasks need an IAM task role with explicit permission to call execute-api:Invoke on your specific API Gateway resource.

step 1. Create an IAM Task Role. Step 2. Assign the Role to ECS Task Step 3. Verify API Gateway Authorization, Test.

Hope it will help!

Best,

step1:- Permission for API Gateway:-

{

"Version": "2012-10-17",

"Statement": [{

"Effect": "Allow",

"Action": "execute-api:Invoke",

"Resource": "arn:aws:execute-api:us-east-1:ACCOUNT_ID:API_ID/STAGE/*"

}]

}

Step 2. in your ECS Task definition

"taskRoleArn": "arn:aws:iam::ACCOUNT_ID:role/ECS-APIGateway-Invoke-Role"

profile picture
answered 17 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions