Skip to content

Lambda in VPC gets ForbiddenException on PostToConnection Despite Correct IAM & Networking

0

I have a Python Lambda function in a private VPC subnet that is triggered by a DynamoDB Stream. The function's purpose is to broadcast flight data updates to clients connected via an API Gateway WebSocket API.

The PostToConnection call to the API Gateway Management API is consistently failing with the following error: An error occurred (ForbiddenException) when calling the PostToConnection operation: Forbidden

I believe I have ruled out all common causes and am looking for what I might be missing.

Architecture:

  • DynamoDB Table with Streams enabled
  • Lambda function in a private subnet with a NAT Gateway for internet access
  • API Gateway WebSocket API

Troubleshooting Steps Already Taken:

  • IAM Policy: The Lambda's execution role has an inline IAM policy with **"Effect": "Allow" **for the "execute-api:ManageConnections" action. I have triple-checked that the Resource ARN is correct. For diagnostics, I even set the Resource to "*" but the ForbiddenException persisted.
  • VPC Networking: The Lambda is confirmed to be in a private subnet. There is a NAT Gateway in a public subnet, and the private subnet's route table has a 0.0.0.0/0 route pointing to the NAT Gateway.
  • Internet Connectivity Test: I added temporary code to the Lambda to make an outbound HTTPS request to https://google.com. This test was successful, proving that the Lambda has internet access and DNS is working correctly.
  • Security Group: The security group attached to the Lambda has an outbound rule allowing All traffic to 0.0.0.0/0.
  • Other Checks: I have confirmed that there is no Permissions Boundary set on the IAM role. I have also confirmed that WebSocket APIs do not use resource policies.

Given that networking appears to be correct and a wildcard IAM policy still results in a ForbiddenException, my final assumption is that a Service Control Policy (SCP) is blocking this action at the AWS Organization level.

Is there any other possible cause for this error that I have overlooked?

4 Answers
0
  1. Use the Correct Endpoint URL for PostToConnection The most common cause of ForbiddenException here is that the Lambda is calling PostToConnection using the wrong endpoint. You need to use the WebSocket’s callback URL, not the default API Gateway endpoint or region endpoint.

Make sure your boto3 client is initialized like this:

import boto3

apigw_client = boto3.client(
    'apigatewaymanagementapi',
    endpoint_url="https://{api-id}.execute-api.{region}.amazonaws.com/{stage}"  # Replace with your actual values
)
If you don’t use this format, you’ll get 403 Forbidden even with the right IAM permissions.
  1. Connection ID Might Be Expired or Closed Even with the correct endpoint, you'll get a ForbiddenException if the connection ID is no longer valid (the client disconnected or timed out).

To confirm this, catch the GoneException

try:
    apigw_client.post_to_connection(
        ConnectionId=conn_id,
        Data=b'Your message here'
    )
except apigw_client.exceptions.GoneException:
    print(f"Client {conn_id} disconnected.")
except apigw_client.exceptions.ForbiddenException:
    print("ForbiddenException: Check if you're using the correct endpoint or expired connection ID.")
  1. Double-Check IAM Policy Format Even if your IAM policy includes execute-api:ManageConnections, the Resource ARN must match the exact format: { "Effect": "Allow", "Action": "execute-api:ManageConnections", "Resource": "arn:aws:execute-api:{region}:{account}:{api-id}/{stage}/POST/@connections/*" } If you’re using wildcards or broader ARNs, make sure they align with what your WebSocket API actually uses.

  2. Check for SCP Restrictions (if using AWS Organizations) If you're part of an AWS Org, Service Control Policies (SCPs) might be blocking this action , even if IAM allows it. -->Ask your Org admin to check if there's an SCP that denies execute-api:ManageConnections -->You can test this using the IAM Policy Simulator

  3. VPC + NAT Setup Seems Good (but Confirm DNS too) You've tested outbound HTTPS to Google, so internet access is likely fine. Just confirm: --> Your VPC subnets have DNS resolution enabled --> You're not using VPC endpoints that restrict outbound routes for some services

answered a year ago

  • Thanks again for all your help with this.

    Unfortunately, I created a brand new, self-contained IAM role and policy exactly as you described, but I'm still getting the same ForbiddenException.

    I've now tried every suggestion. Is there anything else I can share, like specific (anonymized) configurations or logs, that would help you understand my situation better?

0

Thanks for confirming the IAM setup, VPC/NAT, and DNS, that narrows things down significantly.

Here's one last critical point to double-check: Are you 100% sure that the endpoint_url for apigatewaymanagementapi matches your actual deployed WebSocket stage URL?

Even one small mismatch (e.g., wrong region, stage name, or missing HTTPS schema) will always result in a ForbiddenException, no matter how perfect your IAM policy is.

Please verify this:

import boto3

apigw_client = boto3.client(
    'apigatewaymanagementapi',
    endpoint_url="https://<your-api-id>.execute-api.<region>.amazonaws.com/<stage>"
)

Tip: You can get the exact WebSocket endpoint from the API Gateway console → WebSocket API → Stages tab → copy the Invoke URL.

And additionally: If you're still getting ForbiddenException, let’s rule out this very specific scenario:

** SCP/Org restriction simulator test:** Use [AWS Policy Simulator]and test:

  •     Action: execute-api:ManageConnections
    
  •     Resource: "arn:aws:execute-api:<region>:<account-id>:<api-id>/<stage>/POST/@connections/*"
    
  •     Use your Lambda role’s full ARN and check if the result is “Allowed”.
    

If it's not allowed, that confirms an SCP or permission boundary is silently blocking the call.

Final suggestion: If all the above checks are green and it still fails, try calling get_connection() before post_to_connection() to test if the connection is still active:

try:
    apigw_client.get_connection(ConnectionId=conn_id)
    apigw_client.post_to_connection(
        ConnectionId=conn_id,
        Data=b'Your message here'
    )
except apigw_client.exceptions.GoneException:
    print(f"Client {conn_id} disconnected.")
except apigw_client.exceptions.ForbiddenException as e:
    print("Still hitting Forbidden — let’s dig deeper.")

Happy to take a look at an anonymized config snippet if you want to share! You're very close, just one detail away.

answered a year ago

  • Hi Manvitha,

    Thank you, I'd appreciate you taking a look. Here is an anonymized summary of my configuration.

    1. Final IAM Policy This is the full, self-contained policy attached to the Lambda's execution role. All other AWS-managed policies were detached for this test.

    { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "dynamodb:Scan", "Resource": "arn:aws:dynamodb:us-east-1:123456789012:table/Your-Connections-Table-Name" }, { "Effect": "Allow", "Action": "execute-api:ManageConnections", "Resource": "arn:aws:execute-api:us-east-1:123456789012:a1b2c3d4e5/prod/POST/@connections/*" }, { "Effect": "Allow", "Action": "dynamodb:Scan", "Resource": "arn:aws:dynamodb:us-east-1:123456789012:table/Your-Flights-Table-Name" } ] }

    Lamdba environment variable: WEBSOCKET_ENDPOINT: https://a1b2c3d4e5.execute-api.us-east-1.amazonaws.com/prod/

    VPC ID: <vpc-097d7e9e525729d37> Lambda Subnet ID: <subnet-0434c227d6cc70d5e> (This is a private subnet) Route Table: The private subnet's route table has a 0.0.0.0/0 route targeted to a NAT Gateway. Security Group: The Lambda's security group (<sg-04f594e9cbb186ad3>) has an outbound rule allowing All traffic to 0.0.0.0/0.

    Internet Test: A test call from the Lambda to https://google.com succeeds every time.

    Despite this configuration, the get_connection and post_to_connection calls still result in a ForbiddenException. Thank you for any insight you can provide.

0

Root Cause #1: SCP Blocking ManageConnections Even with a perfect IAM policy, if your AWS Org enforces Service Control Policies (SCPs) and doesn’t explicitly allow execute-api:ManageConnections, you'll still hit a ForbiddenException. Fix:** Ask your Org admin to verify if the SCPs attached to your account or OU are allowing this action. You can also confirm via the IAM Policy Simulator:

  • Role ARN: Your Lambda execution role
  • Action: execute-api:ManageConnections
  • Resource: arn:aws:execute-api:us-east-1:123456789012:a1b2c3d4e5/prod/POST/@connections/* If the simulator returns “Denied”, that’s your blocker — not IAM, not VPC.

Root Cause #2: Stale or Misaligned WebSocket Endpoint Even if your WEBSOCKET_ENDPOINT looks right, a minor mismatch (wrong stage, old deploy, typo in API ID) will cause 403 Forbidden every time. Fix:

  • Go to API Gateway → WebSocket API → Stages
  • Select the correct stage (e.g., prod) → click “Invoke URL”
  • Copy the exact endpoint (with region + stage) and update your Lambda code/env variable
  • Re-deploy the stage just to avoid any internal drift

Additional: TLS Cert or VPC DNS Issue (Rare) If you’re in a private subnet and using custom DNS, TLS resolution may silently fail. In your Lambda, test explicitly:

import boto3
import urllib3
urllib3.disable_warnings()

apigw_client = boto3.client(
    'apigatewaymanagementapi',
    endpoint_url=os.environ['WEBSOCKET_ENDPOINT'],
    verify=True  # Fail loudly if TLS cert is invalid
)

Try setting verify=False only for testing; if it works, then it's likely a TLS or DNS trust issue. But never disable verification in production.

answered a year ago

  • Hi, thanks for replying.

    After everything I have tried, nothing is working still. For the simulator I get allowed on the resource ARN and the lamdba code seems to be correct. I will try reach out to AWS to see if they can fix my problems.

0

Based on your detailed troubleshooting, there are a few additional considerations that might explain the ForbiddenException you're encountering.

First, it's important to understand that Lambda functions never actually run in your VPC. They always run in the Lambda service VPC, and when you configure VPC access, AWS creates an Elastic Network Interface (ENI) in your VPC and routes all the Lambda's network traffic through that ENI.

When your Lambda function needs to send messages via WebSocket to clients using the API Gateway Management API's PostToConnection operation, it needs to communicate with API Gateway's public @connections URL. While you've confirmed internet access through your NAT Gateway setup, there are some specific aspects to check:

  1. SigV4 Signing: When making requests to API Gateway endpoints, especially from within a VPC, requests often need to be properly signed with AWS Signature Version 4 (SigV4). The "ForbiddenException" could indicate that your request is not being properly authenticated. Ensure your Lambda function is using the AWS SDK correctly to sign the request.

  2. API Gateway Endpoint Configuration: If you're using a private API Gateway endpoint, verify that the endpoint policy allows access from your Lambda function. Even with the correct IAM permissions, endpoint policies can restrict access.

  3. Connection URL Format: Ensure you're using the correct format for the connection URL. It should typically be in the format: https://{api-id}.execute-api.{region}.amazonaws.com/{stage}/@connections/{connection-id}

  4. Connection ID Validity: Verify that the connection IDs you're trying to post to are still valid. WebSocket connections might have been closed or timed out.

  5. Cross-Account Access: If your API Gateway and Lambda are in different AWS accounts, additional permissions might be required.

  6. Service Control Policies (SCPs): As you suspected, SCPs at the organization level could be restricting this action. This is worth investigating with your organization's administrators.

  7. API Gateway Stage Variables: If you're using stage variables in your API Gateway configuration, ensure they're correctly set up and accessible.

If you've truly exhausted all these possibilities, I would recommend enabling detailed CloudWatch Logs for both your Lambda function and API Gateway to capture more specific error information, which might provide additional clues about the exact cause of the ForbiddenException.
Sources
About API Gateway Websocket and Lambda Invoke in VPC | AWS re:Post
Error Invoking Private REST API from Lambda Function | AWS re:Post
Connecting Lambda inside a VPC to resources outside a VPC | AWS re:Post

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.