VPC Interface Endpoints and API Gateway called from Lambdas

Question

I have a work problem, where for security reasons we are moving all Lambdas into VPC, and using PrivateLink for all AWS services, via Interface Endpoints added to the VPC. I am having an issue trying to call a REST API in API Gateway with an Endpoint Interface (private DNS enabled) for the execute-api service.

This is fine for Private APIs, where our VPC is added to API resource policy, and we can call the API with no problems. However we need to call webhooks from one lambda, which may or may not be in AWS. Outside of AWS works fine, but to our unit test webhook (REGIONAL) REST API in the same account fails with a 403 error unless I make it private and attach the endpoint id for API Gateway. The security group on the lambda allows all outgoing traffic, and I can call the webhook from the internet via postman with no issues.

I'm struggling to work out why the 403 error occurs when REGIONAL, and why it needs to be PRIVATE to work. We will be calling random webhook URLs inside and outside AWS, and my understanding was that the endpoint interface was the entry point to Private link, so that all API Gateway calls were not via the public endpoints, but otherwise behaved the same. Does anyone know what I am missing here, I can't seem to work out where I've gone wrong from the docs...

Much appreciated.

Accepted Answer

Hi WearsTheFoxHat,

If you enable "Enable Private DNS Name" when you add an API GW VPC endpoint to your vpc, all REGIONAL and PRIVATE APIs will get routed to the VPC endpoint.

As is stated in [1], "However, you cannot access public APIs from a VPC by using an API Gateway VPC endpoint with private DNS enabled. " (emphasis mine)

You can solve this in two ways:

leave private DNS Name enabled and make sure all your public APIs are accessed from custom domain names that are Edge optimized
disable private DNS name, but that will require that you call your private APIs thru the endpoint setting the Host header explicitly [2] or using a Route 53 alias [3]

Hope this helps!

Best, Alf

[1] Creating a private API in Amazon API Gateway - Create an interface VPC endpoint for API Gateway <code class="code">execute-api</code> - https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-private-apis.html#apigateway-private-api-create-interface-vpc-endpoint

[2] How to invoke a private API - Invoking your private API using endpoint-specific public DNS hostnames - https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-private-api-test-invoke-url.html#apigateway-private-api-public-dns

[3] How to invoke a private API - Accessing your private API using a Route53 alias - https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-private-api-test-invoke-url.html#apigateway-private-api-route53-alias

Answer

Could you please Check the DNS setting on a VPC endpoint:
If the API is invoked from an Amazon VPC that has an interface VPC endpoint, verify that the DNS setting of the interface endpoint is set correctly based on the API type.

Keep in mind the following:

To invoke a Regional API from inside an Amazon VPC, private DNS must be disabled on the interface endpoint. Then, the endpoint's hostname can be resolved by a public DNS. For more information, see Why do I get an HTTP 403 Forbidden error when connecting to my API Gateway APIs from a VPC?
To invoke a private API from inside an Amazon VPC using the API's private DNS name, private DNS must be activated on the interface endpoint. Then, the interface endpoint's hostname can be resolved to the Amazon VPC's local subnet resources. For more information, see How to invoke a private API.

Note: You don't need to set up private DNS if you're invoking the private API using its public DNS name or an Amazon Route 53 alias.

https://aws.amazon.com/premiumsupport/knowledge-center/api-gateway-troubleshoot-403-forbidden/

Answer

To reach a public webhook you need to grant your VPC lambda access to internet.  This is a bit more involved than security group setting. This article might help:

https://aws.amazon.com/premiumsupport/knowledge-center/internet-access-lambda-function/

VPC Interface Endpoints and API Gateway called from Lambdas

Relevant content