Best Practices for API Gateway - what am I missing?

This is a common architectural challenge, and I understand your frustrations with the trade-offs. Let me break down a few approaches and recommendations:

1. HTTP API Approach (with improvements)

Internet -> CloudFront -> HTTP API -> ALB -> ECS

While the header injection + custom authorizer pattern does feel a bit hacky, it's actually quite secure when implemented correctly. You can improve this by:

• Using a complex, randomly generated header value that's stored in Secrets Manager
• Implementing a custom authorizer that validates multiple headers (not just one)
• Using CloudFront Functions to inject the headers (more cost-effective than Lambda@Edge)

2. REST API Approach (recommended)

Internet -> CloudFront -> REST API -> NLB -> ALB -> ECS

For the REST API approach, while you can't use Managed Prefix Lists directly, you can:

• Use a Resource Policy that restricts access to CloudFront IPs using aws:SourceIp
• Automatically update the Resource Policy using a Lambda function that runs periodically to fetch the latest CloudFront IP ranges from AWS's IP ranges JSON file
• Implement WAF rules on both CloudFront and the REST API

Here's a sample Resource Policy for the REST API:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "execute-api:Invoke",
            "Resource": "arn:aws:execute-api:region:account-id:api-id/*",
            "Condition": {
                "IpAddress": {
                    "aws:SourceIp": [
                        "CLOUDFRONT-IP-RANGES"
                    ]
                }
            }
        }
    ]
}

3. Alternative Approach

Internet -> CloudFront -> ALB -> ECS

Have you considered eliminating the API Gateway altogether? If your use case doesn't specifically require API Gateway features (like request/response transformation, complex authorization, or API key management), you could:

• Put CloudFront directly in front of your ALB
• Use CloudFront Functions or Lambda@Edge for any request/response modifications
• Implement WAF on CloudFront
• Use security groups on the ALB to restrict access to CloudFront IPs

This approach would:

• Reduce latency
• Simplify the architecture
• Reduce costs
• Still provide good security controls

The decision really comes down to your specific requirements:

1. If you need detailed API logging and don't mind the additional network hop, go with the REST API approach
2. If performance is critical and you don't need API Gateway features, consider the alternative approach
3. If you want to minimize infrastructure complexity and can live with limited logging, the HTTP API approach is viable

For logging end-to-end requests:

• Use X-Ray tracing across all components
• Implement request ID propagation through headers
• Use CloudWatch Logs Insights to correlate logs across services
• Consider using AWS OpenSearch (formerly Elasticsearch) for centralized logging

Remember that you can also implement additional security measures at the CloudFront level:

• Custom headers required for origin requests
• Origin Access Identity (OAI) or Origin Access Control (OAC)
• Geo-restrictions
• Custom SSL certificates

You're on the right track with your approach, but there are indeed some nuances to consider when implementing a secure API Gateway solution. Let's address your concerns and provide some best practices:

1. Using a private API: Instead of a public API Gateway, consider using a private API. This allows you to restrict access to your API from within your VPC or through AWS Direct Connect, adding an extra layer of security.

2. VPC Endpoint for API Gateway: Create a VPC Endpoint for API Gateway in your VPC. This allows your resources within the VPC to communicate with API Gateway without going through the public internet.

3. Resource Policy: Use a resource policy on your API to restrict access to only your CloudFront distribution. This can be done by specifying the AWS account ID of your CloudFront distribution in the policy.

4. REST API vs HTTP API: You're correct that REST APIs offer more features, including custom authorizers and better logging capabilities. The trade-off with using a Network Load Balancer is generally worth it for the additional security and customization options.

5. Logging: With a REST API, you can indeed implement custom logging to capture the CloudFront Request ID and other important information. This is a significant advantage over HTTP APIs.

6. WAF Integration: REST APIs allow you to use AWS WAF, which can provide additional protection against common web exploits.

7. API Key and Usage Plans: Consider implementing API keys and usage plans to control and monitor access to your API.

8. Encryption: Ensure that all communication is encrypted in transit using HTTPS.

9. IAM Roles: Use IAM roles and policies to control access between your services, including between API Gateway and your backend services.

10. Monitoring: Implement comprehensive monitoring using Amazon CloudWatch to track API usage, latency, and errors.

While your current setup is close to best practices, consider implementing these additional measures to further enhance security and maintainability. The slight added complexity of using a REST API with a Network Load Balancer is generally justified by the improved security and logging capabilities it provides.

Remember, security is about layers. Each of these practices adds another layer of protection to your architecture, making it more resilient against potential threats.
Sources
Security best practices in Amazon API Gateway - Amazon API Gateway
Best Practices for Designing Amazon API Gateway Private APIs and Private Integration - Best Practices for Designing Amazon API Gateway Private APIs and Private Integration
Private REST APIs in API Gateway - Amazon API Gateway