[ApiGateway] Status code 500 an InternalServerErrorException and Error from cloudfront

We are encountering an intermittent issue within our team's technical infrastructure. Our team operates an API Gateway that connects to Lambda functions, and this entire system is positioned behind a CloudFront distribution.

In recent investigations, our team has delved into approximately 1000 requests. Among these requests, there have been a few (about 10) that have resulted in responses status 500 (Internal Server Error). During our analysis of these errors, we observed specific response headers:

X-Amzn-Errortype: InternalServerErrorException X-Cache: Error from CloudFront

Our CloudFront distribution, which is associated with this API Gateway, is configured with a Cache Policy named "Managed-CachingDisabled." As such, our team has ruled out any issues related to caching. When we scrutinized CloudWatch metrics, we found no instances of timeouts occurring in the Lambda functions. In fact, it seems that these problematic requests never actually reached the Lambda functions. This conclusion was drawn after investigating the request IDs and finding no traces of these requests in the Lambda logs.

Our team is actively working to comprehend the root cause of this issue. It is our collective belief that the problem may lie somewhere in the communication between the API Gateway and CloudFront. However, given that caching is not a factor, we are contemplating whether this might be an internal matter within the AWS infrastructure. This is a question we seek to address as a team to gain a more profound understanding.