CloudFront 502 Error - Previously working setup with a Google Cloud Run origin suddenly starts failing, experienced across 2 separate google accounts and servers, the common denominator is CloudFront.

Question

I'm experiencing a 502 error with my CloudFront distribution that was previously working fine. This is the **second time** this has happened to me with a Google Cloud Run origin connected to AWS CloudFront. The issue seems to occur randomly without any configuration changes on my end.

**Background:**
- Setup: Custom domain → AWS CloudFront → Google Cloud Run service
- This configuration was working perfectly before
- No changes were made to CloudFront or Cloud Run configurations
- This is the second occurrence of this issue with the same setup

**Diagnostic Results:**

1. **SSL Certificate**: Valid and properly configured
   - Issued by Amazon (Amazon RSA 2048 M03)
   - Valid until 2027
   - Correctly includes domain in CN and Subject Alternative Names

2. **DNS Resolution**: Working correctly
   - Domain resolves to CloudFront distribution
   - Multiple IP addresses returned from CloudFront

3. **Origin Server**: Accessible and responding
   - Direct access to origin returns 200
   - Origin is a Google Cloud Run service
   - Not blocking CloudFront IP ranges (tested with X-Forwarded-For header)

**What I've Tried:**
- Verified SSL certificate validity
- Confirmed DNS resolution
- Tested direct origin access
- Checked for IP blocking (origin accepts CloudFront IPs)

The first time I ran into this issue I could not resolve it after 5 hours of attempts. I ended up moving to a different Google Cloud account with a fresh server, which fixed it. I am now seeing the same issue appear in my test environment within my new GCP account.

**Question**
Since this was working before and the origin server is still accessible, what could cause CloudFront to suddenly start returning 502 errors? Has anyone else experienced this intermittent issue with Google Cloud Run + AWS CloudFront? Are there known issues or specific troubleshooting steps for this combination? I've been driving myself insane with this issue, so any suggestions are appreciated.

re:Post Agent · Answer

A 502 error from CloudFront indicates that the service couldn't establish a connection with your origin server (Google Cloud Run in this case). Since this happened suddenly without configuration changes and has occurred multiple times, there are several potential causes to investigate:

1. **SSL/TLS Certificate Mismatch**: Even though your certificate is valid, there might be a mismatch between the domain name in the certificate and the Origin Domain specified in your CloudFront distribution. The SSL/TLS certificate on your origin must include the domain name that matches either the Origin Domain in your CloudFront distribution or the Host header of the origin request.

2. **SSL/TLS Protocol or Cipher Incompatibility**: CloudFront and Google Cloud Run might be using incompatible SSL/TLS protocols or ciphers. This can happen if Google Cloud Run updates its supported protocols/ciphers without notice.

3. **Origin Connection Timeout**: Your Google Cloud Run service might be experiencing intermittent delays in responding, causing CloudFront to time out while waiting for a response.

4. **Origin Configuration Changes**: Google Cloud Run might have made backend changes that affect how it handles requests from CloudFront, such as changes to request validation or header handling.

5. **Network Path Issues**: There could be intermittent network issues between AWS and Google Cloud infrastructure.

Troubleshooting steps:

1. **Check Origin Protocol Policy**: Ensure your CloudFront distribution is configured with the correct protocol (HTTP or HTTPS) to communicate with your origin.

2. **Test SSL/TLS Connection**: Use OpenSSL to test the connection from an EC2 instance to your Google Cloud Run service to verify SSL/TLS handshake is working properly.

3. **Review CloudFront Logs**: Look for specific error messages like "OriginConnectError" that might provide more details.

4. **Check for Google Cloud Run Updates**: Verify if there have been any recent updates or maintenance to Google Cloud Run that might affect external connections.

5. **Adjust Origin Response Timeout**: Try increasing the origin response timeout in your CloudFront distribution settings.

6. **Test with Different Edge Locations**: Check if the issue occurs consistently across all CloudFront edge locations or only specific ones.

Since this has happened across different Google Cloud accounts, it suggests the issue might be related to how CloudFront interacts with Google Cloud Run specifically, rather than an issue with your particular configuration.  
**Sources**  
[HTTP 502 status code (Bad Gateway) - Amazon CloudFront](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/http-502-bad-gateway.html)  
[502 - Bad Gateway : CloudFront wasn't able to connect to the origin | AWS re:Post](https://repost.aws/questions/QUquWPtuN4TH2DIZ_xb3Ug_w/502-bad-gateway-cloudfront-wasn-t-able-to-connect-to-the-origin)  
[cloudfront cache behavior causes 504? | AWS re:Post](https://repost.aws/questions/QUgH7BcheDS_2vVQzLTYe61g/cloudfront-cache-behavior-causes-504)

CloudFront 502 Error - Previously working setup with a Google Cloud Run origin suddenly starts failing, experienced across 2 separate google accounts and servers, the common denominator is CloudFront.

Add your answer

Relevant content