By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

What conditions are causing CloudFront to return unexpected 503 errors with no request ID?

0

Hi,

I have a pretty standard CloudFront distro (eg. low traffic, no Functions/Lambda@Edge, two origins, one S3 and one ALB), which I have recently seen it behaving in an unexpected manner.

Most (~99.9%) traffic to CloudFront going to the ALB is fine, however, CloudFront is returning HTML 503 errors to some requests, the issues/unexpected behaviours are:

  • A custom 503 error page is configured in the distro, but is not used
  • The default HTML 503 error response which is returned contains no request ID
  • The 503 errors do not appear in the distribution metrics "5xx Error rate breakdown" graphs
  • The 503 errors are not logged in the standard distro logs being sent to S3

An example of the CloudFront response as captured by Sentry:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
</HEAD><BODY>
<H1>503 Service Unavailable ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
<BR clear="all">
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
<BR clear="all">
<HR noshade size="1px">
<PRE>
Generated by cloudfront (CloudFront) HTTP3 Server
Request ID: 
</PRE>
<ADDRESS>
</ADDRESS>
</BODY></HTML>

With all the information above it seems there is an issue with CloudFront, for some unknown reason. So with all this in mind, my questions are:

  • Under what situation should this scenario occur?
  • How can I better handle such scenarios, considering there are no logs for such an event in AWS?

Thanks.

Topics
Networking & Content DeliveryManagement & Governance
Tags
Networking & Content DeliveryAmazon CloudFrontAmazon CloudWatch Logs
Language
English
Unlearn
asked 7 months ago336 views
2 Answers
0

I would check your ALB and target groups for the 503 errors.

I think this will be what’s throwing the errors and causing the 503 with cloudfront as the origin.

Even though you have turned of http3 the alb may still be throwing the error back because there’s something there that doesn’t like http 3

profile picture
EXPERT
Gary Mclean
answered 7 months ago
  • Unlearn
    7 months ago

    Under the circumstances where the ALB and backend services are returning 503's it's easy to trace the error through other parts of the system.

    This is not the case in what I've outlined above. The request was terminated at CloudFront. It never reached the origin. Note the lack of request ID.

0

To follow up, as part of the troubleshooting process I wanted to tweak the CloudFront config to see if I could get the error to change.

Since the only hint of what was going on was Generated by cloudfront (CloudFront) HTTP3 Server. I disabled http3 support for the distro. Thinking potentially, that the next error would identify as a different server type.

The errors stopped. So, I would have to conclude there are some issues with the new HTTP 3 service, as it's very new I would not at all be surprised. Perhaps there are scale or load issues.

Consider this a PSA for anyone experiencing this.

Unlearn
answered 7 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content