By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

ongoing 502 (Bad Gateway) error

0

I am consistently getting a 502 (Bad Gateway) error on an AWS website which, up until recently, had been running quite reliably for more than three years. I had one episode a month or so ago where I got the 502 for a few hours, and then it went away, not due to anything I did as far as I know. Now I have been getting the 502 consistently for several weeks. The architecture is:

Bitnami WordPress EC2 (t3a.small) with an Elastic IP CloudFront Certificate Manager R53 S3 (only for W3 Total Cache)

(no Application Load Balancers)

I have reviewed the AWS documentation on this error and tried to investigate. Here are some details on my efforts:

In Chrome DevTools when going to the site I see:

failed to load source map: Could not load content for chrome-extension://gighmmpiobklfepjocnamgkkbiglidom/browser-polyfill.js.map: System error: net::ERR_FILE_NOT_FOUND

I tried going to the domain in Safari and still got the 502.

I tried going to the elastic IP directly, which redirects me to mydomain.org and the 502.

The EC2 instance has been consistently running, no alarms, 2/2 checks passed. I stopped and restarted the EC2 instance and continued getting the 502.

I invalidated the cache for the CloudFront distribution associated with mydomain.org, but continued receiving the 502.

I invalidated the cache for the CloudFront distribution associated with W3 Total Cache, but continued receiving the 502.

One thing at I don't understand is when checking SSL certificate at sslshopper.com, it came back telling me that the domain resolves to an IP I don't recognize:

https://www.sslshopper.com/ssl-checker.html#hostname=www.mydomain.org

www.mydomain.org resolves to 12.34.567.8
 	
Server Type: CloudFront
 	
The certificate should be trusted by all major web browsers (all the correct intermediate certificates are installed).
 	
The certificate will expire in 324 days.	
 	
The hostname (www.mydomain.org) is correctly listed in the certificate.

Going to the above IP (12.34.567.8), I get:

403 ERROR
The request could not be satisfied.

Bad request. We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
Generated by cloudfront (CloudFront)
Request ID: QELc0fvzKvqx58Xrw2S6g3KWMAbkHx66hyxbkDne2Y1GOu7wGb6yGg==

I am able to log in to the apache server on the EC2 instance using SSH using the Mac Terminal app

There I confirmed that MySQL is running

test -d /opt/bitnami/mariadb && echo "MariaDB" || echo "MySQL"

shows MySQL

I stopped and restarted apache

I stopped and restarted all services

I ran the bitnami diagnostic tool (sudo /opt/bitnami/bndiagnostic-tool) and got:

 ✓ Mysql: No issues found
    ? Apache: Found possible issues
    ? Resources: Found possible issues
    ✓ Php: No issues found
    ✓ Connectivity: No issues found
 

[Apache]

Found recent error or warning messages in the Apache error log.

[Sun Nov 06 08:50:26.178264 2022] [authz_core:error] [pid 19192:tid 139683803522816] [client 41.216.183.144:58606] AH01630: client denied by server configuration: /opt/bitnami/apps/phpmyadmin/htdocs/scripts
 [Sun Nov 06 14:55:55.162136 2022] [authz_core:error] [pid 18963:tid 139684130838272] [client 129.151.230.90:54786] AH01630: client denied by server configuration: /opt/bitnami/apps/phpmyadmin/htdocs/index.php
 [Tue Nov 08 00:55:34.423247 2022] [authz_core:error] [pid 18962:tid 139683702810368] [client 70.73.26.178:45923] AH01630: client denied by server configuration: /opt/bitnami/apps/phpmyadmin/htdocs/index.php


Please check the following guide to troubleshoot server issues:
 
https://docs.bitnami.com/general/apps/wordpress/troubleshooting/debug-errors-apache/

A high number of incoming requests originate from one or more unique IP addresses. This could indicate a bot attack. The following guide shows how to check for and block suspicious IP addresses.
 
https://docs.bitnami.com/bch/apps/moodle/troubleshooting/deny-connections-bots-apache/

[Resources]

Your instance has little available RAM memory.

 total used free shared buff/cache available Mem: 1968 713 192 83 1062 960 Swap: 0 0 0

You could try to increase your instance's memory. Please check your cloud provider's documentation for more information.

You can also enable swap memory to improve performance.
 
https://docs.bitnami.com/installer/faq/linux-faq/administration/increase-memory-linux/

Following up on some of the info from the bitnami diagnostic tool, I disabled pagespeed by commenting out two lines in /opt/bitnami/apache2/conf/httpd.conf

#Include conf/pagespeed.conf
#Include conf/pagespeed_libraries.conf

That did not seem to help or change anything.

Also in the httpd.conf file I saw this:

ServerName mydomain.org:80

and wondered whether that is correct considering that the website is only accessed via HTTPS. Should that perhaps be:

ServerName mydomain.org:443

I can get into phpMyAdmin by creating an SSH tunnel as described here:

https://docs.bitnami.com/aws/faq/get-started/access-phpmyadmin/

Everything looks normal to me in phpMyAdmin.

I have not tried to get help from bitnami yet because it seems like this is an AWS problem?

Topics
ComputeNetworking & Content Delivery
Tags
Website HostingAmazon EC2Amazon CloudFront
Language
English
mikeh100
asked a year ago932 views
3 Answers
0

In general, 5xx and 4xx errors indicate that the website is running but there is some internal error that is happening. The exception here is when using CloudFront where it cannot contact the origin (your website). But in this case it looks like you've confirmed that everything is running.

My first impression is that the error message in the logs client denied by server configuration is the key here. Apache is getting the request but it is somehow configured to deny that request.

You can probably confirm this by browsing directly to the website and bypassing CloudFront. If that works then there may be something in the Apache config which is explicitly denying CloudFront access. For example, there might be some configuration which looks at the UserAgent header and denies access based on that.

profile pictureAWS
EXPERT
Brettski-AWS
answered a year ago
  • mikeh100
    a year ago

    Thank you, Brettski-AWS, for taking the time to help me!

    I assume I can browse directly to the website and bypass CloudFront by going to the CloudFront origin, e.g.

    ''ec2-12-345-67-89.us-west-2.compute.amazonaws.com''

    When I do that, initially I get

    ''Your connection is not private''

    with a link:

    ''Go to advanced''

    When I click "Go to advanced"

    I see

    ''This server could not prove that it is ec2-12-345-67-89.us-west-2.compute.amazonaws.com; its security certificate is not trusted by your computer's operating system. This may be caused by a misconfiguration or an attacker intercepting your connection.

    Proceed to ec2-12-345-67-89.us-west-2.compute.amazonaws.com (unsafe)''

    When I click the "proceed" link, I get

    ''This site can’t be reached

    Check if there is a typo in www.ec2-12-345-67-89.us-west-2.compute.amazonaws.com.

    DNS_PROBE_FINISHED_NXDOMAIN''

    I'm not sure what else to try beyond that.

  • Brettski-AWS EXPERT
    a year ago

    The first part of your output is all about the certificate on the web server not matching the hostname. For testing purposes that's ok but it does show that the instance is responding to requests. The "site can't be reached" part is interesting because clearly it can be reached. Your webserver is redirecting you to the host that you reached but is prepending www onto the beginning. And that lookup is failing. So check your configuration because I think your webserver is doing a rewrite of the URL, adding www to the beginning and that's causing the web browser to fail because that DNS entry doesn't exist.

0

Once again, thank you Brettski. I really appreciate your efforts on my behalf. I double-checked, and in the DNS in R53, there is a CNAME record with the name www.mydomain.org and the value mydomain.org. The two mydomain.org records (A and AAAA records) have the CloudFront distribution as their value. So the request only goes to CloudFront after the www has been eliminated. So I don't see how the www could be causing a problem for CloudFront.

mikeh100
answered a year ago
  • Brettski-AWS EXPERT
    a year ago

    That's because the redirect is going to www.ec2-12-345-67-89.us-west-2.compute.amazonaws.com - not to www.mydomain.org. Check your Apache config to see if it is picking up the local hostname or doing some sort of reverse DNS lookup. Either way, the web server is trying to redirect you which may not be what you want.

0

I also looked at the free memory, since the bndiagnostic tool mentioned that. I'm not really sure what's considered normal or adequate. I guess it depends on the particular website. Somewhere I read that this is generally only a problem if you have 1GB or less of RAM. I have 2GB.

with pagespeed disabled:

free -m

         total    used    free  shared  buff/cache   available

Mem:     1968     755     166      92    1046         910

Swap:      0       0       0

with pagespeed enabled:

free -m

          total    used    free  shared  buff/cache   available

Mem:       1968     714     121     142    1131         900

Swap:      0       0       0
mikeh100
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content