Questions tagged with Amazon EC2

Content language: English

Sort by most recent

Browse through the questions and answers listed below or filter and sort to narrow down your results.

ongoing 502 (Bad Gateway) error

I am consistently getting a 502 (Bad Gateway) error on an AWS website which, up until recently, had been running quite reliably for more than three years. I had one episode a month or so ago where I got the 502 for a few hours, and then it went away, not due to anything I did as far as I know. Now I have been getting the 502 consistently for several weeks. The architecture is: Bitnami WordPress EC2 (t3a.small) with an Elastic IP CloudFront Certificate Manager R53 S3 (only for W3 Total Cache) (no Application Load Balancers) I have reviewed the AWS documentation on this error and tried to investigate. Here are some details on my efforts: **** In Chrome DevTools when going to the site I see: ```` failed to load source map: Could not load content for chrome-extension://gighmmpiobklfepjocnamgkkbiglidom/browser-polyfill.js.map: System error: net::ERR_FILE_NOT_FOUND ```` *** I tried going to the domain in Safari and still got the 502. **** I tried going to the elastic IP directly, which redirects me to mydomain.org and the 502. **** The EC2 instance has been consistently running, no alarms, 2/2 checks passed. I stopped and restarted the EC2 instance and continued getting the 502. **** I invalidated the cache for the CloudFront distribution associated with mydomain.org, but continued receiving the 502. **** I invalidated the cache for the CloudFront distribution associated with W3 Total Cache, but continued receiving the 502. **** One thing at I don't understand is when checking SSL certificate at sslshopper.com, it came back telling me that the domain resolves to an IP I don't recognize: https://www.sslshopper.com/ssl-checker.html#hostname=www.mydomain.org ```` www.mydomain.org resolves to 12.34.567.8 Server Type: CloudFront The certificate should be trusted by all major web browsers (all the correct intermediate certificates are installed). The certificate will expire in 324 days. The hostname (www.mydomain.org) is correctly listed in the certificate. ```` **** Going to the above IP (12.34.567.8), I get: ```` 403 ERROR The request could not be satisfied. Bad request. We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner. If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation. Generated by cloudfront (CloudFront) Request ID: QELc0fvzKvqx58Xrw2S6g3KWMAbkHx66hyxbkDne2Y1GOu7wGb6yGg== ```` **** I am able to log in to the apache server on the EC2 instance using SSH using the Mac Terminal app There I confirmed that MySQL is running ```` test -d /opt/bitnami/mariadb && echo "MariaDB" || echo "MySQL" ```` shows MySQL I stopped and restarted apache I stopped and restarted all services I ran the bitnami diagnostic tool (sudo /opt/bitnami/bndiagnostic-tool) and got: ```` ✓ Mysql: No issues found ? Apache: Found possible issues ? Resources: Found possible issues ✓ Php: No issues found ✓ Connectivity: No issues found [Apache] Found recent error or warning messages in the Apache error log. [Sun Nov 06 08:50:26.178264 2022] [authz_core:error] [pid 19192:tid 139683803522816] [client 41.216.183.144:58606] AH01630: client denied by server configuration: /opt/bitnami/apps/phpmyadmin/htdocs/scripts [Sun Nov 06 14:55:55.162136 2022] [authz_core:error] [pid 18963:tid 139684130838272] [client 129.151.230.90:54786] AH01630: client denied by server configuration: /opt/bitnami/apps/phpmyadmin/htdocs/index.php [Tue Nov 08 00:55:34.423247 2022] [authz_core:error] [pid 18962:tid 139683702810368] [client 70.73.26.178:45923] AH01630: client denied by server configuration: /opt/bitnami/apps/phpmyadmin/htdocs/index.php Please check the following guide to troubleshoot server issues: https://docs.bitnami.com/general/apps/wordpress/troubleshooting/debug-errors-apache/ A high number of incoming requests originate from one or more unique IP addresses. This could indicate a bot attack. The following guide shows how to check for and block suspicious IP addresses. https://docs.bitnami.com/bch/apps/moodle/troubleshooting/deny-connections-bots-apache/ [Resources] Your instance has little available RAM memory. total used free shared buff/cache available Mem: 1968 713 192 83 1062 960 Swap: 0 0 0 You could try to increase your instance's memory. Please check your cloud provider's documentation for more information. You can also enable swap memory to improve performance. https://docs.bitnami.com/installer/faq/linux-faq/administration/increase-memory-linux/ ```` **** Following up on some of the info from the bitnami diagnostic tool, I disabled pagespeed by commenting out two lines in /opt/bitnami/apache2/conf/httpd.conf ```` #Include conf/pagespeed.conf #Include conf/pagespeed_libraries.conf ```` That did not seem to help or change anything. Also in the httpd.conf file I saw this: ```` ServerName mydomain.org:80 ```` and wondered whether that is correct considering that the website is only accessed via HTTPS. Should that perhaps be: ```` ServerName mydomain.org:443 ```` **** I can get into phpMyAdmin by creating an SSH tunnel as described here: https://docs.bitnami.com/aws/faq/get-started/access-phpmyadmin/ Everything looks normal to me in phpMyAdmin. **** I have not tried to get help from bitnami yet because it seems like this is an AWS problem?
3
answers
0
votes
56
views
asked 23 days ago

Unable to write to EC2 instance running Apache on shared EFS

I have an Auto-scaling group with the following EFS setup in the Launch Template: ``` sudo yum install -y amazon-efs-utils sudo mount -t efs fs-0f13ef1378a09e59c:/ /efs sudo mount -t efs fs-0f13ef1378a09e59c:/html /var/www/html sudo mount -t efs fs-0f13ef1378a09e59c:/test /home/test # Reference: https://stackoverflow.com/questions/57260276/using-same-aws-efs-to-share-multiple-directories ``` I have PHP8.0 and Apache set up this guide: https://gist.github.com/syad9000/dbc855a11b306cb454b283a83fe479f2. This creates the source AMI that I use to generate two EC2 instances in an Auto-scaling group that uses an EFS to sync the /var/home/html/test folder and the /home/test folder. I have Apache set up to serve port 80 to the /var/www/html/test folder. I am using an ALB to redirect requests to the qualified domain name to the target group I created. I can serve files such as /index.html or /index.php fine. PHP code is working in the browser. My problem is that I am trying to create an API that will run a stored shell script based on GET command. For example, I do a GET request to /index.php?build=true. My PHP script is trying to execute the /var/www/html/test/build.sh script. I get an error message stating: ``` <br /> <b>Warning</b>: file_put_contents(/var/www/html/test/.build/error.log): Failed to open stream: Permission denied in <b>/var/www/html/test/_resources/php/functions.php</b> on line <b>11</b><br /> ``` When I log in via the console I can run the script with no error. When I try to run using either the web browser or curl I get the above error. When I log into the console and run as the apache user using: ``` sudo su -s /bin/bash -c '/var/www/html/build.sh' apache ``` Where is the permissions issue originating in the Apache config? The EFS or something else like the ALB?
1
answers
0
votes
60
views
Bryan C
asked 24 days ago

Temporary connectivity issues from Win2019 EC2 Instance to metadata and other endpoints

We experience irregular temporary connectivity issues on our Windows Server 2019 EC2 instances. Theses issues only occur on some instances (2 out of 12 machines) and are not specific to subnet/security groups (both machines are in different subnets/security groups, other machines in the same subnet/security groups do not experience the issues). The issues can be found in various logs: ``` # Cloud Watch Agent Log (Excerpt) 2022-11-03T11:34:17Z E! WriteToCloudWatch failure, err: RequestError: send request failed caused by: Post "https://monitoring.eu-central-1.amazonaws.com/": dial tcp 52.94.138.113:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 2022-11-03T11:35:15Z E! cloudwatch: code: RequestError, message: send request failed, original error: Post "https://monitoring.eu-central-1.amazonaws.com/": dial tcp 52.94.138.199:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 2022-11-03T11:35:15Z W! 4 retries, going to sleep 3.2s before retrying. 2022-11-03T11:35:18Z E! WriteToCloudWatch failure, err: RequestError: send request failed caused by: Post "https://monitoring.eu-central-1.amazonaws.com/": dial tcp 52.94.138.199:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 2022-11-03T11:36:11Z W! [processors.ec2tagger] ec2tagger: Error refreshing EC2 tags, keeping old values : +RequestError: send request failed caused by: Post "https://ec2.eu-central-1.amazonaws.com/": dial tcp 54.239.55.167:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 2022-11-03T11:36:11Z W! [processors.ec2tagger] ec2tagger: Error refreshing EC2 tags, keeping old values : +RequestError: send request failed caused by: Post "https://ec2.eu-central-1.amazonaws.com/": dial tcp 54.239.55.167:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 2022-11-03T11:36:14Z E! cloudwatch: code: RequestError, message: send request failed, original error: Post "https://monitoring.eu-central-1.amazonaws.com/": dial tcp 52.94.136.226:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 2022-11-03T11:36:14Z W! 5 retries, going to sleep 6.4s before retrying. 2022-11-03T11:36:20Z E! WriteToCloudWatch failure, err: RequestError: send request failed caused by: Post "https://monitoring.eu-central-1.amazonaws.com/": dial tcp 52.94.136.226:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 2022-11-03T11:37:15Z E! cloudwatch: code: RequestError, message: send request failed, original error: Post "https://monitoring.eu-central-1.amazonaws.com/": dial tcp 52.94.136.211:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 2022-11-03T11:37:15Z W! 6 retries, going to sleep 1m0s before retrying. 2022-11-03T11:38:14Z E! cloudwatch: code: RequestError, message: send request failed, original error: Post "https://monitoring.eu-central-1.amazonaws.com/": dial tcp 52.94.136.211:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 2022-11-03T11:38:14Z W! 7 retries, going to sleep 1m0s before retrying. 2022-11-03T11:38:15Z E! WriteToCloudWatch failure, err: RequestError: send request failed caused by: Post "https://monitoring.eu-central-1.amazonaws.com/": dial tcp 52.94.136.211:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 2022-11-03T11:39:14Z E! WriteToCloudWatch failure, err: RequestError: send request failed caused by: Post "https://monitoring.eu-central-1.amazonaws.com/": dial tcp 52.94.136.211:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ``` ``` # SSM Agent Log (Excerpt) 2022-11-03 12:22:59 ERROR [HandleAwsError @ awserr.go.49] [ssm-agent-worker] [MessageService] [MDSInteractor] error when calling AWS APIs. error details - GetMessages Error: EC2RoleRequestError: no EC2 instance role found caused by: RequestError: send request failed caused by: Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2022-11-03 12:23:02 ERROR [HandleAwsError @ awserr.go.49] [ssm-agent-worker] [MessageService] [MDSInteractor] error when calling AWS APIs. error details - GetMessages Error: EC2RoleRequestError: no EC2 instance role found caused by: RequestError: send request failed caused by: Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2022-11-03 12:23:05 ERROR [HandleAwsError @ awserr.go.49] [ssm-agent-worker] [MessageService] [MDSInteractor] error when calling AWS APIs. error details - GetMessages Error: EC2RoleRequestError: no EC2 instance role found caused by: RequestError: send request failed caused by: Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2022-11-03 12:23:08 ERROR [HandleAwsError @ awserr.go.49] [ssm-agent-worker] [MessageService] [MDSInteractor] error when calling AWS APIs. error details - GetMessages Error: EC2RoleRequestError: no EC2 instance role found caused by: RequestError: send request failed caused by: Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2022-11-03 12:23:11 ERROR [HandleAwsError @ awserr.go.49] [ssm-agent-worker] [MessageService] [MDSInteractor] error when calling AWS APIs. error details - GetMessages Error: EC2RoleRequestError: no EC2 instance role found caused by: RequestError: send request failed caused by: Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2022-11-03 12:23:11 ERROR [checkStopPolicy @ mdsinteractor.go.391] [ssm-agent-worker] [MessageService] [MDSInteractor] MDSInteractor stopped temporarily due to internal failure. We will retry automatically after 15 minutes 2022-11-03 12:33:05 ERROR [HandleAwsError @ awserr.go.49] [ssm-agent-worker] [HealthCheck] error when calling AWS APIs. error details - RequestError: send request failed caused by: Post "https://ssm.eu-central-1.amazonaws.com/": dial tcp 52.119.188.195:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 2022-11-03 12:33:05 ERROR [HandleAwsError @ awserr.go.49] [ssm-agent-worker] [HealthCheck] error when calling AWS APIs. error details - RequestError: send request failed caused by: Post "https://ssm.eu-central-1.amazonaws.com/": dial tcp 52.119.188.195:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 2022-11-03 12:38:07 ERROR [HandleAwsError @ awserr.go.49] [ssm-agent-worker] [HealthCheck] error when calling AWS APIs. error details - RequestError: send request failed caused by: Post "https://ssm.eu-central-1.amazonaws.com/": dial tcp 52.119.188.195:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 2022-11-03 12:38:07 ERROR [HandleAwsError @ awserr.go.49] [ssm-agent-worker] [HealthCheck] error when calling AWS APIs. error details - RequestError: send request failed caused by: Post "https://ssm.eu-central-1.amazonaws.com/": dial tcp 52.119.188.195:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 2022-11-03 12:38:43 ERROR [HandleAwsError @ awserr.go.49] [ssm-agent-worker] [MessageService] [Association] error when calling AWS APIs. error details - RequestError: send request failed caused by: Post "https://ssm.eu-central-1.amazonaws.com/": dial tcp 52.119.190.128:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ``` We found out about these connectivity issues since our Cloud Watch Monitoring had gaps in some metrics provided by the Cloud Watch Agent. These gap seem to occur randomly and last for about 90 minutes. ![Cloud Watch Metric](/media/postImages/original/IMSCtZIvgmR-2L9HmIqjftbg) I could not find any related problems in the windows event log. Surprisingly, our windows docker workloads on the affected machines seem to work normal. Has anyone experienced similar network connectivity problems? Any suggestions on how to further investigate the root for these issues?
1
answers
0
votes
80
views
asked 24 days ago