How do I troubleshoot application accessibility and environment connectivity issues in Elastic Beanstalk?

Resolution

Run diagnostics

Prerequisites:

• Use SSH or AWS Systems Manager Session Manager to access the Amazon Elastic Compute Cloud (Amazon EC2) instances.
• Turn on enhanced health reporting.
• Note your environment type and platform.

Run the Systems Manager runbook

Prerequisite: Make sure that your AWS Identity and Access Management (IAM) user or role has the required permissions to use the runbook. For more information, see the Required IAM permissions section of AWSSupport-TroubleshootElasticBeanstalk.

Complete the following steps:

1. Open the Systems Manager console.
2. In the navigation pane, choose Documents.
3. In the search bar, enter TroubleshootElasticBeanstalk.
4. Select the AWSSupport-TroubleshootElasticBeanstalk document.
5. Choose Execute automation.
6. Choose Execute.
   Note: For a list of the steps that the runbook performs, see the Document steps section of AWSSupport-TroubleshootElasticBeanstalk.
7. Review the detailed results in the Outputs section.

Manually turn on diagnostics

To turn on AI-powered environment analysis, complete the following steps:

1. Open the Elastic Beanstalk console.
2. In the navigation pane, choose Environments, and then select your environment.
3. Choose AI Analysis.
4. Review the output for next steps.

Identify the failure

Check the configuration changes and review logs

Complete the following steps:

1. Open the Elastic Beanstalk console.
2. In the navigation pane, choose Environments, and then select your environment.
3. Review the Events tab and Health & Monitoring tab.
4. Choose Change history to check for recent configuration changes.
5. Retrieve the environment logs and review eb-engine.log for platform events.
6. Check the proxy logs for request errors.
   Note: On Linux, proxy logs are at /var/log/nginx/error.log for Nginx or /var/log/httpd/error_log for Apache. On Windows, review the IIS logs.
7. Review the web.stdout.log for application output.

Run tests

To test the public environment URL from the client to the load balancer, run the following command:

curl -v http://my-env.eba-XXXX.aa-example-1.elasticbeanstalk.com/

Note: Replace my-env.eba-XXXX.aa-example-1.elasticbeanstalk.com with your environment URL. You can find your environment URL on the environment overview page on the Elastic Beanstalk console.

To test your instance proxy on port 80, run the following command:

curl -v http://localhost:80/

To test your instance's application on its configured port, run the following command:

curl -v http://localhost:5000/

Note: If you configured a custom application port with the PORT environment property, then replace 5000 with your configured port number.

If the environment URL fails but localhost on the instance succeeds, then the issue is between the load balancer and the instance.

If the proxy port fails but the application port succeeds, then the issue is in the proxy. If the application port also fails, then the issue is in the application.

Check the VPC and network configuration

If your instance can't reach AWS services, then the environment creation might time out or you might get the following error message:

"None of the instances are sending data"

To check your VPC and network configuration, complete the following steps:

1. Open the Amazon Virtual Private Cloud (Amazon VPC) console.
2. In the navigation pane, choose Route tables.
3. Confirm that you correctly configured the route tables for the instance subnets.
4. In the navigation pane, choose Subnets, and then select your subnet.For public subnets, check that there's a route for 0.0.0.0/0 to an internet gateway and that you turned on public IP address assignment.
   For private subnets, check that there's a route for 0.0.0.0/0 to a NAT gateway or VPC endpoints for the services that the environment uses.
5. In the navigation pane, choose Security groups, and then select your security group. Confirm that the instance security group allows outbound HTTPS on port 443 and HTTP on port 80.
6. Choose Network ACLs, and then select your network access control list (network ACL). Confirm that your network ACL allows the ephemeral port range 1024–65535 for return traffic.
7. In the navigation pane, choose Your VPCs, and then confirm that you turned on DNS resolution and DNS hostnames.
8. Use VPC Flow Logs to confirm whether the network accepts or rejects traffic.

For more information, see Using Elastic Beanstalk with Amazon VPC and How do I resolve the "failed to communicate" error message that I get when my EC2 instances fail to communicate with Elastic Beanstalk?

Check DNS and HTTPS

If the application works on the Elastic Beanstalk URL but not your custom domain or HTTPS, then take the following actions:

• Make sure that your domain record uses the environment URL. Use an Amazon Route 53 alias record for a zone apex domain. For more information, see How do I securely deploy my application and route traffic to my Elastic Beanstalk environment URL?.
• For HTTPS, confirm that the load balancer has a port 443 listener with a valid AWS Certificate Manager (ACM) certificate that covers each domain name. For more information, see How can I configure HTTPS for my Elastic Beanstalk environment?.
• If you use HTTP-to-HTTPS redirection, then confirm that the redirect rule is on the load balancer listener to avoid redirection.

Check security group rules

Make sure that you use the Elastic Beanstalk security groups.

To check the security group rules, complete the following steps:

1. Open the Amazon EC2 console.
2. In the navigation pane, choose Security Groups, and then select the instance security group.
3. Confirm that the instance security group allows inbound traffic from the load balancer security group on the application port and health check port.

If you must modify your security group rules, then see Update your security groups to reference peer security groups.

Check the application and proxy

If the preceding troubleshooting steps don't resolve your issue and the targets remain unhealthy, then the issue is at the application or proxy layer.

To resolve this issue, take the following actions:

• Confirm that the application listens on the port that the environment configuration forwards to. Set the PORT environment property or your platform's process configuration. The application and the environment must specify the same port.
• Extend the proxy with a configuration file to adjust upload sizes or request timeouts. Upload the file under .platform/nginx/conf.d/ for Nginx or .platform/httpd/conf.d/ for Apache. For more information, see Extending Elastic Beanstalk Linux platforms.
• Review web.stdout.log and eb-engine.log for application startup errors.

Also, review one of the following proxy logs for connection or configuration errors:

• For Nginx, review /var/log/nginx/error.log for errors and /var/log/nginx/access.log for requests.
• For Apache, review /var/log/httpd/error_log for errors and /var/log/httpd/access_log for requests.
• For Windows platforms, review the IIS logs.

Check the load balancer and health checks

If your health check fails with an HTTP 502, 503, or 504 message and cycles instances, then complete the following steps:

1. Open the Elastic Beanstalk console.
2. In the navigation pane, choose Environments, and then select your environment.
3. Choose the Configuration tab.
4. Choose Edit, and then set the health check path to an endpoint that returns HTTP 200 without authentication or redirection.
   Note: If the destination endpoint is unhealthy, then the endpoint returns HTTP 301, 302, or 404 messages.
5. Select the health check port that the application listens on.
6. If the application is slow to start, then increase the health check grace period and thresholds.
7. Open the Amazon EC2 console.
8. In the navigation pane, choose Target Groups, and then select your target group.
9. Confirm that the destination endpoints are in a healthy state.

Check Amazon RDS and downstream connectivity

If the application starts and the database calls fail, then network reachability or credentials might be causing the issue.

To check connectivity, complete the following steps:

1. Open the Amazon EC2 console.
2. In the navigation pane, choose Security Groups, and then select your database security group.
3. Confirm that the database security group allows inbound traffic from the instance security group on the database port.
4. Confirm that the instances and database are in the same VPC or have valid routing between them.
5. Make sure that you store the database endpoint, port, and credentials as environment properties.

Related information

Troubleshooting your Elastic Beanstalk environment

Managing Elastic Beanstalk instance profiles

Using Elastic Beanstalk with Amazon Relational Database Service (Amazon RDS)