How do I troubleshoot 503 "Service Unavailable" errors from my Application Load Balancer?

Short description

To determine if the Application Load Balancer is generating the 503 errors, complete the following steps:

• Access your Amazon CloudWatch metrics and look for a metric labeled HTTPCode_ELB_503_Count.

• Run the curl command. If the response contains "503 Service Temporarily Unavailable," then the error is from the Application Load Balancer.

  Replace MY_URL with the URL used to access the Application Load Balancer:

  curl -IkL MY_URL

• Check your Application Load Balancer access logs for an elb_status_code = 503. If you see 503 errors, then the error is generated by your Application Load Balancer.

If the 503 errors are generated by your Application Load Balancer, then complete the following resolution steps.

Resolution

Verify that the target group has registered targets

Use the Amazon EC2 console

Complete the following steps:

1. Access the Amazon Elastic Compute Cloud (Amazon EC2) console.
2. On the navigation pane, under Load Balancing, choose Target Groups.
3. Select the name of the target group to open its details page.
4. Choose the Targets tab.
5. Check whether there are targets listed. If they are listed, then they are registered.

Use the AWS CLI

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

To verify targets, run the describe-target-health command.

Register targets

If you don't see registered targets, then register them through the console or through the AWS CLI.

If you use Amazon EC2 Auto Scaling, then verify that the target group is associated with an Amazon EC2 Auto Scaling group

Check that your target group is correctly associated with your Amazon EC2 Auto Scaling group.

Use the Amazon EC2 console

Complete the following steps:

1. Open the Amazon EC2 console.
2. On the navigation pane, under Auto Scaling, choose Auto Scaling Groups.
3. Select the Auto Scaling group that you want to verify.
4. Under Load balancing, confirm that the target group of the Application Load Balancer is associated with the Auto Scaling group.

Use the AWS CLI

Run the describe-auto-scaling-groups command. Replace MY-ASG with the name of your Amazon EC2 Auto Scaling group. Replace AWS-REGION with your specific AWS Region:

aws autoscaling describe-auto-scaling-groups --auto-scaling-group-name MY-ASG --region AWS-REGION

In the command output, confirm that the target group is listed under TargetGroupARNs.

Attach the target group

If the target groups aren't attached to the Amazon EC2 Auto Scaling group, then associate the groups through the console or through the AWS CLI.

Check target health

After you confirm that your targets are registered, check that they are in a healthy state. For more information, see How do I troubleshoot failed health checks for Application Load Balancers?

Check target status for target groups that have cross-zone load balancing turned off

For Application Load Balancers, you can turn off cross-zone load balancing at a target group level. When cross-zone load balancing is turned off, the Application Load Balancer requires each activated Availability Zone to have targets to serve traffic. If you have an empty Availability Zone, then you get a 503 error for HTTP requests that the node in that Availability Zone receives.

Check that targets are ready

If targets aren’t ready for requests, or there aren’t enough targets, then requests routed to target groups with target optimizer might be rejected.

To detect issues with targets, monitor the following metrics in CloudWatch:

• TargetControlRequestRejectCount should be near zero.
• TargetControlWorkQueueLength should show active processing.

To fix issues with targets, complete the following steps:

1. To increase target capacity, add more EC2 instances to the target group or update auto-scaling groups.

2. To increase the maximum number of concurrent requests on the agent, increase the setting of the TARGET_CONTROL_MAX_CONCURRENCY environment variable. Run one of the following commands:

   Linux

   export TARGET_CONTROL_MAX_CONCURRENCY=1000

   Windows

   set TARGET_CONTROL_MAX_CONCURRENCY=1000

Note: You can turn on target optimizer only during target group creation, and you can’t modify the port that it uses.

Related information

DeregisterTargets

AttachLoadBalancerTargetGroups

Target group health