Skip to content

Target Group Health Check is Healthy, but has 500 error.

0

Hello

I am having an issue with the health check on a Target Group reporting Healthy, even though the site is showing a 500 error. I have since discovered that this is because the Health Check only actually checks codes between 200-499.

The setup is using Elastic Beanstalk, with rolling deployments based on health. The site is also using Docker and Nginx.

I am wondering how I can get around this issue? As we have had recently an update pushed to the site which had a 500 error, but the health check did not pick this up. So we had to wait for the rolling deployments to finish then re-push a previous working version. But it meant the site went down for about 1-1.5 hours.

Any help or advice on how to check for 500 errors before marking an instance as healthy would be massively appreciated.

Thanks

1 Answer
6

1. Modify Health Check Path

Ensure that your health check path is configured to a URL endpoint that is representative of the overall health of your application. This endpoint should return a status code within the range of 200-499 when the application is healthy and a 500 status code if there are any issues.

2. Custom Health Check Endpoint

Create a custom health check endpoint that performs more extensive checks than the default health check. This endpoint should:

Check database connectivity. Verify that all necessary services are running.

Perform any other critical checks specific to your application.

For example, you can create an endpoint /healthz that returns a 200 status code if everything is functioning correctly, or a 500 status code if there are any issues.

Example in Python (Flask)

from flask import Flask, jsonify

app = Flask(__name__)

@app.route('/healthz')
def health_check():
    try:
        # Perform various checks here
        # Check database connectivity
        # Check other service statuses
        # Return 200 if all checks pass
        return jsonify({"status": "healthy"}), 200
    except Exception as e:
        # Return 500 if any check fails
        return jsonify({"status": "unhealthy", "error": str(e)}), 500

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=80)

  1. Update Health Check Settings in Target Group Update the health check settings in your Target Group to point to your custom health check endpoint. This can be done through the AWS Management Console or using the AWS CLI.

Using AWS Management Console:

Navigate to the EC2 Dashboard.

Go to Target Groups.

Select your target group.

Edit the health check settings to point to your custom endpoint, e.g., /healthz.

Using AWS CLI:

aws elbv2 modify-target-group --target-group-arn <your-target-group-arn> --health-check-path /healthz

  1. Implement Blue/Green Deployment Strategy Consider using a Blue/Green deployment strategy to reduce downtime during deployments. This involves having two separate environments (Blue and Green) and switching traffic between them. AWS Elastic Beanstalk supports Blue/Green deployments natively.

  2. Use Enhanced Health Reporting Enable Enhanced Health Reporting for your Elastic Beanstalk environment. This provides more detailed health information and can help you quickly identify issues.

Enabling Enhanced Health Reporting:

Navigate to the Elastic Beanstalk environment in the AWS Management Console. Go to the "Configuration" section. Edit the "Health" configuration and enable Enhanced Health Reporting. 6. Monitor and Alerting Set up monitoring and alerts to notify you of any 500 errors or other critical issues. Use Amazon CloudWatch to create alarms based on logs and metrics.

Example CloudWatch Alarm for 500 Errors:

Navigate to the CloudWatch dashboard.

Create a new alarm.

Select the relevant metric (e.g., HTTPCode_Target_5XX_Count for your Load Balancer).

Set the threshold to an appropriate value to trigger an alarm when 500 errors are detected.

EXPERT
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.