Skip to content

ALB health checks always fail for EC2 “web” instance behind CDK‑created ApplicationLoadBalancer

0

Environment & Goal:

  • I’m using AWS CDK (Python) to deploy:
  • A VPC with two public subnets (one per AZ, no NAT gateways).
  • A Fargate‐based Flask API behind an ApplicationLoadBalancer on ports 80/443.
  • A single EC2 “web” instance running Apache/httpd that serves static HTML from /var/www/html, and is registered as a second target (path prefix /apps) in the same ALB via a separate Target Group.
  • ALB access logs streaming into an S3 bucket.

My goal is to have:

What I’ve Tried:

Security groups:

  • ALB SG allows inbound 80/443 from 0.0.0.0/0.
  • EC2 SG allows ingress on port 80 only from the ALB SG.

Apache config on EC2:

  • Created /var/www/html/health.html with content OK (permissions 644).
  • Verified locally on the instance via SSM:
curl -I http://localhost/health.html  # 200 OK
curl -I http://<private-ip>/health.html  # hangs when called from my Mac
  • Added ServerName localhost, Listen 0.0.0.0:80, and a <VirtualHost> + Alias /apps /var/www/html in /etc/httpd/conf.d/apps.conf.
  • Restarted httpd successfully; apachectl configtest is OK.

Target Group & Health Check:

  • Path: /health.html
  • Protocol: HTTP, port 80
  • Matcher: 200-399
  • Interval 30s, timeout 5s, thresholds 2/5
  • Manually deregistered and re‑registered the instance—status remains UNHEALTHY (Target.Timeout).

Network:

  • VPC Subnet ACLs allow all in/out.
  • Instance has an Elastic IP and Public IP.
  • NACL Passes all traffic.
  • No other firewall on the instance.

Logs & Observations:

  • ALB access logs show repeated 504s when hitting /apps.
  • Apache access_log shows no entries from the ALB health checks (only my curl with custom Host header).
  • Apache error_log shows occasional autoindex:error for missing DirectoryIndex, but not /health.html errors.
  • From my workstation, trying curl -I http://<public‑ip>/health.html times out.

What I Need

  • Why is the ALB health check timing out when it clearly can reach the instance from within?
  • How to configure either the ALB health check or Apache (or security groups/network) so that /health.html on the EC2 instance returns 200 when ALB probes it?

The SSL certificate is already set and everything works except access to the webserver on EC2 from ALB.

This is the code i'm using for that:

# ----------------- Web/Upload EC2 (SSH target, serves /apps) -----------------
web_sg = ec2.SecurityGroup(self, "WebUploadSG",
                           vpc=vpc,
                           description="Allow SSH from ECS tasks; HTTP from ALB",
                           allow_all_outbound=True
                           )

web_role = iam.Role(self, "WebUploadRole",
                    assumed_by=iam.ServicePrincipal("ec2.amazonaws.com"),
                    managed_policies=[
                        iam.ManagedPolicy.from_aws_managed_policy_name("AmazonSSMManagedInstanceCore")
                    ]
                    )

web_instance = ec2.Instance(self, "WebUploadInstance",
                            instance_type=ec2.InstanceType("t3.nano"),
                            machine_image=ec2.MachineImage.latest_amazon_linux2(),
                            vpc=vpc,
                            vpc_subnets=ec2.SubnetSelection(subnet_type=ec2.SubnetType.PUBLIC),
                            security_group=web_sg,
                            role=web_role
                            )

web_instance.user_data.add_commands(
    "#!/bin/bash -xe",
    "yum update -y",
    "yum install -y unzip httpd",

    # Configure Apache properly
    "echo 'ServerName localhost' >> /etc/httpd/conf/httpd.conf",
    "sed -i 's/^Listen 80/Listen 0.0.0.0:80/' /etc/httpd/conf/httpd.conf",

    # Create directory structure
    "install -d -m 2775 -o ec2-user -g apache /var/www/html",
    "chmod 755 /var/www /var/www/html",

    # Health check file
    "echo 'OK' > /var/www/html/health.html",
    "chown root:apache /var/www/html/health.html",
    "chmod 644 /var/www/html/health.html",

    # Apps configuration
    "cat <<'EOF' > /etc/httpd/conf.d/apps.conf",
    "<VirtualHost *:80>",
    "  ServerName localhost",
    "  DocumentRoot /var/www/html",
    "  Alias /apps /var/www/html",
    "  <Directory /var/www/html>",
    "    Options Indexes FollowSymLinks",
    "    Require all granted",
    "    AllowOverride None",
    "  </Directory>",
    "</VirtualHost>",
    "EOF",

    # Verify Apache config
    "apachectl configtest",

    # Start Apache
    "systemctl enable --now httpd",
    "systemctl restart httpd",

    # Verify installation from both local and network perspectives
    "curl -I http://localhost/health.html || exit 1",
    "curl -I http://$(hostname -I | awk '{print $1}')/health.html || exit 1"
)

# SG from Fargate tasks
task_sg = service.service.connections.security_groups[0]

# Allow SSH (22) from tasks to EC2
web_sg.add_ingress_rule(
    ec2.Peer.security_group_id(task_sg.security_group_id),
    ec2.Port.tcp(22),
    "SSH from ECS tasks"
)

# Allow HTTP from ALB to EC2
alb_sg = service.load_balancer.connections.security_groups[0]
web_sg.add_ingress_rule(
    ec2.Peer.security_group_id(alb_sg.security_group_id),
    ec2.Port.tcp(80),
    "HTTP from ALB"
)

# Target group for the EC2 instance
web_tg = elbv2.ApplicationTargetGroup(
    self, "WebAppsTG",
    vpc=vpc,
    port=80,
    protocol=elbv2.ApplicationProtocol.HTTP,
    target_type=elbv2.TargetType.INSTANCE,
    health_check=elbv2.HealthCheck(
        path="/health.html",
        healthy_http_codes="200-399",
        interval=Duration.seconds(30),
        timeout=Duration.seconds(5),
        healthy_threshold_count=2,
        unhealthy_threshold_count=5,
    )
)
web_tg.add_target(InstanceTarget(instance=web_instance, port=80))

# Add rule on the HTTPS listener
service.listener.add_action(
    "AppsRule",
    priority=10,
    conditions=[elbv2.ListenerCondition.path_patterns(["/apps", "/apps/*"])],
    action=elbv2.ListenerAction.forward([web_tg]),
)

Any pointers on CDK health‑check properties, Apache virtual‐host tweaks, or network/SecurityGroup settings would be hugely appreciated. Thank you!

2 Answers
0

Based on your description, there are several potential issues that could be causing the ALB health checks to fail for your EC2 instance:

  1. Apache Configuration Issue: Your Apache configuration might not be correctly handling the health check requests from the ALB. The ALB sends health check requests to the root path you specified (/health.html), but your VirtualHost configuration has an Alias that might be interfering with how these requests are processed.

  2. Security Group Configuration: While you've allowed HTTP traffic from the ALB security group to the EC2 instance, there might be an issue with how the return traffic is handled. Even though you've set allow_all_outbound=True for the web security group, ensure that the ALB security group allows the return traffic.

  3. Health Check Path: The health check is looking for /health.html, but your Apache configuration has an Alias that maps /apps to /var/www/html. This might cause confusion in how the server handles the health check request.

  4. Listening Interface: Your Apache might not be properly listening on all interfaces. While you've set Listen 0.0.0.0:80, check that the server is actually binding to all interfaces.

Here are some recommendations:

  1. Simplify Apache Configuration:
  • Remove the Alias directive temporarily to rule out path mapping issues
  • Ensure your VirtualHost is correctly configured to serve content from the root path
  1. Verify Network Connectivity:
  • Use netstat -ant on the EC2 instance to verify that Apache is listening on port 80 on all interfaces
  • Check if there are any firewall rules (like iptables) on the EC2 instance that might be blocking the ALB health checks
  1. Modify Health Check Configuration:
  • Try using a simpler health check path like just / instead of /health.html
  • Increase the timeout value to give more time for the health check to complete
  1. Check ALB Security Group:
  • Ensure the ALB security group allows outbound traffic to the EC2 instance on port 80
  1. Test Direct Connectivity:
  • Launch another EC2 instance in the same VPC and try to curl the health check endpoint from there to isolate if it's a network issue or an application issue

The fact that you can access the health check page locally on the instance but not from your Mac or the ALB suggests there's either a network configuration issue or an Apache configuration issue that's preventing external access to the server.
Sources
Health check failure with HTTP/2 in an ALB target group but succeeding with HTTP1.1 | AWS re:Post
Target Group health checks failing to private instances | AWS re:Post
Troubleshoot a Classic Load Balancer: Health checks - Elastic Load Balancing
ALB security group with cloudFront origin-facing prefix list ingress setting blocks target group health check for eks fargate | AWS re:Post

answered a year ago

EXPERT

reviewed a year ago

0

Hello.

From my workstation, trying curl -I http://<public‑ip>/health.html times out.

Direct access to EC2 fails because the EC2 security group only allows connections from the ALB.
Try adding your IP address to the EC2 security group.
Using this method, first try to make sure that you can access the health check path from outside the EC2 instance.

web_sg.add_ingress_rule(
    ec2.Peer.ipv4('your ip address/32'),
    ec2.Port.tcp(80)
)
EXPERT

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.