Elastic Beanstalk Health degrading due to Web ACL rules blocking bad requests

Hi, Jorgen!

Thanks for sharing your issue. Let's dive deeper into resolving the Elastic Beanstalk health degradation problem due to Web ACL blocking. I'll include detailed examples where applicable for better clarity. 😊

---

Clarifying the Issue

Your application is correctly using Web ACL to block malicious traffic. However, the Elastic Load Balancer (ELB) health checks don't distinguish between malicious requests and legitimate service health. As a result, Elastic Beanstalk interprets the 403 errors caused by ACL blocks as a failure, even though the servers are healthy. We aim to improve this configuration to balance accurate health checks and robust security.

---

Key Terms

• Elastic Beanstalk (EB): Automates the deployment and scaling of web applications.
• Web ACL: Security layer in AWS WAF for filtering web traffic.
• Load Balancer (LB): Distributes incoming traffic to servers.
• Health Check Path: URL or endpoint checked by the LB to determine application health.

---

The Solution (Our Recipe)

Steps at a Glance

1. Update the Elastic Beanstalk health check path to avoid conflicts with Web ACL blocks.
2. Implement a custom health check endpoint and configure your load balancer to use it.
3. Fine-tune Web ACL rules to reduce false positives.
4. Enable WAF logging for dynamic rule refinement.

---

Step-by-Step Details with Examples

1. Update Elastic Beanstalk Health Check Path

Elastic Beanstalk uses a default health check path (/) for ELB. If this path triggers WAF rules, modify it to point to a dedicated health check endpoint.

Update Health Check via CLI

Use the AWS CLI to update the health check path:

aws elbv2 modify-target-group \
    --target-group-arn <your-target-group-arn> \
    --health-check-path "/health" \
    --health-check-interval-seconds 30 \
    --health-check-protocol HTTP

Replace <your-target-group-arn> with your ELB Target Group ARN.

---

2. Implement a Custom Health Check Endpoint

Create a lightweight health check endpoint that returns a 200 OK status. Here's an example for a Node.js application:

Sample Code (Node.js Express)

const express = require('express');
const app = express();

app.get('/health', (req, res) => {
    res.status(200).send('OK');
});

app.listen(3000, () => {
    console.log('Health check running on port 3000');
});

Deploy the /health route to your application and redeploy it to Elastic Beanstalk.

---

3. Fine-Tune Web ACL Rules

Reduce false positives by refining your WAF rules. For example, allowlist trusted IP ranges or paths used by health checks.

Add an Allow Rule for Health Check Path

In AWS WAF, create a rule to allow requests to /health.

1. Go to the WAF console and navigate to your Web ACL.
2. Add a new rule with the following conditions:
   • Match Method: GET
   • Match Path: /health
3. Set the rule action to Allow.

This will ensure that the health check traffic is allowed and does not interfere with Web ACL rules.

---

4. Enable WAF Logging

Enable logging to analyze blocked requests and refine your ACL rules.

Configure WAF Logging via CLI

aws wafv2 put-logging-configuration \
    --resource-arn <your-web-acl-arn> \
    --logging-configuration '{
        "LogDestinationConfigs": ["arn:aws:logs:region:account-id:log-group:my-waf-logs"],
        "RedactedFields": []
    }'

Replace <your-web-acl-arn> with your Web ACL ARN and configure CloudWatch to view logs of allowed and blocked requests.

---

Closing Thoughts

By updating the health check path, creating a custom endpoint, and refining WAF rules, you can maintain strong security while preventing false degradation alarms. Let me know where you'd like to add specific inputs/outputs for additional clarity, or if you'd like examples for a different programming language. Best of luck resolving this issue! 🚀

---

Cheers, Aaron 😊

Hi Jorgen,

Thanks for following up and providing more context on your Elastic Beanstalk issue. Let’s focus on resolving the false degradation alarms caused by Web ACL blocking, based on the logs and observations you’ve shared. 😊

---

Clarifying the Issue

The main problem lies in how Elastic Beanstalk interprets Web ACL blocking. While the servers are healthy, the Elastic Load Balancer (ELB) logs a high percentage of 4xx errors due to malicious traffic being blocked by the ACL. This causes Elastic Beanstalk to misinterpret these errors as a health issue at the environment level, even though individual health checks are passing.

Our solution aims to separate health check evaluations from Web ACL blocking and fine-tune the health thresholds to prevent unnecessary “Severe” status changes.

---

Key Terms

• Elastic Beanstalk (EB): AWS service for deploying and scaling web applications.
• Web ACL (Access Control List): A set of rules in AWS WAF for filtering traffic.
• Elastic Load Balancer (ELB): Balances incoming traffic to ensure reliability and availability.
• Health Check Path: The endpoint used by ELB to monitor application health.

---

The Solution (Our Recipe)

Steps at a Glance:

1. Update the Elastic Beanstalk health check path to use a dedicated endpoint.
2. Exclude health check traffic from Web ACL evaluation.
3. Use AWS WAF logging to analyze and refine Web ACL rules.
4. Adjust Elastic Beanstalk health thresholds to reduce sensitivity to transient errors.

---

Step-by-Step Guide:

1. Update the Elastic Beanstalk Health Check Path
   Configure a dedicated endpoint (/health) to handle health check traffic. Ensure this endpoint always returns a 200 OK status, independent of application load.

   Update the health check path via the AWS CLI:

   aws elbv2 modify-target-group \
       --target-group-arn <your-target-group-arn> \
       --health-check-path "/health" \
       --health-check-interval-seconds 30 \
       --health-check-protocol HTTP

   Replace <your-target-group-arn> with your Target Group ARN.

---

2. Exclude Health Check Traffic from Web ACL Evaluation
   Prevent Web ACL from interfering with health check traffic by allowlisting the /health endpoint.

   In the AWS WAF console:

   • Add a rule with the following conditions:
     • HTTP Method: GET
     • Path: /health
   • Set the rule action to Allow.

   This ensures health check requests bypass Web ACL evaluation and are not blocked.

---

3. Use AWS WAF Logging to Refine Rules
   Enable WAF logging to capture detailed logs of blocked requests. Analyze the logs to identify false positives and refine rules accordingly.

   Enable WAF logging using the CLI:

   aws wafv2 put-logging-configuration \
       --resource-arn <your-web-acl-arn> \
       --logging-configuration '{
           "LogDestinationConfigs": ["arn:aws:logs:region:account-id:log-group:my-waf-logs"]
       }'

   Use CloudWatch logs to adjust rules for common patterns that may trigger false positives.

---

4. Adjust Elastic Beanstalk Health Thresholds
   Modify Elastic Beanstalk’s health reporting settings to reduce sensitivity to transient 4xx errors:

   • Navigate to the Environment Settings in Elastic Beanstalk.
   • Lower the threshold for “Severe” status to allow for brief spikes in 4xx errors.
   • Increase the evaluation interval to smooth out short-term fluctuations.

   For example, increasing the allowable error percentage or evaluation time frame can prevent premature “Severe” status changes.

---

Closing Thoughts

By updating the health check path, excluding it from Web ACL evaluation, and fine-tuning both Web ACL rules and Elastic Beanstalk health settings, you can maintain strong security without false degradation alarms. Let us know how these steps work or if you’d like additional clarification! 🚀

---

Best of luck, Jorgen! Keep us posted on your progress. 😊

---

Cheers, Aaron 😊