ELB returns 4xx when I run too many automated tests

As per Troubleshoot your Application Load Balancers, 403 errors are likely due to AWS WAF

You may want to dis-associate the WAF, or allow-list your source IP during testing.

The issue you're experiencing with your Elastic Load Balancer (ELB) returning 4xx errors, specifically 403 Forbidden, when running a high volume of automated tests is likely related to the ELB's built-in protection mechanisms against potential abuse or unusual traffic patterns.

When you're generating a large number of requests in a short time (3K-4K HTTP requests), the ELB may interpret this as suspicious activity and start blocking requests as a protective measure. This is especially true if the requests are coming from a limited number of IP addresses or if they're arriving at a rate that's significantly higher than normal traffic patterns.

While there isn't a direct setting in the ELB to adjust for this specific scenario, you can try the following approaches to mitigate the issue:

1. Gradually increase the rate of requests: Instead of sending all 3K-4K requests at once, try ramping up the request rate over time. This may help avoid triggering the ELB's protective mechanisms.

2. Distribute requests: If possible, distribute your test requests across multiple source IP addresses or clients. This can help prevent the traffic from appearing as a potential attack from a single source.

3. Increase the capacity of your backend: Ensure that your EC2 instances can handle the load. If the backend is struggling to process requests quickly enough, it might cause the ELB to return errors.

4. Monitor ELB metrics: Keep an eye on metrics like RequestCount, HTTPCode_ELB_4XX_Count, and SurgeQueueLength. These can provide insights into how the ELB is handling the increased load.

5. Consider using Application Load Balancer (ALB): If you're using a Classic Load Balancer, switching to an Application Load Balancer might provide more flexibility and better handling of high-volume traffic.

6. Contact AWS Support: If the issue persists and is significantly impacting your testing process, it may be worth reaching out to AWS Support. They might be able to provide more specific guidance or adjust certain limits on your account.

Remember, while these steps may help, it's important to design your tests in a way that simulates real-world traffic patterns as closely as possible to avoid triggering protective measures.
Sources
Resolve RequestCount metric drop in Application Load Balancer | AWS re:Post
Troubleshoot a Classic Load Balancer: HTTP errors - Elastic Load Balancing