Elastic beanstalk event status issue: Ok to Severe, Severe to shutdown

Question

I have a webapp running on Elastic beanstalk. (Platform - Tomcat 8.5 with Corretto 11 running on 64bit Amazon Linux 2/4.3.7)

At beginning it's works fine. But after a few days, I started to get some error event notifications 
like below:
```
May 28, 2023 17:04:13 (UTC+8)	INFO	Environment health has transitioned from Severe to Ok.
May 28, 2023 17:03:13 (UTC+8)	WARN	Environment health has transitioned from Ok to Severe. 100.0 % of the requests are erroring with HTTP 4xx.
May 28, 2023 19:41:28 (UTC+8)	INFO	Environment health has transitioned from Severe to Ok.
May 28, 2023 19:39:28 (UTC+8)	WARN	Environment health has transitioned from Ok to Severe. 100.0 % of the requests are erroring with HTTP 4xx.
...
```

Some times, the server health could recover from a ‘Severe’ to ‘Ok’, but sometimes it cannot recover to 'Ok' and turns to shutdown.

I checked the server backend logs(/var/log/nginx/access.log), I belived that my webapp has been attacked.

The attacker send lots of bad request during a period of time, to make my web server fail to respond properly.
The logs as below:

```
128.199.16.76 - - [28/May/2023:17:02:22 +0000] "GET /db/webdb/index.php?lang=en HTTP/1.1" 404 777 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" "-"
128.199.16.76 - - [28/May/2023:17:02:22 +0000] "GET /mysqlmanager/index.php?lang=en HTTP/1.1" 404 777 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" "-"
128.199.16.76 - - [28/May/2023:17:02:23 +0000] "GET /administrator/db/index.php?lang=en HTTP/1.1" 404 785 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" "-"
128.199.16.76 - - [28/May/2023:17:02:23 +0000] "GET /sql/websql/index.php?lang=en HTTP/1.1" 404 779 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" "-"
128.199.16.76 - - [28/May/2023:17:02:23 +0000] "GET /admin/web/index.php?lang=en HTTP/1.1" 404 778 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" "-"
128.199.16.76 - - [28/May/2023:17:02:23 +0000] "GET /database/index.php?lang=en HTTP/1.1" 404 773 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" "-"
128.199.16.76 - - [28/May/2023:17:02:24 +0000] "GET /phppma/index.php?lang=en HTTP/1.1" 404 771 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" "-"
128.199.16.76 - - [28/May/2023:17:02:24 +0000] "GET /phpMyAdmin2/index.php?lang=en HTTP/1.1" 404 776 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" "-"
128.199.16.76 - - [28/May/2023:17:02:24 +0000] "GET /administrator/pma/index.php?lang=en HTTP/1.1" 404 786 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" "-"
128.199.16.76 - - [28/May/2023:17:02:24 +0000] "GET /php-my-admin/index.php?lang=en HTTP/1.1" 404 777 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" "-"
128.199.16.76 - - [28/May/2023:17:02:25 +0000] "GET /phpmyadmin2022/index.php?lang=en HTTP/1.1" 404 779 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" "-"
128.199.16.76 - - [28/May/2023:17:02:25 +0000] "GET /db/phpmyadmin4/index.php?lang=en HTTP/1.1" 404 783 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" "-"
128.199.16.76 - - [28/May/2023:17:02:25 +0000] "GET /mysql/pma/index.php?lang=en HTTP/1.1" 404 778 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" "-"
128.199.16.76 - - [28/May/2023:17:02:25 +0000] "GET /index.php?lang=en HTTP/1.1" 404 760 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" "-"

...
```

I think at that moment my server is still alive but the EC2 heath check found that in one minute all the request were responded as 404, so AWS set my server into 'Severe'.

What can I do on ElasticBeanstalk to make my webApp not go fail?

May I change EC2 heath check rule ?  Or dose AWS support any service to protect the webApp like firewall?

Answer

How about deploying AWS WAF to protect your web applications?  
AWS WAF can be configured on ALB or CloudFront and can be used to prevent attacks on web applications.  
Also, AWS WAF can be configured with rate-based rules, so it is possible to have it deal with attacks such as DDoS.  
https://docs.aws.amazon.com/waf/latest/developerguide/getting-started.html

If you want to specialize in DDoS countermeasures, you can also consider deploying AWS Shield Advanced as a countermeasure.  
https://docs.aws.amazon.com/waf/latest/developerguide/ddos-advanced-summary.html

It would be better to start with AWS WAF, which can be easily configured.

Elastic beanstalk event status issue: Ok to Severe, Severe to shutdown

相关内容