Baby DDOS Attacks Triggering AWS Shield Standard Throttling ? Causing ICMP to Fail.



So, it may turn out that the problem all along is caused by the very thing designed to prevent it.....

We run or are trying to run gaming servers on AWS due to the new regions offered However, these gaming servers that we don't have a lot of control over reply upon ICMP for player clients to locate / confirm accessibility.

Despite us running very powerful servers (c5n.x9large + ENA) that can eat these baby DDOS attacks for breakfast (40 Mbps/180K PPS Vs a 2% Core Usage & the x9large's 50 Gbps connection) it seems there is some AWS background activity - maybe from AWS Shield Standard that is blocking ICMP during these ddos attacks (despite the server's ability to clearly handle it).

The best info I could find on the topic was from here:

"Shield calculates capacity for some AWS resources is based on the available capacity of the underlying AWS infrastructure. These resource types include Network Load Balancers (NLBs) and resources that route traffic through Gateway Load Balancers or AWS Network Firewall."

So what it appears to be happening is these little ddos attacks are triggering the ddos mitigation which is causing our services to fail. Below is the flow.

  1. desktop continuously pings EC2 without issue
  2. DDOS starts
  3. desktop ping fails (Request time out.)
  4. DDOS stops
  5. desktop ping resumes automatically

The server is not receiving these pings during the DDOS (we have verified they aren't being dropped due to capacity on the server side - they simply don't reach it).

Note that the players on the server have Zero impact / lag despite this being a UDP intensive game. It's as smooth as butter for them. But new players cannot find the server due to the ICMP requirement that is failing during the DDOS.

So the big question is what is preventing the ICMP functionality during DDOS despite our server being able to handle the traffic and how can we fix this?

Notes: -we want to avoid paying for AWS Shield Advanced and even if we purchase it, it might do the same thing. -we also have an AWS Network firewall that we have utilized but the same problem exists with and without the AWS Firewall. The AWS firewall is removing all of the harmful packets and with it the server traffic is very low like 2 Mbps (40 Mbps if we let the ddos through the network firewall), yet the ICMP still is failing during any ddos.
-we are limited to this setup due to the constraints of the gaming servers, we don't have the luxury of NAT / GA as the gaming server will only allow the Elastic IP not any other IP.

2 Answers

To answer your query regarding what is preventing ICMP during DDOS despite your server being able to handle the traffic . Here are a few possible explanations and suggestions for addressing the issue:

  1. Network congestion: DDoS attacks often flood the network with a massive volume of traffic, overwhelming the available bandwidth. Even if your service can handle the traffic at the application layer, network congestion can still impact ICMP functionality. To address this, you can place your EC2 instance behind ELB ( Elastic Load Balancing ), which will provide Shield Standard DDoS resiliency. Additionally, if your application is being hit by DDoS attack, then you need the ELB to be behind a CloudFront Distribution

  2. Firewall settings: It's possible that your firewalls are configured to drop or rate-limit ICMP traffic during a DDoS attack. This is often done to conserve network resources and prioritize other types of traffic. Review your firewall and IPS settings to ensure that ICMP traffic is not being blocked or heavily restricted during an attack. Adjustments may be needed to allow ICMP traffic for diagnostic purposes or to ensure proper network functionality.

  3. Attack targeting ICMP specifically: Some DDoS attacks may specifically target the ICMP protocol to disrupt network connectivity and monitoring. This type of attack is known as an ICMP flood attack. In such cases, it can be challenging to maintain ICMP functionality even if your service can handle the overall traffic. Consider implementing additional mitigation techniques such as traffic filtering, rate limiting, or using specialized DDoS protection services that can detect and mitigate ICMP flood attacks.

In any case, mitigating the impact on ICMP functionality during a DDoS attack requires a combination of network infrastructure optimization, fine-tuning of security measures, and potentially implementing specialized DDoS protection services.

answered a year ago

We don't make layer 3/4 detections unless total traffic towards the target instance threatens to exceed the network capacity of the target instance type, and the mitigation placed is intended to protect other customers on the same hardware from 'noisy neighbor' impact. Do you know what the traffic composition of the attack was? If it contained ICMP then our default Shield Standard mitigations would not necessarily be able to tell 'good' ICMP from malicious ICMP (if the malicious traffic was well-formed).

If you have any level of technical support you can raise a case in the Shield or DDoS queue and ask to know whether or not we made a detection for instance/ip at date/time - we do have that information available, even for Shield Standard customers. If we did make a detection then we would have placed a mitigation which would have dropped some traffic, however if we did not make a detection then we would not have interfered with your traffic.

Note that the gaming sector are some of our biggest users of Shield Advanced. We specialize in protecting multi-player games on large EC2 instances using UDP. We can create custom mitigations that fit your specific use-case. Again, please raise a technical support case to discuss your options.

answered 6 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions