Help improve AWS Support Official channel in re:Post and share your experience - complete a quick three-question survey to earn a re:Post badge!
Traffic Mirror ICMP Type 3 anomaly
Hello. I have setup a small network in my VPC with 3 Ubuntu EC2 machines and 2 traffic mirror sessions all in the same subnet. The target for both sessions is 10.10.10.129. The 2 sources are 10.10.10.249 and 10.10.10.105. My traffic mirror filter is set to only send outbound traffic from the mirror sources to .129. I am observing a strange anomaly where when I send a ping command from one of my sources to the other source, my target gets the outbound traffic from each source but then the target sends out an ICMP Type 3 packet to both sources.
In the screen grab I have arkime (by way of of malcolm) displaying a single ping command from .249 to .105. I should note that the .249 is where my arkime application is running, so all traffic on .249 is being seen by arkime (traffic mirror stuff and anything else coming or going on .249). For some reason, at the same time the expected ICMP traffic (line items 7 and 4), there is also an ICMP Type 3 packet sent from .249 to .105 and .249 (line items 6 and 8). I have confirmed this behavior using TCPDUMP on the .249 interface.
Can anyone explain this behavior? I am convinced it is not an anomaly with my arkime application and suspect that after the .129 decapsulates the VXLAN packet it then tries to send it's own malformed ICMP packet (malformed because it is a Type 3 ICMP).
- Newest
- Most votes
- Most comments
The behavior you're observing is likely related to how Traffic Mirroring handles certain types of network traffic. When you set up Traffic Mirroring in Amazon VPC, there are some limitations and specific behaviors to be aware of:
-
Traffic Mirroring encapsulates the mirrored packets using VXLAN (Virtual Extensible LAN) headers. This encapsulation is necessary to route the mirrored traffic to the target.
-
When the mirrored traffic reaches the target (in your case, 10.10.10.129), the target needs to decapsulate the VXLAN packets to access the original traffic.
-
The ICMP Type 3 messages you're seeing are likely "Destination Unreachable" messages. These are generated when a packet cannot be delivered to its intended destination for various reasons.
-
In your setup, it appears that after decapsulation, the target (10.10.10.129) is attempting to process or route the original ICMP packets. However, since these packets were mirrored and not intended for actual delivery at the target, the target generates ICMP Type 3 messages in response.
-
This behavior is not necessarily an anomaly, but rather a side effect of how the mirrored traffic is processed at the target.
It's important to note that inbound traffic that is dropped at the traffic mirror source due to security group rules or network ACL rules is not mirrored. However, mirrored outbound traffic is not subject to the outbound security group rules for the traffic mirror source.
To mitigate this behavior, you might consider adjusting your traffic mirror filter to be more specific about the types of traffic you want to mirror, or you may need to configure your monitoring tools (like arkime) to ignore or filter out these ICMP Type 3 messages if they're not relevant to your analysis.
Sources
Understand traffic mirror target concepts - Amazon Virtual Private Cloud
Understand traffic mirror filter concepts - Amazon Virtual Private Cloud
Understanding traffic mirror packet format - Amazon Virtual Private Cloud
Relevant content
- asked 5 years ago
AWS OFFICIALUpdated 2 years ago
