Skip to content

UDP Multicast Behaviour

0

We have the following setup:

  1. An EC2 node that has a UDP multicast producer application running inside it. This publishes UDP multicast packets with TTL=0
  2. This EC2 is registered as a "source" in our AWS Multicast Transit Gateway.
  3. We have few subscribers in this Multicast domain which receive the UDP feed.

This setup has been working for more than a year now. Consumers are able to receive the feed continuously.

Recently we saw some issue where the outbound traffic from the EC2 stopped completely and we found that this was caused by running a local consumer command on the same node (e.g., ./consume-mcast --maddr 239.230.255.250 --port 6381 --iface 172.0.181.165 -- this is joining the --iface to the multicast domain as a consumer)..

What we are not able to understand is the root cause for this behaviour:

  1. Ideally with TTL=0, the packets should not be leaving the host at all. How this has been working with TTL=0?
  2. How does running a multicast consumer affect outbound traffic. It's as if when running a local consumer, the TTL=0 is honoured, but without that, the packets are leaving the host.
  3. If we set the TTL config to be 1 or higher (instead of 0), the outgoing UDP packets do not get affected even with a local consumer.

Is there some behaviour specific to AWS here? Is the act of joining the single interface (which is used for multicast outbound) affecting the group status of this node as a source?

1 Answer
0

The behavior you're experiencing with UDP multicast in AWS has several technical explanations:

Regarding TTL=0: In standard IP networking, a TTL of 0 should indeed prevent packets from leaving the host. However, in AWS networking environments, there may be some virtualization layer handling that allows these packets to be processed differently than on traditional hardware. The fact that your setup worked for over a year with TTL=0 suggests that AWS's networking implementation might be interpreting or handling the TTL differently than expected in a traditional network stack.

Regarding the local consumer issue: When you run a local consumer on the same EC2 instance as your producer, you're joining that interface to the multicast group. This likely changes how the Linux kernel's multicast routing handles those packets. With a local consumer present, the kernel may be enforcing the TTL=0 restriction more strictly, keeping those packets local to the host as technically should happen with TTL=0.

When you set TTL to 1 or higher, you're explicitly telling the network stack that these packets can leave the local host, which is why the behavior works regardless of whether a local consumer is present.

The Transit Gateway Multicast Domain is designed to handle multicast traffic across VPCs and subnets. When you register your EC2 as a source in the multicast domain, AWS is expecting properly configured multicast packets with appropriate TTL values. The interaction between your local interface joining a multicast group and the Transit Gateway's handling of your instance as a source is likely causing the behavior you're seeing.

To resolve this issue, the correct approach would be to set your TTL to at least 1 (or higher depending on how many network hops your multicast traffic needs to traverse). This would ensure proper multicast behavior regardless of local consumers.

The behavior isn’t a defect but a result of architectural layering. AWS’s virtual network behaves more like an overlay multicast distribution system that begins replication at the ENI boundary, not a physical router. Hence, any TTL-based limitation that works within host-local contexts can seemingly be overridden by design, unless the local OS forces the rule internally through active membership. Adjusting TTL to one or greater synchronizes both the AWS network layer and Linux’s behavior, ensuring expected multicast transmission flows even with local consumers present.

Sources
Amazon EC2 examples using AWS CLI - AWS SDK Code Examples

AWS Transit Gateway Multicast Overview

Running Multicast-Enabled Containers on AWS

Integrating External Multicast Services with AWS

answered a year ago

AWS
SUPPORT ENGINEER

revised a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.