next-hop is VPC with multiple subnets

If I got it right, the question is - When the traffic from Client/Server VPCs is being directed by the Transit Gateway (TGW) to the service VPC to get inspected using Firewall, how does the TGW go about choosing which subnet (where it has its network interface (ENI) present ) or AZ (You choose 1 subnet per AZ) it routes the traffic through?

The TGW will always prefer to send traffic from a Transit Gateway ENI in the same Availability Zone (AZ ID) as the sender (Source of traffic - Instance in Client/Server VPC). But if traffic is sourced from an Availability Zone (AZ ID) that the destination attachment is not present in, AWS Transit Gateway will internally route that traffic to a random Availability Zone where the attachment is present. There is no additional transit gateway charge for this type of cross-Availability Zone traffic. Reference - AZ Section in TGW Documentation

When the TGW is delivering an incoming packet in the subnet where the ENI of the TGW attachment is connected, I believe the next hop will be determined simply by the regular VPC route table of the subnet where the attachment is connected.

When the packet exits the firewall after it's been permitted to pass, it will follow the VPC route table of the subnet of the firewall's ENI to find the next hop, which would be the transit gateway attachment.

I'm not quite sure if that's the full scope of what you're asking, though. In addition to the two regular VPC route tables inside the service VPC, you'll also need two separate TGW route tables. One of the TGW route tables will be as you described, sending the client and server VPCs' CIDRs to the respective VPCs directly. This would be needed for the packets already permitted by the firewall to reach their destinations. This TGW route table will need to be attached to the TGW attachment of the service VPC.

The second TGW route table would be attached to the TGW attachments of both the server and client VPCs, with a single default route pointing to the service VPC for the firewall to process.

I did more tests based on the topology above and noticed when I ping IP configured on ge-0/0/0, the packet does go to that interface. But when I pinged the server from the client, ping works, but not via the firewall. The only difference I did this time is I specified Transit gateway CIDR blocks with TransitGateway subnet as listed above. I have no clue how service VPC sends the packets received from the Transit Gateway.

thanks so much !!

I have two route tables with subnet associations with 10.1.1.0/24 (ge-0/0/0) and 10.1.2.0/24 (ge-0/0/1)

Trust route table

   subnet association 10.1.2.0/24 (ge-0/0/1)
   
   static route    0.0.0.0/0 ---> TGW service VPC 

Untrust route table

   subnet association 10.1.1.0/24 (ge-0/0/0)
    
   static route  0.0.0.0/0 ---> network interface (ge-0/0/0)

Based on the above, if the client pings the server, the packet to service VPC should be directed to 10.1.2.0/24 (ge-0/0/1) Am I right ? Appreciate your coaching !!

thanks so much !! Yes. you got what I meant. But I still do not understand how exactly the firewall will receive the traffic. Say my firewall has untrust interface and trust interface, when the traffic from the transit gateway arrives at the service VPC, the packet might be sent to either trust or untrust interfaces, if both interfaces are in the same zone?

Say ge-0/0/0 and ge-0/0/1 are in the same zone, the packet arriving at service VPC is delivered to which interface, randomly, or based on some criteria ? thanks !!

Both your firewall interfaces are in the service VPC. The service VPC has one or more TGW attachments in it. Those TGW attachments are in certain subnets, which should be dedicated to TGW attachments only, but that's not vital for this case.

When the TGW has delivered a packet to the service VPC, the next hop will not be determined by the TGW. The next hop will be determined by the VPC route table that you've attached to the subnet where the TGW attachment is.

This is the subnets info

trust (ge-0/0/1 10.1.2.0/24) and untrust (ge-0/0/1 10.1.1.0/24) are the firewall interfaces.

This is TGW Route table

This is trust route table

This untrust route table

I expect the traffic between client and server goes via the firewall ge-0/0/1 to ge-0/0/0.

thanks so so so much !!

service-RT only is associated only with server vpc and client vpc, the default route points to service vpc. main-RT is associated with service vpc. I think this route table is not used in this case.

I assumed the traffic from the client will go via TGW to service VPC, targeting to ge-0/0/1. as ge-0/0/1 subnet is associated with trust-RT which has a default route with the next-hop pointing to TSW. Is my assumption right ?

I also did some tests with removing the default route from trust route table, the traffic stops. Based on this, I assumed the traffic is directed to 10.1.2.0/24 subnet which is associated with trust RT.

what puzzles me is

if I removed the default route, the traffic stops, but if I only disabled ge-0/0/1, the traffic continues. It seems to me the incoming traffic hits 10.1.2.0/24 subnet, but routed by cloud internal router, not by the firewall.

Any special configuration on subnet is required to direct the traffic to the firewall interface ?

So confusing !!

thanks so much !!

You won't want to use the TGW route table "service-RT" for the attachments for the client VPC or the server VPC.

That's because "service-RT" contains direct routes to both the client VPC and the server VPC, so if you attach the route table to the client VPC, for example, then a packet arriving from the client VPC and on its way to the server VPC will get sent straight to the server VPC -- and not to the service VPC that contains the firewall.

To both the client VPC and the server VPC, you'll need to attach a different TGW route table, such as "other-vpc-RT". That route table should contain only one route for 0.0.0.0/0 with the service VPC as the next hop. This will cause all traffic from both the client and server VPCs to be sent towards the service VPC.

Inside the service VPC, is the TGW attachment in the "svc-tgw-Subnet" or one of the other subnets? Which one? Which VPC route table is attached to that subnet? That route table will determine where to send the packets that are coming through the TGW from the client VPC (towards the server VPC or on-prem) or from the server VPC (towards the client VPC or on-prem). That VPC route table should contain a 0.0.0.0/0 route towards the firewall interface that should receive traffic from the client VPC towards the server VPC or vice versa.

thanks a lot !! I will try.

really appreciate !!!

My understanding is that if the server attachment and the client attachment both are associated with TGW routing table, but not propagate, they are not in the TGW route table, so they can't communicate to each other directly.

Am I right ?

thanks !!!

If you only have one TGW route table, and it's used for all three attachments (the client, server, service VPCs), then logically, there are only two options:

1. the direct routes to the destinations exist in the route table => nothing will go through the firewall
2. the direct routes to the destinations don't exist in the route table => traffic that finished passing through the firewall can't reach its destination

That's why there's no way to make it work with one route table for the TGW, nor with one route table for the VPC. You need at least two of both. One route table to take traffic in the client->server or server->client direction to the firewall, and the other route table to take that same traffic back from the firewall towards the destination VPC (client or server VPC).

You need to keep in mind that the IP frames between the client VPC and server VPC, in either direction, will never have a 10.1.x.x destination IP. When an IP frame in the direction client VPC -> server VPC is sent towards the firewall, the destination IP address won't change to one of the firewall's 10.1.x.x IPs. The destination IP will continue to be 10.2.0.x (in the server VPC). But you want that packet to go in the direction of the firewall first, and you want that exact same packet to do exactly the opposite next, moving away from the firewall and towards its final destination. A single route table can only have one next hop for any given destination, making it impossible for a single route table to satisfy the requirement to take the same IP frame in opposite directions depending on which subnet or attachment it's coming from.

thanks so much !!

I am trying now.

I removed all the lab and rebuilt it. This time it worked with the original configuration. I haven't figured out yet. I am still confused how those route tables work. This is TGW route table

This service RT is associated with server and client attachments and default route to service VPC.

My understanding is that this table is used when the traffic between the server and the client is forwarded to Service VPC. right ? I do not know how service VPC knows the packet should be directed to firewall interface ge-0/0/0 (untrust). Is it related to how the untrust RT is configured ?

The following is seen from the firewall ec2-user# run show security flow session | refresh 2
---(refreshed at 2024-06-01 07:08:18 UTC)---

Session ID: 4634, Policy name: allow-ssh-to-internal-server/4, Timeout: 2, Session State: Valid

In: 10.2.0.250/32288 --> 10.3.232.173/2008;icmp, Conn Tag: 0x0, If: ge-0/0/0.0, Pkts: 1, Bytes: 84,

Out: 10.3.232.173/2008 --> 10.2.0.250/32288;icmp, Conn Tag: 0x0, If: ge-0/0/1.0, Pkts: 0, Bytes: 0,

The following are the Route tables for trust and untrust interfaces

I am still studying these.

A million thanks to all !!

Right now, I think my question is narrowed down to the following: When the traffic arrives service VPC which has multiple route tables (trust, untrust, mgmt). Each of them has different next hop. How does VPC select which route table to use to direct the traffic ?

Sorry for lots of digressive questions. After a couple of lab tests, I am stuck on the above question.

thanks for any coaching !!

https://aws.amazon.com/blogs/architecture/field-notes-working-with-route-tables-in-aws-transit-gateway/

Good stuff !!