Cross-AZ Cost Reduction in ALB --> TGW --> NLB Architecture

When optimizing your multi-account architecture to reduce cross-AZ data transfer costs, here's what you need to know:

1. TGW Zonal Affinity Confirmation: Yes, Transit Gateway does maintain zonal affinity by default. Traffic entering TGW in AZ-A will exit to the same AZ in the destination VPC when possible. This means that traffic from your ALB in AZ-A will be routed to the NLB's ENI in AZ-A, assuming there's a TGW attachment in that AZ. This behavior helps minimize cross-AZ data transfer costs at the TGW layer.

2. NLB Downside Validation: Disabling cross-zone load balancing on your NLB will indeed eliminate the additional cross-AZ data transfer costs at the NLB layer. The primary operational downside is that each NLB node will only distribute traffic to targets within its own AZ. This means:

• If one AZ has fewer targets than another (due to scaling events or instance failures), the targets in that AZ will receive proportionally more traffic per instance.
• If all targets in one AZ become unhealthy, the NLB node in that AZ won't be able to route traffic to healthy targets in other AZs, potentially causing service disruption for clients connected to that node.

Your assumption about Round Robin distribution is partially correct - it will prevent hot-spotting among targets within the same AZ, but won't help balance load across AZs if they have uneven numbers of targets.

3. Best Practice Guidance: For your specific multi-VPC, multi-load balancer scenario, consider these optimization approaches:

• Keep cross-zone load balancing enabled on your public-facing ALB to ensure high availability, but disable it on your internal NLB to reduce costs.
• Ensure your Auto Scaling Group maintains balanced distribution of instances across AZs to mitigate the impact of disabling cross-zone load balancing on the NLB.
• Monitor the health and traffic distribution across your AZs regularly to ensure balanced performance.
• Consider using AWS Cost Explorer to analyze your data transfer costs and identify optimization opportunities.
• For programmatic advertising or high-volume traffic scenarios, co-locating components within the same AZ can significantly reduce cross-AZ data transfer costs.

Remember that there's always a trade-off between cost optimization and high availability. Your specific approach should be based on your application's resilience requirements and traffic patterns.
Sources
Optimizing ad exchange infrastructure: Sharethrough’s journey from ALB to NLB | AWS for Industries
ADVCOST03-BP02 When integrating SSPs and DSPs for programmatic advertising, co-locate the platforms - Video Streaming Advertising Lens
Cost optimization pillar - Hybrid Networking Lens
Conditional cross-zone loadbalancing | AWS re:Post

Yes, Transit Gateway keeps traffic in the same Availability Zone when you have attachments in each AZ. That means traffic from an ALB node in AZ A into the TGW will be routed out of the TGW into the Prod VPC in AZ A, so you avoid a second cross AZ transfer at the NLB side. You will still pay Transit Gateway data processing and any other TGW related charges.

If you disable cross zone on the NLB you can reduce NLB side cross AZ transfer costs. The tradeoffs are that each NLB node will only forward to targets in its own AZ, which requires you to make sure each AZ has enough targets. If one AZ gets more client connections than the targets it contains those targets can get overloaded. With cross zone enabled nodes can spread traffic across all targets and failover is smoother. To go without cross zone you must size and monitor per AZ capacity, test AZ failures, and have fast scaling or health checks to recover.

NeXT steps you can take right away

1. Confirm per AZ TGW attachments exist in both Network and Prod VPCs. TGW needs a subnet in each AZ to keep traffic zonal.
2. Validate with a small test. Send traffic from the ALB to the NLB while capturing VPC Flow Logs and CloudWatch metrics to prove traffic stays within the same AZ. Look at source and destination ENI ids and AZ tags.
3. If cost is the priority, consider turning off NLB cross zone after you verify TGW is keeping traffic zonal. Then make sure each ASG has enough instances in each AZ. Set health checks and scaling policies so a busy AZ can scale out quickly.
4. Run an AZ-failure test to confirm behavior. Simulate or remove an AZ and confirm clients failover in an acceptable way.
5. Revisit the NLB target group attribute about unhealthy connections and test whether connection termination behavior affects your failover or connection drain. Do not change that setting in production without testing.
6. Model costs for your actual traffic. Compare current TGW plus NLB costs versus TGW with NLB cross zone off, PrivateLink, co-locating ALB and NLB in the same VPC, or VPC sharing. TGW has per GB data processing charges that matter at scale. PrivateLink has endpoint hourly and data costs that may or may not be cheaper depending on volume.

Alternatives to consider and their tradeoffs PrivateLink offers private endpoints that are created in each AZ and can keep traffic local, and it removes the TGW hop for that path. PrivateLink has its own pricing so compare numbers for your expected GBs per month. Co-locating the ALB and the backend NLB in the same VPC or using VPC sharing removes TGW data processing charges entirely, but that may not fit your account separation and security requirements. VPC Lattice is another option when you want managed service level connectivity across accounts and VPCs at the application layer. Evaluate these if TGW costs plus operational complexity become significant.

Operational checklist before you flip any settings

1. Verify TGW has attachments in every AZ used by your ALB and NLB.
2. Capture flow logs and metrics to prove traffic path and cost behaviour.
3. Make sure ASGs are balanced per AZ and can scale quickly.
4. Disable NLB cross zone in a test or canary environment first and monitor per AZ target CPU, request rates, latency, and error rates.
5. Run an AZ outage test and validate real client experience.
6. Use real traffic numbers to compare costs between your current architecture, PrivateLink, and co-location.

Short final recommendation If you want the lowest risk path with some cost savings, validate TGW AZ locality with tests, then disable NLB cross zone in a controlled environment while ensuring even per AZ capacity and good autoscaling. If you want to remove TGW processing costs entirely, evaluate co-location or PrivateLink and run a cost comparison using your expected data volumes.

Note however, @Sonic, that VPC interface endpoints continue to charge a data processing fee for all data passing through them, whether or not an AZ boundary is crossed. The cost that was eliminated was for cross-AZ traffic needed for a client to reach their VPC interface endpoint in a different AZ. "Traffic processing" continues to be charged for all data passing through the VPC interface endpoint to the NLB on the service provider VPC's side.

The cheapest alternative would be to use Resource Access Manager (RAM) to share subnets from the production account with the network account. The network account would place the ALB in the shared subnets. This way, the ALB and NLB would be in the same VPC, despite residing in different accounts. Cross-AZ traffic fees aren't charged between an ALB and its target, because ALBs don't support preventing cross-AZ load balancing.

Another alternative would be to establish a VPC peering connection between the production and network VPCs. This may feel simpler for operators unfamiliar with seeing someone else's subnets in their account. However, cross-AZ traffic will be charged for all traffic passing through the VPC peering connection to a different AZ, even when the traffic is passing through an ALB. Traffic within an AZ over the VPC peering connection is free, the same as within a VPC. VPC peering connections are "set it and forget it" types of constructs, requiring no monitoring or maintenance once created, and they are fully highly available with requiring any additional configuration or design.

Like a VPC interface endpoint, a Transit Gateway also charges a fee for all traffic passing through it, regardless of which AZs the source and destination are in. The fee that was eliminated was the cross-AZ data "transfer" (not "processing") charge between a resource and the TGW attachment. For example, if data from a server or container in AZ 1 in the production account arrived in the network account through a TGW attachment ENI in AZ 1, and the target ENI of the ALB in the network account resided in AZ 3, this cross-AZ data transfer is now free, but the TGW still continues to charge $0.02/GB for every byte either the production or network account sends to the TGW.