Intermittent session drops through a stateful firewall after enabling a second Local Gateway BGP peer on AWS Outposts

Short description

You run AWS Outposts behind a stateful firewall and enable both Local Gateway (LGW) uplinks for redundancy. Long-lived sessions (RDP, SMB, persistent HTTP, DNS over TCP) begin failing intermittently. With only one uplink active, everything works. Routing tables look stable, with no BGP flapping and no link errors.

The cause is an asymmetric return path. When both uplinks are up and your firewall advertises your on-premises prefixes to both with an equal AS-path length, the Outpost sees two equal-cost return paths and performs per-flow ECMP across them. Your firewall sends a session out one uplink and registers the session on that interface, but the Outpost returns some flows on the other uplink. Those return packets arrive on an interface the session table does not expect, fail the stateful inspection check, and are dropped.

Disabling the second uplink restores symmetry and appears to fix the issue, which misleads operators into treating the second uplink as faulty. In reality, nothing is wrong with the second uplink; the asymmetric return path is what breaks the sessions.

The following diagram shows both states side by side: the asymmetric flow that causes the intermittent drops, and the symmetric flow that results once the fix is applied.

Figure 1: Return-path asymmetry through a stateful firewall with redundant Local Gateway uplinks, and the AS-path prepend that restores a single symmetric path.

Resolution

Work through the following steps in order: confirm the asymmetry, apply the prepend on the backup peer, verify the advertised routes, and validate failover before you call it done.

Step 1: Confirm the asymmetry

Two checks separate this from the causes it is often mistaken for (flapping, MTU, multipath tuning).

Rule out route flapping. Compare the best path and its uptime for a VM prefix in the working (one uplink) and failing (two uplinks) states. If the best path and uptime are unchanged when the second uplink comes up, the prefixes are stable and flapping is not the cause.

Confirm the split directly. Capture on both LGW uplink sub-interfaces at once, filtered to one failing session's 5-tuple. On a Cisco device:

capture asym1 interface <UPLINK1_SUBIF> match tcp host <client-ip> host <vm-ip>
capture asym2 interface <UPLINK2_SUBIF> match tcp host <client-ip> host <vm-ip>
show capture asym1
show capture asym2

If the SYN leaves on one uplink and the SYN-ACK returns on the other, the asymmetry is confirmed.

Step 2: Apply an outbound AS-path prepend on the backup LGW peer

Make the Outpost prefer one uplink for all return traffic by advertising your on-premises prefixes with a longer AS-path on the backup peer.

One prepend is enough to break a simple equal-length tie: it makes the backup path one AS hop longer, so the Outpost stops seeing the two paths as equal and settles on the primary. Three is a common convention because it keeps the backup reliably less preferred even when other autonomous systems between the Outpost and your firewall add their own path length, and it reads clearly as a deliberate choice rather than an accident. Use one for a straightforward primary and backup pair, and use three when there are extra hops or ASes in the path.

Cisco IOS / FTD:

route-map BACKUP-PREPEND-OUT permit 10
 set as-path prepend <own-ASN> <own-ASN> <own-ASN>
!
router bgp <own-ASN>
 address-family ipv4 unicast
  neighbor <backup-LGW-peer-IP> route-map BACKUP-PREPEND-OUT out

FortiOS:

config router route-map
    edit "BACKUP-PREPEND-OUT"
        config rule
            edit 1
                set set-aspath "<own-ASN>" "<own-ASN>" "<own-ASN>"
            next
        end
    next
end
config router bgp
    config neighbor
        edit "<backup-LGW-peer-IP>"
            set route-map-out "BACKUP-PREPEND-OUT"
        next
    end
end

Palo Alto (PAN-OS):

PAN-OS applies the prepend through a BGP export rule; import rules do not support prepend. On the virtual router, go to Network > Virtual Routers > [your VR] > BGP > Export, add a rule scoped to the backup LGW peer group, and on the Action tab set AS Path to Prepend with a value of 3. On firewalls running the Advanced Routing engine, apply the same prepend through a BGP route-map bound to the backup peer's outbound policy.

Then soft-clear the session outbound so the new AS-path is advertised immediately (clear ip bgp <backup-LGW-peer-IP> soft out on IOS, execute router clear bgp ip <backup-LGW-peer-IP> soft out on FortiOS).

Important: if your firewall already applies an outbound route-map to the LGW peers (for prefix filtering, communities, or other attributes), do not replace it. Copy it, add the prepend to the copy, and apply the copy to the backup peer only. Replacing an existing outbound policy is the most common way a prepend accidentally withdraws prefixes and takes the link down.

Step 3: Verify

• Both BGP sessions remain established.
• The advertised-routes list to the backup peer shows your prefixes with the extended AS-path, and contains the same prefixes as before the change. If any prefix is missing versus the pre-change baseline, roll back.

show bgp neighbors <backup-LGW-peer-IP> advertised-routes

• All Outpost VM return routes resolve to the primary uplink's next-hop.
• The intermittent drops stop.

Step 4: Validate failover

Shut down the primary LGW peer. Confirm the Outpost converges onto the backup uplink and traffic recovers. New sessions use the backup immediately; existing sessions re-establish after convergence. Bring the primary back and confirm preference returns.

Rollback

Remove the route-map from the neighbor (or restore the original), then soft-clear. The change is reversible in seconds, which makes it safe to apply in a short maintenance window.

Deploying at multiple sites

The configuration is identical at any site with the same topology (stateful firewall plus redundant LGW uplinks). Only the local peer IP and ASN change.

Alternatives

• MED achieves the same result with a cleaner failure mode (a misconfigured MED tends to have no effect rather than withdrawing prefixes) when both LGW peers are in the same ASN.
• A more-specific prefix on the preferred uplink pins return traffic by longest match.

A note on firewall-side "fixes"

Enabling asymmetric routing on the firewall (set asymroute enable on FortiGate, asymmetric-path bypass on Palo Alto, TCP state bypass on FTD) also stops the drops, but it does so by relaxing stateful inspection. In regulated environments this is typically prohibited, and correctly so. The BGP-side fix preserves full inspection and is the recommended approach.

Related information

• AWS Outposts High Availability Design and Architecture Considerations (see the Networking section)
• AWS Outposts rack network requirements (covers the service link and firewall rules)