BGP Negotiation over AWS Site-to-Site VPN and Direct Connect: Troubleshooting Strategies for Efficient Networking
Abstract: This article provides a detailed BGP negotiation process over AWS Site-to-Site VPN and Direct Connect connections. It explores the significance of BGP in dynamic routing between on-premises networks and AWS VPCs. The document addresses common BGP-related issues and offers basic to advanced troubleshooting steps. Network Engineers and Admins benefit from this resource, enabling them to maintain efficient networking infrastructures and optimize data transfer with AWS cloud services.
An Overview of BGP
BGP (Border Gateway Protocol) is a widely-used routing protocol that enables the exchange of routing information between different networks on the Internet. BGP is designed to provide highly scalable and reliable routing for large-scale networks and is the de facto standard for inter-domain routing. BGP is a complex protocol that requires a high degree of expertise to configure and manage. In AWS (Amazon Web Services) environments, BGP is used to establish and manage network connectivity between different VPCs (Virtual Private Clouds), as well as between VPCs and on-premises networks.
BGP is particularly important in AWS networking because it provides a flexible and scalable way to manage network traffic and routing. With BGP, network administrators can easily control the flow of traffic between different network segments, ensure high availability and redundancy, and enable seamless connectivity between on-premises and cloud resources.
By using BGP in AWS, organizations can achieve greater flexibility and control over their network architecture, which can lead to improved performance, security, and overall user experience. BGP is used in conjunction with AWS Direct Connect, Transit Gateway Connect and site-to-site VPN are dedicated network connections between a customer's on-premises infrastructure and an AWS Cloud.
This article would cover a range of topics related to the use and implementation of BGP in AWS networking. Some of the key topics we might cover include configuring BGP routing in an AWS environment, best practices for using BGP in AWS, including tips for optimizing routing and achieving high availability, using BGP to enable connectivity between VPCs and on-premises networks and troubleshooting common issues with BGP in AWS, including issues related to BGP establishment, routing and network connectivity.
BGP in AWS VPN
AWS site-to-site VPN is a managed service that enables customers to establish secure and private connections between their on-premises network and an AWS VPC over the public internet. VPN is used to connect an on-premises network to an AWS VPC using IPsec VPN tunnels. BGP is used to exchange routing information between the customer's on-premises network and the AWS VPC over the VPN tunnels. BGP enables the customer to advertise their own IP prefixes or AWS-assigned IP prefixes to the on-premises network, and vice versa. This allows the customer to control the routing of traffic between the on-premises network and the AWS VPC.
Customers can use BGP to advertise their own IP prefixes or AWS-assigned IP prefixes over the Direct Connect connection and site-to-site VPN. BGP also supports route filtering and policy management, allowing customers to control the flow of traffic between their on-premises network and AWS. BGP plays a critical role in enabling customers to establish private and dedicated network connectivity between their on-premises infrastructure and AWS cloud services, providing a highly available and scalable network architecture that meets the demands of modern cloud-based applications and services.
BGP in AWS Direct Connect
Direct Connect is a service provided by Amazon Web Services (AWS) that allows users to establish a dedicated network connection between their on-premises data center and AWS infrastructure. Direct Connect provides a more reliable, consistent, and secure connection compared to an internet-based connection. Direct Connect offers a range of benefits, including lower latency, improved network performance, and increased security.
One of the primary benefits of Direct Connect is lower latency. Direct Connect provides a dedicated connection between a user's on-premises data center and AWS infrastructure, which reduces the number of network hops required to transfer data between the two locations. This can result in faster data transfer speeds and reduced network latency.
Direct Connect also offers increased security compared to internet-based connections. Direct Connect connections are private, dedicated, and encrypted. This means that data transferred over the connection is not exposed to the public internet, which reduces the risk of data breaches and other security threats.
- Border Gateway Protocol (BGP) is used in Direct Connect to exchange routing information between a user's on-premises network and their Virtual Private Clouds (VPCs) in AWS. BGP is a protocol used to exchange routing information between different networks. BGP allows networks to dynamically learn about and advertise routes to other networks.
- Dynamic Routing: BGP is a dynamic routing protocol that allows networks to learn about and advertise routes to other networks in real-time. This means that network changes, such as new VPCs being added or removed, are automatically detected and routes are updated accordingly.
- Efficient Traffic Routing: BGP allows for efficient traffic routing, ensuring that traffic takes the shortest and fastest path between the on-premises data center and AWS. This results in faster data transfer speeds and reduced network latency.
- Redundancy: BGP allows for redundant paths to be created, which ensures that if a primary path fails, traffic can automatically be rerouted to a secondary path. This provides higher availability and reliability for mission-critical applications.
- Granular Control: BGP provides granular control over routing decisions, allowing users to customize routing policies to meet their specific needs. This includes controlling how traffic is routed based on factors such as latency, cost, and network path.
- Authentication:
Security: BGP can be used to establish secure connections between the on-premises data center and AWS by using BGP communities to identify and tag specific routes. This helps ensure that traffic is properly segregated and only flows through authorized paths.
BGP in TGW Connect:
AWS Transit Gateway Connect enables native integration of Software-Defined Wide Area Network (SD-WAN) appliances into AWS. Customers can now seamlessly extend their SD-WAN edge into AWS using standard protocols such as Generic Routing Encapsulation (GRE) and Border Gateway Protocol (BGP). It provides customers with added benefits such as improved bandwidth and supports dynamic routing with increased route limits, thus removing the need to set up multiple IPsec VPNs between the SD-WAN appliances and Transit Gateway. TGW Connect provides several advantages, including:
- Simplified network architecture: TGW Connect enables customers to simplify their network architecture by providing a centralized hub for routing between on-premises networks and VPCs. This reduces the complexity and potential for errors in configuring multiple VPN connections or VPC peering connections.
- Scalability: TGW Connect can scale to support large numbers of VPCs and on-premises networks, making it suitable for large-scale deployments.
- Performance: TGW Connect provides high-speed, low-latency connectivity between VPCs and on-premises networks using AWS's highly available and resilient global network infrastructure.
- Security: TGW Connect supports advanced security features such as AWS Transit Gateway firewall integration and VPC isolation, enabling customers to enforce network segmentation and protect their workloads from external threats.
- Cost-effectiveness: TGW Connect can help reduce networking costs by consolidating multiple connections into a single transit gateway, reducing the need for expensive hardware and reducing data transfer costs.
Troubleshooting BGP in AWS
For troubleshooting a BGP session that can't establish a connection or is in an idle state over a VPN tunnel please go through following troubleshooting steps:
- Check the underlying VPN connection because, BGP session can be established only if the VPN tunnel is up. If the VPN tunnel is down or flapping, you'll experience issues with establishing the BGP session. Make sure that the VPN is up and stable.
- Check the BGP configuration on your customer gateway device and make sure the IP addresses and Autonomous System Numbers (ASN) of the local and remote BGP peers must be configured with the downloaded VPN configuration file.
- If the configuration settings are correct, verify the connectivity between BGP peers by pinging the remote BGP peer IP from your local BGP peer IP. If ping is not working please check the followings:
- If BGP session is flapping between active and connect states, verify that TCP port 179 and other relevant ephemeral ports are not blocked.
Understanding BGP Protocol Negotiation and troubleshoot steps:
Idle: This is the first state where BGP waits for a “start event”. The start event occurs when someone configures a new BGP neighbor or when we reset an established BGP peering. After the start event, BGP will initialize some resources, resets a ConnectRetry timer and initiates a TCP connection to the remote BGP neighbor.
Connect: BGP is waiting for the TCP three-way handshake to complete. When it is successful, it will continue to the OpenSent state. In case it fails, we continue to the Active state. If the ConnectRetry timer expires then we will remain in this state.
TCP Handshake Between BGP peers (CGW side logs):
^58 2021-07-04 22:50:20.699007 169.254.60.146 169.254.60.145 TCP 74 34516 → 179 [SYN] Seq=0 Win=2920 Len=0 MSS=1460 SACK_PERM TSval=3030456 TSecr=0 WS=1
59 2021-07-04 22:50:20.719228 169.254.60.145 169.254.60.146 TCP 74 179 → 34516 [SYN, ACK] Seq=0 Ack=1 Win=26844 Len=0 MSS=1375 TSval=64921081 TSecr=3030456 WS=128
60 2021-07-04 22:50:20.719453 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=1 Ack=1 Win=2920 Len=0 TSval=3030476 TSecr=64921081
61 2021-07-04 22:50:20.719490 169.254.60.146 169.254.60.145 BGP 115 OPEN Message
62 2021-07-04 22:50:20.740519 169.254.60.145 169.254.60.146 TCP 66 179 → 34516 [ACK] Seq=1 Ack=50 Win=26880 Len=0 TSval=64921086 TSecr=3030476
63 2021-07-04 22:50:20.743818 169.254.60.145 169.254.60.146 BGP 138 OPEN Message, KEEPALIVE Message
64 2021-07-04 22:50:20.743918 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=50 Ack=73 Win=2848 Len=0 TSval=3030501 TSecr=64921087
65 2021-07-04 22:50:20.744297 169.254.60.146 169.254.60.145 BGP 85 KEEPALIVE Message
66 2021-07-04 22:50:20.765323 169.254.60.145 169.254.60.146 BGP 85 KEEPALIVE Message
67 2021-07-04 22:50:20.765458 169.254.60.146 169.254.60.145 BGP 89 UPDATE Message
68 2021-07-04 22:50:20.825693 169.254.60.145 169.254.60.146 TCP 66 179 → 34516 [ACK] Seq=92 Ack=92 Win=26880 Len=0 TSval=64921108 TSecr=3030522
69 2021-07-04 22:50:21.765838 169.254.60.145 169.254.60.146 BGP 141 UPDATE Message, UPDATE Message
70 2021-07-04 22:50:21.805586 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=92 Ack=167 Win=2848 Len=0 TSval=3031563 TSecr=64921343
>Frame 58: 74 bytes on wire (592 bits), 74 bytes captured (592 bits)
>Ethernet II, Src: ba:db:ee:fb:ad:04 (ba:db:ee:fb:ad:04), Dst: 00:70:76:69:66:00 (00:70:76:69:66:00)
>Internet Protocol Version 4, Src: 169.254.60.146, Dst: 169.254.60.145
^Transmission Control Protocol, Src Port: 34516, Dst Port: 179, Seq: 0, Len: 0
Source Port: 34516
Destination Port: 179
[Stream index: 57]
[Conversation completeness: Incomplete, DATA (15)]
[TCP Segment Len: 0]
Sequence Number: 0 (relative sequence number)
Sequence Number (raw): 1887105697
[Next Sequence Number: 1 (relative sequence number)]
Acknowledgment Number: 0
Acknowledgment number (raw): 0
1010 .... = Header Length: 40 bytes (10)
Flags: 0x002 (SYN)
Window: 2920
[Calculated window size: 2920]
Checksum: 0x46f5 [unverified]
[Checksum Status: Unverified]
Urgent Pointer: 0
Options: (20 bytes), Maximum segment size, SACK permitted, Timestamps, No-
Operation (NOP), Window scale
[Timestamps]
Active: BGP will try another TCP three-way handshake to establish a connection with the remote BGP neighbor. If it is successful, it will move to the OpenSent state. If the ConnectRetry timer expires then we move back to the Connect state.
OpenSent: After sending an OPEN message to the peer, BGP waits in this state for the OPEN reply. If a successful reply comes in, the BGP state moves to OpenConfirm and a keepalive is sent to the peer. Failure can result in sending the BGP state back to Idle or Active.
CGW Side Peer Sending Open Message:
58 2021-07-04 22:50:20.699007 169.254.60.146 169.254.60.145 TCP 74 34516 → 179 [SYN] Seq=0 Win=2920 Len=0 MSS=1460 SACK_PERM TSval=3030456 TSecr=0 WS=1
59 2021-07-04 22:50:20.719228 169.254.60.145 169.254.60.146 TCP 74 179 → 34516 [SYN, ACK] Seq=0 Ack=1 Win=26844 Len=0 MSS=1375 TSval=64921081 TSecr=3030456 WS=128
60 2021-07-04 22:50:20.719453 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=1 Ack=1 Win=2920 Len=0 TSval=3030476 TSecr=64921081
^61 2021-07-04 22:50:20.719490 169.254.60.146 169.254.60.145 BGP 115 OPEN Message
62 2021-07-04 22:50:20.740519 169.254.60.145 169.254.60.146 TCP 66 179 → 34516 [ACK] Seq=1 Ack=50 Win=26880 Len=0 TSval=64921086 TSecr=3030476
63 2021-07-04 22:50:20.743818 169.254.60.145 169.254.60.146 BGP 138 OPEN Message, KEEPALIVE Message
64 2021-07-04 22:50:20.743918 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=50 Ack=73 Win=2848 Len=0 TSval=3030501 TSecr=64921087
65 2021-07-04 22:50:20.744297 169.254.60.146 169.254.60.145 BGP 85 KEEPALIVE Message
66 2021-07-04 22:50:20.765323 169.254.60.145 169.254.60.146 BGP 85 KEEPALIVE Message
67 2021-07-04 22:50:20.765458 169.254.60.146 169.254.60.145 BGP 89 UPDATE Message
68 2021-07-04 22:50:20.825693 169.254.60.145 169.254.60.146 TCP 66 179 → 34516 [ACK] Seq=92 Ack=92 Win=26880 Len=0 TSval=64921108 TSecr=3030522
69 2021-07-04 22:50:21.765838 169.254.60.145 169.254.60.146 BGP 141 UPDATE Message, UPDATE Message
70 2021-07-04 22:50:21.805586 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=92 Ack=167 Win=2848 Len=0 TSval=3031563 TSecr=64921343
Frame 61: 115 bytes on wire (920 bits), 115 bytes captured (920 bits)
Ethernet II, Src: ba:db:ee:fb:ad:04 (ba:db:ee:fb:ad:04), Dst: 00:70:76:69:66:00 (00:70:76:69:66:00)
Internet Protocol Version 4, Src: 169.254.60.146, Dst: 169.254.60.145
^Transmission Control Protocol, Src Port: 34516, Dst Port: 179, Seq: 1, Ack: 1, Len: 49
Source Port: 34516
Destination Port: 179
[Stream index: 57]
[Conversation completeness: Incomplete, DATA (15)]
[TCP Segment Len: 49]
Sequence Number: 1 (relative sequence number)
Sequence Number (raw): 1887105698
[Next Sequence Number: 50 (relative sequence number)]
Acknowledgment Number: 1 (relative ack number)
Acknowledgment number (raw): 4246846231
1000 .... = Header Length: 32 bytes (8)
Flags: 0x018 (PSH, ACK)
Window: 2920
[Calculated window size: 2920]
[Window size scaling factor: 1]
Checksum: 0x46cb [unverified]
[Checksum Status: Unverified]
Urgent Pointer: 0
Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
[Timestamps]
[SEQ/ACK analysis]
TCP payload (49 bytes)
^Border Gateway Protocol - OPEN Message
Marker: ffffffffffffffffffffffffffffffff
Length: 49
Type: OPEN Message (1)
Version: 4
My AS: 65000
Hold Time: 90
BGP Identifier: 54.241.242.80
Optional Parameters Length: 20
^Optional Parameters
^Optional Parameter: Capability
Parameter Type: Capability (2)
Parameter Length: 18
Capability: Multiprotocol extensions capability
Capability: Route refresh capability
Capability: Route refresh capability (Cisco)
Capability: Graceful Restart capability
AWS Side Peer Acknowledging Open Message:
58 2021-07-04 22:50:20.699007 169.254.60.146 169.254.60.145 TCP 74 34516 → 179 [SYN] Seq=0 Win=2920 Len=0 MSS=1460 SACK_PERM TSval=3030456 TSecr=0 WS=1
59 2021-07-04 22:50:20.719228 169.254.60.145 169.254.60.146 TCP 74 179 → 34516 [SYN, ACK] Seq=0 Ack=1 Win=26844 Len=0 MSS=1375 TSval=64921081 TSecr=3030456 WS=128
60 2021-07-04 22:50:20.719453 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=1 Ack=1 Win=2920 Len=0 TSval=3030476 TSecr=64921081
61 2021-07-04 22:50:20.719490 169.254.60.146 169.254.60.145 BGP 115 OPEN Message
^62 2021-07-04 22:50:20.740519 169.254.60.145 169.254.60.146 TCP 66 179 → 34516 [ACK] Seq=1 Ack=50 Win=26880 Len=0 TSval=64921086 TSecr=3030476
63 2021-07-04 22:50:20.743818 169.254.60.145 169.254.60.146 BGP 138 OPEN Message, KEEPALIVE Message
64 2021-07-04 22:50:20.743918 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=50 Ack=73 Win=2848 Len=0 TSval=3030501 TSecr=64921087
65 2021-07-04 22:50:20.744297 169.254.60.146 169.254.60.145 BGP 85 KEEPALIVE Message
66 2021-07-04 22:50:20.765323 169.254.60.145 169.254.60.146 BGP 85 KEEPALIVE Message
67 2021-07-04 22:50:20.765458 169.254.60.146 169.254.60.145 BGP 89 UPDATE Message
68 2021-07-04 22:50:20.825693 169.254.60.145 169.254.60.146 TCP 66 179 → 34516 [ACK] Seq=92 Ack=92 Win=26880 Len=0 TSval=64921108 TSecr=3030522
69 2021-07-04 22:50:21.765838 169.254.60.145 169.254.60.146 BGP 141 UPDATE Message, UPDATE Message
70 2021-07-04 22:50:21.805586 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=92 Ack=167 Win=2848 Len=0 TSval=3031563 TSecr=64921343
Frame 62: 66 bytes on wire (528 bits), 66 bytes captured (528 bits)
Ethernet II, Src: 06:2e:f7:16:d1:fd (06:2e:f7:16:d1:fd), Dst: 06:6d:ff:ac:b8:63 (06:6d:ff:ac:b8:63)
Internet Protocol Version 4, Src: 169.254.60.145, Dst: 169.254.60.146
^Transmission Control Protocol, Src Port: 179, Dst Port: 34516, Seq: 1, Ack: 50, Len: 0
Source Port: 179
Destination Port: 34516
[Stream index: 57]
[Conversation completeness: Incomplete, DATA (15)]
[TCP Segment Len: 0]
Sequence Number: 1 (relative sequence number)
Sequence Number (raw): 4246846231
[Next Sequence Number: 1 (relative sequence number)]
Acknowledgment Number: 50 (relative ack number)
Acknowledgment number (raw): 1887105747
1000 .... = Header Length: 32 bytes (8)
Flags: 0x010 (ACK)
Window: 210
[Calculated window size: 26880]
[Window size scaling factor: 128]
Checksum: 0x25e5 [unverified]
[Checksum Status: Unverified]
Urgent Pointer: 0
Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
[Timestamps]
[SEQ/ACK analysis]
AWS Side Peer Sending Open Message:
58 2021-07-04 22:50:20.699007 169.254.60.146 169.254.60.145 TCP 74 34516 → 179 [SYN] Seq=0 Win=2920 Len=0 MSS=1460 SACK_PERM TSval=3030456 TSecr=0 WS=1
59 2021-07-04 22:50:20.719228 169.254.60.145 169.254.60.146 TCP 74 179 → 34516 [SYN, ACK] Seq=0 Ack=1 Win=26844 Len=0 MSS=1375 TSval=64921081 TSecr=3030456 WS=128
60 2021-07-04 22:50:20.719453 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=1 Ack=1 Win=2920 Len=0 TSval=3030476 TSecr=64921081
61 2021-07-04 22:50:20.719490 169.254.60.146 169.254.60.145 BGP 115 OPEN Message
62 2021-07-04 22:50:20.740519 169.254.60.145 169.254.60.146 TCP 66 179 → 34516 [ACK] Seq=1 Ack=50 Win=26880 Len=0 TSval=64921086 TSecr=3030476
^63 2021-07-04 22:50:20.743818 169.254.60.145 169.254.60.146 BGP 138 OPEN Message, KEEPALIVE Message
64 2021-07-04 22:50:20.743918 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=50 Ack=73 Win=2848 Len=0 TSval=3030501 TSecr=64921087
65 2021-07-04 22:50:20.744297 169.254.60.146 169.254.60.145 BGP 85 KEEPALIVE Message
66 2021-07-04 22:50:20.765323 169.254.60.145 169.254.60.146 BGP 85 KEEPALIVE Message
67 2021-07-04 22:50:20.765458 169.254.60.146 169.254.60.145 BGP 89 UPDATE Message
68 2021-07-04 22:50:20.825693 169.254.60.145 169.254.60.146 TCP 66 179 → 34516 [ACK] Seq=92 Ack=92 Win=26880 Len=0 TSval=64921108 TSecr=3030522
69 2021-07-04 22:50:21.765838 169.254.60.145 169.254.60.146 BGP 141 UPDATE Message, UPDATE Message
70 2021-07-04 22:50:21.805586 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=92 Ack=167 Win=2848 Len=0 TSval=3031563 TSecr=64921343
Frame 63: 138 bytes on wire (1104 bits), 138 bytes captured (1104 bits)
Ethernet II, Src: 06:2e:f7:16:d1:fd (06:2e:f7:16:d1:fd), Dst: 06:6d:ff:ac:b8:63 (06:6d:ff:ac:b8:63)
Internet Protocol Version 4, Src: 169.254.60.145, Dst: 169.254.60.146
^Transmission Control Protocol, Src Port: 179, Dst Port: 34516, Seq: 1, Ack: 50, Len: 72
Source Port: 179
Destination Port: 34516
[Stream index: 57]
[Conversation completeness: Incomplete, DATA (15)]
[TCP Segment Len: 72]
Sequence Number: 1 (relative sequence number)
Sequence Number (raw): 4246846231
[Next Sequence Number: 73 (relative sequence number)]
Acknowledgment Number: 50 (relative ack number)
Acknowledgment number (raw): 1887105747
1000 .... = Header Length: 32 bytes (8)
Flags: 0x018 (PSH, ACK)
Window: 210
[Calculated window size: 26880]
[Window size scaling factor: 128]
Checksum: 0xfadf [unverified]
[Checksum Status: Unverified]
Urgent Pointer: 0
Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
[Timestamps]
[SEQ/ACK analysis]
TCP payload (72 bytes)
^Border Gateway Protocol - OPEN Message
Marker: ffffffffffffffffffffffffffffffff
Length: 53
Type: OPEN Message (1)
Version: 4
My AS: 64512
Hold Time: 30
BGP Identifier: 169.254.60.145
Optional Parameters Length: 24
^Optional Parameters
Optional Parameter: Capability
Parameter Type: Capability (2)
Parameter Length: 6
Capability: Multiprotocol extensions capability
Optional Parameter: Capability
Parameter Type: Capability (2)
Parameter Length: 2
Capability: Route refresh capability (Cisco)
Optional Parameter: Capability
Parameter Type: Capability (2)
Parameter Length: 2
Capability: Route refresh capability
Optional Parameter: Capability
Parameter Type: Capability (2)
Parameter Length: 6
Capability: Support for 4-octet AS number capability
Border Gateway Protocol - KEEPALIVE Message
Marker: ffffffffffffffffffffffffffffffff
Length: 19
Type: KEEPALIVE Message (4)
CGW Side Peer Acknowledging Open Message:
58 2021-07-04 22:50:20.699007 169.254.60.146 169.254.60.145 TCP 74 34516 → 179 [SYN] Seq=0 Win=2920 Len=0 MSS=1460 SACK_PERM TSval=3030456 TSecr=0 WS=1
59 2021-07-04 22:50:20.719228 169.254.60.145 169.254.60.146 TCP 74 179 → 34516 [SYN, ACK] Seq=0 Ack=1 Win=26844 Len=0 MSS=1375 TSval=64921081 TSecr=3030456 WS=128
60 2021-07-04 22:50:20.719453 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=1 Ack=1 Win=2920 Len=0 TSval=3030476 TSecr=64921081
61 2021-07-04 22:50:20.719490 169.254.60.146 169.254.60.145 BGP 115 OPEN Message
62 2021-07-04 22:50:20.740519 169.254.60.145 169.254.60.146 TCP 66 179 → 34516 [ACK] Seq=1 Ack=50 Win=26880 Len=0 TSval=64921086 TSecr=3030476
63 2021-07-04 22:50:20.743818 169.254.60.145 169.254.60.146 BGP 138 OPEN Message, KEEPALIVE Message
^64 2021-07-04 22:50:20.743918 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=50 Ack=73 Win=2848 Len=0 TSval=3030501 TSecr=64921087
65 2021-07-04 22:50:20.744297 169.254.60.146 169.254.60.145 BGP 85 KEEPALIVE Message
66 2021-07-04 22:50:20.765323 169.254.60.145 169.254.60.146 BGP 85 KEEPALIVE Message
67 2021-07-04 22:50:20.765458 169.254.60.146 169.254.60.145 BGP 89 UPDATE Message
68 2021-07-04 22:50:20.825693 169.254.60.145 169.254.60.146 TCP 66 179 → 34516 [ACK] Seq=92 Ack=92 Win=26880 Len=0 TSval=64921108 TSecr=3030522
69 2021-07-04 22:50:21.765838 169.254.60.145 169.254.60.146 BGP 141 UPDATE Message, UPDATE Message
70 2021-07-04 22:50:21.805586 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=92 Ack=167 Win=2848 Len=0 TSval=3031563 TSecr=64921343
Frame 64: 66 bytes on wire (528 bits), 66 bytes captured (528 bits)
Ethernet II, Src: ba:db:ee:fb:ad:04 (ba:db:ee:fb:ad:04), Dst: 00:70:76:69:66:00 (00:70:76:69:66:00)
Internet Protocol Version 4, Src: 169.254.60.146, Dst: 169.254.60.145
^Transmission Control Protocol, Src Port: 34516, Dst Port: 179, Seq: 50, Ack: 73, Len: 0
Source Port: 34516
Destination Port: 179
[Stream index: 57]
[Conversation completeness: Incomplete, DATA (15)]
[TCP Segment Len: 0]
Sequence Number: 50 (relative sequence number)
Sequence Number (raw): 1887105747
[Next Sequence Number: 50 (relative sequence number)]
Acknowledgment Number: 73 (relative ack number)
Acknowledgment number (raw): 4246846303
1000 .... = Header Length: 32 bytes (8)
Flags: 0x010 (ACK)
Window: 2848
[Calculated window size: 2848]
[Window size scaling factor: 1]
Checksum: 0x1b35 [unverified]
[Checksum Status: Unverified]
Urgent Pointer: 0
Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
[Timestamps]
[SEQ/ACK analysis]
OpenConfirm: The BGP state machine is one step away from reaching its final state (Established). BGP waits in this state for keepalives from the peer. If successful, the state moves to Established; otherwise, the state moves back to Idle based on the errors.
CGW Side Peer Sending Keep-Alive:
58 2021-07-04 22:50:20.699007 169.254.60.146 169.254.60.145 TCP 74 34516 → 179 [SYN] Seq=0 Win=2920 Len=0 MSS=1460 SACK_PERM TSval=3030456 TSecr=0 WS=1
59 2021-07-04 22:50:20.719228 169.254.60.145 169.254.60.146 TCP 74 179 → 34516 [SYN, ACK] Seq=0 Ack=1 Win=26844 Len=0 MSS=1375 TSval=64921081 TSecr=3030456 WS=128
60 2021-07-04 22:50:20.719453 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=1 Ack=1 Win=2920 Len=0 TSval=3030476 TSecr=64921081
61 2021-07-04 22:50:20.719490 169.254.60.146 169.254.60.145 BGP 115 OPEN Message
62 2021-07-04 22:50:20.740519 169.254.60.145 169.254.60.146 TCP 66 179 → 34516 [ACK] Seq=1 Ack=50 Win=26880 Len=0 TSval=64921086 TSecr=3030476
63 2021-07-04 22:50:20.743818 169.254.60.145 169.254.60.146 BGP 138 OPEN Message, KEEPALIVE Message
64 2021-07-04 22:50:20.743918 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=50 Ack=73 Win=2848 Len=0 TSval=3030501 TSecr=64921087
^65 2021-07-04 22:50:20.744297 169.254.60.146 169.254.60.145 BGP 85 KEEPALIVE Message
66 2021-07-04 22:50:20.765323 169.254.60.145 169.254.60.146 BGP 85 KEEPALIVE Message
67 2021-07-04 22:50:20.765458 169.254.60.146 169.254.60.145 BGP 89 UPDATE Message
68 2021-07-04 22:50:20.825693 169.254.60.145 169.254.60.146 TCP 66 179 → 34516 [ACK] Seq=92 Ack=92 Win=26880 Len=0 TSval=64921108 TSecr=3030522
69 2021-07-04 22:50:21.765838 169.254.60.145 169.254.60.146 BGP 141 UPDATE Message, UPDATE Message
70 2021-07-04 22:50:21.805586 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=92 Ack=167 Win=2848 Len=0 TSval=3031563 TSecr=64921343
Frame 65: 85 bytes on wire (680 bits), 85 bytes captured (680 bits)
Ethernet II, Src: ba:db:ee:fb:ad:04 (ba:db:ee:fb:ad:04), Dst: 00:70:76:69:66:00 (00:70:76:69:66:00)
Internet Protocol Version 4, Src: 169.254.60.146, Dst: 169.254.60.145
^Transmission Control Protocol, Src Port: 34516, Dst Port: 179, Seq: 50, Ack: 73, Len: 19
Source Port: 34516
Destination Port: 179
[Stream index: 57]
[Conversation completeness: Incomplete, DATA (15)]
[TCP Segment Len: 19]
Sequence Number: 50 (relative sequence number)
Sequence Number (raw): 1887105747
[Next Sequence Number: 69 (relative sequence number)]
Acknowledgment Number: 73 (relative ack number)
Acknowledgment number (raw): 4246846303
1000 .... = Header Length: 32 bytes (8)
Flags: 0x018 (PSH, ACK)
Window: 2848
[Calculated window size: 2848]
[Window size scaling factor: 1]
Checksum: 0x1707 [unverified]
[Checksum Status: Unverified]
Urgent Pointer: 0
Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
[Timestamps]
[SEQ/ACK analysis]
TCP payload (19 bytes)
Border Gateway Protocol - KEEPALIVE Message
Marker: ffffffffffffffffffffffffffffffff
Length: 19
Type: KEEPALIVE Message (4)
AWS Side Peer Sending Keep-Alive:
58 2021-07-04 22:50:20.699007 169.254.60.146 169.254.60.145 TCP 74 34516 → 179 [SYN] Seq=0 Win=2920 Len=0 MSS=1460 SACK_PERM TSval=3030456 TSecr=0 WS=1
59 2021-07-04 22:50:20.719228 169.254.60.145 169.254.60.146 TCP 74 179 → 34516 [SYN, ACK] Seq=0 Ack=1 Win=26844 Len=0 MSS=1375 TSval=64921081 TSecr=3030456 WS=128
60 2021-07-04 22:50:20.719453 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=1 Ack=1 Win=2920 Len=0 TSval=3030476 TSecr=64921081
61 2021-07-04 22:50:20.719490 169.254.60.146 169.254.60.145 BGP 115 OPEN Message
62 2021-07-04 22:50:20.740519 169.254.60.145 169.254.60.146 TCP 66 179 → 34516 [ACK] Seq=1 Ack=50 Win=26880 Len=0 TSval=64921086 TSecr=3030476
63 2021-07-04 22:50:20.743818 169.254.60.145 169.254.60.146 BGP 138 OPEN Message, KEEPALIVE Message
64 2021-07-04 22:50:20.743918 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=50 Ack=73 Win=2848 Len=0 TSval=3030501 TSecr=64921087
65 2021-07-04 22:50:20.744297 169.254.60.146 169.254.60.145 BGP 85 KEEPALIVE Message
^66 2021-07-04 22:50:20.765323 169.254.60.145 169.254.60.146 BGP 85 KEEPALIVE Message
67 2021-07-04 22:50:20.765458 169.254.60.146 169.254.60.145 BGP 89 UPDATE Message
68 2021-07-04 22:50:20.825693 169.254.60.145 169.254.60.146 TCP 66 179 → 34516 [ACK] Seq=92 Ack=92 Win=26880 Len=0 TSval=64921108 TSecr=3030522
69 2021-07-04 22:50:21.765838 169.254.60.145 169.254.60.146 BGP 141 UPDATE Message, UPDATE Message
70 2021-07-04 22:50:21.805586 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=92 Ack=167 Win=2848 Len=0 TSval=3031563 TSecr=64921343
Frame 66: 85 bytes on wire (680 bits), 85 bytes captured (680 bits)
Ethernet II, Src: 06:2e:f7:16:d1:fd (06:2e:f7:16:d1:fd), Dst: 06:6d:ff:ac:b8:63 (06:6d:ff:ac:b8:63)
Internet Protocol Version 4, Src: 169.254.60.145, Dst: 169.254.60.146
^Transmission Control Protocol, Src Port: 179, Dst Port: 34516, Seq: 73, Ack: 69, Len: 19
Source Port: 179
Destination Port: 34516
[Stream index: 57]
[Conversation completeness: Incomplete, DATA (15)]
[TCP Segment Len: 19]
Sequence Number: 73 (relative sequence number)
Sequence Number (raw): 4246846303
[Next Sequence Number: 92 (relative sequence number)]
Acknowledgment Number: 69 (relative ack number)
Acknowledgment number (raw): 1887105766
1000 .... = Header Length: 32 bytes (8)
Flags: 0x018 (PSH, ACK)
Window: 210
[Calculated window size: 26880]
[Window size scaling factor: 128]
Checksum: 0x213d [unverified]
[Checksum Status: Unverified]
Urgent Pointer: 0
Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
[Timestamps]
[SEQ/ACK analysis]
TCP payload (19 bytes)
^Border Gateway Protocol - KEEPALIVE Message
Marker: ffffffffffffffffffffffffffffffff
Length: 19
Type: KEEPALIVE Message (4)
Established: This is the state in which BGP can exchange information between the peers. The information can be updates, keepalives, or notification.
CGW Side Peer Advertising its routes:
58 2021-07-04 22:50:20.699007 169.254.60.146 169.254.60.145 TCP 74 34516 → 179 [SYN] Seq=0 Win=2920 Len=0 MSS=1460 SACK_PERM TSval=3030456 TSecr=0 WS=1
59 2021-07-04 22:50:20.719228 169.254.60.145 169.254.60.146 TCP 74 179 → 34516 [SYN, ACK] Seq=0 Ack=1 Win=26844 Len=0 MSS=1375 TSval=64921081 TSecr=3030456 WS=128
60 2021-07-04 22:50:20.719453 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=1 Ack=1 Win=2920 Len=0 TSval=3030476 TSecr=64921081
61 2021-07-04 22:50:20.719490 169.254.60.146 169.254.60.145 BGP 115 OPEN Message
62 2021-07-04 22:50:20.740519 169.254.60.145 169.254.60.146 TCP 66 179 → 34516 [ACK] Seq=1 Ack=50 Win=26880 Len=0 TSval=64921086 TSecr=3030476
63 2021-07-04 22:50:20.743818 169.254.60.145 169.254.60.146 BGP 138 OPEN Message, KEEPALIVE Message
64 2021-07-04 22:50:20.743918 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=50 Ack=73 Win=2848 Len=0 TSval=3030501 TSecr=64921087
65 2021-07-04 22:50:20.744297 169.254.60.146 169.254.60.145 BGP 85 KEEPALIVE Message
66 2021-07-04 22:50:20.765323 169.254.60.145 169.254.60.146 BGP 85 KEEPALIVE Message
^67 2021-07-04 22:50:20.765458 169.254.60.146 169.254.60.145 BGP 89 UPDATE Message
68 2021-07-04 22:50:20.825693 169.254.60.145 169.254.60.146 TCP 66 179 → 34516 [ACK] Seq=92 Ack=92 Win=26880 Len=0 TSval=64921108 TSecr=3030522
69 2021-07-04 22:50:21.765838 169.254.60.145 169.254.60.146 BGP 141 UPDATE Message, UPDATE Message
70 2021-07-04 22:50:21.805586 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=92 Ack=167 Win=2848 Len=0 TSval=3031563 TSecr=64921343
Frame 67: 89 bytes on wire (712 bits), 89 bytes captured (712 bits)
Ethernet II, Src: ba:db:ee:fb:ad:04 (ba:db:ee:fb:ad:04), Dst: 00:70:76:69:66:00 (00:70:76:69:66:00)
Internet Protocol Version 4, Src: 169.254.60.146, Dst: 169.254.60.145
^Transmission Control Protocol, Src Port: 34516, Dst Port: 179, Seq: 69, Ack: 92, Len: 23
Source Port: 34516
Destination Port: 179
[Stream index: 57]
[Conversation completeness: Incomplete, DATA (15)]
[TCP Segment Len: 23]
Sequence Number: 69 (relative sequence number)
Sequence Number (raw): 1887105766
[Next Sequence Number: 92 (relative sequence number)]
Acknowledgment Number: 92 (relative ack number)
Acknowledgment number (raw): 4246846322
1000 .... = Header Length: 32 bytes (8)
Flags: 0x018 (PSH, ACK)
Window: 2848
[Calculated window size: 2848]
[Window size scaling factor: 1]
Checksum: 0x18bf [unverified]
[Checksum Status: Unverified]
Urgent Pointer: 0
Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
[Timestamps]
[SEQ/ACK analysis]
TCP payload (23 bytes)
^Border Gateway Protocol - UPDATE Message
Marker: ffffffffffffffffffffffffffffffff
Length: 52
Type: UPDATE Message (2)
Withdrawn Routes Length: 0
Total Path Attribute Length: 26
^Path attributes
Path Attribute - ORIGIN: INCOMPLETE
Path Attribute - AS_PATH: 65000
Path Attribute - NEXT_HOP: 169.254.60.146
^Network Layer Reachability Information (NLRI)
192.168.0.0/16
AWS Side Peer Acknowledging received routes:
58 2021-07-04 22:50:20.699007 169.254.60.146 169.254.60.145 TCP 74 34516 → 179 [SYN] Seq=0 Win=2920 Len=0 MSS=1460 SACK_PERM TSval=3030456 TSecr=0 WS=1
59 2021-07-04 22:50:20.719228 169.254.60.145 169.254.60.146 TCP 74 179 → 34516 [SYN, ACK] Seq=0 Ack=1 Win=26844 Len=0 MSS=1375 TSval=64921081 TSecr=3030456 WS=128
60 2021-07-04 22:50:20.719453 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=1 Ack=1 Win=2920 Len=0 TSval=3030476 TSecr=64921081
61 2021-07-04 22:50:20.719490 169.254.60.146 169.254.60.145 BGP 115 OPEN Message
62 2021-07-04 22:50:20.740519 169.254.60.145 169.254.60.146 TCP 66 179 → 34516 [ACK] Seq=1 Ack=50 Win=26880 Len=0 TSval=64921086 TSecr=3030476
63 2021-07-04 22:50:20.743818 169.254.60.145 169.254.60.146 BGP 138 OPEN Message, KEEPALIVE Message
64 2021-07-04 22:50:20.743918 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=50 Ack=73 Win=2848 Len=0 TSval=3030501 TSecr=64921087
65 2021-07-04 22:50:20.744297 169.254.60.146 169.254.60.145 BGP 85 KEEPALIVE Message
66 2021-07-04 22:50:20.765323 169.254.60.145 169.254.60.146 BGP 85 KEEPALIVE Message
67 2021-07-04 22:50:20.765458 169.254.60.146 169.254.60.145 BGP 89 UPDATE Message
^68 2021-07-04 22:50:20.825693 169.254.60.145 169.254.60.146 TCP 66 179 → 34516 [ACK] Seq=92 Ack=92 Win=26880 Len=0 TSval=64921108 TSecr=3030522
69 2021-07-04 22:50:21.765838 169.254.60.145 169.254.60.146 BGP 141 UPDATE Message, UPDATE Message
70 2021-07-04 22:50:21.805586 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=92 Ack=167 Win=2848 Len=0 TSval=3031563 TSecr=64921343
Frame 68: 66 bytes on wire (528 bits), 66 bytes captured (528 bits)
Ethernet II, Src: 06:2e:f7:16:d1:fd (06:2e:f7:16:d1:fd), Dst: 06:6d:ff:ac:b8:63 (06:6d:ff:ac:b8:63)
Internet Protocol Version 4, Src: 169.254.60.145, Dst: 169.254.60.146
^Transmission Control Protocol, Src Port: 179, Dst Port: 34516, Seq: 92, Ack: 92, Len: 0
Source Port: 179
Destination Port: 34516
[Stream index: 57]
[Conversation completeness: Incomplete, DATA (15)]
[TCP Segment Len: 0]
Sequence Number: 92 (relative sequence number)
Sequence Number (raw): 4246846322
[Next Sequence Number: 92 (relative sequence number)]
Acknowledgment Number: 92 (relative ack number)
Acknowledgment number (raw): 1887105789
1000 .... = Header Length: 32 bytes (8)
Flags: 0x010 (ACK)
Window: 210
[Calculated window size: 26880]
[Window size scaling factor: 128]
Checksum: 0x251c [unverified]
[Checksum Status: Unverified]
Urgent Pointer: 0
Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
[Timestamps]
[SEQ/ACK analysis]
AWS Side Peer Advertising its routes:
58 2021-07-04 22:50:20.699007 169.254.60.146 169.254.60.145 TCP 74 34516 → 179 [SYN] Seq=0 Win=2920 Len=0 MSS=1460 SACK_PERM TSval=3030456 TSecr=0 WS=1
59 2021-07-04 22:50:20.719228 169.254.60.145 169.254.60.146 TCP 74 179 → 34516 [SYN, ACK] Seq=0 Ack=1 Win=26844 Len=0 MSS=1375 TSval=64921081 TSecr=3030456 WS=128
60 2021-07-04 22:50:20.719453 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=1 Ack=1 Win=2920 Len=0 TSval=3030476 TSecr=64921081
61 2021-07-04 22:50:20.719490 169.254.60.146 169.254.60.145 BGP 115 OPEN Message
62 2021-07-04 22:50:20.740519 169.254.60.145 169.254.60.146 TCP 66 179 → 34516 [ACK] Seq=1 Ack=50 Win=26880 Len=0 TSval=64921086 TSecr=3030476
63 2021-07-04 22:50:20.743818 169.254.60.145 169.254.60.146 BGP 138 OPEN Message, KEEPALIVE Message
64 2021-07-04 22:50:20.743918 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=50 Ack=73 Win=2848 Len=0 TSval=3030501 TSecr=64921087
65 2021-07-04 22:50:20.744297 169.254.60.146 169.254.60.145 BGP 85 KEEPALIVE Message
66 2021-07-04 22:50:20.765323 169.254.60.145 169.254.60.146 BGP 85 KEEPALIVE Message
67 2021-07-04 22:50:20.765458 169.254.60.146 169.254.60.145 BGP 89 UPDATE Message
68 2021-07-04 22:50:20.825693 169.254.60.145 169.254.60.146 TCP 66 179 → 34516 [ACK] Seq=92 Ack=92 Win=26880 Len=0 TSval=64921108 TSecr=3030522
^69 2021-07-04 22:50:21.765838 169.254.60.145 169.254.60.146 BGP 141 UPDATE Message, UPDATE Message
70 2021-07-04 22:50:21.805586 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=92 Ack=167 Win=2848 Len=0 TSval=3031563 TSecr=64921343
Frame 69: 141 bytes on wire (1128 bits), 141 bytes captured (1128 bits)
Ethernet II, Src: 06:2e:f7:16:d1:fd (06:2e:f7:16:d1:fd), Dst: 06:6d:ff:ac:b8:63 (06:6d:ff:ac:b8:63)
Internet Protocol Version 4, Src: 169.254.60.145, Dst: 169.254.60.146
^Transmission Control Protocol, Src Port: 179, Dst Port: 34516, Seq: 92, Ack: 92, Len: 75
Source Port: 179
Destination Port: 34516
[Stream index: 57]
[Conversation completeness: Incomplete, DATA (15)]
[TCP Segment Len: 75]
Sequence Number: 92 (relative sequence number)
Sequence Number (raw): 4246846322
[Next Sequence Number: 167 (relative sequence number)]
Acknowledgment Number: 92 (relative ack number)
Acknowledgment number (raw): 1887105789
1000 .... = Header Length: 32 bytes (8)
Flags: 0x018 (PSH, ACK)
Window: 210
[Calculated window size: 26880]
[Window size scaling factor: 128]
Checksum: 0x1a09 [unverified]
[Checksum Status: Unverified]
Urgent Pointer: 0
Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
[Timestamps]
[SEQ/ACK analysis]
TCP payload (75 bytes)
Border Gateway Protocol - UPDATE Message
Marker: ffffffffffffffffffffffffffffffff
Length: 52
Type: UPDATE Message (2)
Withdrawn Routes Length: 0
Total Path Attribute Length: 26
^Path attributes
Path Attribute - ORIGIN: IGP
Path Attribute - AS_PATH: 64512
Path Attribute - NEXT_HOP: 169.254.60.145
^Network Layer Reachability Information (NLRI)
172.17.0.0/16
CGW Side Peer Acknowledging received routes:
58 2021-07-04 22:50:20.699007 169.254.60.146 169.254.60.145 TCP 74 34516 → 179 [SYN] Seq=0 Win=2920 Len=0 MSS=1460 SACK_PERM TSval=3030456 TSecr=0 WS=1
59 2021-07-04 22:50:20.719228 169.254.60.145 169.254.60.146 TCP 74 179 → 34516 [SYN, ACK] Seq=0 Ack=1 Win=26844 Len=0 MSS=1375 TSval=64921081 TSecr=3030456 WS=128
60 2021-07-04 22:50:20.719453 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=1 Ack=1 Win=2920 Len=0 TSval=3030476 TSecr=64921081
61 2021-07-04 22:50:20.719490 169.254.60.146 169.254.60.145 BGP 115 OPEN Message
62 2021-07-04 22:50:20.740519 169.254.60.145 169.254.60.146 TCP 66 179 → 34516 [ACK] Seq=1 Ack=50 Win=26880 Len=0 TSval=64921086 TSecr=3030476
63 2021-07-04 22:50:20.743818 169.254.60.145 169.254.60.146 BGP 138 OPEN Message, KEEPALIVE Message
64 2021-07-04 22:50:20.743918 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=50 Ack=73 Win=2848 Len=0 TSval=3030501 TSecr=64921087
65 2021-07-04 22:50:20.744297 169.254.60.146 169.254.60.145 BGP 85 KEEPALIVE Message
66 2021-07-04 22:50:20.765323 169.254.60.145 169.254.60.146 BGP 85 KEEPALIVE Message
67 2021-07-04 22:50:20.765458 169.254.60.146 169.254.60.145 BGP 89 UPDATE Message
68 2021-07-04 22:50:20.825693 169.254.60.145 169.254.60.146 TCP 66 179 → 34516 [ACK] Seq=92 Ack=92 Win=26880 Len=0 TSval=64921108 TSecr=3030522
69 2021-07-04 22:50:21.765838 169.254.60.145 169.254.60.146 BGP 141 UPDATE Message, UPDATE Message
^70 2021-07-04 22:50:21.805586 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=92 Ack=167 Win=2848 Len=0 TSval=3031563 TSecr=64921343
Frame 70: 66 bytes on wire (528 bits), 66 bytes captured (528 bits)
Ethernet II, Src: ba:db:ee:fb:ad:04 (ba:db:ee:fb:ad:04), Dst: 00:70:76:69:66:00 (00:70:76:69:66:00)
Internet Protocol Version 4, Src: 169.254.60.146, Dst: 169.254.60.145
^Transmission Control Protocol, Src Port: 34516, Dst Port: 179, Seq: 92, Ack: 167, Len: 0
Source Port: 34516
Destination Port: 179
[Stream index: 57]
[Conversation completeness: Incomplete, DATA (15)]
[TCP Segment Len: 0]
Sequence Number: 92 (relative sequence number)
Sequence Number (raw): 1887105789
[Next Sequence Number: 92 (relative sequence number)]
Acknowledgment Number: 167 (relative ack number)
Acknowledgment number (raw): 4246846397
1000 .... = Header Length: 32 bytes (8)
Flags: 0x010 (ACK)
Window: 2848
[Calculated window size: 2848]
[Window size scaling factor: 1]
Checksum: 0x1587 [unverified]
[Checksum Status: Unverified]
Urgent Pointer: 0
Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
[Timestamps]
[SEQ/ACK analysis]
Both Peers exchanging and acknowledging Keepalives:
58 2021-07-04 22:50:20.699007 169.254.60.146 169.254.60.145 TCP 74 34516 → 179 [SYN] Seq=0 Win=2920 Len=0 MSS=1460 SACK_PERM TSval=3030456 TSecr=0 WS=1
59 2021-07-04 22:50:20.719228 169.254.60.145 169.254.60.146 TCP 74 179 → 34516 [SYN, ACK] Seq=0 Ack=1 Win=26844 Len=0 MSS=1375 TSval=64921081 TSecr=3030456 WS=128
60 2021-07-04 22:50:20.719453 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=1 Ack=1 Win=2920 Len=0 TSval=3030476 TSecr=64921081
61 2021-07-04 22:50:20.719490 169.254.60.146 169.254.60.145 BGP 115 OPEN Message
62 2021-07-04 22:50:20.740519 169.254.60.145 169.254.60.146 TCP 66 179 → 34516 [ACK] Seq=1 Ack=50 Win=26880 Len=0 TSval=64921086 TSecr=3030476
63 2021-07-04 22:50:20.743818 169.254.60.145 169.254.60.146 BGP 138 OPEN Message, KEEPALIVE Message
64 2021-07-04 22:50:20.743918 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=50 Ack=73 Win=2848 Len=0 TSval=3030501 TSecr=64921087
65 2021-07-04 22:50:20.744297 169.254.60.146 169.254.60.145 BGP 85 KEEPALIVE Message
66 2021-07-04 22:50:20.765323 169.254.60.145 169.254.60.146 BGP 85 KEEPALIVE Message
67 2021-07-04 22:50:20.765458 169.254.60.146 169.254.60.145 BGP 89 UPDATE Message
68 2021-07-04 22:50:20.825693 169.254.60.145 169.254.60.146 TCP 66 179 → 34516 [ACK] Seq=92 Ack=92 Win=26880 Len=0 TSval=64921108 TSecr=3030522
69 2021-07-04 22:50:21.765838 169.254.60.145 169.254.60.146 BGP 141 UPDATE Message, UPDATE Message
70 2021-07-04 22:50:21.805586 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=92 Ack=167 Win=2848 Len=0 TSval=3031563 TSecr=64921343
71 2021-07-04 22:50:29.871032 169.254.60.146 169.254.60.145 BGP 85 KEEPALIVE Message
72 2021-07-04 22:50:29.891713 169.254.60.145 169.254.60.146 TCP 66 179 → 34516 [ACK] Seq=167 Ack=111 Win=26880 Len=0 TSval=64923374 TSecr=3039628
73 2021-07-04 22:50:30.742335 169.254.60.145 169.254.60.146 BGP 85 KEEPALIVE Message
74 2021-07-04 22:50:30.742464 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=111 Ack=186 Win=2848 Len=0 TSval=3040499 TSecr=64923587
75 2021-07-04 22:50:37.437250 169.254.60.146 169.254.60.145 BGP 85 KEEPALIVE Message
76 2021-07-04 22:50:37.458148 169.254.60.145 169.254.60.146 TCP 66 179 → 34516 [ACK] Seq=186 Ack=130 Win=26880 Len=0 TSval=64925266 TSecr=3047194
77 2021-07-04 22:50:40.745810 169.254.60.145 169.254.60.146 BGP 85 KEEPALIVE Message
78 2021-07-04 22:50:40.745915 169.254.60.146 169.254.60.145 TCP 66 34516 → 179 [ACK] Seq=130 Ack=205 Win=2848 Len=0 TSval=3050503 TSecr=64926087
Advanced troubleshooting:
1. Ping between BGP peers works but session is not establishing over an AWS site-to-site VPN connection even though all BGP setting are configured correctly on both ends:
External BGP (EBGP) multi-hop is disabled on AWS so, if BGP peers are not directly connected to each other BGP session will not establish. To check whether peers are directly connected, run the ping test with TTL value 1 and if ping works then peers are directly connected and if not, multi-hops are present in between which is not supported on AWS side.
Normal ping test is working:
CSR#ping 169.254.60.145
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 169.254.60.145, timeout is 2 seconds:
! ! ! ! !
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 m
Ping with TTL value 1 is not working:
CSR#ping -1 1 169:254.60.145
Type escape sequence to abort
Sending 5, 100-byte ICMP Echos to 169.254.60.145, timeout is 2 seconds:
....
Success rate is 0 percent (0/5)
2. BGP Keep Flapping Due to "Hold Timer Expired:
Hold timer expiry is a common cause for flapping BGP peer. It means that the router didn’t receive or process a keepalive messages or any update message before the hold timer expired. So, it sends a notification message (4/0) and closes the session. On IOS, the keepalive messages are sent by the BGP I/O process, and the BGP router process interprets the incoming keepalive messages. BGP flaps due to hold timer expiry can be caused by one of the following reasons:
A. WAN Interface issues: Various interface issues such as a physical layer concern or drops on the WAN interface can lead to a BGP session flapping because of hold time expiry. Performing MTR and traceroute over public path (internet) and private path (VPN) would help to identify the losses.
B. Input hold queue: Check if packets are arriving to the CGW but dropped in the input hold queue of the incoming interface (WAN) and usually these packets are intended to be processed by router’s CPU utilization. The hold-queue size is a finite size and most of the CGW’s default input hold queue size value is 75 packets and can be configured to higher value to resolve the issue.
Check current interface Hold Queue and drops:
CSR#show interface GigabitEthernet1
GigabitEtherneti is up, line protocol is up
Hardware is CSR vNIC, address is 02b7.159d.784a (bia 02b7.159d.784a)
Internet address is 172.16.1.186/24
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full Duplex, 1000Mbps, link type is auto, media type is Virtual
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 93/75/15/3 (size/max/drops/flushes) Total output drops: 153
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 2000 bits/sec, 5 packets/sec
5 minute output rate 3000 bits/sec, 3 packets/sec
3502 packets input, 240805 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
9 watchdog, 0 multicast, 0 pause input
2805 packets output, 289428 bytes, 0 underruns
Output 0 broadcasts (0 IP multicasts)
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
Configure input Hold with desired value:
CSR#config terminal
CSR(config)#interface GigabitEthernet1
CSR(config-if)#hold-queue 1500 in
CSR(config-if)#exit
C. TCP receive queue and BGP InQ: Check if BGP keepalive arrived at the TCP queue but are not being processed and moved to the BGP InQ due to huge TCP messages are already waiting in the queue. If so, BGP session would flap with an error “hold time expiry” since BGP I/O process do not get a chance to run, BGP I/O process is in charge of putting messages from TCP receiving queue into BGP InQ and these usually happens when BGP hold timer values are very low, there are many neighbors and CPU is running high.
Check receive queue and BGP InQ:
CSR#show bgp ipv4 unicast summary
BGP router identifier 169.254.60.144, local AS number 65000
BGP table version is 4, main routing table version 4
3 network entries using 744 bytes of memory
5 path entries using 680 bytes of memory
2/2 BGP path/bestpath attribute entries using 576 bytes of memory
1 BGP AS-PATH entries using 24 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 2024 total bytes of memory
BGP activity 3/0 prefixes, 5/0 paths, scan interval 60 secs
3 networks peaked at 00:26:52 Aug 11 2021 UTC (01:20:49.049 ago)
Neighbor V AS MsgRcvd MsgSent TbIver IngQ OutQ Up/Down State/PfRed
169.254.60.145 4 64512 259453 157389 8701 207 0 01:20:49 2
- The solution would be reduce the CPU utilization by disabling unnecessary running tasks and increase the BGP Hold timer to appropriate value.
- Also, some customer gateway devices are offering unique features, instead of queueing data once a second, BGP now queues data aggressively from the BGP OutQ to the TCP socket for each peer until the OutQs have drained completely. Since BGP now sends at a faster rate, BGP converges more quickly.
D. Maximum Transmission Unit (MTU) mismatch: MTU mismatch issue cause BGP session to flap. BGP sends updates based on the maximum segment size (MSS) value calculated by TCP. If Path-MTU-Discovery (PMTUD) is not enabled and the destination is remote, the BGP MSS value defaults to 536 bytes as defined in RFC 879. So, if there are a huge number of updates getting exchanged between the two routers at the MSS value of 536 bytes, convergence issues are detected, which cause inefficient use of the network.
Check Current MTU configuration:
CSR#show ip bp neighbors | include max data
Datagrams (max data segment is 536 bytes):
Datagrams (max data segment is 536 bytes):
The solution is to enable the Path MTU (PMTU) feature and use it to dynamically determine how large the MSS value can be without creating packets that need to be fragmented. PMTU allows TCP to determine the smallest MTU size among all links in a TCP session.
Enable Path MTU:
CS#configure terminal
CSR(config)#interface GigabitEthernet1
CSR(config-if)#ip top path-mtu-discovery
CSR(config)#exit.
CSR#show ip bgp neighbors | include max data
Datagrams (max data segment is 1436 bytes):
Datagrams (max data segment is 1436 bytes):
Conclusion
In conclusion, understanding the BGP negotiation process over AWS Site-to-Site VPN and Direct Connect connections is crucial for building and maintaining efficient networking infrastructures in the cloud. By employing BGP for dynamic routing, enterprises can optimize data transfer and ensure resilient connectivity between on-premises networks and AWS VPCs. Additionally, this paper's comprehensive troubleshooting steps offer valuable insights for addressing common BGP-related challenges, empowering network administrators and cloud practitioners to swiftly resolve issues and maintain seamless data flow. Embracing these strategies will enhance the overall performance and reliability of AWS-based networks, enabling organizations to leverage the full potential of cloud services while delivering an exceptional user experience.
References:
Relevant content
asked 7 months ago
AWS OFFICIALUpdated 5 months ago- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 7 months ago