BGP Negotiation over AWS Site-to-Site VPN and Direct Connect: Troubleshooting Strategies for Efficient Networking

36 分的閱讀內容
內容層級:專家
1

Abstract: This article provides a detailed BGP negotiation process over AWS Site-to-Site VPN and Direct Connect connections. It explores the significance of BGP in dynamic routing between on-premises networks and AWS VPCs. The document addresses common BGP-related issues and offers basic to advanced troubleshooting steps. Network Engineers and Admins benefit from this resource, enabling them to maintain efficient networking infrastructures and optimize data transfer with AWS cloud services.

An Overview of BGP

BGP (Border Gateway Protocol) is a widely-used routing protocol that enables the exchange of routing information between different networks on the Internet. BGP is designed to provide highly scalable and reliable routing for large-scale networks and is the de facto standard for inter-domain routing. BGP is a complex protocol that requires a high degree of expertise to configure and manage. In AWS (Amazon Web Services) environments, BGP is used to establish and manage network connectivity between different VPCs (Virtual Private Clouds), as well as between VPCs and on-premises networks.

BGP is particularly important in AWS networking because it provides a flexible and scalable way to manage network traffic and routing. With BGP, network administrators can easily control the flow of traffic between different network segments, ensure high availability and redundancy, and enable seamless connectivity between on-premises and cloud resources.

By using BGP in AWS, organizations can achieve greater flexibility and control over their network architecture, which can lead to improved performance, security, and overall user experience. BGP is used in conjunction with AWS Direct Connect, Transit Gateway Connect and site-to-site VPN are dedicated network connections between a customer's on-premises infrastructure and an AWS Cloud.

This article would cover a range of topics related to the use and implementation of BGP in AWS networking. Some of the key topics we might cover include configuring BGP routing in an AWS environment, best practices for using BGP in AWS, including tips for optimizing routing and achieving high availability, using BGP to enable connectivity between VPCs and on-premises networks and troubleshooting common issues with BGP in AWS, including issues related to BGP establishment, routing and network connectivity.

BGP in AWS VPN

Enter image description here

AWS site-to-site VPN is a managed service that enables customers to establish secure and private connections between their on-premises network and an AWS VPC over the public internet. VPN is used to connect an on-premises network to an AWS VPC using IPsec VPN tunnels. BGP is used to exchange routing information between the customer's on-premises network and the AWS VPC over the VPN tunnels. BGP enables the customer to advertise their own IP prefixes or AWS-assigned IP prefixes to the on-premises network, and vice versa. This allows the customer to control the routing of traffic between the on-premises network and the AWS VPC.

Customers can use BGP to advertise their own IP prefixes or AWS-assigned IP prefixes over the Direct Connect connection and site-to-site VPN. BGP also supports route filtering and policy management, allowing customers to control the flow of traffic between their on-premises network and AWS. BGP plays a critical role in enabling customers to establish private and dedicated network connectivity between their on-premises infrastructure and AWS cloud services, providing a highly available and scalable network architecture that meets the demands of modern cloud-based applications and services.

BGP in AWS Direct Connect

Enter image description here

Direct Connect is a service provided by Amazon Web Services (AWS) that allows users to establish a dedicated network connection between their on-premises data center and AWS infrastructure. Direct Connect provides a more reliable, consistent, and secure connection compared to an internet-based connection. Direct Connect offers a range of benefits, including lower latency, improved network performance, and increased security.

One of the primary benefits of Direct Connect is lower latency. Direct Connect provides a dedicated connection between a user's on-premises data center and AWS infrastructure, which reduces the number of network hops required to transfer data between the two locations. This can result in faster data transfer speeds and reduced network latency.
Direct Connect also offers increased security compared to internet-based connections. Direct Connect connections are private, dedicated, and encrypted. This means that data transferred over the connection is not exposed to the public internet, which reduces the risk of data breaches and other security threats.

  1. Border Gateway Protocol (BGP) is used in Direct Connect to exchange routing information between a user's on-premises network and their Virtual Private Clouds (VPCs) in AWS. BGP is a protocol used to exchange routing information between different networks. BGP allows networks to dynamically learn about and advertise routes to other networks.
  2. Dynamic Routing: BGP is a dynamic routing protocol that allows networks to learn about and advertise routes to other networks in real-time. This means that network changes, such as new VPCs being added or removed, are automatically detected and routes are updated accordingly.
  3. Efficient Traffic Routing: BGP allows for efficient traffic routing, ensuring that traffic takes the shortest and fastest path between the on-premises data center and AWS. This results in faster data transfer speeds and reduced network latency.
  4. Redundancy: BGP allows for redundant paths to be created, which ensures that if a primary path fails, traffic can automatically be rerouted to a secondary path. This provides higher availability and reliability for mission-critical applications.
  5. Granular Control: BGP provides granular control over routing decisions, allowing users to customize routing policies to meet their specific needs. This includes controlling how traffic is routed based on factors such as latency, cost, and network path.
  6. Authentication:

Security: BGP can be used to establish secure connections between the on-premises data center and AWS by using BGP communities to identify and tag specific routes. This helps ensure that traffic is properly segregated and only flows through authorized paths.

BGP in TGW Connect:

Enter image description here

AWS Transit Gateway Connect enables native integration of Software-Defined Wide Area Network (SD-WAN) appliances into AWS. Customers can now seamlessly extend their SD-WAN edge into AWS using standard protocols such as Generic Routing Encapsulation (GRE) and Border Gateway Protocol (BGP). It provides customers with added benefits such as improved bandwidth and supports dynamic routing with increased route limits, thus removing the need to set up multiple IPsec VPNs between the SD-WAN appliances and Transit Gateway. TGW Connect provides several advantages, including:

  1. Simplified network architecture: TGW Connect enables customers to simplify their network architecture by providing a centralized hub for routing between on-premises networks and VPCs. This reduces the complexity and potential for errors in configuring multiple VPN connections or VPC peering connections.
  2. Scalability: TGW Connect can scale to support large numbers of VPCs and on-premises networks, making it suitable for large-scale deployments.
  3. Performance: TGW Connect provides high-speed, low-latency connectivity between VPCs and on-premises networks using AWS's highly available and resilient global network infrastructure.
  4. Security: TGW Connect supports advanced security features such as AWS Transit Gateway firewall integration and VPC isolation, enabling customers to enforce network segmentation and protect their workloads from external threats.
  5. Cost-effectiveness: TGW Connect can help reduce networking costs by consolidating multiple connections into a single transit gateway, reducing the need for expensive hardware and reducing data transfer costs.

Troubleshooting BGP in AWS

For troubleshooting a BGP session that can't establish a connection or is in an idle state over a VPN tunnel please go through following troubleshooting steps:

  1. Check the underlying VPN connection because, BGP session can be established only if the VPN tunnel is up. If the VPN tunnel is down or flapping, you'll experience issues with establishing the BGP session. Make sure that the VPN is up and stable.
  2. Check the BGP configuration on your customer gateway device and make sure the IP addresses and Autonomous System Numbers (ASN) of the local and remote BGP peers must be configured with the downloaded VPN configuration file.
  3. If the configuration settings are correct, verify the connectivity between BGP peers by pinging the remote BGP peer IP from your local BGP peer IP. If ping is not working please check the followings:
  4. If BGP session is flapping between active and connect states, verify that TCP port 179 and other relevant ephemeral ports are not blocked.

 Understanding BGP Protocol Negotiation and troubleshoot steps:

Enter image description here

Idle: This is the first state where BGP waits for a “start event”. The start event occurs when someone configures a new BGP neighbor or when we reset an established BGP peering. After the start event, BGP will initialize some resources, resets a ConnectRetry timer and initiates a TCP connection to the remote BGP neighbor.

Connect: BGP is waiting for the TCP three-way handshake to complete. When it is successful, it will continue to the OpenSent state. In case it fails, we continue to the Active state. If the ConnectRetry timer expires then we will remain in this state.

TCP Handshake Between BGP peers (CGW side logs):

^58	2021-07-04 22:50:20.699007	169.254.60.146	169.254.60.145	TCP	74	34516 → 179 [SYN] Seq=0 Win=2920 Len=0 MSS=1460 SACK_PERM TSval=3030456 TSecr=0 WS=1
 59	2021-07-04 22:50:20.719228	169.254.60.145	169.254.60.146	TCP	74	179 → 34516 [SYN, ACK] Seq=0 Ack=1 Win=26844 Len=0 MSS=1375 TSval=64921081 TSecr=3030456 WS=128
 60	2021-07-04 22:50:20.719453	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=1 Ack=1 Win=2920 Len=0 TSval=3030476 TSecr=64921081
 61	2021-07-04 22:50:20.719490	169.254.60.146	169.254.60.145	BGP	115	OPEN Message
 62	2021-07-04 22:50:20.740519	169.254.60.145	169.254.60.146	TCP	66	179 → 34516 [ACK] Seq=1 Ack=50 Win=26880 Len=0 TSval=64921086 TSecr=3030476
 63	2021-07-04 22:50:20.743818	169.254.60.145	169.254.60.146	BGP	138	OPEN Message, KEEPALIVE Message
 64	2021-07-04 22:50:20.743918	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=50 Ack=73 Win=2848 Len=0 TSval=3030501 TSecr=64921087
 65	2021-07-04 22:50:20.744297	169.254.60.146	169.254.60.145	BGP	85	KEEPALIVE Message
 66	2021-07-04 22:50:20.765323	169.254.60.145	169.254.60.146	BGP	85	KEEPALIVE Message
 67	2021-07-04 22:50:20.765458	169.254.60.146	169.254.60.145	BGP	89	UPDATE Message
 68	2021-07-04 22:50:20.825693	169.254.60.145	169.254.60.146	TCP	66	179 → 34516 [ACK] Seq=92 Ack=92 Win=26880 Len=0 TSval=64921108 TSecr=3030522
 69	2021-07-04 22:50:21.765838	169.254.60.145	169.254.60.146	BGP	141	UPDATE Message, UPDATE Message
 70	2021-07-04 22:50:21.805586	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=92 Ack=167 Win=2848 Len=0 TSval=3031563 TSecr=64921343

>Frame 58: 74 bytes on wire (592 bits), 74 bytes captured (592 bits)
>Ethernet II, Src: ba:db:ee:fb:ad:04 (ba:db:ee:fb:ad:04), Dst: 00:70:76:69:66:00 (00:70:76:69:66:00)
>Internet Protocol Version 4, Src: 169.254.60.146, Dst: 169.254.60.145
^Transmission Control Protocol, Src Port: 34516, Dst Port: 179, Seq: 0, Len: 0
    Source Port: 34516
    Destination Port: 179
    [Stream index: 57]
    [Conversation completeness: Incomplete, DATA (15)]
    [TCP Segment Len: 0]
    Sequence Number: 0    (relative sequence number)
    Sequence Number (raw): 1887105697
    [Next Sequence Number: 1    (relative sequence number)]
    Acknowledgment Number: 0
    Acknowledgment number (raw): 0
    1010 .... = Header Length: 40 bytes (10)
    Flags: 0x002 (SYN)
    Window: 2920
    [Calculated window size: 2920]
    Checksum: 0x46f5 [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    Options: (20 bytes), Maximum segment size, SACK permitted, Timestamps, No- 
    Operation (NOP), Window scale
    [Timestamps]

Active: BGP will try another TCP three-way handshake to establish a connection with the remote BGP neighbor. If it is successful, it will move to the OpenSent state. If the ConnectRetry timer expires then we move back to the Connect state.

OpenSent: After sending an OPEN message to the peer, BGP waits in this state for the OPEN reply. If a successful reply comes in, the BGP state moves to OpenConfirm and a keepalive is sent to the peer. Failure can result in sending the BGP state back to Idle or Active.

CGW Side Peer Sending Open Message:

 58	2021-07-04 22:50:20.699007	169.254.60.146	169.254.60.145	TCP	74	34516 → 179 [SYN] Seq=0 Win=2920 Len=0 MSS=1460 SACK_PERM TSval=3030456 TSecr=0 WS=1
 59	2021-07-04 22:50:20.719228	169.254.60.145	169.254.60.146	TCP	74	179 → 34516 [SYN, ACK] Seq=0 Ack=1 Win=26844 Len=0 MSS=1375 TSval=64921081 TSecr=3030456 WS=128
 60	2021-07-04 22:50:20.719453	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=1 Ack=1 Win=2920 Len=0 TSval=3030476 TSecr=64921081
^61	2021-07-04 22:50:20.719490	169.254.60.146	169.254.60.145	BGP	115	OPEN Message
 62	2021-07-04 22:50:20.740519	169.254.60.145	169.254.60.146	TCP	66	179 → 34516 [ACK] Seq=1 Ack=50 Win=26880 Len=0 TSval=64921086 TSecr=3030476
 63	2021-07-04 22:50:20.743818	169.254.60.145	169.254.60.146	BGP	138	OPEN Message, KEEPALIVE Message
 64	2021-07-04 22:50:20.743918	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=50 Ack=73 Win=2848 Len=0 TSval=3030501 TSecr=64921087
 65	2021-07-04 22:50:20.744297	169.254.60.146	169.254.60.145	BGP	85	KEEPALIVE Message
 66	2021-07-04 22:50:20.765323	169.254.60.145	169.254.60.146	BGP	85	KEEPALIVE Message
 67	2021-07-04 22:50:20.765458	169.254.60.146	169.254.60.145	BGP	89	UPDATE Message
 68	2021-07-04 22:50:20.825693	169.254.60.145	169.254.60.146	TCP	66	179 → 34516 [ACK] Seq=92 Ack=92 Win=26880 Len=0 TSval=64921108 TSecr=3030522
 69	2021-07-04 22:50:21.765838	169.254.60.145	169.254.60.146	BGP	141	UPDATE Message, UPDATE Message
 70	2021-07-04 22:50:21.805586	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=92 Ack=167 Win=2848 Len=0 TSval=3031563 TSecr=64921343

 Frame 61: 115 bytes on wire (920 bits), 115 bytes captured (920 bits)
 Ethernet II, Src: ba:db:ee:fb:ad:04 (ba:db:ee:fb:ad:04), Dst: 00:70:76:69:66:00 (00:70:76:69:66:00)
 Internet Protocol Version 4, Src: 169.254.60.146, Dst: 169.254.60.145
^Transmission Control Protocol, Src Port: 34516, Dst Port: 179, Seq: 1, Ack: 1, Len: 49
    Source Port: 34516
    Destination Port: 179
    [Stream index: 57]
    [Conversation completeness: Incomplete, DATA (15)]
    [TCP Segment Len: 49]
    Sequence Number: 1    (relative sequence number)
    Sequence Number (raw): 1887105698
    [Next Sequence Number: 50    (relative sequence number)]
    Acknowledgment Number: 1    (relative ack number)
    Acknowledgment number (raw): 4246846231
    1000 .... = Header Length: 32 bytes (8)
    Flags: 0x018 (PSH, ACK)
    Window: 2920
    [Calculated window size: 2920]
    [Window size scaling factor: 1]
    Checksum: 0x46cb [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
    [Timestamps]
    [SEQ/ACK analysis]
    TCP payload (49 bytes)
^Border Gateway Protocol - OPEN Message
    Marker: ffffffffffffffffffffffffffffffff
    Length: 49
    Type: OPEN Message (1)
    Version: 4
    My AS: 65000
    Hold Time: 90
    BGP Identifier: 54.241.242.80
    Optional Parameters Length: 20
    ^Optional Parameters
        ^Optional Parameter: Capability
            Parameter Type: Capability (2)
            Parameter Length: 18
            Capability: Multiprotocol extensions capability
            Capability: Route refresh capability
            Capability: Route refresh capability (Cisco)
            Capability: Graceful Restart capability

AWS Side Peer Acknowledging Open Message:

 58	2021-07-04 22:50:20.699007	169.254.60.146	169.254.60.145	TCP	74	34516 → 179 [SYN] Seq=0 Win=2920 Len=0 MSS=1460 SACK_PERM TSval=3030456 TSecr=0 WS=1
 59	2021-07-04 22:50:20.719228	169.254.60.145	169.254.60.146	TCP	74	179 → 34516 [SYN, ACK] Seq=0 Ack=1 Win=26844 Len=0 MSS=1375 TSval=64921081 TSecr=3030456 WS=128
 60	2021-07-04 22:50:20.719453	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=1 Ack=1 Win=2920 Len=0 TSval=3030476 TSecr=64921081
 61	2021-07-04 22:50:20.719490	169.254.60.146	169.254.60.145	BGP	115	OPEN Message
^62	2021-07-04 22:50:20.740519	169.254.60.145	169.254.60.146	TCP	66	179 → 34516 [ACK] Seq=1 Ack=50 Win=26880 Len=0 TSval=64921086 TSecr=3030476
 63	2021-07-04 22:50:20.743818	169.254.60.145	169.254.60.146	BGP	138	OPEN Message, KEEPALIVE Message
 64	2021-07-04 22:50:20.743918	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=50 Ack=73 Win=2848 Len=0 TSval=3030501 TSecr=64921087
 65	2021-07-04 22:50:20.744297	169.254.60.146	169.254.60.145	BGP	85	KEEPALIVE Message
 66	2021-07-04 22:50:20.765323	169.254.60.145	169.254.60.146	BGP	85	KEEPALIVE Message
 67	2021-07-04 22:50:20.765458	169.254.60.146	169.254.60.145	BGP	89	UPDATE Message
 68	2021-07-04 22:50:20.825693	169.254.60.145	169.254.60.146	TCP	66	179 → 34516 [ACK] Seq=92 Ack=92 Win=26880 Len=0 TSval=64921108 TSecr=3030522
 69	2021-07-04 22:50:21.765838	169.254.60.145	169.254.60.146	BGP	141	UPDATE Message, UPDATE Message
 70	2021-07-04 22:50:21.805586	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=92 Ack=167 Win=2848 Len=0 TSval=3031563 TSecr=64921343

 Frame 62: 66 bytes on wire (528 bits), 66 bytes captured (528 bits)
 Ethernet II, Src: 06:2e:f7:16:d1:fd (06:2e:f7:16:d1:fd), Dst: 06:6d:ff:ac:b8:63 (06:6d:ff:ac:b8:63)
 Internet Protocol Version 4, Src: 169.254.60.145, Dst: 169.254.60.146
^Transmission Control Protocol, Src Port: 179, Dst Port: 34516, Seq: 1, Ack: 50, Len: 0
    Source Port: 179
    Destination Port: 34516
    [Stream index: 57]
    [Conversation completeness: Incomplete, DATA (15)]
    [TCP Segment Len: 0]
    Sequence Number: 1    (relative sequence number)
    Sequence Number (raw): 4246846231
    [Next Sequence Number: 1    (relative sequence number)]
    Acknowledgment Number: 50    (relative ack number)
    Acknowledgment number (raw): 1887105747
    1000 .... = Header Length: 32 bytes (8)
    Flags: 0x010 (ACK)
    Window: 210
    [Calculated window size: 26880]
    [Window size scaling factor: 128]
    Checksum: 0x25e5 [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
    [Timestamps]
    [SEQ/ACK analysis]

AWS Side Peer Sending Open Message:

 58	2021-07-04 22:50:20.699007	169.254.60.146	169.254.60.145	TCP	74	34516 → 179 [SYN] Seq=0 Win=2920 Len=0 MSS=1460 SACK_PERM TSval=3030456 TSecr=0 WS=1
 59	2021-07-04 22:50:20.719228	169.254.60.145	169.254.60.146	TCP	74	179 → 34516 [SYN, ACK] Seq=0 Ack=1 Win=26844 Len=0 MSS=1375 TSval=64921081 TSecr=3030456 WS=128
 60	2021-07-04 22:50:20.719453	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=1 Ack=1 Win=2920 Len=0 TSval=3030476 TSecr=64921081
 61	2021-07-04 22:50:20.719490	169.254.60.146	169.254.60.145	BGP	115	OPEN Message
 62	2021-07-04 22:50:20.740519	169.254.60.145	169.254.60.146	TCP	66	179 → 34516 [ACK] Seq=1 Ack=50 Win=26880 Len=0 TSval=64921086 TSecr=3030476
^63	2021-07-04 22:50:20.743818	169.254.60.145	169.254.60.146	BGP	138	OPEN Message, KEEPALIVE Message
 64	2021-07-04 22:50:20.743918	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=50 Ack=73 Win=2848 Len=0 TSval=3030501 TSecr=64921087
 65	2021-07-04 22:50:20.744297	169.254.60.146	169.254.60.145	BGP	85	KEEPALIVE Message
 66	2021-07-04 22:50:20.765323	169.254.60.145	169.254.60.146	BGP	85	KEEPALIVE Message
 67	2021-07-04 22:50:20.765458	169.254.60.146	169.254.60.145	BGP	89	UPDATE Message
 68	2021-07-04 22:50:20.825693	169.254.60.145	169.254.60.146	TCP	66	179 → 34516 [ACK] Seq=92 Ack=92 Win=26880 Len=0 TSval=64921108 TSecr=3030522
 69	2021-07-04 22:50:21.765838	169.254.60.145	169.254.60.146	BGP	141	UPDATE Message, UPDATE Message
 70	2021-07-04 22:50:21.805586	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=92 Ack=167 Win=2848 Len=0 TSval=3031563 TSecr=64921343

 Frame 63: 138 bytes on wire (1104 bits), 138 bytes captured (1104 bits)
 Ethernet II, Src: 06:2e:f7:16:d1:fd (06:2e:f7:16:d1:fd), Dst: 06:6d:ff:ac:b8:63 (06:6d:ff:ac:b8:63)
 Internet Protocol Version 4, Src: 169.254.60.145, Dst: 169.254.60.146
^Transmission Control Protocol, Src Port: 179, Dst Port: 34516, Seq: 1, Ack: 50, Len: 72
    Source Port: 179
    Destination Port: 34516
    [Stream index: 57]
    [Conversation completeness: Incomplete, DATA (15)]
    [TCP Segment Len: 72]
    Sequence Number: 1    (relative sequence number)
    Sequence Number (raw): 4246846231
    [Next Sequence Number: 73    (relative sequence number)]
    Acknowledgment Number: 50    (relative ack number)
    Acknowledgment number (raw): 1887105747
    1000 .... = Header Length: 32 bytes (8)
    Flags: 0x018 (PSH, ACK)
    Window: 210
    [Calculated window size: 26880]
    [Window size scaling factor: 128]
    Checksum: 0xfadf [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
    [Timestamps]
    [SEQ/ACK analysis]
    TCP payload (72 bytes)
^Border Gateway Protocol - OPEN Message
    Marker: ffffffffffffffffffffffffffffffff
    Length: 53
    Type: OPEN Message (1)
    Version: 4
    My AS: 64512
    Hold Time: 30
    BGP Identifier: 169.254.60.145
    Optional Parameters Length: 24
    ^Optional Parameters
        Optional Parameter: Capability
            Parameter Type: Capability (2)
            Parameter Length: 6
            Capability: Multiprotocol extensions capability
        Optional Parameter: Capability
            Parameter Type: Capability (2)
            Parameter Length: 2
            Capability: Route refresh capability (Cisco)
        Optional Parameter: Capability
            Parameter Type: Capability (2)
            Parameter Length: 2
            Capability: Route refresh capability
        Optional Parameter: Capability
            Parameter Type: Capability (2)
            Parameter Length: 6
            Capability: Support for 4-octet AS number capability
Border Gateway Protocol - KEEPALIVE Message
    Marker: ffffffffffffffffffffffffffffffff
    Length: 19
    Type: KEEPALIVE Message (4)

CGW Side Peer Acknowledging Open Message:

 58	2021-07-04 22:50:20.699007	169.254.60.146	169.254.60.145	TCP	74	34516 → 179 [SYN] Seq=0 Win=2920 Len=0 MSS=1460 SACK_PERM TSval=3030456 TSecr=0 WS=1
 59	2021-07-04 22:50:20.719228	169.254.60.145	169.254.60.146	TCP	74	179 → 34516 [SYN, ACK] Seq=0 Ack=1 Win=26844 Len=0 MSS=1375 TSval=64921081 TSecr=3030456 WS=128
 60	2021-07-04 22:50:20.719453	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=1 Ack=1 Win=2920 Len=0 TSval=3030476 TSecr=64921081
 61	2021-07-04 22:50:20.719490	169.254.60.146	169.254.60.145	BGP	115	OPEN Message
 62	2021-07-04 22:50:20.740519	169.254.60.145	169.254.60.146	TCP	66	179 → 34516 [ACK] Seq=1 Ack=50 Win=26880 Len=0 TSval=64921086 TSecr=3030476
 63	2021-07-04 22:50:20.743818	169.254.60.145	169.254.60.146	BGP	138	OPEN Message, KEEPALIVE Message
^64	2021-07-04 22:50:20.743918	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=50 Ack=73 Win=2848 Len=0 TSval=3030501 TSecr=64921087
 65	2021-07-04 22:50:20.744297	169.254.60.146	169.254.60.145	BGP	85	KEEPALIVE Message
 66	2021-07-04 22:50:20.765323	169.254.60.145	169.254.60.146	BGP	85	KEEPALIVE Message
 67	2021-07-04 22:50:20.765458	169.254.60.146	169.254.60.145	BGP	89	UPDATE Message
 68	2021-07-04 22:50:20.825693	169.254.60.145	169.254.60.146	TCP	66	179 → 34516 [ACK] Seq=92 Ack=92 Win=26880 Len=0 TSval=64921108 TSecr=3030522
 69	2021-07-04 22:50:21.765838	169.254.60.145	169.254.60.146	BGP	141	UPDATE Message, UPDATE Message
 70	2021-07-04 22:50:21.805586	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=92 Ack=167 Win=2848 Len=0 TSval=3031563 TSecr=64921343

 Frame 64: 66 bytes on wire (528 bits), 66 bytes captured (528 bits)
 Ethernet II, Src: ba:db:ee:fb:ad:04 (ba:db:ee:fb:ad:04), Dst: 00:70:76:69:66:00 (00:70:76:69:66:00)
 Internet Protocol Version 4, Src: 169.254.60.146, Dst: 169.254.60.145
^Transmission Control Protocol, Src Port: 34516, Dst Port: 179, Seq: 50, Ack: 73, Len: 0
    Source Port: 34516
    Destination Port: 179
    [Stream index: 57]
    [Conversation completeness: Incomplete, DATA (15)]
    [TCP Segment Len: 0]
    Sequence Number: 50    (relative sequence number)
    Sequence Number (raw): 1887105747
    [Next Sequence Number: 50    (relative sequence number)]
    Acknowledgment Number: 73    (relative ack number)
    Acknowledgment number (raw): 4246846303
    1000 .... = Header Length: 32 bytes (8)
    Flags: 0x010 (ACK)
    Window: 2848
    [Calculated window size: 2848]
    [Window size scaling factor: 1]
    Checksum: 0x1b35 [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
    [Timestamps]
    [SEQ/ACK analysis]

OpenConfirm: The BGP state machine is one step away from reaching its final state (Established). BGP waits in this state for keepalives from the peer. If successful, the state moves to Established; otherwise, the state moves back to Idle based on the errors.

CGW Side Peer Sending Keep-Alive:

 58	2021-07-04 22:50:20.699007	169.254.60.146	169.254.60.145	TCP	74	34516 → 179 [SYN] Seq=0 Win=2920 Len=0 MSS=1460 SACK_PERM TSval=3030456 TSecr=0 WS=1
 59	2021-07-04 22:50:20.719228	169.254.60.145	169.254.60.146	TCP	74	179 → 34516 [SYN, ACK] Seq=0 Ack=1 Win=26844 Len=0 MSS=1375 TSval=64921081 TSecr=3030456 WS=128
 60	2021-07-04 22:50:20.719453	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=1 Ack=1 Win=2920 Len=0 TSval=3030476 TSecr=64921081
 61	2021-07-04 22:50:20.719490	169.254.60.146	169.254.60.145	BGP	115	OPEN Message
 62	2021-07-04 22:50:20.740519	169.254.60.145	169.254.60.146	TCP	66	179 → 34516 [ACK] Seq=1 Ack=50 Win=26880 Len=0 TSval=64921086 TSecr=3030476
 63	2021-07-04 22:50:20.743818	169.254.60.145	169.254.60.146	BGP	138	OPEN Message, KEEPALIVE Message
 64	2021-07-04 22:50:20.743918	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=50 Ack=73 Win=2848 Len=0 TSval=3030501 TSecr=64921087
^65	2021-07-04 22:50:20.744297	169.254.60.146	169.254.60.145	BGP	85	KEEPALIVE Message
 66	2021-07-04 22:50:20.765323	169.254.60.145	169.254.60.146	BGP	85	KEEPALIVE Message
 67	2021-07-04 22:50:20.765458	169.254.60.146	169.254.60.145	BGP	89	UPDATE Message
 68	2021-07-04 22:50:20.825693	169.254.60.145	169.254.60.146	TCP	66	179 → 34516 [ACK] Seq=92 Ack=92 Win=26880 Len=0 TSval=64921108 TSecr=3030522
 69	2021-07-04 22:50:21.765838	169.254.60.145	169.254.60.146	BGP	141	UPDATE Message, UPDATE Message
 70	2021-07-04 22:50:21.805586	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=92 Ack=167 Win=2848 Len=0 TSval=3031563 TSecr=64921343

 Frame 65: 85 bytes on wire (680 bits), 85 bytes captured (680 bits)
 Ethernet II, Src: ba:db:ee:fb:ad:04 (ba:db:ee:fb:ad:04), Dst: 00:70:76:69:66:00 (00:70:76:69:66:00)
 Internet Protocol Version 4, Src: 169.254.60.146, Dst: 169.254.60.145
^Transmission Control Protocol, Src Port: 34516, Dst Port: 179, Seq: 50, Ack: 73, Len: 19
    Source Port: 34516
    Destination Port: 179
    [Stream index: 57]
    [Conversation completeness: Incomplete, DATA (15)]
    [TCP Segment Len: 19]
    Sequence Number: 50    (relative sequence number)
    Sequence Number (raw): 1887105747
    [Next Sequence Number: 69    (relative sequence number)]
    Acknowledgment Number: 73    (relative ack number)
    Acknowledgment number (raw): 4246846303
    1000 .... = Header Length: 32 bytes (8)
    Flags: 0x018 (PSH, ACK)
    Window: 2848
    [Calculated window size: 2848]
    [Window size scaling factor: 1]
    Checksum: 0x1707 [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
    [Timestamps]
    [SEQ/ACK analysis]
    TCP payload (19 bytes)
Border Gateway Protocol - KEEPALIVE Message
    Marker: ffffffffffffffffffffffffffffffff
    Length: 19
    Type: KEEPALIVE Message (4)

AWS Side Peer Sending Keep-Alive:

 58	2021-07-04 22:50:20.699007	169.254.60.146	169.254.60.145	TCP	74	34516 → 179 [SYN] Seq=0 Win=2920 Len=0 MSS=1460 SACK_PERM TSval=3030456 TSecr=0 WS=1
 59	2021-07-04 22:50:20.719228	169.254.60.145	169.254.60.146	TCP	74	179 → 34516 [SYN, ACK] Seq=0 Ack=1 Win=26844 Len=0 MSS=1375 TSval=64921081 TSecr=3030456 WS=128
 60	2021-07-04 22:50:20.719453	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=1 Ack=1 Win=2920 Len=0 TSval=3030476 TSecr=64921081
 61	2021-07-04 22:50:20.719490	169.254.60.146	169.254.60.145	BGP	115	OPEN Message
 62	2021-07-04 22:50:20.740519	169.254.60.145	169.254.60.146	TCP	66	179 → 34516 [ACK] Seq=1 Ack=50 Win=26880 Len=0 TSval=64921086 TSecr=3030476
 63	2021-07-04 22:50:20.743818	169.254.60.145	169.254.60.146	BGP	138	OPEN Message, KEEPALIVE Message
 64	2021-07-04 22:50:20.743918	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=50 Ack=73 Win=2848 Len=0 TSval=3030501 TSecr=64921087
 65	2021-07-04 22:50:20.744297	169.254.60.146	169.254.60.145	BGP	85	KEEPALIVE Message
^66	2021-07-04 22:50:20.765323	169.254.60.145	169.254.60.146	BGP	85	KEEPALIVE Message
 67	2021-07-04 22:50:20.765458	169.254.60.146	169.254.60.145	BGP	89	UPDATE Message
 68	2021-07-04 22:50:20.825693	169.254.60.145	169.254.60.146	TCP	66	179 → 34516 [ACK] Seq=92 Ack=92 Win=26880 Len=0 TSval=64921108 TSecr=3030522
 69	2021-07-04 22:50:21.765838	169.254.60.145	169.254.60.146	BGP	141	UPDATE Message, UPDATE Message
 70	2021-07-04 22:50:21.805586	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=92 Ack=167 Win=2848 Len=0 TSval=3031563 TSecr=64921343

 Frame 66: 85 bytes on wire (680 bits), 85 bytes captured (680 bits)
 Ethernet II, Src: 06:2e:f7:16:d1:fd (06:2e:f7:16:d1:fd), Dst: 06:6d:ff:ac:b8:63 (06:6d:ff:ac:b8:63)
 Internet Protocol Version 4, Src: 169.254.60.145, Dst: 169.254.60.146
^Transmission Control Protocol, Src Port: 179, Dst Port: 34516, Seq: 73, Ack: 69, Len: 19
    Source Port: 179
    Destination Port: 34516
    [Stream index: 57]
    [Conversation completeness: Incomplete, DATA (15)]
    [TCP Segment Len: 19]
    Sequence Number: 73    (relative sequence number)
    Sequence Number (raw): 4246846303
    [Next Sequence Number: 92    (relative sequence number)]
    Acknowledgment Number: 69    (relative ack number)
    Acknowledgment number (raw): 1887105766
    1000 .... = Header Length: 32 bytes (8)
    Flags: 0x018 (PSH, ACK)
    Window: 210
    [Calculated window size: 26880]
    [Window size scaling factor: 128]
    Checksum: 0x213d [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
    [Timestamps]
    [SEQ/ACK analysis]
    TCP payload (19 bytes)
^Border Gateway Protocol - KEEPALIVE Message
    Marker: ffffffffffffffffffffffffffffffff
    Length: 19
    Type: KEEPALIVE Message (4)

Established: This is the state in which BGP can exchange information between the peers. The information can be updates, keepalives, or notification.

CGW Side Peer Advertising its routes:

 58	2021-07-04 22:50:20.699007	169.254.60.146	169.254.60.145	TCP	74	34516 → 179 [SYN] Seq=0 Win=2920 Len=0 MSS=1460 SACK_PERM TSval=3030456 TSecr=0 WS=1
 59	2021-07-04 22:50:20.719228	169.254.60.145	169.254.60.146	TCP	74	179 → 34516 [SYN, ACK] Seq=0 Ack=1 Win=26844 Len=0 MSS=1375 TSval=64921081 TSecr=3030456 WS=128
 60	2021-07-04 22:50:20.719453	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=1 Ack=1 Win=2920 Len=0 TSval=3030476 TSecr=64921081
 61	2021-07-04 22:50:20.719490	169.254.60.146	169.254.60.145	BGP	115	OPEN Message
 62	2021-07-04 22:50:20.740519	169.254.60.145	169.254.60.146	TCP	66	179 → 34516 [ACK] Seq=1 Ack=50 Win=26880 Len=0 TSval=64921086 TSecr=3030476
 63	2021-07-04 22:50:20.743818	169.254.60.145	169.254.60.146	BGP	138	OPEN Message, KEEPALIVE Message
 64	2021-07-04 22:50:20.743918	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=50 Ack=73 Win=2848 Len=0 TSval=3030501 TSecr=64921087
 65	2021-07-04 22:50:20.744297	169.254.60.146	169.254.60.145	BGP	85	KEEPALIVE Message
 66	2021-07-04 22:50:20.765323	169.254.60.145	169.254.60.146	BGP	85	KEEPALIVE Message
^67	2021-07-04 22:50:20.765458	169.254.60.146	169.254.60.145	BGP	89	UPDATE Message
 68	2021-07-04 22:50:20.825693	169.254.60.145	169.254.60.146	TCP	66	179 → 34516 [ACK] Seq=92 Ack=92 Win=26880 Len=0 TSval=64921108 TSecr=3030522
 69	2021-07-04 22:50:21.765838	169.254.60.145	169.254.60.146	BGP	141	UPDATE Message, UPDATE Message
 70	2021-07-04 22:50:21.805586	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=92 Ack=167 Win=2848 Len=0 TSval=3031563 TSecr=64921343

 Frame 67: 89 bytes on wire (712 bits), 89 bytes captured (712 bits)
 Ethernet II, Src: ba:db:ee:fb:ad:04 (ba:db:ee:fb:ad:04), Dst: 00:70:76:69:66:00 (00:70:76:69:66:00)
 Internet Protocol Version 4, Src: 169.254.60.146, Dst: 169.254.60.145
^Transmission Control Protocol, Src Port: 34516, Dst Port: 179, Seq: 69, Ack: 92, Len: 23
    Source Port: 34516
    Destination Port: 179
    [Stream index: 57]
    [Conversation completeness: Incomplete, DATA (15)]
    [TCP Segment Len: 23]
    Sequence Number: 69    (relative sequence number)
    Sequence Number (raw): 1887105766
    [Next Sequence Number: 92    (relative sequence number)]
    Acknowledgment Number: 92    (relative ack number)
    Acknowledgment number (raw): 4246846322
    1000 .... = Header Length: 32 bytes (8)
    Flags: 0x018 (PSH, ACK)
    Window: 2848
    [Calculated window size: 2848]
    [Window size scaling factor: 1]
    Checksum: 0x18bf [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
    [Timestamps]
    [SEQ/ACK analysis]
    TCP payload (23 bytes)
^Border Gateway Protocol - UPDATE Message
    Marker: ffffffffffffffffffffffffffffffff
    Length: 52
    Type: UPDATE Message (2)
    Withdrawn Routes Length: 0
    Total Path Attribute Length: 26
    ^Path attributes
        Path Attribute - ORIGIN: INCOMPLETE
        Path Attribute - AS_PATH: 65000 
        Path Attribute - NEXT_HOP: 169.254.60.146 
    ^Network Layer Reachability Information (NLRI)
        192.168.0.0/16

AWS Side Peer Acknowledging received routes:

 58	2021-07-04 22:50:20.699007	169.254.60.146	169.254.60.145	TCP	74	34516 → 179 [SYN] Seq=0 Win=2920 Len=0 MSS=1460 SACK_PERM TSval=3030456 TSecr=0 WS=1
 59	2021-07-04 22:50:20.719228	169.254.60.145	169.254.60.146	TCP	74	179 → 34516 [SYN, ACK] Seq=0 Ack=1 Win=26844 Len=0 MSS=1375 TSval=64921081 TSecr=3030456 WS=128
 60	2021-07-04 22:50:20.719453	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=1 Ack=1 Win=2920 Len=0 TSval=3030476 TSecr=64921081
 61	2021-07-04 22:50:20.719490	169.254.60.146	169.254.60.145	BGP	115	OPEN Message
 62	2021-07-04 22:50:20.740519	169.254.60.145	169.254.60.146	TCP	66	179 → 34516 [ACK] Seq=1 Ack=50 Win=26880 Len=0 TSval=64921086 TSecr=3030476
 63	2021-07-04 22:50:20.743818	169.254.60.145	169.254.60.146	BGP	138	OPEN Message, KEEPALIVE Message
 64	2021-07-04 22:50:20.743918	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=50 Ack=73 Win=2848 Len=0 TSval=3030501 TSecr=64921087
 65	2021-07-04 22:50:20.744297	169.254.60.146	169.254.60.145	BGP	85	KEEPALIVE Message
 66	2021-07-04 22:50:20.765323	169.254.60.145	169.254.60.146	BGP	85	KEEPALIVE Message
 67	2021-07-04 22:50:20.765458	169.254.60.146	169.254.60.145	BGP	89	UPDATE Message
^68	2021-07-04 22:50:20.825693	169.254.60.145	169.254.60.146	TCP	66	179 → 34516 [ACK] Seq=92 Ack=92 Win=26880 Len=0 TSval=64921108 TSecr=3030522
 69	2021-07-04 22:50:21.765838	169.254.60.145	169.254.60.146	BGP	141	UPDATE Message, UPDATE Message
 70	2021-07-04 22:50:21.805586	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=92 Ack=167 Win=2848 Len=0 TSval=3031563 TSecr=64921343

 Frame 68: 66 bytes on wire (528 bits), 66 bytes captured (528 bits)
 Ethernet II, Src: 06:2e:f7:16:d1:fd (06:2e:f7:16:d1:fd), Dst: 06:6d:ff:ac:b8:63 (06:6d:ff:ac:b8:63)
 Internet Protocol Version 4, Src: 169.254.60.145, Dst: 169.254.60.146
^Transmission Control Protocol, Src Port: 179, Dst Port: 34516, Seq: 92, Ack: 92, Len: 0
    Source Port: 179
    Destination Port: 34516
    [Stream index: 57]
    [Conversation completeness: Incomplete, DATA (15)]
    [TCP Segment Len: 0]
    Sequence Number: 92    (relative sequence number)
    Sequence Number (raw): 4246846322
    [Next Sequence Number: 92    (relative sequence number)]
    Acknowledgment Number: 92    (relative ack number)
    Acknowledgment number (raw): 1887105789
    1000 .... = Header Length: 32 bytes (8)
    Flags: 0x010 (ACK)
    Window: 210
    [Calculated window size: 26880]
    [Window size scaling factor: 128]
    Checksum: 0x251c [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
    [Timestamps]
    [SEQ/ACK analysis]

AWS Side Peer Advertising its routes:

 58	2021-07-04 22:50:20.699007	169.254.60.146	169.254.60.145	TCP	74	34516 → 179 [SYN] Seq=0 Win=2920 Len=0 MSS=1460 SACK_PERM TSval=3030456 TSecr=0 WS=1
 59	2021-07-04 22:50:20.719228	169.254.60.145	169.254.60.146	TCP	74	179 → 34516 [SYN, ACK] Seq=0 Ack=1 Win=26844 Len=0 MSS=1375 TSval=64921081 TSecr=3030456 WS=128
 60	2021-07-04 22:50:20.719453	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=1 Ack=1 Win=2920 Len=0 TSval=3030476 TSecr=64921081
 61	2021-07-04 22:50:20.719490	169.254.60.146	169.254.60.145	BGP	115	OPEN Message
 62	2021-07-04 22:50:20.740519	169.254.60.145	169.254.60.146	TCP	66	179 → 34516 [ACK] Seq=1 Ack=50 Win=26880 Len=0 TSval=64921086 TSecr=3030476
 63	2021-07-04 22:50:20.743818	169.254.60.145	169.254.60.146	BGP	138	OPEN Message, KEEPALIVE Message
 64	2021-07-04 22:50:20.743918	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=50 Ack=73 Win=2848 Len=0 TSval=3030501 TSecr=64921087
 65	2021-07-04 22:50:20.744297	169.254.60.146	169.254.60.145	BGP	85	KEEPALIVE Message
 66	2021-07-04 22:50:20.765323	169.254.60.145	169.254.60.146	BGP	85	KEEPALIVE Message
 67	2021-07-04 22:50:20.765458	169.254.60.146	169.254.60.145	BGP	89	UPDATE Message
 68	2021-07-04 22:50:20.825693	169.254.60.145	169.254.60.146	TCP	66	179 → 34516 [ACK] Seq=92 Ack=92 Win=26880 Len=0 TSval=64921108 TSecr=3030522
^69	2021-07-04 22:50:21.765838	169.254.60.145	169.254.60.146	BGP	141	UPDATE Message, UPDATE Message
 70	2021-07-04 22:50:21.805586	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=92 Ack=167 Win=2848 Len=0 TSval=3031563 TSecr=64921343

 Frame 69: 141 bytes on wire (1128 bits), 141 bytes captured (1128 bits)
 Ethernet II, Src: 06:2e:f7:16:d1:fd (06:2e:f7:16:d1:fd), Dst: 06:6d:ff:ac:b8:63 (06:6d:ff:ac:b8:63)
 Internet Protocol Version 4, Src: 169.254.60.145, Dst: 169.254.60.146
^Transmission Control Protocol, Src Port: 179, Dst Port: 34516, Seq: 92, Ack: 92, Len: 75
    Source Port: 179
    Destination Port: 34516
    [Stream index: 57]
    [Conversation completeness: Incomplete, DATA (15)]
    [TCP Segment Len: 75]
    Sequence Number: 92    (relative sequence number)
    Sequence Number (raw): 4246846322
    [Next Sequence Number: 167    (relative sequence number)]
    Acknowledgment Number: 92    (relative ack number)
    Acknowledgment number (raw): 1887105789
    1000 .... = Header Length: 32 bytes (8)
    Flags: 0x018 (PSH, ACK)
    Window: 210
    [Calculated window size: 26880]
    [Window size scaling factor: 128]
    Checksum: 0x1a09 [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
    [Timestamps]
    [SEQ/ACK analysis]
    TCP payload (75 bytes)
Border Gateway Protocol - UPDATE Message
    Marker: ffffffffffffffffffffffffffffffff
    Length: 52
    Type: UPDATE Message (2)
    Withdrawn Routes Length: 0
    Total Path Attribute Length: 26
    ^Path attributes
        Path Attribute - ORIGIN: IGP
        Path Attribute - AS_PATH: 64512 
        Path Attribute - NEXT_HOP: 169.254.60.145 
    ^Network Layer Reachability Information (NLRI)
        172.17.0.0/16

CGW Side Peer Acknowledging received routes:

 58	2021-07-04 22:50:20.699007	169.254.60.146	169.254.60.145	TCP	74	34516 → 179 [SYN] Seq=0 Win=2920 Len=0 MSS=1460 SACK_PERM TSval=3030456 TSecr=0 WS=1
 59	2021-07-04 22:50:20.719228	169.254.60.145	169.254.60.146	TCP	74	179 → 34516 [SYN, ACK] Seq=0 Ack=1 Win=26844 Len=0 MSS=1375 TSval=64921081 TSecr=3030456 WS=128
 60	2021-07-04 22:50:20.719453	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=1 Ack=1 Win=2920 Len=0 TSval=3030476 TSecr=64921081
 61	2021-07-04 22:50:20.719490	169.254.60.146	169.254.60.145	BGP	115	OPEN Message
 62	2021-07-04 22:50:20.740519	169.254.60.145	169.254.60.146	TCP	66	179 → 34516 [ACK] Seq=1 Ack=50 Win=26880 Len=0 TSval=64921086 TSecr=3030476
 63	2021-07-04 22:50:20.743818	169.254.60.145	169.254.60.146	BGP	138	OPEN Message, KEEPALIVE Message
 64	2021-07-04 22:50:20.743918	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=50 Ack=73 Win=2848 Len=0 TSval=3030501 TSecr=64921087
 65	2021-07-04 22:50:20.744297	169.254.60.146	169.254.60.145	BGP	85	KEEPALIVE Message
 66	2021-07-04 22:50:20.765323	169.254.60.145	169.254.60.146	BGP	85	KEEPALIVE Message
 67	2021-07-04 22:50:20.765458	169.254.60.146	169.254.60.145	BGP	89	UPDATE Message
 68	2021-07-04 22:50:20.825693	169.254.60.145	169.254.60.146	TCP	66	179 → 34516 [ACK] Seq=92 Ack=92 Win=26880 Len=0 TSval=64921108 TSecr=3030522
 69	2021-07-04 22:50:21.765838	169.254.60.145	169.254.60.146	BGP	141	UPDATE Message, UPDATE Message
^70	2021-07-04 22:50:21.805586	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=92 Ack=167 Win=2848 Len=0 TSval=3031563 TSecr=64921343

 Frame 70: 66 bytes on wire (528 bits), 66 bytes captured (528 bits)
 Ethernet II, Src: ba:db:ee:fb:ad:04 (ba:db:ee:fb:ad:04), Dst: 00:70:76:69:66:00 (00:70:76:69:66:00)
 Internet Protocol Version 4, Src: 169.254.60.146, Dst: 169.254.60.145
^Transmission Control Protocol, Src Port: 34516, Dst Port: 179, Seq: 92, Ack: 167, Len: 0
    Source Port: 34516
    Destination Port: 179
    [Stream index: 57]
    [Conversation completeness: Incomplete, DATA (15)]
    [TCP Segment Len: 0]
    Sequence Number: 92    (relative sequence number)
    Sequence Number (raw): 1887105789
    [Next Sequence Number: 92    (relative sequence number)]
    Acknowledgment Number: 167    (relative ack number)
    Acknowledgment number (raw): 4246846397
    1000 .... = Header Length: 32 bytes (8)
    Flags: 0x010 (ACK)
    Window: 2848
    [Calculated window size: 2848]
    [Window size scaling factor: 1]
    Checksum: 0x1587 [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
    [Timestamps]
    [SEQ/ACK analysis]

Both Peers exchanging and acknowledging Keepalives:

58	2021-07-04 22:50:20.699007	169.254.60.146	169.254.60.145	TCP	74	34516 → 179 [SYN] Seq=0 Win=2920 Len=0 MSS=1460 SACK_PERM TSval=3030456 TSecr=0 WS=1
59	2021-07-04 22:50:20.719228	169.254.60.145	169.254.60.146	TCP	74	179 → 34516 [SYN, ACK] Seq=0 Ack=1 Win=26844 Len=0 MSS=1375 TSval=64921081 TSecr=3030456 WS=128
60	2021-07-04 22:50:20.719453	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=1 Ack=1 Win=2920 Len=0 TSval=3030476 TSecr=64921081
61	2021-07-04 22:50:20.719490	169.254.60.146	169.254.60.145	BGP	115	OPEN Message
62	2021-07-04 22:50:20.740519	169.254.60.145	169.254.60.146	TCP	66	179 → 34516 [ACK] Seq=1 Ack=50 Win=26880 Len=0 TSval=64921086 TSecr=3030476
63	2021-07-04 22:50:20.743818	169.254.60.145	169.254.60.146	BGP	138	OPEN Message, KEEPALIVE Message
64	2021-07-04 22:50:20.743918	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=50 Ack=73 Win=2848 Len=0 TSval=3030501 TSecr=64921087
65	2021-07-04 22:50:20.744297	169.254.60.146	169.254.60.145	BGP	85	KEEPALIVE Message
66	2021-07-04 22:50:20.765323	169.254.60.145	169.254.60.146	BGP	85	KEEPALIVE Message
67	2021-07-04 22:50:20.765458	169.254.60.146	169.254.60.145	BGP	89	UPDATE Message
68	2021-07-04 22:50:20.825693	169.254.60.145	169.254.60.146	TCP	66	179 → 34516 [ACK] Seq=92 Ack=92 Win=26880 Len=0 TSval=64921108 TSecr=3030522
69	2021-07-04 22:50:21.765838	169.254.60.145	169.254.60.146	BGP	141	UPDATE Message, UPDATE Message
70	2021-07-04 22:50:21.805586	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=92 Ack=167 Win=2848 Len=0 TSval=3031563 TSecr=64921343
71	2021-07-04 22:50:29.871032	169.254.60.146	169.254.60.145	BGP	85	KEEPALIVE Message
72	2021-07-04 22:50:29.891713	169.254.60.145	169.254.60.146	TCP	66	179 → 34516 [ACK] Seq=167 Ack=111 Win=26880 Len=0 TSval=64923374 TSecr=3039628
73	2021-07-04 22:50:30.742335	169.254.60.145	169.254.60.146	BGP	85	KEEPALIVE Message
74	2021-07-04 22:50:30.742464	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=111 Ack=186 Win=2848 Len=0 TSval=3040499 TSecr=64923587
75	2021-07-04 22:50:37.437250	169.254.60.146	169.254.60.145	BGP	85	KEEPALIVE Message
76	2021-07-04 22:50:37.458148	169.254.60.145	169.254.60.146	TCP	66	179 → 34516 [ACK] Seq=186 Ack=130 Win=26880 Len=0 TSval=64925266 TSecr=3047194
77	2021-07-04 22:50:40.745810	169.254.60.145	169.254.60.146	BGP	85	KEEPALIVE Message
78	2021-07-04 22:50:40.745915	169.254.60.146	169.254.60.145	TCP	66	34516 → 179 [ACK] Seq=130 Ack=205 Win=2848 Len=0 TSval=3050503 TSecr=64926087

Advanced troubleshooting:

 1.     Ping between BGP peers works but session is not establishing over an AWS site-to-site VPN connection even though all BGP setting are configured correctly on both ends:

External BGP (EBGP) multi-hop is disabled on AWS so, if BGP peers are not directly connected to each other BGP session will not establish. To check whether peers are directly connected, run the ping test with TTL value 1 and if ping works then peers are directly connected and if not, multi-hops are present in between which is not supported on AWS side.

Normal ping test is working:

CSR#ping 169.254.60.145
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 169.254.60.145, timeout is 2 seconds:

! ! ! ! !
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 m

Ping with TTL value 1 is not working:

CSR#ping -1 1 169:254.60.145
Type escape sequence to abort
Sending 5, 100-byte ICMP Echos to 169.254.60.145, timeout is 2 seconds:
....
Success rate is 0 percent (0/5)

 2.     BGP Keep Flapping Due to "Hold Timer Expired:

Hold timer expiry is a common cause for flapping BGP peer. It means that the router didn’t receive or process a keepalive messages or any update message before the hold timer expired.  So, it sends a notification message (4/0) and closes the session. On IOS, the keepalive messages are sent by the BGP I/O process, and the BGP router process interprets the incoming keepalive messages. BGP flaps due to hold timer expiry can be caused by one of the following reasons:

A. WAN Interface issues: Various interface issues such as a physical layer concern or drops on the WAN interface can lead to a BGP session flapping because of hold time expiry. Performing MTR and traceroute over public path (internet) and private path (VPN) would help to identify the losses. 

B. Input hold queue: Check if packets are arriving to the CGW but dropped in the input hold queue of the incoming interface (WAN) and usually these packets are intended to be processed by router’s CPU utilization. The hold-queue size is a finite size and most of the CGW’s default input hold queue size value is 75 packets and can be configured to higher value to resolve the issue.

Check current interface Hold Queue and drops:

CSR#show interface GigabitEthernet1
GigabitEtherneti is up, line protocol is up
  Hardware is CSR vNIC, address is 02b7.159d.784a (bia 02b7.159d.784a)
  Internet address is 172.16.1.186/24
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec, 
           reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full Duplex, 1000Mbps, link type is auto, media type is Virtual 
  output flow-control is unsupported, input flow-control is unsupported
 ARP type: ARPA, ARP Timeout 04:00:00
 Last input 00:00:00, output 00:00:00, output hang never
 Last clearing of "show interface" counters never
 Input queue: 93/75/15/3 (size/max/drops/flushes) Total output drops: 153
 Queueing strategy: fifo
 Output queue: 0/40 (size/max)
 5 minute input rate 2000 bits/sec, 5 packets/sec
 5 minute output rate 3000 bits/sec, 3 packets/sec
     3502 packets input, 240805 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     9 watchdog, 0 multicast, 0 pause input
     2805 packets output, 289428 bytes, 0 underruns
     Output 0 broadcasts (0 IP multicasts)
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

Configure input Hold with desired value:

CSR#config terminal 
CSR(config)#interface GigabitEthernet1
CSR(config-if)#hold-queue 1500 in
CSR(config-if)#exit

C. TCP receive queue and BGP InQ: Check if BGP keepalive arrived at the TCP queue but are not being processed and moved to the BGP InQ due to huge TCP messages are already waiting in the queue. If so, BGP session would flap with an error “hold time expiry” since BGP I/O process do not get a chance to run, BGP I/O process is in charge of putting messages from TCP receiving queue into BGP InQ and these usually happens when BGP hold timer values are very low, there are many neighbors and CPU is running high.

Check receive queue and BGP InQ:

CSR#show bgp ipv4 unicast summary
BGP router identifier 169.254.60.144, local AS number 65000
BGP table version is 4, main routing table version 4 
3 network entries using 744 bytes of memory
5 path entries using 680 bytes of memory
2/2 BGP path/bestpath attribute entries using 576 bytes of memory
1 BGP AS-PATH entries using 24 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 2024 total bytes of memory
BGP activity 3/0 prefixes, 5/0 paths, scan interval 60 secs 
3 networks peaked at 00:26:52 Aug 11 2021 UTC (01:20:49.049 ago)

Neighbor           V       AS      MsgRcvd      MsgSent   TbIver   IngQ  OutQ   Up/Down  State/PfRed
169.254.60.145     4     64512     259453       157389     8701     207    0   01:20:49      2
  • The solution would be reduce the CPU utilization by disabling unnecessary running tasks and increase the BGP Hold timer to appropriate value. 
  • Also, some customer gateway devices are offering unique features, instead of queueing data once a second, BGP now queues data aggressively from the BGP OutQ to the TCP socket for each peer until the OutQs have drained completely. Since BGP now sends at a faster rate, BGP converges more quickly.

D. Maximum Transmission Unit (MTU) mismatch: MTU mismatch issue cause BGP session to flap. BGP sends updates based on the maximum segment size (MSS) value calculated by TCP. If Path-MTU-Discovery (PMTUD) is not enabled and the destination is remote, the BGP MSS value defaults to 536 bytes as defined in RFC 879. So, if there are a huge number of updates getting exchanged between the two routers at the MSS value of 536 bytes, convergence issues are detected, which cause inefficient use of the network.

Check Current MTU configuration:

CSR#show ip bp neighbors | include max data 
Datagrams (max data segment is 536 bytes):
Datagrams (max data segment is 536 bytes):

The solution is to enable the Path MTU (PMTU) feature and use it to dynamically determine how large the MSS value can be without creating packets that need to be fragmented. PMTU allows TCP to determine the smallest MTU size among all links in a TCP session.

Enable Path MTU:

CS#configure terminal
CSR(config)#interface GigabitEthernet1
CSR(config-if)#ip top path-mtu-discovery
CSR(config)#exit.

CSR#show ip bgp neighbors | include max data
Datagrams (max data segment is 1436 bytes):
Datagrams (max data segment is 1436 bytes):

Conclusion

In conclusion, understanding the BGP negotiation process over AWS Site-to-Site VPN and Direct Connect connections is crucial for building and maintaining efficient networking infrastructures in the cloud. By employing BGP for dynamic routing, enterprises can optimize data transfer and ensure resilient connectivity between on-premises networks and AWS VPCs. Additionally, this paper's comprehensive troubleshooting steps offer valuable insights for addressing common BGP-related challenges, empowering network administrators and cloud practitioners to swiftly resolve issues and maintain seamless data flow. Embracing these strategies will enhance the overall performance and reliability of AWS-based networks, enabling organizations to leverage the full potential of cloud services while delivering an exceptional user experience.

References:

  1. What Is BGP?
  2. How do I troubleshoot BGP connection failure over AWS VPN or Direct Connect?
  3. What is AWS Site-to-Site VPN?
  4. What is AWS Direct Connect?
  5. Transit gateway Connect attachments and Transit Gateway Connect peers
  6. BGP Neighbor Adjacency States
  7. Dynamic Negotiation of BGP Capabilities