How can I use Cloudwatch logs insight to analyze Transit Gateway flow logs?

Short description

When you enable Transit Gateway flow logs in the target CloudWatch Logs, a separate log stream is generated for each attachment type (VPC, VPN, Direct Connect, and Peering). You can use CloudWatch Logs Insights to troubleshoot issues like connectivity between VPCs, VPN connections, Direct Connect, and peering attachments, analyze traffic patterns between attachments, and By monitoring specific events like 'packetsLostNoRoute' and 'packetsLostBlackhole', you can effectively identify and resolve routing problems between the attachments.

Resolution:

Enable Transit Gateway flow logs and use CloudWatch console to run queries for traffic analysis. Access previous queries via History, and export results in desired format.

1. Enable the transit gateway flow logs
2. Go to CloudWatch console
3. In the navigation pane, choose Log Insights
4. Select the Log Group name used for transit gateway logs and use the below queries as per use case

Example queries:

Scenario 1: Identify packet loss due to blackhole routes

You are observing packets being dropped due to blackhole routes in your Transit Gateway, and need to identify the affected traffic flows and root cause using CloudWatch Logs Insights to analyze Transit Gateway flow logs.

parse @message "* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *" as version, resourcetype, accountid, tgwid, tgwattachmentid, tgwsrcvpcaccountid, tgwdstvpcaccountid, tgwsrcvpcid, tgwdstvpcid, tgwsrcsubnetid, tgwdstsubnetid, tgwsrceni, tgwdsteni, tgwsrcazid, tgwdstazid, tgwpairattachmentid, srcaddr, dstaddr, srcport, dstport, protocol, packets, bytes, start, end, logstatus, type, packetslostnoroute, packetslostblackhole, packetslostmtuexceeded, packetslostttlexpired, tcpflags, region, flowdirection, pktsrcawsservice, pktdstawsservice 
 | filter packetslostblackhole not like "0"
 | sort @message desc
 | limit 50

One of the most common causes: Static routes in Transit Gateway route tables are pointing to a VPN attachment where both tunnels are down, causing the routes to be blackholed.

Scenario 2: Identify packet loss due to no route

You are observing packets being dropped due to missing routes in your Transit Gateway, and need to identify the affected traffic flows and root cause using CloudWatch Logs Insights to analyze Transit Gateway flow logs.

parse @message "* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *" as version, resourcetype, accountid, tgwid, tgwattachmentid, tgwsrcvpcaccountid, tgwdstvpcaccountid, tgwsrcvpcid, tgwdstvpcid, tgwsrcsubnetid, tgwdstsubnetid, tgwsrceni, tgwdsteni, tgwsrcazid, tgwdstazid, tgwpairattachmentid, srcaddr, dstaddr, srcport, dstport, protocol, packets, bytes, start, end, logstatus, type, packetslostnoroute, packetslostblackhole, packetslostmtuexceeded, packetslostttlexpired, tcpflags, region, flowdirection, pktsrcawsservice, pktdstawsservice 
 | filter packetslostnoroute not like "0"
 | sort @message desc
 | limit 50

One of the most common causes: Route propagation is disabled on the Transit Gateway route table, preventing routes from being learned from attached VPCs or other networks, resulting in dropped traffic due to no valid route to destination.

This scenario helps identify:

• Traffic flows being dropped (source/destination IPs, ports, protocols)
• Missing routes in Transit Gateway route tables
• Route propagation configuration issues

Scenario 3: Identify the top talkers

You are observing high utilization across your Transit Gateway and need to identify the top traffic generators to understand bandwidth consumption patterns and optimize network resources using CloudWatch Logs Insights to analyze Transit Gateway flow logs.

parse @message '* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *' as Version,restype,accID,tgwid,tgwattid,tgwsrcvpcaccid,tgwdstvpcaccid,tgwsrcvpcid,tgwdstvpcid,tgwsrcsubid,tgwdstsubid,tgwsrceni,tgwdsteni,tgwsrcaz,tgwdstaz,tgwpairattid,srcip,dstip,sport,dport,protocol,pkts,bytes,start,end,status,type,pktlossnort,pktlossblkhol,pktlossmtu,pktlossttl,tcpflags,region,flowdir,pktsrcservice,pktdstservice 
 | stats sum(bytes) as Data_Transferred by tgwpairattid 
 | sort by Data_Transferred desc 
 | limit 50

One of the most common use cases: Analyze Transit Gateway flow logs through CloudWatch Logs Insights to identify top bandwidth consumers and optimize network resources.

This scenario helps identify:

• Highest bandwidth consuming attachments
• Traffic patterns between attachments
• Resource utilization trends
• Potential network optimization opportunities

Scenario 4: Detailed Traffic Flow Analysis by Source-Destination Pairs

You want to perform detailed traffic analysis to understand the specific source-destination pairs generating high volumes of traffic across your Transit Gateway environment. Using CloudWatch Logs Insights will help identify not just the top talkers, but also their associated VPC details and attachment information.

parse @message '* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *' as Version,restype,accID,tgwid,tgwattid,tgwsrcvpcaccid,tgwdstvpcaccid,tgwsrcvpcid,tgwdstvpcid,tgwsrcsubid,tgwdstsubid,tgwsrceni,tgwdsteni,tgwsrcaz,tgwdstaz,tgwpairattid,srcip,dstip,sport,dport,protocol,pkts,bytes,start,end,status,type,pktlossnort,pktlossblkhol,pktlossmtu,pktlossttl,tcpflags,region,flowdir,pktsrcservice,pktdstservice 
 | stats sum(bytes) as Data_Transferred by srcip, dstip, tgwpairattid, tgwdstvpcaccid, tgwdstvpcid 
 | sort by Data_Transferred desc 
 | limit 50

A typical use case: Perform granular analysis of Transit Gateway traffic patterns by examining source-destination pairs, VPC details, and attachment information using CloudWatch Logs Insights for comprehensive network visibility.

This helps organizations:

• Track specific IP-level communications
• Validate application communication flows
• Identify unexpected traffic sources or destinations

Scenario 5: Troubleshoot connectivity between source and destination You are investigating reported connectivity issues between two specific endpoints (source and destination IPs) across different VPCs connected through Transit Gateway. Using CloudWatch Logs Insights will help analyze if traffic is flowing between these specific endpoints, what ports are being used, and whether connections are being established successfully.

parse @message '* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *' as Version,restype,accID,tgwid,tgwattid,tgwsrcvpcaccid,tgwdstvpcaccid,tgwsrcvpcid,tgwdstvpcid,tgwsrcsubid,tgwdstsubid,tgwsrceni,tgwdsteni,tgwsrcaz,tgwdstaz,tgwpairattid,srcip,dstip,sport,dport,protocol,pkts,bytes,start,end,status,type,pktlossnort,pktlossblkhol,pktlossmtu,pktlossttl,tcpflags,region,flowdir,pktsrcservice,pktdstservice 
| filter (srcip like '10.x.x.x' and dstip like '10.x.x.x') or (srcip like '10.5.2.30' and dstip like '10.x.x.x')
| stats count(*) as flow_count 
by srcip, dstip, sport, dport, protocol, flowdir, tgwsrcvpcid, tgwdstvpcid
| sort by srcip, dstip

A typical use case: When troubleshooting reported connectivity issues between any two endpoints, you need to validate traffic flow, identify any blocked ports, or determine if traffic is following the expected network path.“

This helps validate:

• Traffic flow between specific IP pairs
• Connection status and port usage
• Communication patterns between endpoints

References:

---

[1] Transit Gateway Flow Logs records in Amazon CloudWatch Logs - https://docs.aws.amazon.com/vpc/latest/tgw/flow-logs-cwl.html

[2] Supported logs and discovered fields - https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_AnalyzeLogData-discoverable-fields.html

[3] CloudWatch Logs Insights language query syntax - https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html