How do I find the top contributors of inter-AZ traffic or zonal data transfers in Amazon VPC?

Short description

To analyze spikes in your charges or locate resources that contribute to your charges, use Amazon VPC custom flow logs. If you already use custom flow logs but the logs don't include the AZ ID, then you must create a new flow log with a custom format. Then, include the az-id field.

To find the resources that contribute to your inter-AZ traffic or data transfer charges, use one of the following methods:

• Publish the flow log data to Amazon CloudWatch Logs, and then query the logs.
• Publish the flow logs to an Amazon Simple Storage Service (Amazon S3) bucket, and then use Amazon Athena to query the logs.

Note: Amazon VPC flow logs and queries support only the AZ ID. For more information, see AZ IDs.

Resolution

Query logs in CloudWatch Logs

1.    Open the CloudWatch console.

2.    In the navigation pane, choose Logs Insights.

3.    From the dropdown list, select the destination log group for your Amazon VPC flow logs.

4.    To view the top contributors of inter-AZ traffic, run the following command for each Availability Zone in the AWS Region:

Note: Replace example-az-id with your Availability Zone ID and example-y.y. with the first two octets of the Amazon VPC CIDR range.

filter (azId not like 'example-az-id') and (dstAddr like 'example-y.y.' and srcAddr like 'example-y.y.')
| stats sum(bytes) as bytesTransferred by srcAddr,dstAddr,azId
| sort bytesTransferred desc
| limit 20

5.    View resources in other Availability Zones that send or receive traffic from a specific IP address in an Availability Zone:

Note: Replace example-az-id with your Availability Zone ID and example-private-ip with the private IP address for the destinations that you want to check.

filter (azId not like '<example-az-id>') and (dstAddr like 'example-private-ip' or srcAddr like 'example-private-ip')
| stats sum(bytes) as bytesTransferred by srcAddr,dstAddr,azId
| sort bytesTransferred desc
| limit 20

Use Athena to query logs in an Amazon S3 bucket

1.    Open the Amazon VPC console or the Athena console.

2.    Create a table. Note the table name and the database name.

3.    View the resources that send inter-AZ traffic for each Availability Zone in the Region:

Note: Replace example-database.example-table-name with the database and table name and example-az-id with your Availability Zone ID. Replace example-y.y. with the first two octets of the Amazon VPC CIDR range. For an Athena query, use example-y.y.% instead of example-y.y.

SELECT srcaddr,dstaddr,az_id, sum(bytes) as bytesTransferred FROM "example-database"."example-table-name"
WHERE (az_id not like 'example-az-id') AND (dstaddr like 'example-y.y.' AND srcaddr like 'example-y.y.') group by 1,2,3 order by 4 desc
limit 20

4.    View the resources in other Availability Zones that send or receive traffic from a specific IP address in the Amazon VPC:

Note: Replace example-database.example-table-name with the database and table name and example-az-id with your Availability Zone ID. Replace example-private-ip with the private IP address for the destinations that you want to check.

SELECT srcaddr,dstaddr,az_id, sum(bytes) as bytesTransferred FROM "example-database"."example-table-name"
WHERE (az_id not like 'example-az-id') AND (dstaddr like 'example-private-ip' or srcaddr like 'example-private-ip') group by 1,2,3 order by 4 desc
limit 20

5.    View the top zonal data transfer contributors in a specific time period:

Note: Replace example-database.example-table-name with the database and table name and example-yyyy-mm-dd-hh:mm:ss with your date and time, for example, 2000-01-01 12:23:34. Replace example-az-id with your Availability Zone ID and example-y.y. with the first two octets of the Amazon VPC CIDR range. For an Athena query, use example-y.y.% instead of example-y.y.

SELECT srcaddr,dstaddr,az_id, sum(bytes) as bytesTransferred FROM "example-database"."example-table-name"
WHERE from_unixtime(start) > parse_datetime 'example-yyyy-MM-dd-HH:mm:ss'
AND from_unixtime(start)< parse_datetime 'example-yyyy-MM-dd-HH:mm:ss'
AND (az_id not like 'example-az-id') AND (dstaddr like 'example-y.y.' AND srcaddr like 'example-y.y.') group by 1,2,3 order by 4 desc
limit 20

Related information

Data transfer within an AWS Region

Analyze VPC flow logs with point-and-click Amazon Athena integration

How do I find the top contributors to NAT gateway traffic in my Amazon VPC?

Overview of data transfer costs for common architectures