Transit Gateway Logs - Bug report

0

I created an Athena table for vpc flow logs according to the documentation. However I noticed an inconsistency when comparing TG flow logs to other VPC flow logs.

For other VPC flow logs the column account_id contains my account id (int) and interface_id contains eg 'eni-0eb0ac3356dee8030'.
For TG logs account_id contains the string 'TransitGateway' and the interface_id contains my account id.

To me this seems like a bug. Especially since it will mess up all tables that expect an int for account_id...

asked 2 years ago292 views
2 Answers
1

This can be done using the below query:

CREATE EXTERNAL TABLE IF NOT EXISTS tgwflowlogspqttbl ( version int, resource_type string, account_id string, tgw_id string, tgw_attachment_id string, tgw_src_vpc_account_id string, tgw_dst_vpc_account_id string, tgw_src_vpc_id string, tgw_dst_vpc_id string, tgw_src_subnet_id string, tgw_dst_subnet_id string, tgw_src_eni string, tgw_dst_eni string, tgw_src_az_id string, tgw_dst_az_id string, tgw_pair_attachment_id string, srcaddr string, dstaddr string, srcpor int, dstport int, protocol bigint, packet bigint, bytes bigint, start bigint, end bigint, log_status string, type string, packets_lost_no_route bigint, packets_lost_blackhole bigint, packets_lost_mtu_exceeded bigint, packets_lost_ttl_expired bigint, tcp_flags int, region string, flow_direction string, pkt_src_aws_service string, pkt_dst_aws_service string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' LOCATION 's3://bucket/tgw-logs-parquet/' TBLPROPERTIES ( 'skip.header.line.count'='1' )

AWS
answered 11 days ago
0

Hi There

Querying TGW flow logs with Athena is not currently supported

See https://docs.aws.amazon.com/vpc/latest/tgw/tgw-flow-logs.html

You can search TGW flow logs using Cloudwatch

See https://docs.aws.amazon.com/vpc/latest/tgw/working-with-flow-logs.html#search-flow-log-records

profile pictureAWS
EXPERT
Matt-B
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions