Is CloudWatch "Vended Log" cost for VPC Flow Log delivery to S3 calculated on GB before or after compression?

0

Background

When VPC flow logs are enabled and delivered to S3, you incur a CloudWatch "Vended Log" cost for "each GB of Data Ingested". CloudWatch delivers these flow logs to S3 as gzip-compressed text by default and you can instead optionally choose Parquet. In either case, this means stored S3 object size will be smaller than the raw log files.

Question

At what point in the data flow does CloudWatch calculate the size (in GB) of flow logs delivered to S3? i.e is it before or after the results are compressed?

The pricing docs say its based on "Data Ingested", which to me suggests pre-compression logs. However, when I look at an example line item from the AWS Cost & Usage Report, the line description says "$0.xxx per GB of log data delivered to S3", and to me, "delivered", would imply usage is measured post-compression.

Before anyone asks, I do not currently have the ability to look at the destination bucket myself; otherwise, I could just compare size of objects in S3 to the corresponding flow log billing detail.

2 Answers
0

I can't speak to VPC Flow Logs, but we recently had that same question about WAF logs, also sent as CloudWatch Vended Logs with "Delivery to S3" (doc), gzip-compressed (relevant-ish doc).

A large event on our WAF resulted in a corresponding billing surge, and AWS support helped us to clarify: in the case of WAF, the pre-compressed log volume is billed, showing in Cost Explorer for Cloudwatch under Usage type [Region]-S3-Egress-Bytes. It doesn't seem the post-compressed logs are billed, not under Cloudwatch nor S3.

Noteworthy context: in our WAF situation, the gzip compression reached a 40× ratio, resulting in a massive difference between billed usage, and the size of gz logs reaching S3.

_

I agree the language "delivered" suggests post-compression measurements, and even more so the …-S3-**Egress**-Bytes billing Usage type. Although this might be why they associate this "Data Ingested" charge with CloudWatch and not S3 with its own ingestion costs.

answered 9 months ago
-1

Hello, It should not matter whether the data is Pre/Post compressed. S3 operates and calculates the data cost on the data it has as you already mentioned the data stored is already compressed. However, if you want to consider, you can consider it as post compressed from the cost perspective. The charge would be for per GB data in the S3 (post compressed). Hope this helps :)

profile pictureAWS
SUPPORT ENGINEER
answered 2 years ago
  • You misunderstand, I am not asking about the GB-mo storage cost of S3. When VPC flow logs are delivered to S3, CloudWatch charges a one-time "vended logs" fee per GB delivered... and this is what I am referring to in my question. Please see "Vended Logs" section of "Logs" tab at https://aws.amazon.com/cloudwatch/pricing/

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions