Is there a method to automatically name the files with the date or a specified name when exporting from CloudWatch to S3?

Question

Hello,
I was wondering if there is a way to automatically name the exported files to S3 with the current date or a specified name when exporting from CloudWatch? Currently, I am looking to send ECS logs to both CloudWatch and S3. My initial approach involves sending ECS logs to CloudWatch and then exporting them to S3. However, I noticed that the exported file is named '0000.gz'. Ideally, I would prefer the file to be named with the current date or month, such as '20240401.gz'. Is there a method to automatically name the files with the date or a specified name when exporting from CloudWatch to S3? Alternatively, are there better approaches to achieve the goal of sending ECS logs to both CloudWatch and S3?

Accepted Answer

Hello.

How about setting up an S3 event notification to notify Lambda when it is exported to S3 and renaming the file in Lambda?  
If you can program with Python etc., you can use the Python AWS SDK called boto3.  
https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html

It is possible to do something similar to rename by combining "copy_object" and "delete_object".  
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/copy_object.html  
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/delete_object.html

If you use Kinesis Data Firehose, you can also output files by date hierarchy.  
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html

Is there a method to automatically name the files with the date or a specified name when exporting from CloudWatch to S3?

関連するコンテンツ