Cost-Effective Solutions for Collecting and Accessing Large EC2 Instance Logs in S3

I have been working on collecting EC2 instance logs into an S3 bucket. My primary goal is to collect these logs and provide developers with the ability to debug them through a user interface or dashboard, rather than manually accessing the EC2 instance for log analysis.

Due to project requirements, we need to gather all types of logs—debug, info, and error logs.

Previously, we used CloudWatch by deploying the aws.greengrass.LogManager component, which sent all EC2 logs to CloudWatch. However, this approach became costly due to the*** large volume of EC2 log files.***

I am now looking for a more cost-effective solution that allows us to collect EC2 logs into S3. Real-time collection is not necessary; a solution that collects logs at the end of each day would be sufficient.

Let me summarize what approached we've tried so far:

• We trigger a Lambda function via EventBridge at the end of each day. This Lambda function sends a command to copy all EC2 log files to an S3 bucket. However, the issue with this approach is the high likelihood of log file duplication on subsequent triggers (e.g., on day two) since logs are generated continuously.

• We also tried triggering a command/document through AWS Systems Manager (SSM), which sends a command to copy all EC2 log files to an S3 bucket.

• We've been exploring the Kinesis Firehose approach, where a Kinesis agent on the EC2 instances streams logs to a Firehose stream, which then sends the log files to S3.

Could you please AWS-managed, cost-effective options for handling large EC2 log files in S3, and I am also interested in technologies that could enable developers to access these logs easily.