Sending data from EC2 to S3

0

Hii everyone, I am running Websocket Server in python in EC2 and i am storing exchanged messages in as a .log file. I want to store the exchanged messages to S3 continuously. can any one please tell me how to do this?? Thanking you in advance

2 Answers
0

How continuously do you mean? A simple approach would be to use Boto3 in Python to write messages as individual objects in S3 as you process them, but I don't know how small these messages are and how many you get.

Another approach is to use the CloudWatch Logs agent on the EC2 instance to push your .log files' messages into CloudWatch, then set a subscription there feeding to Kinesis Firehose which can deliver to S3.

EXPERT
answered a year ago
  • i am sending sensors data to websocket server continuously for a month and from server i want to store in s3. could you please tell me the best approach that will work?

  • It sounds like you've got a stream of small bits of data so I think Kinesis Firehose delivering to S3 is best, as it can look after buffering etc. Either have your code directly Put to the Kinesis stream, or go via CloudWatch Logs as I mentioned.

0

Configure FluentBit or FluentD and simply send to S3.

It is recommended to configure the log agent separately so that it does not affect the application. https://docs.fluentbit.io/manual/pipeline/outputs/s3

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions