Session Manager Logs in S3 in a structured format

0

Hello,

I'm playing around with Session Manager and I enabled logging to S3 Bucket, however logs are sent to the S3 bucket in a raw format and that makes it unpossible to query through the Athena service. Is there any nice way to send them in a more structured format, so I can parse them easily to find information that I need?

1 Answer
1

You're right, the default Session Manager logs stored in S3 are just raw text files which don't work well for querying in Athena. There are a couple options to get them into a more structured format:

  • Use Amazon CloudWatch Logs instead of S3. The session logs will be sent to CloudWatch Logs and you can then enable the CloudWatch Logs integration in Athena. This will automatically create tables mapped to your log groups that are queryable.
  • Keep using S3 but enable S3 event notifications on the bucket to trigger a Lambda function. Have the Lambda parse the log files and transform them into JSON/Parquet and save back to S3. Then query those structured files.
  • Use a log processing service like Logstash or Fluentd agent on the EC2 instances. Have it tail the session manager logs locally and ship them to S3 in JSON format.
  • Switch to using AWS Systems Manager OpsCenter for aggregating and querying the session logs instead of Athena. OpsCenter has built-in intepretation of session logs.

The CloudWatch Logs integration is probably the easiest way to quickly enable querying in Athena. But any of these options can work to get the logs into a structured format that Athena can understand.

profile pictureAWS
answered 8 months ago
profile pictureAWS
EXPERT
kentrad
reviewed 8 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions