1 Answer
- Newest
- Most votes
- Most comments
1
You're right, the default Session Manager logs stored in S3 are just raw text files which don't work well for querying in Athena. There are a couple options to get them into a more structured format:
- Use Amazon CloudWatch Logs instead of S3. The session logs will be sent to CloudWatch Logs and you can then enable the CloudWatch Logs integration in Athena. This will automatically create tables mapped to your log groups that are queryable.
- Keep using S3 but enable S3 event notifications on the bucket to trigger a Lambda function. Have the Lambda parse the log files and transform them into JSON/Parquet and save back to S3. Then query those structured files.
- Use a log processing service like Logstash or Fluentd agent on the EC2 instances. Have it tail the session manager logs locally and ship them to S3 in JSON format.
- Switch to using AWS Systems Manager OpsCenter for aggregating and querying the session logs instead of Athena. OpsCenter has built-in intepretation of session logs.
The CloudWatch Logs integration is probably the easiest way to quickly enable querying in Athena. But any of these options can work to get the logs into a structured format that Athena can understand.
answered 8 months ago
Relevant content
- asked 8 months ago
- AWS OFFICIALUpdated a month ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 10 months ago