Session Manager Logs in S3 in a structured format

0

Hello,

I'm playing around with Session Manager and I enabled logging to S3 Bucket, however logs are sent to the S3 bucket in a raw format and that makes it unpossible to query through the Athena service. Is there any nice way to send them in a more structured format, so I can parse them easily to find information that I need?

1 Risposta
1

You're right, the default Session Manager logs stored in S3 are just raw text files which don't work well for querying in Athena. There are a couple options to get them into a more structured format:

  • Use Amazon CloudWatch Logs instead of S3. The session logs will be sent to CloudWatch Logs and you can then enable the CloudWatch Logs integration in Athena. This will automatically create tables mapped to your log groups that are queryable.
  • Keep using S3 but enable S3 event notifications on the bucket to trigger a Lambda function. Have the Lambda parse the log files and transform them into JSON/Parquet and save back to S3. Then query those structured files.
  • Use a log processing service like Logstash or Fluentd agent on the EC2 instances. Have it tail the session manager logs locally and ship them to S3 in JSON format.
  • Switch to using AWS Systems Manager OpsCenter for aggregating and querying the session logs instead of Athena. OpsCenter has built-in intepretation of session logs.

The CloudWatch Logs integration is probably the easiest way to quickly enable querying in Athena. But any of these options can work to get the logs into a structured format that Athena can understand.

profile pictureAWS
con risposta 9 mesi fa
profile pictureAWS
ESPERTO
kentrad
verificato 9 mesi fa

Accesso non effettuato. Accedi per postare una risposta.

Una buona risposta soddisfa chiaramente la domanda, fornisce un feedback costruttivo e incoraggia la crescita professionale del richiedente.

Linee guida per rispondere alle domande