By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

How to read older S3 logs

0

Hi Team,

We are managing the all logs in S3 bucket but I am getting an issue to read logs files. How to read logs files from S3 bucket ? Is there any way ?

Thanks,

Topics
Storage
Tags
Amazon Simple Storage Service
Language
English
AnujG
asked 2 years ago453 views
3 Answers
1

AWS Native Way: Most effective way of reading/querying logs stored in S3 is, crawl them through Glue crawler and have athena tables created through this.

Once you'd have athena tables, you can easily query those logs.

If you can create schema by yourself then glue crawler wouldn't be required, you can simply create tables in athena with s3 source and start querying, however it may be tedious for you to handle this, so glue crawler is perfect tool, which does the job.

References:

Parent Document:

Sub-documents:

  1. Querying AWS WAF logs

  2. Querying AWS Cloudtrail logs

  3. Querying Web Server logs

Reading logs through s3 querying won't be scalable and effective, it'd be quite overwhelming if you need to find any specific event, error etc.

It's always advisable to use athena service to query s3 logs as long as your preference is not to use any external log monitoring solution.

Hope you find this useful.

Abhishek

profile pictureAWS
EXPERT
secondabhi_aws
answered 2 years ago
  • AnujG
    2 years ago

    Hi Abhishek,

    How to fetch data from athena. We have some custom logs. Do we need to create table manually for these logs file to read data ? Is there automatic way to read the logs ?

  • secondabhi_aws EXPERT
    2 years ago

    Hi Anuj Creating tables manually would be very overwhelming process. I'd suggest you to go with glue crawler. It works this way:

    1. Glue console -> Data Catalog -> Crawlers -> Create Crawler -> Add data source -> Use s3 source
    2. Glue crawler would create tables and these tables would be there in Athena
    3. Once Athena tables would be created, go to Athena console -> setup workgroup -> Use this workgroup while querying -> Based on this workgroup setting query results would be saved to s3 bucket as well.
    4. You can query these tables and these tables would be fetching the data from S3 source.

    Comment here, happy to assist.

0

What are the log files (WAF/ALB) etc and what issues are you facing?

profile picture
EXPERT
Gary Mclean
answered 2 years ago
  • AnujG
    2 years ago

    I have multiple logs files application logs file in txt format, ALB and WAF logs files also.

    Could you please explain me to read the logs in easy and quick way ?

    Thanks, Anuj

  • Gary Mclean EXPERT
    2 years ago

    Use Athena to query. You don’t need to use glue crawler you can just create the tables and query straight off the bat

0

Use Athena to create a table while pointing to the S3 bucket and query using native SQL

https://docs.aws.amazon.com/athena/latest/ug/application-load-balancer-logs.html

profile picture
EXPERT
Gary Mclean
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content