- Newest
- Most votes
- Most comments
AWS Native Way: Most effective way of reading/querying logs stored in S3 is, crawl them through Glue crawler and have athena tables created through this.
Once you'd have athena tables, you can easily query those logs.
If you can create schema by yourself then glue crawler wouldn't be required, you can simply create tables in athena with s3 source and start querying, however it may be tedious for you to handle this, so glue crawler is perfect tool, which does the job.
References:
Parent Document:
Sub-documents:
Reading logs through s3 querying won't be scalable and effective, it'd be quite overwhelming if you need to find any specific event, error etc.
It's always advisable to use athena service to query s3 logs as long as your preference is not to use any external log monitoring solution.
Hope you find this useful.
Abhishek
What are the log files (WAF/ALB) etc and what issues are you facing?
I have multiple logs files application logs file in txt format, ALB and WAF logs files also.
Could you please explain me to read the logs in easy and quick way ?
Thanks, Anuj
Use Athena to query. You don’t need to use glue crawler you can just create the tables and query straight off the bat
Use Athena to create a table while pointing to the S3 bucket and query using native SQL
https://docs.aws.amazon.com/athena/latest/ug/application-load-balancer-logs.html
Relevant content
- asked 8 months ago
AWS OFFICIALUpdated a month ago- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 6 months ago
- AWS OFFICIALUpdated 4 months ago
Hi Abhishek,
How to fetch data from athena. We have some custom logs. Do we need to create table manually for these logs file to read data ? Is there automatic way to read the logs ?
Hi Anuj Creating tables manually would be very overwhelming process. I'd suggest you to go with glue crawler. It works this way:
Comment here, happy to assist.