Parse application logs via Athena and Glue catalog
I have log files in my onpremise application, Individual file size is 100 MB (file rollover/appender defined in Log4J in onpremise application) and 50 such files get generated on daily basis which is 5 GB total. We then zip these files at day end and push to S3 bucket manually today.
- We dont want to use cloudwatch agent in onpremise application to push logs to CloudWatch (as onpremise applications are not having CPU/Memory and running at peak)
- We zip the files at day end else we have to do 50-55 S3 Copy manually each day (yes we can create some script for this - but again not elegant)
- Yes Elasticsearch is an option but we are building solution which will take 2-3 months to ingest data to Elasticsearch and use ELK stack
Now till 1 year, whenever we get customer complaint we have to extract specific file. We know the date, but unfortunately we have to download all 5 GB for that day and extract the required file and content.
As part of this usecase wanted to check: 1. If Athena works for Log4J log files (from JBoss, Websphere) - do we have Serde's (serialization / deserialization libraries) for the same, glue catalog for the same
Have you had a look at the GROK SerDe? This may help you? https://docs.aws.amazon.com/athena/latest/ug/grok-serde.html Example 2 uses log4j format. Hope this helps.
Using AWS PHP SDK on Elastic Beanstalkasked 2 years ago
What is the sizekey parameter in AWS Glue Catalog Data properties?Accepted Answerasked 2 years ago
How to escape a comma in a csv file in AWS Glue?Accepted AnswerMODERATORasked 3 years ago
AWS ELB log analyzerAccepted Answerasked 4 months ago
Parse application logs via Athena and Glue catalogasked 5 months ago
Greengrass device logging online/offline?asked 2 years ago
How can i upload 50 MB file to s3 from edge device in multiple of 1 MB chunked size?asked 2 months ago
Logs only updated on redeployasked 2 years ago
Why is my AWS S3 Usage So High?asked 5 months ago
Policies needed for creating and writing files in simulation applicationAccepted Answerasked 2 years ago