By using AWS re:Post, you agree to the Terms of Use
AWS re:Postre:Post

Where to find detailed AWS Glue Logs

0

Hello All,

I am trying to implement solution mentioned at below link :

https://medium.com/analytics-vidhya/multithreading-parallel-job-in-aws-glue-a291219123fb

In this solution they have shown AWS logs showing of Scheduler settings . I am not getting where will I get these complete logs . I am running Glue job from console and there I see 3 types of logs All logs Output Logs Error logs

When I am opening all logs > I don't get anything Output logs > getting output what I am printing in my script . This also sure it also shows something related to pyspark application . Error log > I am not sure about

  • Sanjeeb a month ago

    When you are configuring your glue job and if it is a spark job, you can mention the spark log path in your job configuration. Also if you are having simple etl pipelines, try glue studio and under the config, you can set up the log files path to S3 and when a job is failed or success, you can see the logs. Error logs is important when the job is failed so that you can track the issue.

Topics
AnalyticsManagement & Governance
Tags
AWS GlueAmazon CloudWatch Logs
Language
English
AWS-User-6494745
asked a month ago47 views
1 Answer
0

All logs will show when you enable "Continuous logging" in the job settings.
Err log really shows all the Spark/Glue logging. If you don't see those scheduler log entries is probably because they are omitted, by default most INFO logging are disabled. You can try chaging sparkContext.setLogLevel()

profile picture
AWS
Gonzalo Herreros
answered a month ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content