Where to find detailed AWS Glue Logs

0

Hello All,

I am trying to implement solution mentioned at below link :

https://medium.com/analytics-vidhya/multithreading-parallel-job-in-aws-glue-a291219123fb

In this solution they have shown AWS logs showing of Scheduler settings . I am not getting where will I get these complete logs . I am running Glue job from console and there I see 3 types of logs All logs Output Logs Error logs

When I am opening all logs > I don't get anything Output logs > getting output what I am printing in my script . This also sure it also shows something related to pyspark application . Error log > I am not sure about

  • When you are configuring your glue job and if it is a spark job, you can mention the spark log path in your job configuration. Also if you are having simple etl pipelines, try glue studio and under the config, you can set up the log files path to S3 and when a job is failed or success, you can see the logs. Error logs is important when the job is failed so that you can track the issue.

demandé il y a un an644 vues
1 réponse
0

All logs will show when you enable "Continuous logging" in the job settings.
Err log really shows all the Spark/Glue logging. If you don't see those scheduler log entries is probably because they are omitted, by default most INFO logging are disabled. You can try chaging sparkContext.setLogLevel()

profile pictureAWS
EXPERT
répondu il y a un an

Vous n'êtes pas connecté. Se connecter pour publier une réponse.

Une bonne réponse répond clairement à la question, contient des commentaires constructifs et encourage le développement professionnel de la personne qui pose la question.

Instructions pour répondre aux questions