Durch die Nutzung von AWS re:Post stimmt du den AWS re:Post Nutzungsbedingungen
AWS re:Postre:Post

Where to find detailed AWS Glue Logs

0

Hello All,

I am trying to implement solution mentioned at below link :

https://medium.com/analytics-vidhya/multithreading-parallel-job-in-aws-glue-a291219123fb

In this solution they have shown AWS logs showing of Scheduler settings . I am not getting where will I get these complete logs . I am running Glue job from console and there I see 3 types of logs All logs Output Logs Error logs

When I am opening all logs > I don't get anything Output logs > getting output what I am printing in my script . This also sure it also shows something related to pyspark application . Error log > I am not sure about

  • Sanjeeb
    vor einem Jahr

    When you are configuring your glue job and if it is a spark job, you can mention the spark log path in your job configuration. Also if you are having simple etl pipelines, try glue studio and under the config, you can set up the log files path to S3 and when a job is failed or success, you can see the logs. Error logs is important when the job is failed so that you can track the issue.

Themen
AnalysenManagement & Unternehmensführung
Tags
AWS GlueAmazon CloudWatch-Protokolle
Sprache
English
AWS-User-6494745
gefragt vor einem Jahr720 Aufrufe
1 Antwort
0

All logs will show when you enable "Continuous logging" in the job settings.
Err log really shows all the Spark/Glue logging. If you don't see those scheduler log entries is probably because they are omitted, by default most INFO logging are disabled. You can try chaging sparkContext.setLogLevel()

profile pictureAWS
EXPERTE
Gonzalo Herreros
beantwortet vor einem Jahr

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen

Relevanter Inhalt