Usando AWS re:Post, accetti AWS re:Post Termini di utilizzo
AWS re:Postre:Post

Where to find detailed AWS Glue Logs

0

Hello All,

I am trying to implement solution mentioned at below link :

https://medium.com/analytics-vidhya/multithreading-parallel-job-in-aws-glue-a291219123fb

In this solution they have shown AWS logs showing of Scheduler settings . I am not getting where will I get these complete logs . I am running Glue job from console and there I see 3 types of logs All logs Output Logs Error logs

When I am opening all logs > I don't get anything Output logs > getting output what I am printing in my script . This also sure it also shows something related to pyspark application . Error log > I am not sure about

  • Sanjeeb
    un anno fa

    When you are configuring your glue job and if it is a spark job, you can mention the spark log path in your job configuration. Also if you are having simple etl pipelines, try glue studio and under the config, you can set up the log files path to S3 and when a job is failed or success, you can see the logs. Error logs is important when the job is failed so that you can track the issue.

Argomenti
AnalisiGestione e amministrazione
Tag
AWS GlueRegistri di Amazon CloudWatch
Lingua
English
AWS-User-6494745
posta un anno fa731 visualizzazioni
1 Risposta
0

All logs will show when you enable "Continuous logging" in the job settings.
Err log really shows all the Spark/Glue logging. If you don't see those scheduler log entries is probably because they are omitted, by default most INFO logging are disabled. You can try chaging sparkContext.setLogLevel()

profile pictureAWS
ESPERTO
Gonzalo Herreros
con risposta un anno fa

Accesso non effettuato. Accedi per postare una risposta.

Una buona risposta soddisfa chiaramente la domanda, fornisce un feedback costruttivo e incoraggia la crescita professionale del richiedente.

Linee guida per rispondere alle domande

Contenuto pertinente