使用AWS re:Post即您表示您同意 AWS re:Post 使用条款
AWS re:Postre:Post

Where to find detailed AWS Glue Logs

0

Hello All,

I am trying to implement solution mentioned at below link :

https://medium.com/analytics-vidhya/multithreading-parallel-job-in-aws-glue-a291219123fb

In this solution they have shown AWS logs showing of Scheduler settings . I am not getting where will I get these complete logs . I am running Glue job from console and there I see 3 types of logs All logs Output Logs Error logs

When I am opening all logs > I don't get anything Output logs > getting output what I am printing in my script . This also sure it also shows something related to pyspark application . Error log > I am not sure about

  • Sanjeeb
    1 年前

    When you are configuring your glue job and if it is a spark job, you can mention the spark log path in your job configuration. Also if you are having simple etl pipelines, try glue studio and under the config, you can set up the log files path to S3 and when a job is failed or success, you can see the logs. Error logs is important when the job is failed so that you can track the issue.

主题
分析管理与治理
标签
AWS GlueAmazon CloudWatch 日志
语言
English
AWS-User-6494745
已提问 1 年前720 查看次数
1 回答
0

All logs will show when you enable "Continuous logging" in the job settings.
Err log really shows all the Spark/Glue logging. If you don't see those scheduler log entries is probably because they are omitted, by default most INFO logging are disabled. You can try chaging sparkContext.setLogLevel()

profile pictureAWS
专家
Gonzalo Herreros
已回答 1 年前

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈，并能促进提问者的职业发展。

回答问题的准则

相关内容