- Newest
- Most votes
- Most comments
Two things you can set the spark log in session or log4j files which will take effect for all the invocations.
- for session level continue to use the sc.setLogLevel(newLevel) - newLevel like OFF and see it that helps.
- secondly if session setting is not working use log4j file to tuen off . go to file like $SPARK_HOME/conf/log4j.properties or $SPARK_HOME/conf/log4j.properties.template ( move it to $SPARK_HOME/conf/log4j.properties) and set log4j.rootCategory to OFF (default was DEBUG change to OFF). more details on this check the below link :- https://jaceklaskowski.gitbooks.io/mastering-spark-sql/content/spark-logging.html https://community.cloudera.com/t5/Support-Questions/Config-log4j-in-Spark/td-p/34968
For editing log4j either you log into EMR master by SSH to ec2 machine for existing running EMR cluster and use VI editor for editing the log4j.properties. if you are spinning up a new EMR and configure the log4j without logging into master node then use EMR configuration. Below link state how to use the EMR configuration https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-configure-apps.html EMR property is "spark-log4j". Please refer the section "Configuration classifications" in below link https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-650-release.html ( you can browse the configuration based on the EMR version you are using)
Thank you for the useful information.
Relevant content
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 7 months ago
- AWS OFFICIALUpdated a year ago
Thank you for your answers. How can we make these changes from the AWS EMR console?