- Le plus récent
- Le plus de votes
- La plupart des commentaires
Two things you can set the spark log in session or log4j files which will take effect for all the invocations.
- for session level continue to use the sc.setLogLevel(newLevel) - newLevel like OFF and see it that helps.
- secondly if session setting is not working use log4j file to tuen off . go to file like $SPARK_HOME/conf/log4j.properties or $SPARK_HOME/conf/log4j.properties.template ( move it to $SPARK_HOME/conf/log4j.properties) and set log4j.rootCategory to OFF (default was DEBUG change to OFF). more details on this check the below link :- https://jaceklaskowski.gitbooks.io/mastering-spark-sql/content/spark-logging.html https://community.cloudera.com/t5/Support-Questions/Config-log4j-in-Spark/td-p/34968
For editing log4j either you log into EMR master by SSH to ec2 machine for existing running EMR cluster and use VI editor for editing the log4j.properties. if you are spinning up a new EMR and configure the log4j without logging into master node then use EMR configuration. Below link state how to use the EMR configuration https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-configure-apps.html EMR property is "spark-log4j". Please refer the section "Configuration classifications" in below link https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-650-release.html ( you can browse the configuration based on the EMR version you are using)
Thank you for the useful information.
Contenus pertinents
- demandé il y a un an
- demandé il y a un an
- AWS OFFICIELA mis à jour il y a 2 ans
- AWS OFFICIELA mis à jour il y a 3 ans
- AWS OFFICIELA mis à jour il y a 2 ans
Thank you for your answers. How can we make these changes from the AWS EMR console?