1 réponse
- Le plus récent
- Le plus de votes
- La plupart des commentaires
2
I found how to fix this issue.
I had to use the "spark-defaults" classifications and in the properties have something like {"spark.yarn.appMasterEnv.YOUR_ENV_VARIABLE": "the value"}.
This is explained in the Spark documentation (https://spark.apache.org/docs/latest/configuration.html#environment-variables), however I don't understand why it was working differently in EMR 5.x
répondu il y a 3 ans
Contenus pertinents
- demandé il y a un an
- demandé il y a 6 mois
- Réponse acceptéedemandé il y a un an
- demandé il y a 2 mois
- AWS OFFICIELA mis à jour il y a un an
- AWS OFFICIELA mis à jour il y a un an
- AWS OFFICIELA mis à jour il y a 2 ans
Ran into an issue with EMR 6.60 where the encoding of Jupyter Spark was for some reason in ASCII instead of UTF8 like in Zeppelin and others. The problem only existed in Jupyter for whatever reason. The settings I used for EMR 5.x for the same issue stopped working, using your suggest of spark-defaults fixed the issue. Thanks!