- 최신
- 최다 투표
- 가장 많은 댓글
Hey,
Thanks for using SageMaker! This is an issue in pyspark3 with latest Livy. Starting with version 0.5.0-incubating, session kind “pyspark3” is removed, instead users require to set PYSPARK_PYTHON to python3 executable[1].
So there're two options:
- You can switch to use pyspark kernel instead of pyspark3.
- You can set PYSPARK_PYTHON variable in EMR's config file for spark: spark-env.sh
[
{
"Classification": "spark-env",
"Configurations": [
{
"Classification": "export",
"Configurations": [],
"Properties": {
"PYSPARK_PYTHON": "/usr/bin/python3"
}
}
],
"Properties": {}
}
]
Let us know if you have any other question.
Thanks,
Han
[1]https://livy.incubator.apache.org/docs/latest/rest-api.html#pyspark
[2]https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-configure.html
Submitting these EMR configuration options at the time of cluster creation worked for me:
[
{
"Classification": "spark-env",
"Configurations": [
{
"Classification": "export",
"Properties": {
"PYSPARK_PYTHON": "/usr/bin/python3"
}
}
]
},
{
"Classification": "yarn-env",
"Properties": {},
"Configurations": [
{
"Classification": "export",
"Properties": {
"PYSPARK_PYTHON": "/usr/bin/python3",
}
}
]
}
]
관련 콘텐츠
- AWS 공식업데이트됨 일 년 전
- AWS 공식업데이트됨 2년 전
- AWS 공식업데이트됨 일 년 전