By using AWS re:Post, you agree to the Terms of Use

Problem Sagemaker and Spark

0

Hi there,

I followed this tutorial to set up Sagemaker Notebook with Spark (EMR): https://aws.amazon.com/blogs/machine-learning/build-amazon-sagemaker-notebooks-backed-by-spark-in-amazon-emr/

I launched a notebook with sparkmagic (pyspark3) and tried to call the Spark context but got the following error:
"""
The code failed because of a fatal error:
Invalid status code '400' from http://xxx.xx.xx.xx:8998/sessions with error payload: "Invalid kind: pyspark3 (through reference chain: org.apache.livy.server.interactive.CreateInteractiveRequest["kind"])".

Some things to try:
a) Make sure Spark has enough available resources for Jupyter to create a Spark context.
b) Contact your Jupyter administrator to make sure the Spark magics library is configured correctly.
c) Restart the kernel.
"""

Anyone encountered the same issue?

asked 4 years ago485 views
2 Answers
0

Hey,

Thanks for using SageMaker! This is an issue in pyspark3 with latest Livy. Starting with version 0.5.0-incubating, session kind “pyspark3” is removed, instead users require to set PYSPARK_PYTHON to python3 executable[1].

So there're two options:

  1. You can switch to use pyspark kernel instead of pyspark3.
  2. You can set PYSPARK_PYTHON variable in EMR's config file for spark: spark-env.sh
    [
    {
    "Classification": "spark-env",
    "Configurations": [
    {
    "Classification": "export",
    "Configurations": [],
    "Properties": {
    "PYSPARK_PYTHON": "/usr/bin/python3"
    }
    }
    ],
    "Properties": {}
    }
    ]

Let us know if you have any other question.

Thanks,
Han

[1]https://livy.incubator.apache.org/docs/latest/rest-api.html#pyspark
[2]https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-configure.html

answered 4 years ago
0

Submitting these EMR configuration options at the time of cluster creation worked for me:
[
{
"Classification": "spark-env",
"Configurations": [
{
"Classification": "export",
"Properties": {
"PYSPARK_PYTHON": "/usr/bin/python3"
}
}
]
},
{
"Classification": "yarn-env",
"Properties": {},
"Configurations": [
{
"Classification": "export",
"Properties": {
"PYSPARK_PYTHON": "/usr/bin/python3",
}
}
]
}
]

answered 4 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions