Problem Sagemaker and Spark

Question

Hi there,  
  
I followed this tutorial to set up Sagemaker Notebook with Spark (EMR): https://aws.amazon.com/blogs/machine-learning/build-amazon-sagemaker-notebooks-backed-by-spark-in-amazon-emr/  
  
I launched a notebook with sparkmagic (pyspark3) and tried to call the Spark context but got the following error:  
"""  
The code failed because of a fatal error:  
	Invalid status code '400' from http://xxx.xx.xx.xx:8998/sessions with error payload: "Invalid kind: pyspark3 (through reference chain: org.apache.livy.server.interactive.CreateInteractiveRequest\["kind"])".  
  
Some things to try:  
a) Make sure Spark has enough available resources for Jupyter to create a Spark context.  
b) Contact your Jupyter administrator to make sure the Spark magics library is configured correctly.  
c) Restart the kernel.  
"""  
  
Anyone encountered the same issue?

AWS-User-2543952 · Answer

Hey,  
  
Thanks for using SageMaker! This is an issue in pyspark3 with latest Livy. Starting with version 0.5.0-incubating, session kind “pyspark3” is removed, instead users require to set PYSPARK_PYTHON to python3 executable\[1].   
  
So there're two options:  
1) You can switch to use pyspark kernel instead of pyspark3.  
2) You can set PYSPARK_PYTHON variable in EMR's config file for spark: spark-env.sh  
\[  
    {  
        "Classification": "spark-env",   
        "Configurations": \[  
            {  
                "Classification": "export",   
                "Configurations": \[],   
                "Properties": {  
                    "PYSPARK_PYTHON": "/usr/bin/python3"  
                }  
            }  
        ],   
        "Properties": {}  
    }  
]  
  
Let us know if you have any other question.  
  
Thanks,  
Han  
  
\[1]https://livy.incubator.apache.org/docs/latest/rest-api.html#pyspark  
\[2]https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-configure.html

vzaws · Answer

Submitting these EMR configuration options at the time of cluster creation worked for me:  
\[  
  {  
     "Classification": "spark-env",  
     "Configurations": \[  
       {  
         "Classification": "export",  
         "Properties": {  
            "PYSPARK_PYTHON": "/usr/bin/python3"  
          }  
       }  
    ]  
  },  
  {  
   "Classification": "yarn-env",  
   "Properties": {},  
   "Configurations": \[  
       {  
         "Classification": "export",  
         "Properties": {  
             "PYSPARK_PYTHON": "/usr/bin/python3",  
         }  
       }  
   ]  
 }  
]

Problem Sagemaker and Spark

Relevant content