Problem Sagemaker and Spark

Question

Hi there,  
  
I followed this tutorial to set up Sagemaker Notebook with Spark (EMR): https://aws.amazon.com/blogs/machine-learning/build-amazon-sagemaker-notebooks-backed-by-spark-in-amazon-emr/  
  
I launched a notebook with sparkmagic (pyspark3) and tried to call the Spark context but got the following error:  
"""  
The code failed because of a fatal error:  
	Invalid status code '400' from http://xxx.xx.xx.xx:8998/sessions with error payload: "Invalid kind: pyspark3 (through reference chain: org.apache.livy.server.interactive.CreateInteractiveRequest\["kind"])".  
  
Some things to try:  
a) Make sure Spark has enough available resources for Jupyter to create a Spark context.  
b) Contact your Jupyter administrator to make sure the Spark magics library is configured correctly.  
c) Restart the kernel.  
"""  
  
Anyone encountered the same issue?

Answer

Hey,  
  
Thanks for using SageMaker! This is an issue in pyspark3 with latest Livy. Starting with version 0.5.0-incubating, session kind “pyspark3” is removed, instead users require to set PYSPARK_PYTHON to python3 executable\[1].   
  
So there're two options:  
1) You can switch to use pyspark kernel instead of pyspark3.  
2) You can set PYSPARK_PYTHON variable in EMR's config file for spark: spark-env.sh  
\[  
    {  
        "Classification": "spark-env",   
        "Configurations": \[  
            {  
                "Classification": "export",   
                "Configurations": \[],   
                "Properties": {  
                    "PYSPARK_PYTHON": "/usr/bin/python3"  
                }  
            }  
        ],   
        "Properties": {}  
    }  
]  
  
Let us know if you have any other question.  
  
Thanks,  
Han  
  
\[1]https://livy.incubator.apache.org/docs/latest/rest-api.html#pyspark  
\[2]https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-configure.html

Answer

Submitting these EMR configuration options at the time of cluster creation worked for me:  
\[  
  {  
     "Classification": "spark-env",  
     "Configurations": \[  
       {  
         "Classification": "export",  
         "Properties": {  
            "PYSPARK_PYTHON": "/usr/bin/python3"  
          }  
       }  
    ]  
  },  
  {  
   "Classification": "yarn-env",  
   "Properties": {},  
   "Configurations": \[  
       {  
         "Classification": "export",  
         "Properties": {  
             "PYSPARK_PYTHON": "/usr/bin/python3",  
         }  
       }  
   ]  
 }  
]

Problem Sagemaker and Spark

Relevant content