I am trying to control a Spark cluster (using SparkR) from a Sagemaker notebook. I followed these instructions closely: https://aws.amazon.com/blogs/machine-learning/build-amazon-sagemaker-notebooks-backed-by-spark-in-amazon-emr/ and got it to work.
Today when I try to run the SageMaker notebook (using the exact same code as before) I inexplicably get this error:
An error was encountered:
[1] "Error in callJMethod(sparkSession, \"read\"): Invalid jobj 1. If SparkR was restarted, Spark operations need to be re-executed."
Does anyone know why this is? I terminated the SparkR kernel and am still getting this error.