How to choose which Spark kernel to use in SageMaker Studio?

Question

Available Amazon SageMaker Kernels include [the following two Spark kernels](https://docs.aws.amazon.com/sagemaker/latest/dg/notebooks-available-kernels.html):

- PySpark (SparkMagic) with Python 3.7
- Spark (SparkMagic) with Python 3.7
- Spark Analytics 1.0
- Spark Analytics 2.0

And at re:Invent 2022 there was [an announcement](https://aws.amazon.com/about-aws/whats-new/2022/09/sagemaker-studio-supports-glue-interactive-sessions/) that "SageMaker Studio now supports Glue Interactive Sessions." "The built-in Glue PySpark or Glue Spark kernel for your Studio notebook to initialize interactive, serverless Spark sessions."

It seems like the benefits of using one of the Glue Spark kernels are that you can "quickly browse the Glue data catalog, run large queries, and interactively analyze and prepare data using Spark, right in your Studio notebook." But can't you already do all that with the existing two SageMaker kernels?

In other words, how do you choose whether to use one of the existing two SparkMagic kernels in SageMaker Studio notebooks or to use this new Glue Interactive Sessions feature?

Answer

The difference is that with SparkMagic you would need to provide a Spark cluster and link to it using SparkMagic configuration.  
With Glue Interactive Sessions all that time consuming work is taken case for you, you can easily create and destroy Spark clusters as you need.

How to choose which Spark kernel to use in SageMaker Studio?

Contenido relevante