[AWS EMR] HTTP 403: Forbidden (Workspace is not attached to cluster]

Hi everyone,

I am a bit new to AWS so pardon me on this. I am trying to run Spark on a notebook in EMR. I have created a cluster to achieve this effect. However, when I tried to select Pyspark or Spark as the kernel, the aforementioned error pops out.

I have tried creating workspaces in EMR Studio but it doesn't seem to solve the problem.

The next thing I have tried was to try other versions of EMR Release versions. The ones that have succeeded are

1. emr-6.0.0 (JupyterHub 1.0.0, Hive 3.1.2, Hue 4.4.0, Livy 0.6.0, Spark 2.4.4)
2. emr-5.18.0 (Hive 2.3.3, Pig 0.17.0, Hue 4.2.0, JupyterHub 0.8.1, Spark 2.3.2)
3. emr-5.2.3 (Hive 2.1.0, Pig 0.16.0, Hue 3.10.0, Spark 2.0.2)

However, the following failed and gave the error as shown in the picture.

1. emr-6.3.1 (JupyterHub 1.2.0, Hive 3.1.2, Pig 0.17.0, Hue 4.9.0, Spark 3.1.1, JupyterEnterpriseGateway 2.1.0)
2. emr-5.36.0 (Hive 2.3.9, Pig 0.17.0, Hue 4.10.0, JupyterEnterpriseGateway 2.1.0, JupyterHub 1.4.1, Spark 2.4.8)

What could be the issue behind this?

Thanks so much everyone!