EMR Studio with EMR on EKS

1

Hi,

I have created a workspace in EMR studio using an endpoint on EMR on EKS as per below (based on the instructions here https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-studio-create-eks-cluster.html) however after attaching the EMR on EKS cluster to the workspace i don't see or can choose any kernels. Also the icons to create a python/pyspark/spark/sparkr notebook

aws emr-containers create-managed-endpoint
--type JUPYTER_ENTERPRISE_GATEWAY
--virtual-cluster-id $VIRTUAL_CLUSTER_ID
--name testme
--execution-role-arn $EMR_ROLE_ARN
--release-label emr-6.6.0-latest
--certificate-arn arn:aws:acm:eu-central-1:<accoubnt>:certificate/e6dcd6e5-3aac-4247-bc26-1d1236b06861
--configuration-overrides '{ "applicationConfiguration": [ { "classification": "spark-defaults", "properties": { "spark.hadoop.hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory", "spark.sql.catalogImplementation": "hive" } } ] }'

  • I have a similar issue where I get a pop up "error starting kernel. HTTP 403: Forbidden (workspace is not attached to cluster. Click Ok to continue" even though on the left side bar it seems to be attached. This is specifically EMR studio on EC2 for me

asked 3 months ago66 views
1 Answer
0

Hi,

Thanks for writing to re:Post.

Can you please confirm your EMR version ?

Spark pod template functionality with managed endpoints is not working in EMR on EKS release 6.4, 6.5, and 6.6. 1

Generally such issues can also occur due to network setup so please ensure that the security groups are setup correctly and please ensure to use latest EMR version (6.7.0)

Sujay_M
answered 3 months ago
  • Unfortunately used quite a few versions, including 6.7.0 and all of them give me the same error.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions