EMR Studio with EMR on EKS

1

Hi,

I have created a workspace in EMR studio using an endpoint on EMR on EKS as per below (based on the instructions here https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-studio-create-eks-cluster.html) however after attaching the EMR on EKS cluster to the workspace i don't see or can choose any kernels. Also the icons to create a python/pyspark/spark/sparkr notebook

aws emr-containers create-managed-endpoint
--type JUPYTER_ENTERPRISE_GATEWAY
--virtual-cluster-id $VIRTUAL_CLUSTER_ID
--name testme
--execution-role-arn $EMR_ROLE_ARN
--release-label emr-6.6.0-latest
--certificate-arn arn:aws:acm:eu-central-1:<accoubnt>:certificate/e6dcd6e5-3aac-4247-bc26-1d1236b06861
--configuration-overrides '{ "applicationConfiguration": [ { "classification": "spark-defaults", "properties": { "spark.hadoop.hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory", "spark.sql.catalogImplementation": "hive" } } ] }'

  • I have a similar issue where I get a pop up "error starting kernel. HTTP 403: Forbidden (workspace is not attached to cluster. Click Ok to continue" even though on the left side bar it seems to be attached. This is specifically EMR studio on EC2 for me

질문됨 2년 전601회 조회
1개 답변
0

Hi,

Thanks for writing to re:Post.

Can you please confirm your EMR version ?

Spark pod template functionality with managed endpoints is not working in EMR on EKS release 6.4, 6.5, and 6.6. 1

Generally such issues can also occur due to network setup so please ensure that the security groups are setup correctly and please ensure to use latest EMR version (6.7.0)

AWS
Sujay_M
답변함 2년 전
  • Unfortunately used quite a few versions, including 6.7.0 and all of them give me the same error.

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인