EMR Studio with EMR on EKS

Question

Hi,

I have created a workspace in EMR studio using an endpoint on EMR on EKS as per below (based on the instructions here https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-studio-create-eks-cluster.html)  however after attaching the EMR on EKS cluster to the workspace i don't see or can choose any kernels. Also the icons to create a python/pyspark/spark/sparkr notebook

aws emr-containers create-managed-endpoint \
--type JUPYTER_ENTERPRISE_GATEWAY \
--virtual-cluster-id $VIRTUAL_CLUSTER_ID \
--name testme \
--execution-role-arn $EMR_ROLE_ARN \
--release-label emr-6.6.0-latest \
--certificate-arn arn:aws:acm:eu-central-1::certificate/e6dcd6e5-3aac-4247-bc26-1d1236b06861 \
--configuration-overrides '{ 
    "applicationConfiguration": [ 
      { 
        "classification": "spark-defaults",
        "properties": {
          "spark.hadoop.hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
          "spark.sql.catalogImplementation": "hive"
        }
      }
    ]
  }'

Answer

Hi,

Thanks for writing to re:Post.

Can you please confirm your EMR version ?

Spark pod template functionality with managed endpoints is not working in EMR on EKS release 6.4, 6.5, and 6.6. [1]

Generally such issues can also occur due to network setup so please ensure that the security groups are setup correctly and  please ensure to use latest EMR version  (6.7.0)

[1]: https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/emr-eks-6.6.0.html

EMR Studio with EMR on EKS

Relevanter Inhalt