1 Answer
- Newest
- Most votes
- Most comments
3
Hello,
I tried EMR 7.0.0 Serverless application with Interactive workload that attached to workspace in Studio Notebook working fine without any issues. Pyspark & spark kernel are enabled as expected.
Please make sure that your EMR Studio user role has appropriate permissions for your selected compute type and interactive endpoint enabled for the application. Also check if custom AMI configured that causing issue ?
Relevant content
- asked 2 years ago
- asked a year ago
AWS OFFICIALUpdated 2 years ago- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated a year ago
I had my EMR Serverless vCPU quota limited to 16 vCPU, so had to limit the EMR Serverless application size to that, and had to set spark.dynamicAllocation.enabled to false. With those constrained limits, the PySpark kernel of EMR 7.0.0 was not working, but the one from EMR 6.15.0 did.
Now I got approved a quota increase and have 512 vCPU available. I increased the resources of the EMR Application and enabled the dynamic allocation, and the PySpark kernel works fine in 7.0.0 .
So I think somehow 7.0.0 needs more resources to work properly? Could you try 7.0.0 without dynamicAllocation and limiting to less than 16 vCPU and see if the PySpark kernel stays in "unknown" state for you too?
@tomtastic, Good to hear that you are able to resolve the issue. I understand that default limit was 16vCPU and that driven slowness/hung in the notebook which makes sense depends on what you have executed with this capacity. Also, it is depends on how much has configured with pre-initialized capacity unless you dont specify it otherwise. You can certainly check the vCPU utilization for the given period of time in the service quota page.