How do I troubleshoot notebook kernel issues in Amazon EMR Studio?

Short description

After you create a Workspace in Amazon EMR Studio, you might receive the following errors:

• _xsrf argument is missing from POST.
• No kernel starting error is indicated in the Amazon EMR notebook.
• HTTP 599: Unknown (Workspace is not attached to cluster).

Resolution

_xsrf argument missing from POST

This error occurs when you try to open the notebook in the Amazon EMR Studio Workspace and the Jupyter notebook is idle for too long. This error is an open source Jupyter notebook bug and isn't specific to Amazon EMR Studio, the Workspace, or Jupyter. For more information, see Remote Kernels: "'_xsrf' argument missing from POST" when token is empty string on the GitHub website.

To resolve this issue, complete the following steps:

1. Copy the URL into a new tab, and then replace lab with tree.
2. Refresh the Jupyter Home page (tree), and then refresh the page.
3. Navigate to the old notebook and save it.

No kernel starting error is indicated in the Amazon EMR notebook

This error occurs when the YarnAvailableMemory cluster is low and the Jupyter notebook services don't start.

To resolve this issue, run the df -lh command on the primary node in the instance state logs and disk space utilization. Also, use Amazon CloudWatch metrics to check the YarnAvailableMemory. Then, check the status of the services. If the services aren't running, then run the following commands to restart the services:

sudo systemctl stop emr-notebook-agent.service

sudo systemclt start emr-notebook-agent.service

sudo systemctl stop jupyter_enterprise_gateway.service

sudo systemclt start jupyter_enterprise_gateway.service

HTTP 599: Unknown (Workspace is not attached to cluster)

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

This error occurs in the Amazon EMR notebook that's attached to the Amazon EMR Amazon Elastic Kubernetes Service (Amazon EKS) cluster. You receive this error when there are connectivity issues on the managed endpoint. To resolve this issue, the status of the managed endpoint must show ACTIVE.

To check the status of the managed endpoint, run the describe-managed-endpoint command:

aws emr-containers describe-managed-endpoint --id example-endpoint-id --virtual-cluster-id $example-emr-eks-cluster-id

Note: Replace example-endpoint-id with the managed endpoint ID and example-emr-eks-cluster-id with the Amazon EMR on Amazon EKS cluster ID.

Example output:

{ 
 "id": "a403sutr6ltm1", "name": "entergy-bdi-ds-cpu-managed-endpoint-2", 
 "arn": "arn:aws:emr-containers:us-east-1:308359645995:/virtualclusters/hzxh8ezpwzpr1vvdy5f439xij/endpoints/a403sutr6ltm1", 
 "virtualClusterId": "hzxh8ezpwzpr1vvdy5f439xij", 
 "type": "JUPYTER_ENTERPRISE_GATEWAY", 
 "state": "ACTIVE", 
 "releaseLabel": "emr-6.15.0-latest", 
 "executionRoleArn": "arn:aws:iam::308359645995:role/data-scientist-app-1-emr-role" }

Related information

Creating an interactive endpoint for your virtual cluster