How do I mount a workspace directory or resolve the "Unable to mount the workspace" error in Amazon EMR?

Resolution

Mount a workspace directory

To mount a workspace directory to a cluster, use the %mount_workspace_dir magic command in Amazon EMR. When you use this command, you can import and run code from other files in the workspace when you mount your workspace directory.

Resolve the error Unable to mount the workspace

You might receive the following errors when you use the %mount_workspace_dir magic command to mount a workspace:

"UsageError: Unable to mount the Workspace. stdout= stderr=fuse: unknown option compat_dir"

"UsageError: S3-fuse is not installed"

To resolve the preceding errors, use one of the following methods:

Install s3fs-fuse

If you use Amazon EMR release versions 5.3.x or 6.x, then complete the following steps:

1. Set up the Amazon EMR cluster, workspace, and Spark magic with your Spark, Livy, and Jupyter Enterprise Gateway applications. For more information, see Enhance kernels with magic commands.
2. Use SSH to connect to the Amazon EMR cluster node.
3. Install s3fs-fuse:

   sudo amazon-linux-extras install epel -y
   sudo yum install s3fs-fuse -y

If you use Amazon EMR release versions 7.x, then complete the following steps:

1. To install s3fs-fuse, see s3fs-fuse from the GitHub website. Or, run the following command:

   sudo yum install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel -y
   
   git clone https://github.com/s3fs-fuse/s3fs-fuse.git
   cd s3fs-fuse
   ./autogen.sh
   ./configure --prefix=/usr --with-openssl
   make
   sudo make install

2. Install emr-notebooks-magics:
   Note: Before you install Amazon EMR Notebooks magic, see Enhance kernels with magic commands.

   %pip install emr-notebooks-magics --upgrade 

3. Mount the workspace directory in the notebook:

   %mount_workspace_dir.

   Note: For Amazon EMR releases 6.0 through 6.9.0 and 5.0 through 5.36.0, only emr-notebooks-magics package versions 0.2.0 and later support the %mount_workspace_dir magic command.

4. Confirm that your workspace directory mounted successfully:

   %%sh
   ls

Install the s3fs package

To install the s3fs package, run the following command:

wget https://rpmfind.net/linux/epel/8/Everything/aarch64/Packages/s/s3fs-fuse-1.93-1.el8.aarch64.rpm -P /home/hadoop/;sudo yum localinstall /home/hadoop/s3fs-fuse-1.93-1.el8.aarch64.rpm -y;rm /home/hadoop/s3fs-fuse-1.93-1.el8.aarch64.rpm  

You receive the following output:

--2024-05-12 15:49:13-- https://rpmfind.net/linux/epel/8/Everything/aarch64/Packages/s/s3fs-fuse-1.93-1.el8.aarch64.rpm
Resolving rpmfind.net (rpmfind.net)... 195.220.108.108
Connecting to rpmfind.net (rpmfind.net)|195.220.108.108|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 292036 (285K) [application/x-rpm]
Saving to: "/home/hadoop/s3fs-fuse-1.93-1.el8.aarch64.rpm"

The preceding command downloads the s3fs-fuse package version from the EPEL repository, installs the downloaded version, and then cleans up the downloaded file. For more information, see RPM resource s3fs-fuse(aarch-64) on the RPM find website. Also, see How do I install my local rpm files using yum? on the Red Hat website.

Note: If you still experience issues after you complete the previous steps, then change your instance type to an instance that's built on x86_64 architecture. Or, install a newer version of the s3fs-fuse package.