Zeppelin Notebook on EMR - how to do pip install

Question

I am trying to install happybase package on Zeppelin notebook ( or for that matter any package ) . How do I do a pip install  from a Zeppelin cell . 
%pip or !pip is not recognized

Answer

Hello,

You can follow the below steps in zeppelin to install the packages at runtime. This method works in client deploy mode.

1. Provide access to home directory for other user(zeppelin) using following command using a Bootstrap script so that zeppelin service can install the packages on /home/.local directory.

```
sudo chmod 757 /home
```

2. Add below settings in spark interpreter from Zeppelin UI and restart the interpreter.

```
spark.pyspark.virtualenv.enabled	true	
spark.pyspark.virtualenv.bin.path	/usr/bin/virtualenv	
spark.pyspark.virtualenv.type           native
spark.pyspark.python                    python3
```

3. Now try installing the packages using below command from notebook,

```
%spark.pyspark
sc.install_pypi_package("xgboost")
```

Answer

Hi,

Have a look at https://medium.com/@techboomph/getting-zeppelin-to-work-with-emr-93e237ac446a

The author proposes a solution to do the pip install for a Zepplin notebook on EMR that you need.

Didier

Zeppelin Notebook on EMR - how to do pip install

Relevant content