How to shut down zeppelin notebook and restart later

Question

Hi, I followed the guide here to create a zeppelin notebook: https://docs.aws.amazon.com/glue/latest/dg/dev-endpoint-notebook-server-considerations.html  
  
I am just wondering how can I shut a zeppelin notebook down so that I can start it up again at a later date without going through all the hassle of setting it up from scratch.  
  
I tried just stopping the EC2 instance and starting it up again later. I restarted the zeppelin daemon and logged into zeppelin notebook ok. However when it came to executing pyspark statements, I received a java connection error:  
  
####################  
java.net.ConnectException: Connection refused (Connection refused)  
	at java.net.PlainSocketImpl.socketConnect(Native Method)  
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)  
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)  
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)  
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)  
	at java.net.Socket.connect(Socket.java:589)  
	at org.apache.thrift.transport.TSocket.open(TSocket.java:182)  
	at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)  
	at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)  
	at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)  
	at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)  
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)  
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)  
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:62)  
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:133)  
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:165)  
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:132)  
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:299)  
	at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:407)  
	at org.apache.zeppelin.scheduler.Job.run(Job.java:188)  
	at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)  
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)  
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)  
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)  
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  
	at java.lang.Thread.run(Thread.java:748)  
####################  
  
I could not resolve this without having to create a notebook from scratch, which is a pain. This is the second time I had to go through the process.  
  
Thanks for any help.  
  
Edited by: freesolo on Dec 12, 2019 8:31 PM  
  
Edited by: freesolo on Dec 12, 2019 8:32 PM

Accepted Answer

In order to establish connection from Zeppelin EC2 instance to Glue dev endpoint, SSH tunnel is required.  
This SSH tunnel and Zeppelin daemons are started within the "setup_notebook_server.py" script.  
If you restart the EC2 instance, we would recommend you to rerun the script. Interactive shell asks a few questions but you can answer with just 'n' key.  
  
I understand that you do not want to rerun the script. There is another option to use SageMaker notebook instead. It does not require running scripts manually.

Answer

Thanks for the helpful replies. Rerunning the "setup_notebook_server.py" after starting up the EC2 instance was the simple solution that works! Thanks NoritakaS-AWS!  
  
Edited by: freesolo on Jan 6, 2020 6:09 PM

Answer

Hi,  
  
I'm working on local zeppelin connected to an Dev EndPoint and sometimes when I set up the end point and start using zeppelin I get the same error as you. After some investigation I've seen that nothing is listening on port 9007, the solution is to create a new DevEndPoint and then almost every time it works.  
  
Furthermore, I've got the same error when I execute a job from a workflow but this doesn't happen when I execute the job directly. The problem has appeared only three weeks before now.

How to shut down zeppelin notebook and restart later

Relevant content