Commit f82de7ab authored by Éamonn Murray's avatar Éamonn Murray

Add details on using remote servers

parent f0325b11
......@@ -171,8 +171,13 @@
"source": [
"### Installing sci-kit learn\n",
"\n",
"The first thing you'll need to do is make sure you have sci-kit learn installed. You can do this in several ways, with some examples given at its [documentation page](http://scikit-learn.org/stable/install.html). If you're using your own ubuntu install or virtual machine, `sudo pip3 install --upgrade scikit-learn` in a terminal should work. Make sure `python3 -c \"import sklearn, numpy, scipy, matplotlib\"` in a terminal doesn't give any errors before proceeding.\n",
"\n",
"The first thing you'll need to do is make sure you have sci-kit learn installed. You can do this in several ways, with some examples given at its [documentation page](http://scikit-learn.org/stable/install.html). If you're using your own ubuntu install or virtual machine, `sudo pip3 install --upgrade scikit-learn` in a terminal should work. Make sure `python3 -c \"import sklearn, numpy, scipy, matplotlib\"` in a terminal doesn't give any errors before proceeding."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Virtual environments\n",
"\n",
"If you're on a system where you don't have root/sudo access and want to install a particular package, such as sci-kit learn, or if you want to create an environment with some specific versions of some python libraries, virtual environments offer a very convenient way to do this. The capability is built into python from version 3.5.\n",
......@@ -185,6 +190,27 @@
"- To deactivate your environment, you can type `deactivate`. Your prompt will return to normal."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Using remote servers\n",
"\n",
"Another useful thing to be able to do when training and testing machine learning models (or any other type of analysis too heavy to perform on your laptop/desktop) is interact with a jupyter server running somewhere else, such as from a cloud provider. This is commonly done for neural networks and other deep learning techniques which are much faster with a powerful GPU, which is not usually found in a laptop or consumer desktop.\n",
"\n",
"Because of jupyters split between the server and frontend notebook running in your browser, you can do this efficiently using any machine you have ssh access to. You do this by starting the jupyter server on the remote machine, and then starting an ssh tunnel to redirect the port used by the server over an ssh connection and make it available on a port on your local computer. So the steps are as follows:\n",
"\n",
"1. *On the remote machine* (e.g. in an ssh session) start the jupyter notebook as e.g. `jupyter notebook --no-browser --port=8889` where we pass extra options to tell jupyter not to try to open the browser, and specify the port to use.\n",
"2. *On your local machine* start an ssh tunnel with e.g. `ssh -N -f -L localhost:8888:localhost:8889 remote_user@remote_host` where `-N` tells ssh that you're not running any remote command, `-f` puts ssh into the background after it starts, and `-L` gives the ports to map (remote host port 8889 to local host port 8888).\n",
"3. In your browser *on your local machine* either type `localhost:8888` in the address bar and copy and paste the token listed from the jupyter server you started in 1. or copy the whole link, but replace 8889 with 8888.\n",
"\n",
"Once you're done with your calculations, you'll need to find and kill the ssh process manually. This could be done with e.g. `pkill -15 -f localhost:8889`. Or if you omit the `-f` option to ssh when you start the tunnel, it will remain in the foreground and you can use `ctrl+c` to close the ssh process.\n",
"\n",
"If you're running on a cloud service, don't forget to also kill the jupyter server, log out from the ssh session you started it in and stop your remote server in whatever way is appropriate to the service you're using.\n",
"\n",
"> If you have access to Imperial's Research Computing services, they have a web portal at https://jupyter.rcs.imperial.ac.uk:8000/hub/login that will automatically spawn a jupyter notebook with a server running on their hardware, vastly simplifying this whole process."
]
},
{
"cell_type": "markdown",
"metadata": {},
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment