Skip to content

Jupyter Notebook

Jupyter-Notebook is available on the cluster. You can run notebooks on the cluster but visualization of results and interaction through the web interface will take place on your local computer.

Currently jupyter-notebook is on a test phase, some things may need to be improved and may change in the near future.

I will happily accept input//feedbacks//remarks and so on.

Prepare your Jupyter environment#

  1. Connect to maestro as usual

Code Block (text)

ssh login@maestro.pasteur.fr
  1. Choose the python version you want to work with in your notebook

Code Block (text)

login@maestro-submit ~ $ module av python
--------------------------------------------------------------- /opt/gensoft/devmodules ----------------------------------------------------------------------------
Python/2.7.17  Python/3.8.1  Python/3.8.3  Python/3.8.18  Python/3.9.16  Python/3.9.18  Python/3.10.7  Python/3.10.13  Python/3.11.0  Python/3.11.5  
login@maestro-submit ~ $ module load Python/3.11.5
  1. Create a virtual environment in which you will be able to install everything you will need

Code Block (text)

login@maestro-submit ~ $ python3 -m venv venvs/myjupyter3.11env
login@maestro-submit ~ $ source venvs/myjupyter3.11env/bin/activate
(myjupyter3.11env) login@maestro-submit ~ $
  1. Then install the python kernel for Jupyter notebooks in your virtual env and make the python version you chose available in Jupyter

Code Block (text)

(myjupyter3.11env) login@maestro-submit ~ $  pip install ipykernel
[…]
Installing collected packages: wcwidth, pure-eval, ptyprocess, traitlets, tornado, six, pyzmq, pygments, psutil, prompt-toolkit, platformdirs, pexpect, parso, packaging, nest-asyncio, executing, decorator, debugpy, python-dateutil, matplotlib-inline, jupyter-core, jedi, comm, asttokens, stack-data, jupyter-client, ipython, ipykernel
Successfully installed asttokens-2.4.1 comm-0.1.4 debugpy-1.8.0 decorator-5.1.1 executing-2.0.1 ipykernel-6.26.0 ipython-8.17.2 jedi-0.19.1 jupyter-client-8.5.0 jupyter-core-5.5.0 matplotlib-inline-0.1.6 nest-asyncio-1.5.8 packaging-23.2 parso-0.8.3 pexpect-4.8.0 platformdirs-3.11.0 prompt-toolkit-3.0.39 psutil-5.9.6 ptyprocess-0.7.0 pure-eval-0.2.2 pygments-2.16.1 python-dateutil-2.8.2 pyzmq-25.1.1 six-1.16.0 stack-data-0.6.3 tornado-6.3.3 traitlets-5.13.0 wcwidth-0.2.9
(myjupyter3.11env) login@maestro-submit ~ $ 
(myjupyter3.11env) login@maestro-submit ~ $  python -m ipykernel install --name my-python-3.11.5 --display-name "Python-3.11.5" --user
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
Installed kernelspec my-python-3.11.5 in /pasteur/appa/homes/<your login>/.local/share/jupyter/kernels/my-python-3.11.5

The last line shows you where the kernel for the chosen python version has been installed. 5. Then install in your virtual environment any other package required by your notebook

Code Block (text)

(myjupyter3.11env) login@maestro-submit ~ $ pip install pandas
Collecting pandas
[…]
Successfully installed numpy-1.26.1 pandas-2.1.2 pytz-2023.3.post1 tzdata-2023.3
  1. Once done, you can leave the virtual environment

Code Block (text)

(myjupyter3.11env) login@maestro-submit ~ $ deactivate
login@maestro-submit ~ $

Launch Jupyter#

  1. load the module providing jupyter-notebook

Code Block (text)

login@maestro-submit ~ $  module load Jupyter-Notebook

NB you may want to check the Jupyter-Notebook available version using the usual command module av Jupyter-Notebook and adapt accordingly. 2. once the modules are loaded, run the notebook with the following mandatory options

Code Block (text)

login@maestro-submit ~ $ srun jupyter-notebook --no-browser --ip='0.0.0.0'

of course you can dispatch the execution on a specific QoS, partition, etc ... as usual using the corresponding srun options. 3. after a short time, you will see the following output

Code Block (text)

login@maestro-submit ~ $ srun -p dedicated --qos=fast jupyter-notebook --no-browser --ip='0.0.0.0'  [I 11:02:19.284 NotebookApp] 

Writing notebook server cookie secret to /pasteur/appa/homes/<your login>/.local/share/jupyter/runtime/notebook_cookie_secret
[I 11:02:21.152 NotebookApp] Serving notebooks from local directory: /pasteur/appa/homes/<your login>
[I 11:02:21.152 NotebookApp] Jupyter Notebook 6.4.3 is running at:
[I 11:02:21.153 NotebookApp] http://maestro-2023.maestro.pasteur.fr:8888/?token=b128ab13b00d9ae3cd9080a64c28f2e27e055cc929a52d43
[I 11:02:21.153 NotebookApp]  or http://192.168.154.25:8888/?token=b128ab13b00d9ae3cd9080a64c28f2e27e055cc929a52d43
[I 11:02:21.153 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 11:02:21.163 NotebookApp] 

    To access the notebook, open this file in a browser:
        file:///pasteur/appa/homes/<your login>/.local/share/jupyter/runtime/nbserver-2954421-open.html
    Or copy and paste one of these URLs:
        http://maestro-2023.maestro.pasteur.fr:8888/?token=b128ab13b00d9ae3cd9080a64c28f2e27e055cc929a52d43
     or http://192.168.154.25:8888/?token=b128ab13b00d9ae3cd9080a64c28f2e27e055cc929a52d43

keep that terminal open to let your jupyter-notebook run on the maestro node. 4. from your desktop computer, launch your browser and copy/paste one of the URLs given in the instructions above. Then hit the Enter key.

Code Block (text)

http://maestro-2023.maestro.pasteur.fr:8888/?token=b128ab13b00d9ae3cd9080a64c28f2e27e055cc929a52d43

There you go! You now have jupyter-notebook displaying its interface on your desktop computer.

You can then move inside the directories tree in your home on Maestro and import a notebook or create a new one. If you click on New , you can see that you can create a notebook in the Python version installed in your virtual environment earlier.

If you click on New , a new tab opens with the usual Jupyter Notebook interface. The packages installed in your virtual environment are also available as expected 5. As you can see in your Terminal, the notebook is saved in the place you chose in your tree

Code Block (text)

[I 11:40:44.492 NotebookApp] Saving file at /mynotebooks/Untitled.ipynb

You can also see it in the first tab of the browser.

Once you have finished your notebook session, don't forget to stop the jupyter-notebook instance running on the node (you will need to issue Ctrl-C twice in a row to kill the srun)

Articles connexes#

Les articles connexes apparaissent ici en fonction des étiquettes que vous avez sélectionnées. Cliquez pour modifier la macro et ajouter ou modifier des étiquettes.

false5FAQAfalsemodifiedtruepagelabel = "jupyter-notebook" and type = "page" and space = "FAQA"jupyter-notebook

true
Demandes lié