How to Customize JupyterLab with conda Environments

Note

If you have a conda environment of your own set up and ready, skip to step 6: “Add your environment to the list of jupyter kernels available to jupyterlab and Open OnDemand.”.

  1. Open a DeltaAI terminal (direct SSH).

  2. Load python/anaconda3_pytorch/<version>. (Pick which version you want to use as your base installation, the example uses python/anaconda3_pytorch/jax.)

    arnoldg@gh001:~> module load python/miniforge3_pytorch/jax
    arnoldg@gh001:~> which python
    /sw/user/python/miniforge3-pytorch-jax-0.4.29/bin/python
    
  3. Initialize your default login shell to use conda environments.

    arnoldg@gh001:~> conda init
    ...
    modified      /u/arnoldg/.bashrc
    
    ==> For changes to take effect, close and re-open your current shell. <==
    
    arnoldg@gh001:~> bash
    (base) arnoldg@gh001:~>
    
  4. Create your custom environment by making an empty environment or cloning your chosen environment or module (10 to 30 minutes). A few things to consider:

    • If you omit --clone, your new environment will be mostly empty except for conda commands.

    • You can customize the environment using conda and/or pip to install software (be sure to install jupyter into the environment if not cloning).

    • A cloned environment will contain everything from base, so it can take up to 30 minutes to deploy.

    • Large conda installations may exceed your home directory size. This can be avoided by relocating your .conda directory to your project space, which has a larger quota than your home directory.

    The example environment name is myjax.

    (base) arnoldg@gh001:~>  conda create --prefix /u/arnoldg/myjax --clone=base
    
    Collecting package metadata (current_repodata.json): done
    Solving environment: done
    
    ## Package Plan ##
    
        environment location: /u/arnoldg/myjax
    
        Proceed ([y]/n)? y
    
        Preparing transaction: done
        Verifying transaction: done
        Executing transaction: done
          #
          # To activate this environment, use
          #
          #     $ conda activate /u/arnoldg/myjax
          #
          # To deactivate an active environment, use
          #
          #     $ conda deactivate
    
  5. Activate the new environment.

    (base) arnoldg@gh001:~> conda activate /u/arnoldg/myjax
    (/u/arnoldg/myjax) arnoldg@gh001:~>
    
  6. Add your environment to the list of jupyter kernels available to jupyterlab and Open OnDemand.

    The purpose of this step is to make sure your new environment kernel is added to $HOME/.local, as shown. It will then appear along with the default Lmod modules that are loaded when Open OnDemand starts jupyterlab. Your $PATH should contain a jupyter-notebook by this step.

    (/u/arnoldg/myjax) arnoldg@gh002:~> which jupyter-notebook
    /u/arnoldg/myjax/bin/jupyter-notebook
    
    (/u/arnoldg/myjax) arnoldg@gh001:~> python -m ipykernel install --user --name myjax --display-name myjax
    Installed kernelspec myjax in /u/arnoldg/.local/share/jupyter/kernels/myjax
    
    (/u/arnoldg/myjax) arnoldg@gh001:~> jupyter kernelspec list
    Available kernels:
    python3    /u/arnoldg/myjax/share/jupyter/kernels/python3
    myjax      /u/arnoldg/.local/share/jupyter/kernels/myjax
    
  7. OPTIONAL

    Note

    If you want to keep the environment modifications with your default login environment, skip this step.

    Restore your default environment with the following commands. You may repeat the steps above to add additional environment kernels to jupyterlab.

    (/u/arnoldg/myjax) arnoldg@gh001:~> conda init --reverse
    (/u/arnoldg/myjax) arnoldg@gh001:~> conda deactivate
    (base) arnoldg@gh001:~> conda deactivate
    arnoldg@gh001:~> module reset
    Running "module reset". Resetting modules to system default. The following $MODULEPATH directories have been removed: None
    arnoldg@gh001:~>
    
    # note that the default anaconda used by jupyterlab now contains your new kernel
    arnoldg@gh001:~> module load python/anaconda3_pytorch
    arnoldg@gh001:~> jupyter kernelspec list
    Available kernels:
    myjax      /u/arnoldg/.local/share/jupyter/kernels/myjax
    python3    /sw/user/python/miniforge3-pytorch-2.5.0/share/jupyter/kernels/python3
    
  8. Start an Open OnDemand Jupyter session.

  9. In JupyterLab, you can hover over items in the Launcher to see which environment will be used; select the one you want for this session.

    JupyterLab Launcher window with multiple environments to choose from.

References