Software
Core Software and Libraries
The Campus Cluster maintains a variety of core software and libraries to support computing activities. Software can be added to a user’s environment using modules.
Campus Cluster core software and libraries list (click to expand/collapse)
Software |
Version |
|---|---|
Anaconda |
Distribution 2024.06 Includes conda 24.5.0 and python 3.12.4 |
Distribution 2024.10 Includes conda 24.9.2 and python 3.12.7 |
|
Apptainer |
1.3.4-1.el9 |
Boost |
1.8.7 |
CMake |
3.26.5 |
Cuda |
12.4 12.6 |
dos2unix |
7.4.2 |
Emacs |
27.2 |
FFTW |
3.3.10 |
GCC |
11.4.1 (default) 12.4.0 13.3.0 |
Git |
2.43.5 |
GSL |
2.8 |
Intel compilers |
|
Intel Math Kernel Library (MKL) |
2025.0.0 |
Intel MPI |
|
JDK |
Open JDK 23 JDK Development Kit 23.0.1 |
Mathematica |
TBA |
MATLAB |
R2024a (24.1) |
Miniconda |
12.9.2 |
MVAPICH2 |
2.3.7 |
Nano |
5.6.1 |
OpenBLAS |
TBA |
OpenMPI |
5.0.1 |
PAPI |
7.1.0 |
PETSc |
TBA |
Python |
3.9.18 (OS) 3.10.16 (python/3.10.16) 3.11.11 (python/3.11.11) 3.12.4 (anaconda3/2024.06) 3.12.17 (anaconda3/2024.10) 3.12.17 (miniconda3/24.9.2) |
R |
4.2.3 4.4.1-5 |
SVN |
1.14.1 |
Valgrind |
3.22.0 |
VIM |
8.2 |
If the software you want isn’t listed above, review the investor-specific and Documentation Hub - Install Software/Libraries pages for more information.
Managing Your Environment (Modules)
The module command is a user interface to the Modules package. The Modules package provides for the dynamic modification of the user’s environment via modulefiles. A modulefile contains the information needed to configure the shell for an application. Modules are independent of the user’s shell, so both tcsh and bash users can use the same commands to change the environment.
Command |
Description |
|---|---|
|
lists all available modulefiles |
|
lists currently loaded modulefiles |
|
help on module |
|
display information about |
|
load |
|
remove |
|
unload |
To include particular software in the environment for all new shells, edit your shell configuration file ($HOME/.bashrc for bash users and $HOME/.cshrc for tcsh users) by adding the module commands to load the software that you want to be a part of your environment.
After saving your changes, you can source your shell configuration file or log out and then log back in for the changes to take effect.
Note: Order is important. With each module load, the changes are prepended to your current environment paths.
For additional information on Modules, see the module and modulefile man pages or visit the Modules SourceForge page.
R on the Campus Cluster
Introduction
R is a programming language and software environment for statistical computing and graphics. R and its libraries implement a wide variety of statistical and graphical techniques, including linear and non-linear modelling, classical statistical tests, time-series analysis, classification, clustering, and others. R is easily extensible through functions and extensions, and the R community is noted for its active contributions in terms of packages. R is also an interpreted language in which users typically access it through a command line interpreter.
R and R packages are available via the Comprehensive R Archive Network (CRAN), a collection of sites which carry identical material, consisting of the R distribution(s), the contributed extensions, documentation for R, and binaries.
Versions
See the Core Software and Libraries for the versions of R installed on the Campus Cluster.
Adding R to Your Environment
Each R installation on the Campus Cluster has a modulefile that you can use to load a specific version of R into your user environment.
You can see the available versions of R by typing module avail R on the command line.
The latest version of R available on the Campus Cluster can be loaded into your environment by typing module load R.
To load a specific version, you will need to load the corresponding modulefile. See Managing Your Environment (Modules), for more information about modules.
Installing Add-on Packages (in user specified locations)
Any R add-on package not available in the system installation (see Viewing Installed R Packages for information on viewing the list) can be installed from the CRAN in a user specified location that you have write access to with just a few easy steps.
Generally, when installing any software there are a few things to consider:
Home directory disk usage quotas (see Storage Areas).
Will the software be used by multiple users in my group?
Project space is the recommended location to install software (see Investor-Specific Software Installation).
Does the desired software have dependencies/requirements?
Is the prerequisite software already installed?
Installation Command Syntax
Note
The install.packages() command is run within R. You need to load R (module load R) and run it (R) prior to using install.packages().
R package installations, for the most part, are straight forward. Minimally, all that is needed is the name of the package that is to be installed, but installation location and the repository can also be specified.
Command (“Package Name”, “Location”, “Repository”)
install.packages('package_name', '/path/to/r_libraries', 'Repository URL')Behavior: Installs the package downloaded from the specified repository in the specified location.
Command (“Local Filename”, “Location”, “Repository”)
install.packages('package_name.tar.gz', '/path/to/r_libraries', repos = NULL)Behavior: Installs the local package(file) in the specified location specifying no repository(NULL)
When the installation location and the repository URL are not specified, R packages are installed in a default location and the R installation process prompts to choose from a list of repositories.
R packages downloaded manually from the CRAN can be installed by specifying the local filename and omitting the repository URL (specifying NULL).
Using Rscript
To run R commands without starting an R session the command Rscript can be used (as we show in the examples on this page).
As a scripting front end for R, Rscript enables the use of R via shell scripts and scripting applications.
Installation Steps
Below are steps with examples of the actual commands that can be run on the Campus Cluster.
In these steps ~/Rlibs is used for the location to install user specific add-on packages (The tilde “~” means the user’s home directory—i.e., $HOME).
Note
It is assumed that these steps are being run under the default BASH shell. When using a different shell, the syntax for that shell should be used where appropriate.
Create a directory for your R packages:
mkdir ~/Rlibs
Load the R modulefile:
module load R/4.2.3
Set the R library environment variable (R_LIBS) to include your R package directory:
export R_LIBS=~/Rlibs:${R_LIBS}Use the
install.packagesfunction to install your R package:Rscript -e "install.packages('RCurl', '~/Rlibs', 'https://cran.r-project.org')"
If the environment variable R_LIBS is not set and a directory is not specified with the install.packages command, then R packages will be installed under ~/R/x86_64-unknown-linux-gnu-library by default. This R subdirectory structure is created automatically.
The R_LIBS environment variable will need to be set every time when logging in to the Campus Cluster if the user’s R package location is to be visible to an R session.
The following can be added to a user’s ~/.bashrc file to remove the need to set the R_LIBS environment variable with every login session to the Campus Cluster:
if [ -n $R_LIBS ]; then
export R_LIBS=~/Rlibs:$R_LIBS
else
export R_LIBS=~/Rlibs
fi
Warnings and Error Messages
R packages that are not available in the current CRAN (Comprehensive R Archive Network) or if the name of the package is misspelled, tend to generate a message similar to the following:
[cc-login1 ~]$ Rscript -e "install.packages('phybase','~/Rlibs', 'https://cran.r-project.org')"
Warning message:
package 'phybase' is not available (for R version 3.2.2)
Searching the CRAN site for your desired R package may provide links to archived versions that are not available in the current CRAN. In this case the specific archived R package can be downloaded and installed from the local file using the same command but omitting the repository URL (specifying NULL).
Some R packages have dependencies and require them to be installed first and will generate an error message similar to the following:
[cc-login1 ~]$ Rscript -e "install.packages('phybase_1.1.tar.gz', '~/Rlibs', repos = NULL)"
ERROR: dependency 'ape' is not available for package 'phybase'
* removing '/home/jdoe/Rlibs/phybase'
Warning message:
In install.packages("phybase_1.1.tar.gz", repos = NULL) :
installation of package 'phybase_1.1.tar.gz' had non-zero exit status
Installing the required R package first, then the desired R package resolves this issue.
Viewing Installed R Packages
The library() command can be used to view all user and system installed R packages (user installed packages are only visible to R when the $R_LIBS environment variable is set).
[cc-login1 ~]$ Rscript -e "library()"
Packages in library '/home/jdoe/Rlibs':
R6 Classes with reference semantics
RCurl General network (HTTP/FTP/...) client interface
for R
...
stringr Simple, Consistent Wrappers for Common String
Operations
whisker {{mustache}} for R, logicless templating
Packages in library '/usr/local/R/4.2.2/lib64/R/library':
KernSmooth Functions for kernel smoothing for Wand & Jones
(1995)
MASS Support Functions and Datasets for Venables and
Ripley's MASS
...
tools Tools for Package Development
utils The R Utils Package
Sample R Job Batch Script
A sample R job batch script is available at /sw/cc.users/slurm/r.sbatch.
Python on the Campus Cluster
Introduction
Python is an interpreted, high-level, general-purpose programming language.
Python and Python packages are available via the The Python Package Index (PyPI), which hosts thousands of third-party modules for Python. Both Python’s standard library and the community-contributed modules allow for endless possibilities.
Anaconda also provides a Python environment with python packages.
Versions
See the Core Software and Libraries for the versions of Python and Anaconda installed on the Campus Cluster.
Adding Python to Your Environment
Each Python installation on the Campus Cluster has a corresponding modulefile that can be used to load a specific version of Python into your user environment.
You can see the available versions of Python by typing module avail python and/or module avail anaconda on the command line.
See Managing Your Environment (Modules) for more information about modules.
Installing Python Packages (in user specified locations)
Generally, any Python package not available in the system installation (see Viewing Installed Python Packages for information on viewing the list) can be installed from the Python Package Index (PyPI) in a user specified location that you have write access to with just a few easy steps.
Generally, when installing any software there are a few things to consider:
Home directory disk usage quotas (see Storage Areas).
Will the software be used by multiple users in my group?
Project space is the recommended location to install software (see Investor-Specific Software Installation).
Does the desired software have dependencies/requirements?
Is the prerequisite software already installed?
Campus Cluster users must install software/libraries into user write-able locations like their home directory, their group’s project space or their scratch space.
Note
The example instructions below use scratch space. Please note that software installed in scratch space is not permanent and may be removed at any time. We recommend that you install in your group’s project space. For a persistent installation, users should replace the path to their scratch space with the path to their project space in the instructions below`
Warning
You should use one of the installation methods listed below (Python or Anaconda). Combining installation methods can cause unexpected behavior/errors.
Installation Command Syntax
Note
The following use ${HOME}, you can also use /u/${USER} instead.
Under Python:
To Install a specific python package into a user location, use the pip3 install command with the -t option.
module load python/3
mkdir -p ${HOME}/scratch/mypython3
pip3 install -t ${HOME}/scratch/mypython3 phonopy
export PYTHONPATH=${HOME}/scratch/mypython3:${PYTHONPATH}
Users will have to run the above “export…” command after loading python into their user environment with the module command on future login sessions.
This allows the Campus Cluster’s python installation to see the user’s specific python packages.
Alternatively, users can add two commands to their ${HOME}/.bashrc file so that Python is setup for their user environment every time they login to the Campus Cluster.
module load python/3
export PYTHONPATH=${HOME}/scratch/mypython3:${PYTHONPATH}
You can run the “pip3 install -t…” command for each python package/library you want to install.
Under Anaconda:
Note
The following commands will create an anaconda environment in your scratch directory (which is temporary), we recommend that you use your group’s project space to install software/libraries specific to your needs.
To create a minimal anaconda environment with python and any dependencies, type the following. Other packages can be used in place of “python” if you want to base your environment around that specific package, for example, “tensorflow”.
mkdir -p /scratch/users/$USER/my.conda.dir
cd ${HOME}
ln -s /scratch/users/${USER}/my.conda.dir .conda
module load anaconda3
conda create -n my.anaconda python
conda info -e
source activate my.anaconda
conda info -e
conda install pytorch
conda list
Use the command source activate environment_name instead of conda init or conda activate environment_name to activate your anaconda environment.
To deactivate the anaconda environment, type:
conda deactivate
To create a complete clone anaconda environment, replace
conda create -n my.anaconda python
with
conda create -n my.anaconda anaconda
Viewing Installed Python Packages
After enabling Python in your user environment by loading a Python or Anaconda modulefile, one can view a list of the Python packages installed under the Python/Anaconda installation (including your own installed packages) by typing pip list if you have loaded a Python modulefile or conda list if you have loaded an Anaconda modulefile.
Sample Python Job Batch Script
A sample python job batch script is available at /sw/cc.users/slurm/python.sbatch.
MATLAB
Introduction
MATLAB (MATrix LABoratory) is a high-level language and interactive environment for numerical computation, visualization, and programming. Developed by MathWorks, MATLAB allows you to analyze data, develop algorithms, and create models and applications.
MATLAB is available on the Campus Cluster along with a collection of toolboxes all of which are covered by a campus concurrent license.
Versions
See Core Software and Libraries for the version(s) of MATLAB installed on the cluster.
Adding MATLAB to Your Environment
Each MATLAB installation on the Campus Cluster has a module that you can use to load a specific version of MATLAB into your user environment.
You can see the available versions of MATLAB by typing module avail matlab on the command line.
The latest version of MATLAB can be loaded into your environment by typing module load matlab.
To load a specific version, you will need to load the corresponding module.
See the Managing Your Environment (Modules) section for more information about modules.
The MATLAB modules make the corresponding MATLAB product as well as all the installed toolboxes available to the user environment.
To verify which toolboxes are available (and the MATLAB version), type ver at the prompt of an interactive MATLAB session.
Running MATLAB Batch Jobs
Execution of MATLAB should be restricted to compute nodes that are part of a batch job. For detailed information about running jobs on the Campus Cluster, see Running Jobs.
Standard batch job
A sample batch script that runs a single MATLAB task with a single m-file is available in /sw/cc.users/slurm/matlab.sbatch that you can copy and modify for your own use. Submit the job with:
[cc-login1 ~]$ sbatch matlab.sbatch
Interactive batch job
For the GUI (which will display on your local machine), use the -x11 option with the srun command. Replace account_name with the name of an account available to you. If you don’t know the account(s) available to you, ask your technical representative or submit a support request.
srun -A account_name --x11 --export=All --time=00:30:00 --nodes=1 --cpus-per-task=16 --partition=secondary --pty /bin/bash
Once the batch job starts, you will have an interactive shell prompt on a compute node. Then type:
module load matlab
matlab
An X-Server must be running on your local machine with X11 forwarding enabled within your SSH connection in order to display X-Apps, GUIs, and so on, back on your local machine.
Generally, users on Linux-based machines only have to enable X11 forwarding by passing an option (
-Xor-Y) to the SSH command.Users on Windows machines will need to ensure that their SSH client has X11 forwarding enabled and have an X-Server running.
A list of SSH clients (which includes a combo packaged SSH client and X-Server) can be found in the SSH Clients section.
Additional information about running X applications can be found on the Using the X Window System page.
For the command line interface:
srun -A account_name --export=All --time=00:30:00 --nodes=1 --cpus-per-task=16 --partition=secondary --pty /bin/bash
(Replace account_name with the name of an account available to you. If you don’t know the account(s) available to you, ask your technical representative or submit a support request.)
Once the batch job starts, you will have an interactive shell prompt on a compute node. Then type:
module load matlab
matlab -nodisplay
Parallel MATLAB
The Parallel Computing Toolbox (PCT) lets you solve computationally and data-intensive problems using multicore processors. High level constructs, parallel for loops, special array types, and parallelized numerical algorithms let you parallelize MATLAB applications without MPI programming. Under MATLAB versions 8.4 and earlier, this toolbox provides 12 workers (MATLAB computational engines) to execute applications locally on a single multicore node of the Campus Cluster. Under MATLAB version 8.5 the number of workers available is equal to the number of cores on a single node (up to a maximum of 512). See MATLAB Errors for error messages generated when violating this limit.
When submitting multiple parallel MATLAB jobs on the Campus Cluster a race condition to write temporary MATLAB job information to the same location can occur if two or more jobs start at the same time. This race condition can cause one or more of the parallel MATLAB jobs fail to use the parallel functionality of the toolbox. See MATLAB Errors for error messages generated when this occurs. Note that non-parallel MATLAB jobs do not suffer from this race condition.
To avoid this behavior, the start times of the parallel MATLAB jobs can be staggered by submitting each subsequent job to the batch system with the -W depend=after:JobID option (see the Job Dependencies section for more information about this option).
sbatch parallel.ML-job.sbatch
sbatch --dependency=after:JobID.01 parallel.ML-job.sbatch
sbatch --dependency=after:JobID.02 parallel.ML-job.sbatch
...
sbatch --dependency=after:JobID.NN parallel.ML-job.sbatch
Note
The MATLAB Distributed Computing Server (MDCS) is not installed on the Campus Cluster because the latest versions of MDCS are not covered under the campus concurrent license. Therefore, MATLAB jobs are restricted to the parallel computing functionality of MATLAB’s Parallel Computing Toolbox.
The UI WebStore offers MDCS for release 2010b - you can contact them directly at webstore@illinois.edu for information and download instructions (for use with release 2010b only - it is not compatible for use with other versions).
MATLAB matlabpool is no longer available
The matlabpool function is not available in beginning with version 8.5(R2015a).
The parpool function should be used instead.
Additional information can be found in the Parallel Computing Toolbox release notes.
MATLAB Errors
The following are some example errors encountered when running MATLAB on the Campus Cluster using MATLAB versions >= 8.5.
Trying to start a matlabpool (or parpool) with more than 12 workers
Error message generated when trying to start a parpool with more than 12 workers
parpool('local', 24)
>> Starting parallel pool (parpool) using the 'local' profile ...
Error using parpool (line 103)
You requested a minimum of 24 workers, but the cluster "local" has the
NumWorkers property set to allow a maximum of 12 workers. To run a
communicating job on more workers than this (up to a maximum of 512 for the
Local cluster), increase the value of NumWorkers property for the cluster.
The default value of NumWorkers for a Local cluster is the number of cores on
the local machine.
Trying to start a matlabpool (or parpool) with 12 workers using 2 nodes and 6 ppn/node
Error message generated when trying to start a parpool with 12 workers using 2 nodes and 6 ppn/node
parpool('local', 12)
>> Starting parallel pool (parpool) using the 'local' profile ...
Error using parpool (line 103)
You requested a minimum of 12 workers, but the cluster "local" has the
NumWorkers property set to allow a maximum of 6 workers. To run a communicating
job on more workers than this (up to a maximum of 512 for the Local cluster),
increase the value of the NumWorkers property for the cluster. The default
value of NumWorkers for a Local cluster is the number of cores on the local
machine.
When 2 or more parallel MATLAB jobs start at the same time (see the Parallel MATLAB section for details)
Example 1
>>Starting parallel pool (parpool) using the 'local' profile ... Error using parpool (line 103) Failed to start a parallel pool. (For information in addition to the causing error, validate the profiles 'local' in the Cluster Profile Manager.) Caused by: Error using parallel.internal.pool.InteractiveClient>iThrowWithCause (line 667) Failed to start pool. Error using parallel.Job/createTask (line 277) Only one task may be created on a communicating Job.Example 2
>> Starting parallel pool (parpool) using the 'local' profile ... Error using parpool (line 103) Failed to start a parallel pool. (For information in addition to the causing error, validate the profile 'local' in the Cluster Program Manager.) Caused by: Error using parallel.internal.pool.InteractiveClient>iThrowWithCause (line 667) Failed to start pool. Error using parallel.Cluster/createCommunicatingJob (line 92) The storage metadata file is corrupt. Please delete all files in the JobStorageLocation and try again.
Mathematica
Note
Mathematica is not currently available on the Campus Cluster.