Software
Core Software and Libraries
The Campus Cluster maintains a variety of core software and libraries to support computing activities. Software can be added to a user’s environment using modules.
Campus Cluster core software and libraries list (click to expand/collapse)
Software |
Version |
---|---|
Anaconda |
5.2.0 anaconda/2 (python 2.7.15) anaconda/3 (python 3.6.5) |
2018.12 anaconda/2018-Dec/2 (python 2.7.15) anaconda/2018-Dec/3 (python 3.6.8) |
|
2019.10 anaconda/2019-Oct/2 (python 2.7.16) anaconda/2019-Oct/3 (python 3.7.4) |
|
2020.07 anaconda/2020-Jul/3 (python 3.8.3) |
|
2021.05 anaconda/2021-May/3 (python 3.8.8) |
|
2022.05 anaconda/2022-May/3 (python 3.9.12) |
|
2023.03 anaconda/2023-Mar/3 (python 3.10.9) |
|
Boost |
1.68.0 1.71.0 |
CMake |
3.12.0 3.18.4 |
Cuda |
9.2 10.0 11.6 |
Emacs |
26.1 27.1 |
FFTW |
3.3.8 (Intel MPI, MVAPICH2, OpenMPI) |
3.3.9 (Intel MPI, MVAPICH2, OpenMPI) |
|
GCC |
7.2.0 (default) 8.2.0 9.2.0 11.2.0 12.2.0 |
Git |
2.19.0 |
GSL |
2.5 |
Intel compilers |
18.0 19.0 20.4 |
Intel MKL |
2018 (Intel 18 compilers) 2019 (Intel 19 compilers) 2020 (Intel 20 compilers) |
Intel MPI |
2018 (Intel 18 compilers) 2019 (Intel 19 compilers) 2020 (Intel 20 compilers) |
JDK (Oracle/Sun) |
1.8 (default) 9 10 11 12 13 |
Mathematica |
11 12 |
MATLAB |
9.4 9.5 9.7 |
MVAPICH2 |
2.3 2.3.4 |
OpenBLAS |
0.2.20 0.3.7 0.3.12 |
OpenMPI |
3.1.1 4.0.5 4.1.0 4.1.1 4.1.4 |
PAPI |
5.6.0 |
PETSc |
3.10.0 3.11.1 3.15.4 |
Python |
2.7.15 (python/2) 3.7.0 (python/3) 3.9.16 (python/3.9.16) |
R |
3.5.1 3.6.2 4.0.1 4.0.2 4.0.3 4.1.1 4.2.2 |
Singularity |
3.2.0 3.8.5 |
SVN |
1.10.3 |
Valgrind |
3.13.0 |
VIM |
8.1 |
Managing Your Environment (Modules)
The module command is a user interface to the Modules package. The Modules package provides for the dynamic modification of the user’s environment via modulefiles. A modulefile contains the information needed to configure the shell for an application. Modules are independent of the user’s shell, so both tcsh and bash users can use the same commands to change the environment.
Command |
Description |
---|---|
|
lists all available modulefiles |
|
lists currently loaded modulefiles |
|
help on module |
|
display information about |
|
load |
|
remove |
|
unload |
To include particular software in the environment for all new shells, edit your shell configuration file ($HOME/.bashrc
for bash users and $HOME/.cshrc
for tcsh users) by adding the module commands to load the software that you want to be a part of your environment.
After saving your changes, you can source your shell configuration file or log out and then log back in for the changes to take effect.
Note: Order is important. With each module load, the changes are prepended to your current environment paths.
For additional information on Modules, see the module and modulefile man pages or visit the Modules SourceForge page.
R on the Campus Cluster
Introduction
R is a programming language and software environment for statistical computing and graphics. R and its libraries implement a wide variety of statistical and graphical techniques, including linear and non-linear modelling, classical statistical tests, time-series analysis, classification, clustering, and others. R is easily extensible through functions and extensions, and the R community is noted for its active contributions in terms of packages. R is also an interpreted language in which users typically access it through a command line interpreter.
R and R packages are available via the Comprehensive R Archive Network (CRAN), a collection of sites which carry identical material, consisting of the R distribution(s), the contributed extensions, documentation for R, and binaries.
Versions
See the Core Software and Libraries for the versions of R installed on the Campus Cluster.
Adding R to Your Environment
Each R installation on the Campus Cluster has a modulefile that you can use to load a specific version of R into your user environment.
You can see the available versions of R by typing module avail R
on the command line.
The latest version of R available on the Campus Cluster can be loaded into your environment by typing module load R
.
To load a specific version, you will need to load the corresponding modulefile. See Managing Your Environment (Modules), for more information about modules.
Installing Add-on Packages (in user specified locations)
Any R add-on package not available in the system installation (see Viewing Installed R Packages for information on viewing the list) can be installed from the CRAN in a user specified location that you have write access to with just a few easy steps.
Generally, when installing any software there are a few things to consider:
Home directory disk usage quotas (see Storage Areas).
Will the software be used by multiple users in my group?
Project space is the recommended location to install software (see Investor-Specific Software Installation).
Does the desired software have dependencies/requirements?
Is the prerequisite software already installed?
Installation Command Syntax
Note
The install.packages()
command is run within R. You need to load (module load r
) and run (r
) R prior to using install.packages()
.
R package installations, for the most part, are straight forward. Minimally, all that is needed is the name of the package that is to be installed, but installation location and the repository can also be specified.
Command (“Package Name”, “Location”, “Repository”)
install.packages('package_name', '/path/to/r_libraries', 'Repository URL')Behavior: Installs the package downloaded from the specified repository in the specified location.
Command (“Local Filename”, “Location”, “Repository”)
install.packages('package_name.tar.gz', '/path/to/r_libraries', repos = NULL)Behavior: Installs the local package(file) in the specified location specifying no repository(NULL)
When the installation location and the repository URL are not specified, R packages are installed in a default location and the R installation process prompts to choose from a list of repositories.
R packages downloaded manually from the CRAN can be installed by specifying the local filename and omitting the repository URL (specifying NULL).
Using Rscript
To run R commands without starting an R session the command Rscript
can be used (as we show in the examples on this page).
As a scripting front end for R, Rscript
enables the use of R via shell scripts and scripting applications.
Installation Steps
Below are steps with examples of the actual commands that can be run on the Campus Cluster.
In these steps ~/Rlibs
is used for the location to install user specific add-on packages (The tilde “~” means the user’s home directory—i.e., $HOME
).
Note
It is assumed that these steps are being run under the default BASH shell. When using a different shell, the syntax for that shell should be used where appropriate.
Create a directory for your R packages:
mkdir ~/Rlibs
Load the R modulefile:
module load R/4.2.2_sandybridge
Set the R library environment variable (R_LIBS) to include your R package directory:
export R_LIBS=~/Rlibs:${R_LIBS}
Use the
install.packages
function to install your R package:Rscript -e "install.packages('RCurl', '~/Rlibs', 'https://cran.r-project.org')"
If the environment variable R_LIBS
is not set and a directory is not specified with the install.packages
command, then R packages will be installed under ~/R/x86_64-unknown-linux-gnu-library
by default. This R subdirectory structure is created automatically.
The R_LIBS
environment variable will need to be set every time when logging in to the Campus Cluster if the user’s R package location is to be visible to an R session.
The following can be added to a user’s ~/.bashrc
file to remove the need to set the R_LIBS environment variable with every login session to the Campus Cluster:
if [ -n $R_LIBS ]; then
export R_LIBS=~/Rlibs:$R_LIBS
else
export R_LIBS=~/Rlibs
fi
Warnings and Error Messages
R packages that are not available in the current CRAN (Comprehensive R Archive Network) or if the name of the package is misspelled, tend to generate a message similar to the following:
[cc-login1 ~]$ Rscript -e "install.packages('phybase','~/Rlibs', 'https://cran.r-project.org')"
Warning message:
package 'phybase' is not available (for R version 3.2.2)
Searching the CRAN site for your desired R package may provide links to archived versions that are not available in the current CRAN. In this case the specific archived R package can be downloaded and installed from the local file using the same command but omitting the repository URL (specifying NULL).
Some R packages have dependencies and require them to be installed first and will generate an error message similar to the following:
[cc-login1 ~]$ Rscript -e "install.packages('phybase_1.1.tar.gz', '~/Rlibs', repos = NULL)"
ERROR: dependency 'ape' is not available for package 'phybase'
* removing '/home/jdoe/Rlibs/phybase'
Warning message:
In install.packages("phybase_1.1.tar.gz", repos = NULL) :
installation of package 'phybase_1.1.tar.gz' had non-zero exit status
Installing the required R package first, then the desired R package resolves this issue.
Viewing Installed R Packages
The library()
command can be used to view all user and system installed R packages (user installed packages are only visible to R when the $R_LIBS
environment variable is set).
[cc-login1 ~]$ Rscript -e "library()"
Packages in library '/home/jdoe/Rlibs':
R6 Classes with reference semantics
RCurl General network (HTTP/FTP/...) client interface
for R
...
stringr Simple, Consistent Wrappers for Common String
Operations
whisker {{mustache}} for R, logicless templating
Packages in library '/usr/local/R/4.2.2/lib64/R/library':
KernSmooth Functions for kernel smoothing for Wand & Jones
(1995)
MASS Support Functions and Datasets for Venables and
Ripley's MASS
...
tools Tools for Package Development
utils The R Utils Package
Python on the Campus Cluster
Introduction
Python is an interpreted, high-level, general-purpose programming language.
Python and Python packages are available via the The Python Package Index (PyPI), which hosts thousands of third-party modules for Python. Both Python’s standard library and the community-contributed modules allow for endless possibilities.
Anaconda also provides a Python environment with python packages.
Versions
See the Core Software and Libraries for the versions of Python and Anaconda installed on the Campus Cluster.
Adding Python to Your Environment
Each Python installation on the Campus Cluster has a corresponding modulefile that can be used to load a specific version of Python into your user environment.
You can see the available versions of Python by typing module avail python
and/or module avail anaconda
on the command line.
See Managing Your Environment (Modules) for more information about modules.
Installing Python Packages (in user specified locations)
Generally, any Python package not available in the system installation (see Viewing Installed Python Packages for information on viewing the list) can be installed from the Python Package Index (PyPI) in a user specified location that you have write access to with just a few easy steps.
Generally, when installing any software there are a few things to consider:
Home directory disk usage quotas (see Storage Areas).
Will the software be used by multiple users in my group?
Project space is the recommended location to install software (see Investor-Specific Software Installation).
Does the desired software have dependencies/requirements?
Is the prerequisite software already installed?
Campus Cluster users must install software/libraries into user write-able locations like their home directory, their group’s project space or their scratch space.
Note
The example instructions below use scratch space. Please note that software installed in scratch space is not permanent and may be removed at any time. We recommend that you install in your group’s project space. For a persistent installation, users should replace the path to their scratch space with the path to their project space in the instructions below`
Warning
There is a 5GB quota (disk usage) on all user’s home directories, so we do not recommend that users install software/libraries in their home directories since multiple Python packages (or an Anaconda installation) can push the disk usage over quota very quickly.
Additionally, users should use one of the installation methods listed below (Python or Anaconda). Combining installation methods can cause unexpected behavior/errors.
Installation Command Syntax
Under Python:
To Install a specific python package into a user location, use the pip install
command with the -t
option.
module load python/3
mkdir -p /home/${USER}/scratch/mypython3
pip install -t /home/${USER}/scratch/mypython3 phonopy
export PYTHONPATH=/home/${USER}/scratch/mypython3:${PYTHONPATH}
Users will have to run the above “export
…” command after loading python into their user environment with the module command on future login sessions.
This allows the Campus Cluster’s python installation to see the user’s specific python packages.
Alternatively, users can add two commands to their ${HOME}/.bashrc
file so that Python is setup for their user environment every time they login to the Campus Cluster.
module load python/3
export PYTHONPATH=/home/${USER}/scratch/mypython3:${PYTHONPATH}
You can run the “pip install -t
…” command for each python package/library you want to install.
Under Anaconda:
Note
The following commands will create an anaconda environment in your scratch directory (which is temporary), we recommend that you use your group’s project space to install software/libraries specific to your needs.
To create a minimal anaconda environment with python and any dependencies, type the following. Other packages can be used in place of “python” if you want to base your environment around that specific package, for example, “tensorflow”.
mkdir -p /scratch/users/$USER/my.conda.dir
cd ${HOME}
ln -s /scratch/users/${USER}/my.conda.dir .conda
module load anaconda/2023-Mar/3
conda create -n my.anaconda python
conda info -e
source activate my.anaconda
conda info -e
conda install pytorch
conda list
Use the command source activate environment_name
instead of conda init
or conda activate environment_name
to activate your anaconda environment.
To deactivate the anaconda environment, type:
conda deactivate
To create a complete clone anaconda environment, replace
conda create -n my.anaconda python
with
conda create -n my.anaconda anaconda
Viewing Installed Python Packages
After enabling Python in your user environment by loading a Python or Anaconda modulefile, one can view a list of the Python packages installed under the Python/Anaconda installation (including your own installed packages) by typing pip list
if you have loaded a Python modulefile or conda list
if you have loaded an Anaconda modulefile.