Programming Environment (Building Software)

The RAILS programming environment supports the GNU, Intel and NVIDIA HPC compilers.

Modules provide access to the compiler + MPI environment.

The default environment includes the GCC compiler + OpenMPI with support for cuda and gdrcopy. nvcc is in the cuda module and is loaded by default.

Recommended compiler flags for GNU and Intel compilers for Sapphire Rapids processors include “-march=sapphirerapids -mtune=sapphirerapids” when using GNU and “-xsapphirerapids” with the Intel compiler. Both are recommended to optimize at the -O3 or -Ofast level.

Compiling

Serial Programs

To build (compile and link) a serial program in Fortran, C, and C++:

Serial Program Commands
GNU	Intel	NVIDIA
gfortran myprog.f	ifort myprog.f	nvfortran myprog.f
gcc myprog.c	icc myprog.c	nvc myprog.c
g++ myprog.cc	icpc myprog.cc	nvc++ myprog.cc

MPI Programs

To build (compile and link) an MPI program, use the mpicc, mpiCC, mpif77 or mpif90 compiler wrappers to automatically include the OpenMPI libraries.

For example:

mpicc -o mpi_hello mpi_hello.c

Python

If you want a basic, recent Python setup, use the python installation under the gcc module. You can add modules via pip3 install --user <modulename>, setup virtual environments, and customize as needed for your workflow, but starting from an installed base of Python smaller than Anaconda.

$ module load gcc python
$ which python
/sw/spack/v1/apps/python/3.11.6-gcc-11.4.0-apaxxj5/bin/python
$ module list

Currently Loaded Modules:
  1) scripts/script_paths   3) StdEnv       5) cuda/11.8.0     7) python/3.11.6
  2) user/license_file      4) gcc/11.4.0   6) openmpi/4.1.6

View the python packages installed in this environment using pip3 list

OpenMP Programs

To build an OpenMP program, use the -fopenmp/-mp option:

OpenMP Program Commands
GNU	Intel	NVIDIA
gfortran -fopenmp myprog.f	ifort -fopenmp myprog.f	nvfortran -mp myprog.f
gcc -fopenmp myprog.c	icc -fopenmp myprog.c	nvc -mp myprog.c
g++ -fopenmp myprog.cc	icpc -fopenmp myprog.cc	nvc++ -mp myprog.cc

Hybrid MPI/OpenMP Programs

To build an MPI/OpenMP hybrid program, use the -fopenmp / -mp option with the MPI compiling commands:

Hybrid MPI/OpenMP Program Commands
GNU, Intel	NVIDIA
mpif77 -fopenmp myprog.f	mpif77 -mp myprog.f
mpif90 -fopenmp myprog.f90	mpif90 -mp myprog.f90
mpicc -fopenmp myprog.c	mpicc -mp myprog.c
mpic++ -fopenmp myprog.cc	mpic++ -mp myprog.cc

OpenACC Programs

To build an OpenACC program, use the -acc option and the -mp option for multi-threaded, under the NVIDIA compilers:

OpenACC Program Commands
NON-MULTITHREADED	MULTITHREADED
nvfortran -acc myprog.f	nvfortran -acc -mp myprog.f
nvc -acc myprog.c	nvc -acc -mp myprog.c
nvc++ -acc myprog.cc	nvc++ -acc -mp myprog.cc

CUDA

The cuda compiler (nvcc) is included in the cuda module, which is loaded by default. For access to the cuda fortran compiler, cuda c++ compiler and other Nvidia development tools, load the “nvhpc” module.

[cmendes@railsl1 /]$ nv
nv-fabricmanager         nvcpuid                  nvidia-debugdump         nvlink
nv-hostengine            nvcudainit               nvidia-modprobe          nvprepro
nv-nsight-cu             nvdecode                 nvidia-persistenced      nvprof
nv-nsight-cu-cli         nvdisasm                 nvidia-powerd            nvprune
nvaccelerror             nvextract                nvidia-settings          nvsize
nvaccelinfo              nvfortran                nvidia-sleep.sh          nvswitch-audit
nvc                      nvidia-bug-report.sh     nvidia-smi               nvunzip
nvc++                    nvidia-cuda-mps-control  nvidia-xconfig           nvvp
nvcc                     nvidia-cuda-mps-server   nvjtag_discovery         nvzip