Latest Changes On Delta

This page lists the new versions of software and new features available as of when Delta transitioned to RedHat 9 in 2025.

NVIDIA Driver and CUDA

The NVIDIA driver and base CUDA is updated to provide more recent (but not the latest) release as shown in the following table:

	NVIDIA Driver	Base CUDA
New	570.148.08	12.8

Default Programming Environment

Programming Environment (PE)

The compiler, MPI implementation and other base packages will be provided by the Cray Programming Environment (CrayPE), similar to how the default programming environment is provided on DeltaAI.

The default environment will be based on the GNU CrayPE PrgEnv-gnu. The default MPI implementation will be Cray’s MPICH.

	GCC compiler	Module names	CUDA	MPI	Module name
PE	gcc	PrgEnv-gnu gcc-native/13.2	cudatoolkit/25.3_12.8	Cray MPICH	cray-mpich

Other Cray PEs are available such as PrgEnv-nvidia (NVIDIA HPC SDK compilers) and PrgEnv-cray (Cray compilers). All programming environments are set by default to the the cray-mpich module.

Use the modules command to view the default loaded modules

[gbauer@dt-login04 ~]$ module list

Currently Loaded Modules:
  1) gcc-native/13.2      6) cray-libsci/25.03.0    11) craype-accel-nvidia80
  2) craype/2.7.34        7) PrgEnv-gnu/8.6.0       12) cue-login-env/1.1
  3) libfabric/1.22.0     8) cray-dsmml/0.3.1       13) slurm-env/0.1
  4) craype-network-ofi   9) craype-x86-milan       14) default
  5) cray-mpich/8.1.32   10) cudatoolkit/25.3_12.8

Use of Compiler Wrappers

The CrayPE compiler wrappers cc, CC and ftn are recommended when building C, C++ and Fortran libraries and applications. The wrappers automatically include paths to include files and libraries for MPI and GPU RDMA and CUDA.

The CrayPE provides MPI compiler wrappers mpicc, mpicxx/mpic++ and mpifort/mpif77/mpif90 can be used with CPU MPI codes but require some additional include path and libraries when compiling GPU aware libraries for GPU RDMA as mentioned in the mpi man page (see: man mpi).

The following environment variables have been set to help use the compiler wrappers:

Environment Variable	Default Setting
CC	cc
CXX	CC
FC	ftn
MPICC	mpicc
MPICXX	mpicxx
MPIF77	mpif77
MPIF90	mpif90
CMAKE_C_COMPILER	cc
CMAKE_CXX_COMPILER	CC
CMAKE_Fortran_COMPILER	ftn

Support for GPU RDMA

The Cray Programming Environments: PrgEnv-gnu, PrgEnv-nvidia and PrgEnv-cray support GPU RDMA. Compiler and runtime support is configured by default for PrgEnv-gnu and PrgEnv-nvidia.

To enable support for GPU RDMA the environment variable MPICH_GPU_SUPPORT_ENABLED needs to be set

export MPICH_GPU_SUPPORT_ENABLED=1

If you see

aborting job:
MPIDI_CRAY_init: GPU_SUPPORT_ENABLED is requested, but GTL library is not linked

then you set the environment variable but did not properly link the executable with -lmpi_gtl_cuda or use the cc, CC or ftn compiler wrappers.

NCCL

Please load the aws-ofi-nccl module so that NCCL will use the appropriate high-speed network provider for NCCL.

This module provides the AWS OFI network transport plugin for NCCL,
optimized for Cray systems with CXI interconnect.

Dependencies (must be loaded first):
- cudatoolkit/25.3_12.8
- libfabric/1.22.0

Python

Several python packages are available for use:

miniforge3-python
pytorch-conda/2.8
tensorflow-conda/2.18

Use the module spider command to find packages with modules.

When installing Python packages, especially mpi4py with GPU support we recommend setting the MPICC environment variable as follows:

for GPU Python

MPICC="cc -shared" pip install mpi4py

MPI

Only the cray-mpich module is supported. OpenMPI performance is less than one-half of what we see with the cray-mpich implementation, so priority to redeploy OpenMPI is reduced.