Job Testing, Bencharmarking, Optimization, and Scaling

When you start running a new application on an HPC system, one that runs on many cores or many nodes (or especially both), the overall most efficient way to run the application is dependent on the details of the system configuration, the node architecture, and sometimes even how busy the system is. After you get an application built and runable, but before you launch your production runs, it’s often useful to run some tests to make sure you’re using the system to your best advantage. This page cannot be a full guide to HPC system application tuning, but we can give you a few hints about things to look for.

Single-Core Configuration

If you can run your workflow on a single CPU configuration, and if the individual items of work are small enough to run on a single CPU in less than the maximum job walltime, then it’s worth configuring your workload to run in a single-core configuration. Then you can run many copies of this workflow on different work items, and that may well be your most efficient configuration.

Single-node Configuration

If your application can run on all the CPU cores (or the GPUs) on just a single node, that will often be your most efficient configuration. Running on a single node means you don’t have to worry about communication delays between nodes, and it also means you can run the node in “exclusive” mode, so you don’t have to worry about anyone else sharing that compute node and taking up file system or memory bandwidth. The tradeoff is if you request exclusive mode on a node, then you’re paying out of your budget for every processor on the node, even if you don’t use them.

It is also worth running your application in an exclusive mode on a node but only using half the CPUs (spread out). Sometimes this can actually yield a higher speed for the entire application, because the application can use the node’s memory bandwidth, file system bandwidth, and network bandwidth (and cache) more effectively.

If your code isn’t an MPI code; if it only implements threading via a compiler-level threading library (like pthreads or openMP) then using all the processing on a single node is the fastest.

Multi-node Configuration

After you establish the best single-node configuration, then scale that configuration up. Scale up to 2 or 4 or maybe 8 nodes. It will be important to measure the speed per workload item. It’s easy to lose track of this and optimize for the shortest run time, but for instance, if you double the number of cores but total runtime only improves by 20%, it’s probably not worth it.

When you’re determined the best single-node configuration, and the best multi-node configuration, then use the testing speeds to decide how to run your overall workflow.