Job Accounting

The charge unit for Delta is the service unit (SU). This corresponds to the equivalent use of one compute core utilizing less than or equal to 2G of memory for one hour, or 1 GPU or fractional GPU using less than the corresponding amount of memory or cores for 1 hour (see table below). Charges are based on the resources that are reserved for your job and do not necessarily reflect how the resources are used. Charges are based on either the number of cores or the fraction of the memory requested, whichever is larger.

Service Unit Equivalents by Node Type

Node Type

Service Unit Equivalence

Cores

GPU Fraction

Host Memory

CPU Node

1

N/A

2 GB

GPU Node

Quad A100

16

1 A100

62.5 GB

Quad A40

16

1 A40

62.5 GB

8-way A100

16

1 A100

250 GB

8-way MI100

16

1 MI100

250 GB

Note that a weighting factor will discount the charge for the reduced-precision A40 nodes, as well as the novel AMD MI100 based node - this will be documented through the ACCESS SU converter.

Also note that 1 GB of memory here means 1e9 bytes (1,000,000,000), not 2^30 bytes (1,073,741,824).

Local Account Charging

Use the accounts command to list the accounts available for charging. CPU and GPU resources will have individual charge names. For example, in the following, bbka-delta-cpu and bbka-delta-gpu are available for user kingda to use for the CPU and GPU resources.

$ accounts
Project Summary for User 'kingda':

Project         Description                    Balance (Hours)    Deposited (Hours)
--------------  ---------------------------  -----------------  -------------------
bbka-delta-gpu  ncsa/delta staff allocation            5000000              5000000
bbka-delta-cpu  ncsa/delta staff allocation          100000000            100000000

Job Accounting Considerations

  • A node-exclusive job that runs on a compute node for one hour will be charged 128 SUs (128 cores x 1 hour)

  • A node-exclusive job that runs on a 4-way GPU node for one hour will be charged 4 SUs (4 GPU x 1 hour)

  • A node-exclusive job that runs on a 8-way GPU node for one hour will be charged 8 SUs (8 GPU x 1 hour)

QOSGrpBillingMinutes

If you see QOSGrpBillingMinutes under the Reason column for the squeue command, as in:

  JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
1204221       cpu    myjob     .... PD       0:00      5 (QOSGrpBillingMinutes)

Then the resource allocation specified for the job (e.g., xyzt-delta-cpu) does not have sufficient balance to run the job based on the number of resources requested and the wall-clock time. Sometimes it may be other jobs from the same project, also in the QOSGrpBillingMinutes state, using the same resource allocation, that are preventing a job that would normally “fit” from running. To resolve this, the PI of the project needs to put in a supplement request using the same XRAS proposal system that was used for the current award, see Allocation Supplements and Extensions.

Reviewing Job Charges for a Project (jobcharge)

jobcharge in /sw/user/scripts/ will show job charges by user for a project. Example usage:

jobcharge_grp.py (click to expand/collapse)

The example jobcharge commands are showing results for the bbka-delta-gpu account. Accounts available to you are listed under “Project” when you run the accounts command.

[arnoldg@dt-login03 ~]$ jobcharge bbka-delta-gpu -d 10 --detail | tail -15
141  3374332  gpuA40x4-interactive           1372  billing=1000,cpu=2,gres/gpu:a40=1,gres/gpu=1,mem=16G,node=1       arnoldg          ondemand/sys/dashbo
ard/sys/jupyter-lab  2024-04-10T09:08:27       1000      0.38
142  3375013  gpuA100x4                        11  billing=1000,cpu=1,gres/gpu:a100=1,gres/gpu=1,mem=1000M,node=1    svcdeltajenkins  bandwidthTest
                     2024-04-10T17:20:07       1000      0
143  3376558  gpuA100x4-interactive           443  billing=2000,cpu=1,gres/gpu:a100=1,gres/gpu=1,mem=16G,node=1      svcdeltajenkins  python
                     2024-04-11T03:48:25       2000      0.25
144  3376599  gpuA40x4                          5  billing=2000,cpu=4,gres/gpu:a40=4,gres/gpu=4,mem=128G,node=4      svcdeltajenkins  mpi_small
                     2024-04-11T05:18:35       2000      0
145  3376600  gpuA100x4                         5  billing=4000,cpu=4,gres/gpu:a100=4,gres/gpu=4,mem=128G,node=4     svcdeltajenkins  mpi_small
                     2024-04-11T05:18:41       4000      0.01
146  3376623  gpuA100x4-interactive           141  billing=2000,cpu=1,gres/gpu:a100=1,gres/gpu=1,mem=16G,node=1      svcdeltajenkins  python3
                     2024-04-11T07:45:22       2000      0.08
____________SUMMARY____________
User               Charge (SU)
---------------  -------------
arnoldg                   1.7
gbauer                    0.37
jlong                     0.01
rbrunner                  0.89
svcdeltajenkins           3.52
Total                     6.49

[arnoldg@dt-login03 ~]$ jobcharge bbka-delta-gpu -d 10
Charges for account bbka-delta-gpu from 2024-04-01-08:21:18 through 2024-04-11-08:21:18.
User               Charge (SU)
---------------  -------------
arnoldg                   1.7
gbauer                    0.37
jlong                     0.01
rbrunner                  0.89
svcdeltajenkins           3.52
Total                     6.49

[arnoldg@dt-login03 ~]$ jobcharge -h
usage: jobcharge [-h] [-s STARTTIME] [-e ENDTIME] [--detail] [-m MONTH] [-y YEAR] [-d DAYSBACK] account

positional arguments:
  account               Name of the account to get jobcharges for. "accounts" command can be used to list your valid accounts.

optional arguments:
  -h, --help            show this help message and exit
  -s STARTTIME, --starttime STARTTIME
                        Get jobcharges after this time. Default is one month before the current time.
  -e ENDTIME, --endtime ENDTIME
                        Get jobcharges before this time. Default is the current time.
  --detail              detail output, per-job
  -m MONTH, --month MONTH
                        Get jobcharges for a specific month (1-12). Will override start end time arguments
  -y YEAR, --year YEAR  Get jobcharges for a specific year. Will override start end time arguments
  -d DAYSBACK, --daysback DAYSBACK
                        Get jobcharges from the previous N days. Will take precedence over all other time search arguments

Refunds

Refunds are considered, when appropriate, for jobs that failed due to circumstances beyond user control.

To request a refund, submit a support request. Please include the batch job IDs and the standard error and output files produced by the job(s).