Data Management

File Systems

File System Specs
File System	Path	Quota	Snapshots / Backups	Purged	Key Features
HOME	`/u`	90 GB. 600,000 files per user.	No	No	Area for software, scripts, job files, and so on. Not intended as a source or destination for I/O during jobs.
PROJECTS	`/projects`	500 GB. Up to 1-25 TB by allocation request. Large requests may have a monetary fee.	No	No	Area for shared data for a project, common data sets, software, results, and so on.
WORK - HDD	`/work/hdd`	1000 GB. Up to 1-100 TB by allocation request.	No	No	Area for computation, largest allocations, where I/O from jobs should occur. Will be shared between Delta and DeltaAI.
WORK - NVME	`/work/nvme`	NVME space is available upon request; submit a support request.	No	No	Area for computation, NVME is best for lots of small I/O from jobs. Will be shared between Delta and DeltaAI.
`/tmp`	`/tmp`	3.9 TB shared or dedicated depending on node usage by job(s), no quotas in place.	No	After each job	Locally attached disk for fast small file I/O.

File System Notes

Each user has a home directory, $HOME, located at /u/$USER.

For example, a user with the username auser that has an allocated project with a local project serial code abcd will see the following entries in their $HOME, /projects, and /work file systems.

$ ls -ld /u/$USER
drwxrwx---+ 12 root root 12345 Feb 21 11:54 /u/$USER

$ ls -ld /projects/abcd
drwxrws---+  45 root   delta_abcd      4096 Feb 21 11:54 /projects/abcd

$ ls -l /projects/abcd
total 0
drwxrws---+ 2 auser delta_abcd 6 Feb 21 11:54 auser
drwxrws---+ 2 buser delta_abcd 6 Feb 21 11:54 buser
...

$ ls -ld /work/hdd/abcd
drwxrws---+  45 root   delta_abcd      4096 Feb 21 11:54 /work/hdd/abcd

$ ls -l /work/hdd/abcd
total 0
drwxrws---+ 2 auser delta_abcd 6 Feb 21 11:54 auser
drwxrws---+ 2 buser delta_abcd 6 Feb 21 11:54 buser
...

Determine the mapping of ACCESS project to local project using the accounts command.
Directory access changes can be made using the facl command. Submit a support request if you need assistance enabling access for specific users and projects.
To avoid issues when file systems become unstable or non-responsive, do not put symbolic links from $HOME to the /projects and /work spaces.

/tmp on Compute Nodes (Job Duration)

The high performance SSD storage is available in /tmp (*unique to each node but shared with jobs on the node) and may contain less than the expected free space if the node(s) are running multiple jobs. Codes that need to perform I/O to many small files should target /tmp on each node of the job and save results to other file systems before the job ends.

Quota Usage

The quota command allows you to view your use of the file systems and use by your projects. Below is a sample output for a person, “<user>”, who is in one project, “aaaa”. The home directory quota does not depend on which project group the file is written with.

[<user>@dt-login01 ~]$ quota
Quota usage for user <user>:
-------------------------------------------------------------------------------------------
| Directory Path  | User | User | User  | User | User   | User |
|                 | Block| Soft | Hard  | File | Soft   | Hard |
|                 | Used | Quota| Limit | Used | Quota  | Limit|
--------------------------------------------------------------------------------------
| /u/<user>       | 20k  | 90G  | 95G   | 5    | 600000 | 660000 |
--------------------------------------------------------------------------------------
Quota usage for groups user <user> is a member of:
-------------------------------------------------------------------------------------
| Directory Path  | Group | Group | Group | Group | Group  | Group |
|                 | Block | Soft  | Hard  | File  | Soft   | Hard  |
|                 | Used  | Quota | Limit | Used  | Quota  | Limit |
-------------------------------------------------------------------------------------------
| /projects/aaaa  | 8k    | 500G  | 550G  | 2     | 300000 | 330000 |
-------------------------------------------------------------------------------------------
| /work/hdd/aaaa  | 8k    | 552G  | 607.2G| 2     | 500000 | 550000 |
-------------------------------------------------------------------------------------------
| /work/nvme/aaaa | 24k   | 9.766T| 10.74T| 6     | 500000 | 550000 |
------------------------------------------------------------------------------------------

Transferring Data

Note

GUI applications need to support Duo multi-factor authentication (MFA)

Many GUI apps that support ssh/scp/sftp will work with Duo MFA. A good first step is to use the interactive (not stored/saved) password option with these apps. The interactive login should present you with the first password prompt (your NCSA/Kerberos password) followed by the second password prompt for Duo (push to device or passcode from the Duo app).

Secure Copy (scp)

Use scp for small to modest transfers to avoid impacting the usability of the DeltaAI login node. Go to Transferring Files - scp for instructions on using scp on NCSA computing resources.

rsync

Use rsync for small to modest transfers to avoid impacting the usability of the DeltaAI login node. Go to Transferring Files - rsync for instructions on using rsync on NCSA computing resources.

Globus - Coming Soon!

Use Globus for large data transfers. Globus is a web-based file transfer system that works in the background to move files between systems with Globus endpoints.