Data Management

File Systems

File System Specs

File System	Path	Quota	Snapshots / Backups	Purged	Key Features
HOME	/u	90 GB. 600,000 files per user.	No	No	Area for software, scripts, job files, and so on. Not intended as a source or destination for I/O during jobs.
PROJECTS	/projects	500 GB. Up to 1-25 TB by allocation request. Large requests may have a monetary fee.	No	No	Area for shared data for a project, common data sets, software, results, and so on.
WORK - HDD	/work/hdd	1000 GB. Up to 1-100 TB by allocation request. submit a support request.	No	No	Area for computation, largest allocations, where I/O from jobs should occur. Will be shared between Delta and DeltaAI.
WORK - NVME	/work/nvme	1000 GB. Up to 1-100 TB by allocation request for DeltaAI projects; submit a support request.	No	No	Area for computation, NVME is best for lots of small I/O from jobs. Will be shared between Delta and DeltaAI.
/tmp	/tmp	3.9 TB shared or dedicated depending on node usage by job(s), no quotas in place.	No	After each job	Locally attached disk for fast small file I/O.

File System Notes

• Each user has a home directory, $HOME, located at /u/$USER.

  For example, a user with the username auser that has an allocated project with a local project serial code abcd will see the following entries in their $HOME, /projects, and /work file systems.

  $ ls -ld /u/$USER
  drwxrwx---+ 12 root root 12345 Feb 21 11:54 /u/$USER
  
  $ ls -ld /projects/abcd
  drwxrws---+  45 root   delta_abcd      4096 Feb 21 11:54 /projects/abcd
  
  $ ls -l /projects/abcd
  total 0
  drwxrws---+ 2 auser delta_abcd 6 Feb 21 11:54 auser
  drwxrws---+ 2 buser delta_abcd 6 Feb 21 11:54 buser
  ...
  
  $ ls -ld /work/hdd/abcd
  drwxrws---+  45 root   delta_abcd      4096 Feb 21 11:54 /work/hdd/abcd
  
  $ ls -l /work/hdd/abcd
  total 0
  drwxrws---+ 2 auser delta_abcd 6 Feb 21 11:54 auser
  drwxrws---+ 2 buser delta_abcd 6 Feb 21 11:54 buser
  ...
  

• Determine the mapping of ACCESS project to local project using the accounts command.

• Directory access changes can be made using the facl command. Submit a support request if you need assistance enabling access for specific users and projects.

• To avoid issues when file systems become unstable or non-responsive, do not put symbolic links from $HOME to the /projects and /work spaces.

• A “module reset” in a job script populates $WORK and $SCRATCH environment variables automatically, or you may set them as WORK=/projects/<account>/$USER, SCRATCH=/scratch/<account>/$USER.

/tmp on Compute Nodes (Job Duration)

The high performance SSD storage is available in /tmp (*unique to each node but shared with jobs on the node) and may contain less than the expected free space if the node(s) are running multiple jobs. Codes that need to perform I/O to many small files should target /tmp on each node of the job and save results to other file systems before the job ends.

Quota Usage

The quota command allows you to view your use of the file systems and use by your projects. Below is a sample output for a person, “<user>”, who is in one project, “aaaa”. The home directory quota does not depend on which project group the file is written with.

[<user>@dt-login01 ~]$ quota
Quota usage for user <user>:
-------------------------------------------------------------------------------------------
| Directory Path  | User | User | User  | User | User   | User |
|                 | Block| Soft | Hard  | File | Soft   | Hard |
|                 | Used | Quota| Limit | Used | Quota  | Limit|
--------------------------------------------------------------------------------------
| /u/<user>       | 20k  | 90G  | 95G   | 5    | 600000 | 660000 |
--------------------------------------------------------------------------------------
Quota usage for groups user <user> is a member of:
-------------------------------------------------------------------------------------
| Directory Path  | Group | Group | Group | Group | Group  | Group |
|                 | Block | Soft  | Hard  | File  | Soft   | Hard  |
|                 | Used  | Quota | Limit | Used  | Quota  | Limit |
-------------------------------------------------------------------------------------------
| /projects/aaaa  | 8k    | 500G  | 550G  | 2     | 300000 | 330000 |
-------------------------------------------------------------------------------------------
| /work/hdd/aaaa  | 8k    | 552G  | 607.2G| 2     | 500000 | 550000 |
-------------------------------------------------------------------------------------------
| /work/nvme/aaaa | 24k   | 9.766T| 10.74T| 6     | 500000 | 550000 |
------------------------------------------------------------------------------------------

File Sharing

Users may share files from the /projects file system on DeltaAI to external users via Globus.

Create a directory to share from in your /projects directory. If your four-character allocation code is “XXXX” then do something like:

mkdir /projects/XXXX/globus_shared/
mkdir /projects/XXXX/globus_shared/my_data/

Then move or copy whatever data you want to share to that directory.

Follow the instructions on this Globus sharing page to share that directory. You will need to authenticate and connect to the ACCESS Delta endpoint to make this work. Share the collection from the directory you created; in the above example: /projects/XXXX/globus_shared/my_data/.

Transferring Data

Note

GUI applications need to support Duo multi-factor authentication (MFA)

Many GUI apps that support ssh/scp/sftp will work with Duo MFA. A good first step is to use the interactive (not stored/saved) password option with these apps. The interactive login should present you with the first password prompt (your NCSA/Kerberos password) followed by the second password prompt for Duo (push to device or passcode from the Duo app).

Secure Copy (scp)

Use scp for small to modest transfers to avoid impacting the usability of the DeltaAI login node. Go to Transferring Files - scp for instructions on using scp on NCSA computing resources.

rsync

Use rsync for small to modest transfers to avoid impacting the usability of the DeltaAI login node. Go to Transferring Files - rsync for instructions on using rsync on NCSA computing resources.

Globus

Use Globus for large data transfers. Globus is a web-based file transfer system that works in the background to move files between systems with Globus endpoints.

Go to Transferring Files - Globus for instructions on using Globus with NCSA computing resources.

The DeltaAI and Delta endpoints are the same. Delta and DeltaAI can see /work/hdd and /work/nvme, and the Delta and DeltaAI home directories will be visible. The DeltaAI endpoint collection names are:

• “NCSA Delta” (authenticates with your NCSA identity)

• “ACCESS Delta” (authenticates with your ACCESS identity)

The endpoint landing location (/) in Globus shows both home directories, and folders for shared project and work file systems.

• The u and delta folders navigate to /u/${USER} on Delta.

• The deltaai folder navigates to /u/${USER} on DeltaAI.

• Note that Delta and DeltaAI have different home directories; the CPUs are different, incompatible architectures.