Data Management
File Systems
File System |
Path |
Quota |
Snapshots / Backups |
Purged |
Key Features |
|---|---|---|---|---|---|
HOME |
|
90 GB. 600,000 files per user. |
No |
No |
Area for software, scripts, job files, and so on. Not intended as a source or destination for I/O during jobs. |
PROJECTS |
|
500 GB. Up to 1-25 TB by allocation request. Large requests may have a monetary fee. |
No |
No |
Area for shared data for a project, common data sets, software, results, and so on. |
WORK - HDD |
|
1000 GB. Up to 1-100 TB by allocation request. |
No |
No |
Area for computation, largest allocations,
where I/O from jobs should occur.
Will be shared between Delta and DeltaAI. |
WORK - NVME |
|
NVME space is available upon request; submit a support request. |
No |
No |
Area for computation, NVME is best for lots of
small I/O from jobs.
Will be shared between Delta and DeltaAI. |
|
|
3.9 TB shared or dedicated depending on node usage by job(s), no quotas in place. |
No |
After each job |
Locally attached disk for fast small file I/O. |
File System Notes
Each user has a home directory,
$HOME, located at/u/$USER.For example, a user with the username auser that has an allocated project with a local project serial code abcd will see the following entries in their
$HOME,/projects, and/workfile systems.$ ls -ld /u/$USER drwxrwx---+ 12 root root 12345 Feb 21 11:54 /u/$USER $ ls -ld /projects/abcd drwxrws---+ 45 root delta_abcd 4096 Feb 21 11:54 /projects/abcd $ ls -l /projects/abcd total 0 drwxrws---+ 2 auser delta_abcd 6 Feb 21 11:54 auser drwxrws---+ 2 buser delta_abcd 6 Feb 21 11:54 buser ... $ ls -ld /work/hdd/abcd drwxrws---+ 45 root delta_abcd 4096 Feb 21 11:54 /work/hdd/abcd $ ls -l /work/hdd/abcd total 0 drwxrws---+ 2 auser delta_abcd 6 Feb 21 11:54 auser drwxrws---+ 2 buser delta_abcd 6 Feb 21 11:54 buser ...
Determine the mapping of ACCESS project to local project using the
accountscommand.Directory access changes can be made using the facl command. Submit a support request if you need assistance enabling access for specific users and projects.
To avoid issues when file systems become unstable or non-responsive, do not put symbolic links from
$HOMEto the/projectsand/workspaces.
/tmp on Compute Nodes (Job Duration)
The high performance SSD storage is available in /tmp (*unique to each node but shared with jobs on the node) and may contain less than the expected free space if the node(s) are running multiple jobs.
Codes that need to perform I/O to many small files should target /tmp on each node of the job and save results to other file systems before the job ends.
Quota Usage
The quota command allows you to view your use of the file systems and use by your projects.
Below is a sample output for a person, “<user>”, who is in one project, “aaaa”.
The home directory quota does not depend on which project group the file is written with.
[<user>@dt-login01 ~]$ quota
Quota usage for user <user>:
-------------------------------------------------------------------------------------------
| Directory Path | User | User | User | User | User | User |
| | Block| Soft | Hard | File | Soft | Hard |
| | Used | Quota| Limit | Used | Quota | Limit|
--------------------------------------------------------------------------------------
| /u/<user> | 20k | 90G | 95G | 5 | 600000 | 660000 |
--------------------------------------------------------------------------------------
Quota usage for groups user <user> is a member of:
-------------------------------------------------------------------------------------
| Directory Path | Group | Group | Group | Group | Group | Group |
| | Block | Soft | Hard | File | Soft | Hard |
| | Used | Quota | Limit | Used | Quota | Limit |
-------------------------------------------------------------------------------------------
| /projects/aaaa | 8k | 500G | 550G | 2 | 300000 | 330000 |
-------------------------------------------------------------------------------------------
| /work/hdd/aaaa | 8k | 552G | 607.2G| 2 | 500000 | 550000 |
-------------------------------------------------------------------------------------------
| /work/nvme/aaaa | 24k | 9.766T| 10.74T| 6 | 500000 | 550000 |
------------------------------------------------------------------------------------------
File Sharing
Users may share files from the /projects file system on DeltaAI to external users via Globus.
Create a directory to share from in your /projects directory. If your four-character allocation code is “XXXX” then do something like:
mkdir /projects/XXXX/globus_shared/
mkdir /projects/XXXX/globus_shared/my_data/
Then move or copy whatever data you want to share to that directory.
Follow the instructions on this Globus sharing page to share that directory. You will need to authenticate and connect to the ACCESS Delta endpoint to make this work. Share the collection from the directory you created; in the above example: /projects/XXXX/globus_shared/my_data/.
Transferring Data
Note
ssh/scp/sftp will work with Duo MFA. A good first step is to use the interactive (not stored/saved) password option with these apps. The interactive login should present you with the first password prompt (your NCSA/Kerberos password) followed by the second password prompt for Duo (push to device or passcode from the Duo app).Secure Copy (scp)
Use scp for small to modest transfers to avoid impacting the usability of the DeltaAI login node. Go to Transferring Files - scp for instructions on using scp on NCSA computing resources.
rsync
Use rsync for small to modest transfers to avoid impacting the usability of the DeltaAI login node. Go to Transferring Files - rsync for instructions on using rsync on NCSA computing resources.
Globus - Coming Soon!
Use Globus for large data transfers. Globus is a web-based file transfer system that works in the background to move files between systems with Globus endpoints.