Data Management

Globus

Globus is the recommended way to move datasets in and out of HAL.

The HAL endpoint collection name is “ncsa#HAL”.

Visit Globus Connect Personal for instructions on setting up a Globus personal client on a machine. Go to Transferring Files - Globus for instructions on using Globus with NCSA computing resources.

Rclone

Rclone is a command line tool that supports file transfers between a wide variety of services. Rclone is installed on the hal-data host (hal-data.ncsa.illinois.edu). Go to the Rclone documentation for instructions on how to set up various endpoint location types.

You can SSH into hal-data directly or from the HAL login node.

Rsync/SCP

You can also use conventional command line transfer tools like rsync and scp. It is best to perform these processes on hal-data.ncsa.illinois.edu as that keeps the traffic off the login node and gives other users there a better experience.

SFTP

You can also use conventional command line transfer tools like FileZilla or CyberDuck to upload and/or download data from hal-data machine.

sftp <username>@hal-data.ncsa.illinois.edu
put myfile.dat # upload data file
get myfile.dat # download data file