In this exercise we will try out transferring files from your personal computer (PC) to Horizon using command-line tools. These tools are already present in Linux and Mac systems; on Windows, you may want to install WSL to get the same functionality or use the PowerShell application on any modern Windows computer. First we will go over a few simple examples in which a couple of tools are used on individual files, and then a more detailed example to be used for larger files and directories.

A good place to start is to check your allocation and storage usage. Log in to Horizon and try /usr/local/etc/taccinfo (or simply view its output, which appears automatically upon login). Examine the information there, especially the "Disk quotas for user" section. You'll want to remain logged in so you can complete some of the steps in this exercise.

First, let's get the data we're going to transfer set up. We have some small example files for you to use that you can download, which are the words of Rev. Dr. Martin Luther King Jr. in his famous "I Have a Dream" speech. Make a new directory in your home directory on your PC named speech and download part1.txt, part2.txt, and part3.txt into this directory. Use cat or a text editor to view the contents if you wish to become familiar with what you are transferring.

The next step is to prepare a directory on Horizon to receive the files you will be transferring:

Now, from the PC, transfer a single file to the new $SCRATCH/speech directory on Horizon with scp (note, the commands to be executed on your PC will be prefaced with the prompt [myPC]$):

Remember to replace <username> with your username and <number> with the number in your $SCRATCH path. Be aware that TACC multi-factor authentication (MFA) will be required for each transfer operation. Subsequently, you may verify the arrival of the file on Horizon using ls, and inspect its contents with cat if you like.

Next, move the second file using rsync (note,

Note that you can use Horizon environment variables in the path within the commands by putting a \ in front of them; depending on your local OS, you may also need to add a flag, such as scp -O or rsync --old-args, to get the variables to expand properly on Horizon.

Since rsync only transfers updates or changes, you can transfer the remaining file by using rsync to synchronize the directories as follows:

You may wish to further explore the flags for rsync as well.

The above example is fine for small files, and small numbers of files. Otherwise, you should use compression, so let's walk through it. Let's imagine each file in our directory is large (though these example files are small), and that the directory has many files (even though we will only work with these 3). So, our first step would be to compress the directory, for which we will use tar and gzip (via the tar -z flag). On your PC:

Next, we'll make a new directory called "Test" under $SCRATCH on Horizon (login first):

Note that this is just an example, but TACC's rule of thumb for Horizon is "allow at least one stripe for each 100GB in the file" so adjust accordingly. Now, on your PC:

On Horizon, you can reverse the compression (from the Test directory):

$ cds
$ cd Test
$ tar -xzf civil_rights.tar.gz

You should be able to see all 3 parts of the famous speech in the uncompressed directory. This second method can be used for any single large file, even one that did not begin as a directory. If you are curious, you can check /usr/local/etc/taccinfo to see how your storage usage has changed.

To clean up when you are done: