Rclone is a mature, free command line program for performing data transfers and managing remote files. It combines the security and copying functionality of SCP with the remote file management of SFTP and the syncing ability of rsync. Plus, it comes with features to help you connect to data in many cloud storage platforms.

Installing Rclone

Rclone can be downloaded and installed on Windows, Mac and Linux systems. Note that some of the provided downloads contain true "installers" (the .deb and .rpm options). These installers place the software into a system-wide location and can only be run by those with administrator privileges.

The other downloads merely provide a folder containing the rclone executable and some documentation. You can place these folders anywhere on your system and they do not require administrator privileges. Users of high performance computing systems like TACC's Stampede2 and Frontera will want to use this second option and place the downloaded folder in their home directory.

Configuring Remotes

Rclone can transfer data between your local system and a remote system, or between two remote systems. Both the local and remote systems must be configured for use by Rclone before you can use them. You can edit the remote configuration file yourself, or use the interactive "config" subcommand to guide you through the process. This interactive configuration is described in detail for each type of provider and is recommended for new users. Begin configuring by running the configuration sub-command:

rclone config

The command will first ask you the name and type of your provider, then step through the process of setting a number of configuration properties that are specific to that provider. For each property, a set of choices will be offered, and in many cases choosing the default value will suffice. Once a configuration has been created, it can be edited using this same command.

SFTP Remotes

To replicate the functionality of SCP, SFTP and rsync for transfers to a remote system using SSH, you will want to create an SFTP provider. Since there are three ways that you can provide your SSH credentials for use with the remote system, the config command will ask you about all three. You should probably only configure one of the ways, which are:

Password

Setting this value stores your login password for the remote system in the configuration. If you want to enter your password each time it is needed, you can instead run rclone with the option --sftp-ask-password.

Key File

Instead of a password, rclone can provide your private key when authenticating. You have two choices here - either paste the text of your key when config command asks you to set property key_pem, or provide the location of your PEM key file when it asks you to set property key_file. If you provide the key text, be sure to remove all carriage returns from it so it is on a single line.

SSH Agent

If you are running an SSH key agent on your system, you can set property key_use_agent to force rclone to ask the agent for your SSH key. If you haven't set up a key agent (such Pageant) on your system, this option will not be available to you.

Configurations for remote systems that use two factor authentication will probably not work with Rclone. After your password or private key is supplied to the remote system, it may want to prompt you to enter a token. This behavior is not currently supported by Rclone, and in these situations your best option is to install Rclone locally on the remote system and initiate your data transfers from there. This advice pertains to the TACC systems Stampede2 and Frontera.

Cloud-Based Remotes

Configuring a cloud-based remote, such as for Amazon Web Service (AWS) S3, will require setting some completely different properties. One of the first properties will choose between the various providers that can supply access to the cloud platform you have chosen. Selecting the cloud's own provider is a good default choice. Then you must also set some properties to provide your credentials to the cloud service. For AWS, this may take the form of setting the access_key_id and secret_access_key properties using values you obtain through the AWS S3 web site.

It is important to note that retrieving content listings or transferring data from a cloud storage service will incur the same costs through Rclone as you would expect to pay when using some other method.

Using Rclone

The Rclone documentation contains good examples of using Rclone. It also contains explanations of the many command line options (flags) that are available. It is important to note that Rclone performs operations more like a cloud storage access mechanism than like a Linux command line utility. One example of this is that it copies directories by default, without the need to specify a "recursive" option. Another is that when operating on a directory, its ls command doesn't just list the files and subfolders of that directory. Instead, it lists all of the "objects" found under that directory, with each entry including the object's path.

But Rclone also provides many convenient variants of its commands, with some behaving more like Linux commands and others that are handy when writing scripts. For example, the lsf command lists a directory contents in the way you would expect in Linux. And lsjson returns JSON data containing a directory's contents in an easily parsable form.