Tips for Speed
Ben Trumbore, Adam Brazier, Linda Woodard (original author)
Cornell Center for Advanced Computing
Revisions: 11/2020, 2/2015, 7/2013 (original)
Once you are familiar with all of the data transfer tools at your disposal and you understand your requirements and priorities for a particular data transfer, you can make choices that will impact the time it takes to perform your data transfer. These choices involve:
- Choosing the transfer tool you will use
- Determining whether you should consolidate your files and compress your data
- Determining whether you should sync your data and verify the transfer
- Some additional considerations
Objectives
After you complete this segment, you should be able to:
- List important considerations when choosing a data transfer utility
- Explain the purpose of creating a tar file
- Distinguish between synching and copying
- Explain how to verify transferred files
- List variables that can affect transfer speeds
Prerequisites
- Have some familiarity with the file systems and access requirements for high performance computing environments. Example environments would be those of Stampede2 and Frontera, described in the Stampede2 Environment and the Getting Started on Frontera topics. In particular, see the sections on "Storing and Moving Data" for Stampede2 and Frontera, respectively.