Scalability and Speedup
Steve Lantz, Andrew Dolgert (original)
Cornell Center for Advanced Computing
Revisions: 10/2023, 10/2014, 3/2012 (original)
To justify running your application on a major HPC resource like Frontera, you need to demonstrate that your application will scale up to use a significant fraction of the resource's capability—and use it efficiently. This generally involves making an extrapolation based on a plausible model of the application's performance as the number of processing elements increases. Such a model can be developed from known principles of parallel computing combined with relevant benchmarks. This topic explains how to put together the ingredients to produce a model of this type.
Objectives
After you complete this topic, you should be able to:
- Define what is meant by linear speedup
- Explain why super-linear speedup is sometimes possible, even though sub-linear speedup is what is usually observed
- List ways to define scalability
- Explain how Amdahl's Law limits strong scaling
- Define the term weak scaling
- Describe how Gustafson's Law quantifies scalability, and how it differs from Amdahl's Law
- Define the term trivially parallelizable
- Distinguish between capacity and capability, in relation to HPC resources
- Demonstrate estimating an application's scalability by running benchmarks
- List three types of benchmarks to use when an application is under development
- Define the term parallel overhead
- Explain how a speedup could become a slowdown
- Explain how an an isoefficiency measure can influence optimum core count
- Define the term latency hiding
Prerequisites
There are no prerequisites for this topic. However, before you attempt to optimize your code for any particular HPC system, you should become familiar with the specific architecture and hardware characteristics of that system. You should also be aware of profiling and debugging tools that are available for it.
In the case of Frontera at TACC, good references would be the Virtual Workshop roadmaps on Parallel Programming Concepts and Vectorization. But it is not necessary to complete the roadmaps on those topics before diving into this one.