In the previous parts, we learned that training deep learning models with multiple GPUs is critical as the number of parameters for neural networks grows. The same logic of using more compute resources to speed things up motivates us to scale further up beyond resources on one node or one machine.

Before we learn about multi-node training, let’s first review some notions from the previous part of the tutorial.