Distributed training involves using parallelism to distribute the training phase of a deep learning pipeline across multiple computational resources, either to accelerate the training process, access sufficient memory for a very large model, or both. In this topic, we consider the data-parallel training of deep learning models across multiple GPUs, a typical use case that can make effective use of the multiple attached GPU nodes on Frontera at TACC.