MPI, the Message Passing Interface, is the standard way that parallel programs communicate across multiple compute nodes. Apptainer recognizes two different models of using MPI with containers: the hybrid model and the bind model. In this topic, we briefly cover the two models and what is required to run MPI applications in a container.

Hybrid Model

In the hybrid model, apptainer is run under the host's MPI launcher, such as ibrun at TACC, which works together with the MPI runtime inside the container. The same MPI implementation must be present in two places: inside the container alongside your application, and on the host system, where it is maintained by the system administrators and loaded through a module (e.g. TACC). On the host, the launcher starts the apptainer commands, each of which runs the containerized MPI application.

For the two MPI implementations to cooperate, the container's implementation must match the host's implementation and must be the same version or newer (and preferably the two match exactly). MPI applications must also be built to support the same interconnect that is present on the host, such as InfiniBand or Omni-Path, and the same process-management interface, such as PMI2 or PMIx.

Bind Model

The bind model is more flexible in that the container is not required to have MPI installed. Instead, the host's MPI is mounted into the container at run time with the --bind option. Again, the host's launcher starts one container instance per rank, where the apptainer command further specifies how to run the desired MPI application. In this case, the MPI application must either be ABI (Application Binary Interface) compatible with the host's MPI implementation, or it must have been rebuilt inside the container (or perhaps on the host itself) using the host's MPI implementation, as provided through --bind.

Note that it is necessary to alter the container's LD_LIBRARY_PATH environment variable so the application will find the bind-mounted host libraries preferentially when it starts.

If you are running a containerized Python application that makes use of mpi4py, then you should ensure that the container's mpi4py package is linked against the host's MPI implementation. Usually, an mpi4py package that was installed with pip or conda is accompanied by an MPI implementation that was installed as a dependency, and this MPI may not be compatible with the MPI on the host cluster. At TACC, a suitably linked mpi4py library is present in any of the python3_mpi modules (which also include CUDA); however, it may be tricky to get a TACC-built mpi4py library to take precedence over one that is pre-installed in the container. In the end, you may need to create your own virtual environment to pull in a custom (or copied) build of mpi4py that is linked properly. The way to do this is covered on the next page.