Below is a different example of a hybrid code that requires MPI_THREAD_FUNNELED support or higher. At this level of MPI thread support, only the main thread may make MPI calls (i.e., MPI calls must be "funneled" through the main thread).

Example of MPI_THREAD_FUNNELED: thread selected by omp master calls MPI_Send and MPI_Recv
With the MPI_THREAD_FUNNELED level of thread support,
only the main thread can call MPI

Some notes on this program:

  • In OpenMP, the main thread can be selected by the omp master construct
  • This will work, provided that the current thread team includes the main thread of the whole program
  • The omp master construct does not create implicit barriers for the other threads, so it may also be necessary to use omp barrier
  • As illustrated in the example, barriers may be required in two places:
    • Before the MPI call, in case the MPI call needs to wait on the results of other threads
    • After the MPI call, in case other threads immediately need the results of the MPI call
  • The example shows the main thread executing a single MPI call, while the other threads are idle

Note, the main thread can also be selected by testing the result of a call to MPI_Is_thread_main(). This test is appropriate if MPI_Init_thread() was called from an arbitrary thread, say the one chosen by omp single in a multithreaded region. The thread selected by this test is the main thread from the point of view of MPI, even if it is not the main thread from the point of view of OpenMP. Similarly, the MPI_Is_thread_main() test is appropriate inside two or more nested omp parallel constructs, because in that case there is not a single, unique OpenMP main thread. Calling MPI_Is_thread_main() is always thread-safe.

 
©  |   Cornell University    |   Center for Advanced Computing    |   Copyright Statement    |   Inclusivity Statement