MPI_Wait

MPI_Wait is a blocking call that returns only when a specified operation has been completed (e.g., the send buffer is safe to access). This call should be inserted at the point where the next section of code depends on the buffer because it forces the process to block until the buffer is ready. Intuitively, then, the following code is equivalent to a blocking (MPI_Send) call:

Inserting the Wait call immediately after the Isend call emulates a blocking send, and the extra call doesn't even add very much overhead.

However, it's easy to see that by further separating these two operations, we permit the sending process to continue doing work in places where a blocking call might merely be waiting:

We see that because we decoupled the send and the wait, we are now able to perform computation in between. Of course, we have to be careful with our newfound power—are the contents of the message safe, in the example above?

MPI_Test

MPI_Test is the nonblocking counterpart to MPI_Wait. Instead of blocking until the specified message is complete, this function returns immediately with a flag that says whether the requested message is complete (true) or not (false). MPI_Test is basically a safe polling mechanism, and this means we can again emulate blocking behavior by executing MPI_Test inside of a while-loop.

More practically, MPI_Test permits an event-driven style of programming, in which events are fired whenever a MPI_Test comes back true. This is particularly useful on the receiving side. A receiver can post a nonblocking receive early, then periodically check the status of the receive with MPI_Test. When the buffer becomes available, the content can be acted on, temporarily delaying other processing. This type of strategy makes it easier on the senders as well: there should be a low synchronization cost to communicate with the process. The usual caveats about careful use of buffers (or perhaps better, non-use) apply here.

MPI_Test should not be confused with the similar-sounding MPI_Probe, which is covered in a subsequent section.

MPI_Request_free

This call simply nulls out the request handle and deletes the associated request object. (If communication is already in progress, the request object is deleted after it completes.) Often, MPI_Request_free serves as a companion to MPI_Cancel. Why? A call to MPI_Cancel is just the first step in terminating a pending, nonblocking send or receive operation. It must be completed, and MPI_Request_free provides a natural way to do so.

What's the reason for terminating a communication like this? One possible scenario is that the send or receive was issued speculatively, but later it may be determined that the communication will never occur. MPI_Cancel ensures that any resources reserved for the speculative operation, such as buffers, etc., are released for future use.

The cancel call is entirely local, however. It returns immediately, possibly before the communication is really canceled. It is still necessary to complete the cancel operation, and ultimately, to free the request object. This can be achieved with a call to MPI_Request_free, like this:

The above assumes the MPI_Cancel will eventually complete successfully. If you want to trap any errors in the cancellation, then you must use MPI_Wait or MPI_Test (or some variant) instead.