When you send an MPI message, the MPI runtime code handles the details of opening an appropriate network connection with the remote task and moving the bytes of data from the buffer you have designated in the send call into an appropriate buffer at the receiving end. MPI provides a variety of send and receive calls in its interface. These calls can be classified into one of two groups: those that cause a task to pause while it waits for messages to be sent and received, and those that do not.

The call with the most straightforward name, MPI_Send(), uses blocking communication. In other words, the function call does not return control to your program until the data you are sending have been copied out of your buffer. The advantage of blocking communication is that you can modify the data in that buffer after the call returns without concern that it will affect what has been sent. The disadvantage is that if no destination is available for copying the data, your program will be stuck at that point until a place to put the data becomes available.

On the receiving end, a call to MPI_Recv() — or a variant — provides the ultimate destination for the data in an MPI_Send() call. If the receiving task is running more slowly than the sender, the sender will wait because the matching receive will not have been posted yet. This is not a good situation in general because the process that is running the sending task will sit idle for a while.

Many implementations of MPI get around this problem in the case of small messages by caching unmatched messages in a buffer on the receiving task until the matching receive is posted. Caching allows the blocking send call to return and for the sending program to continue without waiting even if the receiving program isn't ready to receive yet. The maximum size and number of messages that the "eager" send mechanism can cache varies among MPI implementations, but the buffer is usually limited.

An alternative is the non-blocking send MPI_Isend(). This call always returns quickly but gives the sending program a handle to a request object that reports the data transfer status. While there is never any delay for the sender at this call, the program is honor-bound not to modify the data in the buffer without checking the request handle to verify that the message has actually been sent. If the sender ignores this contract, then the data sent to the receiver may depend on exactly when the data was copied from the sender's memory. Thus, when you use non-blocking communications, extra programming is necessary to keep track of outstanding requests and to call MPI_Test() or MPI_Wait() as needed to check their status.

There are similar considerations on the receiving side. A call to MPI_Recv() will not return until the buffer contains the requested data. The MPI implementation cannot help you here because it has no way to eagerly receive data that haven't been sent yet! A call to MPI_Irecv() returns a request object that can be used later to determine when the data are in the buffer.