MPI provides both blocking and nonblocking point-to-point communication:

• Blocking communication means that the process waits to ensure the message data have achieved a particular state before processing can continue.
• Nonblocking communication means that the processor merely requests to start an operation and continues processing.

Nonblocking calls merely initiate the communication process. The status of the data transfer, and the success of the communication, must be verified at a later point in the program. The purpose of a nonblocking send is mostly to notify the system of the existence of an outgoing message: the actual transfer might take place later. It is up to the programmer to keep the send buffer intact until it can be verified that the message has actually been copied someplace else. Likewise, a nonblocking receive signals the system that a buffer is prepared for an incoming message, without waiting for the actual data to arrive. In either case, before trusting the contents of the message buffer, the programmer must check its status using the MPI_Wait or MPI_Test functions. We'll cover those methods shortly, but for now, just remember that the programmer is responsible for using API functions to query the status of any message passed, prior to accessing or altering data in the message buffer.

It is fairly intuitive why you need to check the status of a nonblocking receive: you probably want the message, not an empty buffer. It is less obvious why you would want to check the status of a nonblocking send. There are three good reasons why you shouldn't think of a nonblocking send as a fire-and-forget mechanism. The first is, as discussed above, if you ever wish to re-use the buffer, you'll need to determine when it is safe to do so. Secondly, even if the send buffer is never re-used, you will want to release whatever system resources are involved, which is really just a different form of buffer reuse. Lastly, it can be important to know whether your send operation has completed successfully, which of course requires a query of some sort.