When deciding which communication mode to use, the main consideration is overhead, that is to say, how much time your process will spend waiting for the blocking send (or receive) to return. Each mode has different overhead characteristics. In order to tighten our meaning, it will be useful to define the overhead as having two sources:

1. System overhead - the time spent transferring buffer contents
2. Synchronization overhead - the time spent waiting for another process

We can classify all of the time spent waiting as either system or synchronization overhead.

System overhead is incurred from transferring the message data from the sender's message buffer onto the network (directly or indirectly), and from transferring the message data from the network into the receiver's message buffer. It's worth noting that this overhead does not generally encompass waiting for the message to be received, just for the sender's buffer to be clear.

Synchronization overhead is the time spent waiting for an event to occur on another task. In certain modes, the sender must wait for the receive to be executed and for the handshake to arrive before the message can be transferred. The receiver also incurs some synchronization overhead in waiting for the handshake to complete. Synchronization overhead can be significant, not surprisingly, in synchronous mode. As we shall see, the other modes try different strategies for reducing this overhead.

This means that the speed of an MPI program is dependent on good network connections (low system overhead) and intelligent programming (low synchronization overhead). Generally speaking, MPI communications operate in what is known as the rendezvous protocol, which involves a handshake procedure in order to establish communication. Next, we'll explore the 4 different communication modes that MPI provides, all of which perform slight modifications of the generalized procedure shown here.