Blocking communication in MPI involves calls that do not return until the operation is completed. When a process calls a blocking communication routine like MPI_Send, MPI_Recv, or MPI_Bcast, it will block or stop execution and wait for the operation to finish before continuing. Control is not returned to the calling process until the data is safely received by the destination process or processes. This synchronous mode of communication is simplest to program as the calling process is guaranteed the operation has finished when the call returns. It can limit parallelism as the calling process is unable to do useful work while waiting for the communication to finish.
Nonblocking communication allows overlapping of computation and communication. MPI routines prefixed with MPI_I (for initiate) are nonblocking in nature. For example, MPI_Isend, MPI_Irecv, MPI_Ibcast. When a process calls a nonblocking routine, it initiates the communication but does not wait for it to complete. Control is returned immediately to the calling process allowing it to do useful work like computations while the communication proceeds in the background. The process needs to later check if the nonblocking operation has finished through a test call like MPI_Test or MPI_Wait.
There are some key differences in how blocking and nonblocking communication functions:
With blocking sends like MPI_Send, the send buffer can be reused or changed immediately after the call returns as the data is no longer needed. But for nonblocking sends with MPI_Isend, the send buffer must remain unchanged until the send completes, which is checked by a subsequent test/wait call. Similarly, for blocking receives with MPI_Recv, the receive buffer is immediately available for reuse after the call returns containing the received data. But for nonblocking receives with MPI_Irecv, the receive buffer contents are undefined until the receive finishes as checked by a later test/wait.
Blocking communication provides simpler code as it automatically synchronizes the processes involved. But nonblocking offers more flexibility to overlap communication with useful computation. For example, in a blocking broadcast with MPI_Bcast, all processes must enter the call before any process can return. But with nonblocking MPI_Ibcast, the root can initiate sends to all others and then start useful work while the receives proceed asynchronously on other processes.
The performance benefits of nonblocking communication come from two aspects – overlapping communication and computation allows hiding communication latency, and also pipelining of communication where subsequent nonblocking operations can be initiated without waiting for previous ones to finish. Nonblocking code is more complex due to need for explicit synchronization using test/wait calls as well as rules regarding modified send/receive buffers. It also requires careful programming to avoid deadlocks caused by unexpected message arrivals.
In summary:
Blocking communication synchronizes processes, simpler to program but limits parallelism
Nonblocking overlaps communication and computation for improved performance
Blocking allows reusing buffers after call, nonblocking needs buffers unchanged until test/wait
Blocking automatically synchronizes processes, nonblocking requires explicit synchronization
Nonblocking code is more complicated and prone to deadlocks if not implemented carefully
The choice between blocking and nonblocking depends on the communication characteristics and computation pattern in the parallel algorithm. Applications with intensive computation are good candidates for using nonblocking communication to overlap communication wait times with useful work. Applications with frequent small messages may also benefit more from nonblocking to pipeline communications. While applications with infrequent large synchronized communication phases are well suited for the simpler blocking approach. A hybrid scheme also works by using nonblocking communication within a parallel section and synchronizing at the end with blocking calls.
Andrew Stroud