Collective operation

Collective operations are standard building blocks for structuring the communication and interaction patterns used inside SPMD, single program multiple data, algorithms in parallel programming. Rather than every process in a parallel computation exchanging data through ad hoc point to point messages, a collective operation defines a common pattern, such as distributing data from one process to many or combining data from many processes into one, that many parallel algorithms rely on repeatedly. Because these patterns recur so often, there is a strong incentive to implement them efficiently rather than leaving each program to reinvent them. The Message Passing Interface, MPI, is a widely used standard that provides working implementations of these collective operations for parallel programs to call directly.