The essence of computing is instructions operating on data. Parallelization can be achieved by using multiple instruction streams and/or multiple data streams. This insight is formalized in Flynn's taxonomy (1966), which classifies different types of parallel computing architectures.

The upper-left corner corresponds to the simplest computer, in which a single instruction is executed at a time, and data elements are also dealt with one at a time. The acronym for this is SISD, meaning "Single Instruction Single Data."

In the upper-right corner, we have Single Instruction Multiple Data (SIMD) which is exemplified by a vector processor. A single core of a modern processor is basically SISD, but it incorporates some vector processors on the chip accessible via special instructions. Thus, although the computer as a whole may not be SIMD, using compiler flags that exploit on-chip vector processors is an important part of performance optimization.

In the lower-left corner, we have Multiple Instruction Single Data (MISD). This is the least frequently seen type of computer. One example of a MISD processor is a set of digital filters that operate in parallel on a stream of data.

The lower-right corner, Multiple Instruction Multiple Data (MIMD) includes cluster computers, which consist of separate nodes that can operate independently but are connected by a high speed/high capacity communication network so they can be made to work together effectively. Moreover, any one of the modern multi-core processors typically found within a node is essentially a cluster on a chip and may be considered a MIMD computer.

Although the nodes of a cluster operate independently and one could run a different executable on every core of every node in a parallel job, users typically organize their code into a single executable. This programming technique is called SPMD for "Single Program Multiple Data." Although the acronym looks like those that describe computer architectures, it is actually a programming technique. The execution of this program differs from a SIMD program in that different instances of it run independently (even if they're on the same node) and are in sync only to the extent that is imposed by blocking communications. It is common practice in SPMD programs to have conditionals that cause some sections of the program to be executed by some instances and not by others.