If your data fits on a single node's memory, it is relatively straightforward to introduce shared memory parallelism into existing serial code. The easiest way to exploit shared memory parallelism is to insert OpenMP directives into your code to execute specific loops in parallel. One advantage of using OpenMP is that parallel code may still be compilable as a serial code; unless the compiler is instructed to honor the OpenMP directives, it will ignore them and produce a serial program. It is also common to explicitly use threads to create a new shared memory parallel program based on an existing serial program. If you choose to create and manage threads explicitly, ensure that access to shared data from different threads is appropriately synchronized.

Distributed memory parallel programs are much more difficult to write as simple modifications of serial programs. In distributed memory programs, often only one or a few tasks will be doing tasks responsible for I/O may need to distribute and collect data from the other tasks.

Another major concern with distributed memory programs is global data. While fixed-value global objects can be replicated across all tasks, the program must redistribute any updates to mutable global data values to all tasks that use those values. Code that depends on globally updated data (e.g., time-stepping) typically reevaluates global values at the end of each time step and broadcasts the results to all tasks before the next time step begins.