At this point, reviewing program design for parallel programming may help you decide whether you should begin with a serial program or design a parallel program from scratch.

1. Prepare a serial version of your program:
   • If you already have a serial version of your program, you can modify it to use MPI.
   • If you are writing an MPI program from scratch, and it's not much extra work to write a serial version first (without MPI calls), then you should do that.
2. Make sure the serial code is designed in such a way that you can later take advantage of any parallelism inherent in it. For instance, you may want to adopt an "owner computes" rule by dividing the data into smaller pieces that can be processed independently. Ask yourself: Are there large arrays that can be broken down into sub-arrays? Is there a large search space that you can subdivide over processes?
3. Make sure your serial version is thoroughly debugged before going parallel. This will make debugging your parallel version much easier.
4. Add calls to MPI routines in the appropriate places in your program. Transitioning serial code to MPI is a process of identifying where the parallel treatment of subproblems requires exchanging data and adding the appropriate MPI commands to set up communication and data exchange.
5. Always scale up your MPI program gradually:
   • Ensure it works on 2, 4, and 8 CPUs before you move to 128.
   • This will reduce the amount of machine time you waste finding bugs and give you the opportunity to make sure your parallelization scheme is working as you expect.
   • During your initial testing, run your job on a small set of CPUs to ensure that your test job gets scheduled and executed quickly.