In the absence of a well-designed suite of software, parallel I/O would be hard to coordinate and optimize. It would place a tremendous burden on an applications programmer to have to choreograph all the required data movement. But over time, the solution that has emerged is to have I/O specialists implement a number of intermediate layers. That way, the application developer only has to deal with a high-level interface. It is built on top of a software stack that goes right down to the hardware level.

The layers work roughly as follows:

• The high-level I/O library maps application abstractions to a structured, portable file format (e.g., HDF5, Parallel netCDF).
• The middleware layer deals with organizing access by many processes (e.g., MPI-IO, UPC-IO).
• The parallel file system maintains logical file space and provides efficient access to data (e.g., Lustre, PVFS, GPFS).

Other topics in this roadmap address the specifics of the Lustre parallel file system and the MPI-IO middleware layer. Two of the more popular high-level libraries are described in the Parallel I/O Libraries roadmap.