TAU is able to profile any executable that has been appropriately instrumented at compilation time. The TAU distribution provides scripts to instrument and compile code in various languages, such as tau_f90.sh, tau_cc.sh, and tau_cxx.sh.

In this exercise, we will compile a simple C MPI program with tau_cc.sh and profile it using some of TAU's many capabilities. Our example code, matmult.c, performs distributed matrix multiplication via MPI. Before instrumenting or compiling code with TAU, be sure that TAU is available in your environment. On Frontera, that is achieved by loading the TAU module,

The easiest way to profile an instrumented program with TAU is to use TAU's MPI wrapper execution system. On Frontera, this can be done by inserting tauex in between ibrun and the program filename. Thus, ibrun ./matmult becomes

On Frontera, you can run the job in a non-interactive queue by creating a job script job.sh like the following.

Submit the job and wait for it to complete:

When the MPI run is complete, directories containing a number of profile.#.#.# files will be generated. These directories are named according to the metrics identified in the TAU_METRICS environment variable. At TACC, that variable is preset for you, as explained in module spider tau/2.28.2. Let us look at the files in one of the directories:

The profiling information can be viewed with pprof:

In addition to the text interface of pprof, TAU offers a GUI tool called ParaProf for analyzing the run output. See man paraprof for how to get started.