OpenMP contains a set of directives that modify the execution of parallel regions. Many of these will be familiar to anyone has done other shared-memory programming. Each begins with the OpenMP sentinel and some take optional modifying clauses as specified in the list below. All of the C directives must be followed by a block of code enclosed in curly braces { … }. Fortran code requires an END <control> directive at the end of the block.

Master and Synchronization Directives

• master
• critical (name)
• barrier
• atomic
• flush (list)
• ordered

master

The master directive precedes a structured block that must be executed by the primary thread of the current innermost nested parallel region. The other threads will bypass this block. There is no implied barrier either at the beginning or the end of a master region. (Note, the master construct is now deprecated; with compilers that support OpenMP 5.1 and later, the new masked construct, minus its filter clause, may be used as a substitute, with no change in behavior. The compilers on Stampede2 and Frontera do not yet support 5.1 features.)

!$OMP MASTER
⋮
!$OMP END MASTER

critical

The critical directive precedes a structured block that must be executed by a single thread at a time. This enables you to avoid having two threads trying to update a shared variable at the same time. This directive applies to all of the threads in the program without regard for nesting. The optional name clause allows you to distinguish critical sections when you have more than one that might be encountered at the same time. When a thread reaches a critical region, it waits until no other thread is executing the critical region with that name. The name of a critical region is a global entity. These names can conflict with the names of other global entities in Fortran, but they are in a separate namespace from labels, tags, members, and ordinary variables in C. The name (if any) in the END CRITICAL directive of a Fortran program must match the original directive.

!$OMP CRITICAL [name]
⋮
!$OMP END CRITICAL [name]

barrier

The barrier directive specifies an explicit barrier at the location where it occurs in the program. The directive doesn't pertain to a particular code block, so it needn't be followed by curly braces in C or an END directive in Fortran; simply place it on its own line so it doesn't invalidate the syntax of the program. The barrier causes each thread in the innermost nested parallel team to wait at that point until all have reached it. It is an error to put the barrier directive at a place in a program where some threads in the team might not encounter it, such as in a master region. A barrier that isn't encountered by any threads is allowable — one might occur in the unused branch of an if-else, for example. Also, all threads must encounter each barrier in the same order, because deadlock would result otherwise.

⋮
!$OMP BARRIER

atomic

The atomic directive specifies that the immediately following assignment statement of a scalar entity will be done as a transaction. In other words, execution of the assignment cannot be initiated by another thread until the assignment in progress is complete. It is most commonly used for update assignments, though clauses are available for other types of assignments. This is a lighter-weight way of avoiding conflict in updating shared variables as compared with using a critical directive. However, only the load and store of the updated variable is atomic, not the evaluation of the updating expression, so the atomic directive must be used with care in cases where the updating expression is complicated. The atomic directive applies to all threads that exist when it is encountered, not just the ones in the innermost parallel region. The atomic directive does not enforce exclusive access with respect to critical and ordered regions where the same variable is accessed.

!$OMP ATOMIC
X = X + 13

flush

OpenMP uses a weak-ordering model of memory consistency. The flush directive requires that the calling thread's view of the listed variables be synchronized with the shared memory so that the internal view of those variables by the thread is consistent with what is in memory. The flush directive applies only to the thread in which it is encountered. Flushes are necessary if the value of a variable is changed by one thread and the changed value of that variable needs to be read in another thread. After the change is made in thread A, that thread has to flush the variable to get the new value into memory, and thread B, which needs to use the variable, must perform a flush after A's flush to be sure that it sees the new value in its internal state before using it.

If a flush operation appears without a list, all variables visible to that thread are flushed. There are numerous situations where an implicit flush operation without a list occurs:

• During a barrier region
• At entry to or exit from a parallel, critical, or ordered region
• At exit from workshare regions, unless nowait is present
• During omp_set_lock and omp_unset_lock regions
• During omp_test_lock, omp_set_nest_lock, omp_unset_nest_lock, and omp_test_nest_lock regions if the region causes the lock's status to be changed

An implicit flush operation is implied at the entry and exit from an atomic region with a list containing only the object being updated.

!$OMP FLUSH X,Y

ordered

An ordered directive specifies a loop region where the code will be executed in the order of the loop iterations. The ordered directive binds to the threads of the innermost set of nested parallel regions. When the threads in a loop encounter an ordered directive, the thread executing the first iteration of the loop can go ahead, but other threads must wait their turn to execute the code in the region. The ordered directive may not be used in an OpenMP loop that doesn't have an ordered clause in its declaration.

!$OMP PARALLEL DO ORDERED[,clause[,clause]]
⋮
!$OMP ORDERED
⋮
!$OMP END ORDERED
⋮
!$OMP END PARALLEL DO [NOWAIT]