To compile code with OpenMP offloading to NVIDIA GPUs, you should include both the standard -fopenmp flag and a compiler-specific flag. TACC recommends the following:

The above flags for GCC only work only if NVIDIA GPU support was enabled at the time that GCC was installed. You can check this with the command gcc -v 2>&1 | grep "offload". If NVIDIA support was enabled, you should see --enable-offload-targets=nvptx-none included in the output. At TACC, this is true for all GCC versions on Horizon and Vista, but not on older clusters.

OMP_TARGET_OFFLOAD

The OMP_TARGET_OFFLOAD environment variable controls what happens when a target region is encountered at runtime. It accepts three values: default, disabled, and mandatory. The default behavior of OpenMP is to offload to the device if one is available, and fall back to the host otherwise. This behavior can be explicitly defined with OMP_TARGET_OFFLOAD=default. If it is set to disabled, it runs the target region on the host, and if it is set to mandatory, it aborts the program if offloading is not possible.