What does it take to build a Leadership-Class Computing Facility? What exactly goes into a system designed to meet the biggest challenges of computational researchers? Since more and more cycles are being delivered by GPUs, especially for AI applications, it seems that the ideal building block would be an energy-efficient platform that allows CPUs and GPUs to share their memory seamlessly. Horizon is intended to prove that concept.

Horizon at TACC hosts 1,000 NVIDIA "Grace Blackwell" (GB) nodes, where each node consists of a Grace Blackwell NVL4 server containing four Blackwell GPUs and two Grace CPUs per node. Horizon also hosts 4,752 "Vera Vera" (VV) nodes containing two next-generation Vera CPUs per node. Even though NVIDIA is best known for its GPUs, they have more recently employed the ARM architecture to produce Grace and Vera CPUs as well. The combination of node types in Horizon, featuring NVIDIA CPUs as well as GPUs, is intended to support a wide spectrum of scientific applications—from machine learning to AI to HPC simulations—all at a massive scale.

There are two GB200 Grace Blackwell Superchips in the NVL4, each with 2 GPUs and 1 CPU. Each Grace processor holds 240 GB of usable LPDDR5X memory with up to 0.5 TB/s of bandwidth, while each Blackwell GPU has 186 GB of usable HBM3e memory with up to 8.0 TB/s of bandwidth. NVLink connects the chip types in the GB200 at speeds up to 0.9 TB/s; between GPUs, this doubles to 1.8 TB/s. Given these memory and interconnect properties, the GB200 becomes a very effective implementation of NVIDIA's Unified Memory technology, in which memory transfers are managed automatically so that CPU and GPU threads can transparently access both CPU and GPU memory.

The per-node compute performance is excellent, too, as one would expect. A GB node provides 160 TFlops of FP64 performance (~320 TFlops DGEMM performance using NVML) and 40 PFlops of FP16 performance for ML workflows. Each Vera-Vera node should provide over 13 TFlops of double precision performance overall, together with memory bandwidth of 1.2 TB/s per CPU, where each CPU is equipped with LPDDR5X memory.

The interconnect for Horizon's nodes is an NVIDIA InfiniBand XDR 800 Gb/s fabric that uses a fully connected two-level fat-tree topology. Login nodes are Grace Grace (GG) nodes that are compatible with the NVIDIA and GNU software stacks installed for the GB and VV nodes. Horizon is built from Dell Integrated Rack Scalable Systems and liquid-cooled Dell PowerEdge servers. Its 400 PB of shared storage is manufactured by VAST Data.

For a more complete discussion of the capabilities of NVIDIA GPUs, see the Understanding GPU Architecture roadmap.