Compute Nodes
The following tables compare the compute nodes on Horizon to those on Vista and Frontera, the predecessors to Horizon. All three systems are supported by the National Science Foundation, and they all are intended to enable large-scale computational research in a wide variety of fields. Horizon serves as the centerpiece of the Leadership-Class Computing Facility, so its NVIDIA processors have overall better individual capabilities than those in Vista and Frontera.
The bigger contrast, though, comes in the sheer number of processor cores. Like Frontera before it, size is really what distinguishes Horizon as a leadership-class computer system. Horizon is configured to have a million cores in its CPU subsystem—and yet, the 4,000 Blackwell GPUs in its GPU subsystem collectively outperform all of its Vera CPUs in doing FP64 calculations!
The first table below emphasizes the numbers and key properties of CPUs found in representative CPU compute nodes of each system.
| Horizon1 | Vista2 | Frontera3 | |
|---|---|---|---|
| Processor Type in CPU Compute Nodes |
NVIDIA Vera ("Olympus", ARM v9.2) |
NVIDIA Grace (ARM Neoverse v2) |
Intel Xeon Platinum 8280 ("Cascade Lake") |
| Total CPU Compute Nodes | 4,752 | 256 | 8,368 |
| Cores/Node | 176 (88 cores/processor with 2 processors) |
144 (72 cores/processor with 2 processors) |
56 (28 cores/processor with 2 processors) |
| Hardware Threads/Core4 | 2 | 1 | 1 |
| Clock Rate5 | likely 3.2–3.6 GHz (TBA) |
3.4 GHz | 2.7 GHz Max Turbo: 4.0 GHz |
| Memory per Processor | ? GB LPDDR5X | 120 GB LPDDR5X | 96 GB DDR4-2933 |
| Peak Memory Bandwidth per Processor |
1.2 TB/s | 512 GB/s | 141 GB/s |
| L1 Cache | 96 KB per core | 64 KB per core | 32 KB per core |
| L2 Cache | 2 MB per core | 1 MB per core | 1 MB per core |
| L3 Cache | 164 MB per processor | 114 MB per processor | 38.5 MB per socket |
| Local Storage (/tmp) | ? GB | 286 GB partition | 144 GB partition on 240 GB SSD |
The second table compares the GPUs found in the GPU compute nodes in the same three systems. Because NVIDIA GPU architectures have advanced significantly across generations, some precision types and features are only available on newer hardware, as indicated by N/A entries and in the notes. All performance values are per GPU.
| Horizon1 | Vista2 | Frontera3 | |
|---|---|---|---|
| GPU Type in GPU Compute Nodes |
NVIDIA Blackwell B200 (in GB200) |
NVIDIA Hopper H100 (in GH200) |
NVIDIA Quadro RTX 5000 (Turing) |
| FP4 Tensor Core6 | 20 Pflop/s | N/A | N/A |
| FP8/FP6 Tensor Core6 | 10 Pflop/s | ~4 Pflop/s | N/A |
| INT8 Tensor Core6 | 10 Pop/s | ~4 Pop/s | 178.4 Top/s |
| FP16/BF16 Tensor Core6 | 5 Pflop/s | ~2 Pflop/s | 89.2 Tflop/s |
| TF32 Tensor Core | 2.5 Pflop/s | ~1 Pflop/s | N/A |
| FP32 | 80 Tflop/s | 67 Tflop/s | 11.2 Tflop/s |
| FP64 | 40 Tflop/s | 34 Tflop/s | ~0.35 Tflop/s7 |
| FP64 Tensor Core | 40 Tflop/s | 67 Tflop/s | N/A |
| GPU Memory | 186 GB HBM3e | 96 GB HBM3 | 16 GB GDDR6 |
| GPU Peak Memory Bandwidth | 8 TB/s | 4 TB/s | 448 GB/s |
| NVLink Bandwidth | 3.6 TB/s | 900 GB/s | 50 GB/s8 |
1 Horizon User Guide: System Architecture and NVIDIA GB200 NVL72 spec page (plus other sources). Bear in mind that each GB200 includes two GPUs. Vista also has some GB200 NVL4 nodes with Blackwell GPUs.
2 Vista User Guide: System Architecture, NVIDIA Grace CPU Superchip, and NVIDIA Grace Hopper GH200 Superchip datasheet. Note, the GB200 model in Vista has the H100 GPU. NVIDIA also makes a GB200 model with the H200 GPU, which has identical specs except for memory (the H100 has HBM3, the H200 has HBM3e).
3 Frontera User Guide: System Architecture and NVIDIA Quadro RTX 5000 specifications.
4 Frontera did not have hyperthreading enabled, and Vista's Grace processors are not capable of it; Horizon's Vera processors support two hardware threads per core.
5 Max Turbo applies when only 1 core is active; the Turbo Boost rate declines if more cores are active and/or AVX instructions are used.
6 Tensor Core performance values for FP4, FP8/FP6, INT8, and FP16/BF16 rows include the effect of structured sparsity, which can double throughput over dense operation for eligible workloads. The Quadro RTX 5000 (Turing) does not support structured sparsity, nor does it support BF16 in its tensor cores. Hopper does not support FP6.
7 FP64 performance is not listed in the NVIDIA Quadro RTX 5000 datasheet; the value is estimated from the Turing architecture's characteristic 1/32 FP64-to-FP32 ratio.
8 The Quadro RTX 5000's NVLink is an optional workstation add-on for connecting two cards and is not a datacenter-grade interconnect comparable to the NVLink on Horizon or Vista.
CVW material development is supported by NSF OAC awards 1854828, 2321040, 2323116 (UT Austin) and 2005506 (Indiana University)