HPC Ecosystem
Most researchers do not have direct institutional access to a high performance cluster, and even fewer are at an institution with a flagship supercomputer. To make HPC accessible to researchers, universities partnered with the National Science Foundation to establish cross-institutional groups to manage collaboration and access to supercomputing systems. These cross-institutional groups grew out of earlier efforts to develop grid computing, where computations are distributed among multiple sites and executed concurrently.
These organizations establish common standards for user authentication, develop fair use guidelines and grant computing allocations for eligible researchers. Though individual provider organizations select and manage their own cluster hardware, they employ a fairly standard software stack to manage remote system access, job submission, parallel computation, and file transfer. See the CVW topics about the Frontera or Stampede2 clusters to learn how to access and use a flagship supercomputer.
In the United States, Open Science Grid (OSG) and ACCESS are the two largest cross-institutional organizations that manage supercomputing access. Institutions participating in OSG organize themselves into virtual organizations (VO) with similar computing interests. They then install OSG software, typically on clusters of compute nodes, and share resources over the internet. While ACCESS resources use software very similar to that of OSG, ACCESS resources are connected by a dedicated high-performance communications network spanning the country.
Recently, commercial cloud providers like Google, Microsoft, and Amazon have begun to offer HPC services in the cloud. Although these systems are rapidly evolving, compute nodes in the cloud tend to have slower node-to-node communication than traditional supercomputer installations.