Lustre Components

The primary components of a Lustre file system are the Metadata Server (MDS) and the Object Storage Server (OSS) nodes, as illustrated in the diagram below.

  • The MDS contains the filename-to-object map (stored on the MDT).
  • The OSSs contain the actual data (stored on the OSTs).
Lustre Clients are connected to one another and to Metadata Servers and Object Storage Servers via a high speed interconnect.
The main Lustre components are the Metadata Servers and Object Storage Servers.

TACC's shared file systems are all based on Lustre and nodes of all types host Lustre clients. In each TACC supercomputer, Lustre provides access to multiple globally available, distributed systems for I/O operations on the compute nodes. Compute nodes, as well as login nodes, share the same high-speed interconnect (either InfiniBand or Omni-Path) as the MDS and the OSSs, as indicated.

A key feature of Lustre is its metadata server (MDS), which is responsible for maintaining namespace information such as filenames, directories, access rights, and the layout of distributed files. These metadata are stored in a single local disk file system on the MDS (possibly with a mirrored twin for failover). This means the MDS may become a bottleneck under metadata-intensive workloads.

However, compared to block-based distributed file systems such as GPFS and PanFS, Lustre has an advantage in that its MDS is not directly involved in any file I/O operations. Lustre achieves this by storing the file data on many separate Object Storage Servers (OSSs). Since the MDS is not in charge of block allocation on the OSSs, it is not a source of contention for file I/O. Thus the Lustre architecture avoids some of the worst potential I/O scalability problems, while preserving the coherence of the overall file system.

 
©  |   Cornell University    |   Center for Advanced Computing    |   Copyright Statement    |   Inclusivity Statement