Hyperion To Provide a Peek at Storage, File System Usage with Global Site Survey

Curious how the market for distributed file systems, interconnects, and high-end storage is playing out in 2024? Then you might be interested in the market analysis that Hyperion Research is planning on rolling out over the next several months.

Hyperion Research is the HPC-focused analyst group that separated from IDC before IDC’s parent company was The group, which is based in St. Paul, Minnesota, conducts periodic comprehensive global HPC site surveys to get a better idea of the compute, storage, and networking investments that some of the biggest HPC sites around the world.

Mark Nossokoff, the Hyperion analyst who tracks storage, recently sat down with Datanami to chat about the firm’s research and the 2024 global HPC site survey that’s currently underway. The company plans to start releasing results from the 2024 site survey around the International Supercomputing Conference (ISC), which takes place next month in Hamburg, Germany.

“We have over 1,000 systems at over 100 sites who we ask about the whole gamut of the HPC, storage, and interconnect realm,” Nossokoff said. “We ask about file system usage utilization, capacities, on prem storage versus capacity and storage in the cloud, preferred cloud storage providers, preferred on-prem storage vendors. So its has lots of statistics.”

As in previous years, the 2024 report will gauge who’s winning in the battle among HPC storage file systems. Overall and within the NAS file system space, NFS was the clear winner in 2021, according to 2021 HPC site survey  report shared by Hyperion, with about a 53% share across 2,006 HPC systems running at 141 supercomputer sites spanning government, industry, and academia. That was followed overall by several parallel file systems led by Lustre at about 35%, and GPFS/Spectrum Scale at about 25%. , and HDFS at about 20%. ZFS, XFS, and other.

Hyperion will be looking to see how the file system landscape has changed. NFS figures to continue to fair well, particularly among industry sites, while Lustre will likely maintain its lead at government sites. Nossokoff will also be looking to see whether relative newcomers, such as Distributed Asynchronous Object Storage (DAOS), the open source object store spearheaded by Intel in 2012, have gained any traction.

“It will be interesting to see if DAOS has gained any traction beyond Argonne and Google, where it is the foundation for its Parallelstore storage service offering,” Nossokoff said.

Hyperion will also be exploring other traditional NFS-based systems and emerging data platform architectures, including those from DDN, Dell, Hammerspace, HPE, IBM, NetApp, Qumulo, VAST, Weka, and others.

(Pavel Ignatov/Shutterstock)

Hyperion will be looking to see how the surge of investment in AI is impacting storage. Are they using different storage types for training and inference workloads, or is it the same? Which vendors are moving ahead in the race, and is there anybody new or unexpected gaining traction?

“We are asked a lot of about interconnect adoption,” Nossokoff said. “What architecture is it? Is it a single [converged] interconnect that handles both storage and the MPI traffic?, Or are they independent interconnects, one dedicated to server-to-server and the other dedicated to system-to-storage?”

Finally, Hyperion will analyze the network preference, which is typically a horserace between InfiniBand and Ethernet. For the 2021 site survey, Ethernet was ahead with a 45% share, with 100Gb being the most common speed, while InfiniBand had a 36% share of the overall network market, with 100Gb and 200Gb sharing an identical 31% share within the InfiniBand cohort. And how will  OmniPath fare, and will it climb above the 4.5% share it had in the last site survey?

The site survey also delves into the compute side of the house, which allows Hyperion to gather very specific data about how many systems are in use, how many processors, etc. The company also examines user’s spending on HPC resources in the cloud, providing insight into cloud spending patterns on compute, storage, networking, and software applications.

Hyperion doesn’t typically release the entire results from the site survey all at once. The study is funded by a handful of clients, who get first dibs at the good data. But eventually the research trickles down to the industry at large, giving us a detailed peak behind the HPC curtain.

Related Items:

Object and File Storage Have Merged, But Product Differences Remain, Gartner Says

The Past and Future of HPDA: A Q&A with Steve Conway

Hammerspace Hits the Market with Global Parallel File System