Storage I/O performance is the number one issue in HPC Systems for Big Data

HPC end users are elevating the discussion about data storage and recognizing the importance of storage I/O – not just compute – in an effort to hit the exascale threshold by the end of the decade according to the results of a DataDirect Networks (DDN) High Performance Computing (HPC) Trends survey.

  • 10 years ago Posted in

The survey polled a cross-section of 60 DDN HPC end users representing hundreds of petabytes of storage investment, which included individuals responsible for HPC allocations encompassing high availability storage and file systems across a variety of industries. Of those surveyed, 60 percent manage or use at least one petabyte of storage, and over 20 percent manage or use 10 PBs or more.

According to the survey results, more than two-thirds (68%) of those polled agree that data and data storage has become the most strategic part of the HPC datacenter. Moreover, by a margin of two to one respondents, the survey also reveals that today’s storage technologies will need to undergo massive change to hit exascale proportions.

In addressing this challenge, the overwhelming majority of respondents (78%) agree that hybrid storage is the evolutionary next step for HPC storage, combining both the performance and cost efficiency benefits required of storage at exascale.


Future Advances in Storage Performance to Deliver Massive I/O Gains and Reduction in Storage Costs for HPC, Analytics and Cloud Computing Environments
· With a nod to future technology requirements, the survey also shines light on the biggest storage and/or Big Data challenges for HPC users today, with nearly half of respondents citing mixed I/O performance the primary issue (47%), followed by managing rapid data growth (22%), performance (13%) and multi-site replication or collaboration (5%). Thirteen percent of those polled name ‘other’ concerns, including the need to manage the overall total cost of storage without compromising storage scalability and performance.
· With storage performance a critical requirement especially at today’s petascale levels, of those respondents with an opinion, 85 percent agree that site-wide file systems like those at at Oakridge National Laboratory (ORNL), National Energy Research Scientific Computing Center (NERSC) and Texas Advanced Computing Center (TACC) are the next infrastructure trend in HPC environments.
· With HPC no longer limited to data centers at research institutions of the Top500 Supercomputing sites, the survey also examines end users opinions on analytics and integrating computing workloads into large scale HPC cloud environments. More than two-thirds (70%) of those polled point to storage I/O as the biggest bottleneck in analytics workflows, suggesting that compute is no longer the main challenge for analytics platforms. While multi-site HPC collaboration may be on the rise, security and data sharing complexity remain the biggest impediments according to 75 percent of those surveyed.
· Finally, HPC organizations continue to find themselves under increasing funding scrutiny and according to nearly three quarters of the survey respondents, it has become more difficult for HPC sites to obtain funding for compute infrastructure without demonstrating a clear return on investment (ROI).
 

Quest Software has signed a definitive agreement with Clearlake Capital Group, L.P. (together with...
Infinidat has achieved significant milestones in an aggressive expansion of its channel...
Nearly all senior business decision-makers (96%) surveyed report data strategies as essential to...
SharePlex 10.1.2 enables customers to move data in near real-time to MySQL and PostgreSQL.
NetApp extends its collaboration to accelerate Ducati Corse’s digital transformation and deliver...
Partnership to be featured at COP26, highlighting how data-driven solutions and predictive...
Next-Gen solutions to deliver market-leading enterprise cloud scalability, cyber resilience and...
he EMEA external storage systems market value was up 3.3% year on year in dollars but down 5.5% in...