NVMe is gaining speed in the data centre

As enterprises struggle to balance performance and sustainability in their data centre, NVMe flash is gaining popularity and is replacing disk-based storage for high-performance workloads. The trend is already slowly squeezing HDDs out of the market, which might soon become irrelevant as tape storage is gaining traction for cost-effective storage of cold data. To reap the benefits NVMe promises, enterprises need a new storage platform on the software layer that is designed specifically to take advantage of new technologies like advances in NVMe.

The new era of data

Unstructured data - things like video and imagery - is growing exponentially, resulting in trillions of files and objects and exabytes of data, all moving fluidly within and between data centres and the cloud. Organisations are grappling with how to deliver the performance, the management, and processing of these types of data require. Legacy storage systems, based on HDDs and/or SSDs, often run on inefficient Serial Advanced Technology Attachment (SATA) and are not performant enough. To meet the performance requirements of modern unstructured data workloads like AI/ML, VFX animation, and modern data lakes, organisations are increasingly looking at the advantages of NVMe flash based storage.

HDDs are already being phased out

It is expected that the cost per gigabyte of flash storage will, in 2023 and for the first time, be cheaper than the cost for higher end HDDs. The price trend for flash has pointed down for more than 15 years and 2023 seems to be the year when the cost for flash and HDDs become more in line.

Shipped flash-capacity grew at 37% in 2022 and will overall account for 32% of all storage capacity shipped by 2026. Price is an important factor for the choice of storage medium and certainly a cheaper price point for flash storage will accelerate the adoption of flash-based storage technologies. As organisations look for ways to improve their storage system’s performance, they are looking at NVMe-based storage as it has the highest performance potential of all storage mediums. Paired with RDMA, NVMe can also reduce cost for networking infrastructure and complexity as it does not need Fibre Channel but runs on easy and inexpensive ethernet.

NVMe: designed for solid state memory

Most SSDs used in the data centre today still use SATA. The protocol was once designed to work with mechanical HDDs and, as SSDs have become faster and faster, has become increasingly inadequate, as enterprises try to squeeze every bit of performance out of their storage systems. The NVMe protocol is an advancement over traditional SAS and SATA protocols as it was specifically designed for solid-state memory. This design allows host hardware and software to calculate processes simultaneously, which reduces I/O-overhead, improves performance, and reduces latency. With

NVMe it is possible to process not only thousands but millions of concurrent operations, which is an order of magnitude of performance improvement versus legacy architectures using SAS or SATA.

This increase of performance is already utilised in use cases to increase read-/write-speeds for large files or lower rendering times for higher resolution content like 4K and 8K video. As performance requirements keep on rising for emerging use cases like AI and ML, business intelligence, data analytics, DevOps, imaging in healthcare and life sciences, or modern data lakes, the demand for new NVMe all-flash architectures is consistently growing.

Next to performance, ESG has also become an important factor for many enterprises. Compared to hard drives, NVMe flash can reduce not only a system’s footprint by up to 80% percent, due to the reduced amount of hardware required, but also use much less energy. This reduces not only energy cost but also carbon emissions.

HDD is getting squeezed out of the market from both ends

With the need for more performance, ESG-considerations, and potential savings for networking technology, it is not surprising that organisations are shifting more and more of their active workloads to NVMe flash storage. At the same time, as more data is being generated, organisations need a secure, cost-effective, and “green” way to archive that data, which increasingly is tape storage.

In 2022, for the first time ever, the amount of shipped HDD-capacity has decreased, whilst the ratio and capacity of flash and tape have both increased. The trend is undoubtedly going into a direction where HDD is getting squeezed out of the market by flash from the top and by tape from the bottom. Today the vast majority of total HDD capacity is already shipped as slower high-capacity HDD drives - only confirming the trend that organisations are moving away from disk for their higher performing workloads and archive data.

Innovation is in the software

Without question, NVMe flash offers many advantages over legacy storage solutions. To utilise those advantages fully, however, it needs to be paired with a modern storage platform on the software layer. Current storage solutions on the market utilising NVMe often require custom hardware. As a result, these products will not run in the cloud. And as they are custom-built, customers are unable to benefit from falling costs for flash hardware and cloud services.

A software-defined storage solution (SDS) developed for NVMe can solve this problem. It separates hard- and software layers and gives enterprises the possibility to benefit from the best possible solutions on both. The ideal solution would be a modern cloud-native distributed file and object storage platform built for the latest flash and memory technologies like NVMe and RDMA. These protocols unlock the true performance potential of flash by enabling more parallel reads and writes, as well as dramatically reducing network latencies for flash storage nodes communicating via an NVMe fabric. Truly distributed storage architectures also eliminate bottlenecks related to metadata management and overhead, providing consistent low latency performance for both high bandwidth and high IOPs applications.

Organisations are facing a deluge of unstructured data and need more performance out of their storage systems. They also need to take costs and ease of use into consideration. As flash prices continue to come down, NVMe-flash storage systems are increasingly replacing disk-based storage for high-performance workloads. Organisations with digital transformation initiatives require faster performance, more scale, and more flexibility to operate in hybrid and multi-cloud environments; legacy NAS platforms will never meet these requirements. An investment in a modern NVMe flash storage platform will provide the performance, scale, and simplicity to power the unstructured data applications of the next decade.