Delivering advanced enterprise SSD capabilities with NVMe

THERE ARE SEVERAL good interfaces available for enterprise SSDs but PCIe has become one of the best. Its direct connection to the host CPU provides near-zero latency. The current PCIe 3.0 version provides 1GB/s of bandwidth per lane enabling PCIe SSDs to use 4 or 8 lanes concurrently, equating to 4GB/s or 8GB/s of achievable bandwidth from each device.

Today’s enterprise servers have several PCIe slots available for SSD deployment and can accommodate up to 40 lanes of PCIe per CPU. A two-socket server can support 80 lanes of PCIe. With an ongoing performance gap between server processors and HDD subsystems, PCIe-SSDs are highly desirable and represent a fast-growing multi-billion dollar global opportunity over the next five years. At present, PCIe SSD deployments are at introductory levels.

To achieve these expected deployment levels, PCIe required a host-controlled interface standard. Without this, each SSD vendor has to develop their own proprietary drivers on how the SSD will interface with the host. By standardizing the interface, only one driver development is required. As such, the Non-Volatile Memory Express (NVMe) specification was created to deliver the full potential of non-volatile memory for enterprise and client platforms in support of PCIe-based SSDs.
NVMe overview
The interfaces historically used for SSDs to present day (SATA, SAS, Fibre Channel, etc.) were designed for spinning disks. As SSD technology continued to improve over the years, these HDD interfaces have become inefficient for flash-based storage. With NVMe, SSD developers for the first time are afforded a standard flash-only host controller interface supported by an industry consortium and designed to access large amounts of data at high-speed.

NVMe is a scalable host controller interface specification for accessing SSDs on a PCIe bus within enterprise and client systems. The specification is an ongoing development effort coordinated through an open industry consortium of 90+ members under the direction of a 13-company promoter group. The initial NVMe 1.0 specification was published in March 2011. The current specification of NVMe is version 1.1, published in October 2012.

Architected for performance, NVMe features a streamlined and optimized register interface and command set that uses the minimum number of CPU clocks per I/O. The results are faster access to critical data across multiple cores, lower latency and a reduction in power consumption as NVMe supports massive parallelism and high scalability so it can meet current and future storage needs. The standard is also designed to support 2.5-inch form factors enabling NVMe-based PCIe SSDs to be easily hot-swapped in the event of a failure by simply pulling the drive out of the server chassis without shutting down the server. This hot swap serviceability provides a significant advantage over PCIe edge cards that require the server to be shut down and repowered in order to replace failed cards.

The NVMe standard also supports dual port capabilities as a form of High Availability (HA). This provides two redundant data paths in case of a sudden power loss or system failure, so if one of the data paths becomes compromised, the available data path continues operation.

Inner workings
Enterprise I/O requests spend much of their time in the hardware infrastructure (i.e. NAND flash media, flash controller, host bus adapter, related hardware, etc.) as only a small portion of time is spent working through the software I/O stack. NVMe uses a streamlined stack that reduces software stack latency and supports many more commands per queue than other legacy protocols as outlined in Figure 1.

The SAS protocol can support 254 commands per queue while SATA supports 32. NVMe supports 64,000 commands per queue, as well as 64,000 queues, so it will enable extremely fast hardware.

In a published test, the Linux NVMe software stack reduced software overhead by more than 50% (from 6µs to 2.8µs) and the number of instruction cycles from 19,500 to 9,100 when compared to a Linux SCSI/SAS software stack.1

NVMe versus AHCI
The Advanced Host Controller Interface (AHCI) was designed for high latency rotating media back in 2004 versus low latency non-volatile storage required today. As such, AHCI cannot fully utilize SSD technology. NVMe is designed for PCIe SSDs and future memory technologies delivering low latency through a streamlined storage stack. It does not require register reads to issue a command as AHCI requires four non-cached register reads per command that adds approximately 2.5µs of latency.2

With NVMe, support for multiple queues and high queue depths ensure that the CPU can be utilized to its full potential as I/O contention and system bottlenecks are greatly reduced.
Data protection
NVMe features a Data Integrity Field (DIF) to protect data and support redundancy. Extra bytes of ECC are used to check data characteristics once a read or write operation has been initiated by the host. The data block includes the extra bytes of ECC as two bytes are dedicated to Cyclic Redundancy Checking (CRC) that checks for transmitted data errors. A CRC value is stored with each logical block written and compared to a computed value when the block is read.

Product availability
NVMe is an emerging interface specification that provides an opportunity for SSD vendors to be an early provider with the key strategic driver being time-to-market as business momentum behind NVMe-based PCIe SSDs continues to grow.

OCZ Storage Solutions - A Toshiba Group Company and Pioneer of SSD technology just introduced its next generation Z-Drive 6000 PCIe Series that utilizes the NVMe interface to dramatically boost the number of random I/O operations per second that a system can process, and concurrently reduces latency and power consumption by streamlining the software storage stack. The new NVMe SSD series includes 2.5-inch small form factor (SFF) models as well as Add-In Card (AIC) models.

The Z-Drive 6000 PCIe SSD Series has successfully passed NVMe compliance testing at UNH and certified as NVMe-compliant devices by the NVMe Consortium. This includes all models ranging in usable storage capacities from 800GB to 3.2TB and future 6.4TB models. This compliance provides OEMs and third party SSD developers with confidence that their development efforts will support NVMe.

Conclusion
The promise of NVMe is real and its emerging momentum is signifying a new age of SSD technology that exposes flash as memory and unleashes non-volatile memory performance delivering data faster to meet the needs of enterprise and client applications. From an SSD vendor perspective, NVMe has a number of benefits. Standardization makes the development effort easier with a common protocol between client and enterprise, as well as a driver that has new capabilities for firmware differentiation.

Protocol efficiency streamlines the I/O stack to reduce software stack latency. Extensive commands per queue and queues supported enables very fast hardware. DIF delivers redundancy checks for transmitted read/write errors initiated by the host, and the ability to hot-swap SSDs reduces system downtime as well as total operating cost.

Footnote References:
1. First NVMe SSDs announced,Dennis Martin, Storage Newsletter, September 20, 2013
2. Testing SATA express and why we need faster SSDs, Kristian Vatto, AnandTech, March 13, 2013

Additional References:
NVMe Powers SSDs in the Enterprise, Kam Eshghi, EE Times, November 27, 2012
Introduction to NVMe Technology, OSR Consulting, The NT Insider, Nov./Dec. 2014 issue
Why NVM Express, NVM Express, http://www.nvmexpress.org
NVM Express, Wikipedia, http://en.wikipedia.com/wiki/NVM_Express