How Software Ushered in a New Age of Data Storage

With data increasingly stored in cloud-based architectures or in off-prem locations, discussions that once purely focused on data storage hardware have notably shifted towards software. This includes software-defined storage, software managing virtualisation, and automation capabilities utilising artificial intelligence (AI) and machine learning (ML) integration to improve storage optimisation. Here we explore the drivers behind the evolution of software, its game-changing capabilities, its challenges, and how far along organisations are in realising the full potential of software over hardware in the data storage industry. We asked a number of industry influencers and vendors for their views, culminating in a future outlook of software in data storage. 

Drivers behind the evolution of software in data storage

Software in data storage is clearly a mainstay today, but what were the catalysts triggering its development? Expert opinions across the industry reveal a variety of disparate factors coalesced to drive the adoption of data storage software.  

According to Randy Kerns, Senior Strategist and Analyst at Futurum Group, “The trend of software in data storage initially arose from a perception that savings could be made utilising commodity hardware.” He adds, “Interestingly, data services in software were not an initial consideration and developed over time. The evolution drivers continue to be simplicity, stability/reliability, advanced data services, and support/managed services option.”  

Alexander Ivanyuk, Senior Director, Technology, at Acronis agrees, “Convenience and cost are the drivers behind software evolution. Traditional storage is a monolithic bundle of hardware and software. You depend on this hardware and quite often on software that may only work with this hardware. Software-defined storage (SDS) allows abstract storage resources from the underlying hardware platform and that results in greater flexibility, efficiency, and scalability.” 

Another major driver in the development of software in data storage is, of course, the emergence of cloud and hybrid infrastructure. As Fred Lherault, Field CTO, EMEA and Emerging Markets, Pure Storage, tells A3 Communications, “Almost every customer in the world uses hybrid infrastructure these days and that’s a big driver in terms of changes in software development and deployment.” Enrico Signoretti, VP of Product and Partnerships, Cubbit, agrees: “Everything is now about having the same platform on different clouds and on-prem. Enabling users to move data and access it where/when/how they need it.” 

‘Game-changing’ data storage capabilities of a software-led approach 

In exploring the capabilities of a software-led approach that have the greatest industry impact, Sergei Serdyuk, VP of Product Management at NAKIVO eulogises the performance potential enabled by software, “The most ground-breaking capability is perhaps the ability to optimise performance – both by using a logical layer for inter-operable hardware for data movement and processing and an additional ‘intelligent abstraction’ of AI-managed storage provisioning and management. In combination, these two capabilities show the greatest potential for enabling cost-efficient operations.” 

Cubbit’s Signoretti tells us about the importance of “visibility across environments with a single domain approach.” He adds, “On the other hand, you want flexibility (multiple tiers) for better data placement and cost optimisation.” 

Paul Speciale, Chief Marketing Officer, Scality, highlights another attribute: “Initially, the major game changer in software-defined storage was the ability to deliver enterprise levels of data durability and high-availability, but way below the multi-million dollar price points required for custom built legacy systems. This made it possible to deploy systems at cloud-scale that could be trusted with enterprise and user data.” 

Pure Storage’s Lherault agrees, “All storage needs the ability to work in a hybrid and multi cloud manner. For vendors, this allows them to release new software faster and adopt new generations of hardware faster. It’s important to make the distinction that software defined doesn’t necessarily mean software only with commodity hardware. Modern storage arrays are defined and driven by their software capabilities but leverage hardware innovation to enhance the software and deliver greater efficiency.” 

Indeed, software has taken data storage capabilities to another level. As Tim Klein, President, CEO, and Co-founder, at ATTO Technology states, “Without a doubt the two game-changing ‘characteristics’, would be cost and flexibility. The cost savings with software-defined storage versus hardware platforms speaks for itself. With flexibility, we’re talking about the ability to take relatively any storage platform and define what it is and how it can be used – virtually or otherwise.” 

And there is yet another key advantage, as Shawn Meyers, Field CTO at Tintri highlights the value of software to gather data insights: “A software-led approach can provide valuable insight into each managed object and each I/O to help determine the best way to service these with the available hardware.” 

However, when it comes to software’s potential in data storage, there is still much more yet to be explored and realised, as Roy Illsley, Chief Analyst at Omdia reminds us, “In a software-defined world the potential is for storage to be deployed where it is needed to meet the customer demand. The real ‘game changer’ would be if this was universal and could support any technology and include with it the ability to find and secure all data.” 

Navigating the differentiation of ‘software-defined’ 

The term “software-defined” is being used to describe so many storage solutions today (with generally some aspect of commodity hardware at the platform level), but when it comes to software, there are many varying degrees. With software-defined, open-source, and infrastructure-as-a-service, how are data storage vendors differentiating their solutions at the software level? And how do customers navigate the path of data comparison, when there are so many overlapping capabilities between storage software solutions? 

According to Omdia’s Illsley, “This is the million-dollar question: if it is done correctly and is truly agnostic then it is differentiated. Too many software defined solutions are only able to work with a sub-set of infrastructure and are not universal – which is a reality we have to live with.” Illsley suggests the key to ascertaining how software is differentiated is to ask: “How easy is it to extend its capabilities by the customer to meet their specific needs?” 

NAKIVO’s Serdyuk recommends that customers analyse approaches and outcomes: “The offerings can look similar as they address the same pain points. However, the solutions themselves use different approaches, which might be a good starting point. For example, storage virtualisation and ML-powered storage provisioning are both defined as a part of the software-defined storage concept or SDS. Focusing on the outcomes of specific solutions could help differentiate between vendors and their solutions.” 

Tintri’s Meyers goes a step further: “Vendors need to differentiate by reducing complexity and lowering labour costs for their customers. This is done by ease of use, automation, and leveraging AI and ML to automatically tune IT infrastructure based on active usage patterns.” 

“The only sensible way to differentiate,” adds David Norfolk, Practice Leader, Development and Governance at Bloor: “is by capability (performance, functionality, security etc). It doesn’t matter much what you call it, it is what it does that matters – and even if it has a flashy new acronym, you have to evaluate its actual capabilities and whether they are fit for your purposes.” 

Jeff Whitaker, VP of Product Strategy and Marketing, Panasas, points out the reality that there is often a balancing act between attributes of a solution, where one must be forfeited for another: “Software defined is a very nebulous term…Customers often ask for three things: flexibility to choose the best-in-class solution with the highest reliability and at the best price point. But these three elements don’t always work together.”  

How the transformation from hardware to software impacts data storage  

The shift from hardware to software has had a tremendous impact on the industry, encouraging innovation and causing traditional vendors to continue to rethink their product lines and approaches. 

Spectra Logic’s Director of Product Marketing Deanna Hoover reflects: “Prior to the adoption of software-defined storage, we saw fewer new vendors entering the market. The engineering and development of storage hardware is more complex and costly than that of software-defined storage,” She adds, “The shift has motivated many hardware-centric vendors to modernise their offerings by focusing on a software-led approach. The adoption of software-defined storage has enabled more vendors to enter the storage market. The increase in storage vendors is raising the bar for competition and driving innovation.”  

Jimmy Tam, CEO of Peer Software, shares other advantages, “Decoupling hardware and software has enabled freedom and flexibility for customers adopting software-defined storage technologies. Included in these benefits are independence from proprietary systems from a single vendor, enhanced scalability, and increased data mobility across arrays locally as well as geographically dispersed.” 

Molly Presley, Head of Global Marketing, Hammerspace, raises an important point: “The advent of the cloud meant that hybrid environments were inevitable. Economies of scale are different, hardware was commoditised faster. Then supply chains became constrained and customers no longer can be certain they can get hardware from preferred vendors. All of these trends have led to an increased importance in software being the strategic differentiator in how a data strategy is architected. No longer are most new architecture initiatives storage hardware centric and designed around storage features. There is an expectation the software above the storage will provide the required data services across more than one vendor’s storage.” 

The shift has also resulted in a balancing act for vendors between offering freedom of choice vs. complexity for customers, as Scality’s Specialeclarifies: “For vendors, it does force some choices in the degree of hardware freedom. The more degrees of freedom offered, the higher the complexity in software development and testing. Customers ultimately value freedom of choice, not only for the initial deployments - but to be able to buy the most cost effective hardware at any later time when scaling out the system, or to be able to get preferential pricing from other platform vendors.” 

ATTO Technology’s Klein takes a broader view and sees the transition as opening up opportunities for vendors and customers alike, “It’s like any other aspect of technology – vendors and customers adapt. For vendors it presents a path to create and develop new products. For customers it presents new solutions that potentially work better for them and at lower cost.” 

Software Challenges 

It is clear that the journey from data storage hardware to software is not an entirely smooth one, as it means choices need to be made on the part of customers as to how much complexity they wish to manage in exchange for the freedom of choice inherent in software. Here, we delve further into this conundrum, and other issues that the transition presents. 

According to Whitaker at Panasas, “The impacts will be felt the strongest across reliability, performance, and supportability. The challenge with software-defined is with performance. Being abstracted from the underlying hardware means there are limitations on how that hardware can be utilised.” 

Acronis’ Ivanyuk shares further pain points: “Software is also more prone to vulnerabilities, so theoretically software-defined storage is more vulnerable to cyberattacks. This is something to look for, but not a showstopper.” 

Scality’s Speciale raises the importance of staff ‘know-how’ during the transition from hardware to software, “Mainly, there is a shift in skill sets in going from legacy and proprietary systems management to newer solutions. Kerns from Futurum Group, agrees as to the impact on staff, and feels forward planning is required; ”Ultimately it becomes a staffing issue. There are increased support requirements and it may require someone with more capabilities than a storage administrator.” Andy Buss, Senior Research Director, IDC, also sees training is required: “For software-defined storage to be a success, companies need to be driving the full benefits of automation and AIOps that come with it. IDC research shows that only around a quarter of companies in Europe today have the skills as well as sufficiently transformed IT infrastructure to be able to achieve this today. To change this takes time, investment and training and high-level management support.”  

Outlook: software’s future impact on data storage  

As software’s evolution has caused much disruption thus far, we ask: “What does the future of software in data storage look like?” 

Bloor’s Norfolk sees further disruption as positive, and foretells, “Established players will need to evolve fast; new players will have opportunities - and may then get bought by the big established players.” 

IDC Europe’s Buss sees a shift from data storage to data management: “As storage becomes more software defined, we also expect to see more emphasis on data management and manipulation rather than focusing on architecting complex storage systems.”

Similarly, Spectra Logic’s Hoover foretells the further growth of systems capable of unifying data, wherever the location, “The option for organisations to deploy a one-vendor software storage solution for applications running in multiple locations will simplify the procurement and management of storage, while reducing costs. This will in turn drive storage vendors to be competitive by focusing on innovation and more rapidly delivering advanced features. With the increased adoption of software-defined/software-based storage, we will likely notice a decline in hardware-only storage solutions.” Acronis’ Ivanyuk agrees, “The industry has to evolve and the hardware solutions market will shrink. Software-defined solutions will grow in numbers and features, dominating the market.” 

Peer Software’s Jimmy Tam predicts, “As much as most companies want a one-size-fits-all approach for enterprise storage the reality is that the future of storage will be even more distributed across edge, datacentre, and cloud since for performance reasons the data will need to be close to where the end-user or application resides or where the data is either created or analysed.”  

Hammerspace’s Presley sees this future as already here, “Most organisations are already running workloads and applications in multiple locations. The evolution of these software packages from vendors will help reduce the manual work needed to be done by IT, automate a lot of the data placement decisions, and reduce the need for custom/in-house built data orchestration scripting.” 

We are on a journey to fully explore the capabilities of software in data storage and there is no going back. It is in our nature to evolve and evolve we must. Software in data storage has been a disruptive force thus far, causing dynamic competitive shifts resulting in significant innovation, the liberation of resources, greater efficiency, and the freedom to access and manage data from any location. Vendors and end users alike will need to adapt to the impact this brings. As for the challenges, it seems they will naturally equalise - as Aristotle wrote, “Water finds its own level.”