The buzz around Big Data has risen into an audible crescendo over the past twelve months, as businesses everywhere have begun to realise its potential to make them more competitive in the digital business economy. As proven use-cases have started to emerge, we’ve moved beyond the early days of hype. Businesses are now investing heavily in practical applications of Big Data to improve processes and provide better support for their customers. Google is a prime example of a company putting Big Data into action, with its Maps now using real-time mobile data to help users avoid traffic-jams. Illustrating the likely scale of this progress, IDC recently forecast that the Big Data technology and services market will grow at a compound annual growth rate of 23.1% from 2014 to 2019, with annual spending reaching $48.6 billion by 2019. Clearly, Big Data is set to get bigger and bigger.
All aboard the Big Data train
It’s not all that surprising that most businesses fall into one of two buckets; those that are using Big Data and those that are assessing how they could be using it. For those in the latter category, it’s important to realise that the step to Big Data might not be as big a leap as they think. Indeed, for many, it’s a natural transition as an enhancement of their existing business intelligence initiatives. Augmenting traditional Business Intelligence platforms with Big Data will simply enable those businesses already harnessing advanced analytics to add a huge wealth of new, unstructured data from sources such as social media, blogs and Internet of Things devices, to their existing structured datasets. This will provide much more context around why people behave in a certain way and identify the relationships between datasets to enhance customer profiling with a 360 degree view.
However, it’s still far from an easy undertaking, and the pressure to start using Big Data can be a distraction that sends businesses careening off the rails towards bloated costs and delays in realising its full potential. As such, it’s vital to have a clear destination in mind for Big Data. Rather than just hopping aboard the first train to arrive at the station, businesses should identify what it is that they want to achieve from Big Data, to ensure the journey takes them where they want to go. This will help determine which data is required and how it can be collected, which ultimately ensures projects are completed in an economical manner.
Getting out of the platform
Once the objective has been set, the business needs to look at its existing capabilities and assets to identify where improvements and efficiencies are needed to set Big Data off on the right track. They need to take into account what size their Big Data platform should be, how much compute power they need, how data will be migrated and how much storage space will be required. However, the first thing to look at is the infrastructure used to store the organisation’s data. One of the most cost-effective approaches to storing Big Data can be to start building a data lake. This approach enables businesses to reduce their storage costs by building their data repository on commodity servers, which are significantly cheaper than those needed for more traditional data warehousing platforms. Indeed, this cost-reduction is one of the primary drivers for many businesses when considering their motivations for Big Data adoption.
However, using a data lake also removes the siloes that separate data sets in relational database environments. This enables businesses to create a vast reservoir where data can truly become Big Data, generating far richer insights by identifying patterns and contextualising the relationships between different data sets. As an added benefit, it is much easier to get data into a data lake in comparison to relational database environments, as there is no need to prepare or remodel the data before it can be inputted.
Getting the right crew aboard
Once the storage platform has been built, businesses will need to ensure they have the right people on board to make their Big Data journey a success. As well as the IT team, it’s important to have digitalisation specialists and managers from every level of the business involved to identify how analytics can be integrated seamlessly with the processes it’s intended to support. It’s also vital to have people with the skills needed to implement and use Big Data tools and manage their operation effectively. However, this can often be a major stumbling block, as the relative newness of Big Data means there is a scarcity of ‘data scientists’ with these skillsets available. Whilst many universities are now offering courses to address this shortage, this younger generation of workers is new to the business world and lacks the experience of their more senior counterparts.
As such, rather than trying to navigate uncharted territories alone, many are bringing on board technology partners who already have the skills and experience of implementing Big Data projects. As well as removing the barrier created by the shortage of available skills internally, this enables businesses to learn from the experiences of others that have already completed a Big Data journey. Many partners that have managed similar projects elsewhere have also been able to build accelerators to reduce the time needed to get a Big Data initiative off the ground. For example, frameworks for completing major milestone tasks such as data integration can provide a series of modular building-blocks that can reduce much of the complexity and enable businesses to move more rapidly on their journey to Big Data.
Wherever they are on their journey, there’s a very strong chance that Big Data will remain fixed on the radar of every business. However, it’s important not to get caught up in the hype. Having a fully defined roadmap, a clear destination and the right balance of internal skills and external expertise will be imperative to success. Those that are able to achieve this level of forward-thinking will surely reap the biggest rewards as their business enters a new, data-driven future.
