From interest to implementation – a Big deal?

In the survey carried out in April 2013 and sponsored by Talend, less than a quarter of respondents (24%) said there was no interest in big data within their organisation – a huge drop from a similar survey carried out less than a year ago, which reported 61% as having no interest. At the same time, the proportion of respondents engaged in a preliminary discussion about using a big data approach has increased from 24% to 36% over the same time period.

Despite this clear growing enthusiasm, a significant percentage of organisations have so far taken no practical steps to roll-out a big data strategy. While 19% of those surveyed (up from 8% in the previous year) were at the planning and appraisal stage, the proportion engaged in a pilot of big data remains static at 4%. One in ten respondents in the latest survey are engaged in a large scale roll-out, up from 2% last year, but still a relatively low figure.

“It is encouraging that the number of businesses rolling out big data strategies has increased but overall adoption of big data strategies remains slow,” says Yves de Montcheuil, vice president, marketing, Talend. “There is still a significant gap between those businesses expressing an interest and those taking the plunge and actually implementing the approach. It is a gap that the industry needs to address and close if the promise of big data is to be fulfilled.”

Currently significant barriers still exist to the prevalence of big data project roll-outs. Two of the top three constraints are budgetary restrictions and skills shortages, often identified as key barriers to any IT endeavour.

The potential scale of most big data projects explains the financial concerns, especially given the on-going downturn. Also, skills are key because big data requires people to be able to integrate any number of large and inflexible disparate data sources and skilled data scientists to analyse the collective data streams.

“Contrary to popular belief, big data projects do not have to come with a massive price-tag,” adds de Montcheuil. “Being able to run open source databases and integration tools over an open-source Hadoop platform allows big data integration and analysis applications to be run cost-effectively across commodity server clusters, thereby reducing hardware spend.”

The report argues that tools are available today capable of exploiting Hadoop’s MapReduce framework, breaking the vendor lock-in of proprietary ETL engines and escalating licence fees. Graphical tools have been developed which can be used by Java programmers, eliminating the need for rare and expensive data scientists. Also, these open-source tools can be downloaded and tried for free, which cuts the cost of sinking the big data equivalent of a test well, by removing the limitations on technical and economic scalability.

On the reverse side of the coin, a range of factors are positively driving big data adoption. For 24% of respondents to the survey, it is the increasing volume of data that is driving uptake. For around one in five (19%), it is the business requirement to increase revenue. Pressures of compliance and product and service development motivate 11% each; 6% say big data in their organisation is driven by the requirement to find new customers, keep up with competitors or collaborate more closely with business partners.

“The long-term future for big data adoption by businesses is extremely positive, with the technology in place, the key drivers likely to become stronger over time and growing levels of interest among businesses,” says de Montcheuil. “In the short term, organisations need to address the barriers and the issues holding many of them back – and once again technology can be key in streamlining the process and making it more effective to implement big data strategies.”