How to build functional data stacks in the age of generative AI

Unquestionably, there remains a lot of hype around how generative AI (GenAI) can be utilised to drive strategic innovation or create competitive advantages. Equally, key challenges remain in harnessing the hugely valuable insight held in an organisation’s disparate data sources.  Despite this, the impression that breakout applications like Chat-GPT have made on individuals can’t be overlooked. According to a global Oliver Wyman study published earlier this year, 96% of workers believe generative AI can help them in their jobs.

This is good news for decision-makers looking to reap promised GenAI efficiency productivity gains through data-driven insight automation. However, organisations that keep data and analytics siloed risk AI's benefits remaining elusive for most employees and its overall impact being underwhelming.

While refining raw data for insights will remain the lifeblood of intelligent decision-making, doing so at speed and scale requires enabling everyone to use the data and empowering all to take advantage of accessible analytics. Only then will enterprises be in a position to use GenAI to unlock the full capacity of their ever-increasing data to make decisions faster, better, and more effectively than the competition.

Modernising data management and empowering all to extract value from data at the speed and scale requires employing the right data tech stack. Too often, overcomplicated data stacks combined with tool sprawl, data siloes, and rigid data environments aren’t configured to make the most of the technologies available to accelerate the data journey. Lacking the ability to support multiple personas, different skill sets and unlimited use cases to empower the entire workforce to take full advantage of the technology to drive business value, limit the business use cases of GenAI.

Foundational priorities.

Well-designed, modern data stacks empower a range of personas roles found in today’s businesses to get the most value from data. Organisations with such multifunctional, flexible, usable data stacks in place are the ones set to benefit most from leveraging AI and data capabilities. When designing data stacks CIOs should focus on three key factors as part of their planning process:  

Revenue: how is this going to improve revenue or margins? 

Customer experience: how do the technologies give customers what they need, given the constant evolving nature of their journey and path to purchase? 

Employee experience: how is this going to ensure our colleagues have access to the right data and insights they need to be productive and make the company better every day? 

After all, those capable of using their data and analytics the best and the fastest will deliver more revenue, better customer experiences, and stronger employee productivity than their competitors. 

Just as buildings start with foundations, analytics stacks should begin with a firm base of data essentials – storage, data governance frameworks, ETL, analysis and reporting. Putting the best foot forward with these will result in quick value unlocked right off the bat. For builders, it’s worth keeping in mind that data analytics stacks should also be built with scalability and flexibility in mind. So, consider the growing needs of the business as well as accommodation for increasing volumes of data.

Choose multifunctional technology for integration and interoperability.

Data can be stored, accessed and analysed in many different places. While the cloud’s benefits are extensive and well understood, on-premises analytics are still expected to be used in some capacity by 85% of businesses next year. Your data stack should be sufficiently flexible to support multiple deployment scenarios, for instance, managing data pipelines whether they are on-premise, private, public, or multi-cloud. It should also allow operatives to transform their data in the data warehouse of their choice and enable them to build data workflows in one place and execute them in another.

Platforms capable of both on-prem and SaaS-based deployments on multiple clouds as well as the ability to collaboratively develop and manage data workflows that run directly through programmes such as Databricks, Snowflake and AWS, are therefore useful elements in a functional data stack. This kind of flexible functionality and smooth flow of data, through seamless integration with existing tools ultimately helps speed up the analytic process. Further accelerating the path to AI-driven insights as teams build effective AI models and applications based on the right data more quickly.

Meet employees where they are.

Managers know all too well that for generative AI to be successful internally, all employees need to “speak data” and be data-skilled to solve business challenges and deliver decision intelligence, not just the tech workers.

While data enables outcomes for every area of a business, many tools used for analytics weren’t designed to be used by employees of all skills. As a result, data analytics has historically been reserved for very specific roles and specific knowledge workers. The ultimate goal of stack modernisation is empowering the entire workforce to take full advantage data, compute, and automation resources available.

Whether it’s developing personalised offerings for customers, reducing churn, or ensuring regulatory compliance, each line of business should have the ability to creatively solve its own analytic problems and apply its own domain knowledge to relevant and impactful use cases. For this to be achieved, accessible and easy-to-use data interfaces need to be prioritised. They should be accessible – ideally, low-code or even no-code – making it easy for anyone, not just data scientists or engineers, to compile and analyse data. 

Low and no-code applications are an effective way of putting such data interfaces into more hands. Gartner has projected that 70% of new applications will use low-code apps by 2025 to make it possible for workers with functional domain knowledge to contribute to analytics. Prioritising this means everyone who needs to work with data benefits from the resources the organisation has invested in.

It’s an exciting time for CIOs and IT leaders to be shaping how businesses can effectively harness AI technology to drive strategic innovation and create competitive advantages. By nailing the essentials of a modern tech stack, accelerating the data journey and improving the data culture, organisations are best placed to build winning analytics programmes and join the GenAI wave to start getting the answers to ever more complex analytical problems.