Survey reveals “massive productivity drain” in data engineering

Complexity, scalability and compatibility remain challenging - 70% data workers struggle with pipeline management.

Managing data pipelines continues to present significant challenges for organizations, with an overwhelming 70% of respondents rating pipeline management as 'somewhat' or 'extremely' complex according to a recent data integration and AI report.

According to the Data Integration and AI-Readiness Survey commissioned by intelligent data integration platform Matillion, more than two thirds of respondents are struggling with pipeline management.

The survey revealed an alarming picture of data integration, an industry facing ever-increasing demand thanks to the growing business need for artificial intelligence (AI) and generative intelligence (GenAI).

Matillion CEO Matthew Scullion said: “Data engineering is boring, gritty and repetitive. Data teams are wasting valuable hours on low-value build, maintenance and management. Instead this time could and should be spent building valuable data products that can be used for new business impacting initiatives.“

The survey highlighted a substantial productivity drain with 64% of organizations reporting that their data teams spent more than 50% of their time working on repetitive or manual tasks.

Rather than increasing staff (and increasing the overheads associated with that), organizations need to identify data integration solutions that empower their current teams to work more efficiently.

Scalability emerges as another critical concern, with 89% of organizations noting issues with their current data engineering platform’s ability to scale pipelines to meet data processing needs.

Scullion added: “This survey highlights the need for a unified solution that is accessible to a wider scope of the organization - easing the load on data engineers and data teams, while empowering business leaders to make data-driven decisions. Bringing AI into this mix creates an incredibly compelling vision of data engineering of the future, where data engineers are able to focus their time on innovation and doing what they do best - solving business problems, rather than manual, laborious pipeline management.”

This Data Integration and AI-Readiness Survey was conducted in partnership with Perspectus Global in January 2025. Respondents included 307 data decision-makers and data user titles, based across the UK and the US.

Zendesk extends its analytics capabilities by acquiring HyperArc, aiming to provide GenAI-powered...
The Connected Conservation Foundation and Cisco introduce a free training program for aspiring...
The IoA's Professional Portfolio offers a dynamic, evidence-based framework to assess and hire...
Integrity360 enhances public sector cyber security amidst Ireland's growing data breaches.
AI adoption is transforming cybersecurity, reshaping roles, and influencing hiring, as highlighted...
Filo Systems joins NetApp Excellerator to refine its AI-driven data compression solutions.
Amidst rising demand for AI solutions, TigerGraph secures investment to enhance its enterprise AI...
Hyland's new Knowledge Enrichment transforms unstructured content into AI-ready data, optimizing AI...