Testing is the most valuable AI investment across the software development lifecycle

Tricentis has released the findings of its most recent AI report, “AI-augmented DevOps: Trends Shaping the Future.” This year’s research aims to understand to what extent the anticipated benefits of AI in DevOps have been realized today, and how a lack of trust, skills, or other challenges could affect its adoption.

When asked to evaluate the most impactful areas for AI investments across the delivery cycle - such as planning, coding, deploying and releasing - DevOps practitioners ranked testing as the most valuable (60%). This result was foreshadowed in the Tricentis’ 2022 study, which found that testing is where organizations expected the greatest value from AI-augmented DevOps, with nearly 70% of respondents having rated the potential of AI-augmented testing as extremely or very valuable.

The research finds that DevOps teams are realizing the benefits of AI, with mature DevOps teams who have adopted AI significantly more likely (30%) to rate their teams as either extremely or very effective. The biggest challenges DevOps teams are using AI to address are developer team efficiency (60%), reducing the skills gap (54%), cost reduction (47%), and software quality (42%). In fact, almost one third (32%) of respondents estimate AI-augmented DevOps tools will save teams over 40 hours per month—equivalent to an entire workweek.

The 2024 results show that teams use AI to augment a wide range of testing tasks, including test planning/deciding what to test (47.5%), test case generation (44%), and analyzing test results (32%). Additionally, nearly half (42%) of respondents expect AI to perform a risk analysis of code changes, helping QA teams focus on code areas with the greatest risk of errors to quality.

The report surveys 500+ DevOps practitioners, managers, and executives from small, mid-size, and enterprise organizations across the globe and in several industries, including financial services, healthcare, and manufacturing.

Other 2024 findings reveal:

Regulation is expected to help build trust, but others worry it will stifle innovation and outcomes. Nearly two thirds (63%) of those surveyed view increased regulation as a way to build confidence in AI across their organization, while a smaller—but not insignificant—number of DevOps practitioners (16%) feel increased regulation will hinder or stifle the potential impact of AI in organizations.

Humans remain vital to ensure software quality, but there’s a shortfall in AI skills. Data shows that humans are still very much “in the loop,” with over two thirds (71%) of respondents checking outputs at least half of the time, and almost one in five (19%) claiming to check AI outputs all of the time. However, a lack of AI skills (28%) is seen as the greatest hurdle to AI adoption in DevOps.

Generative AI (Gen AI) and AI copilots are key drivers of AI adoption: GenAI (45%) is now the most widely adopted type of AI used by DevOps practitioners. In particular, AI copilots are also on the rise, with use cases in planning, code development, and software testing.

“AI is an exciting technology and growing at a pace unlike anything we’ve seen in our industry,” said Mav Turner, Chief Product and Strategy Officer, Tricentis. “As AI technology is further developed, however, training software development and quality engineering teams with the necessary skills to effectively work with AI will be absolutely critical.”

​

“DevOps teams looking to get started with AI should look no further than their testing processes. AI in testing helps to detect, auto-heal and predict defects during development, as well as identify which tests need to be run based on high risk. When coupled with low-code/no-code technology, this means that, regardless of a team’s technical expertise, AI can significantly contribute to overall software quality. As DevOps teams continue to mature, testing will be pivotal to realizing their investment in AI-augmented DevOps tools and practices.”