Speed

To be able to measure the real value of using AI/ML in testing, the measurements need to be a mix of the time it takes to create a similar test scenario though code vs. code-free, together with test maintainability/reliability over time/software iterations.

It is evident that AI-powered test creation takes about ten times less than writing the same test in code. If we add to that the time spent on maintaining the same test across two or three software releases, we can get to a test creation efficiency metric, more powerful that just measuring the time.

In the same category of speed, we could also look at the E2E test execution time, comparing code-based testing to code-free. The result is that code-free delivers better results than code-based test execution.

Quality

In the quality category, we can assess test automation coverage metrics, defect percentage, as well as the manual test case percentage before and after employing AI/ML.

When combining code-free AI-powered tools on top of existing code-based test automation, we can identify a decrease in manual testing activities, which means greater test automation coverage, as well as less defects slipping to production. Looking at it the other way around, it means that more defects are discovered during the development cycle and initial stages where they are less expensive to fix.

Cost

The cost category is connected with the ROI, so some of the important metrics to look at here, are the cost of software, cost of testers/developers, and the cost of defects.

We can understand that a software test engineer/developer that writes tests with code gets a higher salary than a business tester. What organisations need to do here is to balance the overall quality cost by employing few of these roles. For example, business testers would be able to leverage code-free, AI-powered tools, whereas developers can continue to write code.

In addition, code-free automation powered by AI and ML enables to detect defects faster and allow for continuous feedback and collaboration between testers and developers, reducing the overall DevOps cycle costs.

What is important, is not only to measure these metrics, but also facilitate continuous alignment as well as continuous improvement.

Additional Metrics to Consider

AI and ML should be applied to additional processes throughout the software development lifecycle, to increase productivity and enhance overall quality:

• Risk Coverage Optimisation and Impact Analysis based on historical and defect trends you can create an efficient regression test pack that is more focused on the software iteration scope.
• Data automation and monitoring using AI to detect trends, common use cases, pitfalls, and other real users’ activities to make more informed decisions.
• Predict software delivery by identifying what needs to be tested, and also identify high-risk/defect-prone areas and act upon them.
• Source control systems analysis for buggy source files, code change velocity, and other productivity measurements.