Where AI Helps and Where It Falls Short in QA

There is a lot of talk about what AI can do in software testing, and much of it is vague. Vendors promise transformation without getting specific about what that means in practice. The reality is that AI does some things well and struggles with others, and knowing the difference is more useful than general enthusiasm. We have spent the last few years working with AI testing tools across a range of client projects, and what follows is what we have learned about where the technology actually delivers and where it does not.

The Stochastic Problem

Before getting into specific capabilities, it helps to understand something fundamental about how AI testing tools work. Traditional test automation is deterministic: run the same script against the same application state, and you get the same result every time. AI does not work that way. The models powering these tools are probabilistic, which means the same input can produce different outputs depending on how the model interprets things.

Phil Lew puts it bluntly: if a vendor tells you their AI tool produces the same results every time, they are either not actually using AI or they are not being honest with you. This is not a flaw in any particular tool. It is how the technology works. The same test case run twice might flag different things, find a defect one time and miss it the next, or produce duplicate reports of the same issue.

This has practical implications. You cannot treat AI output the way you treat traditional automation output. It requires review. It requires human judgment about what is real and what is noise. Organizations that expect AI to run unattended and produce reliable results end up disappointed.

Where AI Actually Helps

That said, AI is producing real value in specific areas. The key is matching the technology to problems it is suited for.

Test case generation is probably the clearest win right now. Given requirements, user stories, or existing test cases, AI can suggest additional scenarios much faster than a human could enumerate them manually. An ISG case study of a large e-commerce platform found that test case creation time dropped by 50 percent after integrating generative AI into their process. Gartner research reports that AI can improve test accuracy by 43 percent and widen test coverage by 40 percent.

The output is not perfect. It requires review, and some suggestions will be redundant or miss the point. But as a starting point, it is significantly faster than starting from scratch. Teams that used to spend days brainstorming scenarios can now produce a first draft in hours and spend their time refining rather than generating.

Self-healing test scripts address one of the most persistent headaches in traditional automation. When a UI element changes, locators break, and tests fail even though the underlying functionality still works. AI can recognize when an element has moved or been renamed and adjust automatically. This does not mean maintenance disappears. Significant application changes still require human attention. But the routine churn of minor UI updates breaking tests is dramatically reduced, which frees up time for more valuable work.

Visual regression testing is another area where AI performs well. Comparing screenshots to catch unintended changes is tedious for humans and well-suited to machine analysis. AI-powered visual testing can identify meaningful differences while filtering out noise like minor rendering variations across browsers or dynamic content that changes legitimately between runs.

Test optimization and prioritization benefit from AI’s ability to analyze patterns across large datasets. Which tests are most likely to catch defects given recent code changes? Which tests are redundant? Which have become flaky and should be retired? These are questions that would take humans a long time to answer by digging through execution history, but AI can surface insights relatively quickly.

Where AI Falls Short

The areas where AI struggles are just as important to understand. Deploying it against problems it cannot solve well wastes time and erodes trust in the technology.

Deep domain knowledge remains difficult for AI to replicate. Lew makes this point often: our clients sometimes tell us we know their software better than they do, because we have spent years not just learning the application but understanding the domain it operates in, whether that is banking, healthcare, or something else. A tester who has spent a decade working with medical records software understands which edge cases carry regulatory risk and what realistic clinical data looks like. AI does not have access to that kind of embedded experience. It might generate test cases that look plausible but miss the scenarios that actually matter.

Complex multi-step workflows expose AI’s limitations. Testing that requires understanding how multiple system components interact over time, where the meaning of a result depends on what happened in previous steps, challenges models that process inputs without deep contextual awareness. There are simply too many permutations for AI to handle alone. Humans are still better at tracking state across complex sequences and recognizing when something subtle has gone wrong.

The oracle problem remains unsolved. AI can tell you what happened during a test. It cannot reliably tell you whether what happened is correct. Determining correctness requires understanding the intended behavior of the system, which is context AI does not have. A test might pass because the AI could not tell the difference between correct and incorrect behavior, or it might fail because the AI flagged something that is actually working as designed.

Exploratory testing does not translate well to AI. The skilled practice of investigating software to find problems that scripted tests would miss requires intuition, creativity, and the ability to follow hunches. Tools that claim to automate exploratory testing are generally doing something different, like generating random inputs or systematically varying parameters. That is not the same as the judgment-rich investigation that experienced testers do.

False positives remain a persistent problem. Lew considers precision the most important metric for evaluating AI testing tools: if 30 to 40 percent of the defects the tool flags turn out not to be real defects, that is a serious problem. You end up spending significant time investigating non-issues, which erodes the efficiency gains AI is supposed to provide. When evaluating tools, ask for precision data, and ask to see all the results, not just the cherry-picked successes.

“Probably the most important feature of a very successful AI testing tool is the precision or accuracy. If you’ve got 30-40% of the defects that the AI testing tool finds and they’re basically not defects, then that’s a problem.” — Phil Lew

AI as Supplement, Not Replacement

The framing that works best is AI as a supplement to human testing, not a replacement for it. AI handles certain tasks faster and at greater scale, which frees skilled testers to focus on work that requires judgment, creativity, and domain expertise.

Lew describes it this way: AI agents work around the clock and never get tired, so they can do useful preliminary work before human testers dig in. They can cover ground in areas like accessibility, performance, and security that many teams do not have the bandwidth to address thoroughly. But the deep bugs, the ones that require understanding how real users actually work with the software, still get found by humans.

“AI agents are out there doing their thing and they never get tired. So it’s great that they can do some work prior to the work that we do, which is more value added in finding some of the really deep bugs.” — Phil Lew

The practical implication is that you should deploy AI for high-volume, repetitive work: generating initial test case candidates, running visual regression across hundreds of pages, maintaining locators as UIs evolve, analyzing test execution history. These are areas where AI adds clear value and where human time is better spent elsewhere.

The work that stays human includes defining what quality means for your application, understanding which risks matter most, interpreting ambiguous results, and exploring the software in ways that scripts and models cannot. It also includes reviewing AI output, because the probabilistic nature of the technology means that validation is not optional.

Evaluating Claims Against Reality

When vendors describe what their AI can do, the categories above provide a useful filter. Claims about test case generation, visual regression, self-healing, and optimization line up with what AI demonstrably does well. Claims about replacing domain expertise, automating exploratory testing, or eliminating the need for human review should prompt skepticism.

When you evaluate tools, focus on precision. Ask for data from representative use cases, not curated success stories. Ask to see all results from a test run, not just the highlights. The difference between a tool with 90 percent precision and one with 60 percent precision is the difference between a productivity gain and a new source of overhead.