Published: March 6, 2026
Updated: March 19, 2026
The conversation about AI in testing often jumps straight to tool selection. Which vendor should we go with? What features do we need? These are reasonable questions, but they come too early. The more important question is whether your organization has the prerequisites in place for AI to actually deliver value, or whether adopting it now will just surface problems you need to solve first.
AI testing tools amplify whatever they encounter. If you have clear requirements, well-documented test cases, and mature processes, AI will reflect those strengths in its output. If you have ambiguous requirements, tribal knowledge that lives in people’s heads, and ad hoc practices, AI will amplify those weaknesses. The tool does not fix underlying problems. It makes them more visible.
Phil Lew sees this play out regularly with clients who expect AI to interpret vague inputs and figure things out on its own. That approach, he says, is almost disastrous. It is like asking ChatGPT a general question when you have something very specific in mind. You get answers that are not even close to what you wanted.
The World Quality Report 2024-25 found that only 29 percent of organizations have fully integrated generative AI into their test automation, while 42 percent are still exploring. The gap often comes down to readiness. Organizations that have their fundamentals in order can move forward. Organizations that do not end up with disappointing pilots and abandoned initiatives.
When we work with organizations considering AI adoption, we start with questions that get at actual readiness rather than aspirations.
What is the state of your test documentation? Do you have documented test cases with explicit acceptance criteria, or does your team rely on informal understanding of what tests should verify? AI tools interpret inputs literally. If your test cases require context that exists only in someone’s head, AI will produce results that are technically responsive but practically useless.
Lew puts it directly: the more concise and unambiguous your test cases are, the better AI will perform. If you have junior testers who do not know how to write clear test cases, or workflows that require institutional knowledge to understand, that is a sign you may not be ready.
“The more direct you are and concise and unambiguous in putting in the test cases, the better it’s gonna do. If they’ve got juniors that don’t really know how to write test cases or have very ambiguous workflows, then that’s an indication they may not be ready.” — Phil Lew
What does your test data situation look like? AI tools need access to test data that is realistic, properly structured, and available programmatically. Many organizations discover their test data is scattered across systems, inconsistently formatted, or dependent on manual setup. If getting test data ready requires someone to spend half a day on manual preparation, AI tools will struggle to operate effectively.
How mature are your processes? Is there a defined workflow for how test cases get created, reviewed, and maintained? Are test results tracked systematically, or do they live in scattered reports and emails? AI adoption introduces new activities: reviewing AI output, configuring tools, integrating results into existing systems. Organizations with mature processes can incorporate these smoothly. Organizations with informal processes find that AI creates confusion about who is responsible for what.
Who will work with the AI? This is a critical question that often gets overlooked. Someone needs to review AI-generated test cases and decide which ones are valid. Someone needs to investigate when AI flags a potential defect. Someone needs to configure the tool, tune its behavior, and troubleshoot when things go wrong.
These tasks require experience and judgment. Junior testers typically lack the background to evaluate whether AI output is correct. They may accept suggestions that an experienced tester would recognize as flawed, or reject valid suggestions because they do not understand the reasoning. You need senior people involved, and you need them to have time for this work.
Certain patterns consistently predict difficulty with AI adoption.
High activity, low measurement. Some organizations run a lot of tests without ever asking whether those tests are catching real problems or influencing release decisions. Lew has seen this repeatedly: teams focused on how many test cases they automated or how many passed, without measuring whether any of it was actually improving quality. If you do not know what your current testing is accomplishing, you will not be able to tell whether AI is making it better.
Test suites that only grow. Over time, test suites accumulate cases that no longer provide value: tests for features that have changed, redundant tests covering the same ground, tests that always pass and never find anything. If your organization lacks the discipline to prune and maintain test suites, adding AI-generated tests will make the problem worse, not better.
Unrealistic expectations about what AI will do. Organizations that expect AI to replace testers, operate autonomously, or eliminate the need for human review are setting themselves up for disappointment. The technology is not there. Lew points out that juniors lack the experience and knowledge to tell AI what to do or evaluate its results effectively. If your plan depends on AI working without senior oversight, it will not work.
Organizations that are prepared for AI adoption tend to share certain characteristics.
They have documented test cases with clear acceptance criteria. Someone unfamiliar with the application could read a test case and understand what it is supposed to verify.
They have test data that can be provisioned programmatically. Setup does not depend on manual steps or knowledge that lives only in specific people’s heads.
They have defined processes for how testing work flows through the organization, with clear expectations about creation, review, execution, and maintenance.
They have experienced testers available to evaluate AI output critically. These are people who understand the application well enough to recognize when AI suggestions are valid and when they miss the mark.
They have realistic expectations. They understand that AI supplements human testing rather than replacing it, that output requires review, and that benefits take time to materialize.
If your assessment reveals gaps, the question is whether to address them before adopting AI or try to work on both at the same time.
Some gaps are foundational enough that you need to fix them first. If test cases are undocumented or require extensive tribal knowledge to interpret, AI will produce output that no one can validate. The work of documenting and clarifying test cases needs to happen before AI adoption, not during it.
Other gaps can be addressed in parallel. Process improvements can evolve alongside early AI pilots. The pilot itself may reveal issues that were not visible before, which gives you motivation and direction for improvement.
The key is being honest about where you are starting from. Organizations that acknowledge their gaps and make plans to address them do better than organizations that proceed with inflated assessments of their own readiness. Vendors will not tell you that you are not ready. They have every incentive to sell regardless of your situation. The honest assessment has to come from inside.
The readiness conversations we have with clients often produce surprises. Organizations that seemed well-prepared discover gaps they had not considered. Organizations that doubted their readiness turn out to be further along than they thought. What we bring to these conversations is pattern recognition from working across many different situations. We know which gaps tend to be critical and which can be managed, what realistic timelines look like, and what expectations are achievable.
See the full picture Understanding what AI can and cannot do is one part of deciding whether and how to adopt it. The pillar guide covers readiness, evaluation, economics, and implementation.
Explore AI-Informed QA: Going Beyond the Hype
Talk through your context Where AI helps and where it falls short varies by application type, team composition, and what you are trying to accomplish. A conversation can help clarify what applies to your situation.
Looking for more insights on Agile, DevOps, and quality practices? Explore our latest articles for practical tips, proven strategies, and real-world lessons from QA teams around the world.