AI can generate 100 test cases in minutes.
That sounds impressive. But it raises a more important question:
Are they the right 100 tests?
Speed is useful. Volume can look convincing. Neither one guarantees quality.
In Part 1, we examined how AI is changing traditional QA activities. This part focuses on the skill that becomes more valuable when content is easy to generate: judging whether it is correct, relevant and sufficient.
Test generation is not quality assurance
A test case is an artefact. Quality assurance is a purpose.
Generating tests does not automatically answer:
- Did we understand the real requirement?
- Is an important business rule missing?
- Which failure would hurt customers most?
- Are the expected results correct?
- Does the test reflect how the complete system behaves?
- What evidence is enough to accept the remaining risk?
AI can create a polished list even when its understanding is incomplete. Professional formatting can make weak output appear trustworthy.
Do not measure AI by the number of tests it produces. Measure whether it helps uncover meaningful risks.
A simple example: transferring money
Suppose the requirement says:
A customer can transfer money to another account.
AI may quickly propose:
- Transfer with valid account details
- Transfer with an invalid destination account
- Transfer with insufficient balance
- Transfer the minimum and maximum amounts
- Verify confirmation and transaction history
These are useful tests. They are also only a beginning.
An experienced tester may ask:
- What if the customer submits the transfer twice?
- Can two simultaneous requests spend the same available balance?
- What if the debit succeeds but the credit fails?
- What if the network drops after submission?
- Is the daily limit applied to one transaction or the combined total?
- Can the recipient be changed by modifying an API request?
- What audit evidence and recovery process are required?
The difference is not typing speed. It is the ability to understand systems, risk and consequences.
Where AI-generated tests can fail
1. Hidden assumptions
When information is missing, a model may fill the gap with a plausible assumption. It might assume that a transfer limit applies per transaction when the business means per day.
A useful QA response is not merely to correct the test. It is to expose the ambiguity and get the rule clarified.
2. Duplicate coverage
Ten differently worded tests may exercise the same condition. A long list can create an illusion of coverage without increasing confidence.
3. Missing context
The model may not know the organisation’s undocumented rules, past incidents, architecture or regulatory obligations. Its output can only reflect the context available to it.
4. Incorrect expected results
An expected result written confidently is not necessarily correct. If the source requirement is incomplete or wrong, AI can convert that weakness into many professional-looking tests.
5. Happy-path bias
Common scenarios are easy to generate. Failures involving timing, concurrency, recovery, state changes or multiple systems require deeper reasoning and better context.
6. Unmaintainable automation
AI-generated automation may work once while violating the project’s architecture or standards. It may introduce brittle selectors, fixed waits, duplicated utilities or weak assertions.
“The script runs” is not the same as “the automation is reliable.”
The new bottleneck is evaluation
When creating content becomes fast, reviewing it can become the expensive part.
Stack Overflow’s 2025 Developer Survey shows this tension: AI use is widespread, while distrust in the accuracy of AI output exceeds trust. Adoption does not remove the need for verification; it increases the need for people who can verify intelligently. Stack Overflow 2025 Developer Survey
QA professionals must evaluate:
- Accuracy: Is the output correct?
- Coverage: What important scenarios are missing?
- Relevance: Does each test address a meaningful risk?
- Traceability: Which requirement or risk supports it?
- Maintainability: Does generated code fit the framework?
- Safety: Has sensitive information been exposed?
- Value: Does it save effort after review and correction?
Blindly accepting AI output is dangerous. But manually reviewing everything without a strategy can remove much of the expected productivity gain. The skill lies in creating an efficient validation method.
Give AI the right context
Better prompts help, but prompting alone is not enough. Useful context may include:
- Business rules and acceptance criteria
- Architecture and API contracts
- Existing coverage and project standards
- Past defects and production incidents
- Security and compliance expectations
AI can also be instructed to separate facts from assumptions, identify unanswered questions, map tests to requirements and flag expected results that lack supporting information.
Better context does not guarantee correctness. It makes the output easier to evaluate and more likely to be useful.
Human judgement becomes more valuable
When AI can generate tests and code quickly, producing content becomes less difficult. Deciding whether that content is useful becomes more valuable.
The following human capabilities become stronger differentiators.
Product and domain knowledge
AI understands common patterns, but it does not automatically know:
- Undocumented business rules
- Past production incidents
- Customer expectations
- Regulatory obligations
- Operational workarounds
- Exceptions specific to the organisation
For example, AI may generate standard tests for cancelling an insurance policy. An experienced tester may know that the rules change after a claim is submitted or while a payment is being processed.
That knowledge can reveal risks no generic prompt will uncover.
Risk-based thinking
When AI can generate hundreds of tests, running everything is not always practical or valuable. Someone must decide:
- Which failures would cause the greatest damage?
- Which areas are most likely to fail?
- Which customer journeys are business-critical?
- What should be tested first when time is limited?
- Which risks can be accepted or deferred?
If only five tests can be executed before a release, selecting the right five is more valuable than generating another hundred.
AI can suggest priorities. A QA professional must validate whether they reflect the actual business risk.
Critical evaluation
AI output can be detailed, confident and professionally formatted while still being incomplete or incorrect.
QA professionals must be able to detect:
- Invented requirements
- Unsupported assumptions
- Incorrect expected results
- Duplicate tests presented as additional coverage
- Missing negative and boundary scenarios
- Automation that works but is difficult to maintain
- Suggestions that violate security or privacy expectations
The important skill is not accepting or rejecting AI automatically. It is knowing how to verify its output.
Exploratory investigation
AI works well when the problem and expected output are clearly defined. Real defects are not always found that way.
During exploratory testing, a tester observes unexpected behaviour, asks a new question and changes direction. One unusual response may lead to another user flow, API call or deeper investigation.
AI can suggest ideas and analyse evidence, but the tester provides the curiosity, purpose and judgement that guide the investigation.
System thinking
A feature may work correctly on one screen and still fail across the complete system.
A payment may appear successful in the user interface while:
- The account is debited twice
- The database contains an incomplete transaction
- A downstream service never receives the event
- The confirmation is sent before completion
- A retry creates a duplicate order
- Recovery fails after a partial outage
QA professionals must look beyond individual screens and scripts. They need to understand how APIs, databases, queues, permissions, external services and recovery mechanisms interact.
AI can analyse each component. Human judgement is needed to identify which connections and failures matter most.
The shift in value
| Earlier emphasis | Growing emphasis |
|---|---|
| Producing more test cases | Selecting meaningful coverage |
| Following documented steps | Investigating unexpected behaviour |
| Checking individual features | Understanding end-to-end risk |
| Writing automation code | Evaluating and improving generated code |
| Reporting failures | Explaining their business impact |
| Accepting requirements | Challenging assumptions and ambiguity |
AI can increase the speed of testing activities. Human judgement ensures that the increased speed produces useful quality evidence.
When AI can generate more, QA professionals must become better at deciding what matters.
A practical human–AI review loop
1. Understand and expose ambiguity
Identify the goal, users, dependencies, business rules and missing information before generating tests.
2. Generate structured output
Ask for assumptions, risks, traceability, priority and clear expected results—not merely a long list.
3. Challenge the result
Look for omissions, inventions, duplicates, weak priorities and failures across system boundaries.
4. Add expertise and measure value
Include domain knowledge, past failures and customer behaviour. Measure useful coverage, correction effort, defects found and total time saved.
Use AI as a challenger, not only a writer
Many teams use AI only to produce more content. Its more valuable role may be to challenge existing work.
Ask it to:
- Find contradictions in a requirement
- Identify missing negative scenarios
- Review an existing test suite for gaps
- Compare tests with API specifications or find weak automation assertions
- Suggest failure and recovery scenarios
- Critique a proposed test strategy
This changes AI from a test-case factory into a thinking aid.
AI can expand the questions. QA expertise decides which answers can be trusted.
AI can recommend a test, but it cannot accept organisational accountability for a missed financial risk, privacy breach or unsafe release. Humans must still decide whether the evidence is reliable and the remaining risk is acceptable.
Quality expertise combined with AI capability offers the strongest position.
The next change is bigger than test generation
Software itself is changing. Applications will increasingly contain agents that interpret goals, make decisions, use tools and take actions. Their behaviour may vary even when the input appears similar.
Testing such systems requires new approaches to correctness, reliability, security, performance, cost and control.
In Part 3, we explore this emerging opportunity: from testing software to testing AI agents.
Ask yourself
When AI gives you a polished test suite, do you know how to prove what it missed?