A QA engineer gives an AI assistant a password reset story and asks it to create test cases.

The answer looks thorough. It includes expired links, reused links, invalid email addresses—and this test:

“After three failed reset attempts, verify that the account is locked.”

There is only one problem. The story says nothing about locking the account.

A separate policy may define such a rule. But the assistant has not found or shown that policy. It has presented a convincing detail without supporting evidence.

This is the practical face of an AI hallucination: a false or unsupported claim presented as though it were an established fact.

The word hallucination is widely used, although it is not perfect. NIST uses the term confabulation for confidently stated false content.

For QA, the risk goes beyond an embarrassing answer. An unsupported rule can become:

  1. an expected result;
  2. a test case;
  3. a failed test; and
  4. a defect report that sends the team investigating behaviour the product may never have promised.

Why can an incorrect answer sound so confident?

A language model learns patterns from large amounts of text. It uses those patterns, along with your input, to generate a response.

But these are two different tasks:

  • producing fluent, convincing language; and
  • verifying a claim against your current product specification.

A model can produce a fluent answer without reliably completing the second task.

If you do not provide the requirement, the model cannot be expected to know your private product rules. Even when you provide the requirement, it may still add a familiar rule that the document does not contain.

In other words:

Having access to the requirement and remaining faithful to it are separate questions.

Exact expiry times, customer-specific configurations, recent release rules and decisions made in internal meetings cannot be reliably inferred from general language patterns.

Research on hallucinations also points to the way models are trained and evaluated. If giving a specific answer is rewarded more than saying “I don’t know,” guessing can appear more useful than admitting that evidence is missing.

This is why a polished answer is not evidence that the claim is correct.

Not every incorrect answer has the same cause

An AI response may be wrong because:

  • the model did not have the necessary information;
  • it misunderstood information you provided;
  • it overlooked a condition buried in a long document;
  • a search system retrieved the wrong passage; or
  • it correctly repeated an outdated source.

The last case is especially important for QA.

An answer can accurately reflect the document the assistant read and still be wrong for the release you are testing.

You must check both:

  1. whether the claim is supported by a source; and
  2. whether that source is current and authoritative.

What does hallucination look like in QA work?

An invented business rule

The lockout rule in our password reset example is unsupported.

An unsafe test says:

“After three failed resets, verify that the account is locked.”

A safer response says:

“The story does not define a lockout rule. Confirm with the product owner before adding this test.”

This keeps a potentially useful risk visible without presenting it as an approved requirement.

An assumed API response

An AI assistant may generate an API test that expects a 201 response because 201 Created is common for creation requests.

But your API contract may specify 202 Accepted.

The generated test can look technically correct while checking the wrong expected behaviour.

An unsupported root cause

An assistant may claim that a test failed because a token expired—even though it has not inspected the response, server log or trace.

That explanation may be plausible. Plausibility is not evidence.

Across all three examples, the same thing has happened:

The answer moved from evidence to assumption without making that transition visible.

Code can be correct and still test the wrong thing

AI-generated code requires two different reviews.

First, check whether the code is technically valid:

  • Does it compile?
  • Does the helper method exist?
  • Is the configuration setting valid?
  • Does the test run?

These checks can reveal invented methods, parameters or framework features.

Second, check whether the code reflects the correct product behaviour:

  • Is the expected result supported by a requirement?
  • Is the rule current for this release?
  • Is the assertion testing the intended behaviour?

A test can compile and pass while enforcing a business rule that the product never defined.

Execution proves that the code ran. It does not prove that the expected result is correct.

Hallucination and missing coverage are different problems

Suppose the AI lists valid password reset tests but forgets the single-use rule.

The response may contain no false statement. But it is still incomplete and gives you poor coverage.

QA must therefore ask two separate questions:

  1. Support: Is every claim backed by evidence?
  2. Completeness: Did the response include the important conditions?

An answer can be fully supported but incomplete. It can also cover many areas while including invented expected results.

Both problems matter.

More context helps—but it is not verification

Relevant context helps.

Providing the current requirement, API contract or logs gives the assistant evidence it can use. A retrieval system can find relevant material. Connected tools may allow the assistant to inspect an issue, page or build result.

But access to evidence does not guarantee correct use of evidence.

The assistant may retrieve the wrong version, cite a passage that does not support its conclusion, miss an exception elsewhere or misinterpret the material it found.

Consider two versions of the password reset requirement:

  • The old version says the link expires after 30 minutes.
  • The approved version says it expires after 15 minutes.

If the assistant retrieves and cites the old version, it has not invented the number. But the proposed boundary tests are still wrong for the current release.

That is why “It has a citation” cannot be the final QA check. Open the source, check its version and compare the assistant’s specific claim with what the source actually says.

Asking “Are you sure?” is not a substitute for this check. The assistant may change its answer or repeat it with greater confidence, but neither response provides new evidence.

A better approach is to:

  1. provide the authoritative source;
  2. ask the AI to separate documented rules from suggestions;
  3. ask it to identify information that is missing; and
  4. verify important claims yourself or against a trusted system.

When the source is silent, the useful response is a clarification question—not a newly invented expected result.

How should QA respond?

QA encounters hallucination risk in two roles:

  • when using AI to perform QA work; and
  • when testing an AI feature that users will rely on.

When using AI for QA work

Pause whenever an AI suggestion could become a team decision.

If the assistant proposes a test, ask:

“Which requirement supports this expected result?”

If it explains a failed build, ask:

“Which log line or trace supports this conclusion?”

If it says that a browser flow passed, inspect:

  • the actions it performed;
  • the page state it observed; and
  • the assertions it checked.

A tool trace may show which action was performed, while a screenshot may show the resulting page state. Neither one, by itself, proves that the assistant’s conclusion was justified.

When testing an AI feature

Create a small evaluation set containing questions, their expected responses and the source documents that should support those responses. For a question the documentation cannot answer, the expected response should be an explicit statement of uncertainty rather than a guessed fact.

Include cases that encourage guessing:

  • a requirement that does not contain the answer;
  • two conflicting document versions;
  • an ambiguous boundary; and
  • a document containing a plausible but irrelevant rule.

Evaluate four things separately:

  1. Evidence selection: Did the system use the correct and current source?
  2. Claim support: Is each statement in the answer supported by that source?
  3. Completeness: Did the answer omit any critical condition?
  4. Uncertainty: Does the system clearly admit when the available evidence does not determine the answer?

The acceptable answer depends on the task

Different QA activities need different levels of evidence.

Activity What may be acceptable?
Brainstorming risks Clearly labelled suggestions may go beyond the specification.
Creating approved test cases Expected results must be supported by current requirements.
Diagnosing an incident Conclusions need observed data and honest uncertainty.
Reviewing requirements Missing information should become questions, not invented rules.

Use this three-part check during every review:

  • What is documented?
  • What is only a suggestion?
  • What still needs confirmation?

The mistake is not suggesting an idea. The mistake is allowing that idea to silently become a fact.

The question to carry into your next AI-assisted task

Hallucination is not a simple wording error that proofreading will always catch.

It may appear as:

  • a believable business rule;
  • a persuasive root cause;
  • a confident summary; or
  • a neat test assertion.

More context, retrieval and connected tools can reduce the risk. They do not remove the need to verify what the assistant says and which sources it used.

Before accepting any AI-generated QA artefact, ask:

“What evidence supports this expected result—and what did the AI assume?”

That question can stop an invented detail from becoming a permanent part of your test suite.

← Back to Blog