Consider this hypothetical QA example. You ask an AI assistant:

“What should we test in the password reset flow?”

It suggests expired links, invalid email addresses and reused links. These are sensible ideas. But your product has two specific rules:

  • The reset link expires after 15 minutes.
  • The link can be used only once.

The assistant has not seen those rules. Its answer sounds useful, but it does not contain the details needed to create correct tests for your product.

Now imagine that the assistant first finds the approved password reset requirement, reads it and then answers.

That is the basic idea behind RAG, or retrieval-augmented generation.

In simple terms, RAG means:

Find the relevant information first. Then use it to answer the question.

For QA professionals, this matters because a model should not be treated as an authoritative source for the current internal rules of a product. Relevant information may be spread across requirements, acceptance criteria, release notes, defect reports and internal documents.

RAG gives an AI application a way to retrieve project information and supply it as context while preparing an answer.

What does RAG actually do?

In a common application-level RAG pattern, the name can be explained through three steps:

  1. Retrieve: Find information related to the question.
  2. Augment: Add that information to the user’s question.
  3. Generate: Ask the AI model to produce an answer using both.

Before this can happen, documents often need to be prepared for searching. For example, some RAG implementations split documents into passages, create vector representations called embeddings and store them in an index. This is a common implementation, not a requirement that every RAG system must follow.

When a user asks a question, the system:

  1. searches the connected information;
  2. selects passages that appear relevant;
  3. supplies those passages to the AI model; and
  4. asks the model to answer the question.

Different implementations use different retrieval methods and data stores. For a first understanding of RAG, the important point is that external information is retrieved and made available during generation.

The 2020 paper that introduced the term retrieval-augmented generation described specific models that combined a pretrained generator with a dense index of Wikipedia. The term is now also widely used for application pipelines that retrieve private or current information and place it in a model’s prompt. This article explains that common application pattern rather than every RAG architecture.

See the difference in one hypothetical QA example

Suppose the approved requirement says:

“A reset link expires after 15 minutes and can be used only once.”

Without that requirement, the assistant may answer:

“Test expired links and reused links.”

This is a reasonable general suggestion, but it does not reflect the exact product rules.

If the RAG system retrieves the approved requirement, the assistant can answer:

“Test the link before and after 15 minutes. After using it successfully, try the same link again and verify that it is rejected.”

The second answer is more specific to the hypothetical product because the stated tests use details from the supplied requirement.

However, retrieval does not finish the QA work. The requirement does not clearly define what should happen at exactly 15 minutes. A careful response should identify that ambiguity rather than invent the expected behaviour.

RAG can make retrieved source material available to the model. The model still has to interpret that material correctly, and QA still has to determine whether it is sufficient and authoritative.

Why would a QA team use RAG?

One practical reason is simple: the information needed for a QA decision may be spread across several project sources.

Consider a checkout feature. The requirement says that customers can use a discount code. A release note adds another rule: the discount cannot be combined with a loyalty reward.

If an assistant receives only the requirement, it does not receive the combination restriction stated in the release note. If a RAG system retrieves both documents, the model has both pieces of evidence available. It may still interpret them incorrectly, so retrieval improves the basis for an answer rather than proving that the answer is correct.

The same need appears in other QA activities:

QA activity Information that may be spread across sources How RAG can help
Requirement analysis Acceptance criteria and a shared business policy Retrieve both so the answer considers the feature rule and the wider policy.
Defect investigation The reported symptom, a recent release change and a known issue Bring the relevant facts together before suggesting areas to investigate.
Release-impact analysis The previous rule and the approved new rule Surface the change so QA can identify behaviours and tests that may be affected.

In each activity, RAG can help assemble relevant information. It does not make the final QA decision. A defect’s root cause still needs evidence from logs, traces or observed behaviour. A release comparison still depends on choosing the authoritative document versions.

Across these examples, the useful capability is not simply producing more text or test cases. RAG can provide project-specific context and, when the application exposes source references, make important claims easier to trace to retrieved material.

That changes the review question from:

“Does this sound reasonable?”

to:

“Which project information supports this answer, and is that information current?”

What RAG changes—and what it does not

In the application pattern described here, RAG changes the context available to the model at answer time. Supplying retrieved context is different from changing the model’s trained parameters.

This means a RAG system may use an updated requirement without retraining the underlying model. However, the new rule will influence the answer only if the updated document is available, searchable and retrieved for that question. If the system selects an older version, the answer may still be outdated.

Retrieval and generation do not, by themselves, edit external systems. An application may retrieve a Jira story and explain its testing implications, but creating or editing a ticket requires an integration that can perform that action and an authorization decision that permits it.

These boundaries are useful when evaluating an AI product: access to project knowledge, model training and the ability to take action are three different capabilities.

What can go wrong?

For the application pattern used in this article, answer quality depends on at least two stages:

  1. the system must retrieve suitable evidence; and
  2. the model must use that evidence correctly.

A failure in either stage can contribute to a wrong or incomplete answer.

1. The system retrieves an outdated or wrong source

Suppose an old requirement says that reset links remain valid for 30 minutes. The approved version says 15 minutes.

If the system retrieves the old requirement, the answer may accurately repeat the 30-minute rule and still be wrong for the current release.

A document can be relevant to the wording of a query without being the approved source for the current release. Relevance, currency and authority therefore need separate checks.

2. The system retrieves only part of the rule

The selected passage may describe the 15-minute expiry but omit the single-use condition stored elsewhere in the document.

The answer may therefore be supported by the retrieved text and still provide incomplete coverage.

QA must check not only whether the answer is grounded, but also whether the retrieved evidence contains all important conditions.

3. The system finds no useful evidence

The supporting information may be absent from the connected documents, not yet indexed or not ranked highly enough to be selected for the answer. Exact causes depend on the retrieval method and configuration.

If the assistant fills this gap with a plausible guess, missing information becomes false confidence.

For a product-specific question with no supporting evidence in the available sources, a safe expected behaviour is to report that the information was not found rather than invent a product rule.

4. The model goes beyond the evidence

Even when the correct passage is retrieved, the model may add an unsupported exception, overlook a condition or answer a different question.

A source reference makes inspection possible. Its presence alone does not show that the cited passage supports every claim in the answer.

Open the source and compare the claim with what the document actually says.

5. Access controls fail

Project documents may contain confidential information. A production RAG application needs authorization controls that prevent retrieval of material the requesting user is not permitted to access.

The exact enforcement mechanism depends on the system. Examples include document-level access controls and query-time filtering based on the authenticated user. Filtering based only on an unverified user identity is not, by itself, complete authorization.

6. Retrieved content tries to influence the assistant

A connected document can contain malicious instructions intended to influence the AI system. Security guidance calls this indirect prompt injection when the instructions arrive through external content rather than directly from the user.

Retrieved content should therefore cross a trust boundary: the application should not automatically treat text inside a retrieved document as an instruction with the same authority as its system rules.

This is separate from access control: the user may be allowed to read a document, while the content inside it may still be unsafe for the assistant to obey.

How should QA test a RAG assistant?

Test retrieval and answer generation separately. A final response alone may not reveal whether the system retrieved poor evidence or generated an answer that misused good evidence.

Start with a small evaluation set. For every question, define:

  • the expected answer;
  • the source that should support it; and
  • the behaviour expected when no answer is available.

Include clear questions, ambiguous questions, conflicting sources and questions that the documentation cannot answer.

Step 1: Check what it retrieved

Ask:

  • Did it find the relevant document?
  • Did it select the current and authoritative version?
  • Did it retrieve enough text to include conditions and exceptions?
  • Did it avoid documents the user cannot access?

If the correct evidence was not retrieved, changing only the instructions used after retrieval does not correct the missing evidence. The retrieval stage must also be investigated.

Step 2: Check what it concluded

Ask:

  • Does the answer follow the retrieved evidence?
  • Does it address the user’s question?
  • Did it invent a rule or exception?
  • Did it omit an important condition?
  • Does each reference support the claim attached to it?

A fluent answer can still be unsupported, incomplete or wrong.

Step 3: Test failure conditions deliberately

Use cases designed to expose weaknesses:

  • two conflicting versions of the same requirement;
  • a question whose answer is absent from the documents;
  • an important exception stored in a different passage;
  • a restricted document requested by an unauthorised user; and
  • a document containing an instruction that the assistant should not follow.

For each case, define the expected safe behaviour before running the test.

Step 4: Test document updates

Change an important source and repeat the same questions.

Check:

  • when the new version becomes searchable;
  • whether the old version is still retrieved; and
  • whether the answer changes to reflect the approved update.

Using information added after model training is a commonly stated benefit of RAG. Whether a particular system delivers that benefit depends on its ingestion, indexing and retrieval process, so test the complete update-and-answer flow.

The idea to remember

RAG allows an AI application to retrieve project information for use during generation. This can make an answer more specific and relevant to the product, but the result depends on retrieval and generation quality.

But RAG does not guarantee that the system found the right source, retrieved the complete rule or used the evidence correctly.

For QA, the essential review remains:

First check what it found. Then check what it concluded.

← Back to Blog