Ask an AI assistant to summarize a document. It gives you five key points. Ask again, using the same words, and it gives you seven—while leaving out one point you thought mattered. Which answer should you trust?
An AI assistant does not keep one ready-made answer for your question. It creates a response each time, and the words you type may be only part of what it uses. A changed answer can be harmless, useful or a sign that something important was missed. Here is how to tell the difference.
It creates an answer each time
When an AI assistant writes, it chooses each next piece of text from several possibilities. It can take one reasonable route through your question on the first attempt and another on the second. A setting called temperature can affect how much the choices vary. Many people using a chat app never see or control that setting. Even when a developer turns it down, identical results are not guaranteed.
This is easy to see with an open question. Ask for headlines for an article and you might receive several good options. Ask again and you may receive a different set. That is useful: you asked for ideas, and there is no single required headline.
Now ask the assistant to summarize a policy. If one answer says approval is required and another says it is optional, check the policy itself. Different wording is fine; a different rule is a problem. The key is to distinguish a new way of saying something from a new claim about what is true.
The same words may carry different context
Try writing, “Summarize this document” in an existing chat. The assistant may have access to the document you shared earlier, your previous questions, and its own earlier answers. Type those exact words in a fresh chat and that document may be absent. You repeated your sentence, but you did not repeat the full request the assistant received.
But a fresh chat is not always a blank slate. Some assistants can use saved preferences or relevant information from past chats when personalization is turned on. For instance, one account may remember that you prefer concise answers while another does not. That can change the response even when both people start new chats and type the same question. If you want a fair comparison, check whether either account is using memory or saved instructions.
An application can also add instructions behind the scenes or provide an attached document. A QA assistant configured to focus on security may give a different test list from one configured to focus on usability. If you are comparing two runs in an app your team built, the application instructions and files matter as much as the sentence entered by the user.
Context can change within a long conversation too. Some applications leave out older messages when there is too much for the model to process at once. An instruction you gave much earlier, such as “Use only the acceptance criteria in the ticket,” might no longer reach the model. The next answer may include tests based on assumptions that earlier answers avoided. When an answer suddenly changes after a long chat, try a new one with the complete requirement and the important instructions together.
Your question may leave room for several answers
“Write comprehensive tests for login” sounds clear until you ask what comprehensive means. Should the answer cover password rules, account lockout, accessibility, API errors and recovery? Are these in scope for the feature you are testing? The assistant may make different choices each time because you have left those decisions open.
You can narrow that room without turning your request into a long technical prompt. Supply the actual requirement. Say which areas you want covered. Tell the assistant to list uncertainties separately instead of inventing product behavior. For example: “Based only on the acceptance criteria below, suggest positive, negative and boundary tests. If a rule is missing, write a question instead of assuming an expected result.” That will not make every answer identical, but it gives you a clearer basis for judging each one.
The information it reads may have changed
Some assistants do more than answer from the conversation. They can search the web, read a Jira ticket or retrieve documents from a project library. This is useful when the answer needs current information. It also means the same question can be answered using different material.
Suppose a Jira issue gains a new acceptance criterion between Monday and Tuesday. A test list generated on Tuesday ought to include something the Monday list did not. Even when the issue itself has not changed, the assistant might retrieve a different supporting document or decide to use a different available tool. To understand the difference, ask what it read. A saved prompt without the ticket version or retrieved documents tells only part of the story.
Sometimes the answer did not finish
An app can set a limit on how much text an assistant may generate. An answer that stops at test case eight may have reached that limit before it could write cases nine through twelve. It may be visibly cut off, or it may end neatly enough that you do not notice. If the app exposes a completion status, check whether the response finished before concluding that the assistant chose a smaller set of tests.
The AI behind the app may have changed
AI services release new model versions, and an app may change which model it uses. Providers also warn that changes to the system running a model can affect repeatability. This matters most when you compare answers weeks or months apart, or when a team says, “This prompt used to work.”
If your team depends on a repeatable workflow, keep a record of the model, the requirement, the instructions and any documents used. Developers can also record settings and use a fixed seed where a service supports one. These steps make differences easier to investigate, but they do not promise that the same words will always be returned.
How to get more consistent answers
You cannot guarantee identical wording every time. You can reduce avoidable differences and make the answers easier to compare. Start with the steps you can control; the final rows are for teams building an AI application.
| Practice | What to do | Why it helps |
|---|---|---|
| Give the same source material | Include the exact requirement or document each time, or link to a fixed version. | The assistant works from the same facts. |
| Define the scope | Say what to cover and what to leave out. Ask it to flag missing rules instead of guessing. | Fewer decisions are left open to interpretation. |
| Specify the output | Request the same headings or fields, such as scenario, source rule and expected result. | Answers are easier to compare even when the wording differs. |
| Show an example when useful | Provide one example of the kind of answer you want. | The example makes the expected level of detail and structure clearer. |
| Check conversation and personalization | For a fair comparison, use a fresh chat and check whether memory or saved instructions are active. | Earlier or remembered information may otherwise change the full input. |
| Keep external information steady | If the assistant reads tickets, files or web results, record what it used and when. | A changed source can produce a changed answer for a good reason. |
| Keep the model and settings steady | In an application you control, use the same model version and generation settings across runs. | You avoid changing the system while trying to compare its answers. |
| Make sure the answer finished | If your application reports incomplete responses, check the status and allow enough output for the task. | A cut-off answer can look like missing coverage. |
| Review more than one run | Compare the answers with the source, especially for important facts or decisions. | Repetition reveals differences; the source helps you judge them. |
These practices make the task clearer and the comparison fairer. They cannot turn an incorrect answer into a correct one merely by making it repeatable.
So which answer should you trust?
Consider a QA engineer who asks for login test cases. The requirement says: “After five failed attempts, lock the account for 15 minutes.” One answer tests what happens on the fifth attempt. Another misses that boundary and adds an administrator unlock test, even though the requirement says nothing about administrators. The fifth-attempt test follows the stated rule; the administrator test needs clarification. The second answer is not better simply because it contains more tests.
If you want to check a recurring QA task, run it more than once with the same full requirement. Keep the source documents and model consistent where you can. Ask the assistant to connect each proposed test to a stated rule and to separate unanswered questions. Then compare the tests against the requirement. This can reveal both missing coverage and unsupported assumptions. The aim is to make the review dependable, even when the wording varies.
Different answers can bring useful ideas to the surface. A repeated answer can also repeat the same mistake. When two answers disagree, look at the document or requirement behind them and ask three questions:
- Did only the wording change? Two answers may express the same fact in different words.
- Did anything important disappear or appear? Check whether a key point or necessary test is missing from one answer.
- Did the facts or expected behavior change? If an answer adds a rule that the source never stated, treat it as a question to resolve.
Trust the answer that the source supports, not the one that sounds more confident or contains more words.