AI is becoming part of everyday QA work. Testers are using it to review requirements, create test cases, generate automation code, investigate failures and prepare reports. QA professionals are also starting to test applications powered by AI.
You do not need to become a data scientist to work with AI. However, you should understand the common terms used in this field. These ten terms provide a practical starting point.
These ten terms are connected. An LLM processes information as tokens within a context window. Embeddings and RAG help it find relevant information. Tool calling and MCP connect it to external systems. Agents use these capabilities to complete tasks, while guardrails control what the system is allowed to do.
Guardrails protect the request, context, tools and result.
1. Large Language Model (LLM)
A Large Language Model, commonly called an LLM, is an AI model trained on a very large amount of text and other content. It learns patterns in language and generates a response by predicting what should come next. LLMs can summarise text, answer questions, generate code, classify information and perform many other language-based tasks. They do not store knowledge like a traditional database, and they do not understand a business in the same way an experienced employee does.1
For a QA professional, the LLM is the main AI model behind tools such as ChatGPT, Claude and Microsoft Copilot. If you provide a requirement and ask for test scenarios, the LLM identifies patterns in the requirement and produces a likely answer. The quality of that answer depends on the model, the instructions, the supplied context and the complexity of the task. Treat its response as a useful draft that must be reviewed, not as an automatically correct test artifact.
QA example: Give an LLM a password-reset requirement and ask it to identify positive, negative, boundary, security and usability scenarios. Review whether it invented rules that were not present in the requirement and whether it missed important domain behaviour.
Remember: An LLM generates likely answers; it does not guarantee true or correct answers.
2. Tokens
AI models do not read text exactly as people read words and sentences. They divide inputs and outputs into smaller units called tokens. A token may be a whole word, part of a word, punctuation or another small unit, depending on the model. Images, audio and other input types may also be represented and billed using model-specific token systems. Tokens matter because model limits, processing time and usage cost are often measured using them.2
For QA work, a requirement, prompt, source file, execution log and AI response all consume tokens. Copying an entire automation repository or a very large log into every prompt can increase cost and may hide the important evidence inside irrelevant content. A better approach is to provide only the files, log sections and business rules needed for the current task. This is not simply about using fewer words; it is about giving the model focused information.
QA example: Instead of sharing a 20,000-line execution log, provide the failed test name, error, relevant stack trace, screenshot summary, recent code change and the surrounding log entries.
Remember: Tokens are the units an AI model processes, and unnecessary tokens can increase noise, time and cost.
3. Context Window
The context window is the amount of information a model can process at one time. It may include your instructions, current question, earlier messages, attached documents and tool results. If a conversation becomes too large, the application may remove or summarise older information. A useful comparison is short-term working memory: information must be inside the available context for the model to use it directly.3
A large context window can accept lengthy requirements, code and logs, but larger does not automatically mean better. Important instructions can still be overlooked when they are surrounded by unrelated material. Information may also be removed, summarised or become unavailable when a conversation grows beyond the model’s limit. QA professionals should therefore organise context clearly, label the evidence and state which sources take priority.
QA example: When asking AI to review a checkout feature, supply the requirement, acceptance criteria, business rules, relevant API contract and existing test summary. Do not mix unrelated modules and old discussions into the same request.
Remember: The context window defines what the model can consider, while context quality determines how useful that information is.
4. Hallucination
A hallucination is an AI response that appears confident and believable but contains unsupported, inaccurate or invented information. Hallucinations can occur because an LLM generates language from learned patterns rather than performing a guaranteed factual lookup. Even a grammatically perfect answer can contain a nonexistent API, an incorrect assertion, an invented requirement or a false technical explanation. NIST uses the term confabulation for confidently presented false or erroneous content and treats it as a generative-AI risk.4
This is especially important for QA because generated output often looks professional. An AI-created test suite may include a business rule that the product never required. Generated automation might call a library method that does not exist. A failure analysis might confidently blame a timeout without examining the evidence. Verification should include traceability to requirements, compilation and execution of code, validation of expected results, and confirmation against trusted documentation.
QA example: If AI states that an account locks after three invalid passwords, ask it to show the exact source for that rule. If no approved requirement supports the claim, mark it as an assumption or remove it.
Remember: Confidence, detail and good language are not evidence of correctness.
5. Embeddings
An embedding converts content into a list of numbers that represents its important patterns and relationships. AI systems use these numbers to find content with a similar meaning, even when the words are different.5 For example, “login failure” may be considered related to “user cannot sign in.”
QA professionals may not create embeddings directly, but they will encounter systems that depend on them. An AI assistant could use embeddings to locate test cases related to a new requirement, find similar historical defects, group failure messages or retrieve relevant domain documentation. The important testing lesson is that similarity is not the same as correctness. Retrieval quality must be tested using realistic queries, synonyms, abbreviations, domain language and unrelated content.
QA example: Search a defect knowledge base for “payment deducted but order not created.” Verify whether the system retrieves semantically related transaction-consistency defects even when those exact words are absent.
Remember: Embeddings help machines compare meaning, but the closest result may still be irrelevant or incomplete.
6. Retrieval-Augmented Generation (RAG)
Retrieval-Augmented Generation, or RAG, allows an AI application to search an external knowledge source before answering a question. It retrieves relevant information and gives it to the LLM as additional context.6 The knowledge source could be a collection of documents, a database, a company portal or another approved system. RAG helps the application use private or current information that may not be present in the model’s original training.
RAG can reduce hallucinations, but it cannot guarantee correct answers. The system might retrieve the wrong document, miss an important rule, use an outdated version or produce a conclusion that the sources do not support. Testing a RAG application therefore requires more than checking the final response. QA should evaluate document ingestion, chunking, retrieval relevance, access permissions, source citations, answer faithfulness and behaviour when no reliable information is available.7
QA example: Ask an internal QA assistant about the refund period for a specific product. Confirm that it retrieves the correct and current policy, cites the relevant section and refuses to invent an answer if no approved policy exists.
Remember: RAG gives an LLM relevant reference material; it does not automatically make every answer correct.
7. AI Agent
There is no single industry-wide definition of an AI agent. In simple terms, an agent is an AI-powered system that can decide what steps to take, use available tools, examine the results and continue working towards a goal.8
An agent is different from a fixed workflow. In a workflow, code defines the sequence of steps. In an agent, the model can decide which step or tool should come next. Agents provide flexibility, but they can also increase cost, execution time and risk.
For QA, an agent might retrieve a Jira requirement, inspect related documents, generate test cases and add them back to Jira. A more advanced agent could run tests, examine failures and propose code changes. Testing must cover much more than the final answer: tool selection, tool parameters, permissions, intermediate decisions, retries, duplicate actions, stopping conditions, audit records and recovery after partial failure.
QA example: If a test-maintenance agent sees a failing locator, verify that it first confirms a genuine UI change. It should not silently weaken an assertion or change the test simply to make it pass.
Remember: An agent can choose and perform actions; this greater freedom requires stronger observation and control.
8. Tool Calling
Tool calling, also called function calling, is the mechanism that allows an AI model to request data or an action from an external system. The application tells the model which tools are available and what inputs each tool accepts. The model can then request a tool call, the application executes it, and the result is returned to the model. The LLM does not magically access Jira, a database or a browser; approved software integrations perform those actions.9
Tool calling changes AI from a text generator into part of an operational system. A QA team could expose tools such as get_requirement, search_test_cases, run_test_suite and create_jira_issue. Testers should validate correct tool selection, parameter values, input schemas, authentication, authorisation, error handling and side effects. Read and write operations should be treated differently because an incorrect search is inconvenient, while an incorrect update can damage real data.
QA example: When asked to create test cases for PROJ-123, confirm that the model calls the requirement tool with exactly that identifier and never writes to Jira before the required approval step.
Remember: The model requests a tool; the surrounding application controls and executes it.
9. Model Context Protocol (MCP)
The Model Context Protocol, or MCP, is an open standard that helps AI applications connect to external tools and data. For example, it can help an AI assistant work with Jira, GitHub, a browser or a database through a standard connection.10
The AI application acts as the MCP client. An MCP server makes approved tools, data or prompts available to that application. MCP and tool calling are related, but they are not the same. Tool calling is the general ability of a model to request a function. MCP provides a standard way for the application to discover and use that function. It does not give AI unrestricted access; the application still controls permissions and user approval.
MCP is important for QA because it can connect an AI assistant or agent to testing tools and evidence. Examples include browser automation, Jira requirements, GitHub repositories, API specifications, databases and test reports. QA professionals should test server identity, capability discovery, tool schemas, permissions, parameter validation, errors, timeouts, side effects and attempts to access unauthorised resources.
QA example: An AI assistant uses a Jira MCP server to retrieve PROJ-123 and a browser-testing MCP server to examine the related feature. Verify that it reads only the authorised project, sends correct tool parameters and requires approval before creating or updating Jira content.
Remember: MCP standardises the connection between AI applications and external capabilities; it does not remove the need for authentication, authorisation, approval and testing.
10. Guardrails
Guardrails are controls that restrict what an AI system can see, say and do. They can block unsafe input, protect sensitive information, limit access to tools, validate outputs and require human approval before important actions.11 Guardrails are not one feature or prompt; effective protection normally combines application rules, access control, monitoring, testing and human oversight.
For QA professionals, guardrails create a new set of functional and security requirements. Test both the normal path and attempts to bypass controls. Check whether users can manipulate a prompt to expose sensitive data, make the model use an unauthorised tool or perform a destructive action. Also test false positives: a guardrail that blocks valid business activity too often is a quality problem. Controls must work consistently across paraphrased inputs, languages, long conversations and indirect instructions hidden inside retrieved content.
QA example: A defect assistant may read production logs but must not reveal access tokens found in them. A test-healing agent may propose a change but should require human approval before committing it to the repository.
Remember: Guardrails reduce risk, but they must themselves be tested, monitored and improved.