For years, testers have validated systems designed to follow predefined instructions.
Enter a value, click a button and verify the result. Send an API request and compare the response. Run a workflow and confirm that it reaches the expected state.
Now the systems themselves are changing.
AI agents can interpret a goal, choose steps, call tools, read changing information and take actions. Two runs may not follow exactly the same path. A response can sound reasonable and still be wrong. A small error can travel through several tools before anyone notices it.
This does not remove the need for QA. It creates a new quality frontier.
Evolution changes the work
Human progress has repeatedly changed our tools and skills. People once transported goods by walking. Animals, carts, engines and aircraft increased what could be moved and how quickly.
The person did not become faster than the machine. The work evolved: people learned to operate, guide, maintain and control more capable systems.
AI represents a similar shift in knowledge work. QA professionals do not need to compete with AI in producing routine artefacts. The opportunity is to direct AI-enabled systems and verify that they behave safely and usefully.
If AI is the elephant, the practical response is to learn how to ride it—with skill, direction and vigilance.
What makes an AI agent different?
A traditional application normally follows logic written in advance. An agent can be given a broader goal and decide how to pursue it.
An agent may:
- Interpret a request and create a plan
- Choose from available tools
- Read data from other systems
- Take several dependent actions
- Observe results and change direction
- Ask another agent for help
- Stop, retry or continue based on intermediate outcomes
Imagine a QA agent that receives a Jira requirement, analyses it, creates test cases and writes them back to Jira. A more advanced agent might generate automation, run tests, investigate failures and create a pull request.
This is no longer one input mapped to one predictable output. It is a chain of decisions, tool calls and side effects. Each step creates something QA must examine.
Correctness is no longer a simple exact match
For a calculator, 2 + 2 must return 4. For
an agent asked to prepare a risk-based test strategy, several different
answers may be reasonable.
QA therefore needs richer evaluation criteria:
- Did the agent understand the goal?
- Was the final outcome correct and useful?
- Were important constraints followed?
- Were facts separated from assumptions?
- Did it select and use appropriate tools?
- Could it recover when a dependency failed?
- Was human review requested before a high-impact action?
The basis for deciding pass or fail may combine business rules, reference answers, scoring rubrics and expert review rather than one exact expected result.
What QA teams will need to test
| Area | What QA must validate |
|---|---|
| Goal understanding | Intent, ambiguity, constraints and scope |
| Planning | Safe, efficient steps that can achieve the goal |
| Tool usage | Correct tools, parameters, permissions and failure handling |
| Memory and context | Relevant information without leakage or outdated influence |
| Multi-agent work | Hand-offs, conflicts, duplication and accountability |
| Human oversight | Review and approval before high-impact actions |
When multiple agents collaborate, testing each one separately is not enough. Important context can be lost between them, conclusions can conflict and a failed subtask may never reach the coordinating agent. The complete workflow must be tested.
Non-functional testing becomes critical
Functional correctness is only one part of agent quality.
Reliability
Because model output can vary, one successful run proves little. Repeated evaluations should measure task success, constraint following, tool-call accuracy, recovery and human intervention.
Performance and scale
An agent may make several model and tool calls before finishing. Teams must measure end-to-end response time, concurrency, external-tool delays, timeouts and retries.
An agent that works for ten tasks may fail at ten thousand. Testing at scale should examine rate limits, capacity, retry storms, growing context and human-review queues.
Cost
Agent quality includes economic sustainability. Model calls, tokens, tool usage, failed plans and uncontrolled loops all affect cost. An agent that produces the right result at an uncontrolled cost is not production-ready.
Security and control
An agent connected to tools can do more than produce a wrong sentence. It may expose information or perform an unintended action.
Testing should cover:
- Prompt injection and manipulated external content
- Excessive permissions
- Sensitive-data or cross-user leakage
- Unsafe or unauthorised tool calls
- Auditability and safe handling of secrets
- Approval for irreversible actions
- Safe stopping, failure and recovery
NIST’s Generative AI Profile treats trustworthiness as a lifecycle concern, including evaluation, monitoring and risk management. OWASP’s work on agentic applications also highlights that autonomous action and tool use introduce risks beyond ordinary chatbot output. NIST Generative AI Profile, OWASP GenAI Security Project
Testing must move from examples to evaluations
Traditional automation often uses a fixed input and exact expected result. Agent evaluation suites should include:
- Common and difficult user tasks
- Boundary, ambiguous and adversarial inputs
- Tool failures and unavailable dependencies
- Long, multi-step scenarios
- Policy and permission checks
- Repeated runs of the same task
- Production-like data variations
Success may be measured through deterministic checks, business rules, rubric-based scoring, reference outputs and human expert review.
Production monitoring also becomes part of the quality strategy. Pre-release testing cannot predict every combination of language, context, model behaviour and tool state.
New avenues for QA professionals
This shift creates new areas of work:
- Agent evaluation and reliability
- AI security and safety testing
- AI workflow quality engineering
- AI observability and production monitoring
QA professionals already bring relevant strengths:
- Asking “what if?”
- Challenging assumptions
- Thinking in boundaries and failure modes
- Designing systematic coverage
- Investigating unexpected behaviour
- Communicating risk
- Protecting the user’s interests
These strengths remain valuable. They must now be extended to models, agents, tools, data, evaluations and production behaviour.
A practical reskilling path
1. Use AI for familiar QA work
Apply it to one recurring task such as requirement review, test design, automation review or failure analysis. Compare the output with your expert work and record what it misses.
2. Learn to evaluate and challenge it
Practise finding hallucinations, missing coverage, unsupported assumptions, unsafe suggestions and unmaintainable code.
3. Build and test a controlled workflow
Create a small implementation, such as Jira ticket to reviewed test cases or failed test to investigation summary. Test errors and unsafe inputs—not only the happy path.
4. Test NFRs and build evidence
Run repeated and concurrent tasks. Measure reliability, latency, cost, security, recovery and human intervention. Document the failures you found and the controls you added.
The goal is not only to collect certificates. It is to show that you can apply AI to a quality problem and evaluate the result responsibly.
The future belongs to evolving quality professionals
Evolution does not mean discarding everything learned before. Cars did not make safe transportation irrelevant. Aircraft did not remove the need for navigation, maintenance or control. More capable systems created new skills and greater responsibilities.
The same principle applies to QA.
Testing fundamentals, domain knowledge and risk thinking remain the foundation. AI adds a new capability and a new class of systems to test.
Use AI where it adds value. Test its behaviour, control its actions and remain accountable for the outcome.
Quality expertise combined with AI capability offers the strongest position.
The elephant is already in the room. We can debate how powerful it will become, or we can learn how it behaves, how to guide it and how to travel farther with it.
For QA professionals willing to evolve, this may be one of the largest opportunities the field has seen.
Ask yourself
Are you preparing only to use AI—or are you also preparing to test the systems and agents that AI will create?