AI can help QA professionals review requirements, develop test strategies, design test scenarios, generate automation code, prepare test data, analyse failures and summarise test results.
However, getting an answer quickly does not mean that the answer is correct.
Generative AI can produce confidently written but incorrect information. The National Institute of Standards and Technology (NIST) calls this risk “confabulation.” NIST also identifies automation bias and over-reliance as risks in interactions between people and AI systems.
Therefore, the important question is not simply:
Are you using AI in testing?
The better question is:
Are you using AI in a way that preserves evidence, critical thinking and human accountability?
Here are ten signs that you may be using AI incorrectly across the testing lifecycle.
The QA examples below are illustrative, not measured research results.
1. You ask AI to perform a QA activity without giving it the necessary context
Consider prompts such as:
- Review this requirement.
- Create a test strategy.
- Generate test scenarios.
- Analyse this failure.
- Prioritise these regression tests.
These prompts describe what AI should do. They do not necessarily provide the information AI needs to do it correctly.
The relevance of an AI response depends partly on the context available to the model. GitHub’s responsible-use documentation explains that Copilot generates suggestions using the surrounding context, may not identify larger architectural issues and can produce incomplete or inaccurate output. GitHub advises users to review and validate generated suggestions.
Practical QA example
Suppose a tester gives AI a Jira story and asks:
Create a risk-based test plan for this requirement.
The Jira story does not include:
- Business impact
- Number or type of affected users
- Integration dependencies
- Production usage
- Historical defects
- Security considerations
- Recent code changes
AI can still create a professional-looking test plan. However, its product-specific risk assessment cannot reliably account for information that was neither supplied nor retrieved through its tools.
A better way to use AI
Before submitting the prompt:
- Identify the decision the AI output must support.
- Determine what information is required.
- Provide only relevant and approved context.
- Ask AI to list missing information.
- Ask it to state its assumptions separately.
A useful instruction would be:
Before creating the test plan, list the information that is missing. Do not invent business rules. Clearly identify every assumption.
Treat the resulting list as a starting point, not a guarantee of completeness. The tester must independently check for missing information and unsupported assumptions.
2. You treat a confident AI answer as evidence that it is correct
AI can present an answer using clear language, technical terminology and convincing reasoning. None of these qualities proves that the answer is correct.
NIST states that generative AI can produce factually inaccurate or internally inconsistent content. It also warns that generated explanations, logic and citations may themselves be incorrect, potentially causing people to place inappropriate trust in the answer.
Practical QA example
A test fails with a database timeout. AI examines the stack trace and concludes:
The test failed because the database server was overloaded.
The stack trace may prove that a timeout occurred. It may not prove why it occurred.
Other possibilities could include:
- A slow query
- Network interruption
- Connection-pool exhaustion
- Incorrect test data
- An unavailable dependency
- An environment configuration problem
AI may have generated a plausible explanation rather than identifying the actual root cause.
Use an evidence classification
The following is a suggested framework for reviewing AI-generated claims, not a formal standard.
| Classification | Meaning |
|---|---|
| Observed fact | Directly supported by a log, trace, test result, requirement or code |
| Supported inference | A conclusion reasonably connected to available evidence |
| Hypothesis | A possible explanation that still requires investigation |
| Unsupported assumption | A claim for which no supporting evidence was supplied |
| Contradicted claim | An explanation that conflicts with available evidence |
AI is often useful for generating hypotheses. It should not be allowed to silently convert a hypothesis into a confirmed root cause.
3. You measure how much AI produced instead of how much value it added
AI can generate:
- 100 test cases
- Hundreds of lines of automation code
- A complete test plan
- A detailed defect report
- A multi-page execution summary
These numbers measure output volume. They do not establish that the output is correct, useful or effective.
Recent research demonstrates why this distinction matters. The SWE-Mutation benchmark evaluates whether LLM-generated test suites can detect deliberately modified implementations. Its authors found weaknesses in the ability of generated tests to distinguish correct implementations from faulty ones. These results belong to a specific benchmark and should not be generalised to every model, product or testing context.
GitHub evaluates its AI coding assistance using multiple measures, including correctness, contextual relevance, regressions in suggestion quality, latency, token use and qualitative developer feedback—not merely the volume of generated code.
Practical QA example
AI generates 50 test cases for a login feature:
- 20 vary invalid passwords but exercise the same behaviour without adding distinct coverage.
- 15 repeat the same happy path using data that does not exercise different boundaries, user roles or configurations.
- 10 test rules not stated in the requirement.
- 5 examine meaningful security and account-state risks.
The number “50” sounds impressive. The actual value may come primarily from the final five tests.
Different datasets are not inherently redundant. They add value when they exercise distinct equivalence classes, boundaries, roles, configurations or other meaningful risks.
Better measures
Instead of counting only generated output, consider:
- How much generated work was accepted?
- How much required correction?
- Did it identify previously missed risks?
- Did the tests detect defects?
- Were the tests maintainable?
- Did AI reduce or increase review effort?
- Did it improve meaningful coverage?
- How much rework was required later?
More output is not automatically more quality.
4. You allow AI to invent missing business rules
AI can fill missing information with content that sounds reasonable. In testing, a reasonable assumption is not necessarily an approved expected result.
NIST explains that confabulation can include false, erroneous or unsupported output. It notes that the risk can be especially relevant in domains requiring substantial context or specialist knowledge.
Practical QA example
A requirement says:
Lock the user account after repeated failed login attempts.
It does not specify:
- The permitted number of attempts
- How long the account remains locked
- Whether an administrator can unlock it
- Whether successful login resets the failed-attempt counter
AI generates tests assuming:
- Five failed attempts
- A 30-minute lockout
- Automatic unlocking
- Counter reset after successful login
These may be sensible suggestions. They are not confirmed requirements.
If the tester copies them into expected results, the test now validates an unapproved assumption rather than a confirmed requirement.
A better classification
Every AI-generated rule should be labelled as one of the following:
- Supported directly by the requirement
- Supported by an approved domain rule
- Supported by another referenced document
- Requires clarification
- Tester assumption
- AI suggestion
Do not convert an AI suggestion into an expected result until an authorised source confirms it.
5. You share sensitive QA information without checking the AI tool’s data controls
Testing artefacts may contain:
- Customer information
- Production data
- Personally identifiable information
- Access tokens
- Passwords
- Database connection details
- Internal URLs
- Source code
- Security vulnerabilities
- Confidential business rules
NIST identifies unauthorised disclosure, leakage, inference and de-anonymisation of sensitive information as generative AI privacy risks.
OWASP’s guidance on sensitive information disclosure recommends controls such as data sanitisation, access restrictions and clear policies for handling sensitive data used with LLM applications.
The risk is not identical for every AI tool
The actual risk depends on:
- The selected product
- Personal or enterprise account
- Retention settings
- Provider policies
- Contractual protections
- Model-training settings
- Organisation-level controls
- Applicable laws and regulations
Using an organisation-approved enterprise service under defined controls is not the same as pasting the information into an unapproved public service.
A better way to use AI
Before sharing QA material:
- Follow your organisation’s AI and data policies.
- Remove passwords, tokens and API keys.
- Mask personal and customer information.
- Share only the minimum information required.
- Confirm retention and model-training settings.
- Prefer synthetic data when it represents the testing need adequately.
- Do not assume that changing a person’s name makes an entire dataset anonymous.
6. You accept AI-generated automation because it compiles or passes once
Compilation demonstrates that code satisfies certain language and dependency requirements.
A successful run demonstrates that one execution met the implemented assertions.
Neither proves that the test:
- Validates the correct business behaviour
- Contains meaningful assertions
- Is secure
- Is stable
- Contributes to appropriate coverage of negative conditions across the suite
- Is independent
- Fits the automation architecture
GitHub warns that AI-generated code can appear valid while being syntactically or semantically incorrect, may fail to reflect the developer’s intent and may contain security weaknesses. Its guidance recommends reviewing, testing and validating generated code before using it.
Practical QA example
AI generates a UI test that:
- Selects the first button on the page
- Completes a transaction
- Checks only that the next page appears
The test passes.
However, it may not verify:
- Which button was selected
- Whether the transaction succeeded
- Whether the correct amount was processed
- Whether the backend state changed
- Whether the confirmation belongs to the correct user
The script is executable, but the test may still be weak.
The checks required depend on the test’s purpose and layer. Not every UI test needs a direct backend check, and not every individual test needs negative conditions. The suite should provide appropriate coverage without unnecessary duplication.
Review AI-generated automation for
Apply these checks where relevant to the test’s scope and the framework:
- Test intent
- Requirement traceability
- Selector quality
- Wait strategy
- Assertion strength
- Test isolation
- Test-data management
- Cleanup
- Error handling
- Secret handling
- Reusability
- Parallel-execution safety
- Alignment with framework standards
AI-generated test code should go through the same review process as human-written code—and may require additional attention when its author does not fully understand the generated implementation.
7. You ask AI to fix a failed test before identifying why it failed
A test can fail because of:
- A product defect
- A test defect
- Incorrect test data
- Environment instability
- An unavailable dependency
- A timing problem
- A configuration change
- A changed requirement
- An expected product change
If you give AI only the failed script, it may assume that the script is the problem.
Practical QA example
AI “fixes” a failed test by:
- Increasing a timeout
- Adding retries
- Catching and ignoring an exception
- Using a broader locator
- Removing an assertion
- Changing the expected result
The test becomes green, but the underlying problem may still exist.
Timeout adjustments, retries and locator changes can be legitimate fixes when supported by evidence. The problem is making unsupported changes that hide a product defect or weaken the intended check.
GitHub’s responsible-use guidance states that AI-generated suggestions can be incomplete, inaccurate or misaligned with the intended objective and should be reviewed against actual requirements.
A better sequence
- Collect the test evidence.
- Confirm whether the failure is reproducible.
- Review application, test and environment changes.
- Use AI, where helpful, to generate possible hypotheses and identify supporting or conflicting evidence.
- Investigate what would confirm or reject each hypothesis.
- Classify the failure using the available evidence, with AI assistance if useful.
- Apply and verify a fix that addresses a supported cause while preserving the intended test behaviour.
AI can assist throughout this sequence. Failure classification may change as new evidence becomes available.
AI can accelerate investigation. It should not be used merely to make a red test green.
8. You use AI to summarise test results without preserving the underlying evidence
AI can turn raw execution results into an attractive report. But a polished summary can omit important information.
Practical QA example
AI creates this summary:
96% of executed tests passed, so the release is stable.
That statement may be misleading if:
- A critical payment test was blocked
- Important tests were excluded
- Several failures passed only after reruns
- Testing used an environment different from production
- A high-severity defect remains open
- A major code change was not covered
- The failed 4% includes the highest-risk scenarios
A pass percentage alone does not establish release readiness.
A better way to use AI
Treat AI as a drafting layer over traceable evidence.
The final report should preserve:
- Build and application versions
- Test environment
- Executed scope
- Excluded scope
- Blocked tests
- Failed tests
- Rerun information
- Open defects
- Untested changes
- Known limitations
- Residual risks
- Links to detailed evidence
Ask AI to state uncertainty and missing information explicitly.
9. You allow an AI agent to modify QA systems without appropriate controls
The risk changes when AI moves from suggesting an action to performing it.
An AI agent may be able to:
- Read requirements from Jira
- Add test cases to Jira
- Modify automation code
- Run tests
- Change pipeline configuration
- Create test data
- Comment on pull requests
- Commit code
- Trigger deployments
OWASP describes “excessive agency” as a situation in which an LLM-based system has excessive functionality, permissions or autonomy that can result in damaging actions. OWASP recommends minimising functionality, permissions and autonomy and using human approval for high-impact actions.
OWASP also warns against passing model output directly to downstream systems without appropriate validation. It recommends treating model output with zero trust and validating it before use.
Practical QA example
An agent reads a requirement, generates test cases and adds them directly to Jira.
Possible problems include:
- Duplicate test cases
- Invented expected results
- Incorrect project or issue selection
- Overwriting existing information
- Exposure of sensitive data
- Untraceable changes
- Instructions hidden inside untrusted content
Controls to consider
- Least-privilege access
- Separate read and write permissions
- Validation before updates
- Human approval for consequential actions
- Complete audit trails
- Restricted tool access
- Reversible operations
- Clear ownership
- Failure and recovery procedures
Enforce permissions and approval gates in the tools and downstream systems—not solely through instructions to the AI. Treat retrieved requirements, logs and other external content as data, not as authority to change the agent’s permissions.
A low-risk and reversible action may justify more autonomy than changing a production-connected pipeline or approving a release.
10. You allow AI to replace accountable QA judgement
AI can:
- Organise evidence
- Compare alternatives
- Identify inconsistencies
- Generate questions
- Draft recommendations
- Summarise known risks
It does not own the consequences of a business decision.
NIST’s AI Risk Management Framework connects trustworthy AI with governance, accountability, transparency, measurement and appropriate human–AI configuration. It also emphasises that risks and mitigations vary according to the organisation, application and use case.
Practical QA example
A team asks AI:
Should we release this application?
AI may not know:
- Contractual commitments
- Customer expectations
- Legal obligations
- Operational readiness
- Financial impact
- Accepted residual risk
- Who has authority to approve the release
A confident “yes” or “no” is not a substitute for accountable decision-making.
A better way to use AI
Ask AI to prepare a decision brief containing:
- Available evidence
- Test coverage
- Untested scope
- Open defects
- Assumptions
- Conflicting evidence
- Known limitations
- Residual risks
- Questions requiring stakeholder input
Authorised stakeholders should make and record the final decision.
A practical AI-in-QA self-assessment
Before accepting an AI-generated QA output, ask these ten questions:
| Question | What it helps you detect |
|---|---|
| What source information did AI receive? | Missing or weak context |
| What relevant information was unavailable? | Limits of the output |
| Which statements are facts, inferences, hypotheses or assumptions? | Unsupported conclusions |
| Can expected results be traced to an authorised source? | Invented business rules |
| Was sensitive information handled under approved controls? | Privacy and security risks |
| Was generated code reviewed like human-written code? | Automation-quality problems |
| Are we measuring useful outcomes or output volume? | Misleading productivity claims |
| Is the summary linked to underlying evidence? | Unsupported reporting |
| What can the AI change, and with what permissions? | Excessive agency |
| Who remains accountable for the final decision? | Loss of human ownership |
If several answers are unclear, it does not necessarily mean that AI is unsuitable.
It may mean that the surrounding QA process needs better context, evidence, controls or accountability.
Important limitations and exceptions
These ten signs do not prove that a particular person or team is using AI irresponsibly.
The level of risk depends on:
- The QA activity
- The product
- The AI tool
- Available context
- Data sensitivity
- Tool permissions
- Consequences of an incorrect result
- Existing review and governance controls
A brainstorming exercise does not require the same controls as an AI agent that modifies a CI/CD pipeline.
Human review is also not automatically reliable. A reviewer needs sufficient domain knowledge, technical understanding, time and evidence to recognise an incorrect output.
For repeated AI-assisted activities, defined evaluation criteria and regression datasets may be more dependable than informal review alone. OpenAI’s evaluation guidance recommends task-specific criteria, representative evaluation data and continuous evaluation as AI systems change. It cautions against relying on informal “it seems to work” assessments instead of defined criteria.
These evaluations should complement informed human review, not automatically replace it. Anthropic’s guidance for evaluating agents distinguishes checking the final result from examining the agent’s process, tool use and intermediate decisions.
Takeaway for QA professionals
Using AI correctly does not necessarily mean using it less.
It means giving AI the right role.
Use AI to:
- Expand your thinking
- Generate alternatives
- Organise evidence
- Identify questions
- Accelerate well-understood work
- Support investigation
- Improve documentation
Do not allow AI-generated output to replace:
- Approved requirements
- Domain knowledge
- Test evidence
- Technical review
- Privacy controls
- Risk ownership
- Accountable human judgement
A valuable skill in the AI era is knowing:
What evidence is required, what must be verified and where human responsibility must remain.