Does Your AI-Generated Test Code Pass the Maintainability Test?

Reading time: 10 minutes

AI can generate a test script in seconds. The script may compile. It may even pass.

But that does not answer a more important question:

Can another engineer understand, change and debug this code six months from now?

GitHub warns that AI-generated code may be inaccurate, incomplete or inconsistent with the developer's intent. Its guidance says generated suggestions should be reviewed, tested and validated before use. It also recommends checking that the code follows appropriate design patterns and fits the architecture and style of the existing codebase.¹

This does not mean AI-generated test code is automatically bad. It means that passing once is not enough evidence that the code belongs in a long-lived automation suite.

The following 10 checks provide a practical maintainability test. They apply whether the code was written by a person, generated by AI, or created by both.

1. Can You Understand the Test Without Asking the AI?

A maintainable test should make its purpose clear to a human reader. Its name should describe the behaviour being verified, and its main steps should reveal the business flow.

The practical test is simple: hide the prompt and chat history, then give the code to another engineer. Can that person explain:

• What behaviour is being tested?
• What setup is required?
• What action is performed?
• What result proves success?

A structure such as Arrange–Act–Assert or Given–When–Then can make these parts easier to recognise. Fowler's practical test-pyramid guidance recommends short, readable tests and identifies Arrange–Act–Assert and Given–When–Then as useful structures for clean test code.²

test('locked user cannot sign in', async ({ page }) => {
  // Arrange
  await loginPage.open();

  // Act
  await loginPage.signIn('locked_user', process.env.TEST_PASSWORD!);

  // Assert
  await expect(loginPage.errorMessage)
    .toHaveText('Your account is locked');
});

Comments should explain information that the code cannot express clearly—not translate every line into English.

Pass condition: A teammate can understand the test's intention and failure condition without needing the original AI conversation.

2. Do the Locators Describe What the User Sees?

AI may generate a locator that works on the current page but depends on a long CSS path, a position such as nth(3), or an implementation-specific class.

Playwright recommends prioritising user-facing attributes and explicit contracts. Its documentation also warns that CSS and XPath selectors tied to the DOM structure can break when that structure changes.³ Selenium similarly recommends compact locators and notes that IDs are generally preferred when they are available and unique.⁴

// Fragile: coupled to the current DOM structure
page.locator('div.form > div:nth-child(3) > button.btn-primary');

// Clearer: describes the control as a user experiences it
page.getByRole('button', { name: 'Submit order' });

No locator type is perfect for every application. A stable test ID may be a better contract when accessible roles or visible text are ambiguous. The important question is whether the locator has a deliberate reason and a stable relationship with the element.

Pass condition: Locators use the team's agreed strategy and are not unnecessarily coupled to page structure or styling.

3. Does the Test Wait for Conditions—or Merely Wait for Time?

Generated scripts often use fixed delays because they are easy to write:

await page.waitForTimeout(5000);

A fixed delay does not describe what the test is waiting for. If the application responds sooner, the test still waits. If it responds later, the test can still fail.

Playwright locators perform actionability checks before actions, and its web-first assertions retry until their conditions are met or their timeout expires.⁵ Selenium provides explicit waits for waiting until a specific condition becomes true and warns not to mix implicit and explicit waits because the combination can cause unpredictable wait times.⁶

Prefer a meaningful condition:

await expect(page.getByText('Order confirmed')).toBeVisible();

This does not mean every timeout is wrong. Framework and test-level timeouts are necessary safety limits. The maintainability problem is using an unexplained sleep as a substitute for the application condition that matters.

Pass condition: Synchronisation is based on visible, testable conditions; fixed sleeps are rare and justified.

4. Do the Assertions Prove the Requirement?

A test can complete every action and still prove very little. For example, checking only that a button is visible does not prove that clicking it created an order correctly.

Playwright distinguishes generic assertions from asynchronous web-specific assertions. Its documentation recommends web-first assertions for UI conditions because they wait and retry, while non-retrying assertions can evaluate immediately.⁷

Review each assertion against the requirement:

• Does it verify the important business outcome?
• Does it use a specific expected value?
• Would the test fail if the feature were implemented incorrectly?
• Is the failure message understandable?

Avoid the opposite extreme as well: one test that verifies many unrelated behaviours can be difficult to diagnose. Fowler recommends testing one condition per test to keep tests short and easier to reason about.⁸

Pass condition: Every important expected result has a meaningful assertion, and each test has a focused reason to fail.

5. Is Page and Domain Knowledge Kept in the Right Place?

If every test directly contains page selectors, navigation details and low-level UI operations, a small interface change may require edits across many files.

Selenium describes Page Object Model as a way to reduce duplicated code and keep UI changes in one place. It also recommends that page objects model page services and generally keep test assertions in the test code, apart from checks that confirm the expected page was loaded.⁹

class CheckoutPage {
  constructor(private page: Page) {}

  async placeOrder() {
    await this.page.getByRole('button', { name: 'Place order' }).click();
  }

  get confirmation() {
    return this.page.getByRole('heading', { name: 'Order confirmed' });
  }
}

Do not ask AI to create an abstraction for every two lines of code. Playwright explicitly notes that a little duplication can be acceptable when it keeps simple tests clearer and easier to maintain.¹⁰ Abstract stable, meaningful behaviour—not every repeated keystroke.

Pass condition: Reusable page or domain behaviour has a clear home, without hiding the purpose of the test behind unnecessary layers.

6. Are Test Data, Configuration and Secrets Separated From Test Logic?

Environment URLs, account credentials and environment-specific values should not be scattered through test methods. This makes changes harder and can expose sensitive information.

GitHub's guidance for AI-generated code specifically calls for secure coding and warns against hard-coded passwords.¹¹ OWASP explains that a hard-coded password may be visible to anyone who can access the source and cannot be changed without modifying the software.¹²

Use the configuration, environment-variable and secret-management approach approved by your organisation. Test data can be supplied through fixtures, builders, factories or parameterised inputs. Playwright supports parameterising tests and projects, while its fixtures provide tests with the environment and resources they need.¹³

Be careful not to make data so indirect that nobody can understand the scenario. Values that explain the business case can remain visible in the test; credentials and environment settings should not.

Pass condition: The test contains meaningful scenario data, but no embedded credentials or unexplained environment-specific values.

7. Can the Test Run Alone and in Any Order?

A test that depends on another test's data or login state may pass as part of one suite and fail when run alone.

Playwright defines test isolation as each test running independently with its own browser storage, session storage and cookies. Its documentation says isolation prevents failure carry-over and makes individual failures easier to reproduce and debug.¹⁴ Selenium also recommends independent tests, no shared test data, cleanup of stale data, and a new WebDriver instance per test where appropriate.¹⁵

Review AI-generated code for hidden shared state:

• Global mutable variables
• Reused accounts that parallel tests can change
• Tests that assume an execution order
• Setup performed only by an earlier test
• Data that is created but never cleaned up

Some systems contain resources that genuinely cannot be used concurrently. In that case, handle the constraint explicitly rather than allowing an accidental dependency.

Pass condition: The test can run by itself, repeatedly and in a different order, with its prerequisites created or declared explicitly.

8. Has Repetition Been Removed Without Hiding Meaning?

AI can generate several tests independently, producing slightly different copies of the same login, navigation or data-creation steps. That creates more places to update when shared behaviour changes.

However, eliminating every repeated line can produce generic helper methods that are harder to understand than the duplication they replace. Playwright accepts limited duplication when it improves clarity.¹⁶ Fowler also recommends balancing reuse with readable, descriptive test code rather than applying DRY mechanically.¹⁷

Before creating a helper, ask:

• Does this represent one business action?
• Will several tests change for the same reason?
• Can its name clearly describe what it does?
• Does it reduce knowledge duplication, not merely line count?

customerCompletesCheckout() communicates intent. A helper such as clickAndWait(selector, timeout, expectedText) may only hide unrelated details behind a vague interface.

Pass condition: Common behaviour is reused deliberately, while each test still reads like a clear scenario.

9. Will a Failure Tell You What Actually Went Wrong?

Maintainability includes diagnosis. A test that reports only “expected true but received false” forces the team to reconstruct the context.

Use descriptive test names, meaningful assertions and the diagnostics supported by the framework. Playwright's Trace Viewer can show the action timeline, DOM snapshots, network activity and other details from an execution. Its guidance recommends trace collection on the first retry in CI rather than enabling it for every run because full-time tracing is resource intensive.¹⁸

Logs should add useful business or technical context without exposing credentials or personal information. Screenshots, traces and attachments should be captured according to the team's retention and privacy rules.

Pass condition: When the test fails in CI, the report identifies the scenario, failed expectation and relevant evidence without requiring an immediate local rerun.

10. Does the Code Pass the Same Engineering Gates as Human-Written Code?

AI-generated code should not bypass the normal pull-request process. GitHub states that generated suggestions must be reviewed and validated by the user. It also recommends secure coding, code review, testing and validation.¹⁹

A suitable quality gate depends on the technology and organisation, but it may include:

• Peer review
• Compilation or type checking
• Formatting and linting
• Unit tests for custom utilities
• Execution of the affected automation tests
• A focused regression run
• Secret and security scanning
• Review of the test's traceability to its requirement

For Playwright projects, the official guidance recommends linting tests, using TypeScript checks where applicable, and running tests frequently in CI—ideally on commits and pull requests.²⁰

The reviewer should also compare the generated code with the existing framework. A locally clever solution can still be expensive to maintain if it introduces a second configuration style, another assertion library or a new abstraction for a problem the framework already solves.

Pass condition: The code has been reviewed by a responsible owner and passes the project's normal automated and manual checks.

A Quick Maintainability Scorecard

Give the code one point for every Yes answer.

This score is a review aid, not an industry standard or a scientific measurement. A security failure or a missing business assertion can be serious even if the code scores well elsewhere. Teams should treat critical checks as mandatory rather than accepting a simple total.

The Better Way to Ask AI for Test Code

Maintainability begins before generation. Give the AI the standards it must follow:

• The framework, language and approved libraries
• A small example from the existing codebase
• The team's locator strategy
• The preferred page-object or component structure
• Test-data and configuration rules
• The required assertion style
• Naming, linting and formatting conventions
• A request to explain assumptions and uncertain decisions

Then ask for a small, reviewable change rather than an entire framework in one response. This is a practical recommendation, not a guarantee of correct output. The result must still pass the 10 checks above.

Final Takeaway

AI can reduce the time needed to produce a first draft of automation code. But generation speed and maintainability are different questions.

The final responsibility remains with the team that accepts the code. The best test is not simply one that passes today. It is one whose purpose is clear, whose dependencies are controlled, whose failures are diagnosable, and whose design can change with the product.

So, before merging the next AI-generated test, ask:

Does it merely work—or does it pass the maintainability test?

References

← Back to Blog