Risks of AI-Generated Tests — A Review Checklist

Build a systematic review checklist for AI-generated tests to catch hallucinated coverage, missing assertions, brittle selectors, and untested edge cases before they reach production.

#testing #qa-automation #software-engineering

This post is part 3 (final) of the AI for QA series. If you missed Part 2 — AI for Test Maintenance, start there.

Introduction — The Hidden Risks of AI-Generated Tests

AI-generated tests look professional. They have descriptive names, clear assertions, and well-structured code. They pass in CI. They increase coverage metrics.

But they have a dangerous property: they can pass without testing anything meaningful.

This is the risk of hallucinated coverage — tests that exist, that run, that report success, but that don’t actually validate the behavior they claim to test. AI doesn’t understand your application. It guesses at what tests should do based on patterns it’s seen in training data. Sometimes it guesses wrong.

A test that doesn’t test is worse than no test. It creates false confidence. It consumes maintenance effort. It slows down CI. And when the real bug appears, the test passes anyway.

This post will help you build a review checklist for AI-generated tests — a systematic process for catching hallucinated coverage, missing assertions, brittle selectors, and untested edge cases before they reach your main branch.

Risk 1: Hallucinated Coverage

The problem: AI generates a test that looks right but doesn’t actually verify the behavior it claims to test.

Example: Fake Assertion

AI-generated test:

test('user can reset password', async ({ page }) => {
  await page.goto('/forgot-password');
  await page.fill('[data-testid="email"]', 'user@example.com');
  await page.click('[data-testid="submit-button"]');
  
  // AI assumes success message appears
  await expect(page.locator('[data-testid="success-message"]')).toBeVisible();
});

The bug: The success message element exists in the DOM all the time (hidden with CSS display: none). The test passes even if the password reset email is never sent.

How to catch it:

  1. Manually execute the test scenario in the application
  2. Verify the assertion is actually triggered — remove the assertion and confirm the test fails
  3. Check that the element is conditionally rendered, not just conditionally shown

Fix:

test('user can reset password', async ({ page, request }) => {
  await page.goto('/forgot-password');
  await page.fill('[data-testid="email"]', 'user@example.com');
  await page.click('[data-testid="submit-button"]');
  
  // Wait for the success message to be rendered (not just unhidden)
  await expect(page.locator('[data-testid="success-message"]')).toBeVisible();
  
  // Additional verification: check that the reset email was sent
  const emails = await request.get('/api/test/emails'); // test-only endpoint
  const resetEmail = emails.find(e => e.to === 'user@example.com' && e.subject.includes('Reset'));
  expect(resetEmail).toBeDefined();
});

:::warning[Passing Tests ≠ Working Tests] A passing test is not proof that the test works. Always verify that the test fails when it should before trusting it. :::

Risk 2: Missing Negative Tests

AI is biased toward happy path tests. It generates tests that validate success scenarios but often misses failure scenarios.

Example: Missing Error Handling

AI-generated test suite:

test('create order with valid data', async ({ request }) => {
  const response = await request.post('/api/orders', {
    data: { productId: '123', quantity: 2 }
  });
  expect(response.status()).toBe(201);
});

What’s missing:

  • What happens if productId is invalid?
  • What happens if quantity is 0 or negative?
  • What happens if the product is out of stock?
  • What happens if the user is not authenticated?

Review question: For every AI-generated happy path test, ask: “What should fail here, and how?”

Fix: Add negative tests:

test('create order fails with invalid product ID', async ({ request }) => {
  const response = await request.post('/api/orders', {
    data: { productId: 'invalid', quantity: 2 }
  });
  expect(response.status()).toBe(404);
  const body = await response.json();
  expect(body.error).toContain('Product not found');
});

test('create order fails with negative quantity', async ({ request }) => {
  const response = await request.post('/api/orders', {
    data: { productId: '123', quantity: -1 }
  });
  expect(response.status()).toBe(400);
});

Risk 3: Brittle Selectors

AI often generates selectors that are brittle — they work today but will break on minor UI changes.

Example: CSS Class Selector

AI-generated test:

await page.click('button.btn-primary.submit-order');

Problem: This selector depends on CSS classes that can change when the designer updates the button style.

Review question: Does this selector use stable identifiers (data-testid, aria-label, semantic HTML)?

Fix:

await page.click('[data-testid="submit-order-button"]');

Risk 4: Hardcoded Test Data

AI often generates tests with hardcoded data that depends on pre-existing state in the database.

Example: Assuming User Exists

AI-generated test:

test('user can view profile', async ({ page }) => {
  await page.goto('/login');
  await page.fill('[data-testid="email"]', 'testuser@example.com');
  await page.fill('[data-testid="password"]', 'password123');
  await page.click('[data-testid="login-button"]');
  
  await page.goto('/profile');
  await expect(page.locator('h1')).toHaveText('Test User');
});

Problem: This test assumes testuser@example.com exists in the database. If another test modifies or deletes this user, this test fails.

Review question: Does this test create its own test data, or does it depend on pre-existing shared data?

Fix: Seed data per test:

test('user can view profile', async ({ page, request }) => {
  const user = await request.post('/api/test/seed-user', {
    data: { email: `user-${Date.now()}@example.com`, name: 'Test User' }
  });
  
  await page.goto('/login');
  await page.fill('[data-testid="email"]', user.email);
  await page.fill('[data-testid="password"]', user.password);
  await page.click('[data-testid="login-button"]');
  
  await page.goto('/profile');
  await expect(page.locator('h1')).toHaveText('Test User');
});

Risk 5: Arbitrary Waits

AI frequently generates tests with hardcoded timeouts instead of waiting for deterministic state changes.

Example: waitForTimeout

AI-generated test:

await page.click('[data-testid="submit-button"]');
await page.waitForTimeout(3000); // wait 3 seconds
await expect(page.locator('[data-testid="success-message"]')).toBeVisible();

Problem: This test is flaky. Sometimes 3 seconds is enough. Sometimes it’s not. Sometimes it’s too much (test is slower than necessary).

Review question: Does this test use arbitrary waits, or does it wait for observable state changes?

Fix:

await page.click('[data-testid="submit-button"]');
await expect(page.locator('[data-testid="success-message"]')).toBeVisible({ timeout: 10000 });

Or better, wait for a specific state attribute:

await page.click('[data-testid="submit-button"]');
await page.waitForSelector('[data-testid="form-container"][data-state="success"]');
await expect(page.locator('[data-testid="success-message"]')).toBeVisible();

Risk 6: Incomplete Assertions

AI-generated tests sometimes have weak assertions that don’t fully validate the behavior.

Example: Only Checking Status Code

AI-generated test:

test('create user', async ({ request }) => {
  const response = await request.post('/api/users', {
    data: { email: 'user@example.com', name: 'Test User' }
  });
  expect(response.status()).toBe(201);
});

What’s missing:

  • Does the response contain a valid user ID?
  • Does the user actually exist in the database?
  • Can the user log in with the created account?

Review question: Does this assertion prove the operation succeeded, or does it just prove the API returned the expected status code?

Fix: Add stronger assertions:

test('create user', async ({ request }) => {
  const response = await request.post('/api/users', {
    data: { email: 'user@example.com', name: 'Test User' }
  });
  expect(response.status()).toBe(201);
  
  const body = await response.json();
  expect(body.id).toBeDefined();
  expect(body.email).toBe('user@example.com');
  
  // Verify user exists in database
  const fetchResponse = await request.get(`/api/users/${body.id}`);
  expect(fetchResponse.status()).toBe(200);
});

Risk 7: Missing Edge Cases

AI generates obvious test cases but often misses edge cases that matter in production.

Example: Date Boundary Cases

AI-generated tests for “user must be 18 or older”:

test('user over 18 can register', async ({ page }) => {
  await page.fill('[data-testid="birthdate"]', '2000-01-01'); // 26 years old
  await page.click('[data-testid="submit"]');
  await expect(page.locator('[data-testid="success"]')).toBeVisible();
});

test('user under 18 cannot register', async ({ page }) => {
  await page.fill('[data-testid="birthdate"]', '2010-01-01'); // 16 years old
  await page.click('[data-testid="submit"]');
  await expect(page.locator('[data-testid="error"]')).toBeVisible();
});

What’s missing:

  • User exactly 18 years old (boundary case)
  • User born on February 29 (leap year edge case)
  • User with birth year 1900 (minimum date validation)
  • Future birthdates (validation error)

Review question: What edge cases would a human tester think of that AI might miss?

The Review Checklist

Use this checklist to review every AI-generated test before merging:

1. Coverage Verification

  • Does this test actually validate the behavior it claims to test?
  • If I remove the assertion, does the test fail?
  • Is the assertion checking for a condition that’s always true (hallucinated coverage)?

2. Negative Testing

  • Are there corresponding negative tests (error cases, invalid input, edge cases)?
  • Does the test suite cover “what should fail” in addition to “what should succeed”?

3. Selector Stability

  • Does the test use stable selectors (data-testid, semantic HTML, ARIA attributes)?
  • Will this test break if the UI design changes (CSS classes, layout)?

4. Data Independence

  • Does the test create its own data, or does it depend on shared/pre-existing data?
  • Can this test run in parallel with other tests without conflicts?

5. Deterministic Waits

  • Does the test use arbitrary waits (waitForTimeout), or does it wait for observable state?
  • Are all waits necessary, or are some redundant (Playwright auto-waits)?

6. Assertion Strength

  • Do the assertions prove the operation succeeded, or just that the API responded?
  • Are there additional verifications that would increase confidence (e.g., check database state)?

7. Edge Cases

  • Does the test suite cover boundary conditions (min/max values, empty inputs, special characters)?
  • Are there date/time edge cases (leap years, timezones, DST)?
  • Are there concurrency edge cases (race conditions, lock contention)?

8. Readability

  • Is the test easy to understand?
  • Are variable names clear?
  • Is the test structure logical (arrange, act, assert)?

Real-World Example: Reviewing an AI-Generated Test

AI-generated test:

test('checkout flow', async ({ page }) => {
  await page.goto('http://localhost:3000/cart');
  await page.click('button.checkout-btn');
  await page.fill('input[name="email"]', 'test@example.com');
  await page.fill('input[name="card"]', '4242424242424242');
  await page.click('button[type="submit"]');
  await page.waitForTimeout(5000);
  await expect(page.locator('.success')).toBeVisible();
});

Review findings:

  1. ❌ Brittle selectors: button.checkout-btn, input[name="email"], .success
  2. ❌ Hardcoded data: Assumes cart has items
  3. ❌ Arbitrary wait: waitForTimeout(5000)
  4. ❌ Weak assertion: Only checks if success message is visible, not if order was created

Refactored test:

test('checkout flow', async ({ page, request }) => {
  // Seed test data
  const cart = await request.post('/api/test/seed-cart', {
    data: { productId: 123, quantity: 1 }
  });
  
  await page.goto('/cart');
  await page.click('[data-testid="checkout-button"]');
  await page.fill('[data-testid="email-input"]', 'test@example.com');
  await page.fill('[data-testid="card-number-input"]', '4242424242424242');
  await page.click('[data-testid="submit-button"]');
  
  // Wait for deterministic state
  await expect(page.locator('[data-testid="success-message"]')).toBeVisible();
  
  // Verify order was created
  const orders = await request.get('/api/test/orders');
  expect(orders.length).toBeGreaterThan(0);
});

Result: Stable, deterministic, and comprehensive.

Conclusion

AI-generated tests need human review. They often contain:

  • Hallucinated coverage (passing tests that don’t verify behavior)
  • Missing negative tests (only happy paths)
  • Brittle selectors (CSS classes, fragile locators)
  • Hardcoded data (shared state, flaky tests)
  • Arbitrary waits (timeouts instead of state checks)
  • Weak assertions (status codes without verification)
  • Missing edge cases (boundary conditions, special inputs)

Use the review checklist from this post on every AI-generated test. Treat AI output as a first draft, not a finished product.

The goal is not to avoid AI — it’s to use AI safely. AI accelerates test creation, but human review ensures the tests are reliable, maintainable, and valuable.

:::tip[AI + Human Review = Best Results] The best test suites combine AI’s speed (generating boilerplate, surfacing edge cases) with human judgment (prioritization, risk assessment, business context). Don’t use AI alone. Don’t write everything manually. Combine both. :::

Action for this week: Pick one AI-generated test from your suite. Run through the review checklist in this post. Identify at least three improvements. Refactor the test. Measure the improvement in stability and coverage.

This concludes the AI for QA series. For more on test architecture, see the Test Architecture series.