Flaky Tests — Root Causes and Systematic Fixes

Flaky tests destroy trust in automation. Learn the complete taxonomy of flakiness causes and how to build a team process for prevention, quarantine, and systematic elimination.

#qa-automation #testing #test-management

This post is part of the Automation Strategy series. If you missed the previous post, read Part 5 — Designing Maintainable Suites first.

“Just re-run the pipeline.”

If your team says this regularly, you have a flaky test problem. Flaky tests — tests that pass or fail intermittently without code changes — are the single most corrosive force in test automation.

They destroy trust. When a test fails, engineers ask “is this a real bug or a flaky test?” If the answer is “probably flaky,” they ignore it. Once trust is gone, the entire test suite becomes a formality — something you run because CI requires it, not because it provides value.

This post goes deeper than Part 1 of the fundamentals series. We’ll cover a complete taxonomy of flakiness causes, systematic detection and fixes, and how to build a team process for managing flake budgets, quarantine, and prevention.

The Cost of Flaky Tests

Before diving into fixes, understand the cost:

Direct Costs

  • Developer time — Investigating false failures (5–20 minutes per flake)
  • CI cost — Re-running pipelines wastes compute and delays deployments
  • Blocked deployments — Teams can’t ship while waiting for flaky tests to pass

Indirect Costs

  • Eroded trust — Once trust is lost, real bugs are ignored as “probably flaky”
  • Test maintenance — Flaky tests consume disproportionate maintenance effort
  • Team morale — Nothing is more demoralising than debugging a test failure that isn’t a bug

:::warning[The 2% Rule] Research shows that once flake rate exceeds 2%, teams stop trusting the test suite. If 1 in 50 test runs has a false failure, engineers assume every failure is false until proven otherwise. Keep flake rate below 2%. :::

Taxonomy of Flakiness — Root Causes

Flaky tests aren’t random. They fall into predictable categories. Here’s the complete taxonomy.

Category 1: Timing and Async Issues

Symptom: Test passes locally, fails in CI. Fails intermittently with no pattern.

Root causes:

  • Hardcoded waits — await page.waitForTimeout(2000) assumes the page loads in < 2s
  • Race conditions — Test clicks button before JavaScript has attached event listener
  • Polling intervals — App polls API every 5s, test runs before first poll completes
  • Animation delays — UI element animates into position, test clicks before animation finishes

Fixes:

  • Replace waitForTimeout with condition-based waits: 
    // Bad
await page.waitForTimeout(2000);
await expect(page.getByText('Success')).toBeVisible();

// Good
await expect(page.getByText('Success')).toBeVisible({ timeout: 5000 });
  • Use Playwright’s auto-waiting (it retries assertions automatically)
  • Wait for network idle after navigation: await page.waitForLoadState('networkidle')
  • Disable animations in test environments: 
    await page.addStyleTag({ content: '* { animation: none !important; }' });

Category 2: Test Data Pollution

Symptom: Test passes when run alone, fails when run in a suite.

Root causes:

  • Shared fixtures — Multiple tests modify the same database record
  • No cleanup — Tests leave data behind that affects subsequent tests
  • Race conditions in parallel execution — Two tests create/delete the same resource

Fixes:

  • Each test creates and owns its data (covered in Part 4)
  • Use unique identifiers: 
    const email = `test-${Date.now()}-${Math.random().toString(36)}@example.com`;
  • Clean up in afterEach: 
    test.afterEach(async ({ request }) => {
  await deleteUser(request, testUserId);
});
  • Use database transactions for integration tests (rollback after each test)

Category 3: Environment Instability

Symptom: Test fails in CI but passes locally, or vice versa.

Root causes:

  • Network flakiness — External API calls timeout in CI
  • Clock skew — Tests depend on Date.now(), fail across timezones or slow machines
  • File system differences — Tests hardcode paths (C:\temp\) that don’t exist in Linux CI
  • Missing dependencies — Fonts, libraries, or services not installed in CI

Fixes:

  • Mock external APIs: 
    await page.route('https://api.external.com/**', route => {
  route.fulfill({ status: 200, body: JSON.stringify(mockData) });
});
  • Use fixed clocks in tests: 
    vi.useFakeTimers();
vi.setSystemTime(new Date('2025-01-01T12:00:00Z'));
  • Use Docker/TestContainers for consistent environments
  • Ensure CI has same dependencies as local (pin versions, use lockfiles)

Category 4: Flaky Selectors

Symptom: Test fails with “element not found” intermittently.

Root causes:

  • Dynamic IDs — Selecting by ID when IDs are auto-generated (<div id="item-1827364">)
  • Positional selectors — .nth-child(3) breaks when DOM order changes
  • CSS classes — CSS-in-JS generates new class names on each build
  • Text content — Selecting by text that changes (e.g., dynamic dates)

Fixes:

  • Use data-testid attributes for stable selection: 
    <button data-testid="submit-order-btn">Place Order</button>
  • Prefer role-based selectors: 
    await page.getByRole('button', { name: 'Place Order' }).click();
  • Avoid brittle selectors like .nth-child(), > div > div > button

Category 5: Dependency on External Services

Symptom: Test sometimes fails with “network timeout” or “service unavailable.”

Root causes:

  • Third-party APIs — Payment gateway, email service, geocoding API goes down
  • Shared staging environment — Another team deploys breaking change to shared backend
  • Rate limits — Test exceeds API rate limit when run frequently

Fixes:

  • Mock external services (as shown in Category 3)
  • Use contract tests to validate API shape without hitting real service
  • Run integration tests against dedicated test instances (not shared staging)
  • Implement retries with exponential backoff for genuinely flaky services: 
    await expect(async () => {
  const response = await fetch('/api/orders');
  expect(response.status).toBe(200);
}).toPass({ intervals: [1000, 2000, 5000] }); // Playwright retries with backoff

Category 6: Test Order Dependency

Symptom: Test suite passes, but individual test fails when run in isolation.

Root causes:

  • Setup in previous test — Test A creates data, Test B depends on it existing
  • Global state mutation — Test modifies singleton or static variable

Fixes:

  • Every test must be idempotent — runnable in any order
  • Use test fixtures or beforeEach for setup: 
    test.beforeEach(async ({ request }) => {
  testUser = await createUser(request);
});
  • Avoid modifying global state (singletons, environment variables) — reset after each test

Detection: Measuring Flakiness

You can’t fix what you don’t measure. Track flake rate systematically.

Manual Detection

Run tests multiple times in parallel:

# Playwright: run suite 10 times in parallel
npx playwright test --repeat-each=10 --workers=10

# Watch for intermittent failures

If any test fails even once out of 10 runs, it’s flaky.

Automated Detection

Most CI systems can track flake rate:

  • GitHub Actions: Use test reporters that track flaky tests over time
  • Playwright: Built-in flaky test detection — reruns failed tests automatically and marks them as flaky if they pass on retry

Example Playwright config:

export default defineConfig({
  retries: 2, // Retry failed tests twice
  reporter: [
    ['html'],
    ['json', { outputFile: 'test-results.json' }],
  ],
});

Playwright will mark tests as flaky if they fail initially but pass on retry.

Flake Dashboard

Track flake rate over time. Create a dashboard showing:

  • Flake rate — % of test runs with at least one flaky test
  • Flaky tests by name — Identify chronic offenders
  • Trend — Is flake rate increasing or decreasing?

:::tip[Set a Flake Budget] Define an acceptable flake rate (e.g., < 2%) and treat exceeding it as a P0 incident. When you cross the threshold, stop adding new tests until flakes are fixed. :::

Systematic Fixes: The Quarantine Process

When you detect a flaky test, follow this process:

Step 1: Reproduce Locally

Run the test 50–100 times in parallel:

npx playwright test path/to/test.spec.ts --repeat-each=50 --workers=10

If it fails at least once, you’ve reproduced it. Note the failure rate (e.g., fails 3 times out of 50 = 6% flake rate).

Step 2: Quarantine the Test

Mark it as flaky and skip it in CI to prevent blocking deployments:

test.fixme('user can checkout (FLAKY)', async ({ page }) => {
  // Test implementation
});

Or use tags:

test('user can checkout @flaky', async ({ page }) => {
  // Test implementation
});

Then in CI, exclude flaky tests:

npx playwright test --grep-invert @flaky

:::warning[Quarantine, Don’t Delete] Don’t delete flaky tests. Quarantine them. A flaky test that validates critical functionality is still valuable once fixed. Deleting it loses coverage. :::

Step 3: Investigate Root Cause

Use the taxonomy above to identify the category. Common investigation techniques:

  • Add debug logging: Log timestamps, network requests, element states
  • Use trace viewer: Playwright’s trace viewer shows screenshots, network, console logs for each step
  • Compare passing vs failing runs: What differs? Timing? Data? Environment?

Step 4: Fix and Verify

Apply the fix (e.g., replace waitForTimeout, add data isolation, mock external API). Verify by running the test 100 times:

npx playwright test path/to/test.spec.ts --repeat-each=100 --workers=10

If it passes 100/100 times, the fix is valid. If it still fails, repeat Step 3.

Step 5: Un-Quarantine

Remove .fixme() or @flaky tag. Re-enable in CI. Monitor flake rate.

Prevention: Building a Flake-Free Culture

Fixing existing flakes is reactive. Prevention is proactive.

Code Review Checklist for Test PRs

When reviewing a PR with new tests, check:

  • No waitForTimeout or Thread.Sleep
  • No hardcoded IDs, class names, or positional selectors
  • Test creates its own data (no shared fixtures)
  • Cleanup included (afterEach or fixtures)
  • Passed 10+ times locally in parallel
  • No dependencies on external services (or properly mocked)

CI Gating

Don’t merge PRs that introduce flaky tests. Configure CI to:

  1. Run new tests 5–10 times before merge
  2. Block merge if any run fails

Example GitHub Actions workflow:

- name: Run new tests multiple times
  run: npx playwright test --repeat-each=5 --workers=5

Flake Budget

Set a team-wide flake budget (e.g., < 2% flake rate). When you exceed it:

  • Stop adding new tests until flake rate drops below threshold
  • Dedicate time to fixing flakes — treat it as P0 work
  • Hold a post-mortem — Why did flake rate increase? What can we improve?

Advanced: Flake Mitigation When You Can’t Fix

Some flakes are unavoidable (e.g., integration tests against truly flaky third-party services). In these cases, use mitigation strategies:

Strategy 1: Test Retries

Playwright supports automatic retries:

export default defineConfig({
  retries: 2, // Retry failed tests up to 2 times
});

If a test passes on retry, it’s marked flaky but doesn’t block CI.

Use sparingly. Retries mask flakiness — they don’t fix it.

Strategy 2: Flake-Tolerant Assertions

For tests that validate “eventually consistent” behaviour:

await expect(async () => {
  const count = await getOrderCount();
  expect(count).toBeGreaterThan(0);
}).toPass({ timeout: 10000 }); // Retry assertion for up to 10s

This is appropriate for message queues, async processing, or eventual consistency scenarios.

Strategy 3: Separate Flaky Tests from Core Suite

If a test is valuable but unavoidably flaky (e.g., performance test affected by CI load):

  • Move it to a separate suite
  • Run it nightly instead of per-commit
  • Don’t block deployments on it

Conclusion

Flaky tests are not inevitable. They have predictable root causes and systematic fixes.

The taxonomy:

  1. Timing and async issues — Replace hardcoded waits with condition-based waits
  2. Test data pollution — Isolate data per test
  3. Environment instability — Use Docker, mock external services, fix clocks
  4. Flaky selectors — Use data-testid and role-based selection
  5. External service dependencies — Mock or use contract tests
  6. Test order dependency — Make tests idempotent

The process:

  1. Measure — Track flake rate
  2. Quarantine — Isolate flaky tests to prevent blocking CI
  3. Fix — Apply taxonomy-based fixes
  4. Prevent — Code review, CI gating, flake budget

Treat flake rate below 2% as a non-negotiable quality bar. Once trust is lost, the entire test suite becomes worthless.

In the final post of this series, we’ll cover contract tests vs E2E — when to validate integration boundaries with lightweight contract tests instead of heavy E2E automation, and when you still need full user journey coverage.

Action for this week: Measure your current flake rate. Run your full test suite 10 times (manually or in CI). Count how many runs have at least one failure. If flake rate > 2%, identify the three most frequent offenders using CI logs. Quarantine them (.fixme() or @flaky tag). Dedicate 2 hours this week to fixing one using the taxonomy in this post.