Auto-Waiting, Assertions, and Killing Flakes

Master Playwright's web-first assertions and auto-waiting to eliminate flaky tests. Learn why waitForTimeout is a bug, when networkidle helps, how expect.poll works, and whether to retry or fix the.

#playwright #e2e #testing #qa-automation

This post is part of the Playwright Essentials series. Part 3 covered structuring tests with page objects and fixtures. This post focuses on writing tests that don’t fail randomly — the art of killing flakiness.

What Makes a Test Flaky?

A flaky test is a test that passes sometimes and fails sometimes without any code changes. Flakiness is poison for test suites. When engineers lose trust in the suite, they stop acting on failures. Real bugs slip through.

The most common cause of flakiness in E2E tests: timing issues. The test runs faster than the application, or the application runs faster than the test expects.

Playwright’s solution: auto-waiting and web-first assertions. These features eliminate most timing issues without manual waits.

Auto-Waiting — What It Is and Why It Matters

Playwright automatically waits for elements to be actionable before performing actions. Actionable means:

  • Attached to DOM — the element exists in the page
  • Visible — the element is not display: none or visibility: hidden
  • Stable — the element is not animating or moving
  • Receives events — the element is not obscured by another element
  • Enabled — the element is not disabled

Example: Auto-Waiting in Action

await page.getByRole('button', { name: 'Submit' }).click();

Playwright waits until:

  1. The button exists in DOM
  2. The button is visible
  3. The button is not covered by a modal or spinner
  4. The button is enabled (not disabled attribute)
  5. The button is stable (not animating)

Only then does Playwright click. This happens automatically. No waitForTimeout, no manual checks.

What Auto-Waiting Does NOT Cover

Auto-waiting applies to actions (click, fill, check, select), but not to queries (isVisible(), textContent(), getAttribute()).

// ❌ BAD: No auto-waiting on queries
if (await page.getByRole('button').isVisible()) {
  await page.getByRole('button').click();
}

The isVisible() call doesn’t wait. It returns immediately. If the button isn’t in DOM yet, it returns false even though the button will appear 100ms later.

// ✅ GOOD: Use assertions, which DO auto-wait
await expect(page.getByRole('button')).toBeVisible();
await page.getByRole('button').click();

Assertions retry until the condition is met (or timeout).

Web-First Assertions — The Right Way to Check State

Playwright provides web-first assertions that automatically retry until the condition is true. These are the foundation of reliable tests.

Visibility Assertions

// Wait until element is visible (retries for up to 5 seconds)
await expect(page.getByText('Order confirmed')).toBeVisible();

// Wait until element is NOT visible (e.g., loading spinner disappears)
await expect(page.getByTestId('loading-spinner')).not.toBeVisible();

Text Content Assertions

// Wait until element contains specific text
await expect(page.getByRole('heading')).toHaveText('Welcome');

// Partial text match
await expect(page.getByRole('status')).toContainText('Processing');

// Regex match
await expect(page.getByRole('alert')).toHaveText(/error|warning/i);

Attribute Assertions

// Wait until input has specific value
await expect(page.getByLabel('Email')).toHaveValue('user@example.com');

// Wait until button is enabled
await expect(page.getByRole('button', { name: 'Submit' })).toBeEnabled();

// Wait until element has specific class
await expect(page.locator('.status')).toHaveClass(/active/);

Count Assertions

// Wait until exactly 3 items in list
await expect(page.getByRole('listitem')).toHaveCount(3);

// Wait until at least 1 item
await expect(page.getByRole('row')).toHaveCount({ minimum: 1 });

URL Assertions

// Wait until URL matches pattern
await expect(page).toHaveURL(/\/dashboard/);

// Wait until URL is exact
await expect(page).toHaveURL('https://example.com/profile');

Title Assertions

// Wait until page title matches
await expect(page).toHaveTitle('Dashboard | MyApp');

All of these assertions retry automatically. Default timeout: 5 seconds (configurable).

The waitForTimeout Anti-Pattern

If you have waitForTimeout in your tests, treat it as a bug.

Why It’s Wrong

// ❌ BAD: Hardcoded wait
await page.getByRole('button', { name: 'Submit' }).click();
await page.waitForTimeout(2000);  // Hope 2 seconds is enough
await expect(page.getByText('Success')).toBeVisible();

This test:

  • Wastes time — waits 2 seconds even if the response comes in 200ms
  • Still flaky — fails if the response takes 2.1 seconds
  • Environment-dependent — 2 seconds might work locally but fail in CI

The Fix

// ✅ GOOD: Wait for the condition
await page.getByRole('button', { name: 'Submit' }).click();
await expect(page.getByText('Success')).toBeVisible();  // Waits up to 5s

This test:

  • Fast — continues as soon as “Success” appears
  • Reliable — retries for 5 seconds (configurable timeout)
  • Portable — works the same locally and in CI

:::warning[The Only Valid Use of waitForTimeout] Debugging. When you need to see what the page looks like mid-test, await page.waitForTimeout(5000) pauses execution. Remove it before committing. :::

Network Waiting — When and How to Use It

Sometimes you need to wait for network activity, not just DOM changes. Playwright provides tools for this.

Wait for Specific Network Request

// Wait for API call to complete before proceeding
const responsePromise = page.waitForResponse('**/api/orders');
await page.getByRole('button', { name: 'Load orders' }).click();
const response = await responsePromise;

expect(response.status()).toBe(200);

Wait for Navigation

// Wait for navigation after clicking a link
await Promise.all([
  page.waitForNavigation(),
  page.getByRole('link', { name: 'Dashboard' }).click(),
]);

Or use the simpler waitForURL:

await page.getByRole('link', { name: 'Dashboard' }).click();
await page.waitForURL('**/dashboard');

Wait for Load State

// Wait until network is mostly idle
await page.waitForLoadState('networkidle');

// Wait until DOM is loaded (but scripts may still be running)
await page.waitForLoadState('domcontentloaded');

// Wait until page is fully loaded (default for goto)
await page.waitForLoadState('load');

:::info[The networkidle Myth] networkidle waits until there are no network requests for 500ms. This sounds useful but is often a trap. Modern SPAs continuously poll APIs (analytics, websockets, live updates). networkidle may never resolve. Use targeted waits (specific element visible, specific API call completed) instead. :::

Polling with expect.poll — Custom Conditions

Sometimes you need to wait for a condition that isn’t a simple DOM check. expect.poll() lets you define custom retry logic.

Example: Wait for API Data to Update

// Poll an API endpoint until the order status changes
await expect.poll(async () => {
  const response = await page.request.get('/api/orders/123');
  const order = await response.json();
  return order.status;
}, {
  message: 'Order status should become "completed"',
  timeout: 10_000,  // 10 seconds
}).toBe('completed');

expect.poll() retries the callback function until the assertion passes or timeout.

Example: Wait for File to Appear

// Poll filesystem until file exists (useful for download tests)
import fs from 'fs/promises';

await expect.poll(async () => {
  try {
    await fs.access('./downloads/report.pdf');
    return true;
  } catch {
    return false;
  }
}, {
  timeout: 5000,
}).toBe(true);

Use expect.poll() sparingly. Most cases are better handled by web-first assertions.

Configuring Timeouts

Default timeout for assertions: 5 seconds. You can override globally or per-assertion.

Global Timeout (playwright.config.ts)

export default defineConfig({
  expect: {
    timeout: 10_000,  // 10 seconds for all assertions
  },
});

Per-Test Timeout

test('slow operation', async ({ page }) => {
  test.setTimeout(30_000);  // This test gets 30 seconds total
  
  await page.goto('/long-running-operation');
  await expect(page.getByText('Complete')).toBeVisible({ timeout: 15_000 });
});

Per-Assertion Timeout

// Override timeout for a single assertion
await expect(page.getByText('Data loaded')).toBeVisible({ timeout: 15_000 });

:::tip[Timeout Guidelines]

  • Default (5s) — sufficient for 95% of assertions
  • Increase (10-15s) — for known-slow operations (file uploads, report generation)
  • Decrease (1-2s) — for fast-fail scenarios (testing error states)

If you find yourself increasing timeouts frequently, the application is too slow or the test strategy is wrong. :::

Retries — Safety Net or Smell?

Playwright supports test retries. If a test fails, it can automatically re-run.

Enable Retries (playwright.config.ts)

export default defineConfig({
  retries: process.env.CI ? 2 : 0,  // Retry twice in CI, never locally
});

When Retries Are Appropriate

Infrastructure flakiness — CI runners are slower than local machines. Network hiccups, container startup delays, and resource contention can cause intermittent failures. Retries smooth over these issues.

Third-party flakiness — If you’re testing integration with an external service (payment gateway, email provider), their occasional downtime shouldn’t block your pipeline.

When Retries Are Wrong

Application bugs — If a test fails because your code has a race condition, retrying hides the bug. Fix the race condition.

Test bugs — If a test fails because it has a brittle selector, retrying papers over the problem. Fix the selector.

Slow tests — Retrying a 5-minute test multiplies CI time. Fix the test speed instead.

:::warning[Retries Are a Code Smell] If your test suite needs retries to pass, you have flaky tests. Retries are a safety net for infrastructure issues, not a substitute for fixing root causes. Track retry rate. If it’s above 2%, investigate. :::

Real-World Flake Fixes

Let’s refactor common flaky patterns.

Flake #1: Race Condition on Data Load

// ❌ FLAKY: Clicks before data loads
test('view first order', async ({ page }) => {
  await page.goto('/orders');
  await page.getByRole('row').first().click();  // May not exist yet
});
// ✅ FIXED: Wait for data to load
test('view first order', async ({ page }) => {
  await page.goto('/orders');
  await expect(page.getByRole('row')).toHaveCount({ minimum: 1 });
  await page.getByRole('row').first().click();
});

Flake #2: Button Enabled But Not Ready

// ❌ FLAKY: Button enabled, but form validation runs async
test('submit form', async ({ page }) => {
  await page.goto('/contact');
  await page.getByLabel('Email').fill('test@example.com');
  await page.getByRole('button', { name: 'Submit' }).click();  // May fail
});
// ✅ FIXED: Wait for submit button to be stable
test('submit form', async ({ page }) => {
  await page.goto('/contact');
  await page.getByLabel('Email').fill('test@example.com');
  
  // Wait for validation to complete (button re-enables after validation)
  const submitButton = page.getByRole('button', { name: 'Submit' });
  await expect(submitButton).toBeEnabled();
  await submitButton.click();
});

Flake #3: Stale Element After Page Update

// ❌ FLAKY: DOM changes between locator creation and action
test('delete order', async ({ page }) => {
  await page.goto('/orders');
  const deleteButton = page.getByRole('button', { name: 'Delete' }).first();
  await page.getByRole('button', { name: 'Refresh' }).click();
  await deleteButton.click();  // Element stale — page refreshed
});
// ✅ FIXED: Locate element after page stabilizes
test('delete order', async ({ page }) => {
  await page.goto('/orders');
  await page.getByRole('button', { name: 'Refresh' }).click();
  await expect(page.getByRole('row')).toHaveCount({ minimum: 1 });
  
  // Locate after refresh
  await page.getByRole('button', { name: 'Delete' }).first().click();
});

Debugging Flaky Tests

When a test fails intermittently, use these tools:

1. Run with Trace

npx playwright test --trace on

Capture full trace (screenshots, network, DOM) for every test. Open with:

npx playwright show-report

The trace viewer shows exactly what happened, frame by frame.

2. Run in Headed Mode

npx playwright test --headed --workers=1

Watch the browser. Sometimes seeing the test run reveals the issue (animation timing, modal overlay, etc.).

3. Repeat Until Failure

npx playwright test --repeat-each=10 path/to/flaky.spec.ts

Runs the test 10 times. If it fails once, you’ve reproduced the flake.

4. Add Explicit Logging

test('flaky test', async ({ page }) => {
  console.log('Navigating to /orders');
  await page.goto('/orders');
  
  console.log('Waiting for rows');
  await expect(page.getByRole('row')).toHaveCount({ minimum: 1 });
  
  console.log('Clicking first row');
  await page.getByRole('row').first().click();
});

Check the logs to see where the test failed.

Flakiness Checklist

Use this to audit your test suite:

☐ No waitForTimeout in any test
☐ No waitForLoadState('networkidle') unless application truly idles
☐ All queries (isVisible, textContent) replaced with assertions
☐ All assertions use web-first assertions (toBeVisible, toHaveText)
☐ Retry count tracked in CI (should be < 2%)
☐ Timeouts increased only for known-slow operations
☐ Race conditions fixed at root cause, not papered over with retries
☐ Tests pass 10 times in a row locally

Conclusion

Flaky tests are not inevitable. Playwright’s auto-waiting and web-first assertions eliminate 90% of timing issues if used correctly.

Golden rules:

  1. Use web-first assertions (toBeVisible, toHaveText) — never raw queries (isVisible(), textContent())
  2. Never use waitForTimeout except for debugging
  3. Wait for specific conditions, not arbitrary durations
  4. Retries are for infrastructure flakes, not test bugs

A test suite that requires retries to pass is a test suite with bugs. Fix the root cause.

Action for this week: Find one flaky test in your suite. Run it 10 times with --repeat-each=10. Capture a failure with --trace on. Use the trace viewer to identify the root cause. Fix it. Then run it 10 more times to prove the fix.

Previous: Part 3 — Page Objects vs Fixtures
Next: Part 5 — API Testing with Playwright