Test Data Strategies for Reliable Automation

This post is part of the Automation Strategy series. If you missed the previous post, read Part 3 — What to Automate First first.

---

“The tests pass on my machine.”

This phrase is rarely about code. It’s about data.

You run a test locally, it passes. CI runs it, it fails. A colleague runs it, it fails. You investigate: someone deleted the test user. A previous test modified shared data. The staging database was restored from production and now has different IDs.

Bad test data strategy is the most common cause of flaky, unreliable automated tests — and it’s entirely preventable.

In this post, we’ll cover how to design test data for automation: factories, isolation, cleanup, API seeding, and the dangers of shared database fixtures.

---

The Problem: Shared, Polluted Test Data

Most teams start with a simple approach: seed a test database with fixture data, then write tests that assume it exists.

// Bad: depends on data someone else created
test('user can view their order history', async () => {
  await loginAs('testuser@example.com', 'password123');
  await expect(page.getByText('Order #12345')).toBeVisible();
});

This test assumes:

• testuser@example.com exists
• The user has at least one order
• That order has ID #12345

What happens when:

• Someone deletes testuser@example.com while debugging?
• Another test creates order #12345 for a different user?
• The test database is wiped and re-seeded with different order IDs?

The test fails — not because the code is broken, but because the data changed.

:::warning[Shared Data is a Flaky Test Factory] Any test that depends on pre-existing, shared data will eventually fail for reasons unrelated to the code. Tests should own their data: create it, use it, clean it up. :::

---

Principle #1: Each Test Creates and Owns Its Data

The fix is simple in concept: every test creates the data it needs before running, and cleans it up after.

Before

// Bad: depends on pre-existing data
test('can place order', async () => {
  await loginAs('testuser@example.com', 'password123');
  // ...test logic
});

After

// Good: creates its own user
test('can place order', async ({ request }) => {
  const user = await createUser(request, {
    email: `test-${Date.now()}@example.com`,
    password: 'Test1234!'
  });

  await loginAs(user.email, 'Test1234!');
  // ...test logic

  await deleteUser(request, user.id);
});

Now the test is isolated. It doesn’t matter what data exists in the database when it starts. It creates what it needs, uses it, and cleans up.

---

Principle #2: Use Factories, Not Hardcoded Fixtures

Hardcoded test data leads to duplication and maintenance burden. Instead, use factories — functions that generate test data on demand.

Factory Pattern

// testHelpers.ts
export const createUser = async (request: APIRequestContext, overrides = {}) => {
  const defaults = {
    email: `user-${Date.now()}-${Math.random().toString(36)}@example.com`,
    password: 'Test1234!',
    firstName: 'Test',
    lastName: 'User',
  };

  const response = await request.post('/api/users', {
    data: { ...defaults, ...overrides }
  });

  return await response.json();
};

Now you can create users with custom properties without duplicating setup logic:

const adminUser = await createUser(request, { role: 'admin' });
const guestUser = await createUser(request, { role: 'guest' });
const premiumUser = await createUser(request, { subscriptionTier: 'premium' });

Factories for Complex Objects

export const createOrder = async (request: APIRequestContext, userId: string, overrides = {}) => {
  const defaults = {
    userId,
    items: [
      { productId: 'prod-123', quantity: 1, price: 29.99 }
    ],
    shippingAddress: {
      street: '123 Test St',
      city: 'Testville',
      postcode: 'TE5 7ST',
    },
    status: 'pending',
  };

  const response = await request.post('/api/orders', {
    data: { ...defaults, ...overrides }
  });

  return await response.json();
};

Now creating an order with custom properties is one line:

const order = await createOrder(request, user.id, { status: 'completed' });

:::tip[Use Factories from Day One] Even if you only have two tests, use factories. The cost is minimal upfront and saves massive refactoring later when you have 200 tests with duplicated setup code. :::

---

Principle #3: Isolate Data to Avoid Race Conditions

When tests run in parallel (which they should for speed), tests can interfere with each other if they modify shared data.

Example: Race Condition

// Test 1
test('admin can delete user', async () => {
  await deleteUser('testuser@example.com');
  // ...
});

// Test 2
test('user can update profile', async () => {
  await loginAs('testuser@example.com', 'password123');
  // Fails because Test 1 deleted the user
});

Both tests use the same user. If they run in parallel, one deletes the user while the other tries to log in. Flaky failure.

Fix: Unique Data Per Test

// Test 1
test('admin can delete user', async ({ request }) => {
  const user = await createUser(request); // unique user
  await deleteUser(user.id);
});

// Test 2
test('user can update profile', async ({ request }) => {
  const user = await createUser(request); // different unique user
  await loginAs(user.email, 'Test1234!');
  // ...
});

Now each test has its own user. They can run in parallel without interference.

Unique Identifiers

To guarantee uniqueness:

const uniqueEmail = `test-${Date.now()}-${Math.random().toString(36).substr(2, 9)}@example.com`;

This ensures even if two tests run at the exact same millisecond, their data won’t collide.

---

Principle #4: Clean Up After Yourself

Tests that leave data behind pollute the database, slow future tests, and can cause unexpected failures.

Cleanup Strategies

Option 1: Explicit Cleanup (Best for API-Driven Tests)

test('can create order', async ({ request }) => {
  const user = await createUser(request);
  const order = await createOrder(request, user.id);

  // ...test logic

  // Cleanup
  await deleteOrder(request, order.id);
  await deleteUser(request, user.id);
});

Pros: Explicit, easy to debug
Cons: Requires cleanup logic in every test

Option 2: Test Fixtures (Playwright)

import { test as base } from '@playwright/test';

const test = base.extend<{ testUser: User }>({
  testUser: async ({ request }, use) => {
    // Setup
    const user = await createUser(request);
    
    // Use
    await use(user);
    
    // Teardown
    await deleteUser(request, user.id);
  },
});

// Now every test gets an auto-cleaned user
test('can place order', async ({ testUser, request }) => {
  const order = await createOrder(request, testUser.id);
  // Cleanup happens automatically
});

Pros: Automatic, less duplication
Cons: Requires understanding test frameworks’ fixture systems

Option 3: Database Transactions (Integration Tests)

[Fact]
public async Task CanCreateOrder()
{
    using var transaction = await _dbContext.Database.BeginTransactionAsync();
    
    var user = await CreateTestUser();
    var order = await _orderService.CreateOrder(user.Id, items);
    
    Assert.NotNull(order);
    
    // Rollback — nothing persists
    await transaction.RollbackAsync();
}

Pros: Fast, guaranteed cleanup
Cons: Only works for database-only tests (not E2E with UI)

---

Principle #5: Avoid Shared Database State

Some teams use a single “test database” shared across all test runs. This is a disaster waiting to happen.

Problems with Shared Databases

• Slow tests — Database fills with old test data, queries slow down
• Flaky tests — Tests interfere with each other via shared records
• Debugging hell — Hard to reproduce issues locally because your database state differs from CI

Solution: Database Per Test Run

Use TestContainers or Docker Compose to spin up isolated databases per test run.

Example: Playwright with Postgres TestContainer

import { PostgreSqlContainer } from '@testcontainers/postgresql';

let container: PostgreSqlContainer;
let connectionString: string;

beforeAll(async () => {
  container = await new PostgreSqlContainer('postgres:16').start();
  connectionString = container.getConnectionUri();
  
  // Run migrations
  await runMigrations(connectionString);
});

afterAll(async () => {
  await container.stop();
});

Now every test run gets a fresh, isolated database. No pollution, no conflicts.

:::tip[Use Docker for Test Isolation] If your CI environment supports Docker (GitHub Actions, GitLab CI, Azure Pipelines all do), use containers for databases, message queues, and external dependencies. Tests become reproducible across any environment. :::

---

Principle #6: Seed Data via API, Not Direct Database Access

Many teams seed test data by running SQL scripts or directly inserting records via ORM.

Problem: This bypasses application logic — validation, triggers, side effects. Tests pass, but real user flows fail.

Bad: Direct Database Insert

// Bad: bypasses validation, doesn't trigger events
await db.query('INSERT INTO users (email, password) VALUES (?, ?)', 
  ['test@example.com', 'plaintext_password']);

If the real application:

• Hashes passwords
• Sends a welcome email
• Creates related records (user profile, preferences)

…the test won’t catch bugs in those flows.

Good: Seed via API

// Good: goes through the real registration flow
const response = await request.post('/api/users', {
  data: {
    email: 'test@example.com',
    password: 'Test1234!',
  }
});

const user = await response.json();

Now the test validates:

• Password hashing works
• Welcome email triggers
• Related records are created

If any of those break, the test fails.

:::info[Test the Real Path] Automated tests are most valuable when they exercise the same code path users do. Seeding via API ensures you test the full flow, not just the happy-path logic. :::

---

Practical Example: E2E Test with Data Isolation

Let’s put it all together.

Full Test with Factories and Cleanup

import { test, expect } from '@playwright/test';
import { createUser, createProduct, deleteUser } from './testHelpers';

test('user can add product to cart and checkout', async ({ page, request }) => {
  // Setup: create isolated test data
  const user = await createUser(request);
  const product = await createProduct(request, { name: 'Test Widget', price: 29.99 });

  // Test
  await page.goto('/login');
  await page.getByLabel('Email').fill(user.email);
  await page.getByLabel('Password').fill('Test1234!');
  await page.getByRole('button', { name: 'Login' }).click();

  await page.goto(`/products/${product.id}`);
  await page.getByRole('button', { name: 'Add to Cart' }).click();
  await expect(page.getByText('Cart (1)')).toBeVisible();

  await page.getByRole('link', { name: 'Checkout' }).click();
  await expect(page.getByText(`Total: £${product.price}`)).toBeVisible();

  // Cleanup
  await deleteUser(request, user.id);
  // Product cleanup would happen here if needed
});

This test:

• Creates its own user and product
• Doesn’t depend on any pre-existing data
• Can run in parallel with other tests without conflicts
• Cleans up after itself

---

Common Pitfalls and Fixes

Pitfall #1: Forgetting to Clean Up

Symptom: Test database grows to millions of records. Queries slow down. Tests start timing out.

Fix: Automate cleanup. Use test fixtures, database transactions, or teardown hooks. Never rely on manual cleanup.

Pitfall #2: Hardcoding IDs

// Bad: assumes product ID 123 exists
const product = await request.get('/api/products/123');

Fix: Create the product in the test or look it up dynamically.

Pitfall #3: Reusing Data Across Tests

// Bad: multiple tests use the same user
const ADMIN_USER = 'admin@example.com';

test('admin can delete users', () => { /* uses ADMIN_USER */ });
test('admin can view logs', () => { /* uses ADMIN_USER */ });

If tests run in parallel, one might modify ADMIN_USER while another is using it.

Fix: Each test creates its own admin user.

Pitfall #4: No Strategy for Large Datasets

Some tests need realistic data volumes (pagination, performance testing).

Fix: Use seeding scripts that run once per environment (not per test). Separate “seed data” (static, shared, read-only) from “test data” (created, modified, deleted per test).

---

Recommended Test Data Stack

Layer	Tool	Purpose
Factories	Custom helpers or Faker.js	Generate unique test data on demand
API Seeding	Your app’s REST/GraphQL API	Seed data via real flows
Isolation	TestContainers / Docker Compose	Spin up isolated databases per run
Cleanup	Test fixtures or transaction rollback	Automatically remove test data

---

Conclusion

Test data is not an afterthought. Bad data strategy is the silent killer of automation — causing flaky tests, slow CI, and wasted debugging time.

The principles are simple:

1. Each test creates and owns its data — no shared fixtures
2. Use factories, not hardcoded data — generate dynamically
3. Isolate data per test — avoid race conditions
4. Clean up after yourself — prevent database pollution
5. Avoid shared database state — use containers for isolation
6. Seed via API, not direct DB access — test real flows

Apply these principles from day one and your test suite will be faster, more reliable, and easier to maintain.

In the next post, we’ll cover designing maintainable automated suites — folder structure, naming conventions, ownership, and how to review test code like production code.

---

Action for this week: Pick one flaky test in your suite. Check if it depends on pre-existing data. If yes, refactor it to create its own data using a factory function. Run the test 10 times in parallel (Playwright: --workers=10). If it passes every time, you’ve fixed the flake.