AI-Assisted Test Design — What Works Today

Explore how AI tools can help with test charters, test case generation, and edge case brainstorming — and why human review remains essential for quality and coverage.

#testing #qa-automation #software-engineering

This post is the first part of the AI for QA series. We’ll explore how AI-assisted tools fit into test design, where they add value, and where they fall short.

Introduction — The AI Hype in Testing

AI-powered test generation tools promise to revolutionize QA. “Generate comprehensive test suites from requirements in seconds.” “AI that writes tests better than humans.” “Eliminate manual test design forever.”

The reality is more nuanced.

AI tools can help with test design. They’re excellent at brainstorming edge cases, generating boilerplate test cases, and suggesting coverage gaps you might have missed. But they don’t replace human judgment. They don’t understand the business context. They don’t know which edge cases actually matter to your users.

AI-assisted test design is a force multiplier, not a replacement. When used well, it speeds up the boring parts of test design and frees you to focus on the strategic decisions: what to test, why, and how much is enough.

This post will show you what works today, what doesn’t, and how to use AI tools effectively without falling into the trap of trusting generated tests blindly.

Where AI Helps: Test Charters and Exploratory Testing

Test charters define the scope, focus, and risks for an exploratory testing session. They’re short, focused statements like:

Explore the checkout flow to identify payment validation failures and error handling gaps.

Writing good charters requires understanding the feature, the risks, and the user journey. AI can help you brainstorm charters based on requirements or feature descriptions.

Example: AI-Generated Charters

Input (feature description):

We’re adding a new feature that lets users schedule recurring payments. Users can choose weekly, monthly, or yearly intervals. Payments are processed automatically on the scheduled date.

AI Output (generated test charters):

  1. Explore recurring payment setup to identify edge cases around date selection (e.g., Feb 29, end-of-month dates)
  2. Test payment failure scenarios — what happens if the card is declined on a scheduled date?
  3. Verify cancellation workflow — can users cancel recurring payments? Is there confirmation?
  4. Check timezone handling — do scheduled payments respect the user’s timezone or server time?
  5. Test boundary cases — what’s the minimum and maximum interval? Can users schedule payments years in advance?

Value: AI quickly generates a starting point. You review the list, remove irrelevant charters, add missing ones, and prioritize based on risk.

:::tip[AI as a Brainstorming Partner] Use AI to generate ideas, not final test plans. Treat AI output as a first draft that you refine with business knowledge, risk analysis, and user empathy. :::

Where AI Helps: Generating Test Cases from Specifications

If you have structured requirements (user stories, acceptance criteria, API specs), AI can generate test cases automatically.

Example: API Test Case Generation

Input (API specification):

POST /api/orders
Request:
  productId: string (required)
  quantity: integer (required, min: 1, max: 100)
  couponCode: string (optional)

Response:
  201 Created: { orderId, total, discount }
  400 Bad Request: { error }
  404 Not Found: { error }

AI Output (generated test cases):

  1. Happy path: POST with valid productId and quantity → expect 201 and valid orderId
  2. Missing required field: POST without productId → expect 400 with error message
  3. Invalid quantity: POST with quantity: 0 → expect 400
  4. Boundary case: POST with quantity: 100 → expect 201
  5. Boundary case: POST with quantity: 101 → expect 400
  6. Optional field: POST with valid couponCode → expect 201 with discount applied
  7. Invalid coupon: POST with invalid couponCode → expect 400 or 201 (depending on business logic)

Value: AI generates boilerplate test cases quickly. You review them, fix ambiguities (like test case 7), and add business-specific scenarios.

:::info[AI Doesn’t Know Your Business Logic] AI can infer basic validation rules from specs, but it doesn’t know your business logic. “What happens if the coupon is expired?” “What if the product is out of stock?” You still need to add those cases manually. :::

Where AI Helps: Edge Case Brainstorming

Humans are bad at exhaustive edge case enumeration. We think of the obvious cases and miss the obscure ones. AI is good at combinatorial thinking — generating permutations and edge cases systematically.

Example: Edge Cases for a Date Input

Feature: Users enter a date of birth to verify age for an age-restricted product.

Human-generated test cases (typical):

  • Valid date → pass
  • Future date → fail
  • User under 18 → fail
  • User over 18 → pass

AI-generated edge cases (additional):

  • February 29 on a leap year → pass or fail depending on age calculation
  • February 29 on a non-leap year → validation error
  • Date with year 1900 → edge case for minimum age
  • Date with year 2100 → edge case for future validation
  • Date entered in different formats (DD/MM/YYYY vs MM/DD/YYYY) → locale handling
  • Empty date field → validation error
  • Partially filled date (day and month, but no year) → validation error

Value: AI surfaces edge cases you might not think of in a 15-minute manual test design session.

Where AI Struggles: Understanding Business Context

AI doesn’t know your product. It doesn’t know which features are high-risk, which edge cases are common in production, or which user workflows matter most to your business.

Example: Misguided Test Prioritization

Scenario: You’re testing a SaaS billing system. AI generates 50 test cases for the invoicing feature.

AI-generated priority order (alphabetical or spec-based):

  1. Test invoice generation for monthly subscription
  2. Test invoice generation for yearly subscription
  3. Test invoice PDF download
  4. Test invoice email delivery
  5. Test invoice currency formatting … (50 more cases)

Reality: The most critical test is “Does the customer get charged the correct amount?” If that fails, the business loses money or faces chargebacks. The PDF formatting is low-risk.

Human-prioritized order:

  1. Test correct charge amount (high risk, high impact)
  2. Test failed payment handling (high risk, high impact)
  3. Test invoice sent to correct email (medium risk, high impact)
  4. Test invoice PDF download (low risk, low impact)
  5. Test invoice currency formatting (low risk, low impact)

:::warning[AI Can’t Assess Risk] AI doesn’t know which failures cost your business money, which frustrate users most, or which features are actively changing. You need to manually prioritize generated test cases based on risk and impact. :::

Where AI Struggles: Generating Maintainable Test Code

Some AI tools offer to generate full test automation code (Playwright, Selenium, Cypress). The code often works — but it’s not maintainable.

Example: AI-Generated Playwright Test

AI Output:

test('user can checkout', async ({ page }) => {
  await page.goto('http://localhost:3000/cart');
  await page.click('button.btn-primary.checkout');
  await page.fill('input[name="email"]', 'test@example.com');
  await page.fill('input[name="cardNumber"]', '4242424242424242');
  await page.click('button[type="submit"]');
  await page.waitForTimeout(3000);
  await expect(page.locator('div.success-message')).toBeVisible();
});

Problems:

  1. Brittle selectors — button.btn-primary.checkout will break if CSS classes change
  2. Hardcoded timeout — waitForTimeout(3000) is flaky and arbitrary
  3. No page object model — logic is mixed with test code, hard to reuse
  4. No data seeding — depends on existing cart state, will fail if cart is empty

Human-refactored version:

test('user can checkout', async ({ page, request }) => {
  // Seed test data
  const cart = await request.post('/api/test/seed-cart', {
    data: { productId: 123, quantity: 1 }
  });
  
  await page.goto('/cart');
  await page.click('[data-testid="checkout-button"]');
  await page.fill('[data-testid="email-input"]', 'test@example.com');
  await page.fill('[data-testid="card-number-input"]', '4242424242424242');
  await page.click('[data-testid="submit-button"]');
  
  await expect(page.locator('[data-testid="success-message"]')).toBeVisible();
});

Better because:

  • Uses test IDs (stable, won’t break on UI changes)
  • Seeds data (deterministic, no flakiness from pre-existing state)
  • No arbitrary timeout (Playwright’s auto-waiting handles it)

:::tip[Review All AI-Generated Code] Never commit AI-generated test code without review. Treat it as a draft that needs refactoring for maintainability, stability, and adherence to your team’s conventions. :::

Where AI Helps: Documentation and Test Case Templates

AI is excellent at generating structured documentation from unstructured input.

Example: Converting Notes to Test Cases

Input (rough notes from a planning meeting):

Users should be able to reset their password. They enter their email, get a reset link, click it, enter a new password, and log in.

AI Output (structured test cases):

Test CasePreconditionStepsExpected Result
Happy pathUser exists with email user@example.com1. Click “Forgot password”
2. Enter email
3. Submit
4. Open email
5. Click reset link
6. Enter new password
7. Submit		Redirected to login; can log in with new password
Invalid emailUser does not existEnter non-existent emailError: “Email not found”
Expired linkReset link is >24 hours oldClick expired reset linkError: “Link expired”
Weak passwordValid reset linkEnter weak password (e.g., “123”)Error: “Password must be at least 8 characters”

Value: AI structures your notes into a reviewable format. You add missing edge cases and adjust based on actual business rules.

The Human Review Gate

AI-assisted test design is valuable, but it’s not autonomous. Every AI-generated artifact needs human review before it’s trusted.

Review Checklist for AI-Generated Tests

  1. Does this test reflect actual business logic? (AI doesn’t know your product)
  2. Is this test prioritized correctly? (AI doesn’t assess risk)
  3. Does this test use stable locators? (AI often generates brittle selectors)
  4. Is this test deterministic? (AI often uses arbitrary waits)
  5. Does this test follow our conventions? (AI doesn’t know your team’s standards)

Real-World Workflow: AI-Assisted Test Design

Here’s how a QA engineer might use AI effectively:

  1. Feature kickoff: Read the feature spec, paste it into an AI tool, ask for test charters
  2. Charter review: Review AI-generated charters, remove irrelevant ones, add business-specific ones
  3. Test case generation: Use AI to generate boilerplate test cases from API specs
  4. Edge case brainstorming: Ask AI “What edge cases am I missing for date input validation?”
  5. Prioritization: Manually prioritize test cases based on risk, impact, and likelihood
  6. Test implementation: Write test automation using AI-generated cases as a guide, but refactor for maintainability
  7. Code review: Review AI-generated test code, fix brittle selectors, add data seeding, remove arbitrary waits

Total time saved: ~30% faster than writing everything from scratch
Quality: Same or better (AI surfaces edge cases you’d miss, but you fix its mistakes)

Conclusion

AI-assisted test design is valuable when used as a brainstorming partner, not as an autonomous test generator.

What works today:

  • Test charter generation
  • Boilerplate test case generation
  • Edge case brainstorming
  • Documentation and templates

What doesn’t work:

  • Autonomous test prioritization
  • Maintainable test code generation
  • Business context understanding

The key is human review. Treat AI output as a draft, not a final product. Review, refactor, and prioritize based on your knowledge of the product, the users, and the risks.

:::tip[AI Improves Your Baseline, Not Your Ceiling] AI won’t make a novice tester into an expert. But it will make an expert tester faster by handling the mechanical parts of test design and surfacing edge cases they might have missed. :::

Action for this week: Try an AI tool (ChatGPT, GitHub Copilot, or a dedicated test generation tool) on one feature. Generate test charters or test cases. Review the output. Measure how much time you saved vs how much time you spent fixing AI mistakes. Decide if it’s worth integrating into your workflow.

Next in the series: Part 2 — AI for Test Maintenance, where we’ll explore how AI tools help (and hurt) test suite maintenance, selector updates, and refactoring.