When to Automate — and When Not To

“We should automate everything.”

This sentence, uttered with the best intentions, has launched a thousand doomed automation projects. Teams spend months building vast E2E suites that take hours to run, break on every UI change, and catch fewer bugs than a single skilled tester with 30 minutes of exploratory testing time.

The right question is not “can we automate this?” but “should we automate this?” And the answer is often “no.”

This is the first post in the Automation Strategy series. We’ll build a practical framework for deciding what to automate, when to automate it, and — just as importantly — when manual or exploratory testing is the better investment.

The False Promise of “Automate Everything”

Test automation is not free. Every automated test has a lifecycle cost:

Development time — Writing the test, integrating it into CI, handling edge cases
Execution time — Runtime in CI/CD pipelines, compute cost for cloud runners
Maintenance time — Updating tests when the application changes, fixing flaky tests, refactoring brittle selectors
Cognitive load — Understanding what the test actually validates when it fails

When these costs exceed the value the test provides, automation becomes a liability.

:::warning[Automation Debt is Real] Unmaintained automated tests are worse than no tests. They fail for the wrong reasons, erode trust in the suite, and consume engineering time investigating false failures. A test that hasn’t caught a real bug in six months but breaks weekly is technical debt, not an asset. :::

The goal is not “100% automation coverage.” The goal is fast, reliable feedback on the things that matter most.

The Decision Framework — Four Criteria

A test is a strong automation candidate if it satisfies at least three of these four criteria:

1. Stable

The underlying functionality does not change frequently.

Good candidates:

User authentication flow
Password reset process
Core checkout logic
Data import/export functionality

Poor candidates:

A feature currently in active development where requirements change daily
UI components undergoing design iteration
Experimental features with high churn

If the test breaks every sprint because the feature itself is evolving, automation cost exceeds value. Manual testing during the development phase, automation once the feature stabilises.

2. Frequent

The test runs many times per week or per sprint.

Good candidates:

Smoke tests that run on every commit
Regression tests for critical paths (login, checkout, payment)
API contract tests that guard integration boundaries

Poor candidates:

Year-end tax calculation logic that runs once per year
One-time data migration validation
Deprecated features scheduled for removal

If a test runs five times a year, even a perfectly reliable automated test will struggle to justify its development and maintenance cost. Run it manually when needed.

3. High-Risk

Failures here have real business or safety impact.

Good candidates:

Payment processing
User data deletion (GDPR compliance)
Access control and permissions
Billing and invoicing logic

Poor candidates:

Button hover effects
Marketing copy on a landing page
Internal admin tools used by two people

Risk is not uniform across a system. A broken “Add to Cart” button is a business-critical defect. A broken “Export to PDF” button in an internal reporting tool might be low-priority. Allocate automation effort proportional to risk.

4. Tedious Manually

Long, repetitive test sequences that are error-prone when executed by hand.

Good candidates:

Multi-step form wizards (e.g., insurance application, loan approval)
End-to-end user journeys with many clicks and data entry steps
Regression suites covering 20+ integration points
Load/performance testing (hundreds of concurrent users)

Poor candidates:

Visual design review (“does this layout look right?”)
Exploratory testing of a new feature (“what breaks if I…?”)
Subjective UX evaluation (“is this workflow intuitive?”)

If a manual tester can complete the test in 2 minutes with full confidence, automation might cost more than it saves.

When Exploratory Testing Wins

Exploratory testing is not a fallback for “things we haven’t automated yet.” It is a first-class testing technique that finds bugs automation cannot.

Automated tests validate known behaviour against expected outcomes. Exploratory testing discovers unknown problems you didn’t anticipate.

Exploratory Testing is Better For:

New features — A feature that shipped this sprint has not yet proven what “normal” looks like. Automated tests check expected paths. Exploratory testing finds the weird edge case where the payment form accepts emoji characters and crashes the backend.

Complex interactions — Multi-user workflows, race conditions, network failures, mobile/desktop differences. These are hard to script and change as the system evolves. A human tester finds them faster than automating them.

UX and usability — Is this error message helpful? Is this workflow confusing? Does the mobile layout break on a small screen? Automated tests cannot answer subjective questions.

Regression discovery — When a critical bug escapes to production, the question is not “why didn’t we automate this?” but “why didn’t we know this scenario mattered?” Exploratory testing discovers the scenarios you should have automated.

:::tip[Automation and Exploration are Complementary] Automated tests guard known risks. Exploratory testing discovers unknown risks. High-quality QA uses both. The mistake is trying to automate exploratory testing — you end up with either useless tests or no exploration. :::

Real Examples — Automation Decisions

Let’s apply the framework to real scenarios.

Stable? Yes. Login logic rarely changes.
Frequent? Yes. Runs on every deploy (100+ times/month).
High-risk? Yes. Broken login blocks all users.
Tedious? Moderately. 5–10 manual steps per test.

Decision: Automate. Meets all four criteria. High ROI.

Example 2: Admin Dashboard UI Redesign

Stable? No. Design is actively changing.
Frequent? Medium. Used daily by 5 internal users.
High-risk? Low. Internal tool, limited business impact.
Tedious? No. Quick to validate manually.

Decision: Do not automate yet. Exploratory + manual testing during development. Consider automation once the design stabilises.

Example 3: Annual Tax Report Generation

Stable? Yes. Tax logic is governed by regulation.
Frequent? No. Runs once per year.
High-risk? High. Incorrect tax reports have legal consequences.
Tedious? No. Single-button operation.

Decision: Do not automate. High risk argues for automation, but extremely low frequency (1×/year) means ROI is negative. Instead: comprehensive manual checklist, peer review, sandbox testing before production run.

Example 4: Checkout Flow (E-commerce)

Stable? Yes. Core business logic.
Frequent? Yes. Regression tested on every release.
High-risk? Yes. Broken checkout = lost revenue.
Tedious? Yes. Multi-step process with payment integration.

Decision: Automate. This is the textbook case for automation. Write E2E tests covering the critical path, API tests for edge cases, and supplement with exploratory testing for UX and new payment methods.

Calculating Automation ROI

The framework gives you a qualitative answer. For high-stakes decisions, calculate the numbers.

The Formula

ROI = (Time Saved × Cost per Hour) − (Development + Maintenance Cost)

Worked Example

Scenario: Regression suite, 10 hours to run manually, executed 40 times/year. QA cost: £45/hour.

Item	Calculation	Cost
Manual testing cost (annual)	10h × 40 runs × £45	£18,000
Automation development (one-time)	60h × £70/hr (dev rate)	£4,200
Automation maintenance (annual)	25% of dev cost	£1,050/year
Year 1 net saving	£18,000 − £4,200 − £1,050	£12,750
Year 2+ saving	£18,000 − £1,050	£16,950/year

Year 1 ROI: +£12,750 (304% return).

This is compelling. But change the frequency to 5 runs/year and the manual cost drops to £2,250 — less than the development cost. Automation loses money.

:::info[Include Maintenance Cost] A common mistake: estimating only development cost and ignoring maintenance. Realistic maintenance cost is 20–30% of initial development cost per year. For brittle E2E tests on frequently-changing UIs, it can exceed 50%. :::

Common Anti-Patterns

Anti-Pattern #1: Automating to Meet a Coverage Target

“We need 80% test coverage.”

Coverage targets divorced from risk and value lead to useless tests. Teams automate easy, low-value tests (getter/setter unit tests, trivial UI interactions) to hit the number while high-risk integration points remain untested.

Fix: Define coverage in terms of risk, not lines of code. “All payment flows have automated E2E coverage” is better than “80% code coverage.”

Anti-Pattern #2: Automating Tests That Don’t Fail

If a test has never failed — not once in 18 months — it is either testing something trivial or duplicating coverage elsewhere.

Fix: Delete tests that have never caught a real bug. Automation is not free; every test must justify its existence.

Anti-Pattern #3: 1:1 Manual-to-Automated Test Conversion

Transcribing a 50-step manual test case into a 50-step Playwright script. The result: a slow, brittle test that breaks on every minor UI change.

Fix: Automated tests should validate behaviour, not replicate manual steps. A 50-step manual test might become three focused automated tests: one for the happy path, one for validation errors, one for data persistence.

Building Your Automation Backlog

Here’s how to prioritise what to automate first:

List all test scenarios (manual test cases, exploratory charters, regression risks).
Score each scenario using the four criteria (Stable, Frequent, High-Risk, Tedious). 1 point per criterion satisfied.
Sort by score.
Estimate development + annual maintenance cost for the top candidates.
Start with high-score, low-cost tests. Build momentum with quick wins.

You will have tests that score 4/4 and tests that score 0/4. Automate the former. Question why you’re even testing the latter.

Conclusion

Not every test should be automated. The decision framework is simple:

Stable, Frequent, High-Risk, Tedious → Strong automation candidate
Unstable, Rare, Low-Risk, Quick → Manual or exploratory testing

Automation is an investment. Treat it like one: estimate cost, measure ROI, and be willing to say “no” when manual testing is cheaper and faster.

In the next post, we’ll look at the Test Automation Pyramid and how to distribute automation effort across layers (unit, integration, E2E) to maximise value and minimise maintenance cost.

Action for this week: Take three manual tests you’ve considered automating. Score each using the four criteria (Stable, Frequent, High-Risk, Tedious). For the highest-scoring test, estimate development time and run the ROI calculation. If ROI is positive, add it to your backlog. If not, keep it manual and move on.