What to Automate First — A Practical ROI Framework

This post is part of the Automation Strategy series. If you missed the previous post, read Part 2 — The Test Automation Pyramid first.

You have 200 manual test cases. Your manager wants automation. You have two weeks.

Which tests do you automate first?

This is not a hypothetical scenario. This is the reality for most QA teams: infinite test scenarios, finite time, and pressure to “increase automation coverage.”

The wrong answer is “start with test #1 and work down the list.” The right answer requires a framework for prioritising by value.

In this post, we’ll build a scoring system based on risk × frequency × cost that tells you exactly which tests to automate first — and which to leave manual forever.

The Core Insight: Automation is an Investment

Every automated test costs something:

Development time — Writing, reviewing, debugging, integrating into CI
Execution cost — Runtime in CI/CD, compute cost for cloud runners
Maintenance time — Updating selectors, fixing flaky tests, adapting to feature changes
Cognitive load — Understanding what broke when the test fails

Automation only makes sense when value delivered exceeds cost.

The question is not “can we automate this?” but “is this the best use of our automation budget?”

The ROI Formula — Three Factors

We score tests on three dimensions:

Automation Priority = Risk × Frequency × Maintenance Cost

Let’s break each down.

Factor #1: Risk

What breaks if this test fails in production?

Risk is not uniform across your application. A broken login flow blocks every user. A broken “Export to CSV” button in an internal admin tool affects two people.

Risk Scoring

Score	Impact	Examples
5	Critical — business/revenue/data loss/legal	Payment processing, user data deletion, access control
4	High — major user impact	Checkout, login, password reset, critical workflows
3	Medium — affects some users or workflows	Secondary features, internal tools with moderate usage
2	Low — cosmetic or rarely-used	UI polish, deprecated features, low-traffic pages
1	Trivial — no real impact	Button hover effects, marketing copy, tooltips

:::tip[Risk is About Impact, Not Complexity] A simple feature can be high-risk (authentication). A complex feature can be low-risk (admin analytics dashboard used once a month). Don’t confuse technical complexity with business risk. :::

Factor #2: Frequency

How often do you need confidence that this works?

A test executed 100 times per month has 20× the ROI potential of a test executed 5 times per month.

Frequency Scoring

Score	Execution Rate	Examples
5	Every commit (50+ times/week)	Smoke tests, login, core API contracts
4	Daily or per-deploy (10–50 times/week)	Regression suite for critical paths
3	Weekly (sprint regression)	Full regression, edge cases, integrations
2	Monthly or per-release	Uncommon workflows, seasonal features
1	Rarely (quarterly or less)	Year-end reports, one-time migrations, deprecated paths

If a test runs fewer than 5 times per year, automation is almost never justified. Run it manually when needed.

Factor #3: Maintenance Cost (Inverted)

How much effort does this test require to keep working?

Some tests are resilient. Others break every sprint. High-maintenance tests eat ROI.

Maintenance Cost Scoring (Lower is Better)

Score	Stability	Examples
5	Rock-solid — rarely changes	Authentication, core business logic, stable APIs
4	Stable — changes infrequently	Established features with settled UX
3	Moderate — UI tweaks every few sprints	Standard CRUD workflows, common forms
2	High-churn — changes frequently	Features in active development, experimental UX
1	Extremely brittle	Unstable dependencies, frequently redesigned UI, tests with timing issues

:::warning[Don’t Automate Unstable Features Yet] If a feature is changing weekly, automation cost will exceed value. Use manual or exploratory testing during development. Automate once it stabilises. :::

Calculating the Priority Score

For each test scenario, assign scores for Risk (1–5), Frequency (1–5), and Maintenance (1–5). Multiply them:

Priority Score = Risk × Frequency × Maintenance

Maximum score: 5 × 5 × 5 = 125
Minimum score: 1 × 1 × 1 = 1

Worked Examples

Risk: 5 (critical — blocks all users)
Frequency: 5 (every commit, 100+ times/month)
Maintenance: 5 (stable logic, rarely changes)

Priority Score: 5 × 5 × 5 = 125

Decision: Automate immediately. This is the highest-value automation possible.

Example 2: Checkout Flow (E-commerce)

Risk: 5 (broken checkout = lost revenue)
Frequency: 4 (regression tested on every deploy)
Maintenance: 4 (stable, minor UI tweaks occasionally)

Priority Score: 5 × 4 × 4 = 80

Decision: Automate. High ROI. Focus on happy path E2E and edge cases at integration/unit level.

Example 3: Admin Dashboard Redesign (In Progress)

Risk: 3 (internal tool, moderate usage)
Frequency: 3 (weekly manual check)
Maintenance: 1 (UI changing weekly)

Priority Score: 3 × 3 × 1 = 9

Decision: Do not automate yet. Use exploratory testing during development. Revisit after design stabilises and maintenance score improves to 4+.

Example 4: Annual Tax Report Generation

Risk: 5 (legal/compliance consequences)
Frequency: 1 (once per year)
Maintenance: 5 (stable, regulatory-driven logic)

Priority Score: 5 × 1 × 5 = 25

Decision: Do not automate. Despite high risk, extremely low frequency makes ROI negative. Use a comprehensive manual checklist, peer review, and sandbox testing instead.

Example 5: Export to CSV (Admin Tool)

Risk: 2 (low usage, non-critical)
Frequency: 2 (used monthly by 3 people)
Maintenance: 4 (stable export logic)

Priority Score: 2 × 2 × 4 = 16

Decision: Do not automate. Low risk + low frequency = poor ROI. Keep manual.

Building Your Automation Backlog

Here’s the process:

Step 1: List All Test Scenarios

Include:

Existing manual test cases
Exploratory testing charters
Known regression risks
Critical user journeys

You’ll likely have 100–300 items.

Step 2: Score Each Scenario

For each test, assign:

Risk (1–5)
Frequency (1–5)
Maintenance (1–5)

Calculate Priority Score = Risk × Frequency × Maintenance.

Step 3: Sort by Priority Score

Highest scores = highest-value automation candidates.

Step 4: Estimate Development + Maintenance Cost

For the top 20–30 tests (scores above 40), estimate:

Development time — How long to write, integrate, and verify?
Annual maintenance cost — Estimate 20–30% of development cost per year

Step 5: Automate High-Score, Low-Cost Tests First

Start with tests that score 80+ and require < 2 days of development time. Build momentum with quick wins.

Step 6: Revisit Quarterly

Priorities shift. A feature in development (low maintenance score) may stabilise (high maintenance score). A rarely-used feature may become critical. Re-score quarterly and adjust your backlog.

When to Say “No” to Automation

Not every test should be automated. Here’s when to say no:

Case #1: Priority Score Below 20

If a test scores below 20, automation cost almost certainly exceeds value. Keep it manual or delete it entirely.

Example: Testing that a footer link opens in a new tab (Risk: 1, Frequency: 1, Maintenance: 3 → Score: 3). This is noise.

Case #2: Feature is Actively Changing

Even high-risk features can be poor automation candidates during active development.

Example: New checkout wizard in design iteration (Risk: 5, Frequency: 4, Maintenance: 1 → Score: 20). Wait until the design settles.

Case #3: Test Has Never Caught a Real Bug

If a test has run for 12 months and never failed (except due to flakiness or test environment issues), it’s either:

Testing something trivial
Duplicating coverage elsewhere

Delete it or move it to a manual smoke test checklist. Don’t automate it.

Case #4: Manual Execution is Faster Than Maintenance

Some tests are so quick to run manually that automation is never justified.

Example: Visual inspection of a landing page layout (30 seconds manually vs 2 hours to automate with visual regression testing).

Real-World Backlog Example

Here’s a sample automation backlog for an e-commerce application:

Test Scenario	Risk	Freq	Maint	Score	Decision
User login	5	5	5	125	✅ Automate (E2E)
Add to cart	5	5	4	100	✅ Automate (E2E)
Checkout flow	5	4	4	80	✅ Automate (E2E + integration)
Password reset	4	4	5	80	✅ Automate (E2E)
Product search	4	4	4	64	✅ Automate (integration + E2E)
Order history	3	4	4	48	✅ Automate (integration)
Discount codes	4	3	3	36	⚠️ Automate (unit + 1 E2E)
Admin dashboard	3	3	2	18	❌ Manual (in active dev)
Export orders CSV	2	2	4	16	❌ Manual
Footer links	1	1	3	3	❌ Delete test

Start at the top. Automate login, cart, checkout, password reset first. Stop when you hit diminishing returns (scores below 40).

Avoiding Common Pitfalls

Pitfall #1: Automating to Hit a Coverage Metric

“We need 80% automation coverage.”

Coverage targets divorced from value lead to low-ROI automation. Teams automate easy tests (getter/setter unit tests, trivial UI checks) to hit the number while critical paths remain untested.

Fix: Replace coverage targets with risk coverage targets. “All critical user journeys (Risk 4–5) have automated coverage” is better than “80% code coverage.”

Pitfall #2: Starting with Hard Tests

Some teams start with the most complex manual test (“if we can automate this, we can automate anything”). This is backwards.

Fix: Start with high-score, low-complexity tests. Build skills, prove value, and establish patterns before tackling hard cases.

Pitfall #3: Ignoring Maintenance Cost

A test that takes 5 days to write and 3 days/quarter to maintain has a total cost of 5 + (3 × 4 × 3) = 41 days over 3 years.

If that test saves 10 manual test runs/year at 1 hour each, the ROI is negative by year 2.

Fix: Factor maintenance into ROI calculations. Delete tests that cost more to maintain than they save.

Automation Roadmap Template

Here’s a template for building your automation roadmap:

Phase 1: Foundation (Weeks 1–4)

Goal: Prove value with quick wins
Scope: Top 5 tests (scores 80+, dev time < 2 days each)
Outcome: CI pipeline with 5 reliable tests, < 5 min runtime

Phase 2: Critical Paths (Weeks 5–12)

Goal: Automate all critical user journeys
Scope: Tests scoring 60+, covering Risk 4–5 scenarios
Outcome: Automated regression suite for all critical paths

Phase 3: Expansion (Weeks 13–24)

Goal: Broaden coverage to medium-risk areas
Scope: Tests scoring 30–59
Outcome: Full regression suite, 80% of manual effort eliminated

Phase 4: Optimisation (Ongoing)

Goal: Reduce maintenance cost, improve test speed
Scope: Refactor flaky tests, eliminate duplicates, push tests down the pyramid
Outcome: Suite runs in < 15 min, flake rate < 2%

Conclusion

Automation without prioritisation is waste. The ROI framework gives you a systematic way to answer “what should I automate first?”

The formula is simple: Risk × Frequency × Maintenance Cost.

High scores (80+) → Automate immediately
Medium scores (30–79) → Automate after proving value with high-score tests
Low scores (< 30) → Keep manual or delete

Not every test deserves automation. Treat your automation budget as finite and invest it where ROI is highest.

In the next post, we’ll tackle test data strategies — how to create, isolate, and clean up test data so your automated tests don’t fail due to data pollution, race conditions, or environment drift.

Action for this week: Take 10 test scenarios from your manual regression suite. Score each using Risk (1–5), Frequency (1–5), Maintenance (1–5). Calculate Priority Scores. Sort by score. The top 3 are your automation candidates. Estimate development time for the highest-scoring test. If dev time is < 2 days, add it to this week’s sprint.