Risk-Based Regression — What to Run When You Cannot Run Everything
Choose which regression tests run on a pull request, which wait for nightly, and which should leave the suite.
This post is part of the CI/CD Quality series. If you missed the previous post, read Part 6 — Continuous Testing in Continuous Delivery first.
Introduction — The Suite Grew Faster Than the Pipeline
The regression suite used to finish before lunch. Then it grew. Now a pull request waits forty minutes, people restart red jobs, and someone suggests skipping tests “just this once.”
That suggestion is the real decision. You are already choosing what not to run. The only question is whether the choice is written down.
This post is not about what to automate in the first place. It is about a suite that already exists and no longer fits in the time you have.
Score the Test, Not the Feature Story
Give each regression test four numbers, each from 0 to 3:
| Signal | 0 | 3 |
|---|---|---|
| Failure cost | A typo in a footer | Money moves, data leaks, or users cannot sign in |
| Change proximity | The area has not changed in months | The diff touches this code or its contract |
| Recent signal | It has not failed, and has not found a bug, in 90 days | It failed for a real defect in the last month |
| Runtime | Under 10 seconds | Over 2 minutes, or it needs a shared environment |
Add them. A test with a high score earns a pull request. A test with a low score waits for nightly, or leaves the suite.
Proximity changes every pull request. The other three numbers change weekly. Do not pretend the score is permanent.
Three Buckets Are Enough
- Pull request. The journey breaks the business if it breaks, and the test is stable enough to trust. Cap this bucket by time, not by count. Twenty minutes is a reasonable ceiling for most teams.
- Nightly. Real coverage that is too slow, too broad, or too dependent on data you will not refresh on every commit.
- Quarantine. Flaky, or red for a reason nobody owns. A quarantined test does not gate a merge. It has an owner and a date. After two weeks with no owner, delete it.
A test that is always nightly is not a safety net. It is a report nobody reads. If nightly fails and the next action is “rerun,” the bucket is wrong.
Proximity Without a Crystal Ball
You do not need a perfect impact graph. Start with tags that match the areas developers already name:
# .github/workflows/pr.yml
- name: Regression for this change
run: npx playwright test --grep @checkout
if: contains(github.event.pull_request.labels.*.name, 'area:checkout')Until labels exist, run the pull-request bucket whole, and only the pull-request bucket. Adding Firefox, a full data reset, and every edge case to that bucket is how the forty minutes come back.
What “Delete” Means
A test that has not failed, and has not been the reason a bug was caught, for a quarter is a candidate. Read it before you delete it. Sometimes it is the only check of a regulation, a payment rule, or a permission. Those stay, even when they are slow. They move to nightly, they do not disappear.
Deleting a duplicate is safer than deleting a unique assertion. Two end-to-end tests that click the same button and expect the same toast are one test.
Conclusion
A regression suite is a budget. Spend the pull-request minutes on failure cost and on the code that actually changed. Spend the night on the rest. Spend nothing on tests nobody will fix.
Action for this week: Time the pull-request suite. Move every test over two minutes that is not a release blocker to nightly. Quarantine anything that failed without a defect in the last two weeks.
Next in the series: Part 8 — What QA Owns After Deploy.