Quality Gates in CI Pipelines

This post is part 1 of the CI/CD Quality series. We’ll explore practical strategies for building quality into continuous integration and delivery pipelines.

Introduction — The Merge Button Is a Decision Point

Every pull request has a merge button. That button represents a decision: is this code ready for production?

In teams without automation, that decision relies on human judgment, code review, and hope. In teams with CI/CD, that decision is informed by automated quality gates.

A quality gate is a checkpoint in your CI pipeline that evaluates whether code meets specific quality criteria. It can pass or fail. When it fails, the pipeline stops, and the merge is blocked.

The critical question is: which checks should block the merge, and which should only inform?

Block too little, and broken code reaches production. Block too much, and your team works around the pipeline to ship faster.

:::warning[Blocking Everything Is Not Quality] If your CI pipeline fails 40% of the time due to flaky tests, developers will stop trusting it. They’ll click “merge anyway” or disable the check. A gate that fails unreliably is worse than no gate at all. :::

This post defines what quality gates are, which ones should block merges, and how to design gates that protect production without creating false bottlenecks.

What Must Block a Merge

Not every quality check deserves to block a merge. The bar for blocking is high: this code will cause production incidents or violate non-negotiable constraints.

Critical Quality Gates

1. Build must succeed

If the code doesn’t compile or bundle, it cannot run. This is non-negotiable.

# .github/workflows/ci.yml
name: CI

on: [pull_request]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npm run build

Why block: If the build fails, deployment is impossible. Blocking here prevents wasting time on a PR that cannot ship.

2. Linting must pass

Linters catch syntax errors, unused variables, import violations, and style inconsistencies. These are cheap to fix and prevent entire categories of runtime errors.

  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npm run lint

Why block: A linting failure often indicates incomplete refactoring or broken imports. If npm run lint fails locally, the code is not ready.

3. Unit tests must pass

Unit tests validate isolated business logic. If a unit test fails, a developer explicitly broke existing functionality or introduced a regression.

  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npm test

Why block: Unit tests fail deterministically when logic is wrong. A failing unit test is a clear signal: this code is not ready.

4. Smoke tests must pass

Smoke tests are a minimal subset of end-to-end tests that verify the application starts, critical pages render, and authentication works. They run in under 5 minutes.

  smoke-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npm run build
      - run: npx playwright install --with-deps chromium
      - run: npm run test:e2e:smoke

Why block: If the homepage doesn’t load or login is broken, the application is not deployable. Smoke tests catch catastrophic failures before they reach production.

5. Security scans must not find critical vulnerabilities

Dependency scanning tools like npm audit, Snyk, or Dependabot flag known CVEs in your dependencies.

  security-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npm audit --audit-level=high

Why block: A critical vulnerability is a production incident waiting to happen. Block the merge and fix or suppress it explicitly.

:::tip[Audit Levels] Use --audit-level=high to block only high and critical vulnerabilities. Blocking on moderate or low findings creates noise and slows teams down without proportional security benefit. :::

What Should Inform, Not Block

Some checks are valuable but should not block merges. They provide information for decision-making but do not represent a hard constraint.

1. Test coverage percentage

Coverage reports show which lines of code are tested. This is useful context, but a coverage drop does not mean the code is broken.

  coverage:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npm test -- --coverage
      - uses: codecov/codecov-action@v4
        with:
          token: ${{ secrets.CODECOV_TOKEN }}

Why inform only: Coverage is a proxy metric. High coverage does not guarantee quality; low coverage does not guarantee defects. Use it as a signal, not a gate.

2. Full E2E test suites

Full end-to-end test suites take 15–30 minutes to run. They test realistic user journeys across the full application stack.

  e2e-full:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npm run build
      - run: npx playwright install --with-deps
      - run: npm run test:e2e

Why inform only: E2E tests are valuable but slow and occasionally flaky. Block on smoke tests; inform with full E2E. Ship fast, catch edge cases in staging.

3. Performance benchmarks

Performance tests flag regressions in response time, memory usage, or bundle size.

  performance:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npm run build
      - run: npm run perf:benchmark

Why inform only: Performance regressions are often acceptable trade-offs for feature delivery. Flag them, but let the team decide whether to address before or after merge.

The Flaky Gate Anti-Pattern

A flaky gate is a quality check that fails intermittently without code changes. Flaky tests are the single most destructive force in CI/CD pipelines.

Why Flaky Tests Break Teams

Developers stop trusting the pipeline
Teams click “merge anyway” or disable checks
Real failures are ignored because “it’s probably flaky”
Time is wasted re-running builds

:::danger[Flaky Gates Destroy Trust] If your CI pipeline has a 10% flake rate, and developers run it 10 times per day, they see false failures twice daily. After a week, they’ve stopped paying attention. Your gate is now decoration. :::

How to Handle Flaky Tests

1. Do not block on flaky tests

If a test suite has >2% flake rate, do not make it a required check. Move it to an informational check or disable it until it’s fixed.

  e2e-flaky:
    runs-on: ubuntu-latest
    continue-on-error: true  # Does not block merge
    steps:
      - run: npm run test:e2e

2. Quarantine flaky tests

Tag flaky tests with @flaky and skip them in CI. Track them in a backlog. Fix or delete them.

test.skip('checkout flow completes', async ({ page }) => {
  // Flaky due to third-party payment iframe timing
  // TODO: Fix or replace with API test
});

3. Fix the root cause

Most flaky E2E tests fail due to:

Race conditions (missing await, improper waits)
Hardcoded timeouts (page.waitForTimeout(5000))
Shared test data (parallel tests mutating the same DB records)

Playwright’s auto-waiting solves most of these. Use it.

:::tip[Flake-Free E2E Tests] Use Playwright’s built-in locators and assertions. They auto-wait and auto-retry. Avoid page.waitForTimeout() and manual sleep() calls. Read Part 2 of the Playwright series for details. :::

Practical Example — GitHub Actions Required Checks

GitHub allows you to mark specific jobs as required before a PR can be merged.

Repository Settings → Branches → Branch protection rules → Require status checks to pass before merging

Mark these as required:

✅ build
✅ lint
✅ unit-tests
✅ smoke-tests
✅ security-scan

Leave these as optional:

ℹ️ e2e-full
ℹ️ coverage
ℹ️ performance

This configuration protects production without creating false bottlenecks.

Conclusion — Gates That Serve the Team

Quality gates are not about control — they’re about confidence. A well-designed gate tells you: this code will not break production in obvious ways.

The principles:

Block on failures that make deployment impossible (build, lint, unit tests, smoke tests, critical security issues)
Inform on failures that provide useful context (coverage, full E2E, performance)
Never block on flaky tests — they destroy trust faster than they catch bugs

Quality gates should accelerate delivery by catching critical issues early. If your gates slow the team down or get bypassed regularly, the gates are misconfigured.

Action for this week: Review your CI pipeline. Identify one check that blocks merges but fails >5% of the time. Either fix it or move it to informational-only. Trust is built by reliability.

Next in this series: Part 2 — GitHub Actions for Test Automation