Continuous Testing in Continuous Delivery

This post is part 6 of the CI/CD Quality series. Part 5 covered PR checks — shift-left without noise.

---

Introduction — Continuous Delivery Is About Confidence

Continuous Delivery (CD) means your main branch is always deployable to production. Every commit that passes automated checks can be released.

This sounds simple. It’s not.

CD requires confidence that your changes won’t break production. That confidence comes from testing — but not the traditional “QA gates at the end” testing. It comes from continuous testing throughout the delivery pipeline.

This post covers where QA fits in CD, how to build release confidence, and how progressive delivery strategies (canary releases, feature flags, A/B tests) enable fast, safe deployments.

---

Where QA Fits in Continuous Delivery

In traditional waterfall or stage-gate processes, QA is a phase. Code is written, then QA tests it, then it’s released.

In CD, QA is a continuous activity embedded in the pipeline, not a phase.

Testing Stages in a CD Pipeline

Developer → PR Checks → Merge → Post-Merge Tests → Staging → Production → Synthetic Monitoring

Stage	Tests Run	Purpose
PR Checks	Lint, unit tests, smoke tests	Catch obvious defects pre-merge
Post-Merge	Full E2E suite, integration tests	Validate end-to-end flows
Staging	Smoke tests, exploratory testing	Validate in production-like environment
Production	Smoke tests, canary validation	Detect issues before full rollout
Synthetic Monitoring	Scheduled E2E tests	Proactively detect production regressions

Key principle: Testing is layered. Fast, narrow tests run early. Slow, comprehensive tests run post-merge. Production testing validates real-world behaviour.

:::tip[The Testing Pyramid in CD] Unit tests run in seconds and catch most defects. E2E tests run in minutes and catch integration issues. Production monitoring catches edge cases that only appear at scale. Invest most in the base of the pyramid. :::

---

Building Release Confidence

Release confidence is the belief that a deployment will succeed without causing incidents.

Confidence Through Coverage

Code coverage is a proxy metric. High coverage doesn’t guarantee quality, but low coverage guarantees gaps.

Target:

• Unit test coverage: 70–80% for business logic
• E2E coverage: Critical user journeys (login, checkout, core workflows)
• API test coverage: All public endpoints, authentication, error handling

Don’t chase 100% coverage. Diminishing returns set in around 80%. The last 20% often tests trivial code (getters, setters, constructors) with low defect risk.

Confidence Through Observability

Automated tests tell you what you programmed them to check. Observability tells you what’s actually happening in production.

Observability stack:

• Logs — Structured logs with correlation IDs for tracing requests
• Metrics — Request rates, error rates, latency percentiles (P50, P95, P99)
• Traces — Distributed tracing across services (Jaeger, Datadog APM)
• Alerts — Threshold-based alerts on error spikes, latency regressions

Example: Deploy a new feature. Monitor the error rate on the affected endpoint. If it spikes, roll back immediately.

:::warning[Testing Is Not a Substitute for Observability] Tests validate expected behaviour. Observability detects unexpected behaviour. You need both. :::

---

Progressive Delivery — Releasing Features Gradually

Progressive delivery is the practice of releasing features incrementally to subsets of users, validating behaviour, and rolling back if issues arise.

Canary Releases

A canary release deploys the new version to a small percentage of traffic (5–10%) while the old version serves the rest.

Workflow:

1. Deploy new version to canary environment
2. Route 5% of traffic to canary
3. Monitor error rates, latency, and key metrics for 15–30 minutes
4. If metrics are stable, increase to 25%, then 50%, then 100%
5. If metrics degrade, roll back to 0% and investigate

Implementation with Kubernetes:

# Canary deployment with traffic split
apiVersion: v1
kind: Service
metadata:
  name: app-service
spec:
  selector:
    app: myapp
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-stable
spec:
  replicas: 9
  selector:
    matchLabels:
      app: myapp
      version: stable
  template:
    metadata:
      labels:
        app: myapp
        version: stable
    spec:
      containers:
      - name: app
        image: myapp:v1.0
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-canary
spec:
  replicas: 1  # 10% of traffic
  selector:
    matchLabels:
      app: myapp
      version: canary
  template:
    metadata:
      labels:
        app: myapp
        version: canary
    spec:
      containers:
      - name: app
        image: myapp:v1.1

Result: 10% of users see the new version. Monitor for 30 minutes. If stable, scale canary to 5 replicas (50%), then 10 replicas (100%).

---

Feature Flags — Decoupling Deployment from Release

A feature flag (also called feature toggle) is a runtime switch that enables or disables features without deploying new code.

Why Feature Flags Matter for CD

• Deploy dark — Merge code to production with the feature disabled. Enable it later.
• Gradual rollout — Enable the feature for 10% of users, then 50%, then 100%.
• Emergency kill switch — Disable a problematic feature instantly without rolling back the deployment.
• A/B testing — Serve variant A to 50% of users, variant B to the other 50%, measure conversion.

Implementing Feature Flags

Simple in-memory flag:

// config/features.ts
export const features = {
  newCheckoutFlow: process.env.FEATURE_NEW_CHECKOUT === 'true',
  aiRecommendations: process.env.FEATURE_AI_RECS === 'true',
};

Usage:

import { features } from './config/features';

function renderCheckout() {
  if (features.newCheckoutFlow) {
    return <NewCheckout />;
  }
  return <LegacyCheckout />;
}

Production-grade flag service (LaunchDarkly, Split.io, Unleash):

import { LaunchDarkly } from 'launchdarkly-node-server-sdk';

const client = LaunchDarkly.init(process.env.LAUNCHDARKLY_SDK_KEY);

async function shouldShowNewCheckout(user: User): Promise<boolean> {
  return await client.variation('new-checkout-flow', user, false);
}

Result: Feature flags are evaluated per-user. You can target specific users, roll out gradually, or disable instantly.

:::tip[Feature Flag Hygiene] Remove feature flags once the feature is stable and rolled out to 100%. Dead flags accumulate technical debt. Schedule flag cleanup as part of your release process. :::

---

Smoke Tests in Production

Smoke tests aren’t just for pre-production environments. Run them in production to detect issues immediately after deployment.

Post-Deployment Smoke Tests

name: Production Smoke Tests

on:
  workflow_dispatch:

jobs:
  smoke-prod:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: 'npm'
      
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      
      - name: Run smoke tests
        env:
          BASE_URL: https://app.production.com
          AUTH_TOKEN: ${{ secrets.PROD_AUTH_TOKEN }}
        run: npm run test:e2e:smoke
      
      - name: Notify on failure
        if: failure()
        uses: slackapi/slack-github-action@v1
        with:
          webhook-url: ${{ secrets.SLACK_WEBHOOK_URL }}
          payload: |
            {
              "text": "🚨 Production smoke tests failed after deployment"
            }

Trigger this workflow:

• Manually after each production deployment
• Automatically via deployment webhooks
• On a schedule (every 15 minutes) for continuous validation

---

Synthetic Monitoring — Proactive Production Testing

Synthetic monitoring runs automated user journeys against production on a schedule. It detects failures before real users encounter them.

Scheduled E2E Tests in Production

name: Synthetic Monitoring

on:
  schedule:
    - cron: '*/15 * * * *'  # Every 15 minutes

jobs:
  synthetic:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: 'npm'
      
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      
      - name: Run synthetic tests
        env:
          BASE_URL: https://app.production.com
          AUTH_TOKEN: ${{ secrets.PROD_AUTH_TOKEN }}
        run: npm run test:synthetic
      
      - name: Alert on failure
        if: failure()
        uses: slackapi/slack-github-action@v1
        with:
          webhook-url: ${{ secrets.SLACK_WEBHOOK_ONCALL }}
          payload: |
            {
              "text": "🚨 Synthetic monitoring detected failure in production",
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "<${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View run>"
                  }
                }
              ]
            }

Synthetic tests cover:

• User login
• Product search
• Checkout flow
• Payment processing
• Admin workflows

Result: If a deployment breaks production, synthetic monitoring detects it within 15 minutes and alerts the on-call engineer.

:::info[Synthetic vs. Real User Monitoring] Synthetic monitoring is proactive (detects issues before users). Real user monitoring (RUM) is reactive (detects issues as users encounter them). Use both. :::

---

Practical Example — Full CD Pipeline with Testing

Here’s a production CD pipeline with testing at every stage.

name: CD Pipeline

on:
  push:
    branches: [main]

jobs:
  build-and-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: 'npm'
      - run: npm ci
      - run: npm run build
      - run: npm test
      - run: npx playwright install --with-deps chromium
      - run: npm run test:e2e
      
      - name: Upload build
        uses: actions/upload-artifact@v4
        with:
          name: build
          path: dist/

  deploy-staging:
    needs: build-and-test
    runs-on: ubuntu-latest
    steps:
      - name: Download build
        uses: actions/download-artifact@v4
        with:
          name: build
          path: dist/
      
      - name: Deploy to staging
        run: ./scripts/deploy-staging.sh
      
      - name: Run smoke tests on staging
        env:
          BASE_URL: https://staging.app.com
        run: npm run test:e2e:smoke

  deploy-production:
    needs: deploy-staging
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to production (canary)
        run: ./scripts/deploy-canary.sh
      
      - name: Wait for canary validation
        run: sleep 300  # 5 minutes
      
      - name: Check canary metrics
        run: ./scripts/check-canary-metrics.sh
      
      - name: Promote to 100%
        run: ./scripts/promote-canary.sh
      
      - name: Run smoke tests on production
        env:
          BASE_URL: https://app.com
        run: npm run test:e2e:smoke

Pipeline stages:

1. Build and run full test suite (unit + E2E)
2. Deploy to staging and run smoke tests
3. Deploy to production canary (5%)
4. Wait 5 minutes and validate metrics
5. Promote to 100%
6. Run smoke tests on production

Total time: 12–15 minutes from merge to full production rollout.

---

Conclusion — Continuous Testing Enables Continuous Delivery

Continuous delivery is not “release faster and hope for the best.” It’s “release faster with confidence built through continuous testing.”

The principles:

• Test at every stage — PR, post-merge, staging, production
• Layer your tests — Fast unit tests catch most defects; slow E2E tests catch integration issues
• Use progressive delivery — Canary releases, feature flags, gradual rollouts
• Monitor production — Synthetic monitoring and observability detect issues before users do
• Build confidence through data — Coverage, metrics, traces, alerts

When testing is continuous, delivery can be continuous. When delivery is continuous, teams ship faster and with higher quality.

Action for this week: Identify one feature you could deploy behind a feature flag. Merge it to production with the flag disabled. Enable it for 10% of users and monitor metrics for 24 hours before rolling out to 100%.

---

This concludes the CI/CD Quality series. Next series begins 2026-05-17: Architecture & Design Patterns for Testable Systems.