Reports, Artifacts, and Failure Triage in CI

Failed CI runs are only useful if you can diagnose them quickly. Learn how to generate HTML reports, upload trace artifacts, establish ownership loops, and notify teams without creating alert fatigue.

#ci-cd #testing #debugging #playwright #devops

This post is part 4 of the CI/CD Quality series. Part 3 covered Docker and ephemeral environments for reproducible E2E testing.

Introduction — Failures Are Data

A failed CI run without context is noise. “E2E tests failed” tells you nothing. Was it a real bug? A flaky test? An infrastructure timeout?

The value of automated testing is proportional to how quickly you can diagnose and fix failures. This requires:

  • Rich reports — HTML reports with screenshots, traces, and failure summaries
  • Artifacts — Downloadable test results, logs, and debugging data
  • Ownership loops — Clear responsibility for investigating and resolving failures
  • Targeted notifications — Alerts that reach the right people without spamming everyone

This post covers how to generate and upload test reports, establish failure triage workflows, and integrate notifications into Slack and GitHub without creating alert fatigue.

HTML Reports for Test Results

Playwright generates an HTML report showing test results, execution times, screenshots, and videos for each test.

Generating HTML Reports

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  reporter: [
    ['html', { outputFolder: 'playwright-report', open: 'never' }],
    ['list'],
  ],
  use: {
    screenshot: 'only-on-failure',
    video: 'retain-on-failure',
    trace: 'retain-on-failure',
  },
});

Configuration breakdown:

  • html reporter generates an HTML report in playwright-report/
  • list reporter prints test results to console
  • screenshot: 'only-on-failure' captures screenshots only when tests fail
  • video: 'retain-on-failure' saves video recordings only for failed tests
  • trace: 'retain-on-failure' saves trace files only for failures

Uploading Reports as Artifacts

GitHub Actions allows you to upload files as artifacts. Upload the HTML report so it’s accessible from the workflow run.

jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: 'npm'
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - run: npm run test:e2e
      
      - name: Upload Playwright report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: playwright-report
          path: playwright-report/
          retention-days: 7

Key features:

  • if: always() uploads the report even if tests fail
  • retention-days: 7 keeps the artifact for 7 days (GitHub default is 90 days; reduce to save storage)

Accessing the report:

  1. Navigate to the workflow run in GitHub UI
  2. Scroll to Artifacts section
  3. Download playwright-report.zip
  4. Extract and open index.html in a browser

:::tip[Viewing Traces] Traces provide the most detail: full timeline, network requests, DOM snapshots, console logs, and screenshots for every action. Download traces from the report and open with npx playwright show-trace trace.zip. :::

Uploading Logs and Test Results

Beyond HTML reports, upload raw test results and logs for programmatic analysis.

JUnit XML Reports

JUnit XML is a standard format for test results. Many CI dashboards parse it.

// playwright.config.ts
export default defineConfig({
  reporter: [
    ['html', { outputFolder: 'playwright-report' }],
    ['junit', { outputFile: 'test-results/junit.xml' }],
    ['list'],
  ],
});

Upload the JUnit XML file:

      - name: Upload JUnit results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: junit-results
          path: test-results/junit.xml

Uploading Application Logs

If your app logs to a file during tests, upload those logs for debugging.

      - name: Upload app logs
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: app-logs
          path: logs/

Establishing Ownership Loops

A failed test with no owner is a failed test that stays broken.

The Ownership Loop

  1. Test fails in CI
  2. Alert sent to owner (Slack, email, GitHub comment)
  3. Owner investigates (downloads report, views trace)
  4. Owner fixes or disables (merge fix or quarantine flaky test)
  5. Loop closes when CI is green

Assigning Ownership

Define ownership at the test suite level or file level.

Example: Ownership by directory

tests/
  auth/          # Owner: @auth-team
  checkout/      # Owner: @payments-team
  admin/         # Owner: @platform-team

Use GitHub CODEOWNERS to enforce review:

# .github/CODEOWNERS
tests/auth/*       @auth-team
tests/checkout/*   @payments-team
tests/admin/*      @platform-team

When a test fails, the workflow can tag the owning team in a GitHub comment.

Slack Notifications Without Noise

Slack notifications are powerful but dangerous. Unfiltered CI alerts create noise that trains teams to ignore them.

Good Notification Strategy

✅ Notify on failure — Alert when tests fail on main branch (post-merge)
✅ Notify specific channels — Route test failures to team-specific channels
✅ Include context — Link to workflow run, failed test names, and report artifacts
❌ Do not notify on PR failures — Developers already see PR checks; Slack notifications are redundant
❌ Do not notify on flaky test re-runs — Only alert after retries are exhausted

Slack Notification Workflow

Use slackapi/slack-github-action to post to Slack.

name: E2E Tests

on:
  push:
    branches: [main]

jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: 'npm'
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - run: npm run test:e2e
      
      - name: Upload report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: playwright-report
          path: playwright-report/
      
      - name: Notify Slack on failure
        if: failure()
        uses: slackapi/slack-github-action@v1
        with:
          webhook-url: ${{ secrets.SLACK_WEBHOOK_URL }}
          payload: |
            {
              "text": "E2E tests failed on main branch",
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "🔴 *E2E Tests Failed*\n<${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View workflow run>"
                  }
                },
                {
                  "type": "section",
                  "fields": [
                    {
                      "type": "mrkdwn",
                      "text": "*Commit:* <${{ github.event.head_commit.url }}|${{ github.event.head_commit.message }}>"
                    },
                    {
                      "type": "mrkdwn",
                      "text": "*Author:* ${{ github.event.head_commit.author.name }}"
                    }
                  ]
                }
              ]
            }

Result: Slack message with workflow link, commit message, and author.

:::warning[Alert Fatigue] If your main branch CI fails >5% of the time, fix the flaky tests before adding Slack notifications. Frequent false alarms destroy trust in alerts. :::

GitHub PR Comments for Test Failures

Post a comment on the pull request when tests fail, summarising which tests failed and linking to the report.

Using actions/github-script

      - name: Comment on PR with test results
        if: failure() && github.event_name == 'pull_request'
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const reportPath = 'test-results/junit.xml';
            let failedTests = 'Could not parse test results';
            
            if (fs.existsSync(reportPath)) {
              const xml = fs.readFileSync(reportPath, 'utf-8');
              // Parse JUnit XML and extract failed test names
              failedTests = xml.match(/<testcase.*?name="(.*?)".*?<failure/g)
                ?.map(m => m.match(/name="(.*?)"/)[1])
                .join('\n- ') || 'No failures found';
            }
            
            github.rest.issues.createComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.issue.number,
              body: `## ❌ E2E Tests Failed\n\n**Failed tests:**\n- ${failedTests}\n\n[View full report](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})`
            });

Result: GitHub comment listing failed tests and linking to the workflow run.

:::info[Rate Limits] GitHub API has rate limits. If your workflow posts many comments, you may hit the limit. Consolidate updates into a single comment that gets edited on subsequent runs. :::

Practical Example — Full Failure Triage Workflow

Combine HTML reports, artifacts, Slack notifications, and PR comments in a production-ready workflow.

name: E2E Tests

on:
  pull_request:
  push:
    branches: [main]

jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: 'npm'
      
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      
      - name: Run E2E tests
        run: npm run test:e2e
      
      - name: Upload HTML report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: playwright-report
          path: playwright-report/
          retention-days: 7
      
      - name: Upload JUnit results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: junit-results
          path: test-results/junit.xml
      
      - name: Notify Slack on main failure
        if: failure() && github.ref == 'refs/heads/main'
        uses: slackapi/slack-github-action@v1
        with:
          webhook-url: ${{ secrets.SLACK_WEBHOOK_URL }}
          payload: |
            {
              "text": "🔴 E2E tests failed on main",
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "*E2E Tests Failed on Main*\n<${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View run>"
                  }
                }
              ]
            }
      
      - name: Comment on PR
        if: failure() && github.event_name == 'pull_request'
        uses: actions/github-script@v7
        with:
          script: |
            github.rest.issues.createComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.issue.number,
              body: `## ❌ E2E Tests Failed\n\n[View workflow run](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})\n\nDownload the Playwright report artifact for details.`
            });

Features:

  • HTML report and JUnit XML uploaded for every run
  • Slack notification only on main branch failures
  • GitHub PR comment only on PR failures
  • All notifications include links to workflow run

Conclusion — Make Failures Actionable

CI failures are only valuable if they’re actionable. A test suite that fails with no context, no owner, and no notification loop is indistinguishable from a test suite that doesn’t exist.

The principles:

  • Generate rich reports — HTML, screenshots, videos, traces
  • Upload artifacts — Make debugging data accessible
  • Establish ownership — Every test suite has a named owner
  • Notify strategically — Alert the right people at the right time without spamming
  • Close the loop — Failures are investigated and resolved, not ignored

When failures become actionable, CI becomes trusted. When CI is trusted, teams ship faster.

Action for this week: Configure HTML report uploads for your test suite. Trigger one intentional test failure and download the report. Measure how long it takes to identify the root cause from the report alone.

Next in this series: Part 5 — PR Checks That Matter — Shift-Left Without Noise