Flaky-Test Strategy in CI (Retries, Quarantine, Tracking)

📌 Definition:

📖 Detailed Explanation:

Flakiness is corrosive: it erodes trust until developers reflexively re-run red builds, at which point the suite stops catching real regressions. A mature strategy has four parts. Detect: mark tests that pass only on retry as 'flaky' and track the rate as a first-class metric. Contain: quarantine chronically flaky tests into a non-blocking lane so they don't gate merges, while explicitly tracked for repair — not deleted and forgotten. Diagnose: capture traces on first retry so each flake ships evidence. Eliminate: fix root causes (bad waits, shared state, timing/animation races, order dependence) rather than papering over with ever-higher retries. The key judgment is that retries are a diagnostic safety net and a temporary bridge, never a permanent cure — a rising flaky count is a signal to invest, not to raise the retry ceiling.

🔑 Key Points:

• Flaky = non-deterministic result on unchanged code; it destroys suite trust
• Detect: mark pass-on-retry as 'flaky' and track the rate as a metric
• Quarantine chronic flakes into a non-blocking lane, tracked for repair (not deleted)
• Diagnose with trace-on-first-retry; fix root causes (waits, shared state, order)
• Retries are a safety net/bridge, not a cure — rising flakiness means invest, not raise retries

🌍 Real-World Example:

📎 Code Example:

# playwright.config.ts — modest retries + trace on first retry
#   retries: process.env.CI ? 2 : 0,
#   use: { trace: 'on-first-retry' }

# Quarantine lane: run known-flaky tests separately, non-blocking
jobs:
  stable:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npx playwright test --grep-invert @flaky   # blocks merge
  quarantine:
    runs-on: ubuntu-latest
    continue-on-error: true                                       # does NOT block merge
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npx playwright test --grep @flaky