QA metrics are measurements used to understand and improve quality and testing effectiveness. Useful metrics drive better decisions and behavior; vanity metrics look impressive but reveal little or actively encourage the wrong behavior. Knowing the difference is a mark of a mature tester.
The danger with metrics is Goodhart's Law: when a measure becomes a target, it stops being a good measure because people game it. Classic vanity/harmful metrics include raw test-case count (encourages many trivial tests), number of bugs found (can reward writing low-value bugs, or punish QA when devs improve), and code-coverage percentage treated as a quality guarantee (lines executed ≠ behavior verified). More useful, behavior-driving metrics include: escaped defects (bugs found in production — the real measure of test effectiveness), defect escape rate or defect removal efficiency (share of defects caught before release), lead time / feedback time (how fast the pipeline gives signal), flaky-test rate (suite health), and mean time to detect/resolve. Even good metrics need context — none should be used to rank or punish individuals, which corrupts them. The mature stance QA should articulate is: use a small set of trend-based, outcome-focused metrics to inform improvement conversations (especially in retros), never as targets to hit or weapons to assign blame.
A team is rewarded on 'number of test cases written' and soon has thousands of trivial, overlapping tests that are slow to run and maintain but catch little. Switching the focus to escaped-defect rate and feedback time refocuses everyone on tests that actually prevent production bugs, and the bloated case count stops being celebrated.
| Metric | Type | What It Really Tells You |
|---|---|---|
| Test-case count | Vanity | Volume, not value |
| Bugs found by QA | Often misleading | Depends on dev quality & bug value |
| Code coverage % | Misleading if a target | Lines run, not behavior verified |
| Escaped defects | Useful (outcome) | True test effectiveness |
| Defect escape rate | Useful (outcome) | Share of defects caught pre-release |
| Feedback/lead time | Useful (flow) | How fast quality feedback arrives |
| Flaky-test rate | Useful (health) | How trustworthy the suite is |
Scenario: A manager wants to measure QA performance and proposes tracking 'number of test cases written per tester' and 'number of bugs found per tester', with a leaderboard. Why is this risky, and what metrics would you propose instead?