Testing

Coverage measures which lines executed while the tests ran. It says nothing about whether anything was asserted, or whether the assertions mean anything. A test that calls a function and checks nothing gives that function full coverage and catches no regression whatever, which is the whole problem in one sentence.

That makes a coverage target a textbook case of a measure becoming a goal. Mandate eighty per cent and you will get eighty per cent, assembled from tests written to touch lines rather than to verify behaviour, and the number will be defended in review by people who know it means nothing. The metric was fine until it was made a target.

It is genuinely useful in the other direction, as a discovery tool. Sorted by least covered, it shows which modules nobody has tested, and diff coverage on a pull request answers a narrower and more sensible question: is the code being added tested at all. That is a prompt for a conversation rather than a gate.

The variants are worth knowing because they differ in strength. Line coverage is the weakest. Branch coverage checks that both sides of a condition ran, which catches a class of gap line coverage reports as complete. Mutation testing is the strongest by a distance: it changes the code deliberately and checks that a test fails, which measures assertions rather than execution and is the only common technique that can tell you a test suite is decorative.

The honest measures of a suite are elsewhere. Does it catch the regressions you actually have, does it fail for one reason rather than five, does it run fast enough that people wait for it, and does a failure name the problem. None of those has a percentage, which is exactly why coverage gets used instead.

A flaky test passes and fails without the code changing. The causes are a short list and worth knowing by name: timing, where a fixed sleep stands in for waiting on a condition; shared state, where tests leak into each other through a database or a global; order dependence, where the suite passes in one sequence and not another; real network calls; and time itself, which is why tests written on a Tuesday afternoon fail at midnight or during a leap day.

The damage is cultural rather than technical. Once a suite is known to fail randomly, the correct response to any red build becomes pressing retry, and that response is applied to real failures too. Trust does not degrade gracefully: a suite that is ninety per cent reliable is treated exactly like one that is unreliable, because a human cannot tell which failure they are looking at.

So the response has to be immediate and structural. Quarantine the test out of the blocking suite the moment it flakes, so the signal for everyone else stays trustworthy, then fix it or delete it with an owner and a deadline. A quarantine with no expiry is just a slower way of deleting it, and being honest about that is better than pretending otherwise.

Most fixes come from removing the source of nondeterminism rather than from making the test more patient. Wait for a condition instead of sleeping. Inject the clock so time is a value rather than an ambient fact. Give every test its own data, ideally its own schema or transaction, so nothing leaks. Run the suite in a random order deliberately, so order dependence is found by you rather than by a scheduler change six months from now.

Measure it. Flake rate per test and per suite, tracked over time, turns a vague sense that CI is annoying into a number with a trend, and it identifies the small number of tests responsible for most of the noise. In most suites, a handful of tests cause the overwhelming majority of the retries.

The uncomfortable case is a test that is flaky because the system is. A race condition in the code will present as an intermittent test, and quarantining it hides a real defect. The distinction is worth making before reaching for the quarantine label: is the test unreliable, or is it reporting accurately on something that is.