You break the thing the test protects, on purpose, and watch the test fail. If it stays green, it was never testing anything. That is the whole method, and it has a name: mutation testing. I apply it to every check in my projects, not only the unit tests but the linters, the type checker, the secret scanner and the security tests. In the repository behind this site, a script makes more than forty deliberate breakages across every check before anything is pushed, and each one has to turn its check red.
Why isn’t a passing test enough?
Because a passing test looks exactly the same whether it works or not. Green is what you get from a test that asserts nothing, from a check pointed at the wrong folder, and from a scanner that crashed before it scanned. A check that cannot fail is worse than no check at all. It occupies the slot a real one would go in, and everyone stops looking.
I learned this from a lint step in two apps that had no linter configuration and no linter installed. It reported nothing, and it looked like a pass for weeks.
Database constraints are the same. A unique index over columns that can be empty looks as though it forbids duplicates, and in Postgres it forbids nothing while those columns are null. Reading the schema tells you what you wrote. Inserting the row that ought to fail tells you what the database enforces.
Green is what a broken test looks like. The only way to tell it from a working one is to make it fail.
What does breaking it on purpose look like?
For each check, a script makes one small, realistic defect, runs the check, requires red, and puts the file back. A few of mine:
- A loose any type slipped into application code, which the linter must refuse.
- A test with no assertion in it, which a lint rule must catch, so the tests are tested too.
- A new route that nobody added to the list of public ones, which the security tests must notice.
- A dev server told to listen on every network interface instead of only this machine.
- A fake secret committed with the hooks switched off and deleted in the next commit, which the history scan must still find.
- A database column the code writes to that no migration ever created.
Each one names the kind of defect it stands for, in words, beside the break itself. That matters more than it sounds.
Can a break-test be wrong too?
Yes, and that is the subtle part. A break-test written at the same moment as its check tends to share the check’s blind spots. On one project my secret-scanner test used a sample key that the scanner ignores on purpose, because it is the example key in a cloud provider’s public documentation. So a broken scanner passed its own proof. Writing down the class of defect each break stands for puts that assumption where someone can argue with it.
It can go wrong in other ways, each of which I have paid for:
- The anchor moves. A break that finds its place by quoting a line stops applying when that line is rewritten. My prover counts a break it couldn’t apply as a failure, because it proved nothing while the summary would have said it passed.
- A later stage hides the gap. Removing the first smoke test of a staged release didn’t turn the test red, because a later stage ran the same smoke test and stopped the release there. The test now asks which stage stopped it.
- The break escapes. A run interrupted halfway left a mutated file behind: a sign-in gate that let every session through, in the very checkout the dev servers ran from. Breaks now happen in a throwaway clone, never in the working copy.
- The check can’t see its subject. A test that loops over every route passes happily when it finds no routes. Mine assert that they found something before they assert anything about it.
What about tests that fail at random?
Those lie in the other direction, and they teach people to ignore red. The worst I have met counted rows in a database it shared with other test files. Whenever another file happened to run first, its rows were counted as well, and under load the suite failed about half the time. The rule I took from it: a test counts only what it wrote, names its own unique values, and waits for a state it can name, never for a number of seconds.
Run tests in the conditions production runs in, too. On one project, dates were formatted in the machine’s time zone, which was London, while production ran in UTC. Every test passed, because the tests ran in London as well, and every time shown on screen was an hour out all summer.
Isn’t this a lot of effort?
Less than you would think, and less than the alternative. Each break is a few lines. The real cost is running the suite many times over, which is why mine runs at the push rather than on every save. On my largest project, proving everything on every push grew to seven hours, so a push now proves only what its change could reach, and the full run happens overnight.
What it buys is the right to believe your own green ticks. Building with AI makes that worth more. AI writes very convincing tests, and convincing isn’t the same as correct.
Where should you start?
Pick the check you rely on most and break the thing it guards. Delete the line a unit test is about. Add a type error. Commit a fake key to a scratch branch. If the check stays green, you have found something more important than whatever you planned to do today. Then write that break down as a script, so it happens again on every push without anyone remembering to do it, and move on to the next check.
If you want mutations generated for you as well, there are good tools for most languages: Stryker for JavaScript and TypeScript, PIT for Java, mutmut for Python. They change your code in hundreds of small ways and report which changes your tests failed to notice. Hand-written breaks are still worth having beside them, because they cover the checks those tools never touch.