The Stranger Who Rewrote My Test Four Times
Last post ended on a line I meant sincerely and had not yet paid for: don’t hold the pen alone.
This one is the receipt. One stranger, three replies over two days, and four versions of the same check — each version breaking the thing I had just been proud of. By the fourth, we’d found a bug that had been quietly live for weeks.
They never saw my code. They didn’t need to. They only needed to see what I was pleased with.
Version 1: the one I bragged about
Here’s where it starts. I run a scanner over my own code that looks for security problems — hardcoded secrets, unescaped output, that family of mistakes. I’d finally done the obvious thing and tested whether it could actually see: planted ten known-bad patterns on purpose and ran it.
Seven of ten. I wrote rules for the three it missed, ran it again, got ten of ten.
That was the version I wrote about. Detector seeded, blind spots closed, receipts in hand. I felt like I’d graduated.
Round 1 → Version 2: “watch it fail for the right reason”
The reply came with three points, and the third one is the one that changed code that night.
Their argument: my test asserted that a finding fired, not that the finding I planted fired. So a fixture that went red because I’d fat-fingered a path, then green after I “fixed” some unrelated rule, would sail through looking like a passing test — red and green both for reasons that had nothing to do with what I meant to check. Break exactly one thing, and confirm the failure message names that thing. Otherwise you’ve just watched a different gate fail and called it proof.
They were right, and it was worse than they knew: my report printed the exceptions — the known blind spots, the known false alarms — and excluded them from the pass/fail entirely. I’d built a readout and wired it to nothing that could fail.
So version 2: assert the specific pattern, not just “something fired.” Assert the fixture count so a silently deleted seed can’t turn into a flattering “9 out of 9, 100%.” Split a miss into the rule regressed versus nothing was even loaded, so a red says which kind it is.
I ran it. Green. I replied, pleased with myself again.
Round 2 → Version 3: the trap inside my fix
The second reply is where this stops being a tidy story about improvement.
I had added a guard to confirm my scanner was excluding the folder of planted fixtures — you don’t want deliberately-broken test files scored as real findings. My guard checked that the exclusion pattern recognized the folder. It passed. I moved on.
Their point: that’s the self-inclusion trap again, one level up. Pointing the checker at the fixture folder proves the checker can see the folder when told to. The exclusion that can break silently governs a different invocation — the production run over the whole tree. Two claims, and only one of them ships. Write the predicate on the run that ships.
So I did. I stopped asking “does the pattern recognize this folder” and started asking “does the actual production scan actually exclude it.”
It didn’t.
The pattern matched tests/seed but not tests/seed-clean — the trailing -clean fell just outside the boundary the regex anchored on. One character’s worth of gap. Six clean fixtures had been scored as real findings in every run for weeks, and every one of those runs came back green, because the run and the report were written by the same thing in the same breath.
I want to be precise about what happened there, because it’s the whole point of this post: the drill didn’t prevent that bug. It performed the autopsy. A gate that’s silently broken doesn’t just fail to catch things — it removes the possibility of catching them early, because “no failure reported” is exactly what it produces while broken.
And I would not have found it. My guard was green, and a fluent green never asks to be opened.
Round 3 → Version 4: the danger my fix created
Here’s the part I didn’t see coming, and the reason a single round of feedback wouldn’t have been enough.
To fix the leak I widened the exclusion pattern. That’s the obvious repair, and the third reply aimed straight at it: an exclude that fails OPEN produces findings — noisy, catchable. An exclude that gets BROADER produces silence. If the pattern was fragile enough to fail once, the fix that widens it can start swallowing real source paths, and “zero findings under the seed folder” stays trivially true while the run scans half of what it used to.
Their instruction: assert the file count the production run actually scanned, not only what it found. Zero findings because you scanned nothing is the greenest possible run at both ends.
So version 4 asserts the denominator. Specific files that must be present in every scan, and a floor under the total number scanned. Findings alone can’t distinguish “nothing wrong” from “nothing looked at.”
They also caught something subtler in the same reply, which I’ll compress: when I pinned the set of rules each fixture was allowed to trip, I’d quietly changed what the assertion meant — from “this fixture proves the scanner catches X” to “this is what my scanner currently does.” The first legitimate new rule turns that red, the reflex is to re-pin, and that’s the moment the fixture stops being a test. Keep the two claims apart: the intended rule is the coverage claim and must never be re-pinned away; the rest is a change detector.
What came after: making the match automatic
One more turn, and it came from a different direction — from noticing that my own habit had the same defect I’d been diagnosing all week.
My planted fixtures re-run automatically on every change. But the drill — deliberately breaking a gate to confirm it still goes red — was something I did by hand, once, when I happened to think of it. Which makes the drill the one unprotected part of the whole setup, by exactly the argument I’d been making for days.
In test-driven development the red is a one-time event: you see it before the code exists, and afterward the test carries itself. A gate’s red is not an event. It’s a perishable state. It proves the check worked on the day you looked, and says nothing about whether the condition that makes it fire is still reachable next month. Mine rotted precisely there — falsifiable by design, unreachable in practice, so “no failure observed” stayed true for the wrong reason.
So now every guard gets broken automatically, on every run, and has to go red for its own reason — matching the expected message, not just returning a nonzero exit, because exit codes alone let one gate’s failure pass as another’s proof. Then everything is restored, and the restore is itself verified against a snapshot.
The first time I ran it, I broke one guard on purpose to see whether the drill could detect a dead one. It reported: you broke this and it still passed — this guard is dead. Which is the only reason I now believe the other seven.
Why none of this was possible alone
Four versions. Each one fixed a real defect. Each one created or exposed the next.
And here’s the thing I keep turning over: none of the three replies required knowing my codebase. They required standing outside it. Every single time, the flaw was in the thing I had just built and was still proud of — and that’s the exact object a maker can’t examine. The newest fix is the least suspected and has the least evidence behind it. I don’t audit what I finished an hour ago; I audit what I’ve grown suspicious of, and suspicion takes time I hadn’t spent yet.
A stranger has none of that history. They read what I was pleased with, and pleased-with is a reliable marker for under-tested.
I’d have stopped at version 1. Version 1 was checking that a regex could read a folder name. It would have stayed green for months while six clean files were scored as broken, and I’d have gone on telling people I’d seeded my detector — which was true, and which was not the same as it working.
That’s what “you can’t hold the pen alone” actually looks like when you write it down. Not a principle. Three replies from someone who owed me nothing, each one aimed at the last thing I’d bragged about.
Consider this post the fence I’m returning. If you’re running a check you’re proud of: what would happen if you broke it on purpose — and would it fail for the reason you expect, or just fail?
This is part of Stolen from the Feed — what a non-developer carries home from dev.to and actually builds. Sister series to From Zero to Ship.
Comments
Loading comments…