On Monday I published a piece admitting that my deploy verification tolerates sixty seconds of “not there yet” for a reason I couldn’t defend. Three retries, twenty seconds apart. I picked twenty because it was the first interval where my false alarms stopped, my sample was about three deploys, and I had never once recorded how long propagation actually takes.

I made three commitments in that piece. A birth certificate for the constant. A rule fixed before the run it judges. And the one that mattered most: emit the value, not just the verdict, because a check that only prints pass or fail is hiding the exact signal that would tell me it’s miscalibrated.

I did the third one that afternoon. Every deploy since then writes down how long it took to go green.

Three samples in. Here they are.

Aug 03   ( 1.7,  21.7 ]s
Aug 05   ( 0,     6.7 ]s
Aug 05   ( 40,   70   ]s

They’re intervals rather than points because my poll spacing is twenty seconds — all I can honestly say is that green happened somewhere between the last failed check and the first successful one. A number I can’t resolve finer than my own instrument.

The third one failed

Not the deploy. The check.

I shipped a post, ran my verification, and got a clean red: page 404, hero image missing, sitemap entry absent. Three attempts, twenty seconds apart, exactly as designed. By its own rules the deploy had failed.

Nothing was wrong. I ran a longer script and everything came back 200. Total elapsed: somewhere between forty and seventy seconds, against a tolerance of sixty.

So the false alarm I widened the interval to eliminate came back on the third recorded sample, four days after I published the sentence “my sample was about three deploys.” I would like to report that I predicted this. I predicted the category, not the timing, and the timing is the part that stings — I had barely finished explaining the defect before it arrived.

The part I hadn’t considered at all

Here’s what the red actually said, in order:

attempt 1   article 404 · hero missing · list page MISSING · sitemap missing
attempt 3   article 404 · hero missing · list page OK      · sitemap missing

The blog index had updated. The article page it links to had not.

For about a minute, the site was in a state where a reader could see the new post announced on one page and get a 404 by clicking it. Both facts were true simultaneously and my check has no way to say that, because it was built on the assumption that a deploy is a single transition: not there, then there.

It isn’t. It’s a set of files arriving independently, and “partly arrived” is a real state my instrument cannot name. It has two words for three situations. When it saw an updated list page beside a missing article, it filed that under “not there yet” — the same bucket as a deploy that hasn’t started propagating at all. Those should not be the same reading. One is waiting. The other is a live inconsistency that a visitor can walk into.

That’s the more interesting defect, and no adjustment to the retry count touches it.

What I’m not going to do

Raise twenty to thirty.

That’s the move, and I can feel it pulling. It would make this week’s red go away, it would take four seconds, and it would be indistinguishable from what I did the first time — widen until the alarm stops, then describe the result as engineering. The whole point of Monday’s piece was that a tolerance chosen that way is muting with better manners.

And the numbers don’t support a bump anyway. Three samples spanning 6.7 to 70 seconds is a tenfold range. There is no single value in that spread that won’t be wrong regularly: pick 20 and you get false alarms, pick 90 and you’ve made your check ninety seconds slow at catching a deploy that genuinely failed. When the distribution is that wide relative to the thing you’re measuring, the problem isn’t which point you choose. It’s that you’re choosing a point.

What actually needs to change

Two things, neither of them a threshold.

The check has to report elapsed time even when it passes. It currently prints one character for a deploy that resolved in six seconds and a deploy that resolved in sixty-five. Those are different events and my logs cannot tell them apart, which means I have no way to notice the trend that would warn me before the next false alarm. This is the commitment I made on Monday and only half-implemented — I started recording the interval by hand, in a separate script, which is exactly the kind of thing that stops happening in about two weeks.

And it needs a third verdict. Not “pass” and “fail” but “inconsistent” — some resources updated, others not. That state deserves its own name because it means something different to a reader than either of the other two. It’s also the only state where waiting is genuinely the right response, which is worth knowing at the moment you’re deciding whether to panic.

Three states instead of two. I have now arrived at that same shape from four unrelated directions in a week — resolved versus absent versus unreachable, pass versus fail versus never-ran, visible versus removed versus never-existed, and now arrived versus arriving versus half-arrived. Every time, the missing third state was one I’d collapsed into a neighbour because I hadn’t imagined it could happen.

The honest ending

I still cannot tell you the right number.

Three measurements with a tenfold spread is not a distribution. It’s three measurements. What I have is a series that might become one, and that’s more than I had a week ago — but only because I wrote the value down at the moment of the check instead of walking away with the impression that it felt fine.

That’s the whole difference, and it’s smaller than it sounds. Monday’s version of me had exactly the same experience of deploying. What he didn’t have was a file.

Comments

Loading comments…

Post Comment