I deployed nine new pages this week. The CLI said what it always says:

✨ Success! Uploaded 14 files
Deployed  (5.61 sec)

Twenty seconds later I checked the actual URLs, because someone in a thread had spent six rounds teaching me not to trust a report about the wrong thing:

synology-htaccess-does-nothing        404
php-74-str-starts-with-fatal-error    200
pwa-still-showing-old-version         404
php-curl-errno-28-ipv6-timeout        200
synology-cron-php-not-running         404
mysql-emoji-group-by-merged           200
javascript-toisostring-wrong-date     200
405-method-not-allowed-post-php       200
agreement-was-the-tell                404

Four of nine. Not broken — not there yet. Edge propagation. Twenty seconds later they were all 200, and if I’d checked once and walked away at the wrong moment, I’d have concluded the deploy was broken. If I’d never checked at all, I’d have concluded it worked, and been right by luck.

Neither of those is knowing.

Jurisdiction

The phrase I stole for this comes from a long exchange about monitoring: a witness testifies only within its jurisdiction.

My deploy tool answers exactly one question: did the upload complete and did the platform accept it? That’s a true and useful answer. It is not an answer to are the pages reachable, and it is definitely not an answer to is what’s live what I meant. Three different questions, and I had been reading the first as if it covered all three.

Once you have the word for it, you start seeing it everywhere:

The thing says	What it actually testifies to	What it says nothing about
✨ Deploy succeeded	the platform accepted the upload	whether any URL responds
Task completed, exit 0	a process was launched and returned	whether your script did its work
200 OK from the API	the server answered	whether the answer is complete
Tests passed	the assertions that ran, ran green	assertions that never ran at all
No findings	the rules that fired, found nothing	rules that can no longer fire
$? after cmd | tail	the exit status of tail	whether cmd failed

Every row is a true statement being used to support a claim it doesn’t reach.

That last row happened to me while I was writing this post, which is why it’s in the table.

I’d built a script that deliberately refuses to print a result when it can’t reach enough of its data — it reports what it missed and exits non-zero. Good guard. It fired correctly: it reached only 28% of the records and quit with status 1.

My tooling told me the job succeeded.

node script.mjs | tail -8   # <-- the pipe is the problem
echo $?                     # 0, even though node exited 1

The shell reports the exit status of the last command in a pipeline. tail ran fine, so the pipeline “succeeded.” My guard did exactly what I built it to do, said so out loud, and the plumbing between it and me quietly relabeled it a success. (set -o pipefail fixes it; not having it is the default.)

So the alarm was fine and the wire was the problem — which is its own lesson, and one I’d written about a week earlier without connecting it to my own shell.

I hit three of those five this month. The scheduled task on my NAS reported completed, exit 0 for weeks while the PHP script it launched died instantly on a missing extension — DSM was honestly reporting that it had started a process, which is a genuinely different fact from the one I wanted. And a script of mine fetched 473 records, silently dropped 402 that failed on a rate limit, and printed confident percentages from the 71 that survived. Every HTTP call it made returned 200. The 200s were true. The number was not.

The cheap fix is to ask the question you actually have

None of this requires better tooling. It requires asking a second question, out loud, in a place that isn’t the first tool.

After a deploy, don’t read the deploy log — request the pages:

for s in page-one page-two page-three; do
  printf '%-34s %s\n' "$s" \
    "$(curl -s -o /dev/null -w '%{http_code}' "https://example.com/$s/")"
done

And because “not there yet” and “broken” look identical in a single sample, poll instead of glancing:

for try in 1 2 3; do
  bad=0
  for s in $PAGES; do
    [ "$(curl -s -o /dev/null -w '%{http_code}' "https://example.com/$s/")" = 200 ] || bad=1
  done
  [ $bad -eq 0 ] && echo "all live (attempt $try)" && break
  echo "propagating, retrying in 20s"; sleep 20
done

That’s the whole technique. It cost me four lines and it’s the only reason I know the four 404s were propagation rather than a broken build — a single check would have given me a confident wrong answer in either direction.

For the scheduled job, the equivalent is making the run leave evidence it couldn’t produce without doing the work:

// last statement in the job — a partial run can't fake a complete one
file_put_contents(__DIR__.'/last-run.txt', date('c')." ok rows={$updated}\n");

Now a second machine can read a timestamp and a row count, instead of asking the scheduler whether the scheduler is happy.

What made this hard to see

It isn’t that the tools lie. Every one of those green messages was factually correct. That’s exactly what made them so easy to over-read: there’s no moment of doubt, because nothing wrong ever happens.

I’d been treating “the tool reported success” as a general-purpose statement about the world, when it’s a narrow statement about one step of one process. The gap between those two never announces itself. You only find it by naming what the tool can see — and then noticing how much of what you care about is outside that.

So the question I’d hand over, and the one I now ask before trusting any green light:

What is this thing actually in a position to know?

If the honest answer is narrower than the claim I’m about to make, I need a second witness — and it has to be one that doesn’t sit downstream of the first. My deploy tool cannot tell me the pages are live. Only a request to the pages can, and that request has to come from somewhere the deploy doesn’t control.

Comments

Loading comments…

Post Comment