I have a scanner that grades my own projects. It reads the code, applies a list of rules, subtracts points per finding, and hands each project a letter grade. It exists because I build with AI and needed something that wasn’t me deciding whether my work was fine.

Someone in a thread pointed at a flaw in it that I couldn’t argue with.

His point: my rules overlap. If two rules fire on the same underlying defect, the project is charged twice for one mistake. And — this is the part that made me sit up — that error can only land on projects that have the defect. A clean project is never double-charged, because there’s nothing to charge. So the penalty is regressive: it pushes the worst projects further down and leaves the good ones untouched, which means my grade distribution is stretched at the bottom by a bug rather than by reality.

That’s a real argument. My instinct was to fix it that evening.

What I did instead

I’ve been burned enough times this month by fixes that felt obviously right. So before changing the scoring, I wrote something to measure whether the thing he described is actually happening in my data.

The measurement is simple: for every finding, record which rule fired and where. Then count how often two different rules fire at the same file and line.

// count co-occurrence of different rule keys at the same file:line
const seen = new Map();                       // "file:line" -> Set(ruleKey)
for (const f of findings) {
  const at = `${f.file}:${f.line}`;
  if (!seen.has(at)) seen.set(at, new Set());
  seen.get(at).add(f.key);
}

const pairs = new Map();
for (const keys of seen.values()) {
  const ks = [...keys].sort();
  for (let i = 0; i < ks.length; i++)
    for (let j = i + 1; j < ks.length; j++) {
      const p = `${ks[i]}||${ks[j]}`;
      pairs.set(p, (pairs.get(p) ?? 0) + 1);
    }
}

Then: recompute every project’s score with duplicates collapsed, and compare.

What came back

72 projects. Ten rule pairs that ever co-occur. Here are the top four:

Rule pair (co-firing at the same line)	Times
missing social preview + missing standard footer	32
SELECT * + no performance diagnostics	22
suspected N+1 query + no performance diagnostics	16
suspected N+1 query + SELECT *	14

Look at the top one. “No social preview tag” and “no standard footer” are not one defect seen twice. They’re two genuinely separate omissions that happen to live in the same file, because that file is the page template. Collapsing them wouldn’t remove a double charge — it would stop charging for a real second problem.

The same is true further down. SELECT * and missing diagnostics co-occur because both are symptoms of a page written quickly, not because one implies the other.

Then I checked his actual prediction — that the correction should lift the worst projects most:

Original grade band	Projects	Avg. points gained by deduplicating
F (0–59)	2	+0.50
D (60–69)	1	+1.00
C (70–79)	4	+1.50
B (80–89)	11	+0.82
A (90–100)	54	+0.39

The uplift peaks in the middle, not at the bottom. If the regressive error he described were the dominant effect in my data, that F row should be the largest number in the table. It’s the second smallest.

Two projects would have moved from B to A. Not because they improved. Because I applied a correction for a problem I hadn’t confirmed I had.

So I didn’t ship it.

The part I have to admit

Here’s what stops this from being a story about being clever.

Look at the sample sizes. F is two projects. D is one. The bottom of that table, where the entire argument lives, is three data points. “The uplift peaks at C” is a sentence I can write and cannot really defend.

So the honest conclusion isn’t “he was wrong.” It’s narrower and less satisfying: I don’t have enough bad projects to know whether he’s right, and shipping the fix would have changed real grades based on a theory I couldn’t test. The measurement didn’t prove him wrong. It proved I was about to act on faith and call it engineering.

If my next twenty projects include eight bad ones, that F row becomes meaningful and I’ll rerun it. The script is committed, the baseline is stored, and it compares against that baseline on every run — so the day the answer changes, something tells me instead of waiting for me to wonder.

What I actually took from it

Three things, and only one of them is about scoring.

A good argument is not a measurement. His reasoning was sound and might still be correct. Sound reasoning about your data is a hypothesis, and it costs about an hour to find out which one you have.

Fixes have a direction, and you should predict it before you look. The reason this was checkable at all is that his claim made a prediction: the correction should help the worst projects most. That gave me something to compare against. Advice that doesn’t predict anything can only be accepted or ignored — never tested.

Report what the measurement can’t say. My first draft of this post ended at the table, with the uplift peaking at C, sounding conclusive. Adding “n=2” to that row changed my own conclusion while I was writing it.

So, genuinely: thank you for the flaw. It’s on the list, with the measurement attached and a condition that will tell me when it starts to matter. That’s a better outcome than shipping it would have been — and it took longer to write this post than to run the test.

Comments

Loading comments…

Post Comment