calibration-earned-promotion

An automated check earns the right to block only by measured agreement with a human — never by being new.

2 · The failure it offsets

A check born blocking is bypassed the first week it false-alarms: one wrong refusal at a bad moment and everyone learns the override flag instead of the rule. A check born advisory is ignored forever: warnings scroll past, and a warning nobody acts on trains everyone to skip warnings. The two failures look opposite. They are the same failure — nobody measured whether the check was right.

3 · The rule

Every new check starts advisory and keeps score against human ground truth. It may block only after at least twenty-five observed decisions with zero missed violations and at most two false alarms — N ≥ 25, FN = 0, FP ≤ 2. One miss resets the count to zero; the streak starts over. The numbers live next to the check, not in a doc. The thresholds are the lab’s own — steal the shape, tune the constants.

4 · Adopt in an afternoon

A scorecard the check itself appends to, and a promotion predicate short enough to read at review. No dashboard.

steal this

# scorecard — lives next to the check;
# the check appends one line per decision
# format: <date> <check-verdict> <human-verdict>
2026-07-30 flag agree        # true alarm
2026-07-30 pass agree        # clean pass
2026-07-30 flag false-alarm  # FP — two allowed
2026-07-30 pass miss         # FN — streak to 0
# promote advisory -> block when,
# counted since the last miss:
#   N >= 25  AND  FP <= 2  AND  FN == 0

Promotion is an edit the scorecard justifies: flip the check’s level line and cite the streak in the commit. Demotion needs no meeting either — a miss resets the numbers, and the numbers decide.

5 · The refusal test

While the check is advisory, plant one true violation and run the normal path. If the check flags it, the streak grows — good. If the check misses it, the streak observably resets to zero — also good, and more informative: the reset is the deny path firing. Either way the score is proven real. Delete the plant when done.

6 · Retire trigger

When a check has held blocking mode for a quarter with zero overrides, fold its scorecard into normal review and stop counting. Calibration is scaffolding for trust; earned trust retires it — note #1 says how.

captured 2026-07-30 · v1 · stolen from Buddy Lab · CC BY 4.0, snippets MIT