a-pin-outlives-its-world

Nothing retires an expectation when the ruling underneath it changes.

2 · The failure it offsets

A suite carried a handful of failures everyone had agreed to live with. Each had its own line in the known-issues list, each line naming a test, each treated as a separate small debt. Read together they were one defect. Some pins asserted a policy value that a later, properly ratified decision had deliberately changed; the rest asserted defects that had since been fixed. Every pin predated the ruling that invalidated it by weeks.

The mechanism is the part worth taking. The declaration keyed on the test’s name. A name says which test is unhappy. It says nothing about what it is unhappy about. So when a second, unrelated defect landed in an already-declared test, the run reported exactly what it had reported the day before — same name, same count, same summary — and the new failure was absorbed into an old excuse. It can hide there indefinitely, and nothing in the record is false. The record simply is not shaped to notice.

3 · The rule

A declared failure carries a signature, not a name. The signature is four fields: the test id, the error type, the assertion kind, and the diff locus — the thing being compared, named by key or path. Never line numbers, never timings, never stack frames. Those move on edits that have nothing to do with the failure, and a signature that drifts on unrelated work is one everybody learns to regenerate without reading.

Then compare signatures, and make all four outcomes distinguishable — including the one that looks like good news. A declared failure that now passes is not a quiet win. It is the ruling underneath the pin having changed while the pin stayed put. That outcome gets its own name and its own exit code.

4 · Adopt in an afternoon

One file of declarations, one comparison, four exit codes. No framework.

steal this

# expected-failures — one line per DECLARED failure.
# signature = test-id | error-type | assertion-kind | diff-locus
# NEVER a line number, a timing, or a stack frame: those move
# on edits that have nothing to do with the failure.
policy.limits::daily_cap | AssertionError | equals | cap.value
render.cards::alt_text   | AssertionError | raises | alt.missing
 
# compare OBSERVED signatures against DECLARED ones, and give
# all four outcomes a distinct exit code:
#   0  MATCHED     declared, and it failed the declared way
#   3  DRIFTED     declared id failed a DIFFERENT way — a
#                  second defect wearing the first one's name
#   4  UNDECLARED  a failure with no line in this file
#   5  ROTTED      a declared failure now PASSES — the ruling
#                  under the pin changed; delete the line
# ROTTED is the one that must never be silent. Good news that
# exits 0 is exactly how a pin outlives its world.

Two fields do the work. DRIFTED is why the signature exists at all: it is the only outcome a name-keyed list cannot produce. ROTTED is why the file shrinks — a declaration is a liability that has to be argued for every run, not a note that ages in place.

5 · The refusal test

Change one character in a declared signature — flip the assertion kind, the cheapest field to alter — and run the suite. It must exit 3, name the test, and print the declared and observed signatures side by side. If it exits 0, the comparison is still keying on the name and the whole file is decoration. Restore the line when you are done. Run this once, deliberately, the day you install the check.

6 · Retire trigger

Retire the comparison when a ruling emits, as part of ratifying itself, the list of expectations it invalidates. At that point pins die with the decisions that justified them and there is nothing left to catch. Until then this is the only thing standing between a ruling and the tests that outlived it.

captured 2026-08-06 · v1 · stolen from Buddy Lab · CC BY 4.0, snippets MIT