diff --git a/docs/pfi/ops-lessons-playbook.md b/docs/pfi/ops-lessons-playbook.md index 84ec6da..4fec345 100644 --- a/docs/pfi/ops-lessons-playbook.md +++ b/docs/pfi/ops-lessons-playbook.md @@ -359,6 +359,28 @@ themselves: they waste attention and misdirect, but they cannot silently destroy a good artifact. A post-mortem that lumps all three together loses the half that decides what to do about it. +### Filter on the ARTIFACT, not on the name pointing at it + +Measured 2026-09-09 while quantifying how much traffic had reached a tune that +failed a safety gate. The gateway alias `trial` had pointed at three different +artifacts across the day, so the obvious query โ€” *"how many calls to `trial`?"* โ€” +answers a question about a **name**, not about the **thing**: + +| filtered on | rows | +|---|---| +| `model_group = 'trial'` (the alias) | **363** | +| `model = 'hosted_vllm/erp-tune-v7-nvfp4a16'` (the artifact) | **77** | + +**Wrong by 4.7x, in the direction that looks careful.** Reporting 363 would have +overstated the operator's own exposure nearly fivefold, and nothing about the +query would have looked sloppy โ€” an alias is what a caller types, so counting it +feels like counting usage. + +**The rule:** when the question is *"what did this artifact do"*, filter on the +artifact's identity, never on a mutable pointer to it. A name that has been +repointed carries the history of everything it ever pointed at. Same family as ยง3 +(identity, not resemblance): an alias resembles the thing and is not it. + ### A fourth variant: the instrument read a surface MID-TRANSITION Added 2026-09-09 from a near-miss brokkr-smithy-dev caught and did not send.