docs(ops-lessons): filter on the artifact, not the name pointing at it — measured 4.7x inflation

Quantifying exposure to a gate-failed tune: counting by the gateway alias gave 363
rows, counting by the artifact gave 77, because the alias had carried three
different models that day. Wrong in the direction that looks careful.
This commit is contained in:
2026-09-09 17:58:20 -07:00
parent 98da4ec1f1
commit b135adce99
+22
View File
@@ -359,6 +359,28 @@ themselves: they waste attention and misdirect, but they cannot silently destroy
a good artifact. A post-mortem that lumps all three together loses the half that
decides what to do about it.
### Filter on the ARTIFACT, not on the name pointing at it
Measured 2026-09-09 while quantifying how much traffic had reached a tune that
failed a safety gate. The gateway alias `trial` had pointed at three different
artifacts across the day, so the obvious query — *"how many calls to `trial`?"*
answers a question about a **name**, not about the **thing**:
| filtered on | rows |
|---|---|
| `model_group = 'trial'` (the alias) | **363** |
| `model = 'hosted_vllm/erp-tune-v7-nvfp4a16'` (the artifact) | **77** |
**Wrong by 4.7x, in the direction that looks careful.** Reporting 363 would have
overstated the operator's own exposure nearly fivefold, and nothing about the
query would have looked sloppy — an alias is what a caller types, so counting it
feels like counting usage.
**The rule:** when the question is *"what did this artifact do"*, filter on the
artifact's identity, never on a mutable pointer to it. A name that has been
repointed carries the history of everything it ever pointed at. Same family as §3
(identity, not resemblance): an alias resembles the thing and is not it.
### A fourth variant: the instrument read a surface MID-TRANSITION
Added 2026-09-09 from a near-miss brokkr-smithy-dev caught and did not send.