Files
esh-pfi-infrastructure/persistent-memory.d/2026-09-01-wrong-object-measurement.md
T
vh 7142657749 memory: snapshot — gx10 unracked and next up as an inference+training box
Run 3c's new intended home is the GX10 rather than a power triage on ana-ml2: a ~240 W
appliance instead of the kilowatt-class box that tripped the breaker, and 121 GB unified
holds the 49 GB bf16 base comfortably where ana-ml2 was tight. The box is bare, so the
first move is a throughput probe rather than a harness port.

Also banked: the Ada migration settling on zfs send with branch (b) ruled out by irv-ml1
keeping its eight services; the Synapse 39-release upgrade with its one-way schema
migration, the appservice namespace opening and the admin API lockdown; the ratified room
alias convention; and a named failure class — a correct check aimed at the wrong object —
with six instances from one day across three sessions.
2026-09-01 16:32:21 -07:00

43 lines
2.5 KiB
Markdown

# `[2026-09-01]` A named failure class: a correct check aimed at the wrong object
Six instances surfaced across three sessions in a single day, independently, in unrelated
domains. It has a distinguishing property that makes it worth naming separately from
"a bad measurement":
> **Re-running the same check cannot catch it, because the check is correct and the object
> is wrong.** The only move that breaks it is asking what the artifact *is* before trusting
> any metric computed over it.
## The instances
| where | the metric | the artifact nobody opened |
|---|---|---|
| comfy-dev, civitai | a "~25 KB/s throttle" | a **9,685-byte login page** returned on failed auth |
| comfy-dev, render harness | reported "completed" | a 4.7 KB **all-black PNG** |
| comfy-dev, audio | RMS said healthy | degenerate audio where the **loudness WAS the noise** |
| ledger-dev, capability 5 | healthy ping, healthy container, healthy delivery | a **requirement unmet** — a mailbox, not an away-channel |
| infra-ops, gx10 headless | six green verifies | `is-enabled` on a **static unit**; gdm still running |
| infra-ops, synapse admin | an `ipallowlist` comment promising fleet access | fleet traffic **hairpins out the WAN**; 10.0.0.0/8 matches nothing |
## Related lessons banked the same day
- **A caveat plus propagation is decoration.** infra-ops flagged a sample-size problem AND
escalated the claim in the same message. If a number needs re-measuring before it can be
repeated, hold the escalation until it has been. The caveat made the uncareful thing look
examined.
- **Verify with a negative control.** A 200 means nothing without a 401 beside it. Used on
the pewpewstudio key mint; used on the `/_synapse/admin` lock (tested from a genuinely
external vantage via the NH3 residential egress proxy, because testing from a fleet host
sits inside the allow-list and proves nothing).
- **Reasoning from what is VISIBLE to what EXISTS.** infra-ops found no `sm_89` part in the
inventory, saw one new machine, and collapsed "the box I can see" into "the box that is
coming". They were two different machines. Wrote "I cannot resolve it and will not guess",
then built three messages on the guess.
## Disposition
Recommended for a row in `docs/pfi/training-throughput-playbook.md` §4 (the durable home for
"why a run LIES about itself"), with attribution to comfy-dev and ledger-dev.
**NOT YET WRITTEN — awaiting operator.** Tracking surface: this file, plus althing threads
`01M1EYBSYA4QRYK54PX0K1S8CS` (comfy-dev) and `01M1EXVPZT66SSGCRZH3W624R7` (ledger-dev).