memory: snapshot — run 3c gated, run 4 training, DAC revert, GPU rebalance, NAS exposure

2026-09-05 06:36:56 -07:00
parent 5237efa299
commit 82158e7e36
13 changed files with 958 additions and 651 deletions
@@ -0,0 +1,39 @@
+# `[2026-09-05]` A peer's "2.7x stack effect" was a coin flip — the operator caught it and the arithmetic is worth keeping
+
+brokkr-smithy-dev reported that a diversity **noise floor** was 2.7x tighter on ana-ml2 than on
+the GX10 and ranked ana-ml2 the better instrument on it. I relayed it. **The operator rejected it
+on instinct** — *"that makes zero sense. except for speed, serving a model should be identical
+across servers"* — and he was substantially right.
+
+    ana-ml2   0.9688, 0.9574   spread 1.14pp   mean 0.9631
+    gx10      0.9583, 0.9895   spread 3.12pp   mean 0.9739
+
+    between-box LEVEL difference     1.08pp
+    ana-ml2's own replicate spread   1.14pp     <- LARGER than the between-box gap
+    pooled range ignoring box        3.21pp     <- ~= the entire "GX10 floor"
+
+**Each "floor" is `|block0 - block1|` from n=2.** A range over two draws is not an estimate of
+dispersion; the ratio of two such ranges is a ratio of two half-normals — a **half-Cauchy**.
+brokkr computed the tail himself on retraction: `P(ratio >= 2.7) = 1 - (2/pi)*arctan(2.7)`,
+doubled = **0.452**, confirmed by 400,000-draw simulation. He had reported a coin flip as a
+measured effect and ranked hardware on it. Retracted at `97f73dd`.
+
+⚠ **The disconfirming evidence was inside his own sentence.** He wrote "the base level is nearly
+identical, it is the spread that shifts" and offered it as *reassurance*. A real stack effect
+moves the level. Level agreeing while a two-draw range differs is the signature of a noisy range
+estimator — he had the refutation in hand and read it as support.
+
+⚠ **His own diagnosis of why it got through is the transferable part:** he had spent the session
+triaging *my* claims hard — reading the encode path, simulating preflight, recomputing shas from
+bytes — and this one was **his, and flattering**: it made his earlier work look prescient and
+produced a clean recommendation. **Asymmetric scepticism is one error twice, and the flattering
+direction needs the extra pass.**
+
+**What survived, deliberately separated:** re-measuring the floor on whatever stack actually
+serves stays non-negotiable. The mechanism list is sound whether or not those four numbers show
+it — sm_121 vs sm_120 kernel selection, vLLM 0.28.0 against mixed 0.26/nightly, and separately
+measured batch-invariance (3.12pp at jobs=8, same order as the whole claimed effect).
+**Retracting the evidence and keeping the discipline are different acts.** Settling it properly
+wants several blocks per box and is its own probe, not a by-product of a gate.
+
+See [[2026-09-05-vllm-on-sm121-and-run4]].