fix(coldfusion-abliteration): GPU0 seat restore order is load-bearing — correct the claim and the runbook

Restoring the two GPU0 seats with `start meromero; sleep 10; start gen` put
meromero into a 7-restart crash-loop:

  ValueError: Free memory on device cuda:0 (35.3/94.97 GiB) on startup is less
  than desired GPU memory utilization (0.52, 49.38 GiB).

The previous commit's README claimed restore order "is not actually load-bearing"
on the grounds that both seats pass --gpu-memory-utilization as a fraction of
total VRAM. That is half right and the wrong half mattered: the fraction sets the
target, but vLLM gates startup on FREE VRAM and refuses to start unless the whole
target is available. GPU0 runs at ~96.4/97.9 GB with roughly 0.4 GiB of slack, so
the seats coexist only in the order they were originally brought up, and meromero
is the one that does not fit in the remainder. The pre-existing auto-memory note
("gen takes a fraction of free VRAM at startup and will starve meromero") was
pointing at the real effect.

Also: "first" means healthy, not ten seconds earlier. A sleep 10 against a
two-to-three minute weight load is simultaneity, not ordering — gate on observed
state.

Recovery applied: stop gen, wait for meromero healthy, start gen. Verified
against the pre-window baseline rather than against "both green":

  gen KV       14.36 GiB / 403,065 tok / 1.54x  ->  14.34 GiB / 401,550 tok / 1.53x
  meromero KV  542,202 tok                      ->  542,202 tok
  RestartCount 0 on both; summarizer smoke-tested through LiteLLM

Note for the next reader: raw nvidia-smi used-MiB is the wrong check here. It
reads 89,503 now vs 96,376 before, which looks like a 6.9 GB regression and is
allocator slack — serving capacity is unchanged. The anomalous boots were the
high ones (34.95 GiB KV), where gen came up on an empty card mid-window.

persistent-memory.d/2026-08-20-coldfusion-abliteration-landed.md

@@ -101,6 +101,52 @@ means re-measuring the incumbent through this script (one more GPU window).
 Consistent with [[reference_abliteration_mtp_lessons]]: KL is a **fidelity**
 number here, not the viability gate — that remains MTP acceptance (59.1%).
 
+### ⚠️ The restore bit me — GPU0 seat order is load-bearing, and "first" means *healthy*
+
+Restoring with `docker start meromero; sleep 10; docker start gen` put **meromero
+into a 7-restart crash-loop**: gen finished claiming the card while meromero was
+still loading weights, and meromero died on
+
+```
+ValueError: Free memory on device cuda:0 (35.3/94.97 GiB) on startup is less than
+desired GPU memory utilization (0.52, 49.38 GiB).
+```
+
+**I had this half-right and the half I got wrong is what caused it.** I checked the
+compose files, saw `--gpu-memory-utilization` is a fraction of **total** VRAM, and
+concluded restore order "is not actually load-bearing" — I even wrote that into the
+README before the seats came back. Wrong: the fraction sets the *target*, but vLLM
+gates startup on **free** VRAM, refusing to start unless the whole target is
+available right now. GPU0 runs at ~96.4/97.9 GB with ~0.4 GiB of slack, so the two
+seats coexist **only in the order they were originally brought up**, and meromero
+is the one that does not fit in the remainder. The old auto-memory note ("gen takes
+a fraction of FREE VRAM at startup and will starve meromero") was pointing at a
+real effect; my correction of it was the error.
+
+Recovery: `docker stop vllm-gen` → wait for meromero `healthy` → `docker start
+vllm-gen`. Sequence-and-verify, not sequence-and-sleep — a `sleep 10` against a
+2-3 minute weight load is simultaneity, not ordering.
+(Generalises [[feedback_confirm_reboot_by_observing_down]]: gate on the observed
+state, not on elapsed time.)
+
+**Restore verified against the pre-window baseline, not just "it's green."** Both
+seats `healthy`, `RestartCount=0`, and — the check that actually matters — the KV
+pools match what they were before the session:
+
+| | pre-window (18:32) | after restore (20:06) |
+|---|---|---|
+| gen KV | 14.36 GiB, 403,065 tok, **1.54×** | 14.34 GiB, 401,550 tok, **1.53×** |
+| meromero KV | 542,202 tok | 542,202 tok |
+
+⚠️ **Do not read raw `nvidia-smi` used-MiB as the restore check.** GPU0 shows
+89,503 MiB used now vs 96,376 before, which looks like a 6.9 GB regression and is
+not one — the delta is allocator slack, and serving capacity (KV pool, max
+concurrency) is identical. The genuinely anomalous boots were the *high* ones
+(34.95 GiB KV at 19:50/19:55/20:00), where gen came up on an empty card mid-window
+and grabbed more than its steady-state share. Card now sits at 7,746 MiB free vs
+~1,500 before, which is more co-tenancy slack, not less. Summarizer smoke-tested
+end-to-end through LiteLLM after the restore.
+
 ### Three durable process lessons from the measurement
 
 1. **★ Report abliteration KL SPLIT BY PROMPT CLASS.** A single averaged KL over a

services/coldfusion-abliteration/README.md

@@ -444,12 +444,28 @@ aborted window did exactly that and cost nothing but two minutes.
 
 (`--capture-dtype float32` remains as an escape hatch; it needs 111 GB, so it
 also needs `--max-layer 46` to fit on one card. The two agree to 0.0005, so there
-is no reason to reach for it.) **Restore after:** start `vllm-meromero-rp`
+is no reason to reach for it.)
-**first**, then `vllm-gen`. (The stated reason — "gen grabs a fraction of *free*
+
-VRAM" — is not what the configs do: both seats pass `--gpu-memory-utilization` as
+**⚠️ Restore after: start `vllm-meromero-rp` first, and WAIT FOR IT TO GO HEALTHY
-a fraction of **total** (`MEROMERO_GPU_MEM_UTIL=0.52`, `GEN_GPU_MEM_UTIL=0.43`),
+before starting `vllm-gen`.** This ordering is load-bearing, and "first" means
-so restore order is not actually load-bearing. Kept as the runbook order anyway;
+*fully up*, not *ten seconds earlier* — a `docker start meromero; sleep 10; docker
-it costs nothing.)
+start gen` put meromero into a **7-restart crash-loop** on 2026-08-20, because gen
+finished claiming the card while meromero was still loading weights:
+
+```
+ValueError: Free memory on device cuda:0 (35.3/94.97 GiB) on startup is less than
+desired GPU memory utilization (0.52, 49.38 GiB).
+```
+
+The mechanism, stated precisely because a half-right version of it is what caused
+the mistake: `--gpu-memory-utilization` is a fraction of **total** VRAM (0.52 ×
+94.97 = 49.38 GiB is meromero's *target*), but vLLM gates startup on **free** VRAM
+— it refuses to start unless the card currently has the whole target available.
+So the two seats coexist only in the order they were originally brought up. GPU0
+runs at ~96.4/97.9 GB with about 0.4 GiB of slack; whichever seat starts second
+gets whatever the first one left, and meromero is the one that does not fit in the
+remainder. Recovery when it does happen: `docker stop vllm-gen`, wait for meromero
+to report `healthy`, then `docker start vllm-gen`.
 
 **4. fla is irrelevant here — but harmless.** `fla` + `einops` are `--target`
 -installed to `/tank/aimodels/coldfusion-abliteration/pylibs` and reached via