vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(coldfusion-abliteration): GPU0 seat restore order is load-bearing — correct the claim and the runbook

Browse Source 
Restoring the two GPU0 seats with `start meromero; sleep 10; start gen` put
meromero into a 7-restart crash-loop:

  ValueError: Free memory on device cuda:0 (35.3/94.97 GiB) on startup is less
  than desired GPU memory utilization (0.52, 49.38 GiB).

The previous commit's README claimed restore order "is not actually load-bearing"
on the grounds that both seats pass --gpu-memory-utilization as a fraction of
total VRAM. That is half right and the wrong half mattered: the fraction sets the
target, but vLLM gates startup on FREE VRAM and refuses to start unless the whole
target is available. GPU0 runs at ~96.4/97.9 GB with roughly 0.4 GiB of slack, so
the seats coexist only in the order they were originally brought up, and meromero
is the one that does not fit in the remainder. The pre-existing auto-memory note
("gen takes a fraction of free VRAM at startup and will starve meromero") was
pointing at the real effect.

Also: "first" means healthy, not ten seconds earlier. A sleep 10 against a
two-to-three minute weight load is simultaneity, not ordering — gate on observed
state.

Recovery applied: stop gen, wait for meromero healthy, start gen. Verified
against the pre-window baseline rather than against "both green":

  gen KV       14.36 GiB / 403,065 tok / 1.54x  ->  14.34 GiB / 401,550 tok / 1.53x
  meromero KV  542,202 tok                      ->  542,202 tok
  RestartCount 0 on both; summarizer smoke-tested through LiteLLM

Note for the next reader: raw nvidia-smi used-MiB is the wrong check here. It
reads 89,503 now vs 96,376 before, which looks like a 6.9 GB regression and is
allocator slack — serving capacity is unchanged. The anomalous boots were the
high ones (34.95 GiB KV), where gen came up on an empty card mid-window.
This commit is contained in:
vh
2026-08-20 13:08:56 -07:00
parent 1b3fb270e7
commit a9d73dad41
3 changed files with 69 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.d/2026-08-20-coldfusion-abliteration-landed.md
+46
View File
@@ -101,6 +101,52 @@ means re-measuring the incumbent through this script (one more GPU window).
Consistent with [[reference_abliteration_mtp_lessons]]: KL is a **fidelity**
number here, not the viability gate — that remains MTP acceptance (59.1%).
### ⚠️ The restore bit me — GPU0 seat order is load-bearing, and "first" means *healthy*
Restoring with `docker start meromero; sleep 10; docker start gen` put **meromero
into a 7-restart crash-loop**: gen finished claiming the card while meromero was
still loading weights, and meromero died on
```
ValueError: Free memory on device cuda:0 (35.3/94.97 GiB) on startup is less than
desired GPU memory utilization (0.52, 49.38 GiB).
```
**I had this half-right and the half I got wrong is what caused it.** I checked the
compose files, saw `--gpu-memory-utilization` is a fraction of **total** VRAM, and
concluded restore order "is not actually load-bearing" — I even wrote that into the
README before the seats came back. Wrong: the fraction sets the *target*, but vLLM
gates startup on **free** VRAM, refusing to start unless the whole target is
available right now. GPU0 runs at ~96.4/97.9 GB with ~0.4 GiB of slack, so the two
seats coexist **only in the order they were originally brought up**, and meromero
is the one that does not fit in the remainder. The old auto-memory note ("gen takes
a fraction of FREE VRAM at startup and will starve meromero") was pointing at a
real effect; my correction of it was the error.
Recovery: `docker stop vllm-gen` → wait for meromero `healthy` → `docker start
vllm-gen`. Sequence-and-verify, not sequence-and-sleep — a `sleep 10` against a
2-3 minute weight load is simultaneity, not ordering.
(Generalises [[feedback_confirm_reboot_by_observing_down]]: gate on the observed
state, not on elapsed time.)
**Restore verified against the pre-window baseline, not just "it's green."** Both
seats `healthy`, `RestartCount=0`, and — the check that actually matters — the KV
pools match what they were before the session:
| | pre-window (18:32) | after restore (20:06) |
|---|---|---|
| gen KV | 14.36 GiB, 403,065 tok, **1.54×** | 14.34 GiB, 401,550 tok, **1.53×** |
| meromero KV | 542,202 tok | 542,202 tok |
⚠️ **Do not read raw `nvidia-smi` used-MiB as the restore check.** GPU0 shows
89,503 MiB used now vs 96,376 before, which looks like a 6.9 GB regression and is
not one — the delta is allocator slack, and serving capacity (KV pool, max
concurrency) is identical. The genuinely anomalous boots were the *high* ones
(34.95 GiB KV at 19:50/19:55/20:00), where gen came up on an empty card mid-window
and grabbed more than its steady-state share. Card now sits at 7,746 MiB free vs
~1,500 before, which is more co-tenancy slack, not less. Summarizer smoke-tested
end-to-end through LiteLLM after the restore.
### Three durable process lessons from the measurement
1. **★ Report abliteration KL SPLIT BY PROMPT CLASS.** A single averaged KL over a