From a9d73dad411405b96565a57b5c192a203a3b95c3 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Thu, 20 Aug 2026 13:08:56 -0700 Subject: [PATCH] =?UTF-8?q?fix(coldfusion-abliteration):=20GPU0=20seat=20r?= =?UTF-8?q?estore=20order=20is=20load-bearing=20=E2=80=94=20correct=20the?= =?UTF-8?q?=20claim=20and=20the=20runbook?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Restoring the two GPU0 seats with `start meromero; sleep 10; start gen` put meromero into a 7-restart crash-loop: ValueError: Free memory on device cuda:0 (35.3/94.97 GiB) on startup is less than desired GPU memory utilization (0.52, 49.38 GiB). The previous commit's README claimed restore order "is not actually load-bearing" on the grounds that both seats pass --gpu-memory-utilization as a fraction of total VRAM. That is half right and the wrong half mattered: the fraction sets the target, but vLLM gates startup on FREE VRAM and refuses to start unless the whole target is available. GPU0 runs at ~96.4/97.9 GB with roughly 0.4 GiB of slack, so the seats coexist only in the order they were originally brought up, and meromero is the one that does not fit in the remainder. The pre-existing auto-memory note ("gen takes a fraction of free VRAM at startup and will starve meromero") was pointing at the real effect. Also: "first" means healthy, not ten seconds earlier. A sleep 10 against a two-to-three minute weight load is simultaneity, not ordering — gate on observed state. Recovery applied: stop gen, wait for meromero healthy, start gen. Verified against the pre-window baseline rather than against "both green": gen KV 14.36 GiB / 403,065 tok / 1.54x -> 14.34 GiB / 401,550 tok / 1.53x meromero KV 542,202 tok -> 542,202 tok RestartCount 0 on both; summarizer smoke-tested through LiteLLM Note for the next reader: raw nvidia-smi used-MiB is the wrong check here. It reads 89,503 now vs 96,376 before, which looks like a 6.9 GB regression and is allocator slack — serving capacity is unchanged. The anomalous boots were the high ones (34.95 GiB KV), where gen came up on an empty card mid-window. --- ...26-08-20-coldfusion-abliteration-landed.md | 46 +++++++++++++++++++ persistent-memory.md | 2 +- services/coldfusion-abliteration/README.md | 28 ++++++++--- 3 files changed, 69 insertions(+), 7 deletions(-) diff --git a/persistent-memory.d/2026-08-20-coldfusion-abliteration-landed.md b/persistent-memory.d/2026-08-20-coldfusion-abliteration-landed.md index 229e180..5098435 100644 --- a/persistent-memory.d/2026-08-20-coldfusion-abliteration-landed.md +++ b/persistent-memory.d/2026-08-20-coldfusion-abliteration-landed.md @@ -101,6 +101,52 @@ means re-measuring the incumbent through this script (one more GPU window). Consistent with [[reference_abliteration_mtp_lessons]]: KL is a **fidelity** number here, not the viability gate — that remains MTP acceptance (59.1%). +### ⚠️ The restore bit me — GPU0 seat order is load-bearing, and "first" means *healthy* + +Restoring with `docker start meromero; sleep 10; docker start gen` put **meromero +into a 7-restart crash-loop**: gen finished claiming the card while meromero was +still loading weights, and meromero died on + +``` +ValueError: Free memory on device cuda:0 (35.3/94.97 GiB) on startup is less than +desired GPU memory utilization (0.52, 49.38 GiB). +``` + +**I had this half-right and the half I got wrong is what caused it.** I checked the +compose files, saw `--gpu-memory-utilization` is a fraction of **total** VRAM, and +concluded restore order "is not actually load-bearing" — I even wrote that into the +README before the seats came back. Wrong: the fraction sets the *target*, but vLLM +gates startup on **free** VRAM, refusing to start unless the whole target is +available right now. GPU0 runs at ~96.4/97.9 GB with ~0.4 GiB of slack, so the two +seats coexist **only in the order they were originally brought up**, and meromero +is the one that does not fit in the remainder. The old auto-memory note ("gen takes +a fraction of FREE VRAM at startup and will starve meromero") was pointing at a +real effect; my correction of it was the error. + +Recovery: `docker stop vllm-gen` → wait for meromero `healthy` → `docker start +vllm-gen`. Sequence-and-verify, not sequence-and-sleep — a `sleep 10` against a +2-3 minute weight load is simultaneity, not ordering. +(Generalises [[feedback_confirm_reboot_by_observing_down]]: gate on the observed +state, not on elapsed time.) + +**Restore verified against the pre-window baseline, not just "it's green."** Both +seats `healthy`, `RestartCount=0`, and — the check that actually matters — the KV +pools match what they were before the session: + +| | pre-window (18:32) | after restore (20:06) | +|---|---|---| +| gen KV | 14.36 GiB, 403,065 tok, **1.54×** | 14.34 GiB, 401,550 tok, **1.53×** | +| meromero KV | 542,202 tok | 542,202 tok | + +⚠️ **Do not read raw `nvidia-smi` used-MiB as the restore check.** GPU0 shows +89,503 MiB used now vs 96,376 before, which looks like a 6.9 GB regression and is +not one — the delta is allocator slack, and serving capacity (KV pool, max +concurrency) is identical. The genuinely anomalous boots were the *high* ones +(34.95 GiB KV at 19:50/19:55/20:00), where gen came up on an empty card mid-window +and grabbed more than its steady-state share. Card now sits at 7,746 MiB free vs +~1,500 before, which is more co-tenancy slack, not less. Summarizer smoke-tested +end-to-end through LiteLLM after the restore. + ### Three durable process lessons from the measurement 1. **★ Report abliteration KL SPLIT BY PROMPT CLASS.** A single averaged KL over a diff --git a/persistent-memory.md b/persistent-memory.md index 5db926a..f5c7963 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -150,7 +150,7 @@ _As of 2026-08-20 — continued from the 08-19 infra session (DNS/.internal, wat - **🟢 WT #401 (fd-leak deadlock) CLOSED 2026-08-17 — one ping still owed.** worldtree-dev closed it on our demo verify. Layers: **(a) their `e41b139`** pins `ulimits: nofile 65536/65536` in the worldtree compose anchor — **demo VERIFIED** (api + matrix recreated 22:55:34Z, `ulimit -Sn`=65536); **personal/pinned are covered-not-verified**, they inherit at their next promotion/recreate. **(b) our host floor is STAGED, NOT ACTIVE** — `/etc/docker/daemon.json` on corviduo-dev carries `default-ulimits nofile 65536/65536` but **`default-ulimits` is NOT SIGHUP-reloadable** (measured on 29.4.3: post-reload the daemon's own "Reloaded configuration" log omits it and a fresh container still reports 1024). Activation needs a full dockerd restart = bounces all 13 containers; **worldtree-dev explicitly does NOT want one**, and `live-restore:true`-then-restart is PARKED as a separate host-side improvement for the operator to rule on, never folded into #401. Playbook `playbooks/corviduo-dev-docker-default-ulimits.yaml` (verify step 3 fails BY DESIGN until a restart). Hourly fd tripwire on corviduo-dev stays armed. **⏳ OWED: ping worldtree-dev in thread `01M08QQ655XD6VKEV7MA9GX0NS` once worldtree-personal recreates and 65536 is confirmed there.** Commit `7f3f265`. -- **🟢 COLD-FUSION ABLITERATION — LANDED 2026-08-20 (the real work; abliterated model WORKS, verify + quant still owed).** Abliterated `DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1` with the **Robinson formula** (`docs/pfi/abliteration-recipe-qwen38.md`; harness `services/coldfusion-abliteration/`). Output `ana-ml2:/tank/aimodels/qwen38-27b-coldfusion-abliterated-L35-bf16`; bitwise-verified 131/131 targets changed, 333/333 vision byte-identical, 735/735 others untouched. A/B vs stock: explicit-sexual + graphic-torture (the measured stock refusal surface) go refused→complied, self-harm guardrail survives, coherence intact — the Robinson design point. **THREE first-session diagnoses were wrong, all corrected:** (1) **layer selection by two-template |cos| agreement is misleading on a merged base** — its pick (L18) was the *worst*-separating layer and abliterating there was a measured no-op; replaced with harmful/harmless **separation** (Cohen's d/AUC) gated on the sink screen → **L35** (d9.35, AUC0.9997, sink0.094%); (2) **"bf16 NaNs, use fp32" was a misdiagnosis** — the NaN was multi-GPU sharding + `expandable_segments`, not precision; bf16 on ONE GPU is deterministic+coherent at 50 GB, 4.3× faster (now gated, `CUDA_VISIBLE_DEVICES=0`); (3) **corpus-size hypothesis falsified** (8→416 moved agreement 0.594→0.624, nothing). Write is now **shard surgery** (no model object — `AutoModelForCausalLM` is text-only and would drop vision + skip the in-band MTP edit). **🎯 THESIS PROVEN 2026-08-20 (same session, later):** quantized L35 → mixed NVFP4 (`/tank/aimodels/qwen38-27b-coldfusion-L35-nvfp4-mixed`, 22.5 GB) and measured **MTP acceptance 59.1% median** (51–65%) — clears the ≳40% gate AND **beats the incumbent Heretic seat's ~47%.** So Robinson's **in-band** MTP abliteration accepts BETTER than Heretic's byte-identical graft — the additive delta the experiment tested, positive. Abliteration survives the quant (creative refusals drop, self-harm guardrail intact, coherent); decode 118.7 tok/s (faster, image-confounded). Env foot-guns hardened: quant venv drifted (transformers 5.10/llmcompressor 0.12 no longer delegate `num_attention_heads` to text_config → promote-then-restore in `quant_mixed_nvfp4.py`); a <23GB quant saves single-file no-index → build index from the safetensors header (never `safe_open`, ENOMEMs on ZFS). Commit `725c8fd`. **⚠ NOT CUT OVER** — incumbent gen seat untouched; making L35 the `gen` seat is a **separate operator decision** needing the full Stage-3 gate (PPL, prefill, surface 6/6, refusal-probe battery) + real multi-turn hold. **Do NOT delete** `qwen38-27b-heresy-nvfp4-mixed`. **📐 KL MEASURED 2026-08-20 (third session) — the surgery is SELECTIVE.** `kl_divergence.py` (new, beside the harness): first-token KL(stock‖L35) over the full 248,320 vocab, bf16-vs-bf16, held-out prompts. **Answer mode: harmless median 0.0211 / mean 0.0364, harmful median 0.5996 → 28.4× selectivity** (think mode 0.0042 / 0.3068 → 72.8×); top-1 agreement on benign prompts stays **89.8%**; **self-KL noise floor exactly 0.0**, so every digit is signal. Reverse KL on harmful is 1.43 vs forward 0.70 — the mass-where-stock-had-none asymmetry that is abliteration's signature. Vs the Heretic reference figures (0.1191 prior seat, **0.0759 the live `absolute-heresy` seat**) ours looks materially gentler, but ⚠️ **NOT a head-to-head** — those are Heretic's own optimizer output on a different base with its own harmless set/template; a real comparison needs the incumbent re-measured through this script (one more window). KL stays a **fidelity** number, not the gate (that's MTP acceptance, 59.1%). Run cost **2m40s**, both GPU0 seats down. **Three durable process lessons:** (1) report abliteration KL **split by prompt class** — a mixed average is meaningless since the metric is meant to be big on harmful and small on benign; (2) the runbook's "bf16 is 50 GB / only gen must go" was **50.10 GiB mislabelled** — text weights are 51,300 MiB and freeing either GPU0 seat alone leaves ~50,933, so **both must stop**; (3) **a 27B model cannot be released in-process** (`del`+`gc`+`empty_cache` and frame-exit both left 45,287 MiB free; only process exit gave the card back) → one process per model, and the old residency gate read `hf_device_map`, which is empty on single-device loads and therefore **could never fail**. → playbook §3.12. Full saga → `persistent-memory.d/2026-08-20-coldfusion-abliteration-landed.md`. +- **🟢 COLD-FUSION ABLITERATION — LANDED 2026-08-20 (the real work; abliterated model WORKS, verify + quant still owed).** Abliterated `DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1` with the **Robinson formula** (`docs/pfi/abliteration-recipe-qwen38.md`; harness `services/coldfusion-abliteration/`). Output `ana-ml2:/tank/aimodels/qwen38-27b-coldfusion-abliterated-L35-bf16`; bitwise-verified 131/131 targets changed, 333/333 vision byte-identical, 735/735 others untouched. A/B vs stock: explicit-sexual + graphic-torture (the measured stock refusal surface) go refused→complied, self-harm guardrail survives, coherence intact — the Robinson design point. **THREE first-session diagnoses were wrong, all corrected:** (1) **layer selection by two-template |cos| agreement is misleading on a merged base** — its pick (L18) was the *worst*-separating layer and abliterating there was a measured no-op; replaced with harmful/harmless **separation** (Cohen's d/AUC) gated on the sink screen → **L35** (d9.35, AUC0.9997, sink0.094%); (2) **"bf16 NaNs, use fp32" was a misdiagnosis** — the NaN was multi-GPU sharding + `expandable_segments`, not precision; bf16 on ONE GPU is deterministic+coherent at 50 GB, 4.3× faster (now gated, `CUDA_VISIBLE_DEVICES=0`); (3) **corpus-size hypothesis falsified** (8→416 moved agreement 0.594→0.624, nothing). Write is now **shard surgery** (no model object — `AutoModelForCausalLM` is text-only and would drop vision + skip the in-band MTP edit). **🎯 THESIS PROVEN 2026-08-20 (same session, later):** quantized L35 → mixed NVFP4 (`/tank/aimodels/qwen38-27b-coldfusion-L35-nvfp4-mixed`, 22.5 GB) and measured **MTP acceptance 59.1% median** (51–65%) — clears the ≳40% gate AND **beats the incumbent Heretic seat's ~47%.** So Robinson's **in-band** MTP abliteration accepts BETTER than Heretic's byte-identical graft — the additive delta the experiment tested, positive. Abliteration survives the quant (creative refusals drop, self-harm guardrail intact, coherent); decode 118.7 tok/s (faster, image-confounded). Env foot-guns hardened: quant venv drifted (transformers 5.10/llmcompressor 0.12 no longer delegate `num_attention_heads` to text_config → promote-then-restore in `quant_mixed_nvfp4.py`); a <23GB quant saves single-file no-index → build index from the safetensors header (never `safe_open`, ENOMEMs on ZFS). Commit `725c8fd`. **⚠ NOT CUT OVER** — incumbent gen seat untouched; making L35 the `gen` seat is a **separate operator decision** needing the full Stage-3 gate (PPL, prefill, surface 6/6, refusal-probe battery) + real multi-turn hold. **Do NOT delete** `qwen38-27b-heresy-nvfp4-mixed`. **📐 KL MEASURED 2026-08-20 (third session) — the surgery is SELECTIVE.** `kl_divergence.py` (new, beside the harness): first-token KL(stock‖L35) over the full 248,320 vocab, bf16-vs-bf16, held-out prompts. **Answer mode: harmless median 0.0211 / mean 0.0364, harmful median 0.5996 → 28.4× selectivity** (think mode 0.0042 / 0.3068 → 72.8×); top-1 agreement on benign prompts stays **89.8%**; **self-KL noise floor exactly 0.0**, so every digit is signal. Reverse KL on harmful is 1.43 vs forward 0.70 — the mass-where-stock-had-none asymmetry that is abliteration's signature. Vs the Heretic reference figures (0.1191 prior seat, **0.0759 the live `absolute-heresy` seat**) ours looks materially gentler, but ⚠️ **NOT a head-to-head** — those are Heretic's own optimizer output on a different base with its own harmless set/template; a real comparison needs the incumbent re-measured through this script (one more window). KL stays a **fidelity** number, not the gate (that's MTP acceptance, 59.1%). Run cost **2m40s**, both GPU0 seats down. **Three durable process lessons:** (1) report abliteration KL **split by prompt class** — a mixed average is meaningless since the metric is meant to be big on harmful and small on benign; (2) the runbook's "bf16 is 50 GB / only gen must go" was **50.10 GiB mislabelled** — text weights are 51,300 MiB and freeing either GPU0 seat alone leaves ~50,933, so **both must stop**; (3) **a 27B model cannot be released in-process** (`del`+`gc`+`empty_cache` and frame-exit both left 45,287 MiB free; only process exit gave the card back) → one process per model, and the old residency gate read `hf_device_map`, which is empty on single-device loads and therefore **could never fail**. → playbook §3.12. **⚠️ GPU0 SEAT RESTORE ORDER IS LOAD-BEARING and "first" means HEALTHY, not 10s earlier** — `start meromero; sleep 10; start gen` put meromero in a **7-restart crash-loop** (`Free memory on cuda:0 (35.3/94.97 GiB) < desired utilization (0.52, 49.38 GiB)`). I had checked the compose files, seen `--gpu-memory-utilization` is a fraction of **total**, and wrongly concluded order didn't matter — the fraction sets the *target* but vLLM **gates startup on FREE VRAM**, so at ~96.4/97.9 GB the seats coexist only in their original boot order. The old auto-memory note was right; my correction of it was the error. Recovery = stop gen → meromero healthy → start gen. **Restore VERIFIED against the pre-window baseline** (gen KV 14.34 GiB/1.53× vs 14.36/1.54×; meromero 542,202 tok both; restarts=0; summarizer smoke-tested through LiteLLM) — ⚠️ don't use raw `nvidia-smi` used-MiB for this check, it reads 89,503 vs 96,376 and that 6.9 GB is allocator slack, not lost capacity. Full saga → `persistent-memory.d/2026-08-20-coldfusion-abliteration-landed.md`. - **🟢 esh-pve-nas — MIGRATION DONE 2026-08-18. Root is `nvme/ROOT/pve-1` on mirrored NVMe; the USB DOM is out of the runtime I/O path.** All five guests healthy, three pools ONLINE, system `running`, ext4 `pve-root` intact+unmounted as rollback with its own kernel. Boot config: `saved_entry=pve-zfs-root`, no `next_entry`; if grubenv were unreadable GRUB falls to entry 0 which the `/etc/default/grub.d/zfs-root.cfg` drop-in also points at ZFS. **`zfs-import-cache.service` is now the active import path** (the all-three-pools cachefile fix working as intended); vestigial `zfs-import@nvme.service` disabled — it failed every boot as redundant. ⚠ **Device letters shift across reboots** (DOM was `sdq`, now `sdl`) — never key anything to a bare `sdX` here. ⚠ **NO auto-fallback on a failed boot, and no IPMI/BMC/serial** — grubenv on LVM is readable but not writable by GRUB, so `grub-reboot`'s one-shot degrades to a sticky default (verified: `next_entry` survived the boot that consumed it). Recovery = pick the ROLLBACK entry at the console. **PATCHED 2026-08-18: 225 packages installed, pve-manager 8.4.11 -> 8.4.20, corosync 3.1.9 -> 3.1.10-pve2, kernel 6.8.12-42 staged on the /boot LV. dpkg clean, no unapplied conffiles, cluster quorate, 6/6 verify. ⏳ REBOOT DEFERRED at operator request — host still runs 6.8.12-13 until a chosen window; `GRUB_DEFAULT=0` means entry 0 is already the -42 entry with the correct `root=ZFS=nvme/ROOT/pve-1`, so the reboot is the only remaining step. Rollback for the upgrade is the ZFS snapshot `nvme/ROOT/pve-1@pre-upgrade-20260818T141652Z` (409M) — `zfs rollback -r && reboot`. Second confirmation reboot ALREADY DONE (2026-08-18, booted ZFS from GRUB_DEFAULT=0 with no one-shot). ⏳ Still outstanding: refresh the off-box DOM image, since `/boot` changed.** esh-pve is FULLY done (8.4.20 + kernel 6.8.12-42 + corosync 3.1.10, rebooted, quorate). ⚠⚠ **THE WINDOW COST AN UNPLANNED OUTAGE, caused by our own tooling, not the migration.** The staging chroot did `mount --rbind /dev` + `/sys` with **no `--make-rslave`**; on systemd `/` is *shared*, so the cutover's `umount -R` **propagated back into the live host** and stripped the real `/sys/fs/cgroup`, `/dev/pts`, `/dev/shm`. logind could then create no sessions: ping fine, TCP fine, **SSH authenticates**, resident daemons keep serving (pveproxy returned clean 401s) — and **every new exec hangs, including `/sbin/reboot`**, so the reboot never ran. **It is a near-perfect impostor of failing root-disk I/O**, and I misdiagnosed it as the USB DOM dying and told the operator to walk to the machine. **Operator caught it** — the DOM had been fine for years and the wedge began right after a change. The settling evidence was in `dmesg` all along: `[16.00] [sdq] Attached SCSI removable disk` (clean, no errors) and a last-line timestamp of **`[12114881]` = 140 days = the ORIGINAL boot** — the machine had never rebooted. My down-detector never once reported the host down and I read that as a fast reboot rather than *no* reboot. Recovered with **no console access** by hammering an idempotent cgroup2/devpts/shm remount into the brief windows where exec succeeded. Zero data loss. **RULES: (1) always `--make-rslave` after `--rbind` (playbook now guards on `PROPAGATION != shared`); (2) a reboot is not confirmed until the host is observed DOWN — poll for disappearance, not reappearance; (3) before blaming hardware for a wedge that started right after a change, get `dmesg` and check the boot timestamp.** diff --git a/services/coldfusion-abliteration/README.md b/services/coldfusion-abliteration/README.md index 64ed446..771345b 100644 --- a/services/coldfusion-abliteration/README.md +++ b/services/coldfusion-abliteration/README.md @@ -444,12 +444,28 @@ aborted window did exactly that and cost nothing but two minutes. (`--capture-dtype float32` remains as an escape hatch; it needs 111 GB, so it also needs `--max-layer 46` to fit on one card. The two agree to 0.0005, so there -is no reason to reach for it.) **Restore after:** start `vllm-meromero-rp` -**first**, then `vllm-gen`. (The stated reason — "gen grabs a fraction of *free* -VRAM" — is not what the configs do: both seats pass `--gpu-memory-utilization` as -a fraction of **total** (`MEROMERO_GPU_MEM_UTIL=0.52`, `GEN_GPU_MEM_UTIL=0.43`), -so restore order is not actually load-bearing. Kept as the runbook order anyway; -it costs nothing.) +is no reason to reach for it.) + +**⚠️ Restore after: start `vllm-meromero-rp` first, and WAIT FOR IT TO GO HEALTHY +before starting `vllm-gen`.** This ordering is load-bearing, and "first" means +*fully up*, not *ten seconds earlier* — a `docker start meromero; sleep 10; docker +start gen` put meromero into a **7-restart crash-loop** on 2026-08-20, because gen +finished claiming the card while meromero was still loading weights: + +``` +ValueError: Free memory on device cuda:0 (35.3/94.97 GiB) on startup is less than +desired GPU memory utilization (0.52, 49.38 GiB). +``` + +The mechanism, stated precisely because a half-right version of it is what caused +the mistake: `--gpu-memory-utilization` is a fraction of **total** VRAM (0.52 × +94.97 = 49.38 GiB is meromero's *target*), but vLLM gates startup on **free** VRAM +— it refuses to start unless the card currently has the whole target available. +So the two seats coexist only in the order they were originally brought up. GPU0 +runs at ~96.4/97.9 GB with about 0.4 GiB of slack; whichever seat starts second +gets whatever the first one left, and meromero is the one that does not fit in the +remainder. Recovery when it does happen: `docker stop vllm-gen`, wait for meromero +to report `healthy`, then `docker start vllm-gen`. **4. fla is irrelevant here — but harmless.** `fla` + `einops` are `--target` -installed to `/tank/aimodels/coldfusion-abliteration/pylibs` and reached via