diff --git a/persistent-memory.d/2026-08-20-coldfusion-abliteration-landed.md b/persistent-memory.d/2026-08-20-coldfusion-abliteration-landed.md index 229e180..5098435 100644 --- a/persistent-memory.d/2026-08-20-coldfusion-abliteration-landed.md +++ b/persistent-memory.d/2026-08-20-coldfusion-abliteration-landed.md @@ -101,6 +101,52 @@ means re-measuring the incumbent through this script (one more GPU window). Consistent with [[reference_abliteration_mtp_lessons]]: KL is a **fidelity** number here, not the viability gate — that remains MTP acceptance (59.1%). +### ⚠️ The restore bit me — GPU0 seat order is load-bearing, and "first" means *healthy* + +Restoring with `docker start meromero; sleep 10; docker start gen` put **meromero +into a 7-restart crash-loop**: gen finished claiming the card while meromero was +still loading weights, and meromero died on + +``` +ValueError: Free memory on device cuda:0 (35.3/94.97 GiB) on startup is less than +desired GPU memory utilization (0.52, 49.38 GiB). +``` + +**I had this half-right and the half I got wrong is what caused it.** I checked the +compose files, saw `--gpu-memory-utilization` is a fraction of **total** VRAM, and +concluded restore order "is not actually load-bearing" — I even wrote that into the +README before the seats came back. Wrong: the fraction sets the *target*, but vLLM +gates startup on **free** VRAM, refusing to start unless the whole target is +available right now. GPU0 runs at ~96.4/97.9 GB with ~0.4 GiB of slack, so the two +seats coexist **only in the order they were originally brought up**, and meromero +is the one that does not fit in the remainder. The old auto-memory note ("gen takes +a fraction of FREE VRAM at startup and will starve meromero") was pointing at a +real effect; my correction of it was the error. + +Recovery: `docker stop vllm-gen` → wait for meromero `healthy` → `docker start +vllm-gen`. Sequence-and-verify, not sequence-and-sleep — a `sleep 10` against a +2-3 minute weight load is simultaneity, not ordering. +(Generalises [[feedback_confirm_reboot_by_observing_down]]: gate on the observed +state, not on elapsed time.) + +**Restore verified against the pre-window baseline, not just "it's green."** Both +seats `healthy`, `RestartCount=0`, and — the check that actually matters — the KV +pools match what they were before the session: + +| | pre-window (18:32) | after restore (20:06) | +|---|---|---| +| gen KV | 14.36 GiB, 403,065 tok, **1.54×** | 14.34 GiB, 401,550 tok, **1.53×** | +| meromero KV | 542,202 tok | 542,202 tok | + +⚠️ **Do not read raw `nvidia-smi` used-MiB as the restore check.** GPU0 shows +89,503 MiB used now vs 96,376 before, which looks like a 6.9 GB regression and is +not one — the delta is allocator slack, and serving capacity (KV pool, max +concurrency) is identical. The genuinely anomalous boots were the *high* ones +(34.95 GiB KV at 19:50/19:55/20:00), where gen came up on an empty card mid-window +and grabbed more than its steady-state share. Card now sits at 7,746 MiB free vs +~1,500 before, which is more co-tenancy slack, not less. Summarizer smoke-tested +end-to-end through LiteLLM after the restore. + ### Three durable process lessons from the measurement 1. **★ Report abliteration KL SPLIT BY PROMPT CLASS.** A single averaged KL over a diff --git a/persistent-memory.md b/persistent-memory.md index 5db926a..f5c7963 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -150,7 +150,7 @@ _As of 2026-08-20 — continued from the 08-19 infra session (DNS/.internal, wat - **🟢 WT #401 (fd-leak deadlock) CLOSED 2026-08-17 — one ping still owed.** worldtree-dev closed it on our demo verify. Layers: **(a) their `e41b139`** pins `ulimits: nofile 65536/65536` in the worldtree compose anchor — **demo VERIFIED** (api + matrix recreated 22:55:34Z, `ulimit -Sn`=65536); **personal/pinned are covered-not-verified**, they inherit at their next promotion/recreate. **(b) our host floor is STAGED, NOT ACTIVE** — `/etc/docker/daemon.json` on corviduo-dev carries `default-ulimits nofile 65536/65536` but **`default-ulimits` is NOT SIGHUP-reloadable** (measured on 29.4.3: post-reload the daemon's own "Reloaded configuration" log omits it and a fresh container still reports 1024). Activation needs a full dockerd restart = bounces all 13 containers; **worldtree-dev explicitly does NOT want one**, and `live-restore:true`-then-restart is PARKED as a separate host-side improvement for the operator to rule on, never folded into #401. Playbook `playbooks/corviduo-dev-docker-default-ulimits.yaml` (verify step 3 fails BY DESIGN until a restart). Hourly fd tripwire on corviduo-dev stays armed. **⏳ OWED: ping worldtree-dev in thread `01M08QQ655XD6VKEV7MA9GX0NS` once worldtree-personal recreates and 65536 is confirmed there.** Commit `7f3f265`. -- **🟢 COLD-FUSION ABLITERATION — LANDED 2026-08-20 (the real work; abliterated model WORKS, verify + quant still owed).** Abliterated `DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1` with the **Robinson formula** (`docs/pfi/abliteration-recipe-qwen38.md`; harness `services/coldfusion-abliteration/`). Output `ana-ml2:/tank/aimodels/qwen38-27b-coldfusion-abliterated-L35-bf16`; bitwise-verified 131/131 targets changed, 333/333 vision byte-identical, 735/735 others untouched. A/B vs stock: explicit-sexual + graphic-torture (the measured stock refusal surface) go refused→complied, self-harm guardrail survives, coherence intact — the Robinson design point. **THREE first-session diagnoses were wrong, all corrected:** (1) **layer selection by two-template |cos| agreement is misleading on a merged base** — its pick (L18) was the *worst*-separating layer and abliterating there was a measured no-op; replaced with harmful/harmless **separation** (Cohen's d/AUC) gated on the sink screen → **L35** (d9.35, AUC0.9997, sink0.094%); (2) **"bf16 NaNs, use fp32" was a misdiagnosis** — the NaN was multi-GPU sharding + `expandable_segments`, not precision; bf16 on ONE GPU is deterministic+coherent at 50 GB, 4.3× faster (now gated, `CUDA_VISIBLE_DEVICES=0`); (3) **corpus-size hypothesis falsified** (8→416 moved agreement 0.594→0.624, nothing). Write is now **shard surgery** (no model object — `AutoModelForCausalLM` is text-only and would drop vision + skip the in-band MTP edit). **🎯 THESIS PROVEN 2026-08-20 (same session, later):** quantized L35 → mixed NVFP4 (`/tank/aimodels/qwen38-27b-coldfusion-L35-nvfp4-mixed`, 22.5 GB) and measured **MTP acceptance 59.1% median** (51–65%) — clears the ≳40% gate AND **beats the incumbent Heretic seat's ~47%.** So Robinson's **in-band** MTP abliteration accepts BETTER than Heretic's byte-identical graft — the additive delta the experiment tested, positive. Abliteration survives the quant (creative refusals drop, self-harm guardrail intact, coherent); decode 118.7 tok/s (faster, image-confounded). Env foot-guns hardened: quant venv drifted (transformers 5.10/llmcompressor 0.12 no longer delegate `num_attention_heads` to text_config → promote-then-restore in `quant_mixed_nvfp4.py`); a <23GB quant saves single-file no-index → build index from the safetensors header (never `safe_open`, ENOMEMs on ZFS). Commit `725c8fd`. **⚠ NOT CUT OVER** — incumbent gen seat untouched; making L35 the `gen` seat is a **separate operator decision** needing the full Stage-3 gate (PPL, prefill, surface 6/6, refusal-probe battery) + real multi-turn hold. **Do NOT delete** `qwen38-27b-heresy-nvfp4-mixed`. **📐 KL MEASURED 2026-08-20 (third session) — the surgery is SELECTIVE.** `kl_divergence.py` (new, beside the harness): first-token KL(stock‖L35) over the full 248,320 vocab, bf16-vs-bf16, held-out prompts. **Answer mode: harmless median 0.0211 / mean 0.0364, harmful median 0.5996 → 28.4× selectivity** (think mode 0.0042 / 0.3068 → 72.8×); top-1 agreement on benign prompts stays **89.8%**; **self-KL noise floor exactly 0.0**, so every digit is signal. Reverse KL on harmful is 1.43 vs forward 0.70 — the mass-where-stock-had-none asymmetry that is abliteration's signature. Vs the Heretic reference figures (0.1191 prior seat, **0.0759 the live `absolute-heresy` seat**) ours looks materially gentler, but ⚠️ **NOT a head-to-head** — those are Heretic's own optimizer output on a different base with its own harmless set/template; a real comparison needs the incumbent re-measured through this script (one more window). KL stays a **fidelity** number, not the gate (that's MTP acceptance, 59.1%). Run cost **2m40s**, both GPU0 seats down. **Three durable process lessons:** (1) report abliteration KL **split by prompt class** — a mixed average is meaningless since the metric is meant to be big on harmful and small on benign; (2) the runbook's "bf16 is 50 GB / only gen must go" was **50.10 GiB mislabelled** — text weights are 51,300 MiB and freeing either GPU0 seat alone leaves ~50,933, so **both must stop**; (3) **a 27B model cannot be released in-process** (`del`+`gc`+`empty_cache` and frame-exit both left 45,287 MiB free; only process exit gave the card back) → one process per model, and the old residency gate read `hf_device_map`, which is empty on single-device loads and therefore **could never fail**. → playbook §3.12. Full saga → `persistent-memory.d/2026-08-20-coldfusion-abliteration-landed.md`. +- **🟢 COLD-FUSION ABLITERATION — LANDED 2026-08-20 (the real work; abliterated model WORKS, verify + quant still owed).** Abliterated `DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1` with the **Robinson formula** (`docs/pfi/abliteration-recipe-qwen38.md`; harness `services/coldfusion-abliteration/`). Output `ana-ml2:/tank/aimodels/qwen38-27b-coldfusion-abliterated-L35-bf16`; bitwise-verified 131/131 targets changed, 333/333 vision byte-identical, 735/735 others untouched. A/B vs stock: explicit-sexual + graphic-torture (the measured stock refusal surface) go refused→complied, self-harm guardrail survives, coherence intact — the Robinson design point. **THREE first-session diagnoses were wrong, all corrected:** (1) **layer selection by two-template |cos| agreement is misleading on a merged base** — its pick (L18) was the *worst*-separating layer and abliterating there was a measured no-op; replaced with harmful/harmless **separation** (Cohen's d/AUC) gated on the sink screen → **L35** (d9.35, AUC0.9997, sink0.094%); (2) **"bf16 NaNs, use fp32" was a misdiagnosis** — the NaN was multi-GPU sharding + `expandable_segments`, not precision; bf16 on ONE GPU is deterministic+coherent at 50 GB, 4.3× faster (now gated, `CUDA_VISIBLE_DEVICES=0`); (3) **corpus-size hypothesis falsified** (8→416 moved agreement 0.594→0.624, nothing). Write is now **shard surgery** (no model object — `AutoModelForCausalLM` is text-only and would drop vision + skip the in-band MTP edit). **🎯 THESIS PROVEN 2026-08-20 (same session, later):** quantized L35 → mixed NVFP4 (`/tank/aimodels/qwen38-27b-coldfusion-L35-nvfp4-mixed`, 22.5 GB) and measured **MTP acceptance 59.1% median** (51–65%) — clears the ≳40% gate AND **beats the incumbent Heretic seat's ~47%.** So Robinson's **in-band** MTP abliteration accepts BETTER than Heretic's byte-identical graft — the additive delta the experiment tested, positive. Abliteration survives the quant (creative refusals drop, self-harm guardrail intact, coherent); decode 118.7 tok/s (faster, image-confounded). Env foot-guns hardened: quant venv drifted (transformers 5.10/llmcompressor 0.12 no longer delegate `num_attention_heads` to text_config → promote-then-restore in `quant_mixed_nvfp4.py`); a <23GB quant saves single-file no-index → build index from the safetensors header (never `safe_open`, ENOMEMs on ZFS). Commit `725c8fd`. **⚠ NOT CUT OVER** — incumbent gen seat untouched; making L35 the `gen` seat is a **separate operator decision** needing the full Stage-3 gate (PPL, prefill, surface 6/6, refusal-probe battery) + real multi-turn hold. **Do NOT delete** `qwen38-27b-heresy-nvfp4-mixed`. **📐 KL MEASURED 2026-08-20 (third session) — the surgery is SELECTIVE.** `kl_divergence.py` (new, beside the harness): first-token KL(stock‖L35) over the full 248,320 vocab, bf16-vs-bf16, held-out prompts. **Answer mode: harmless median 0.0211 / mean 0.0364, harmful median 0.5996 → 28.4× selectivity** (think mode 0.0042 / 0.3068 → 72.8×); top-1 agreement on benign prompts stays **89.8%**; **self-KL noise floor exactly 0.0**, so every digit is signal. Reverse KL on harmful is 1.43 vs forward 0.70 — the mass-where-stock-had-none asymmetry that is abliteration's signature. Vs the Heretic reference figures (0.1191 prior seat, **0.0759 the live `absolute-heresy` seat**) ours looks materially gentler, but ⚠️ **NOT a head-to-head** — those are Heretic's own optimizer output on a different base with its own harmless set/template; a real comparison needs the incumbent re-measured through this script (one more window). KL stays a **fidelity** number, not the gate (that's MTP acceptance, 59.1%). Run cost **2m40s**, both GPU0 seats down. **Three durable process lessons:** (1) report abliteration KL **split by prompt class** — a mixed average is meaningless since the metric is meant to be big on harmful and small on benign; (2) the runbook's "bf16 is 50 GB / only gen must go" was **50.10 GiB mislabelled** — text weights are 51,300 MiB and freeing either GPU0 seat alone leaves ~50,933, so **both must stop**; (3) **a 27B model cannot be released in-process** (`del`+`gc`+`empty_cache` and frame-exit both left 45,287 MiB free; only process exit gave the card back) → one process per model, and the old residency gate read `hf_device_map`, which is empty on single-device loads and therefore **could never fail**. → playbook §3.12. **⚠️ GPU0 SEAT RESTORE ORDER IS LOAD-BEARING and "first" means HEALTHY, not 10s earlier** — `start meromero; sleep 10; start gen` put meromero in a **7-restart crash-loop** (`Free memory on cuda:0 (35.3/94.97 GiB) < desired utilization (0.52, 49.38 GiB)`). I had checked the compose files, seen `--gpu-memory-utilization` is a fraction of **total**, and wrongly concluded order didn't matter — the fraction sets the *target* but vLLM **gates startup on FREE VRAM**, so at ~96.4/97.9 GB the seats coexist only in their original boot order. The old auto-memory note was right; my correction of it was the error. Recovery = stop gen → meromero healthy → start gen. **Restore VERIFIED against the pre-window baseline** (gen KV 14.34 GiB/1.53× vs 14.36/1.54×; meromero 542,202 tok both; restarts=0; summarizer smoke-tested through LiteLLM) — ⚠️ don't use raw `nvidia-smi` used-MiB for this check, it reads 89,503 vs 96,376 and that 6.9 GB is allocator slack, not lost capacity. Full saga → `persistent-memory.d/2026-08-20-coldfusion-abliteration-landed.md`. - **🟢 esh-pve-nas — MIGRATION DONE 2026-08-18. Root is `nvme/ROOT/pve-1` on mirrored NVMe; the USB DOM is out of the runtime I/O path.** All five guests healthy, three pools ONLINE, system `running`, ext4 `pve-root` intact+unmounted as rollback with its own kernel. Boot config: `saved_entry=pve-zfs-root`, no `next_entry`; if grubenv were unreadable GRUB falls to entry 0 which the `/etc/default/grub.d/zfs-root.cfg` drop-in also points at ZFS. **`zfs-import-cache.service` is now the active import path** (the all-three-pools cachefile fix working as intended); vestigial `zfs-import@nvme.service` disabled — it failed every boot as redundant. ⚠ **Device letters shift across reboots** (DOM was `sdq`, now `sdl`) — never key anything to a bare `sdX` here. ⚠ **NO auto-fallback on a failed boot, and no IPMI/BMC/serial** — grubenv on LVM is readable but not writable by GRUB, so `grub-reboot`'s one-shot degrades to a sticky default (verified: `next_entry` survived the boot that consumed it). Recovery = pick the ROLLBACK entry at the console. **PATCHED 2026-08-18: 225 packages installed, pve-manager 8.4.11 -> 8.4.20, corosync 3.1.9 -> 3.1.10-pve2, kernel 6.8.12-42 staged on the /boot LV. dpkg clean, no unapplied conffiles, cluster quorate, 6/6 verify. ⏳ REBOOT DEFERRED at operator request — host still runs 6.8.12-13 until a chosen window; `GRUB_DEFAULT=0` means entry 0 is already the -42 entry with the correct `root=ZFS=nvme/ROOT/pve-1`, so the reboot is the only remaining step. Rollback for the upgrade is the ZFS snapshot `nvme/ROOT/pve-1@pre-upgrade-20260818T141652Z` (409M) — `zfs rollback -r && reboot`. Second confirmation reboot ALREADY DONE (2026-08-18, booted ZFS from GRUB_DEFAULT=0 with no one-shot). ⏳ Still outstanding: refresh the off-box DOM image, since `/boot` changed.** esh-pve is FULLY done (8.4.20 + kernel 6.8.12-42 + corosync 3.1.10, rebooted, quorate). ⚠⚠ **THE WINDOW COST AN UNPLANNED OUTAGE, caused by our own tooling, not the migration.** The staging chroot did `mount --rbind /dev` + `/sys` with **no `--make-rslave`**; on systemd `/` is *shared*, so the cutover's `umount -R` **propagated back into the live host** and stripped the real `/sys/fs/cgroup`, `/dev/pts`, `/dev/shm`. logind could then create no sessions: ping fine, TCP fine, **SSH authenticates**, resident daemons keep serving (pveproxy returned clean 401s) — and **every new exec hangs, including `/sbin/reboot`**, so the reboot never ran. **It is a near-perfect impostor of failing root-disk I/O**, and I misdiagnosed it as the USB DOM dying and told the operator to walk to the machine. **Operator caught it** — the DOM had been fine for years and the wedge began right after a change. The settling evidence was in `dmesg` all along: `[16.00] [sdq] Attached SCSI removable disk` (clean, no errors) and a last-line timestamp of **`[12114881]` = 140 days = the ORIGINAL boot** — the machine had never rebooted. My down-detector never once reported the host down and I read that as a fast reboot rather than *no* reboot. Recovered with **no console access** by hammering an idempotent cgroup2/devpts/shm remount into the brief windows where exec succeeded. Zero data loss. **RULES: (1) always `--make-rslave` after `--rbind` (playbook now guards on `PROPAGATION != shared`); (2) a reboot is not confirmed until the host is observed DOWN — poll for disappearance, not reappearance; (3) before blaming hardware for a wedge that started right after a change, get `dmesg` and check the boot timestamp.** diff --git a/services/coldfusion-abliteration/README.md b/services/coldfusion-abliteration/README.md index 64ed446..771345b 100644 --- a/services/coldfusion-abliteration/README.md +++ b/services/coldfusion-abliteration/README.md @@ -444,12 +444,28 @@ aborted window did exactly that and cost nothing but two minutes. (`--capture-dtype float32` remains as an escape hatch; it needs 111 GB, so it also needs `--max-layer 46` to fit on one card. The two agree to 0.0005, so there -is no reason to reach for it.) **Restore after:** start `vllm-meromero-rp` -**first**, then `vllm-gen`. (The stated reason — "gen grabs a fraction of *free* -VRAM" — is not what the configs do: both seats pass `--gpu-memory-utilization` as -a fraction of **total** (`MEROMERO_GPU_MEM_UTIL=0.52`, `GEN_GPU_MEM_UTIL=0.43`), -so restore order is not actually load-bearing. Kept as the runbook order anyway; -it costs nothing.) +is no reason to reach for it.) + +**⚠️ Restore after: start `vllm-meromero-rp` first, and WAIT FOR IT TO GO HEALTHY +before starting `vllm-gen`.** This ordering is load-bearing, and "first" means +*fully up*, not *ten seconds earlier* — a `docker start meromero; sleep 10; docker +start gen` put meromero into a **7-restart crash-loop** on 2026-08-20, because gen +finished claiming the card while meromero was still loading weights: + +``` +ValueError: Free memory on device cuda:0 (35.3/94.97 GiB) on startup is less than +desired GPU memory utilization (0.52, 49.38 GiB). +``` + +The mechanism, stated precisely because a half-right version of it is what caused +the mistake: `--gpu-memory-utilization` is a fraction of **total** VRAM (0.52 × +94.97 = 49.38 GiB is meromero's *target*), but vLLM gates startup on **free** VRAM +— it refuses to start unless the card currently has the whole target available. +So the two seats coexist only in the order they were originally brought up. GPU0 +runs at ~96.4/97.9 GB with about 0.4 GiB of slack; whichever seat starts second +gets whatever the first one left, and meromero is the one that does not fit in the +remainder. Recovery when it does happen: `docker stop vllm-gen`, wait for meromero +to report `healthy`, then `docker start vllm-gen`. **4. fla is irrelevant here — but harmless.** `fla` + `einops` are `--target` -installed to `/tank/aimodels/coldfusion-abliteration/pylibs` and reached via