From 08c852792b3ee427937ba7879e535fb8e8d1ba76 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Wed, 9 Sep 2026 23:09:13 -0700 Subject: [PATCH] chore(erp-tune): purge superseded merged models (~499 GB), keep run 6 and the v6 quant Operator: "purge the merged models, keep run06 and the v6 quant." Eleven merges removed with literal paths, one rm per line. pfi-gx10 merged-run03c, merged-run04, merged-run05 3 x 49 GiB -> 145 GB ana-ml2 merged-final, merged-run02, merged-run03, merged-run03-s{025,050,075}, merged-test, nvfp4a16-test 8 dirs -> 354 GB gx10 382G->237G used, 632 GB free. /tank/erp-tune/serve 354G->8.6M with no snapshots holding the blocks; the pool's raw FREE moved 4.80T->5.30T. Combined with the earlier checkpoint purge, ~573 GB reclaimed tonight. The check that made this safe: docker inspect on the live vllm-erp-seat shows it binds /tank/aimodels ONLY, with model arg /tank/aimodels/erp-tune-v6-nvfp4a16, so /tank/erp-tune/serve was never in the serving path. Also confirmed no container mounts that tree and no process held a file open under it, reading /proc/*/fd and /proc/*/maps rather than trusting an empty lsof. All eleven were run-1/2/3-era Gemma-4 merges dated Aug 24-26; the keeper is dated Sep 8 and lives in a different tree, so there was no ambiguity about which was which. Kept and re-verified after the deletion: gx10 serve/merged-run06, /tank/aimodels/erp-tune-v6-{bf16,nvfp4a16}, all eight adapters sha256-unchanged, and the merge/quant tooling and logs under /tank/erp-tune/serve (8.6 MB) that document how the artifacts were built. The live seat never bounced -- Pfish-6 answered a real completion after the delete with finish_reason stop and correct text, container still healthy at 4 h uptime. relaunch-trial-seat.sh now names a deleted model; it was banner-marked RETIRED rather than removed, because its flags carry the FlashInfer JIT/PATH trap and the gpu-clear / never-pkill notes. --- docs/pfi/author-voice-lora-regime.md | 39 +++++++++++++++++++++++----- persistent-memory.md | 24 +++++++++++++---- 2 files changed, 52 insertions(+), 11 deletions(-) diff --git a/docs/pfi/author-voice-lora-regime.md b/docs/pfi/author-voice-lora-regime.md index bfb4927..f1243d5 100644 --- a/docs/pfi/author-voice-lora-regime.md +++ b/docs/pfi/author-voice-lora-regime.md @@ -261,12 +261,39 @@ snapshot` empty so the space is genuinely returned rather than snapshot-held. **All eight adapters re-verified by `sha256` after the deletion**, matching the values recorded during the mirror. -⏳ **Still standing, and it is the bigger prize — ~550 GB of superseded MERGED -models, deliberately NOT touched** because they are not checkpoints: -`gx10:~/erp-tune/serve/merged-run{03c,04,05,06}` at 49 GiB each (196 GiB) and -`ana-ml2:/tank/erp-tune/serve` (~354 GB). Run 6 is the standing seat, so -`merged-run06` plus `/tank/aimodels/erp-tune-v6-{bf16,nvfp4a16}` are the rollback -path and should survive; `merged-run03c/04/05` are dead weight. Separate call. +**And the merged models followed at 23:08 PT**, operator: _"purge the merged +models, keep run06 and the v6 quant."_ Eleven merges, literal paths, one `rm` per +line, after proving what the live seat reads: `docker inspect vllm-erp-seat` shows +it binds **`/tank/aimodels` only** and its model arg is +`/tank/aimodels/erp-tune-v6-nvfp4a16`, so `/tank/erp-tune/serve` was never in the +serving path at all. No container mounts it and no process held a file open under +it (checked `/proc/*/fd` and `/proc/*/maps`, not `lsof` alone). + + pfi-gx10 merged-run03c · merged-run04 · merged-run05 3 x 49 GiB -> 145 GB + ana-ml2 merged-final · merged-run02 · merged-run03 · + merged-run03-s{025,050,075} · merged-test · + nvfp4a16-test 8 dirs -> 354 GB + +All eleven were Gemma-4 merges from the run-1/2/3 era (Aug 24–26); the keeper is +dated Sep 8 and lives in a different tree, so there was no ambiguity about which +was which. gx10 382G→237G used (**632 GB free**); `/tank/erp-tune/serve` 354G→8.6M +with `zfs list -t snapshot` empty, and the pool's raw FREE moved 4.80T→5.30T. + +**Kept, and verified after the deletion:** `gx10:~/erp-tune/serve/merged-run06` +(49 GiB, 9 files), `/tank/aimodels/erp-tune-v6-bf16` (49 G) and +`erp-tune-v6-nvfp4a16` (16 G, 9 files incl. `recipe.yaml`), all eight adapters +sha256-unchanged, and `/tank/erp-tune/serve`'s merge/quant tooling and logs +(`merge_lora.py`, `quant_nvfp4a16.py`, the dry-run logs, `base-arm`, `lora-scales` +— 8.6 MB total, the provenance for how every artifact above was built). + +⚠ **`gx10:~/erp-tune/relaunch-trial-seat.sh` now names a model that is gone.** It +was not deleted — its flags carry the FlashInfer JIT/PATH trap and the +gpu-clear/never-pkill notes, each bought with a real failure — but it gained a +RETIRED banner so nobody hits a confusing missing-model error later. + +**The live seat never bounced.** After the deletion `Pfish-6` answered a real +completion (`finish_reason: stop`, 5 tokens, correct text), container still +`Up 4 hours (healthy)`. ## 7. Adapter disposition — settled, and made real diff --git a/persistent-memory.md b/persistent-memory.md index 822c2a7..ce1bce8 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-09 23:10 PT (**Pfish-6** = run-6 NVFP4 is the standing seat, ana-ml2 :8021 ONLY; run 7 PURGED ~139 GiB; pfi-gx10 is an experimental/TRAINING box and carries no serving seat; all five ERP adapters now MIRRORED to ana-ml2 and inside restic; **BabyBronte / R49 author-voice regime is in PREP on gx10, carrier SETTLED = dense Qwen3**; intermediate checkpoints PURGED both boxes ~74 GB)_ +_Last updated: 2026-09-09 23:15 PT (**Pfish-6** = run-6 NVFP4 is the standing seat, ana-ml2 :8021 ONLY; run 7 PURGED ~139 GiB; pfi-gx10 is an experimental/TRAINING box and carries no serving seat; all five ERP adapters now MIRRORED to ana-ml2 and inside restic; **BabyBronte / R49 author-voice regime is in PREP on gx10, carrier SETTLED = dense Qwen3**; checkpoints AND superseded merges PURGED both boxes, **~573 GB total**; only merged-run06 + the v6 quant survive)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under an hour old, read it (it carries the in-flight @@ -159,10 +159,24 @@ preserved verbatim in `archival-memory.md` § Superseded in-flight snapshots._ free); ana-ml2 `/tank/erp-tune/run-01` 12G + `run-02` 12G + `run-03` 5.9G = **29 GB** (`/tank/erp-tune` 392G→363G, `zfs list -t snapshot` EMPTY so the space is genuinely returned). **~74 GB total. All eight adapters re-verified by sha256 AFTER the deletion**, matching the mirror. - ⏳ **NOT touched, and it is the bigger prize (~550 GB): superseded MERGED models** — they are not - checkpoints, so I stayed literal. gx10 `serve/merged-run{03c,04,05,06}` 49 GiB each (196 GiB) + - ana-ml2 `/tank/erp-tune/serve` ~354 GB. Keep `merged-run06` + `/tank/aimodels/erp-tune-v6-*` (the - standing seat's rollback); `merged-run03c/04/05` are dead weight. Separate operator call. +- **🧹 SUPERSEDED MERGED MODELS PURGED 2026-09-09 23:08 PT** (operator: *"purge the merged models, + keep run06 and the v6 quant"*). **Eleven merges, ~499 GB**, literal paths one `rm` per line: + gx10 `serve/merged-run{03c,04,05}` 3×49 GiB = 145 GB (382G→237G used, **632 GB free**); ana-ml2 + `/tank/erp-tune/serve/{merged-final,merged-run02,merged-run03,merged-run03-s025,merged-run03-s050, + merged-run03-s075,merged-test,nvfp4a16-test}` = 354 GB (`serve/` 354G→8.6M, no snapshots, pool raw + FREE 4.80T→5.30T). **Tonight's two purges total ~573 GB.** + ⭐ **The safety proof that made this cheap: `docker inspect vllm-erp-seat` shows the live seat binds + `/tank/aimodels` ONLY**, model arg `/tank/aimodels/erp-tune-v6-nvfp4a16` — `/tank/erp-tune/serve` was + never in the serving path. Also verified no container mounts it and no process held a file open under + it (`/proc/*/fd` + `/proc/*/maps`, since `lsof` returned nothing and absence-of-output is not proof). + All eleven were run-1/2/3-era Gemma-4 merges (Aug 24–26); the keeper is Sep 8 in a different tree. + **KEPT + verified after deletion:** gx10 `serve/merged-run06` (49 GiB, 9 files), + `/tank/aimodels/erp-tune-v6-{bf16,nvfp4a16}`, all **8 adapters sha256-unchanged**, and + `/tank/erp-tune/serve`'s merge/quant tooling + logs (8.6 MB — the provenance for how the artifacts + were built). **The live seat never bounced**: `Pfish-6` answered a real completion after the delete + (`finish_reason: stop`, correct text), container `Up 4 hours (healthy)`. + ⚠ `gx10:~/erp-tune/relaunch-trial-seat.sh` names a now-deleted model; NOT removed (its flags carry + the FlashInfer JIT/PATH trap + gpu-clear/never-pkill notes) but banner-marked RETIRED. - **🖋 BabyBronte / R49 author-voice LoRA regime — IN PREP on pfi-gx10, nothing training.** Plan + every measured number: [`docs/pfi/author-voice-lora-regime.md`](docs/pfi/author-voice-lora-regime.md). Research target is **brokkr-smithy R49** (`research/R49-author-voice-adapters/`) — brokkr owns