diff --git a/docs/pfi/author-voice-lora-regime.md b/docs/pfi/author-voice-lora-regime.md index bfb4927..f1243d5 100644 --- a/docs/pfi/author-voice-lora-regime.md +++ b/docs/pfi/author-voice-lora-regime.md @@ -261,12 +261,39 @@ snapshot` empty so the space is genuinely returned rather than snapshot-held. **All eight adapters re-verified by `sha256` after the deletion**, matching the values recorded during the mirror. -⏳ **Still standing, and it is the bigger prize — ~550 GB of superseded MERGED -models, deliberately NOT touched** because they are not checkpoints: -`gx10:~/erp-tune/serve/merged-run{03c,04,05,06}` at 49 GiB each (196 GiB) and -`ana-ml2:/tank/erp-tune/serve` (~354 GB). Run 6 is the standing seat, so -`merged-run06` plus `/tank/aimodels/erp-tune-v6-{bf16,nvfp4a16}` are the rollback -path and should survive; `merged-run03c/04/05` are dead weight. Separate call. +**And the merged models followed at 23:08 PT**, operator: _"purge the merged +models, keep run06 and the v6 quant."_ Eleven merges, literal paths, one `rm` per +line, after proving what the live seat reads: `docker inspect vllm-erp-seat` shows +it binds **`/tank/aimodels` only** and its model arg is +`/tank/aimodels/erp-tune-v6-nvfp4a16`, so `/tank/erp-tune/serve` was never in the +serving path at all. No container mounts it and no process held a file open under +it (checked `/proc/*/fd` and `/proc/*/maps`, not `lsof` alone). + + pfi-gx10 merged-run03c · merged-run04 · merged-run05 3 x 49 GiB -> 145 GB + ana-ml2 merged-final · merged-run02 · merged-run03 · + merged-run03-s{025,050,075} · merged-test · + nvfp4a16-test 8 dirs -> 354 GB + +All eleven were Gemma-4 merges from the run-1/2/3 era (Aug 24–26); the keeper is +dated Sep 8 and lives in a different tree, so there was no ambiguity about which +was which. gx10 382G→237G used (**632 GB free**); `/tank/erp-tune/serve` 354G→8.6M +with `zfs list -t snapshot` empty, and the pool's raw FREE moved 4.80T→5.30T. + +**Kept, and verified after the deletion:** `gx10:~/erp-tune/serve/merged-run06` +(49 GiB, 9 files), `/tank/aimodels/erp-tune-v6-bf16` (49 G) and +`erp-tune-v6-nvfp4a16` (16 G, 9 files incl. `recipe.yaml`), all eight adapters +sha256-unchanged, and `/tank/erp-tune/serve`'s merge/quant tooling and logs +(`merge_lora.py`, `quant_nvfp4a16.py`, the dry-run logs, `base-arm`, `lora-scales` +— 8.6 MB total, the provenance for how every artifact above was built). + +⚠ **`gx10:~/erp-tune/relaunch-trial-seat.sh` now names a model that is gone.** It +was not deleted — its flags carry the FlashInfer JIT/PATH trap and the +gpu-clear/never-pkill notes, each bought with a real failure — but it gained a +RETIRED banner so nobody hits a confusing missing-model error later. + +**The live seat never bounced.** After the deletion `Pfish-6` answered a real +completion (`finish_reason: stop`, 5 tokens, correct text), container still +`Up 4 hours (healthy)`. ## 7. Adapter disposition — settled, and made real diff --git a/persistent-memory.md b/persistent-memory.md index 822c2a7..ce1bce8 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-09 23:10 PT (**Pfish-6** = run-6 NVFP4 is the standing seat, ana-ml2 :8021 ONLY; run 7 PURGED ~139 GiB; pfi-gx10 is an experimental/TRAINING box and carries no serving seat; all five ERP adapters now MIRRORED to ana-ml2 and inside restic; **BabyBronte / R49 author-voice regime is in PREP on gx10, carrier SETTLED = dense Qwen3**; intermediate checkpoints PURGED both boxes ~74 GB)_ +_Last updated: 2026-09-09 23:15 PT (**Pfish-6** = run-6 NVFP4 is the standing seat, ana-ml2 :8021 ONLY; run 7 PURGED ~139 GiB; pfi-gx10 is an experimental/TRAINING box and carries no serving seat; all five ERP adapters now MIRRORED to ana-ml2 and inside restic; **BabyBronte / R49 author-voice regime is in PREP on gx10, carrier SETTLED = dense Qwen3**; checkpoints AND superseded merges PURGED both boxes, **~573 GB total**; only merged-run06 + the v6 quant survive)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under an hour old, read it (it carries the in-flight @@ -159,10 +159,24 @@ preserved verbatim in `archival-memory.md` § Superseded in-flight snapshots._ free); ana-ml2 `/tank/erp-tune/run-01` 12G + `run-02` 12G + `run-03` 5.9G = **29 GB** (`/tank/erp-tune` 392G→363G, `zfs list -t snapshot` EMPTY so the space is genuinely returned). **~74 GB total. All eight adapters re-verified by sha256 AFTER the deletion**, matching the mirror. - ⏳ **NOT touched, and it is the bigger prize (~550 GB): superseded MERGED models** — they are not - checkpoints, so I stayed literal. gx10 `serve/merged-run{03c,04,05,06}` 49 GiB each (196 GiB) + - ana-ml2 `/tank/erp-tune/serve` ~354 GB. Keep `merged-run06` + `/tank/aimodels/erp-tune-v6-*` (the - standing seat's rollback); `merged-run03c/04/05` are dead weight. Separate operator call. +- **🧹 SUPERSEDED MERGED MODELS PURGED 2026-09-09 23:08 PT** (operator: *"purge the merged models, + keep run06 and the v6 quant"*). **Eleven merges, ~499 GB**, literal paths one `rm` per line: + gx10 `serve/merged-run{03c,04,05}` 3×49 GiB = 145 GB (382G→237G used, **632 GB free**); ana-ml2 + `/tank/erp-tune/serve/{merged-final,merged-run02,merged-run03,merged-run03-s025,merged-run03-s050, + merged-run03-s075,merged-test,nvfp4a16-test}` = 354 GB (`serve/` 354G→8.6M, no snapshots, pool raw + FREE 4.80T→5.30T). **Tonight's two purges total ~573 GB.** + ⭐ **The safety proof that made this cheap: `docker inspect vllm-erp-seat` shows the live seat binds + `/tank/aimodels` ONLY**, model arg `/tank/aimodels/erp-tune-v6-nvfp4a16` — `/tank/erp-tune/serve` was + never in the serving path. Also verified no container mounts it and no process held a file open under + it (`/proc/*/fd` + `/proc/*/maps`, since `lsof` returned nothing and absence-of-output is not proof). + All eleven were run-1/2/3-era Gemma-4 merges (Aug 24–26); the keeper is Sep 8 in a different tree. + **KEPT + verified after deletion:** gx10 `serve/merged-run06` (49 GiB, 9 files), + `/tank/aimodels/erp-tune-v6-{bf16,nvfp4a16}`, all **8 adapters sha256-unchanged**, and + `/tank/erp-tune/serve`'s merge/quant tooling + logs (8.6 MB — the provenance for how the artifacts + were built). **The live seat never bounced**: `Pfish-6` answered a real completion after the delete + (`finish_reason: stop`, correct text), container `Up 4 hours (healthy)`. + ⚠ `gx10:~/erp-tune/relaunch-trial-seat.sh` names a now-deleted model; NOT removed (its flags carry + the FlashInfer JIT/PATH trap + gpu-clear/never-pkill notes) but banner-marked RETIRED. - **🖋 BabyBronte / R49 author-voice LoRA regime — IN PREP on pfi-gx10, nothing training.** Plan + every measured number: [`docs/pfi/author-voice-lora-regime.md`](docs/pfi/author-voice-lora-regime.md). Research target is **brokkr-smithy R49** (`research/R49-author-voice-adapters/`) — brokkr owns