chore(erp-tune): purge superseded merged models (~499 GB), keep run 6 and the v6 quant

Operator: "purge the merged models, keep run06 and the v6 quant." Eleven merges
removed with literal paths, one rm per line.

  pfi-gx10   merged-run03c, merged-run04, merged-run05        3 x 49 GiB -> 145 GB
  ana-ml2    merged-final, merged-run02, merged-run03,
             merged-run03-s{025,050,075}, merged-test,
             nvfp4a16-test                                    8 dirs     -> 354 GB

gx10 382G->237G used, 632 GB free. /tank/erp-tune/serve 354G->8.6M with no
snapshots holding the blocks; the pool's raw FREE moved 4.80T->5.30T. Combined
with the earlier checkpoint purge, ~573 GB reclaimed tonight.

The check that made this safe: docker inspect on the live vllm-erp-seat shows it
binds /tank/aimodels ONLY, with model arg /tank/aimodels/erp-tune-v6-nvfp4a16, so
/tank/erp-tune/serve was never in the serving path. Also confirmed no container
mounts that tree and no process held a file open under it, reading /proc/*/fd and
/proc/*/maps rather than trusting an empty lsof. All eleven were run-1/2/3-era
Gemma-4 merges dated Aug 24-26; the keeper is dated Sep 8 and lives in a different
tree, so there was no ambiguity about which was which.

Kept and re-verified after the deletion: gx10 serve/merged-run06,
/tank/aimodels/erp-tune-v6-{bf16,nvfp4a16}, all eight adapters sha256-unchanged,
and the merge/quant tooling and logs under /tank/erp-tune/serve (8.6 MB) that
document how the artifacts were built.

The live seat never bounced -- Pfish-6 answered a real completion after the delete
with finish_reason stop and correct text, container still healthy at 4 h uptime.
relaunch-trial-seat.sh now names a deleted model; it was banner-marked RETIRED
rather than removed, because its flags carry the FlashInfer JIT/PATH trap and the
gpu-clear / never-pkill notes.
This commit is contained in:
vh
2026-09-09 23:09:13 -07:00
parent 9f30c12da7
commit 08c852792b
2 changed files with 52 additions and 11 deletions
+33 -6
View File
@@ -261,12 +261,39 @@ snapshot` empty so the space is genuinely returned rather than snapshot-held.
**All eight adapters re-verified by `sha256` after the deletion**, matching the
values recorded during the mirror.
⏳ **Still standing, and it is the bigger prize — ~550 GB of superseded MERGED
models, deliberately NOT touched** because they are not checkpoints:
`gx10:~/erp-tune/serve/merged-run{03c,04,05,06}` at 49 GiB each (196 GiB) and
`ana-ml2:/tank/erp-tune/serve` (~354 GB). Run 6 is the standing seat, so
`merged-run06` plus `/tank/aimodels/erp-tune-v6-{bf16,nvfp4a16}` are the rollback
path and should survive; `merged-run03c/04/05` are dead weight. Separate call.
**And the merged models followed at 23:08 PT**, operator: _"purge the merged
models, keep run06 and the v6 quant."_ Eleven merges, literal paths, one `rm` per
line, after proving what the live seat reads: `docker inspect vllm-erp-seat` shows
it binds **`/tank/aimodels` only** and its model arg is
`/tank/aimodels/erp-tune-v6-nvfp4a16`, so `/tank/erp-tune/serve` was never in the
serving path at all. No container mounts it and no process held a file open under
it (checked `/proc/*/fd` and `/proc/*/maps`, not `lsof` alone).
pfi-gx10 merged-run03c · merged-run04 · merged-run05 3 x 49 GiB -> 145 GB
ana-ml2 merged-final · merged-run02 · merged-run03 ·
merged-run03-s{025,050,075} · merged-test ·
nvfp4a16-test 8 dirs -> 354 GB
All eleven were Gemma-4 merges from the run-1/2/3 era (Aug 24–26); the keeper is
dated Sep 8 and lives in a different tree, so there was no ambiguity about which
was which. gx10 382G→237G used (**632 GB free**); `/tank/erp-tune/serve` 354G→8.6M
with `zfs list -t snapshot` empty, and the pool's raw FREE moved 4.80T→5.30T.
**Kept, and verified after the deletion:** `gx10:~/erp-tune/serve/merged-run06`
(49 GiB, 9 files), `/tank/aimodels/erp-tune-v6-bf16` (49 G) and
`erp-tune-v6-nvfp4a16` (16 G, 9 files incl. `recipe.yaml`), all eight adapters
sha256-unchanged, and `/tank/erp-tune/serve`'s merge/quant tooling and logs
(`merge_lora.py`, `quant_nvfp4a16.py`, the dry-run logs, `base-arm`, `lora-scales`
— 8.6 MB total, the provenance for how every artifact above was built).
⚠ **`gx10:~/erp-tune/relaunch-trial-seat.sh` now names a model that is gone.** It
was not deleted — its flags carry the FlashInfer JIT/PATH trap and the
gpu-clear/never-pkill notes, each bought with a real failure — but it gained a
RETIRED banner so nobody hits a confusing missing-model error later.
**The live seat never bounced.** After the deletion `Pfish-6` answered a real
completion (`finish_reason: stop`, 5 tokens, correct text), container still
`Up 4 hours (healthy)`.
## 7. Adapter disposition — settled, and made real