chore(erp-tune): purge superseded merged models (~499 GB), keep run 6 and the v6 quant
Operator: "purge the merged models, keep run06 and the v6 quant." Eleven merges
removed with literal paths, one rm per line.
pfi-gx10 merged-run03c, merged-run04, merged-run05 3 x 49 GiB -> 145 GB
ana-ml2 merged-final, merged-run02, merged-run03,
merged-run03-s{025,050,075}, merged-test,
nvfp4a16-test 8 dirs -> 354 GB
gx10 382G->237G used, 632 GB free. /tank/erp-tune/serve 354G->8.6M with no
snapshots holding the blocks; the pool's raw FREE moved 4.80T->5.30T. Combined
with the earlier checkpoint purge, ~573 GB reclaimed tonight.
The check that made this safe: docker inspect on the live vllm-erp-seat shows it
binds /tank/aimodels ONLY, with model arg /tank/aimodels/erp-tune-v6-nvfp4a16, so
/tank/erp-tune/serve was never in the serving path. Also confirmed no container
mounts that tree and no process held a file open under it, reading /proc/*/fd and
/proc/*/maps rather than trusting an empty lsof. All eleven were run-1/2/3-era
Gemma-4 merges dated Aug 24-26; the keeper is dated Sep 8 and lives in a different
tree, so there was no ambiguity about which was which.
Kept and re-verified after the deletion: gx10 serve/merged-run06,
/tank/aimodels/erp-tune-v6-{bf16,nvfp4a16}, all eight adapters sha256-unchanged,
and the merge/quant tooling and logs under /tank/erp-tune/serve (8.6 MB) that
document how the artifacts were built.
The live seat never bounced -- Pfish-6 answered a real completion after the delete
with finish_reason stop and correct text, container still healthy at 4 h uptime.
relaunch-trial-seat.sh now names a deleted model; it was banner-marked RETIRED
rather than removed, because its flags carry the FlashInfer JIT/PATH trap and the
gpu-clear / never-pkill notes.
This commit is contained in:
@@ -261,12 +261,39 @@ snapshot` empty so the space is genuinely returned rather than snapshot-held.
|
||||
**All eight adapters re-verified by `sha256` after the deletion**, matching the
|
||||
values recorded during the mirror.
|
||||
|
||||
⏳ **Still standing, and it is the bigger prize — ~550 GB of superseded MERGED
|
||||
models, deliberately NOT touched** because they are not checkpoints:
|
||||
`gx10:~/erp-tune/serve/merged-run{03c,04,05,06}` at 49 GiB each (196 GiB) and
|
||||
`ana-ml2:/tank/erp-tune/serve` (~354 GB). Run 6 is the standing seat, so
|
||||
`merged-run06` plus `/tank/aimodels/erp-tune-v6-{bf16,nvfp4a16}` are the rollback
|
||||
path and should survive; `merged-run03c/04/05` are dead weight. Separate call.
|
||||
**And the merged models followed at 23:08 PT**, operator: _"purge the merged
|
||||
models, keep run06 and the v6 quant."_ Eleven merges, literal paths, one `rm` per
|
||||
line, after proving what the live seat reads: `docker inspect vllm-erp-seat` shows
|
||||
it binds **`/tank/aimodels` only** and its model arg is
|
||||
`/tank/aimodels/erp-tune-v6-nvfp4a16`, so `/tank/erp-tune/serve` was never in the
|
||||
serving path at all. No container mounts it and no process held a file open under
|
||||
it (checked `/proc/*/fd` and `/proc/*/maps`, not `lsof` alone).
|
||||
|
||||
pfi-gx10 merged-run03c · merged-run04 · merged-run05 3 x 49 GiB -> 145 GB
|
||||
ana-ml2 merged-final · merged-run02 · merged-run03 ·
|
||||
merged-run03-s{025,050,075} · merged-test ·
|
||||
nvfp4a16-test 8 dirs -> 354 GB
|
||||
|
||||
All eleven were Gemma-4 merges from the run-1/2/3 era (Aug 24–26); the keeper is
|
||||
dated Sep 8 and lives in a different tree, so there was no ambiguity about which
|
||||
was which. gx10 382G→237G used (**632 GB free**); `/tank/erp-tune/serve` 354G→8.6M
|
||||
with `zfs list -t snapshot` empty, and the pool's raw FREE moved 4.80T→5.30T.
|
||||
|
||||
**Kept, and verified after the deletion:** `gx10:~/erp-tune/serve/merged-run06`
|
||||
(49 GiB, 9 files), `/tank/aimodels/erp-tune-v6-bf16` (49 G) and
|
||||
`erp-tune-v6-nvfp4a16` (16 G, 9 files incl. `recipe.yaml`), all eight adapters
|
||||
sha256-unchanged, and `/tank/erp-tune/serve`'s merge/quant tooling and logs
|
||||
(`merge_lora.py`, `quant_nvfp4a16.py`, the dry-run logs, `base-arm`, `lora-scales`
|
||||
— 8.6 MB total, the provenance for how every artifact above was built).
|
||||
|
||||
⚠ **`gx10:~/erp-tune/relaunch-trial-seat.sh` now names a model that is gone.** It
|
||||
was not deleted — its flags carry the FlashInfer JIT/PATH trap and the
|
||||
gpu-clear/never-pkill notes, each bought with a real failure — but it gained a
|
||||
RETIRED banner so nobody hits a confusing missing-model error later.
|
||||
|
||||
**The live seat never bounced.** After the deletion `Pfish-6` answered a real
|
||||
completion (`finish_reason: stop`, 5 tokens, correct text), container still
|
||||
`Up 4 hours (healthy)`.
|
||||
|
||||
## 7. Adapter disposition — settled, and made real
|
||||
|
||||
|
||||
+19
-5
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-09-09 23:10 PT (**Pfish-6** = run-6 NVFP4 is the standing seat, ana-ml2 :8021 ONLY; run 7 PURGED ~139 GiB; pfi-gx10 is an experimental/TRAINING box and carries no serving seat; all five ERP adapters now MIRRORED to ana-ml2 and inside restic; **BabyBronte / R49 author-voice regime is in PREP on gx10, carrier SETTLED = dense Qwen3**; intermediate checkpoints PURGED both boxes ~74 GB)_
|
||||
_Last updated: 2026-09-09 23:15 PT (**Pfish-6** = run-6 NVFP4 is the standing seat, ana-ml2 :8021 ONLY; run 7 PURGED ~139 GiB; pfi-gx10 is an experimental/TRAINING box and carries no serving seat; all five ERP adapters now MIRRORED to ana-ml2 and inside restic; **BabyBronte / R49 author-voice regime is in PREP on gx10, carrier SETTLED = dense Qwen3**; checkpoints AND superseded merges PURGED both boxes, **~573 GB total**; only merged-run06 + the v6 quant survive)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under an hour old, read it (it carries the in-flight
|
||||
@@ -159,10 +159,24 @@ preserved verbatim in `archival-memory.md` § Superseded in-flight snapshots._
|
||||
free); ana-ml2 `/tank/erp-tune/run-01` 12G + `run-02` 12G + `run-03` 5.9G = **29 GB**
|
||||
(`/tank/erp-tune` 392G→363G, `zfs list -t snapshot` EMPTY so the space is genuinely returned).
|
||||
**~74 GB total. All eight adapters re-verified by sha256 AFTER the deletion**, matching the mirror.
|
||||
⏳ **NOT touched, and it is the bigger prize (~550 GB): superseded MERGED models** — they are not
|
||||
checkpoints, so I stayed literal. gx10 `serve/merged-run{03c,04,05,06}` 49 GiB each (196 GiB) +
|
||||
ana-ml2 `/tank/erp-tune/serve` ~354 GB. Keep `merged-run06` + `/tank/aimodels/erp-tune-v6-*` (the
|
||||
standing seat's rollback); `merged-run03c/04/05` are dead weight. Separate operator call.
|
||||
- **🧹 SUPERSEDED MERGED MODELS PURGED 2026-09-09 23:08 PT** (operator: *"purge the merged models,
|
||||
keep run06 and the v6 quant"*). **Eleven merges, ~499 GB**, literal paths one `rm` per line:
|
||||
gx10 `serve/merged-run{03c,04,05}` 3×49 GiB = 145 GB (382G→237G used, **632 GB free**); ana-ml2
|
||||
`/tank/erp-tune/serve/{merged-final,merged-run02,merged-run03,merged-run03-s025,merged-run03-s050,
|
||||
merged-run03-s075,merged-test,nvfp4a16-test}` = 354 GB (`serve/` 354G→8.6M, no snapshots, pool raw
|
||||
FREE 4.80T→5.30T). **Tonight's two purges total ~573 GB.**
|
||||
⭐ **The safety proof that made this cheap: `docker inspect vllm-erp-seat` shows the live seat binds
|
||||
`/tank/aimodels` ONLY**, model arg `/tank/aimodels/erp-tune-v6-nvfp4a16` — `/tank/erp-tune/serve` was
|
||||
never in the serving path. Also verified no container mounts it and no process held a file open under
|
||||
it (`/proc/*/fd` + `/proc/*/maps`, since `lsof` returned nothing and absence-of-output is not proof).
|
||||
All eleven were run-1/2/3-era Gemma-4 merges (Aug 24–26); the keeper is Sep 8 in a different tree.
|
||||
**KEPT + verified after deletion:** gx10 `serve/merged-run06` (49 GiB, 9 files),
|
||||
`/tank/aimodels/erp-tune-v6-{bf16,nvfp4a16}`, all **8 adapters sha256-unchanged**, and
|
||||
`/tank/erp-tune/serve`'s merge/quant tooling + logs (8.6 MB — the provenance for how the artifacts
|
||||
were built). **The live seat never bounced**: `Pfish-6` answered a real completion after the delete
|
||||
(`finish_reason: stop`, correct text), container `Up 4 hours (healthy)`.
|
||||
⚠ `gx10:~/erp-tune/relaunch-trial-seat.sh` names a now-deleted model; NOT removed (its flags carry
|
||||
the FlashInfer JIT/PATH trap + gpu-clear/never-pkill notes) but banner-marked RETIRED.
|
||||
- **🖋 BabyBronte / R49 author-voice LoRA regime — IN PREP on pfi-gx10, nothing training.** Plan +
|
||||
every measured number: [`docs/pfi/author-voice-lora-regime.md`](docs/pfi/author-voice-lora-regime.md).
|
||||
Research target is **brokkr-smithy R49** (`research/R49-author-voice-adapters/`) — brokkr owns
|
||||
|
||||
Reference in New Issue
Block a user