memory: in-flight rewritten for the close — Pfish-6 on ana-ml2 only, gx10 is a training box with no seat, run 7 purged (~139 GiB), no new run planned

This commit is contained in:
vh
2026-09-09 22:10:07 -07:00
parent 5ea35d86f4
commit 92cf09e397
+17 -12
View File
@@ -1,6 +1,6 @@
# Persistent memory — eshpfi-management
_Last updated: 2026-09-09 21:45 PT (run 7 RETIRED; **Pfish-6** = the run-6 NVFP4 quant is the standing seat on both boxes at 262k ctx; its gate failure was a DETECTOR BUG the operator diagnosed, fixed `cc42d76`; ana-ml2 measured 4.1x FASTER than the GX10; ana-ml2 pool actions landed; Booth gained the ASKS primitive)_
_Last updated: 2026-09-09 22:05 PT (**Pfish-6** = run-6 NVFP4 is the standing seat, ana-ml2 :8021 ONLY; run 7 PURGED ~139 GiB; pfi-gx10 is an experimental/TRAINING box and carries no serving seat; no new run planned)_
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
> `Written:` stamp is under an hour old, read it (it carries the in-flight
@@ -112,17 +112,22 @@ no longer deployed sidecars here. See Recent decisions.)
_As of 2026-09-09 21:45 PT. **Nothing is training and nothing is queued.** Older in-flight blocks are
preserved verbatim in `archival-memory.md` § Superseded in-flight snapshots._
- **`Pfish-6` is the standing seat, on BOTH boxes.** `erp-tune-v6-nvfp4a16` served under that name at
262,144 ctx: ana-ml2 `:8021` (compose `stacks/erp-seat`, healthy) and pfi-gx10 `:8098` (hand-launched,
`~/erp-tune/vllm-pfish6.pid`). LiteLLM alias `trial` is RETIRED, replaced by `Pfish-6`. Run 7 is
retired — its gate failure was a detector bug, but the run was independently poor.
→ `persistent-memory.d/2026-09-09-run7-retired-pfish6.md`
- **⏳ OPEN, operator's call, no tracking surface (untracked by operator choice):**
(a) **what the GX10 seat is for** — it is 4.1x SLOWER than ana-ml2 on interactive decode but holds a
17.6x-context KV pool (4.6M tokens), so it suits long-context or batch work, not latency;
(b) **disposition of the run-7 artifacts** — adapter 315 MB + merged-run07 49 GiB on gx10,
v7-nvfp4a16 16 GiB + v7-bf16 49 GiB on ana-ml2, ~113 GiB total, nothing deleted;
(c) **whether the opening-split idea gets a fresh run** on a clean base (a NEW run, not a re-read).
- **`Pfish-6` is the standing seat — ana-ml2 `:8021` ONLY.** `erp-tune-v6-nvfp4a16` served under that
name at 262,144 ctx, compose `stacks/erp-seat`, max-num-seqs 32, KV 534,649 tok. LiteLLM alias
`trial` is RETIRED, replaced by `Pfish-6`. → `persistent-memory.d/2026-09-09-run7-retired-pfish6.md`
- **pfi-gx10 is an EXPERIMENTAL box, primarily for TRAINING** (operator ruling 2026-09-09). It carries
**no standing serving seat** — the Pfish-6 arm there was taken down 22:00 PT and its GPU is free.
The measured reason it is the wrong serving box: **4.1x SLOWER than ana-ml2** on interactive decode
(54 vs 219 tok/s single-stream, same artifact and context) because decode is bandwidth-bound. Its
advantage is a **17.6x-context KV pool** (4.6M tokens) — capacity for long-context/batch work.
- **⛔ RUN 7 PURGED 2026-09-09 22:00 PT** (operator: *"run 7 can be purged … no new run, we'll roll
with run 6 for now"*). Deleted with literal paths, one per line: gx10 `serve/merged-run07` 49 GiB,
`run-07/checkpoints` 9.2 GiB, `serve/pfish6-nvfp4a16` 16 GiB; ana-ml2 `erp-tune-v7-bf16` 49 GiB,
`erp-tune-v7-nvfp4a16` 16 GiB, `erp-tune-v7-quant-work`. **~139 GiB reclaimed** (gx10 53%→47%).
⚠ **KEPT deliberately: `~/erp-tune/run-07/adapter` 315 MB + provenance + `loss-series-r7.json`** —
the only non-reproducible piece (14 h of training), and it costs nothing. `rm -rf
/home/infra-ops/erp-tune/run-07` finishes the job if wanted; everything else run-7 is already gone.
- **NO NEW TRAINING RUN PLANNED.** The opening-split idea is not being re-tested; run 6 stands.
- **⏳ ana-ml2 storage follow-ups, operator's call** (the three actions themselves LANDED 09-09 02:02):
(a) **ZFS pool-health ALERTING** — `tank` sat DEGRADED 04-23→09-05 with nvme7 physically absent and
nobody knew (ZED mails `root`, no MTA on the box); (b) nvme7 / slot 0-5 keep-vs-replace —