memory: in-flight rewritten for the close — Pfish-6 on ana-ml2 only, gx10 is a training box with no seat, run 7 purged (~139 GiB), no new run planned
This commit is contained in:
+17
-12
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-09-09 21:45 PT (run 7 RETIRED; **Pfish-6** = the run-6 NVFP4 quant is the standing seat on both boxes at 262k ctx; its gate failure was a DETECTOR BUG the operator diagnosed, fixed `cc42d76`; ana-ml2 measured 4.1x FASTER than the GX10; ana-ml2 pool actions landed; Booth gained the ASKS primitive)_
|
||||
_Last updated: 2026-09-09 22:05 PT (**Pfish-6** = run-6 NVFP4 is the standing seat, ana-ml2 :8021 ONLY; run 7 PURGED ~139 GiB; pfi-gx10 is an experimental/TRAINING box and carries no serving seat; no new run planned)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under an hour old, read it (it carries the in-flight
|
||||
@@ -112,17 +112,22 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
_As of 2026-09-09 21:45 PT. **Nothing is training and nothing is queued.** Older in-flight blocks are
|
||||
preserved verbatim in `archival-memory.md` § Superseded in-flight snapshots._
|
||||
|
||||
- **`Pfish-6` is the standing seat, on BOTH boxes.** `erp-tune-v6-nvfp4a16` served under that name at
|
||||
262,144 ctx: ana-ml2 `:8021` (compose `stacks/erp-seat`, healthy) and pfi-gx10 `:8098` (hand-launched,
|
||||
`~/erp-tune/vllm-pfish6.pid`). LiteLLM alias `trial` is RETIRED, replaced by `Pfish-6`. Run 7 is
|
||||
retired — its gate failure was a detector bug, but the run was independently poor.
|
||||
→ `persistent-memory.d/2026-09-09-run7-retired-pfish6.md`
|
||||
- **⏳ OPEN, operator's call, no tracking surface (untracked by operator choice):**
|
||||
(a) **what the GX10 seat is for** — it is 4.1x SLOWER than ana-ml2 on interactive decode but holds a
|
||||
17.6x-context KV pool (4.6M tokens), so it suits long-context or batch work, not latency;
|
||||
(b) **disposition of the run-7 artifacts** — adapter 315 MB + merged-run07 49 GiB on gx10,
|
||||
v7-nvfp4a16 16 GiB + v7-bf16 49 GiB on ana-ml2, ~113 GiB total, nothing deleted;
|
||||
(c) **whether the opening-split idea gets a fresh run** on a clean base (a NEW run, not a re-read).
|
||||
- **`Pfish-6` is the standing seat — ana-ml2 `:8021` ONLY.** `erp-tune-v6-nvfp4a16` served under that
|
||||
name at 262,144 ctx, compose `stacks/erp-seat`, max-num-seqs 32, KV 534,649 tok. LiteLLM alias
|
||||
`trial` is RETIRED, replaced by `Pfish-6`. → `persistent-memory.d/2026-09-09-run7-retired-pfish6.md`
|
||||
- **pfi-gx10 is an EXPERIMENTAL box, primarily for TRAINING** (operator ruling 2026-09-09). It carries
|
||||
**no standing serving seat** — the Pfish-6 arm there was taken down 22:00 PT and its GPU is free.
|
||||
The measured reason it is the wrong serving box: **4.1x SLOWER than ana-ml2** on interactive decode
|
||||
(54 vs 219 tok/s single-stream, same artifact and context) because decode is bandwidth-bound. Its
|
||||
advantage is a **17.6x-context KV pool** (4.6M tokens) — capacity for long-context/batch work.
|
||||
- **⛔ RUN 7 PURGED 2026-09-09 22:00 PT** (operator: *"run 7 can be purged … no new run, we'll roll
|
||||
with run 6 for now"*). Deleted with literal paths, one per line: gx10 `serve/merged-run07` 49 GiB,
|
||||
`run-07/checkpoints` 9.2 GiB, `serve/pfish6-nvfp4a16` 16 GiB; ana-ml2 `erp-tune-v7-bf16` 49 GiB,
|
||||
`erp-tune-v7-nvfp4a16` 16 GiB, `erp-tune-v7-quant-work`. **~139 GiB reclaimed** (gx10 53%→47%).
|
||||
⚠ **KEPT deliberately: `~/erp-tune/run-07/adapter` 315 MB + provenance + `loss-series-r7.json`** —
|
||||
the only non-reproducible piece (14 h of training), and it costs nothing. `rm -rf
|
||||
/home/infra-ops/erp-tune/run-07` finishes the job if wanted; everything else run-7 is already gone.
|
||||
- **NO NEW TRAINING RUN PLANNED.** The opening-split idea is not being re-tested; run 6 stands.
|
||||
- **⏳ ana-ml2 storage follow-ups, operator's call** (the three actions themselves LANDED 09-09 02:02):
|
||||
(a) **ZFS pool-health ALERTING** — `tank` sat DEGRADED 04-23→09-05 with nvme7 physically absent and
|
||||
nobody knew (ZED mails `root`, no MTA on the box); (b) nvme7 / slot 0-5 keep-vs-replace —
|
||||
|
||||
Reference in New Issue
Block a user