From 92cf09e39725ace76269c535601321e9b1070188 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Wed, 9 Sep 2026 22:10:07 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20in-flight=20rewritten=20for=20the=20c?= =?UTF-8?q?lose=20=E2=80=94=20Pfish-6=20on=20ana-ml2=20only,=20gx10=20is?= =?UTF-8?q?=20a=20training=20box=20with=20no=20seat,=20run=207=20purged=20?= =?UTF-8?q?(~139=20GiB),=20no=20new=20run=20planned?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- persistent-memory.md | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-) diff --git a/persistent-memory.md b/persistent-memory.md index 4c7967f..f842df3 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-09 21:45 PT (run 7 RETIRED; **Pfish-6** = the run-6 NVFP4 quant is the standing seat on both boxes at 262k ctx; its gate failure was a DETECTOR BUG the operator diagnosed, fixed `cc42d76`; ana-ml2 measured 4.1x FASTER than the GX10; ana-ml2 pool actions landed; Booth gained the ASKS primitive)_ +_Last updated: 2026-09-09 22:05 PT (**Pfish-6** = run-6 NVFP4 is the standing seat, ana-ml2 :8021 ONLY; run 7 PURGED ~139 GiB; pfi-gx10 is an experimental/TRAINING box and carries no serving seat; no new run planned)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under an hour old, read it (it carries the in-flight @@ -112,17 +112,22 @@ no longer deployed sidecars here. See Recent decisions.) _As of 2026-09-09 21:45 PT. **Nothing is training and nothing is queued.** Older in-flight blocks are preserved verbatim in `archival-memory.md` § Superseded in-flight snapshots._ -- **`Pfish-6` is the standing seat, on BOTH boxes.** `erp-tune-v6-nvfp4a16` served under that name at - 262,144 ctx: ana-ml2 `:8021` (compose `stacks/erp-seat`, healthy) and pfi-gx10 `:8098` (hand-launched, - `~/erp-tune/vllm-pfish6.pid`). LiteLLM alias `trial` is RETIRED, replaced by `Pfish-6`. Run 7 is - retired — its gate failure was a detector bug, but the run was independently poor. - → `persistent-memory.d/2026-09-09-run7-retired-pfish6.md` -- **⏳ OPEN, operator's call, no tracking surface (untracked by operator choice):** - (a) **what the GX10 seat is for** — it is 4.1x SLOWER than ana-ml2 on interactive decode but holds a - 17.6x-context KV pool (4.6M tokens), so it suits long-context or batch work, not latency; - (b) **disposition of the run-7 artifacts** — adapter 315 MB + merged-run07 49 GiB on gx10, - v7-nvfp4a16 16 GiB + v7-bf16 49 GiB on ana-ml2, ~113 GiB total, nothing deleted; - (c) **whether the opening-split idea gets a fresh run** on a clean base (a NEW run, not a re-read). +- **`Pfish-6` is the standing seat — ana-ml2 `:8021` ONLY.** `erp-tune-v6-nvfp4a16` served under that + name at 262,144 ctx, compose `stacks/erp-seat`, max-num-seqs 32, KV 534,649 tok. LiteLLM alias + `trial` is RETIRED, replaced by `Pfish-6`. → `persistent-memory.d/2026-09-09-run7-retired-pfish6.md` +- **pfi-gx10 is an EXPERIMENTAL box, primarily for TRAINING** (operator ruling 2026-09-09). It carries + **no standing serving seat** — the Pfish-6 arm there was taken down 22:00 PT and its GPU is free. + The measured reason it is the wrong serving box: **4.1x SLOWER than ana-ml2** on interactive decode + (54 vs 219 tok/s single-stream, same artifact and context) because decode is bandwidth-bound. Its + advantage is a **17.6x-context KV pool** (4.6M tokens) — capacity for long-context/batch work. +- **⛔ RUN 7 PURGED 2026-09-09 22:00 PT** (operator: *"run 7 can be purged … no new run, we'll roll + with run 6 for now"*). Deleted with literal paths, one per line: gx10 `serve/merged-run07` 49 GiB, + `run-07/checkpoints` 9.2 GiB, `serve/pfish6-nvfp4a16` 16 GiB; ana-ml2 `erp-tune-v7-bf16` 49 GiB, + `erp-tune-v7-nvfp4a16` 16 GiB, `erp-tune-v7-quant-work`. **~139 GiB reclaimed** (gx10 53%→47%). + ⚠ **KEPT deliberately: `~/erp-tune/run-07/adapter` 315 MB + provenance + `loss-series-r7.json`** — + the only non-reproducible piece (14 h of training), and it costs nothing. `rm -rf + /home/infra-ops/erp-tune/run-07` finishes the job if wanted; everything else run-7 is already gone. +- **NO NEW TRAINING RUN PLANNED.** The opening-split idea is not being re-tested; run 6 stands. - **⏳ ana-ml2 storage follow-ups, operator's call** (the three actions themselves LANDED 09-09 02:02): (a) **ZFS pool-health ALERTING** — `tank` sat DEGRADED 04-23→09-05 with nvme7 physically absent and nobody knew (ZED mails `root`, no MTA on the box); (b) nvme7 / slot 0-5 keep-vs-replace —