chore(erp-tune): purge intermediate checkpoints (~74 GB); R49 carrier settled on dense Qwen3
Two operator rulings, 2026-09-09.
"purge intermediate checkpoints" -- seven checkpoints/ directories removed with
literal paths, one rm per line, after confirming none was a symlink and that
every run's final adapter/ is an independent real directory:
pfi-gx10 run-03c 11G run-04 16G run-05 9.2G run-06 9.2G = 45 GB
ana-ml2 run-01 12G run-02 12G run-03 5.9G = 29 GB
gx10 419G->374G used, 496 GB free. /tank/erp-tune 392G->363G with zfs list -t
snapshot empty, so the space is genuinely returned rather than snapshot-held. All
eight adapters re-verified by sha256 after the deletion, matching the values
recorded during the mirror. Merged artifacts deliberately untouched -- they are
not checkpoints, and the ~550 GB of superseded merges stays a separate call.
"use dense qwen3" -- the R49 H02 carrier sweep becomes Qwen3-{0.6,1.7,4}B-Base,
which overrides the Qwen3.5 arms H02 names; brokkr-smithy owns that file and was
told directly. Qwen3-4B-Base staged and benched to complete the family:
Qwen3-0.6B-Base 0.616 B 1.707 s/step 2,399 tok/s spread 0.6%
Qwen3-1.7B-Base 1.755 B 2.895 s/step 1,415 tok/s spread 0.8%
Qwen3-4B-Base 4.089 B 5.714 s/step 717 tok/s spread 0.3%
The dense 4.089 B carrier still trains 33% faster than the hybrid 0.765 B one.
Projected per voice 2.7 / 4.6 / 9.1 h; the three-arm sweep at two seeds is ~33 h
of GPU, ~10 h if H03's corpus floor holds. The three Qwen3.5 checkpoints stay
staged so the decision is reversible behind an fla install.
Also recorded: verified at 22:45-22:48 PT that nothing is training on gx10,
ana-ml2, nh3-dev or irv-ml1, and that brokkr's own run07-gate close states
"Nothing is owed. No battery to run." Run 7 has no servable artifact left. And a
correction to a standing lesson -- the bracketed-class trick does not defeat a
wrapper's argv, since the invoking shell's command line carries the literal
pattern; observe the artifact instead.
This commit is contained in:
+36
-4
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-09-09 22:50 PT (**Pfish-6** = run-6 NVFP4 is the standing seat, ana-ml2 :8021 ONLY; run 7 PURGED ~139 GiB; pfi-gx10 is an experimental/TRAINING box and carries no serving seat; all five ERP adapters now MIRRORED to ana-ml2 and inside restic; **BabyBronte / R49 author-voice regime is in PREP on gx10** — carrier-family decision open)_
|
||||
_Last updated: 2026-09-09 23:10 PT (**Pfish-6** = run-6 NVFP4 is the standing seat, ana-ml2 :8021 ONLY; run 7 PURGED ~139 GiB; pfi-gx10 is an experimental/TRAINING box and carries no serving seat; all five ERP adapters now MIRRORED to ana-ml2 and inside restic; **BabyBronte / R49 author-voice regime is in PREP on gx10, carrier SETTLED = dense Qwen3**; intermediate checkpoints PURGED both boxes ~74 GB)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under an hour old, read it (it carries the in-flight
|
||||
@@ -129,6 +129,22 @@ preserved verbatim in `archival-memory.md` § Superseded in-flight snapshots._
|
||||
already gone; do NOT `rm -rf /home/infra-ops/erp-tune/run-07` — operator ruled **keep the adapter**
|
||||
2026-09-09.
|
||||
- **NO NEW *ERP* TRAINING RUN PLANNED.** The opening-split idea is not being re-tested; run 6 stands.
|
||||
- **⚠ VERIFIED 2026-09-09 22:45–22:48 PT: NOTHING IS TRAINING ANYWHERE.** Checked gx10 (0 GPU compute
|
||||
apps, 0% util, every pidfile dead), ana-ml2 (all 8 GPU processes are vLLM serving seats, no trainer),
|
||||
nh3-dev, irv-ml1 (5 GPU processes, all the known native serving stacks). The last run was **run 7,
|
||||
finished 13:23 PT**, and brokkr's own `research/R47-premium-corpus-gate/run07-gate/GATE-FAILED-run07.md`
|
||||
(rev 19:40 PT) closes it: **"Nothing is owed. No battery to run."** Three findings there worth
|
||||
holding: the CSAM gate fire was a **FALSE POSITIVE** (adjective `minor` in a HARD `child_term` tier,
|
||||
fixed `cc42d76`, selftest 24/24) so the safety result is **VOID on an instrument defect, not a
|
||||
failure**; run 7 is retired **on its merits** (primary flat 69→70.5, both diversity families reduced,
|
||||
coherence 1.0→0.875), which is the operator's own ruling; and ⚠ **do NOT re-run that gate against
|
||||
gx10:8098** — the artifact there changed and then went down entirely. ⚠ **Run 7 has NO servable
|
||||
artifact left**: `serve/merged-run07` and `serve/pfish6-nvfp4a16` are both GONE, so "serve run 7"
|
||||
means a fresh 49 GiB merge + 16 GiB requant, hours, on a tune already judged flat.
|
||||
- **⚠ THE BRACKETED-CLASS pgrep TRICK DOES NOT SAVE A WRAPPER'S ARGV.** `ps -eo cmd | grep '[t]rain'`
|
||||
still matched my own shell command, because the invoking `zsh -c '...'` argv contains the literal
|
||||
bracketed pattern. The bracket only stops *grep itself* from matching. Durable form: **observe the
|
||||
artifact** (GPU compute apps, the seat's own log) or exclude known PIDs — never the process name.
|
||||
- **✅ ALL FIVE gx10 ERP ADAPTERS ARE NOW TWO-COPY AND BACKED UP** (2026-09-09 22:30 PT, operator:
|
||||
*"keep the adapter"*). `run-03c/04/05/06/07` mirrored to `ana-ml2:/tank/erp-tune/run-<N>/adapter`
|
||||
(the layout runs 01–03 already used), byte-totals identical both sides + `sha256` match on every
|
||||
@@ -136,6 +152,17 @@ preserved verbatim in `archival-memory.md` § Superseded in-flight snapshots._
|
||||
regenerable weights), so `configs/restic/ana-ml2/profiles.yaml` gained ONE documented carve-out —
|
||||
`/tank/erp-tune/run-*/adapter` — verified by `resticprofile --dry-run` to expand to exactly those 8
|
||||
paths and nothing else. Live file is byte-identical to the repo canonical; `.bak-20260909` beside it.
|
||||
- **🧹 INTERMEDIATE CHECKPOINTS PURGED 2026-09-09 23:00 PT** (operator: *"purge intermediate
|
||||
checkpoints"*). Seven `checkpoints/` dirs, deleted with LITERAL paths one `rm` per line after
|
||||
confirming none was a symlink and that every final `adapter/` is an independent real dir: gx10
|
||||
`run-03c` 11G + `run-04` 16G + `run-05` 9.2G + `run-06` 9.2G = **45 GB** (419G→374G used, 496 GB
|
||||
free); ana-ml2 `/tank/erp-tune/run-01` 12G + `run-02` 12G + `run-03` 5.9G = **29 GB**
|
||||
(`/tank/erp-tune` 392G→363G, `zfs list -t snapshot` EMPTY so the space is genuinely returned).
|
||||
**~74 GB total. All eight adapters re-verified by sha256 AFTER the deletion**, matching the mirror.
|
||||
⏳ **NOT touched, and it is the bigger prize (~550 GB): superseded MERGED models** — they are not
|
||||
checkpoints, so I stayed literal. gx10 `serve/merged-run{03c,04,05,06}` 49 GiB each (196 GiB) +
|
||||
ana-ml2 `/tank/erp-tune/serve` ~354 GB. Keep `merged-run06` + `/tank/aimodels/erp-tune-v6-*` (the
|
||||
standing seat's rollback); `merged-run03c/04/05` are dead weight. Separate operator call.
|
||||
- **🖋 BabyBronte / R49 author-voice LoRA regime — IN PREP on pfi-gx10, nothing training.** Plan +
|
||||
every measured number: [`docs/pfi/author-voice-lora-regime.md`](docs/pfi/author-voice-lora-regime.md).
|
||||
Research target is **brokkr-smithy R49** (`research/R49-author-voice-adapters/`) — brokkr owns
|
||||
@@ -158,9 +185,14 @@ preserved verbatim in `archival-memory.md` § Superseded in-flight snapshots._
|
||||
sampled-target-changed check in the SERVING path); and **cross-document packing is unsafe** because
|
||||
SSM state ignores the attention mask, which breaks the per-copy name-consistency invariant the design
|
||||
doc calls sacred. None of these exist on the dense line.
|
||||
**⭐ OPEN FOR THE OPERATOR: carrier family — recommend the dense `Qwen3-{0.6,1.7,4}B-Base` line**
|
||||
(the design doc's own original pin) over R49 H02's `Qwen3.5` trio. Reversible: Qwen3.5 stays staged
|
||||
and an `fla` install (pure Triton, plausibly fine on aarch64) could revive it as a follow-up.
|
||||
**✅ CARRIER SETTLED 2026-09-09 — operator: *"use dense qwen3"*.** Sweep is
|
||||
**`Qwen3-{0.6,1.7,4}B-Base`**, all three staged on gx10. ⚠ **This OVERRIDES R49 H02's stated arms**
|
||||
(which name the Qwen3.5 trio) — brokkr-smithy owns that file and was told directly. Full dense sweep
|
||||
measured: 0.6B **2,399**, 1.7B **1,415**, 4B **717** tok/s (spread 0.3–0.8%) → projected **2.7 / 4.6 /
|
||||
9.1 h** per voice; whole 3-arm × 2-seed sweep ≈ 33 h GPU, ≈10 h if H03's ~300k-word floor holds. The
|
||||
sharpest framing: **dense 4.089B still beats hybrid 0.765B by 33%** on 5.3x the params. Qwen3.5 stays
|
||||
staged (14.8 GB) — an `fla` install could revive it as a follow-up experiment, never a prerequisite.
|
||||
If anyone re-opens this, re-run `bench_lora_step.py` first: the argument is a measurement.
|
||||
- **📌 forseti shipped althing 3.6.1 (2026-09-09) — every box's herald needs a restart to pick up the
|
||||
new poke text.** `uv tool install --force --reinstall <checkout>` + `systemctl --user restart
|
||||
althing-po-herald`, expect `3.6.1`; the plugin also went to 0.1.6 (inbox.md no longer calls the
|
||||
|
||||
Reference in New Issue
Block a user