From 36f1b70a88c2c245bd2fd6a538d10ab8a62dc42e Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Wed, 9 Sep 2026 22:55:34 -0700 Subject: [PATCH] chore(erp-tune): purge intermediate checkpoints (~74 GB); R49 carrier settled on dense Qwen3 Two operator rulings, 2026-09-09. "purge intermediate checkpoints" -- seven checkpoints/ directories removed with literal paths, one rm per line, after confirming none was a symlink and that every run's final adapter/ is an independent real directory: pfi-gx10 run-03c 11G run-04 16G run-05 9.2G run-06 9.2G = 45 GB ana-ml2 run-01 12G run-02 12G run-03 5.9G = 29 GB gx10 419G->374G used, 496 GB free. /tank/erp-tune 392G->363G with zfs list -t snapshot empty, so the space is genuinely returned rather than snapshot-held. All eight adapters re-verified by sha256 after the deletion, matching the values recorded during the mirror. Merged artifacts deliberately untouched -- they are not checkpoints, and the ~550 GB of superseded merges stays a separate call. "use dense qwen3" -- the R49 H02 carrier sweep becomes Qwen3-{0.6,1.7,4}B-Base, which overrides the Qwen3.5 arms H02 names; brokkr-smithy owns that file and was told directly. Qwen3-4B-Base staged and benched to complete the family: Qwen3-0.6B-Base 0.616 B 1.707 s/step 2,399 tok/s spread 0.6% Qwen3-1.7B-Base 1.755 B 2.895 s/step 1,415 tok/s spread 0.8% Qwen3-4B-Base 4.089 B 5.714 s/step 717 tok/s spread 0.3% The dense 4.089 B carrier still trains 33% faster than the hybrid 0.765 B one. Projected per voice 2.7 / 4.6 / 9.1 h; the three-arm sweep at two seeds is ~33 h of GPU, ~10 h if H03's corpus floor holds. The three Qwen3.5 checkpoints stay staged so the decision is reversible behind an fla install. Also recorded: verified at 22:45-22:48 PT that nothing is training on gx10, ana-ml2, nh3-dev or irv-ml1, and that brokkr's own run07-gate close states "Nothing is owed. No battery to run." Run 7 has no servable artifact left. And a correction to a standing lesson -- the bracketed-class trick does not defeat a wrapper's argv, since the invoking shell's command line carries the literal pattern; observe the artifact instead. --- docs/pfi/author-voice-lora-regime.md | 70 +++++++++++++------ persistent-memory.md | 40 +++++++++-- .../bench-lora-step-gx10-2026-09-09.jsonl | 1 + 3 files changed, 86 insertions(+), 25 deletions(-) diff --git a/docs/pfi/author-voice-lora-regime.md b/docs/pfi/author-voice-lora-regime.md index ee2a151..70a8c7a 100644 --- a/docs/pfi/author-voice-lora-regime.md +++ b/docs/pfi/author-voice-lora-regime.md @@ -1,7 +1,9 @@ # Author-voice LoRA regime on pfi-gx10 — training-side prep -_Written 2026-09-09 22:45 PT. Status: **PREP. Nothing is training and nothing is -queued.** Two operator decisions open (§6)._ +_Written 2026-09-09 22:45 PT, revised 23:05 PT. Status: **PREP. Nothing is +training and nothing is queued.** Both operator decisions from the first draft are +now **SETTLED** (§6): carrier family is the **dense `Qwen3` line**, and the +intermediate checkpoints are purged._ The research target is **brokkr-smithy R49** (`research/R49-author-voice-adapters/`), whose seed is the operator's **BabyBronte** design doc @@ -58,9 +60,10 @@ Staged on local NVMe under `/home/infra-ops/carriers/` (pulled 2026-09-09, existence API-verified against the HF registry first, with a phantom repo run as the negative control): - Qwen3.5-0.8B-Base 1.7G Qwen3-0.6B-Base 1.2G - Qwen3.5-2B-Base 4.3G Qwen3-1.7B-Base 3.3G - Qwen3.5-4B-Base 8.8G + CHOSEN — dense Qwen3 held, not chosen — hybrid Qwen3.5 + Qwen3-0.6B-Base 1.2G Qwen3.5-0.8B-Base 1.7G + Qwen3-1.7B-Base 3.3G Qwen3.5-2B-Base 4.3G + Qwen3-4B-Base ~8G Qwen3.5-4B-Base 8.8G Probes live at `scripts/training-probes/{probe_carrier.py,bench_lora_step.py}` with raw output in `bench-lora-step-gx10-2026-09-09.jsonl`, so every number below @@ -127,6 +130,7 @@ measured after 3 warmup steps; median reported with the full spread. | `Qwen3.5-0.8B-Base` (no grad-ckpt) | " | 0.765 B | 6.364 | 644 | 38.9 GiB | 1.5% | | `Qwen3-0.6B-Base` | dense | 0.616 B | 1.707 | **2,399** | 9.8 GiB | 0.6% | | `Qwen3-1.7B-Base` | dense | 1.755 B | 2.895 | **1,415** | 12.2 GiB | 0.8% | +| `Qwen3-4B-Base` | dense | 4.089 B | 5.714 | **717** | 17.2 GiB | 0.3% | | `Qwen3-1.7B-Base`, batch 4 | dense | 1.755 B | 11.387 | 1,439 | 38.0 GiB | 0.6% | | `Qwen3.5-0.8B-Base`, batch 4 | hybrid | 0.765 B | 30.030 | 546 | 55.5 GiB | 0.7% | @@ -152,8 +156,14 @@ means the 2.6× gap is the kernel path, not a batching artefact. |---|---| | `Qwen3-0.6B-Base` dense | **2.7 h** | | `Qwen3-1.7B-Base` dense | **4.6 h** | +| `Qwen3-4B-Base` dense | **9.1 h** | | `Qwen3.5-0.8B-Base` hybrid | **12 h** | +The sharpest way to put it: **the dense 4.089 B carrier still trains 33% faster +than the hybrid 0.765 B one**, on 5.3× the parameters. The full three-arm dense +sweep at two seeds each is ~33 h of GPU — about a day and a half for the whole +H02 carrier question, and ~10 h if H03's ~300 k-word corpus floor holds. + The hybrid carrier would make a per-voice run *longer than the 7 h 26B-A4B tune it exists to replace.* At R49 H03's hoped-for corpus floor (~300 k words) the dense 1.7B lands near **1.4 h** — a voice per afternoon, which is the regime the @@ -163,7 +173,7 @@ first real corpus. ## 5. Prep remaining, in order -1. **Carrier family decision** (§6) — everything downstream keys on it. +1. ~~Carrier family decision~~ — **settled: dense `Qwen3`** (§6a). 2. **Corpus D1** — Gutenberg Brontë (Jane Eyre, Villette, Shirley, The Professor), boilerplate stripped, chapter-segmented, typography normalised, character inventory recorded. Public domain, clean under any disposition. @@ -195,22 +205,40 @@ integration contract, the modern in-copyright arsenal, inference latency. ## 6. Open for the operator -**(a) Carrier family — recommend the dense `Qwen3` line.** R49 H02 pins -`Qwen3.5-{0.8,2,4}B-Base`. The measurements say that family costs 2.6–6× the -wall-clock on this box, brings a vision tower and an MTP head to a text job, and -makes cross-document packing unsafe in 18 of 24 layers. `Qwen3-{0.6,1.7,4}B-Base` -— the design doc's own original pin — is plain dense, has none of those, and is -the better instrument for a probe whose whole point is isolating one variable. -The cost is one model generation of base quality. Reversible: the Qwen3.5 -checkpoints stay staged, and a fused-kernel install (`fla` is pure Triton and -would plausibly work on aarch64) could revive them later as a follow-up rather -than a blocker. +**(a) Carrier family — SETTLED 2026-09-09, operator: _"use dense qwen3"._** The +sweep is **`Qwen3-{0.6,1.7,4}B-Base`** — the design doc's own original pin, and +the family the measurements favour by 2.6–6×. **This overrides R49 H02's stated +arms**, which name `Qwen3.5-{0.8,2,4}B-Base`; brokkr-smithy owns that file and +has been told directly. -**(b) 45 GB of intermediate checkpoints on gx10, and 354 GB on ana-ml2.** -`run-03c/04/05/06/checkpoints` total 45 GB on gx10; `/tank/erp-tune/serve` is -354 GB of superseded merged models on ana-ml2. The final adapters are safe (§7) -and run 6 is the standing seat. Purging is the operator's call — 470 GB free on -gx10 means it does not block this regime. +What the ruling buys, restated so it is not re-litigated: no vision tower and no +MTP head to exclude, cross-document packing is safe again, `sdpa`/flash are both +reachable, and the per-voice wall-clock is 2.7–4.6 h instead of 12 h. What it +costs: one model generation of base quality. **Reversible** — the three Qwen3.5 +checkpoints stay staged (14.8 GB), and an `fla` install (pure Triton, plausibly +fine on aarch64) could revive that family as a follow-up experiment rather than a +prerequisite. If anyone re-opens this, re-run `bench_lora_step.py` first; the +argument is a measurement, not a preference. + +**(b) Intermediate checkpoints — PURGED 2026-09-09 23:00 PT, operator: +_"purge intermediate checkpoints"._** Seven `checkpoints/` directories deleted +with literal paths, one `rm` per line, after confirming none was a symlink and +that every run's final `adapter/` is an independent real directory: + + pfi-gx10 run-03c 11G · run-04 16G · run-05 9.2G · run-06 9.2G -> 45 GB + ana-ml2 run-01 12G · run-02 12G · run-03 5.9G -> 29 GB + +gx10 419G→374G used (496 GB free); `/tank/erp-tune` 392G→363G, with `zfs list -t +snapshot` empty so the space is genuinely returned rather than snapshot-held. +**All eight adapters re-verified by `sha256` after the deletion**, matching the +values recorded during the mirror. + +⏳ **Still standing, and it is the bigger prize — ~550 GB of superseded MERGED +models, deliberately NOT touched** because they are not checkpoints: +`gx10:~/erp-tune/serve/merged-run{03c,04,05,06}` at 49 GiB each (196 GiB) and +`ana-ml2:/tank/erp-tune/serve` (~354 GB). Run 6 is the standing seat, so +`merged-run06` plus `/tank/aimodels/erp-tune-v6-{bf16,nvfp4a16}` are the rollback +path and should survive; `merged-run03c/04/05` are dead weight. Separate call. ## 7. Adapter disposition — settled, and made real diff --git a/persistent-memory.md b/persistent-memory.md index 41f02d9..822c2a7 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-09 22:50 PT (**Pfish-6** = run-6 NVFP4 is the standing seat, ana-ml2 :8021 ONLY; run 7 PURGED ~139 GiB; pfi-gx10 is an experimental/TRAINING box and carries no serving seat; all five ERP adapters now MIRRORED to ana-ml2 and inside restic; **BabyBronte / R49 author-voice regime is in PREP on gx10** — carrier-family decision open)_ +_Last updated: 2026-09-09 23:10 PT (**Pfish-6** = run-6 NVFP4 is the standing seat, ana-ml2 :8021 ONLY; run 7 PURGED ~139 GiB; pfi-gx10 is an experimental/TRAINING box and carries no serving seat; all five ERP adapters now MIRRORED to ana-ml2 and inside restic; **BabyBronte / R49 author-voice regime is in PREP on gx10, carrier SETTLED = dense Qwen3**; intermediate checkpoints PURGED both boxes ~74 GB)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under an hour old, read it (it carries the in-flight @@ -129,6 +129,22 @@ preserved verbatim in `archival-memory.md` § Superseded in-flight snapshots._ already gone; do NOT `rm -rf /home/infra-ops/erp-tune/run-07` — operator ruled **keep the adapter** 2026-09-09. - **NO NEW *ERP* TRAINING RUN PLANNED.** The opening-split idea is not being re-tested; run 6 stands. +- **⚠ VERIFIED 2026-09-09 22:45–22:48 PT: NOTHING IS TRAINING ANYWHERE.** Checked gx10 (0 GPU compute + apps, 0% util, every pidfile dead), ana-ml2 (all 8 GPU processes are vLLM serving seats, no trainer), + nh3-dev, irv-ml1 (5 GPU processes, all the known native serving stacks). The last run was **run 7, + finished 13:23 PT**, and brokkr's own `research/R47-premium-corpus-gate/run07-gate/GATE-FAILED-run07.md` + (rev 19:40 PT) closes it: **"Nothing is owed. No battery to run."** Three findings there worth + holding: the CSAM gate fire was a **FALSE POSITIVE** (adjective `minor` in a HARD `child_term` tier, + fixed `cc42d76`, selftest 24/24) so the safety result is **VOID on an instrument defect, not a + failure**; run 7 is retired **on its merits** (primary flat 69→70.5, both diversity families reduced, + coherence 1.0→0.875), which is the operator's own ruling; and ⚠ **do NOT re-run that gate against + gx10:8098** — the artifact there changed and then went down entirely. ⚠ **Run 7 has NO servable + artifact left**: `serve/merged-run07` and `serve/pfish6-nvfp4a16` are both GONE, so "serve run 7" + means a fresh 49 GiB merge + 16 GiB requant, hours, on a tune already judged flat. +- **⚠ THE BRACKETED-CLASS pgrep TRICK DOES NOT SAVE A WRAPPER'S ARGV.** `ps -eo cmd | grep '[t]rain'` + still matched my own shell command, because the invoking `zsh -c '...'` argv contains the literal + bracketed pattern. The bracket only stops *grep itself* from matching. Durable form: **observe the + artifact** (GPU compute apps, the seat's own log) or exclude known PIDs — never the process name. - **✅ ALL FIVE gx10 ERP ADAPTERS ARE NOW TWO-COPY AND BACKED UP** (2026-09-09 22:30 PT, operator: *"keep the adapter"*). `run-03c/04/05/06/07` mirrored to `ana-ml2:/tank/erp-tune/run-/adapter` (the layout runs 01–03 already used), byte-totals identical both sides + `sha256` match on every @@ -136,6 +152,17 @@ preserved verbatim in `archival-memory.md` § Superseded in-flight snapshots._ regenerable weights), so `configs/restic/ana-ml2/profiles.yaml` gained ONE documented carve-out — `/tank/erp-tune/run-*/adapter` — verified by `resticprofile --dry-run` to expand to exactly those 8 paths and nothing else. Live file is byte-identical to the repo canonical; `.bak-20260909` beside it. +- **🧹 INTERMEDIATE CHECKPOINTS PURGED 2026-09-09 23:00 PT** (operator: *"purge intermediate + checkpoints"*). Seven `checkpoints/` dirs, deleted with LITERAL paths one `rm` per line after + confirming none was a symlink and that every final `adapter/` is an independent real dir: gx10 + `run-03c` 11G + `run-04` 16G + `run-05` 9.2G + `run-06` 9.2G = **45 GB** (419G→374G used, 496 GB + free); ana-ml2 `/tank/erp-tune/run-01` 12G + `run-02` 12G + `run-03` 5.9G = **29 GB** + (`/tank/erp-tune` 392G→363G, `zfs list -t snapshot` EMPTY so the space is genuinely returned). + **~74 GB total. All eight adapters re-verified by sha256 AFTER the deletion**, matching the mirror. + ⏳ **NOT touched, and it is the bigger prize (~550 GB): superseded MERGED models** — they are not + checkpoints, so I stayed literal. gx10 `serve/merged-run{03c,04,05,06}` 49 GiB each (196 GiB) + + ana-ml2 `/tank/erp-tune/serve` ~354 GB. Keep `merged-run06` + `/tank/aimodels/erp-tune-v6-*` (the + standing seat's rollback); `merged-run03c/04/05` are dead weight. Separate operator call. - **🖋 BabyBronte / R49 author-voice LoRA regime — IN PREP on pfi-gx10, nothing training.** Plan + every measured number: [`docs/pfi/author-voice-lora-regime.md`](docs/pfi/author-voice-lora-regime.md). Research target is **brokkr-smithy R49** (`research/R49-author-voice-adapters/`) — brokkr owns @@ -158,9 +185,14 @@ preserved verbatim in `archival-memory.md` § Superseded in-flight snapshots._ sampled-target-changed check in the SERVING path); and **cross-document packing is unsafe** because SSM state ignores the attention mask, which breaks the per-copy name-consistency invariant the design doc calls sacred. None of these exist on the dense line. - **⭐ OPEN FOR THE OPERATOR: carrier family — recommend the dense `Qwen3-{0.6,1.7,4}B-Base` line** - (the design doc's own original pin) over R49 H02's `Qwen3.5` trio. Reversible: Qwen3.5 stays staged - and an `fla` install (pure Triton, plausibly fine on aarch64) could revive it as a follow-up. + **✅ CARRIER SETTLED 2026-09-09 — operator: *"use dense qwen3"*.** Sweep is + **`Qwen3-{0.6,1.7,4}B-Base`**, all three staged on gx10. ⚠ **This OVERRIDES R49 H02's stated arms** + (which name the Qwen3.5 trio) — brokkr-smithy owns that file and was told directly. Full dense sweep + measured: 0.6B **2,399**, 1.7B **1,415**, 4B **717** tok/s (spread 0.3–0.8%) → projected **2.7 / 4.6 / + 9.1 h** per voice; whole 3-arm × 2-seed sweep ≈ 33 h GPU, ≈10 h if H03's ~300k-word floor holds. The + sharpest framing: **dense 4.089B still beats hybrid 0.765B by 33%** on 5.3x the params. Qwen3.5 stays + staged (14.8 GB) — an `fla` install could revive it as a follow-up experiment, never a prerequisite. + If anyone re-opens this, re-run `bench_lora_step.py` first: the argument is a measurement. - **📌 forseti shipped althing 3.6.1 (2026-09-09) — every box's herald needs a restart to pick up the new poke text.** `uv tool install --force --reinstall ` + `systemctl --user restart althing-po-herald`, expect `3.6.1`; the plugin also went to 0.1.6 (inbox.md no longer calls the diff --git a/scripts/training-probes/bench-lora-step-gx10-2026-09-09.jsonl b/scripts/training-probes/bench-lora-step-gx10-2026-09-09.jsonl index ae5247e..5513029 100644 --- a/scripts/training-probes/bench-lora-step-gx10-2026-09-09.jsonl +++ b/scripts/training-probes/bench-lora-step-gx10-2026-09-09.jsonl @@ -4,3 +4,4 @@ {"model": "Qwen3-1.7B-Base", "total_params_B": 1.755, "lora_rank": 32, "targets": "attn_mlp", "adapted_modules": 196, "trainable_params_M": 34.87, "trainable_pct": 1.986, "batch": 1, "seq": 4096, "tokens_per_microbatch": 4096, "grad_checkpointing": true, "attn_impl": "sdpa", "dtype": "bfloat16", "device": "NVIDIA GB10", "torch": "2.14.0+cu130", "warmup": 3, "n": 10, "s_per_step_median": 2.8946, "s_per_step_min": 2.8782, "s_per_step_max": 2.901, "s_per_step_spread_pct": 0.8, "tok_per_s_median": 1415.0, "peak_mem_GiB": 12.24} {"model": "Qwen3-1.7B-Base", "total_params_B": 1.755, "lora_rank": 32, "targets": "attn_mlp", "adapted_modules": 196, "trainable_params_M": 34.87, "trainable_pct": 1.986, "batch": 4, "seq": 4096, "tokens_per_microbatch": 16384, "grad_checkpointing": true, "attn_impl": "sdpa", "dtype": "bfloat16", "device": "NVIDIA GB10", "torch": "2.14.0+cu130", "warmup": 3, "n": 10, "s_per_step_median": 11.3872, "s_per_step_min": 11.3401, "s_per_step_max": 11.4128, "s_per_step_spread_pct": 0.6, "tok_per_s_median": 1438.8, "peak_mem_GiB": 37.99} {"model": "Qwen3.5-0.8B-Base", "total_params_B": 0.765, "lora_rank": 32, "targets": "attn_mlp", "adapted_modules": 96, "trainable_params_M": 12.78, "trainable_pct": 1.67, "batch": 4, "seq": 4096, "tokens_per_microbatch": 16384, "grad_checkpointing": true, "attn_impl": "sdpa", "dtype": "bfloat16", "device": "NVIDIA GB10", "torch": "2.14.0+cu130", "warmup": 3, "n": 10, "s_per_step_median": 30.0301, "s_per_step_min": 29.9379, "s_per_step_max": 30.1444, "s_per_step_spread_pct": 0.7, "tok_per_s_median": 545.6, "peak_mem_GiB": 55.49} +{"model": "Qwen3-4B-Base", "total_params_B": 4.089, "lora_rank": 32, "targets": "attn_mlp", "adapted_modules": 252, "trainable_params_M": 66.06, "trainable_pct": 1.616, "batch": 1, "seq": 4096, "tokens_per_microbatch": 4096, "grad_checkpointing": true, "attn_impl": "sdpa", "dtype": "bfloat16", "device": "NVIDIA GB10", "torch": "2.14.0+cu130", "warmup": 3, "n": 10, "s_per_step_median": 5.7145, "s_per_step_min": 5.7085, "s_per_step_max": 5.727, "s_per_step_spread_pct": 0.3, "tok_per_s_median": 716.8, "peak_mem_GiB": 17.22}