chore(erp-tune): purge intermediate checkpoints (~74 GB); R49 carrier settled on dense Qwen3
Two operator rulings, 2026-09-09.
"purge intermediate checkpoints" -- seven checkpoints/ directories removed with
literal paths, one rm per line, after confirming none was a symlink and that
every run's final adapter/ is an independent real directory:
pfi-gx10 run-03c 11G run-04 16G run-05 9.2G run-06 9.2G = 45 GB
ana-ml2 run-01 12G run-02 12G run-03 5.9G = 29 GB
gx10 419G->374G used, 496 GB free. /tank/erp-tune 392G->363G with zfs list -t
snapshot empty, so the space is genuinely returned rather than snapshot-held. All
eight adapters re-verified by sha256 after the deletion, matching the values
recorded during the mirror. Merged artifacts deliberately untouched -- they are
not checkpoints, and the ~550 GB of superseded merges stays a separate call.
"use dense qwen3" -- the R49 H02 carrier sweep becomes Qwen3-{0.6,1.7,4}B-Base,
which overrides the Qwen3.5 arms H02 names; brokkr-smithy owns that file and was
told directly. Qwen3-4B-Base staged and benched to complete the family:
Qwen3-0.6B-Base 0.616 B 1.707 s/step 2,399 tok/s spread 0.6%
Qwen3-1.7B-Base 1.755 B 2.895 s/step 1,415 tok/s spread 0.8%
Qwen3-4B-Base 4.089 B 5.714 s/step 717 tok/s spread 0.3%
The dense 4.089 B carrier still trains 33% faster than the hybrid 0.765 B one.
Projected per voice 2.7 / 4.6 / 9.1 h; the three-arm sweep at two seeds is ~33 h
of GPU, ~10 h if H03's corpus floor holds. The three Qwen3.5 checkpoints stay
staged so the decision is reversible behind an fla install.
Also recorded: verified at 22:45-22:48 PT that nothing is training on gx10,
ana-ml2, nh3-dev or irv-ml1, and that brokkr's own run07-gate close states
"Nothing is owed. No battery to run." Run 7 has no servable artifact left. And a
correction to a standing lesson -- the bracketed-class trick does not defeat a
wrapper's argv, since the invoking shell's command line carries the literal
pattern; observe the artifact instead.
This commit is contained in:
@@ -1,7 +1,9 @@
|
||||
# Author-voice LoRA regime on pfi-gx10 — training-side prep
|
||||
|
||||
_Written 2026-09-09 22:45 PT. Status: **PREP. Nothing is training and nothing is
|
||||
queued.** Two operator decisions open (§6)._
|
||||
_Written 2026-09-09 22:45 PT, revised 23:05 PT. Status: **PREP. Nothing is
|
||||
training and nothing is queued.** Both operator decisions from the first draft are
|
||||
now **SETTLED** (§6): carrier family is the **dense `Qwen3` line**, and the
|
||||
intermediate checkpoints are purged._
|
||||
|
||||
The research target is **brokkr-smithy R49** (`research/R49-author-voice-adapters/`),
|
||||
whose seed is the operator's **BabyBronte** design doc
|
||||
@@ -58,9 +60,10 @@ Staged on local NVMe under `/home/infra-ops/carriers/` (pulled 2026-09-09,
|
||||
existence API-verified against the HF registry first, with a phantom repo run as
|
||||
the negative control):
|
||||
|
||||
Qwen3.5-0.8B-Base 1.7G Qwen3-0.6B-Base 1.2G
|
||||
Qwen3.5-2B-Base 4.3G Qwen3-1.7B-Base 3.3G
|
||||
Qwen3.5-4B-Base 8.8G
|
||||
CHOSEN — dense Qwen3 held, not chosen — hybrid Qwen3.5
|
||||
Qwen3-0.6B-Base 1.2G Qwen3.5-0.8B-Base 1.7G
|
||||
Qwen3-1.7B-Base 3.3G Qwen3.5-2B-Base 4.3G
|
||||
Qwen3-4B-Base ~8G Qwen3.5-4B-Base 8.8G
|
||||
|
||||
Probes live at `scripts/training-probes/{probe_carrier.py,bench_lora_step.py}`
|
||||
with raw output in `bench-lora-step-gx10-2026-09-09.jsonl`, so every number below
|
||||
@@ -127,6 +130,7 @@ measured after 3 warmup steps; median reported with the full spread.
|
||||
| `Qwen3.5-0.8B-Base` (no grad-ckpt) | " | 0.765 B | 6.364 | 644 | 38.9 GiB | 1.5% |
|
||||
| `Qwen3-0.6B-Base` | dense | 0.616 B | 1.707 | **2,399** | 9.8 GiB | 0.6% |
|
||||
| `Qwen3-1.7B-Base` | dense | 1.755 B | 2.895 | **1,415** | 12.2 GiB | 0.8% |
|
||||
| `Qwen3-4B-Base` | dense | 4.089 B | 5.714 | **717** | 17.2 GiB | 0.3% |
|
||||
| `Qwen3-1.7B-Base`, batch 4 | dense | 1.755 B | 11.387 | 1,439 | 38.0 GiB | 0.6% |
|
||||
| `Qwen3.5-0.8B-Base`, batch 4 | hybrid | 0.765 B | 30.030 | 546 | 55.5 GiB | 0.7% |
|
||||
|
||||
@@ -152,8 +156,14 @@ means the 2.6× gap is the kernel path, not a batching artefact.
|
||||
|---|---|
|
||||
| `Qwen3-0.6B-Base` dense | **2.7 h** |
|
||||
| `Qwen3-1.7B-Base` dense | **4.6 h** |
|
||||
| `Qwen3-4B-Base` dense | **9.1 h** |
|
||||
| `Qwen3.5-0.8B-Base` hybrid | **12 h** |
|
||||
|
||||
The sharpest way to put it: **the dense 4.089 B carrier still trains 33% faster
|
||||
than the hybrid 0.765 B one**, on 5.3× the parameters. The full three-arm dense
|
||||
sweep at two seeds each is ~33 h of GPU — about a day and a half for the whole
|
||||
H02 carrier question, and ~10 h if H03's ~300 k-word corpus floor holds.
|
||||
|
||||
The hybrid carrier would make a per-voice run *longer than the 7 h 26B-A4B tune
|
||||
it exists to replace.* At R49 H03's hoped-for corpus floor (~300 k words) the
|
||||
dense 1.7B lands near **1.4 h** — a voice per afternoon, which is the regime the
|
||||
@@ -163,7 +173,7 @@ first real corpus.
|
||||
|
||||
## 5. Prep remaining, in order
|
||||
|
||||
1. **Carrier family decision** (§6) — everything downstream keys on it.
|
||||
1. ~~Carrier family decision~~ — **settled: dense `Qwen3`** (§6a).
|
||||
2. **Corpus D1** — Gutenberg Brontë (Jane Eyre, Villette, Shirley, The
|
||||
Professor), boilerplate stripped, chapter-segmented, typography normalised,
|
||||
character inventory recorded. Public domain, clean under any disposition.
|
||||
@@ -195,22 +205,40 @@ integration contract, the modern in-copyright arsenal, inference latency.
|
||||
|
||||
## 6. Open for the operator
|
||||
|
||||
**(a) Carrier family — recommend the dense `Qwen3` line.** R49 H02 pins
|
||||
`Qwen3.5-{0.8,2,4}B-Base`. The measurements say that family costs 2.6–6× the
|
||||
wall-clock on this box, brings a vision tower and an MTP head to a text job, and
|
||||
makes cross-document packing unsafe in 18 of 24 layers. `Qwen3-{0.6,1.7,4}B-Base`
|
||||
— the design doc's own original pin — is plain dense, has none of those, and is
|
||||
the better instrument for a probe whose whole point is isolating one variable.
|
||||
The cost is one model generation of base quality. Reversible: the Qwen3.5
|
||||
checkpoints stay staged, and a fused-kernel install (`fla` is pure Triton and
|
||||
would plausibly work on aarch64) could revive them later as a follow-up rather
|
||||
than a blocker.
|
||||
**(a) Carrier family — SETTLED 2026-09-09, operator: _"use dense qwen3"._** The
|
||||
sweep is **`Qwen3-{0.6,1.7,4}B-Base`** — the design doc's own original pin, and
|
||||
the family the measurements favour by 2.6–6×. **This overrides R49 H02's stated
|
||||
arms**, which name `Qwen3.5-{0.8,2,4}B-Base`; brokkr-smithy owns that file and
|
||||
has been told directly.
|
||||
|
||||
**(b) 45 GB of intermediate checkpoints on gx10, and 354 GB on ana-ml2.**
|
||||
`run-03c/04/05/06/checkpoints` total 45 GB on gx10; `/tank/erp-tune/serve` is
|
||||
354 GB of superseded merged models on ana-ml2. The final adapters are safe (§7)
|
||||
and run 6 is the standing seat. Purging is the operator's call — 470 GB free on
|
||||
gx10 means it does not block this regime.
|
||||
What the ruling buys, restated so it is not re-litigated: no vision tower and no
|
||||
MTP head to exclude, cross-document packing is safe again, `sdpa`/flash are both
|
||||
reachable, and the per-voice wall-clock is 2.7–4.6 h instead of 12 h. What it
|
||||
costs: one model generation of base quality. **Reversible** — the three Qwen3.5
|
||||
checkpoints stay staged (14.8 GB), and an `fla` install (pure Triton, plausibly
|
||||
fine on aarch64) could revive that family as a follow-up experiment rather than a
|
||||
prerequisite. If anyone re-opens this, re-run `bench_lora_step.py` first; the
|
||||
argument is a measurement, not a preference.
|
||||
|
||||
**(b) Intermediate checkpoints — PURGED 2026-09-09 23:00 PT, operator:
|
||||
_"purge intermediate checkpoints"._** Seven `checkpoints/` directories deleted
|
||||
with literal paths, one `rm` per line, after confirming none was a symlink and
|
||||
that every run's final `adapter/` is an independent real directory:
|
||||
|
||||
pfi-gx10 run-03c 11G · run-04 16G · run-05 9.2G · run-06 9.2G -> 45 GB
|
||||
ana-ml2 run-01 12G · run-02 12G · run-03 5.9G -> 29 GB
|
||||
|
||||
gx10 419G→374G used (496 GB free); `/tank/erp-tune` 392G→363G, with `zfs list -t
|
||||
snapshot` empty so the space is genuinely returned rather than snapshot-held.
|
||||
**All eight adapters re-verified by `sha256` after the deletion**, matching the
|
||||
values recorded during the mirror.
|
||||
|
||||
⏳ **Still standing, and it is the bigger prize — ~550 GB of superseded MERGED
|
||||
models, deliberately NOT touched** because they are not checkpoints:
|
||||
`gx10:~/erp-tune/serve/merged-run{03c,04,05,06}` at 49 GiB each (196 GiB) and
|
||||
`ana-ml2:/tank/erp-tune/serve` (~354 GB). Run 6 is the standing seat, so
|
||||
`merged-run06` plus `/tank/aimodels/erp-tune-v6-{bf16,nvfp4a16}` are the rollback
|
||||
path and should survive; `merged-run03c/04/05` are dead weight. Separate call.
|
||||
|
||||
## 7. Adapter disposition — settled, and made real
|
||||
|
||||
|
||||
Reference in New Issue
Block a user