feat(r49-prep): author-voice LoRA regime prep on gx10 — carriers staged, throughput measured, adapters secured
Prep for the BabyBronte / brokkr-smithy R49 author-voice adapter regime, plus the operator's "keep the adapter" ruling made durable. Measured on pfi-gx10 (GB10, sm_121), n=10 per arm after 3 warmup steps, seq 4096, LoRA r=32 on q/k/v/o + MLP, bf16, sdpa, grad-checkpointing on: Qwen3-0.6B-Base dense 0.616 B 1.707 s/step 2,399 tok/s Qwen3-1.7B-Base dense 1.755 B 2.895 s/step 1,415 tok/s Qwen3.5-0.8B-Base hybrid 0.765 B 7.581 s/step 540 tok/s The dense 1.755 B carrier trains 2.6x faster than the hybrid 0.765 B one on 2.3x the parameters (~6x per parameter), with more LoRA modules adapted (196 vs 96). Spreads of 0.6-2.6% put instrument noise an order of magnitude below the effect. Cause: Qwen3.5 is 18 linear-attention (SSM) layers to 6 attention, and no fused linear-attention kernel is installed on the box. Grad checkpointing is not the culprit (19%, and saves 2.6x memory). Batching is not the lever for either family -- both sit at this box's roofline at batch 1. Projected per voice on a Brontë-scale corpus: dense 0.6B 2.7 h, dense 1.7B 4.6 h, hybrid 0.8B 12 h. The hybrid would take longer than the 7 h 26B-A4B tune the regime exists to replace, so the carrier family is now an open decision with a recommendation for the dense Qwen3 line -- the design doc's original pin. Two further Qwen3.5 findings, both measured rather than read off the config: the Base checkpoints ship a vision tower (153/297 model.visual.* Linear tensors that target_modules="all-linear" would train on text) and an MTP head, both dropped for free by loading through AutoModelForCausalLM -- which renames modules relative to the vLLM serving path, so adapter binding needs the sampled-target-changed check on the serving side; and cross-document packing is unsafe because SSM state ignores the attention mask, breaking the per-copy name-consistency invariant the design doc calls sacred. Neither exists on dense. Adapter disposition, per the operator's ruling: all five gx10-resident ERP adapters (run-03c/04/05/06/07) mirrored to ana-ml2:/tank/erp-tune/run-<N>/adapter matching the layout runs 01-03 already used, byte-totals identical both sides and sha256 matching on every adapter_model.safetensors. /tank/* is deliberately excluded from ana-ml2's restic sources, so the profile gains one documented carve-out for /tank/erp-tune/run-*/adapter, verified by resticprofile --dry-run to expand to exactly those eight paths. Nothing is training and nothing is queued.
This commit is contained in:
+42
-4
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-09-09 22:05 PT (**Pfish-6** = run-6 NVFP4 is the standing seat, ana-ml2 :8021 ONLY; run 7 PURGED ~139 GiB; pfi-gx10 is an experimental/TRAINING box and carries no serving seat; no new run planned)_
|
||||
_Last updated: 2026-09-09 22:50 PT (**Pfish-6** = run-6 NVFP4 is the standing seat, ana-ml2 :8021 ONLY; run 7 PURGED ~139 GiB; pfi-gx10 is an experimental/TRAINING box and carries no serving seat; all five ERP adapters now MIRRORED to ana-ml2 and inside restic; **BabyBronte / R49 author-voice regime is in PREP on gx10** — carrier-family decision open)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under an hour old, read it (it carries the in-flight
|
||||
@@ -125,9 +125,47 @@ preserved verbatim in `archival-memory.md` § Superseded in-flight snapshots._
|
||||
`run-07/checkpoints` 9.2 GiB, `serve/pfish6-nvfp4a16` 16 GiB; ana-ml2 `erp-tune-v7-bf16` 49 GiB,
|
||||
`erp-tune-v7-nvfp4a16` 16 GiB, `erp-tune-v7-quant-work`. **~139 GiB reclaimed** (gx10 53%→47%).
|
||||
⚠ **KEPT deliberately: `~/erp-tune/run-07/adapter` 315 MB + provenance + `loss-series-r7.json`** —
|
||||
the only non-reproducible piece (14 h of training), and it costs nothing. `rm -rf
|
||||
/home/infra-ops/erp-tune/run-07` finishes the job if wanted; everything else run-7 is already gone.
|
||||
- **NO NEW TRAINING RUN PLANNED.** The opening-split idea is not being re-tested; run 6 stands.
|
||||
the only non-reproducible piece (14 h of training), and it costs nothing. Everything else run-7 is
|
||||
already gone; do NOT `rm -rf /home/infra-ops/erp-tune/run-07` — operator ruled **keep the adapter**
|
||||
2026-09-09.
|
||||
- **NO NEW *ERP* TRAINING RUN PLANNED.** The opening-split idea is not being re-tested; run 6 stands.
|
||||
- **✅ ALL FIVE gx10 ERP ADAPTERS ARE NOW TWO-COPY AND BACKED UP** (2026-09-09 22:30 PT, operator:
|
||||
*"keep the adapter"*). `run-03c/04/05/06/07` mirrored to `ana-ml2:/tank/erp-tune/run-<N>/adapter`
|
||||
(the layout runs 01–03 already used), byte-totals identical both sides + `sha256` match on every
|
||||
`adapter_model.safetensors`. ⚠ `/tank/*` is deliberately OUT of ana-ml2's restic sources (TB of
|
||||
regenerable weights), so `configs/restic/ana-ml2/profiles.yaml` gained ONE documented carve-out —
|
||||
`/tank/erp-tune/run-*/adapter` — verified by `resticprofile --dry-run` to expand to exactly those 8
|
||||
paths and nothing else. Live file is byte-identical to the repo canonical; `.bak-20260909` beside it.
|
||||
- **🖋 BabyBronte / R49 author-voice LoRA regime — IN PREP on pfi-gx10, nothing training.** Plan +
|
||||
every measured number: [`docs/pfi/author-voice-lora-regime.md`](docs/pfi/author-voice-lora-regime.md).
|
||||
Research target is **brokkr-smithy R49** (`research/R49-author-voice-adapters/`) — brokkr owns
|
||||
hypotheses/instruments/adjudication, infra-ops owns box+corpus staging+trainer+wall-clock, same split
|
||||
as ERP runs 3c–7. Five carriers staged on gx10 `~/carriers/` (Qwen3.5-0.8/2/4B-Base +
|
||||
Qwen3-0.6/1.7B-Base); probes committed at `scripts/training-probes/{probe_carrier,bench_lora_step}.py`.
|
||||
⚠⚠ **THE HEADLINE, AND IT INVERTS R49's PIN: the newest carrier is the SLOW one.** Measured on gx10
|
||||
(n=10, spread 0.6–2.6%, seq 4096, r=32 attn+mlp, bf16 sdpa, grad-ckpt on): dense `Qwen3-1.7B-Base`
|
||||
**1,415 tok/s** vs hybrid `Qwen3.5-0.8B-Base` **540 tok/s** — the dense model is 2.6x faster on 2.3x
|
||||
the parameters (~6x per-param), because Qwen3.5 is **18 SSM / 6 attention layers** and NO fused
|
||||
linear-attention kernel is installed (`mamba_ssm`/`causal_conv1d`/`fla` all absent; triton 3.8 is
|
||||
there). Projected per voice: dense 0.6B **2.7 h**, dense 1.7B **4.6 h**, hybrid 0.8B **12 h** — the
|
||||
hybrid would take LONGER than the 7 h 26B-A4B tune it exists to replace, killing the regime's whole
|
||||
premise. Grad-ckpt is not the cause (19%, and saves 2.6x memory — keep it); batching is not the lever
|
||||
(dense 1,415→1,439 and hybrid 540→546 tok/s at batch 4 — BOTH already at this box's roofline at batch 1, a bandwidth story, so the 2.6x gap is the kernel path not a batching artefact).
|
||||
⚠ Two more Qwen3.5 landmines, both measured: it **ships a vision tower** (153/297 `model.visual.*`
|
||||
Linear tensors that `all-linear` would train on text — the same defect the gemma-4 audit caught;
|
||||
`AutoModelForCausalLM` drops it and the MTP head for free, BUT that renames modules
|
||||
`model.layers.N.*` vs vLLM's `model.language_model.layers.N.*`, so an adapter may not bind → run the
|
||||
sampled-target-changed check in the SERVING path); and **cross-document packing is unsafe** because
|
||||
SSM state ignores the attention mask, which breaks the per-copy name-consistency invariant the design
|
||||
doc calls sacred. None of these exist on the dense line.
|
||||
**⭐ OPEN FOR THE OPERATOR: carrier family — recommend the dense `Qwen3-{0.6,1.7,4}B-Base` line**
|
||||
(the design doc's own original pin) over R49 H02's `Qwen3.5` trio. Reversible: Qwen3.5 stays staged
|
||||
and an `fla` install (pure Triton, plausibly fine on aarch64) could revive it as a follow-up.
|
||||
- **📌 forseti shipped althing 3.6.1 (2026-09-09) — every box's herald needs a restart to pick up the
|
||||
new poke text.** `uv tool install --force --reinstall <checkout>` + `systemctl --user restart
|
||||
althing-po-herald`, expect `3.6.1`; the plugin also went to 0.1.6 (inbox.md no longer calls the
|
||||
result reply "optional"). NOT DONE — deferred, nothing is blocked on it, and it is a fleet-wide
|
||||
multi-box pass with its own verification burden.
|
||||
- **⏳ ana-ml2 storage follow-ups, operator's call** (the three actions themselves LANDED 09-09 02:02):
|
||||
(a) **ZFS pool-health ALERTING** — `tank` sat DEGRADED 04-23→09-05 with nvme7 physically absent and
|
||||
nobody knew (ZED mails `root`, no MTA on the box); (b) nvme7 / slot 0-5 keep-vs-replace —
|
||||
|
||||
Reference in New Issue
Block a user