vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(r49-prep): author-voice LoRA regime prep on gx10 — carriers staged, throughput measured, adapters secured

Browse Source 
Prep for the BabyBronte / brokkr-smithy R49 author-voice adapter regime, plus
the operator's "keep the adapter" ruling made durable.

Measured on pfi-gx10 (GB10, sm_121), n=10 per arm after 3 warmup steps, seq
4096, LoRA r=32 on q/k/v/o + MLP, bf16, sdpa, grad-checkpointing on:

  Qwen3-0.6B-Base    dense    0.616 B   1.707 s/step   2,399 tok/s
  Qwen3-1.7B-Base    dense    1.755 B   2.895 s/step   1,415 tok/s
  Qwen3.5-0.8B-Base  hybrid   0.765 B   7.581 s/step     540 tok/s

The dense 1.755 B carrier trains 2.6x faster than the hybrid 0.765 B one on 2.3x
the parameters (~6x per parameter), with more LoRA modules adapted (196 vs 96).
Spreads of 0.6-2.6% put instrument noise an order of magnitude below the effect.
Cause: Qwen3.5 is 18 linear-attention (SSM) layers to 6 attention, and no fused
linear-attention kernel is installed on the box. Grad checkpointing is not the
culprit (19%, and saves 2.6x memory). Batching is not the lever for either
family -- both sit at this box's roofline at batch 1.

Projected per voice on a Brontë-scale corpus: dense 0.6B 2.7 h, dense 1.7B
4.6 h, hybrid 0.8B 12 h. The hybrid would take longer than the 7 h 26B-A4B tune
the regime exists to replace, so the carrier family is now an open decision with
a recommendation for the dense Qwen3 line -- the design doc's original pin.

Two further Qwen3.5 findings, both measured rather than read off the config: the
Base checkpoints ship a vision tower (153/297 model.visual.* Linear tensors that
target_modules="all-linear" would train on text) and an MTP head, both dropped
for free by loading through AutoModelForCausalLM -- which renames modules
relative to the vLLM serving path, so adapter binding needs the
sampled-target-changed check on the serving side; and cross-document packing is
unsafe because SSM state ignores the attention mask, breaking the per-copy
name-consistency invariant the design doc calls sacred. Neither exists on dense.

Adapter disposition, per the operator's ruling: all five gx10-resident ERP
adapters (run-03c/04/05/06/07) mirrored to ana-ml2:/tank/erp-tune/run-<N>/adapter
matching the layout runs 01-03 already used, byte-totals identical both sides and
sha256 matching on every adapter_model.safetensors. /tank/* is deliberately
excluded from ana-ml2's restic sources, so the profile gains one documented
carve-out for /tank/erp-tune/run-*/adapter, verified by resticprofile --dry-run
to expand to exactly those eight paths.

Nothing is training and nothing is queued.
This commit is contained in:
vh
2026-09-09 22:41:47 -07:00
parent 92cf09e397
commit 7db6c44bcd
6 changed files with 445 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/pfi/author-voice-lora-regime.md
+229
View File
@@ -0,0 +1,229 @@
# Author-voice LoRA regime on pfi-gx10 — training-side prep
_Written 2026-09-09 22:45 PT. Status: **PREP. Nothing is training and nothing is
queued.** Two operator decisions open (§6)._
The research target is **brokkr-smithy R49** (`research/R49-author-voice-adapters/`),
whose seed is the operator's **BabyBronte** design doc
(`R49/reference/babybronte-design-doc.md`, 2026-08-25). R49 owns the hypotheses,
the instruments and the adjudication. **This document owns the other half — the
box, the stack, the corpus staging, the trainer, the launcher, and the
wall-clock** — the same split that ran ERP-seat runs 3c through 7 on this box.
Read `R49/target.md` before touching the corpus design; several attractive ideas
are already settled *against* there and re-proposing them is the failure mode
this file exists to prevent.
---
## 1. What the regime is, in one paragraph
A voice is a **LoRA adapter on a small non-instruct base model**, trained on that
author's real prose, steered at serving time by a terse beat line rather than a
prompt. The completion in every training pair is 100% authentic author text; the
only synthetic token in the corpus is the beat line, and beat lines are
**loss-masked**, so the distribution the adapter learns to *emit* is the author's
and the style ceiling is the author's own. Names and places are substituted
**deterministically** — seeded RNG over a curated 23,398-name dictionary, never
an LLM — which is what lets one work become 5–8 training copies without teaching
plot memorisation.
**The value being bought is marginal cost per voice, not inference latency.**
That framing is the operator's and it is load-bearing: the anchor for
"expensive" is ERP run 2 at ~7 h for a single 26B-A4B tune on a harness whose
audit found a blocking-mask defect, a vision tower a leaf-name regex would have
trained on text, and 128 experts fused per layer. A dense sub-2B carrier has
none of those failure modes, and at this size the **methodology floor is finally
cheaper than the shortcut** — two seeds per arm and a re-run after every change
are routine rather than unaffordable.
## 2. The box, and what is already staged
`pfi-gx10` (10.100.50.60) — ASUS Ascent GX10, NVIDIA **GB10**, `sm_121`,
aarch64, **121 GB unified** memory, 916 GB NVMe, 470 GB free. Operator ruling
2026-09-09: **experimental box, primarily for training, no serving seat.** Its
GPU is idle.
Training stack already present and current at `/home/infra-ops/ml/.venv`:
| | |
|---|---|
| torch | `2.14.0+cu130`, `torch.cuda.get_device_capability() == (12, 1)` |
| transformers | 5.16.1 (loads `Qwen3_5Config` natively) |
| peft / trl / accelerate | 0.20.0 / 1.12.0 / 1.14.0 |
| datasets / bitsandbytes | 5.0.1 / 0.50.2 |
| **absent** | `mamba_ssm`, `causal_conv1d`, `fla`, `flash_attn`, `kernels` (triton 3.8.0 present) |
Staged on local NVMe under `/home/infra-ops/carriers/` (pulled 2026-09-09,
existence API-verified against the HF registry first, with a phantom repo run as
the negative control):
Qwen3.5-0.8B-Base 1.7G Qwen3-0.6B-Base 1.2G
Qwen3.5-2B-Base 4.3G Qwen3-1.7B-Base 3.3G
Qwen3.5-4B-Base 8.8G
Probes live at `scripts/training-probes/{probe_carrier.py,bench_lora_step.py}`
with raw output in `bench-lora-step-gx10-2026-09-09.jsonl`, so every number below
can be re-derived rather than taken on faith.
## 3. What the carriers actually are — measured, not read off the model card
R49 H02 names the `Qwen3.5` trio. Probing the checkpoints rather than the config
found three things worth knowing before writing a recipe.
**They ship a vision tower and an MTP head.** `model.visual.*` is 153 tensors on
the 0.8B and 297 on the 2B — `attn.qkv`, `attn.proj`, `mlp.linear_fc1/2`, all
`nn.Linear` leaves that `target_modules="all-linear"` would attach LoRA to and
then train on pure text. This is the *same* defect the ERP harness audit caught
on gemma-4. **Mitigation is free:** loading through `AutoModelForCausalLM`
returns `Qwen3_5ForCausalLM` with the vision tower and MTP head dropped
entirely — 0.752 B of text model, module paths `model.layers.N.*`.
⚠ **That mitigation creates a serving trap.** vLLM will load the full
`Qwen3_5ForConditionalGeneration`, where the same weights live at
`model.language_model.layers.N.*`. An adapter trained against the CausalLM
prefix may not bind. **Pre-flight:** load the finished adapter in the serving
path and confirm a sampled target tensor actually changed — the same silent-no-op
check the ERP merge step already uses.
**Three quarters of the layers are not attention.** `layer_types` is 3×
`linear_attention` + 1× `full_attention` repeating: 18 SSM / 6 attention at
0.8B and 2B, 24 / 8 at 4B. The SSM blocks carry `conv1d`, `A_log`, `dt_bias` and
five Linear projections; `mamba_ssm_dtype` is `float32`.
| region | 0.8B | 2B | 4B | LoRA-able leaves |
|---|---|---|---|---|
| MLP | 35.1% | 48.1% | 53.9% | `gate_proj`, `up_proj`, `down_proj` |
| embeddings (tied) | 33.8% | 27.0% | 15.1% | `lm_head` — exclude |
| linear-attn (SSM) | 25.2% | 20.1% | 24.0% | `in_proj_{qkv,a,b,z}`, `out_proj` |
| full attention | 5.9% | 4.7% | 7.0% | `q_proj`, `k_proj`, `v_proj`, `o_proj` |
| **total** | **0.752 B** | **1.882 B** | **4.206 B** | |
The conventional `q,k,v,o` + MLP recipe therefore covers **41%** of the 0.8B and
leaves the SSM stack untouched in 18 of 24 layers. Adding the SSM Linears takes
coverage to 66%. At this scale that is a cheap ablation, not an agonising choice.
⚠ **Packing across document boundaries is unsafe on this architecture.** An SSM
layer carries recurrent state along the sequence and an attention mask does not
reset it, so two renamed copies packed into one 8k window can bleed in 18 of 24
layers — which is precisely the per-copy name-consistency invariant the design
doc calls sacred. Either one document per sequence, or prove the trainer's
sequence-boundary signal is honoured by the linear-attn path. Under a dense
carrier this problem does not exist.
**The tied embedding is a third of the small carrier.** vocab 248,320 × hidden
1024 = 254 M of the 0.752 B. The transformer body being tested at the small end
is ~0.50 B, which matters when reporting "the carrier floor".
## 4. Throughput — and the newest carrier is the slow one
One forward+backward+AdamW microbatch, LoRA r=32/α=64 on `q,k,v,o` + MLP,
bf16, `sdpa`, gradient checkpointing on, seq 4096, on gx10's GB10. n=10
measured after 3 warmup steps; median reported with the full spread.
| carrier | architecture | params | s/step | tok/s | peak | spread |
|---|---|---|---|---|---|---|
| `Qwen3.5-0.8B-Base` | hybrid, 18 SSM / 6 attn | 0.765 B | 7.581 | **540** | 15.1 GiB | 2.6% |
| `Qwen3.5-0.8B-Base` (no grad-ckpt) | " | 0.765 B | 6.364 | 644 | 38.9 GiB | 1.5% |
| `Qwen3-0.6B-Base` | dense | 0.616 B | 1.707 | **2,399** | 9.8 GiB | 0.6% |
| `Qwen3-1.7B-Base` | dense | 1.755 B | 2.895 | **1,415** | 12.2 GiB | 0.8% |
| `Qwen3-1.7B-Base`, batch 4 | dense | 1.755 B | 11.387 | 1,439 | 38.0 GiB | 0.6% |
| `Qwen3.5-0.8B-Base`, batch 4 | hybrid | 0.765 B | 30.030 | 546 | 55.5 GiB | 0.7% |
**The dense 1.755 B carrier trains 2.6× faster than the hybrid 0.765 B one** — on
2.3× the parameters, with *more* LoRA modules adapted (196 vs 96, because dense
has real attention in every layer). Per parameter the dense path is ~6× more
efficient. Spreads of 0.6–2.6% across n=10 put the instrument's noise an order of
magnitude below the effect, so this is not variance.
The cause is almost certainly that **no fused linear-attention kernel is
installed** (§2) so the SSM path runs a reference implementation. Grad
checkpointing is *not* the culprit — turning it off recovers only 19% and costs
2.6× the memory, so leave it on. Batching is not the lever for either family: 1,415 → 1,439
tok/s dense and 540 → 546 tok/s hybrid from batch 1 to 4. **Both architectures
are already at this box's roofline at batch 1**, which is a bandwidth story
(GB10's unified LPDDR5X against an RTX PRO 6000's ~6.6× higher figure) — and it
means the 2.6× gap is the kernel path, not a batching artefact.
**What that does to the regime's premise.** Projecting a Brontë-scale corpus
(~1 M words ≈ 1.3 M tokens, × 6 rename copies, 3 epochs ≈ 23 M tokens):
| carrier | projected wall-clock per voice |
|---|---|
| `Qwen3-0.6B-Base` dense | **2.7 h** |
| `Qwen3-1.7B-Base` dense | **4.6 h** |
| `Qwen3.5-0.8B-Base` hybrid | **12 h** |
The hybrid carrier would make a per-voice run *longer than the 7 h 26B-A4B tune
it exists to replace.* At R49 H03's hoped-for corpus floor (~300 k words) the
dense 1.7B lands near **1.4 h** — a voice per afternoon, which is the regime the
operator asked for. ⚠ These are projections from a synthetic-token throughput
harness, not from a completed run; treat them as sizing, and re-measure on the
first real corpus.
## 5. Prep remaining, in order
1. **Carrier family decision** (§6) — everything downstream keys on it.
2. **Corpus D1** — Gutenberg Brontë (Jane Eyre, Villette, Shirley, The
Professor), boilerplate stripped, chapter-segmented, typography normalised,
character inventory recorded. Public domain, clean under any disposition.
3. **Re-point the R49 deterministic machinery at Brontë.** The entity detector
(corpus-level capitalised-vs-lowercase ratio), identity linking, gender
resolution and the 23,398-name dictionary were all built and hardened against
a *Yarros* sample. Per-work re-derivation needed: entity map, alphabet, and
the `UNRESOLVED_BLOCKING` human pass (~20–40 entities per work).
4. **Beat annotation (D4)** via `gen` inverse-prompting, using F02's hardened
prompt (banned meta-language, three PD worked examples, ≤20-word gate).
5. **Trainer.** `erp_sft_harness` is chat-shaped and carries ERP-specific
eligibility machinery; the author-voice job is plain continuation with a
masked prefix. Decision: a small purpose-built trainer that *keeps* the
harness's §4 disciplines (provenance pin, order manifest, truncation report,
cache key that sees semantic changes, recorded attention backend) rather than
a fork of its corpus logic.
6. **Pre-set the decision threshold before collecting data**, wider than the
measured seed-to-seed spread, per the R49 charter — and run the positive
control the R49 journal's own lesson demands: confirm the stylometric
instrument separates real Brontë from unadapted base output *before* it is
asked to judge an adapter.
7. **Launcher** modelled on `launch-run-07.sh` — its guards were each bought with
a past failure (GPU-clear assertion, pidfile not `pgrep -f`, refuse an
existing log, free-space floor, `setsid` detach).
Deliberately out of scope here, per R49: the Director/critic loop, style
arithmetic and the Pelican test, multi-LoRA arsenal serving, the Skaldsong
integration contract, the modern in-copyright arsenal, inference latency.
## 6. Open for the operator
**(a) Carrier family — recommend the dense `Qwen3` line.** R49 H02 pins
`Qwen3.5-{0.8,2,4}B-Base`. The measurements say that family costs 2.6–6× the
wall-clock on this box, brings a vision tower and an MTP head to a text job, and
makes cross-document packing unsafe in 18 of 24 layers. `Qwen3-{0.6,1.7,4}B-Base`
— the design doc's own original pin — is plain dense, has none of those, and is
the better instrument for a probe whose whole point is isolating one variable.
The cost is one model generation of base quality. Reversible: the Qwen3.5
checkpoints stay staged, and a fused-kernel install (`fla` is pure Triton and
would plausibly work on aarch64) could revive them later as a follow-up rather
than a blocker.
**(b) 45 GB of intermediate checkpoints on gx10, and 354 GB on ana-ml2.**
`run-03c/04/05/06/checkpoints` total 45 GB on gx10; `/tank/erp-tune/serve` is
354 GB of superseded merged models on ana-ml2. The final adapters are safe (§7)
and run 6 is the standing seat. Purging is the operator's call — 470 GB free on
gx10 means it does not block this regime.
## 7. Adapter disposition — settled, and made real
Operator, 2026-09-09: **keep the adapter.** As of 22:30 PT all five
gx10-resident ERP adapters are mirrored to `ana-ml2:/tank/erp-tune/run-<N>/adapter`,
matching the layout runs 01–03 already use there, byte-total identical on both
sides and `sha256` matching on every `adapter_model.safetensors`:
run-03c run-04 run-05 run-06 run-07 315 MB each, 8 files each
`/tank/*` is deliberately **excluded** from ana-ml2's restic sources — terabytes
of regenerable model weights. A trained adapter is the one thing under there
upstream cannot hand back, so `configs/restic/ana-ml2/profiles.yaml` now carries
a single documented carve-out, `/tank/erp-tune/run-*/adapter`, verified by
`resticprofile --dry-run` to expand to exactly those eight paths and nothing
else. The nightly 01:00 run picks them up.