feat(r49-prep): author-voice LoRA regime prep on gx10 — carriers staged, throughput measured, adapters secured
Prep for the BabyBronte / brokkr-smithy R49 author-voice adapter regime, plus the operator's "keep the adapter" ruling made durable. Measured on pfi-gx10 (GB10, sm_121), n=10 per arm after 3 warmup steps, seq 4096, LoRA r=32 on q/k/v/o + MLP, bf16, sdpa, grad-checkpointing on: Qwen3-0.6B-Base dense 0.616 B 1.707 s/step 2,399 tok/s Qwen3-1.7B-Base dense 1.755 B 2.895 s/step 1,415 tok/s Qwen3.5-0.8B-Base hybrid 0.765 B 7.581 s/step 540 tok/s The dense 1.755 B carrier trains 2.6x faster than the hybrid 0.765 B one on 2.3x the parameters (~6x per parameter), with more LoRA modules adapted (196 vs 96). Spreads of 0.6-2.6% put instrument noise an order of magnitude below the effect. Cause: Qwen3.5 is 18 linear-attention (SSM) layers to 6 attention, and no fused linear-attention kernel is installed on the box. Grad checkpointing is not the culprit (19%, and saves 2.6x memory). Batching is not the lever for either family -- both sit at this box's roofline at batch 1. Projected per voice on a Brontë-scale corpus: dense 0.6B 2.7 h, dense 1.7B 4.6 h, hybrid 0.8B 12 h. The hybrid would take longer than the 7 h 26B-A4B tune the regime exists to replace, so the carrier family is now an open decision with a recommendation for the dense Qwen3 line -- the design doc's original pin. Two further Qwen3.5 findings, both measured rather than read off the config: the Base checkpoints ship a vision tower (153/297 model.visual.* Linear tensors that target_modules="all-linear" would train on text) and an MTP head, both dropped for free by loading through AutoModelForCausalLM -- which renames modules relative to the vLLM serving path, so adapter binding needs the sampled-target-changed check on the serving side; and cross-document packing is unsafe because SSM state ignores the attention mask, breaking the per-copy name-consistency invariant the design doc calls sacred. Neither exists on dense. Adapter disposition, per the operator's ruling: all five gx10-resident ERP adapters (run-03c/04/05/06/07) mirrored to ana-ml2:/tank/erp-tune/run-<N>/adapter matching the layout runs 01-03 already used, byte-totals identical both sides and sha256 matching on every adapter_model.safetensors. /tank/* is deliberately excluded from ana-ml2's restic sources, so the profile gains one documented carve-out for /tank/erp-tune/run-*/adapter, verified by resticprofile --dry-run to expand to exactly those eight paths. Nothing is training and nothing is queued.
This commit is contained in:
@@ -11,8 +11,12 @@
|
||||
# - /tank/* is NOT in source paths. That's ~TB of model weights (HF
|
||||
# caches, llama.cpp GGUFs, ComfyUI models, etc.) — all regenerable
|
||||
# from upstream. Backing them up would blow the repo size budget.
|
||||
# ONE carve-out: /tank/erp-tune/run-*/adapter. A trained LoRA adapter is
|
||||
# the only thing under /tank that upstream cannot hand back — each is
|
||||
# ~300 MB and cost 7-14 h of GPU time, and the ERP line's adapters exist
|
||||
# nowhere else but pfi-gx10's single NVMe and here. ~2.4 GB total.
|
||||
# - No DB dumps needed. None of the stacks on this host (llama-swap,
|
||||
# vllm, comfyui, kokoro, parakeet, vibevoice, beszel-agent,
|
||||
# vllm-qwen3, comfyui, kokoro, parakeet, vibevoice, beszel-agent,
|
||||
# dozzle-agent, dockge) store relational data.
|
||||
|
||||
version: "1"
|
||||
@@ -40,6 +44,7 @@ default:
|
||||
- /etc # host config (systemd units, chrony, apparmor, ssh, etc.)
|
||||
- /root # root shell history, ssh keys, any ad-hoc scripts
|
||||
- /var/lib/docker/volumes # named volumes (small; models live on /tank, not here)
|
||||
- /tank/erp-tune/run-*/adapter # non-regenerable trained LoRA adapters (see header)
|
||||
exclude:
|
||||
# Docker internals we never want in a backup
|
||||
- /var/lib/docker/volumes/backingFsBlockDev
|
||||
|
||||
@@ -0,0 +1,229 @@
|
||||
# Author-voice LoRA regime on pfi-gx10 — training-side prep
|
||||
|
||||
_Written 2026-09-09 22:45 PT. Status: **PREP. Nothing is training and nothing is
|
||||
queued.** Two operator decisions open (§6)._
|
||||
|
||||
The research target is **brokkr-smithy R49** (`research/R49-author-voice-adapters/`),
|
||||
whose seed is the operator's **BabyBronte** design doc
|
||||
(`R49/reference/babybronte-design-doc.md`, 2026-08-25). R49 owns the hypotheses,
|
||||
the instruments and the adjudication. **This document owns the other half — the
|
||||
box, the stack, the corpus staging, the trainer, the launcher, and the
|
||||
wall-clock** — the same split that ran ERP-seat runs 3c through 7 on this box.
|
||||
|
||||
Read `R49/target.md` before touching the corpus design; several attractive ideas
|
||||
are already settled *against* there and re-proposing them is the failure mode
|
||||
this file exists to prevent.
|
||||
|
||||
---
|
||||
|
||||
## 1. What the regime is, in one paragraph
|
||||
|
||||
A voice is a **LoRA adapter on a small non-instruct base model**, trained on that
|
||||
author's real prose, steered at serving time by a terse beat line rather than a
|
||||
prompt. The completion in every training pair is 100% authentic author text; the
|
||||
only synthetic token in the corpus is the beat line, and beat lines are
|
||||
**loss-masked**, so the distribution the adapter learns to *emit* is the author's
|
||||
and the style ceiling is the author's own. Names and places are substituted
|
||||
**deterministically** — seeded RNG over a curated 23,398-name dictionary, never
|
||||
an LLM — which is what lets one work become 5–8 training copies without teaching
|
||||
plot memorisation.
|
||||
|
||||
**The value being bought is marginal cost per voice, not inference latency.**
|
||||
That framing is the operator's and it is load-bearing: the anchor for
|
||||
"expensive" is ERP run 2 at ~7 h for a single 26B-A4B tune on a harness whose
|
||||
audit found a blocking-mask defect, a vision tower a leaf-name regex would have
|
||||
trained on text, and 128 experts fused per layer. A dense sub-2B carrier has
|
||||
none of those failure modes, and at this size the **methodology floor is finally
|
||||
cheaper than the shortcut** — two seeds per arm and a re-run after every change
|
||||
are routine rather than unaffordable.
|
||||
|
||||
## 2. The box, and what is already staged
|
||||
|
||||
`pfi-gx10` (10.100.50.60) — ASUS Ascent GX10, NVIDIA **GB10**, `sm_121`,
|
||||
aarch64, **121 GB unified** memory, 916 GB NVMe, 470 GB free. Operator ruling
|
||||
2026-09-09: **experimental box, primarily for training, no serving seat.** Its
|
||||
GPU is idle.
|
||||
|
||||
Training stack already present and current at `/home/infra-ops/ml/.venv`:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| torch | `2.14.0+cu130`, `torch.cuda.get_device_capability() == (12, 1)` |
|
||||
| transformers | 5.16.1 (loads `Qwen3_5Config` natively) |
|
||||
| peft / trl / accelerate | 0.20.0 / 1.12.0 / 1.14.0 |
|
||||
| datasets / bitsandbytes | 5.0.1 / 0.50.2 |
|
||||
| **absent** | `mamba_ssm`, `causal_conv1d`, `fla`, `flash_attn`, `kernels` (triton 3.8.0 present) |
|
||||
|
||||
Staged on local NVMe under `/home/infra-ops/carriers/` (pulled 2026-09-09,
|
||||
existence API-verified against the HF registry first, with a phantom repo run as
|
||||
the negative control):
|
||||
|
||||
Qwen3.5-0.8B-Base 1.7G Qwen3-0.6B-Base 1.2G
|
||||
Qwen3.5-2B-Base 4.3G Qwen3-1.7B-Base 3.3G
|
||||
Qwen3.5-4B-Base 8.8G
|
||||
|
||||
Probes live at `scripts/training-probes/{probe_carrier.py,bench_lora_step.py}`
|
||||
with raw output in `bench-lora-step-gx10-2026-09-09.jsonl`, so every number below
|
||||
can be re-derived rather than taken on faith.
|
||||
|
||||
## 3. What the carriers actually are — measured, not read off the model card
|
||||
|
||||
R49 H02 names the `Qwen3.5` trio. Probing the checkpoints rather than the config
|
||||
found three things worth knowing before writing a recipe.
|
||||
|
||||
**They ship a vision tower and an MTP head.** `model.visual.*` is 153 tensors on
|
||||
the 0.8B and 297 on the 2B — `attn.qkv`, `attn.proj`, `mlp.linear_fc1/2`, all
|
||||
`nn.Linear` leaves that `target_modules="all-linear"` would attach LoRA to and
|
||||
then train on pure text. This is the *same* defect the ERP harness audit caught
|
||||
on gemma-4. **Mitigation is free:** loading through `AutoModelForCausalLM`
|
||||
returns `Qwen3_5ForCausalLM` with the vision tower and MTP head dropped
|
||||
entirely — 0.752 B of text model, module paths `model.layers.N.*`.
|
||||
|
||||
⚠ **That mitigation creates a serving trap.** vLLM will load the full
|
||||
`Qwen3_5ForConditionalGeneration`, where the same weights live at
|
||||
`model.language_model.layers.N.*`. An adapter trained against the CausalLM
|
||||
prefix may not bind. **Pre-flight:** load the finished adapter in the serving
|
||||
path and confirm a sampled target tensor actually changed — the same silent-no-op
|
||||
check the ERP merge step already uses.
|
||||
|
||||
**Three quarters of the layers are not attention.** `layer_types` is 3×
|
||||
`linear_attention` + 1× `full_attention` repeating: 18 SSM / 6 attention at
|
||||
0.8B and 2B, 24 / 8 at 4B. The SSM blocks carry `conv1d`, `A_log`, `dt_bias` and
|
||||
five Linear projections; `mamba_ssm_dtype` is `float32`.
|
||||
|
||||
| region | 0.8B | 2B | 4B | LoRA-able leaves |
|
||||
|---|---|---|---|---|
|
||||
| MLP | 35.1% | 48.1% | 53.9% | `gate_proj`, `up_proj`, `down_proj` |
|
||||
| embeddings (tied) | 33.8% | 27.0% | 15.1% | `lm_head` — exclude |
|
||||
| linear-attn (SSM) | 25.2% | 20.1% | 24.0% | `in_proj_{qkv,a,b,z}`, `out_proj` |
|
||||
| full attention | 5.9% | 4.7% | 7.0% | `q_proj`, `k_proj`, `v_proj`, `o_proj` |
|
||||
| **total** | **0.752 B** | **1.882 B** | **4.206 B** | |
|
||||
|
||||
The conventional `q,k,v,o` + MLP recipe therefore covers **41%** of the 0.8B and
|
||||
leaves the SSM stack untouched in 18 of 24 layers. Adding the SSM Linears takes
|
||||
coverage to 66%. At this scale that is a cheap ablation, not an agonising choice.
|
||||
|
||||
⚠ **Packing across document boundaries is unsafe on this architecture.** An SSM
|
||||
layer carries recurrent state along the sequence and an attention mask does not
|
||||
reset it, so two renamed copies packed into one 8k window can bleed in 18 of 24
|
||||
layers — which is precisely the per-copy name-consistency invariant the design
|
||||
doc calls sacred. Either one document per sequence, or prove the trainer's
|
||||
sequence-boundary signal is honoured by the linear-attn path. Under a dense
|
||||
carrier this problem does not exist.
|
||||
|
||||
**The tied embedding is a third of the small carrier.** vocab 248,320 × hidden
|
||||
1024 = 254 M of the 0.752 B. The transformer body being tested at the small end
|
||||
is ~0.50 B, which matters when reporting "the carrier floor".
|
||||
|
||||
## 4. Throughput — and the newest carrier is the slow one
|
||||
|
||||
One forward+backward+AdamW microbatch, LoRA r=32/α=64 on `q,k,v,o` + MLP,
|
||||
bf16, `sdpa`, gradient checkpointing on, seq 4096, on gx10's GB10. n=10
|
||||
measured after 3 warmup steps; median reported with the full spread.
|
||||
|
||||
| carrier | architecture | params | s/step | tok/s | peak | spread |
|
||||
|---|---|---|---|---|---|---|
|
||||
| `Qwen3.5-0.8B-Base` | hybrid, 18 SSM / 6 attn | 0.765 B | 7.581 | **540** | 15.1 GiB | 2.6% |
|
||||
| `Qwen3.5-0.8B-Base` (no grad-ckpt) | " | 0.765 B | 6.364 | 644 | 38.9 GiB | 1.5% |
|
||||
| `Qwen3-0.6B-Base` | dense | 0.616 B | 1.707 | **2,399** | 9.8 GiB | 0.6% |
|
||||
| `Qwen3-1.7B-Base` | dense | 1.755 B | 2.895 | **1,415** | 12.2 GiB | 0.8% |
|
||||
| `Qwen3-1.7B-Base`, batch 4 | dense | 1.755 B | 11.387 | 1,439 | 38.0 GiB | 0.6% |
|
||||
| `Qwen3.5-0.8B-Base`, batch 4 | hybrid | 0.765 B | 30.030 | 546 | 55.5 GiB | 0.7% |
|
||||
|
||||
**The dense 1.755 B carrier trains 2.6× faster than the hybrid 0.765 B one** — on
|
||||
2.3× the parameters, with *more* LoRA modules adapted (196 vs 96, because dense
|
||||
has real attention in every layer). Per parameter the dense path is ~6× more
|
||||
efficient. Spreads of 0.6–2.6% across n=10 put the instrument's noise an order of
|
||||
magnitude below the effect, so this is not variance.
|
||||
|
||||
The cause is almost certainly that **no fused linear-attention kernel is
|
||||
installed** (§2) so the SSM path runs a reference implementation. Grad
|
||||
checkpointing is *not* the culprit — turning it off recovers only 19% and costs
|
||||
2.6× the memory, so leave it on. Batching is not the lever for either family: 1,415 → 1,439
|
||||
tok/s dense and 540 → 546 tok/s hybrid from batch 1 to 4. **Both architectures
|
||||
are already at this box's roofline at batch 1**, which is a bandwidth story
|
||||
(GB10's unified LPDDR5X against an RTX PRO 6000's ~6.6× higher figure) — and it
|
||||
means the 2.6× gap is the kernel path, not a batching artefact.
|
||||
|
||||
**What that does to the regime's premise.** Projecting a Brontë-scale corpus
|
||||
(~1 M words ≈ 1.3 M tokens, × 6 rename copies, 3 epochs ≈ 23 M tokens):
|
||||
|
||||
| carrier | projected wall-clock per voice |
|
||||
|---|---|
|
||||
| `Qwen3-0.6B-Base` dense | **2.7 h** |
|
||||
| `Qwen3-1.7B-Base` dense | **4.6 h** |
|
||||
| `Qwen3.5-0.8B-Base` hybrid | **12 h** |
|
||||
|
||||
The hybrid carrier would make a per-voice run *longer than the 7 h 26B-A4B tune
|
||||
it exists to replace.* At R49 H03's hoped-for corpus floor (~300 k words) the
|
||||
dense 1.7B lands near **1.4 h** — a voice per afternoon, which is the regime the
|
||||
operator asked for. ⚠ These are projections from a synthetic-token throughput
|
||||
harness, not from a completed run; treat them as sizing, and re-measure on the
|
||||
first real corpus.
|
||||
|
||||
## 5. Prep remaining, in order
|
||||
|
||||
1. **Carrier family decision** (§6) — everything downstream keys on it.
|
||||
2. **Corpus D1** — Gutenberg Brontë (Jane Eyre, Villette, Shirley, The
|
||||
Professor), boilerplate stripped, chapter-segmented, typography normalised,
|
||||
character inventory recorded. Public domain, clean under any disposition.
|
||||
3. **Re-point the R49 deterministic machinery at Brontë.** The entity detector
|
||||
(corpus-level capitalised-vs-lowercase ratio), identity linking, gender
|
||||
resolution and the 23,398-name dictionary were all built and hardened against
|
||||
a *Yarros* sample. Per-work re-derivation needed: entity map, alphabet, and
|
||||
the `UNRESOLVED_BLOCKING` human pass (~20–40 entities per work).
|
||||
4. **Beat annotation (D4)** via `gen` inverse-prompting, using F02's hardened
|
||||
prompt (banned meta-language, three PD worked examples, ≤20-word gate).
|
||||
5. **Trainer.** `erp_sft_harness` is chat-shaped and carries ERP-specific
|
||||
eligibility machinery; the author-voice job is plain continuation with a
|
||||
masked prefix. Decision: a small purpose-built trainer that *keeps* the
|
||||
harness's §4 disciplines (provenance pin, order manifest, truncation report,
|
||||
cache key that sees semantic changes, recorded attention backend) rather than
|
||||
a fork of its corpus logic.
|
||||
6. **Pre-set the decision threshold before collecting data**, wider than the
|
||||
measured seed-to-seed spread, per the R49 charter — and run the positive
|
||||
control the R49 journal's own lesson demands: confirm the stylometric
|
||||
instrument separates real Brontë from unadapted base output *before* it is
|
||||
asked to judge an adapter.
|
||||
7. **Launcher** modelled on `launch-run-07.sh` — its guards were each bought with
|
||||
a past failure (GPU-clear assertion, pidfile not `pgrep -f`, refuse an
|
||||
existing log, free-space floor, `setsid` detach).
|
||||
|
||||
Deliberately out of scope here, per R49: the Director/critic loop, style
|
||||
arithmetic and the Pelican test, multi-LoRA arsenal serving, the Skaldsong
|
||||
integration contract, the modern in-copyright arsenal, inference latency.
|
||||
|
||||
## 6. Open for the operator
|
||||
|
||||
**(a) Carrier family — recommend the dense `Qwen3` line.** R49 H02 pins
|
||||
`Qwen3.5-{0.8,2,4}B-Base`. The measurements say that family costs 2.6–6× the
|
||||
wall-clock on this box, brings a vision tower and an MTP head to a text job, and
|
||||
makes cross-document packing unsafe in 18 of 24 layers. `Qwen3-{0.6,1.7,4}B-Base`
|
||||
— the design doc's own original pin — is plain dense, has none of those, and is
|
||||
the better instrument for a probe whose whole point is isolating one variable.
|
||||
The cost is one model generation of base quality. Reversible: the Qwen3.5
|
||||
checkpoints stay staged, and a fused-kernel install (`fla` is pure Triton and
|
||||
would plausibly work on aarch64) could revive them later as a follow-up rather
|
||||
than a blocker.
|
||||
|
||||
**(b) 45 GB of intermediate checkpoints on gx10, and 354 GB on ana-ml2.**
|
||||
`run-03c/04/05/06/checkpoints` total 45 GB on gx10; `/tank/erp-tune/serve` is
|
||||
354 GB of superseded merged models on ana-ml2. The final adapters are safe (§7)
|
||||
and run 6 is the standing seat. Purging is the operator's call — 470 GB free on
|
||||
gx10 means it does not block this regime.
|
||||
|
||||
## 7. Adapter disposition — settled, and made real
|
||||
|
||||
Operator, 2026-09-09: **keep the adapter.** As of 22:30 PT all five
|
||||
gx10-resident ERP adapters are mirrored to `ana-ml2:/tank/erp-tune/run-<N>/adapter`,
|
||||
matching the layout runs 01–03 already use there, byte-total identical on both
|
||||
sides and `sha256` matching on every `adapter_model.safetensors`:
|
||||
|
||||
run-03c run-04 run-05 run-06 run-07 315 MB each, 8 files each
|
||||
|
||||
`/tank/*` is deliberately **excluded** from ana-ml2's restic sources — terabytes
|
||||
of regenerable model weights. A trained adapter is the one thing under there
|
||||
upstream cannot hand back, so `configs/restic/ana-ml2/profiles.yaml` now carries
|
||||
a single documented carve-out, `/tank/erp-tune/run-*/adapter`, verified by
|
||||
`resticprofile --dry-run` to expand to exactly those eight paths and nothing
|
||||
else. The nightly 01:00 run picks them up.
|
||||
+42
-4
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-09-09 22:05 PT (**Pfish-6** = run-6 NVFP4 is the standing seat, ana-ml2 :8021 ONLY; run 7 PURGED ~139 GiB; pfi-gx10 is an experimental/TRAINING box and carries no serving seat; no new run planned)_
|
||||
_Last updated: 2026-09-09 22:50 PT (**Pfish-6** = run-6 NVFP4 is the standing seat, ana-ml2 :8021 ONLY; run 7 PURGED ~139 GiB; pfi-gx10 is an experimental/TRAINING box and carries no serving seat; all five ERP adapters now MIRRORED to ana-ml2 and inside restic; **BabyBronte / R49 author-voice regime is in PREP on gx10** — carrier-family decision open)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under an hour old, read it (it carries the in-flight
|
||||
@@ -125,9 +125,47 @@ preserved verbatim in `archival-memory.md` § Superseded in-flight snapshots._
|
||||
`run-07/checkpoints` 9.2 GiB, `serve/pfish6-nvfp4a16` 16 GiB; ana-ml2 `erp-tune-v7-bf16` 49 GiB,
|
||||
`erp-tune-v7-nvfp4a16` 16 GiB, `erp-tune-v7-quant-work`. **~139 GiB reclaimed** (gx10 53%→47%).
|
||||
⚠ **KEPT deliberately: `~/erp-tune/run-07/adapter` 315 MB + provenance + `loss-series-r7.json`** —
|
||||
the only non-reproducible piece (14 h of training), and it costs nothing. `rm -rf
|
||||
/home/infra-ops/erp-tune/run-07` finishes the job if wanted; everything else run-7 is already gone.
|
||||
- **NO NEW TRAINING RUN PLANNED.** The opening-split idea is not being re-tested; run 6 stands.
|
||||
the only non-reproducible piece (14 h of training), and it costs nothing. Everything else run-7 is
|
||||
already gone; do NOT `rm -rf /home/infra-ops/erp-tune/run-07` — operator ruled **keep the adapter**
|
||||
2026-09-09.
|
||||
- **NO NEW *ERP* TRAINING RUN PLANNED.** The opening-split idea is not being re-tested; run 6 stands.
|
||||
- **✅ ALL FIVE gx10 ERP ADAPTERS ARE NOW TWO-COPY AND BACKED UP** (2026-09-09 22:30 PT, operator:
|
||||
*"keep the adapter"*). `run-03c/04/05/06/07` mirrored to `ana-ml2:/tank/erp-tune/run-<N>/adapter`
|
||||
(the layout runs 01–03 already used), byte-totals identical both sides + `sha256` match on every
|
||||
`adapter_model.safetensors`. ⚠ `/tank/*` is deliberately OUT of ana-ml2's restic sources (TB of
|
||||
regenerable weights), so `configs/restic/ana-ml2/profiles.yaml` gained ONE documented carve-out —
|
||||
`/tank/erp-tune/run-*/adapter` — verified by `resticprofile --dry-run` to expand to exactly those 8
|
||||
paths and nothing else. Live file is byte-identical to the repo canonical; `.bak-20260909` beside it.
|
||||
- **🖋 BabyBronte / R49 author-voice LoRA regime — IN PREP on pfi-gx10, nothing training.** Plan +
|
||||
every measured number: [`docs/pfi/author-voice-lora-regime.md`](docs/pfi/author-voice-lora-regime.md).
|
||||
Research target is **brokkr-smithy R49** (`research/R49-author-voice-adapters/`) — brokkr owns
|
||||
hypotheses/instruments/adjudication, infra-ops owns box+corpus staging+trainer+wall-clock, same split
|
||||
as ERP runs 3c–7. Five carriers staged on gx10 `~/carriers/` (Qwen3.5-0.8/2/4B-Base +
|
||||
Qwen3-0.6/1.7B-Base); probes committed at `scripts/training-probes/{probe_carrier,bench_lora_step}.py`.
|
||||
⚠⚠ **THE HEADLINE, AND IT INVERTS R49's PIN: the newest carrier is the SLOW one.** Measured on gx10
|
||||
(n=10, spread 0.6–2.6%, seq 4096, r=32 attn+mlp, bf16 sdpa, grad-ckpt on): dense `Qwen3-1.7B-Base`
|
||||
**1,415 tok/s** vs hybrid `Qwen3.5-0.8B-Base` **540 tok/s** — the dense model is 2.6x faster on 2.3x
|
||||
the parameters (~6x per-param), because Qwen3.5 is **18 SSM / 6 attention layers** and NO fused
|
||||
linear-attention kernel is installed (`mamba_ssm`/`causal_conv1d`/`fla` all absent; triton 3.8 is
|
||||
there). Projected per voice: dense 0.6B **2.7 h**, dense 1.7B **4.6 h**, hybrid 0.8B **12 h** — the
|
||||
hybrid would take LONGER than the 7 h 26B-A4B tune it exists to replace, killing the regime's whole
|
||||
premise. Grad-ckpt is not the cause (19%, and saves 2.6x memory — keep it); batching is not the lever
|
||||
(dense 1,415→1,439 and hybrid 540→546 tok/s at batch 4 — BOTH already at this box's roofline at batch 1, a bandwidth story, so the 2.6x gap is the kernel path not a batching artefact).
|
||||
⚠ Two more Qwen3.5 landmines, both measured: it **ships a vision tower** (153/297 `model.visual.*`
|
||||
Linear tensors that `all-linear` would train on text — the same defect the gemma-4 audit caught;
|
||||
`AutoModelForCausalLM` drops it and the MTP head for free, BUT that renames modules
|
||||
`model.layers.N.*` vs vLLM's `model.language_model.layers.N.*`, so an adapter may not bind → run the
|
||||
sampled-target-changed check in the SERVING path); and **cross-document packing is unsafe** because
|
||||
SSM state ignores the attention mask, which breaks the per-copy name-consistency invariant the design
|
||||
doc calls sacred. None of these exist on the dense line.
|
||||
**⭐ OPEN FOR THE OPERATOR: carrier family — recommend the dense `Qwen3-{0.6,1.7,4}B-Base` line**
|
||||
(the design doc's own original pin) over R49 H02's `Qwen3.5` trio. Reversible: Qwen3.5 stays staged
|
||||
and an `fla` install (pure Triton, plausibly fine on aarch64) could revive it as a follow-up.
|
||||
- **📌 forseti shipped althing 3.6.1 (2026-09-09) — every box's herald needs a restart to pick up the
|
||||
new poke text.** `uv tool install --force --reinstall <checkout>` + `systemctl --user restart
|
||||
althing-po-herald`, expect `3.6.1`; the plugin also went to 0.1.6 (inbox.md no longer calls the
|
||||
result reply "optional"). NOT DONE — deferred, nothing is blocked on it, and it is a fleet-wide
|
||||
multi-box pass with its own verification burden.
|
||||
- **⏳ ana-ml2 storage follow-ups, operator's call** (the three actions themselves LANDED 09-09 02:02):
|
||||
(a) **ZFS pool-health ALERTING** — `tank` sat DEGRADED 04-23→09-05 with nvme7 physically absent and
|
||||
nobody knew (ZED mails `root`, no MTA on the box); (b) nvme7 / slot 0-5 keep-vs-replace —
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
{"model": "Qwen3.5-0.8B-Base", "total_params_B": 0.765, "lora_rank": 32, "targets": "attn_mlp", "adapted_modules": 96, "trainable_params_M": 12.78, "trainable_pct": 1.67, "batch": 1, "seq": 4096, "tokens_per_microbatch": 4096, "grad_checkpointing": true, "attn_impl": "sdpa", "dtype": "bfloat16", "device": "NVIDIA GB10", "torch": "2.14.0+cu130", "warmup": 3, "n": 10, "s_per_step_median": 7.5814, "s_per_step_min": 7.4864, "s_per_step_max": 7.6843, "s_per_step_spread_pct": 2.6, "tok_per_s_median": 540.3, "peak_mem_GiB": 15.1}
|
||||
{"model": "Qwen3.5-0.8B-Base", "total_params_B": 0.765, "lora_rank": 32, "targets": "attn_mlp", "adapted_modules": 96, "trainable_params_M": 12.78, "trainable_pct": 1.67, "batch": 1, "seq": 4096, "tokens_per_microbatch": 4096, "grad_checkpointing": false, "attn_impl": "sdpa", "dtype": "bfloat16", "device": "NVIDIA GB10", "torch": "2.14.0+cu130", "warmup": 3, "n": 10, "s_per_step_median": 6.3639, "s_per_step_min": 6.3549, "s_per_step_max": 6.4533, "s_per_step_spread_pct": 1.5, "tok_per_s_median": 643.6, "peak_mem_GiB": 38.91}
|
||||
{"model": "Qwen3-0.6B-Base", "total_params_B": 0.616, "lora_rank": 32, "targets": "attn_mlp", "adapted_modules": 196, "trainable_params_M": 20.19, "trainable_pct": 3.276, "batch": 1, "seq": 4096, "tokens_per_microbatch": 4096, "grad_checkpointing": true, "attn_impl": "sdpa", "dtype": "bfloat16", "device": "NVIDIA GB10", "torch": "2.14.0+cu130", "warmup": 3, "n": 10, "s_per_step_median": 1.7072, "s_per_step_min": 1.6993, "s_per_step_max": 1.7097, "s_per_step_spread_pct": 0.6, "tok_per_s_median": 2399.2, "peak_mem_GiB": 9.75}
|
||||
{"model": "Qwen3-1.7B-Base", "total_params_B": 1.755, "lora_rank": 32, "targets": "attn_mlp", "adapted_modules": 196, "trainable_params_M": 34.87, "trainable_pct": 1.986, "batch": 1, "seq": 4096, "tokens_per_microbatch": 4096, "grad_checkpointing": true, "attn_impl": "sdpa", "dtype": "bfloat16", "device": "NVIDIA GB10", "torch": "2.14.0+cu130", "warmup": 3, "n": 10, "s_per_step_median": 2.8946, "s_per_step_min": 2.8782, "s_per_step_max": 2.901, "s_per_step_spread_pct": 0.8, "tok_per_s_median": 1415.0, "peak_mem_GiB": 12.24}
|
||||
{"model": "Qwen3-1.7B-Base", "total_params_B": 1.755, "lora_rank": 32, "targets": "attn_mlp", "adapted_modules": 196, "trainable_params_M": 34.87, "trainable_pct": 1.986, "batch": 4, "seq": 4096, "tokens_per_microbatch": 16384, "grad_checkpointing": true, "attn_impl": "sdpa", "dtype": "bfloat16", "device": "NVIDIA GB10", "torch": "2.14.0+cu130", "warmup": 3, "n": 10, "s_per_step_median": 11.3872, "s_per_step_min": 11.3401, "s_per_step_max": 11.4128, "s_per_step_spread_pct": 0.6, "tok_per_s_median": 1438.8, "peak_mem_GiB": 37.99}
|
||||
{"model": "Qwen3.5-0.8B-Base", "total_params_B": 0.765, "lora_rank": 32, "targets": "attn_mlp", "adapted_modules": 96, "trainable_params_M": 12.78, "trainable_pct": 1.67, "batch": 4, "seq": 4096, "tokens_per_microbatch": 16384, "grad_checkpointing": true, "attn_impl": "sdpa", "dtype": "bfloat16", "device": "NVIDIA GB10", "torch": "2.14.0+cu130", "warmup": 3, "n": 10, "s_per_step_median": 30.0301, "s_per_step_min": 29.9379, "s_per_step_max": 30.1444, "s_per_step_spread_pct": 0.7, "tok_per_s_median": 545.6, "peak_mem_GiB": 55.49}
|
||||
@@ -0,0 +1,93 @@
|
||||
"""Throughput floor for an R49 author-voice LoRA step on pfi-gx10 (GB10, sm_121).
|
||||
|
||||
Measures the cost of ONE forward+backward+optimizer microbatch on synthetic
|
||||
tokens, so a full-corpus wall-clock can be projected before any corpus exists.
|
||||
|
||||
Deliberately synthetic: random token ids exercise the same kernels at the same
|
||||
shapes as real text, and this is a THROUGHPUT harness only -- it says nothing
|
||||
about loss, quality, or voice transfer. The harness is part of the number, so
|
||||
every knob is printed with the result.
|
||||
|
||||
python bench_lora_step.py <model-dir> --seq 4096 --targets attn_mlp|all_linear_text
|
||||
"""
|
||||
import argparse, json, statistics, time, os
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoConfig
|
||||
from peft import LoraConfig, get_peft_model
|
||||
|
||||
ATTN_MLP = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
|
||||
PLUS_SSM = ATTN_MLP + ["in_proj_qkv", "in_proj_a", "in_proj_b", "in_proj_z", "out_proj"]
|
||||
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("model")
|
||||
ap.add_argument("--seq", type=int, default=4096)
|
||||
ap.add_argument("--batch", type=int, default=1)
|
||||
ap.add_argument("--rank", type=int, default=32)
|
||||
ap.add_argument("--targets", choices=["attn_mlp", "plus_ssm"], default="attn_mlp")
|
||||
ap.add_argument("--warmup", type=int, default=3)
|
||||
ap.add_argument("--steps", type=int, default=10)
|
||||
ap.add_argument("--no-grad-ckpt", action="store_true")
|
||||
ap.add_argument("--attn", default="sdpa")
|
||||
a = ap.parse_args()
|
||||
|
||||
torch.manual_seed(0)
|
||||
cfg = AutoConfig.from_pretrained(a.model)
|
||||
vocab = getattr(getattr(cfg, "text_config", cfg), "vocab_size")
|
||||
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
a.model, dtype=torch.bfloat16, attn_implementation=a.attn,
|
||||
).to("cuda")
|
||||
targets = ATTN_MLP if a.targets == "attn_mlp" else PLUS_SSM
|
||||
peft_cfg = LoraConfig(
|
||||
r=a.rank, lora_alpha=2 * a.rank, lora_dropout=0.0, bias="none",
|
||||
task_type="CAUSAL_LM", target_modules=targets,
|
||||
)
|
||||
model = get_peft_model(model, peft_cfg)
|
||||
if not a.no_grad_ckpt:
|
||||
model.gradient_checkpointing_enable()
|
||||
model.enable_input_require_grads()
|
||||
model.train()
|
||||
|
||||
trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
|
||||
total = sum(p.numel() for p in model.parameters())
|
||||
n_adapted = sum(1 for n, _ in model.named_modules() if n.endswith("lora_A.default"))
|
||||
|
||||
opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=1e-4)
|
||||
ids = torch.randint(0, vocab, (a.batch, a.seq), device="cuda")
|
||||
|
||||
def step():
|
||||
opt.zero_grad(set_to_none=True)
|
||||
out = model(input_ids=ids, labels=ids)
|
||||
out.loss.backward()
|
||||
opt.step()
|
||||
return float(out.loss)
|
||||
|
||||
for _ in range(a.warmup):
|
||||
step()
|
||||
torch.cuda.synchronize()
|
||||
|
||||
lat = []
|
||||
for _ in range(a.steps):
|
||||
t0 = time.perf_counter()
|
||||
step()
|
||||
torch.cuda.synchronize()
|
||||
lat.append(time.perf_counter() - t0)
|
||||
|
||||
tok = a.batch * a.seq
|
||||
res = dict(
|
||||
model=os.path.basename(a.model.rstrip("/")),
|
||||
total_params_B=round(total / 1e9, 3),
|
||||
lora_rank=a.rank, targets=a.targets, adapted_modules=n_adapted,
|
||||
trainable_params_M=round(trainable / 1e6, 2),
|
||||
trainable_pct=round(100 * trainable / total, 3),
|
||||
batch=a.batch, seq=a.seq, tokens_per_microbatch=tok,
|
||||
grad_checkpointing=not a.no_grad_ckpt, attn_impl=a.attn,
|
||||
dtype="bfloat16", device=torch.cuda.get_device_name(0),
|
||||
torch=torch.__version__, warmup=a.warmup, n=a.steps,
|
||||
s_per_step_median=round(statistics.median(lat), 4),
|
||||
s_per_step_min=round(min(lat), 4), s_per_step_max=round(max(lat), 4),
|
||||
s_per_step_spread_pct=round(100 * (max(lat) - min(lat)) / statistics.median(lat), 1),
|
||||
tok_per_s_median=round(tok / statistics.median(lat), 1),
|
||||
peak_mem_GiB=round(torch.cuda.max_memory_allocated() / 2**30, 2),
|
||||
)
|
||||
print(json.dumps(res))
|
||||
@@ -0,0 +1,69 @@
|
||||
"""Read-only structural probe of an R49 candidate carrier.
|
||||
|
||||
Answers, by measurement rather than by reading the config:
|
||||
* does transformers on this box load the checkpoint at all,
|
||||
* which module paths are nn.Linear (the only LoRA-attachable leaves),
|
||||
* how the parameter budget splits across text body / vision tower / MTP /
|
||||
embeddings, so "0.8B carrier" can be reported honestly,
|
||||
* which attention implementations the class accepts.
|
||||
|
||||
Loads on CPU in bf16. No training, no GPU, nothing written but stdout.
|
||||
"""
|
||||
import json, sys, collections, re
|
||||
import torch
|
||||
from transformers import AutoConfig, AutoModelForCausalLM
|
||||
|
||||
path = sys.argv[1]
|
||||
print(f"== {path}")
|
||||
cfg = AutoConfig.from_pretrained(path, trust_remote_code=False)
|
||||
print(" config class :", type(cfg).__name__)
|
||||
print(" architectures :", getattr(cfg, "architectures", None))
|
||||
tc = getattr(cfg, "text_config", None)
|
||||
if tc is not None:
|
||||
lt = getattr(tc, "layer_types", None) or []
|
||||
print(" text layers :", getattr(tc, "num_hidden_layers", "?"),
|
||||
"| full_attention:", lt.count("full_attention"),
|
||||
"| linear_attention:", lt.count("linear_attention"))
|
||||
print(" hidden/inter :", getattr(tc, "hidden_size", "?"), "/", getattr(tc, "intermediate_size", "?"))
|
||||
print(" vocab :", getattr(tc, "vocab_size", "?"), "| tied:", getattr(tc, "tie_word_embeddings", "?"))
|
||||
|
||||
try:
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
path, dtype=torch.bfloat16, device_map="cpu",
|
||||
attn_implementation="sdpa",
|
||||
)
|
||||
except Exception as e:
|
||||
print(" LOAD FAILED:", type(e).__name__, str(e)[:400])
|
||||
raise SystemExit(1)
|
||||
print(" model class :", type(model).__name__)
|
||||
print(" attn impl :", getattr(model.config, "_attn_implementation", "?"))
|
||||
|
||||
# --- parameter budget ------------------------------------------------------
|
||||
buckets = collections.Counter()
|
||||
def bucket(name):
|
||||
if ".visual." in name or name.startswith("visual."): return "vision_tower"
|
||||
if name.startswith("mtp.") or ".mtp." in name: return "mtp_head"
|
||||
if "embed_tokens" in name or name.endswith("lm_head.weight"): return "embeddings"
|
||||
if "linear_attn" in name: return "text_linear_attn"
|
||||
if "self_attn" in name: return "text_full_attn"
|
||||
if ".mlp." in name: return "text_mlp"
|
||||
return "text_other"
|
||||
for n, p in model.named_parameters():
|
||||
buckets[bucket(n)] += p.numel()
|
||||
total = sum(buckets.values())
|
||||
print(f" TOTAL params : {total/1e9:.3f} B")
|
||||
for k, v in sorted(buckets.items(), key=lambda kv: -kv[1]):
|
||||
print(f" {k:<18} {v/1e6:9.1f} M ({100*v/total:5.1f}%)")
|
||||
|
||||
# --- LoRA-attachable leaves ------------------------------------------------
|
||||
lin = collections.defaultdict(list)
|
||||
for name, mod in model.named_modules():
|
||||
if isinstance(mod, torch.nn.Linear):
|
||||
lin[bucket(name + ".weight")].append(name)
|
||||
print(" nn.Linear leaves by region:")
|
||||
for region in sorted(lin):
|
||||
names = lin[region]
|
||||
tmpl = sorted({re.sub(r"\.\d+\.", ".N.", n) for n in names})
|
||||
print(f" {region:<18} {len(names):4d} modules, {len(tmpl)} distinct shapes")
|
||||
for t in tmpl:
|
||||
print(f" {t}")
|
||||
Reference in New Issue
Block a user