feat(r49-prep): author-voice LoRA regime prep on gx10 — carriers staged, throughput measured, adapters secured

Prep for the BabyBronte / brokkr-smithy R49 author-voice adapter regime, plus the operator's "keep the adapter" ruling made durable. Measured on pfi-gx10 (GB10, sm_121), n=10 per arm after 3 warmup steps, seq 4096, LoRA r=32 on q/k/v/o + MLP, bf16, sdpa, grad-checkpointing on: Qwen3-0.6B-Base dense 0.616 B 1.707 s/step 2,399 tok/s Qwen3-1.7B-Base dense 1.755 B 2.895 s/step 1,415 tok/s Qwen3.5-0.8B-Base hybrid 0.765 B 7.581 s/step 540 tok/s The dense 1.755 B carrier trains 2.6x faster than the hybrid 0.765 B one on 2.3x the parameters (~6x per parameter), with more LoRA modules adapted (196 vs 96). Spreads of 0.6-2.6% put instrument noise an order of magnitude below the effect. Cause: Qwen3.5 is 18 linear-attention (SSM) layers to 6 attention, and no fused linear-attention kernel is installed on the box. Grad checkpointing is not the culprit (19%, and saves 2.6x memory). Batching is not the lever for either family -- both sit at this box's roofline at batch 1. Projected per voice on a Brontë-scale corpus: dense 0.6B 2.7 h, dense 1.7B 4.6 h, hybrid 0.8B 12 h. The hybrid would take longer than the 7 h 26B-A4B tune the regime exists to replace, so the carrier family is now an open decision with a recommendation for the dense Qwen3 line -- the design doc's original pin. Two further Qwen3.5 findings, both measured rather than read off the config: the Base checkpoints ship a vision tower (153/297 model.visual.* Linear tensors that target_modules="all-linear" would train on text) and an MTP head, both dropped for free by loading through AutoModelForCausalLM -- which renames modules relative to the vLLM serving path, so adapter binding needs the sampled-target-changed check on the serving side; and cross-document packing is unsafe because SSM state ignores the attention mask, breaking the per-copy name-consistency invariant the design doc calls sacred. Neither exists on dense. Adapter disposition, per the operator's ruling: all five gx10-resident ERP adapters (run-03c/04/05/06/07) mirrored to ana-ml2:/tank/erp-tune/run-<N>/adapter matching the layout runs 01-03 already used, byte-totals identical both sides and sha256 matching on every adapter_model.safetensors. /tank/* is deliberately excluded from ana-ml2's restic sources, so the profile gains one documented carve-out for /tank/erp-tune/run-*/adapter, verified by resticprofile --dry-run to expand to exactly those eight paths. Nothing is training and nothing is queued.
2026-09-09 22:41:47 -07:00
parent 92cf09e397
commit 7db6c44bcd
6 changed files with 445 additions and 5 deletions
@@ -11,8 +11,12 @@
 #   - /tank/* is NOT in source paths. That's ~TB of model weights (HF
 #     caches, llama.cpp GGUFs, ComfyUI models, etc.) — all regenerable
 #     from upstream. Backing them up would blow the repo size budget.
 #     ONE carve-out: /tank/erp-tune/run-*/adapter. A trained LoRA adapter is
 #     the only thing under /tank that upstream cannot hand back — each is
 #     ~300 MB and cost 7-14 h of GPU time, and the ERP line's adapters exist
 #     nowhere else but pfi-gx10's single NVMe and here. ~2.4 GB total.
 #   - No DB dumps needed. None of the stacks on this host (llama-swap,
-#     vllm, comfyui, kokoro, parakeet, vibevoice, beszel-agent,
+#     vllm-qwen3, comfyui, kokoro, parakeet, vibevoice, beszel-agent,
 #     dozzle-agent, dockge) store relational data.
 version: "1"
@@ -40,6 +44,7 @@ default:
      - /etc                 # host config (systemd units, chrony, apparmor, ssh, etc.)
      - /root                # root shell history, ssh keys, any ad-hoc scripts
      - /var/lib/docker/volumes  # named volumes (small; models live on /tank, not here)
      - /tank/erp-tune/run-*/adapter  # non-regenerable trained LoRA adapters (see header)
    exclude:
      # Docker internals we never want in a backup
      - /var/lib/docker/volumes/backingFsBlockDev
@@ -0,0 +1,229 @@
 # Author-voice LoRA regime on pfi-gx10 — training-side prep
 _Written 2026-09-09 22:45 PT. Status: **PREP. Nothing is training and nothing is
 queued.** Two operator decisions open (§6)._
 The research target is **brokkr-smithy R49** (`research/R49-author-voice-adapters/`),
 whose seed is the operator's **BabyBronte** design doc
 (`R49/reference/babybronte-design-doc.md`, 2026-08-25). R49 owns the hypotheses,
 the instruments and the adjudication. **This document owns the other half — the
 box, the stack, the corpus staging, the trainer, the launcher, and the
 wall-clock** — the same split that ran ERP-seat runs 3c through 7 on this box.
 Read `R49/target.md` before touching the corpus design; several attractive ideas
 are already settled *against* there and re-proposing them is the failure mode
 this file exists to prevent.
 ---
 ## 1. What the regime is, in one paragraph
 A voice is a **LoRA adapter on a small non-instruct base model**, trained on that
 author's real prose, steered at serving time by a terse beat line rather than a
 prompt. The completion in every training pair is 100% authentic author text; the
 only synthetic token in the corpus is the beat line, and beat lines are
 **loss-masked**, so the distribution the adapter learns to *emit* is the author's
 and the style ceiling is the author's own. Names and places are substituted
 **deterministically** — seeded RNG over a curated 23,398-name dictionary, never
 an LLM — which is what lets one work become 5–8 training copies without teaching
 plot memorisation.
 **The value being bought is marginal cost per voice, not inference latency.**
 That framing is the operator's and it is load-bearing: the anchor for
 "expensive" is ERP run 2 at ~7 h for a single 26B-A4B tune on a harness whose
 audit found a blocking-mask defect, a vision tower a leaf-name regex would have
 trained on text, and 128 experts fused per layer. A dense sub-2B carrier has
 none of those failure modes, and at this size the **methodology floor is finally
 cheaper than the shortcut** — two seeds per arm and a re-run after every change
 are routine rather than unaffordable.
 ## 2. The box, and what is already staged
 `pfi-gx10` (10.100.50.60) — ASUS Ascent GX10, NVIDIA **GB10**, `sm_121`,
 aarch64, **121 GB unified** memory, 916 GB NVMe, 470 GB free. Operator ruling
 2026-09-09: **experimental box, primarily for training, no serving seat.** Its
 GPU is idle.
 Training stack already present and current at `/home/infra-ops/ml/.venv`:
 | | |
 |---|---|
 | torch | `2.14.0+cu130`, `torch.cuda.get_device_capability() == (12, 1)` |
 | transformers | 5.16.1 (loads `Qwen3_5Config` natively) |
 | peft / trl / accelerate | 0.20.0 / 1.12.0 / 1.14.0 |
 | datasets / bitsandbytes | 5.0.1 / 0.50.2 |
 | **absent** | `mamba_ssm`, `causal_conv1d`, `fla`, `flash_attn`, `kernels` (triton 3.8.0 present) |
 Staged on local NVMe under `/home/infra-ops/carriers/` (pulled 2026-09-09,
 existence API-verified against the HF registry first, with a phantom repo run as
 the negative control):
    Qwen3.5-0.8B-Base   1.7G      Qwen3-0.6B-Base   1.2G
    Qwen3.5-2B-Base     4.3G      Qwen3-1.7B-Base   3.3G
    Qwen3.5-4B-Base     8.8G
 Probes live at `scripts/training-probes/{probe_carrier.py,bench_lora_step.py}`
 with raw output in `bench-lora-step-gx10-2026-09-09.jsonl`, so every number below
 can be re-derived rather than taken on faith.
 ## 3. What the carriers actually are — measured, not read off the model card
 R49 H02 names the `Qwen3.5` trio. Probing the checkpoints rather than the config
 found three things worth knowing before writing a recipe.
 **They ship a vision tower and an MTP head.** `model.visual.*` is 153 tensors on
 the 0.8B and 297 on the 2B — `attn.qkv`, `attn.proj`, `mlp.linear_fc1/2`, all
 `nn.Linear` leaves that `target_modules="all-linear"` would attach LoRA to and
 then train on pure text. This is the *same* defect the ERP harness audit caught
 on gemma-4. **Mitigation is free:** loading through `AutoModelForCausalLM`
 returns `Qwen3_5ForCausalLM` with the vision tower and MTP head dropped
 entirely — 0.752 B of text model, module paths `model.layers.N.*`.
 ⚠ **That mitigation creates a serving trap.** vLLM will load the full
 `Qwen3_5ForConditionalGeneration`, where the same weights live at
 `model.language_model.layers.N.*`. An adapter trained against the CausalLM
 prefix may not bind. **Pre-flight:** load the finished adapter in the serving
 path and confirm a sampled target tensor actually changed — the same silent-no-op
 check the ERP merge step already uses.
 **Three quarters of the layers are not attention.** `layer_types` is 3×
 `linear_attention` + 1× `full_attention` repeating: 18 SSM / 6 attention at
 0.8B and 2B, 24 / 8 at 4B. The SSM blocks carry `conv1d`, `A_log`, `dt_bias` and
 five Linear projections; `mamba_ssm_dtype` is `float32`.
 | region | 0.8B | 2B | 4B | LoRA-able leaves |
 |---|---|---|---|---|
 | MLP | 35.1% | 48.1% | 53.9% | `gate_proj`, `up_proj`, `down_proj` |
 | embeddings (tied) | 33.8% | 27.0% | 15.1% | `lm_head` — exclude |
 | linear-attn (SSM) | 25.2% | 20.1% | 24.0% | `in_proj_{qkv,a,b,z}`, `out_proj` |
 | full attention | 5.9% | 4.7% | 7.0% | `q_proj`, `k_proj`, `v_proj`, `o_proj` |
 | **total** | **0.752 B** | **1.882 B** | **4.206 B** | |
 The conventional `q,k,v,o` + MLP recipe therefore covers **41%** of the 0.8B and
 leaves the SSM stack untouched in 18 of 24 layers. Adding the SSM Linears takes
 coverage to 66%. At this scale that is a cheap ablation, not an agonising choice.
 ⚠ **Packing across document boundaries is unsafe on this architecture.** An SSM
 layer carries recurrent state along the sequence and an attention mask does not
 reset it, so two renamed copies packed into one 8k window can bleed in 18 of 24
 layers — which is precisely the per-copy name-consistency invariant the design
 doc calls sacred. Either one document per sequence, or prove the trainer's
 sequence-boundary signal is honoured by the linear-attn path. Under a dense
 carrier this problem does not exist.
 **The tied embedding is a third of the small carrier.** vocab 248,320 × hidden
 1024 = 254 M of the 0.752 B. The transformer body being tested at the small end
 is ~0.50 B, which matters when reporting "the carrier floor".
 ## 4. Throughput — and the newest carrier is the slow one
 One forward+backward+AdamW microbatch, LoRA r=32/α=64 on `q,k,v,o` + MLP,
 bf16, `sdpa`, gradient checkpointing on, seq 4096, on gx10's GB10. n=10
 measured after 3 warmup steps; median reported with the full spread.
 | carrier | architecture | params | s/step | tok/s | peak | spread |
 |---|---|---|---|---|---|---|
 | `Qwen3.5-0.8B-Base` | hybrid, 18 SSM / 6 attn | 0.765 B | 7.581 | **540** | 15.1 GiB | 2.6% |
 | `Qwen3.5-0.8B-Base` (no grad-ckpt) | " | 0.765 B | 6.364 | 644 | 38.9 GiB | 1.5% |
 | `Qwen3-0.6B-Base` | dense | 0.616 B | 1.707 | **2,399** | 9.8 GiB | 0.6% |
 | `Qwen3-1.7B-Base` | dense | 1.755 B | 2.895 | **1,415** | 12.2 GiB | 0.8% |
 | `Qwen3-1.7B-Base`, batch 4 | dense | 1.755 B | 11.387 | 1,439 | 38.0 GiB | 0.6% |
 | `Qwen3.5-0.8B-Base`, batch 4 | hybrid | 0.765 B | 30.030 | 546 | 55.5 GiB | 0.7% |
 **The dense 1.755 B carrier trains 2.6× faster than the hybrid 0.765 B one** — on
 2.3× the parameters, with *more* LoRA modules adapted (196 vs 96, because dense
 has real attention in every layer). Per parameter the dense path is ~6× more
 efficient. Spreads of 0.6–2.6% across n=10 put the instrument's noise an order of
 magnitude below the effect, so this is not variance.
 The cause is almost certainly that **no fused linear-attention kernel is
 installed** (§2) so the SSM path runs a reference implementation. Grad
 checkpointing is *not* the culprit — turning it off recovers only 19% and costs
 2.6× the memory, so leave it on. Batching is not the lever for either family: 1,415 → 1,439
 tok/s dense and 540 → 546 tok/s hybrid from batch 1 to 4. **Both architectures
 are already at this box's roofline at batch 1**, which is a bandwidth story
 (GB10's unified LPDDR5X against an RTX PRO 6000's ~6.6× higher figure) — and it
 means the 2.6× gap is the kernel path, not a batching artefact.
 **What that does to the regime's premise.** Projecting a Brontë-scale corpus
 (~1 M words ≈ 1.3 M tokens, × 6 rename copies, 3 epochs ≈ 23 M tokens):
 | carrier | projected wall-clock per voice |
 |---|---|
 | `Qwen3-0.6B-Base` dense | **2.7 h** |
 | `Qwen3-1.7B-Base` dense | **4.6 h** |
 | `Qwen3.5-0.8B-Base` hybrid | **12 h** |
 The hybrid carrier would make a per-voice run *longer than the 7 h 26B-A4B tune
 it exists to replace.* At R49 H03's hoped-for corpus floor (~300 k words) the
 dense 1.7B lands near **1.4 h** — a voice per afternoon, which is the regime the
 operator asked for. ⚠ These are projections from a synthetic-token throughput
 harness, not from a completed run; treat them as sizing, and re-measure on the
 first real corpus.
 ## 5. Prep remaining, in order
 1. **Carrier family decision** (§6) — everything downstream keys on it.
 2. **Corpus D1** — Gutenberg Brontë (Jane Eyre, Villette, Shirley, The
   Professor), boilerplate stripped, chapter-segmented, typography normalised,
   character inventory recorded. Public domain, clean under any disposition.
 3. **Re-point the R49 deterministic machinery at Brontë.** The entity detector
   (corpus-level capitalised-vs-lowercase ratio), identity linking, gender
   resolution and the 23,398-name dictionary were all built and hardened against
   a *Yarros* sample. Per-work re-derivation needed: entity map, alphabet, and
   the `UNRESOLVED_BLOCKING` human pass (~20–40 entities per work).
 4. **Beat annotation (D4)** via `gen` inverse-prompting, using F02's hardened
   prompt (banned meta-language, three PD worked examples, ≤20-word gate).
 5. **Trainer.** `erp_sft_harness` is chat-shaped and carries ERP-specific
   eligibility machinery; the author-voice job is plain continuation with a
   masked prefix. Decision: a small purpose-built trainer that *keeps* the
   harness's §4 disciplines (provenance pin, order manifest, truncation report,
   cache key that sees semantic changes, recorded attention backend) rather than
   a fork of its corpus logic.
 6. **Pre-set the decision threshold before collecting data**, wider than the
   measured seed-to-seed spread, per the R49 charter — and run the positive
   control the R49 journal's own lesson demands: confirm the stylometric
   instrument separates real Brontë from unadapted base output *before* it is
   asked to judge an adapter.
 7. **Launcher** modelled on `launch-run-07.sh` — its guards were each bought with
   a past failure (GPU-clear assertion, pidfile not `pgrep -f`, refuse an
   existing log, free-space floor, `setsid` detach).
 Deliberately out of scope here, per R49: the Director/critic loop, style
 arithmetic and the Pelican test, multi-LoRA arsenal serving, the Skaldsong
 integration contract, the modern in-copyright arsenal, inference latency.
 ## 6. Open for the operator
 **(a) Carrier family — recommend the dense `Qwen3` line.** R49 H02 pins
 `Qwen3.5-{0.8,2,4}B-Base`. The measurements say that family costs 2.6–6× the
 wall-clock on this box, brings a vision tower and an MTP head to a text job, and
 makes cross-document packing unsafe in 18 of 24 layers. `Qwen3-{0.6,1.7,4}B-Base`
 — the design doc's own original pin — is plain dense, has none of those, and is
 the better instrument for a probe whose whole point is isolating one variable.
 The cost is one model generation of base quality. Reversible: the Qwen3.5
 checkpoints stay staged, and a fused-kernel install (`fla` is pure Triton and
 would plausibly work on aarch64) could revive them later as a follow-up rather
 than a blocker.
 **(b) 45 GB of intermediate checkpoints on gx10, and 354 GB on ana-ml2.**
 `run-03c/04/05/06/checkpoints` total 45 GB on gx10; `/tank/erp-tune/serve` is
 354 GB of superseded merged models on ana-ml2. The final adapters are safe (§7)
 and run 6 is the standing seat. Purging is the operator's call — 470 GB free on
 gx10 means it does not block this regime.
 ## 7. Adapter disposition — settled, and made real
 Operator, 2026-09-09: **keep the adapter.** As of 22:30 PT all five
 gx10-resident ERP adapters are mirrored to `ana-ml2:/tank/erp-tune/run-<N>/adapter`,
 matching the layout runs 01–03 already use there, byte-total identical on both
 sides and `sha256` matching on every `adapter_model.safetensors`:
    run-03c  run-04  run-05  run-06  run-07     315 MB each, 8 files each
 `/tank/*` is deliberately **excluded** from ana-ml2's restic sources — terabytes
 of regenerable model weights. A trained adapter is the one thing under there
 upstream cannot hand back, so `configs/restic/ana-ml2/profiles.yaml` now carries
 a single documented carve-out, `/tank/erp-tune/run-*/adapter`, verified by
 `resticprofile --dry-run` to expand to exactly those eight paths and nothing
 else. The nightly 01:00 run picks them up.
@@ -1,6 +1,6 @@
 # Persistent memory — eshpfi-management
-_Last updated: 2026-09-09 22:05 PT (**Pfish-6** = run-6 NVFP4 is the standing seat, ana-ml2 :8021 ONLY; run 7 PURGED ~139 GiB; pfi-gx10 is an experimental/TRAINING box and carries no serving seat; no new run planned)_
+_Last updated: 2026-09-09 22:50 PT (**Pfish-6** = run-6 NVFP4 is the standing seat, ana-ml2 :8021 ONLY; run 7 PURGED ~139 GiB; pfi-gx10 is an experimental/TRAINING box and carries no serving seat; all five ERP adapters now MIRRORED to ana-ml2 and inside restic; **BabyBronte / R49 author-voice regime is in PREP on gx10** — carrier-family decision open)_
 > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
 > `Written:` stamp is under an hour old, read it (it carries the in-flight
@@ -125,9 +125,47 @@ preserved verbatim in `archival-memory.md` § Superseded in-flight snapshots._
  `run-07/checkpoints` 9.2 GiB, `serve/pfish6-nvfp4a16` 16 GiB; ana-ml2 `erp-tune-v7-bf16` 49 GiB,
  `erp-tune-v7-nvfp4a16` 16 GiB, `erp-tune-v7-quant-work`. **~139 GiB reclaimed** (gx10 53%→47%).
  ⚠ **KEPT deliberately: `~/erp-tune/run-07/adapter` 315 MB + provenance + `loss-series-r7.json`** —
-  the only non-reproducible piece (14 h of training), and it costs nothing. `rm -rf
+  the only non-reproducible piece (14 h of training), and it costs nothing. Everything else run-7 is
-  /home/infra-ops/erp-tune/run-07` finishes the job if wanted; everything else run-7 is already gone.
+  already gone; do NOT `rm -rf /home/infra-ops/erp-tune/run-07` — operator ruled **keep the adapter**
- **NO NEW TRAINING RUN PLANNED.** The opening-split idea is not being re-tested; run 6 stands.
+  2026-09-09.
 - **NO NEW *ERP* TRAINING RUN PLANNED.** The opening-split idea is not being re-tested; run 6 stands.
 - **✅ ALL FIVE gx10 ERP ADAPTERS ARE NOW TWO-COPY AND BACKED UP** (2026-09-09 22:30 PT, operator:
  *"keep the adapter"*). `run-03c/04/05/06/07` mirrored to `ana-ml2:/tank/erp-tune/run-<N>/adapter`
  (the layout runs 01–03 already used), byte-totals identical both sides + `sha256` match on every
  `adapter_model.safetensors`. ⚠ `/tank/*` is deliberately OUT of ana-ml2's restic sources (TB of
  regenerable weights), so `configs/restic/ana-ml2/profiles.yaml` gained ONE documented carve-out —
  `/tank/erp-tune/run-*/adapter` — verified by `resticprofile --dry-run` to expand to exactly those 8
  paths and nothing else. Live file is byte-identical to the repo canonical; `.bak-20260909` beside it.
 - **🖋 BabyBronte / R49 author-voice LoRA regime — IN PREP on pfi-gx10, nothing training.** Plan +
  every measured number: [`docs/pfi/author-voice-lora-regime.md`](docs/pfi/author-voice-lora-regime.md).
  Research target is **brokkr-smithy R49** (`research/R49-author-voice-adapters/`) — brokkr owns
  hypotheses/instruments/adjudication, infra-ops owns box+corpus staging+trainer+wall-clock, same split
  as ERP runs 3c–7. Five carriers staged on gx10 `~/carriers/` (Qwen3.5-0.8/2/4B-Base +
  Qwen3-0.6/1.7B-Base); probes committed at `scripts/training-probes/{probe_carrier,bench_lora_step}.py`.
  ⚠⚠ **THE HEADLINE, AND IT INVERTS R49's PIN: the newest carrier is the SLOW one.** Measured on gx10
  (n=10, spread 0.6–2.6%, seq 4096, r=32 attn+mlp, bf16 sdpa, grad-ckpt on): dense `Qwen3-1.7B-Base`
  **1,415 tok/s** vs hybrid `Qwen3.5-0.8B-Base` **540 tok/s** — the dense model is 2.6x faster on 2.3x
  the parameters (~6x per-param), because Qwen3.5 is **18 SSM / 6 attention layers** and NO fused
  linear-attention kernel is installed (`mamba_ssm`/`causal_conv1d`/`fla` all absent; triton 3.8 is
  there). Projected per voice: dense 0.6B **2.7 h**, dense 1.7B **4.6 h**, hybrid 0.8B **12 h** — the
  hybrid would take LONGER than the 7 h 26B-A4B tune it exists to replace, killing the regime's whole
  premise. Grad-ckpt is not the cause (19%, and saves 2.6x memory — keep it); batching is not the lever
  (dense 1,415→1,439 and hybrid 540→546 tok/s at batch 4 — BOTH already at this box's roofline at batch 1, a bandwidth story, so the 2.6x gap is the kernel path not a batching artefact).
  ⚠ Two more Qwen3.5 landmines, both measured: it **ships a vision tower** (153/297 `model.visual.*`
  Linear tensors that `all-linear` would train on text — the same defect the gemma-4 audit caught;
  `AutoModelForCausalLM` drops it and the MTP head for free, BUT that renames modules
  `model.layers.N.*` vs vLLM's `model.language_model.layers.N.*`, so an adapter may not bind → run the
  sampled-target-changed check in the SERVING path); and **cross-document packing is unsafe** because
  SSM state ignores the attention mask, which breaks the per-copy name-consistency invariant the design
  doc calls sacred. None of these exist on the dense line.
  **⭐ OPEN FOR THE OPERATOR: carrier family — recommend the dense `Qwen3-{0.6,1.7,4}B-Base` line**
  (the design doc's own original pin) over R49 H02's `Qwen3.5` trio. Reversible: Qwen3.5 stays staged
  and an `fla` install (pure Triton, plausibly fine on aarch64) could revive it as a follow-up.
 - **📌 forseti shipped althing 3.6.1 (2026-09-09) — every box's herald needs a restart to pick up the
  new poke text.** `uv tool install --force --reinstall <checkout>` + `systemctl --user restart
  althing-po-herald`, expect `3.6.1`; the plugin also went to 0.1.6 (inbox.md no longer calls the
  result reply "optional"). NOT DONE — deferred, nothing is blocked on it, and it is a fleet-wide
  multi-box pass with its own verification burden.
 - **⏳ ana-ml2 storage follow-ups, operator's call** (the three actions themselves LANDED 09-09 02:02):
  (a) **ZFS pool-health ALERTING** — `tank` sat DEGRADED 04-23→09-05 with nvme7 physically absent and
  nobody knew (ZED mails `root`, no MTA on the box); (b) nvme7 / slot 0-5 keep-vs-replace —
@@ -0,0 +1,6 @@
 {"model": "Qwen3.5-0.8B-Base", "total_params_B": 0.765, "lora_rank": 32, "targets": "attn_mlp", "adapted_modules": 96, "trainable_params_M": 12.78, "trainable_pct": 1.67, "batch": 1, "seq": 4096, "tokens_per_microbatch": 4096, "grad_checkpointing": true, "attn_impl": "sdpa", "dtype": "bfloat16", "device": "NVIDIA GB10", "torch": "2.14.0+cu130", "warmup": 3, "n": 10, "s_per_step_median": 7.5814, "s_per_step_min": 7.4864, "s_per_step_max": 7.6843, "s_per_step_spread_pct": 2.6, "tok_per_s_median": 540.3, "peak_mem_GiB": 15.1}
 {"model": "Qwen3.5-0.8B-Base", "total_params_B": 0.765, "lora_rank": 32, "targets": "attn_mlp", "adapted_modules": 96, "trainable_params_M": 12.78, "trainable_pct": 1.67, "batch": 1, "seq": 4096, "tokens_per_microbatch": 4096, "grad_checkpointing": false, "attn_impl": "sdpa", "dtype": "bfloat16", "device": "NVIDIA GB10", "torch": "2.14.0+cu130", "warmup": 3, "n": 10, "s_per_step_median": 6.3639, "s_per_step_min": 6.3549, "s_per_step_max": 6.4533, "s_per_step_spread_pct": 1.5, "tok_per_s_median": 643.6, "peak_mem_GiB": 38.91}
 {"model": "Qwen3-0.6B-Base", "total_params_B": 0.616, "lora_rank": 32, "targets": "attn_mlp", "adapted_modules": 196, "trainable_params_M": 20.19, "trainable_pct": 3.276, "batch": 1, "seq": 4096, "tokens_per_microbatch": 4096, "grad_checkpointing": true, "attn_impl": "sdpa", "dtype": "bfloat16", "device": "NVIDIA GB10", "torch": "2.14.0+cu130", "warmup": 3, "n": 10, "s_per_step_median": 1.7072, "s_per_step_min": 1.6993, "s_per_step_max": 1.7097, "s_per_step_spread_pct": 0.6, "tok_per_s_median": 2399.2, "peak_mem_GiB": 9.75}
 {"model": "Qwen3-1.7B-Base", "total_params_B": 1.755, "lora_rank": 32, "targets": "attn_mlp", "adapted_modules": 196, "trainable_params_M": 34.87, "trainable_pct": 1.986, "batch": 1, "seq": 4096, "tokens_per_microbatch": 4096, "grad_checkpointing": true, "attn_impl": "sdpa", "dtype": "bfloat16", "device": "NVIDIA GB10", "torch": "2.14.0+cu130", "warmup": 3, "n": 10, "s_per_step_median": 2.8946, "s_per_step_min": 2.8782, "s_per_step_max": 2.901, "s_per_step_spread_pct": 0.8, "tok_per_s_median": 1415.0, "peak_mem_GiB": 12.24}
 {"model": "Qwen3-1.7B-Base", "total_params_B": 1.755, "lora_rank": 32, "targets": "attn_mlp", "adapted_modules": 196, "trainable_params_M": 34.87, "trainable_pct": 1.986, "batch": 4, "seq": 4096, "tokens_per_microbatch": 16384, "grad_checkpointing": true, "attn_impl": "sdpa", "dtype": "bfloat16", "device": "NVIDIA GB10", "torch": "2.14.0+cu130", "warmup": 3, "n": 10, "s_per_step_median": 11.3872, "s_per_step_min": 11.3401, "s_per_step_max": 11.4128, "s_per_step_spread_pct": 0.6, "tok_per_s_median": 1438.8, "peak_mem_GiB": 37.99}
 {"model": "Qwen3.5-0.8B-Base", "total_params_B": 0.765, "lora_rank": 32, "targets": "attn_mlp", "adapted_modules": 96, "trainable_params_M": 12.78, "trainable_pct": 1.67, "batch": 4, "seq": 4096, "tokens_per_microbatch": 16384, "grad_checkpointing": true, "attn_impl": "sdpa", "dtype": "bfloat16", "device": "NVIDIA GB10", "torch": "2.14.0+cu130", "warmup": 3, "n": 10, "s_per_step_median": 30.0301, "s_per_step_min": 29.9379, "s_per_step_max": 30.1444, "s_per_step_spread_pct": 0.7, "tok_per_s_median": 545.6, "peak_mem_GiB": 55.49}
@@ -0,0 +1,93 @@
 """Throughput floor for an R49 author-voice LoRA step on pfi-gx10 (GB10, sm_121).
 Measures the cost of ONE forward+backward+optimizer microbatch on synthetic
 tokens, so a full-corpus wall-clock can be projected before any corpus exists.
 Deliberately synthetic: random token ids exercise the same kernels at the same
 shapes as real text, and this is a THROUGHPUT harness only -- it says nothing
 about loss, quality, or voice transfer. The harness is part of the number, so
 every knob is printed with the result.
  python bench_lora_step.py <model-dir> --seq 4096 --targets attn_mlp|all_linear_text
 """
 import argparse, json, statistics, time, os
 import torch
 from transformers import AutoModelForCausalLM, AutoConfig
 from peft import LoraConfig, get_peft_model
 ATTN_MLP = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
 PLUS_SSM = ATTN_MLP + ["in_proj_qkv", "in_proj_a", "in_proj_b", "in_proj_z", "out_proj"]
 ap = argparse.ArgumentParser()
 ap.add_argument("model")
 ap.add_argument("--seq", type=int, default=4096)
 ap.add_argument("--batch", type=int, default=1)
 ap.add_argument("--rank", type=int, default=32)
 ap.add_argument("--targets", choices=["attn_mlp", "plus_ssm"], default="attn_mlp")
 ap.add_argument("--warmup", type=int, default=3)
 ap.add_argument("--steps", type=int, default=10)
 ap.add_argument("--no-grad-ckpt", action="store_true")
 ap.add_argument("--attn", default="sdpa")
 a = ap.parse_args()
 torch.manual_seed(0)
 cfg = AutoConfig.from_pretrained(a.model)
 vocab = getattr(getattr(cfg, "text_config", cfg), "vocab_size")
 model = AutoModelForCausalLM.from_pretrained(
    a.model, dtype=torch.bfloat16, attn_implementation=a.attn,
 ).to("cuda")
 targets = ATTN_MLP if a.targets == "attn_mlp" else PLUS_SSM
 peft_cfg = LoraConfig(
    r=a.rank, lora_alpha=2 * a.rank, lora_dropout=0.0, bias="none",
    task_type="CAUSAL_LM", target_modules=targets,
 )
 model = get_peft_model(model, peft_cfg)
 if not a.no_grad_ckpt:
    model.gradient_checkpointing_enable()
    model.enable_input_require_grads()
 model.train()
 trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
 total = sum(p.numel() for p in model.parameters())
 n_adapted = sum(1 for n, _ in model.named_modules() if n.endswith("lora_A.default"))
 opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=1e-4)
 ids = torch.randint(0, vocab, (a.batch, a.seq), device="cuda")
 def step():
    opt.zero_grad(set_to_none=True)
    out = model(input_ids=ids, labels=ids)
    out.loss.backward()
    opt.step()
    return float(out.loss)
 for _ in range(a.warmup):
    step()
 torch.cuda.synchronize()
 lat = []
 for _ in range(a.steps):
    t0 = time.perf_counter()
    step()
    torch.cuda.synchronize()
    lat.append(time.perf_counter() - t0)
 tok = a.batch * a.seq
 res = dict(
    model=os.path.basename(a.model.rstrip("/")),
    total_params_B=round(total / 1e9, 3),
    lora_rank=a.rank, targets=a.targets, adapted_modules=n_adapted,
    trainable_params_M=round(trainable / 1e6, 2),
    trainable_pct=round(100 * trainable / total, 3),
    batch=a.batch, seq=a.seq, tokens_per_microbatch=tok,
    grad_checkpointing=not a.no_grad_ckpt, attn_impl=a.attn,
    dtype="bfloat16", device=torch.cuda.get_device_name(0),
    torch=torch.__version__, warmup=a.warmup, n=a.steps,
    s_per_step_median=round(statistics.median(lat), 4),
    s_per_step_min=round(min(lat), 4), s_per_step_max=round(max(lat), 4),
    s_per_step_spread_pct=round(100 * (max(lat) - min(lat)) / statistics.median(lat), 1),
    tok_per_s_median=round(tok / statistics.median(lat), 1),
    peak_mem_GiB=round(torch.cuda.max_memory_allocated() / 2**30, 2),
 )
 print(json.dumps(res))
@@ -0,0 +1,69 @@
 """Read-only structural probe of an R49 candidate carrier.
 Answers, by measurement rather than by reading the config:
  * does transformers on this box load the checkpoint at all,
  * which module paths are nn.Linear (the only LoRA-attachable leaves),
  * how the parameter budget splits across text body / vision tower / MTP /
    embeddings, so "0.8B carrier" can be reported honestly,
  * which attention implementations the class accepts.
 Loads on CPU in bf16. No training, no GPU, nothing written but stdout.
 """
 import json, sys, collections, re
 import torch
 from transformers import AutoConfig, AutoModelForCausalLM
 path = sys.argv[1]
 print(f"== {path}")
 cfg = AutoConfig.from_pretrained(path, trust_remote_code=False)
 print("   config class     :", type(cfg).__name__)
 print("   architectures    :", getattr(cfg, "architectures", None))
 tc = getattr(cfg, "text_config", None)
 if tc is not None:
    lt = getattr(tc, "layer_types", None) or []
    print("   text layers      :", getattr(tc, "num_hidden_layers", "?"),
          "| full_attention:", lt.count("full_attention"),
          "| linear_attention:", lt.count("linear_attention"))
    print("   hidden/inter     :", getattr(tc, "hidden_size", "?"), "/", getattr(tc, "intermediate_size", "?"))
    print("   vocab            :", getattr(tc, "vocab_size", "?"), "| tied:", getattr(tc, "tie_word_embeddings", "?"))
 try:
    model = AutoModelForCausalLM.from_pretrained(
        path, dtype=torch.bfloat16, device_map="cpu",
        attn_implementation="sdpa",
    )
 except Exception as e:
    print("   LOAD FAILED:", type(e).__name__, str(e)[:400])
    raise SystemExit(1)
 print("   model class      :", type(model).__name__)
 print("   attn impl        :", getattr(model.config, "_attn_implementation", "?"))
 # --- parameter budget ------------------------------------------------------
 buckets = collections.Counter()
 def bucket(name):
    if ".visual." in name or name.startswith("visual."):        return "vision_tower"
    if name.startswith("mtp.") or ".mtp." in name:              return "mtp_head"
    if "embed_tokens" in name or name.endswith("lm_head.weight"): return "embeddings"
    if "linear_attn" in name:                                   return "text_linear_attn"
    if "self_attn" in name:                                     return "text_full_attn"
    if ".mlp." in name:                                         return "text_mlp"
    return "text_other"
 for n, p in model.named_parameters():
    buckets[bucket(n)] += p.numel()
 total = sum(buckets.values())
 print(f"   TOTAL params     : {total/1e9:.3f} B")
 for k, v in sorted(buckets.items(), key=lambda kv: -kv[1]):
    print(f"     {k:<18} {v/1e6:9.1f} M  ({100*v/total:5.1f}%)")
 # --- LoRA-attachable leaves ------------------------------------------------
 lin = collections.defaultdict(list)
 for name, mod in model.named_modules():
    if isinstance(mod, torch.nn.Linear):
        lin[bucket(name + ".weight")].append(name)
 print("   nn.Linear leaves by region:")
 for region in sorted(lin):
    names = lin[region]
    tmpl = sorted({re.sub(r"\.\d+\.", ".N.", n) for n in names})
    print(f"     {region:<18} {len(names):4d} modules, {len(tmpl)} distinct shapes")
    for t in tmpl:
        print(f"        {t}")