diff --git a/configs/restic/ana-ml2/profiles.yaml b/configs/restic/ana-ml2/profiles.yaml index a30284c..9c52378 100644 --- a/configs/restic/ana-ml2/profiles.yaml +++ b/configs/restic/ana-ml2/profiles.yaml @@ -11,8 +11,12 @@ # - /tank/* is NOT in source paths. That's ~TB of model weights (HF # caches, llama.cpp GGUFs, ComfyUI models, etc.) — all regenerable # from upstream. Backing them up would blow the repo size budget. +# ONE carve-out: /tank/erp-tune/run-*/adapter. A trained LoRA adapter is +# the only thing under /tank that upstream cannot hand back — each is +# ~300 MB and cost 7-14 h of GPU time, and the ERP line's adapters exist +# nowhere else but pfi-gx10's single NVMe and here. ~2.4 GB total. # - No DB dumps needed. None of the stacks on this host (llama-swap, -# vllm, comfyui, kokoro, parakeet, vibevoice, beszel-agent, +# vllm-qwen3, comfyui, kokoro, parakeet, vibevoice, beszel-agent, # dozzle-agent, dockge) store relational data. version: "1" @@ -40,6 +44,7 @@ default: - /etc # host config (systemd units, chrony, apparmor, ssh, etc.) - /root # root shell history, ssh keys, any ad-hoc scripts - /var/lib/docker/volumes # named volumes (small; models live on /tank, not here) + - /tank/erp-tune/run-*/adapter # non-regenerable trained LoRA adapters (see header) exclude: # Docker internals we never want in a backup - /var/lib/docker/volumes/backingFsBlockDev diff --git a/docs/pfi/author-voice-lora-regime.md b/docs/pfi/author-voice-lora-regime.md new file mode 100644 index 0000000..ee2a151 --- /dev/null +++ b/docs/pfi/author-voice-lora-regime.md @@ -0,0 +1,229 @@ +# Author-voice LoRA regime on pfi-gx10 — training-side prep + +_Written 2026-09-09 22:45 PT. Status: **PREP. Nothing is training and nothing is +queued.** Two operator decisions open (§6)._ + +The research target is **brokkr-smithy R49** (`research/R49-author-voice-adapters/`), +whose seed is the operator's **BabyBronte** design doc +(`R49/reference/babybronte-design-doc.md`, 2026-08-25). R49 owns the hypotheses, +the instruments and the adjudication. **This document owns the other half — the +box, the stack, the corpus staging, the trainer, the launcher, and the +wall-clock** — the same split that ran ERP-seat runs 3c through 7 on this box. + +Read `R49/target.md` before touching the corpus design; several attractive ideas +are already settled *against* there and re-proposing them is the failure mode +this file exists to prevent. + +--- + +## 1. What the regime is, in one paragraph + +A voice is a **LoRA adapter on a small non-instruct base model**, trained on that +author's real prose, steered at serving time by a terse beat line rather than a +prompt. The completion in every training pair is 100% authentic author text; the +only synthetic token in the corpus is the beat line, and beat lines are +**loss-masked**, so the distribution the adapter learns to *emit* is the author's +and the style ceiling is the author's own. Names and places are substituted +**deterministically** — seeded RNG over a curated 23,398-name dictionary, never +an LLM — which is what lets one work become 5–8 training copies without teaching +plot memorisation. + +**The value being bought is marginal cost per voice, not inference latency.** +That framing is the operator's and it is load-bearing: the anchor for +"expensive" is ERP run 2 at ~7 h for a single 26B-A4B tune on a harness whose +audit found a blocking-mask defect, a vision tower a leaf-name regex would have +trained on text, and 128 experts fused per layer. A dense sub-2B carrier has +none of those failure modes, and at this size the **methodology floor is finally +cheaper than the shortcut** — two seeds per arm and a re-run after every change +are routine rather than unaffordable. + +## 2. The box, and what is already staged + +`pfi-gx10` (10.100.50.60) — ASUS Ascent GX10, NVIDIA **GB10**, `sm_121`, +aarch64, **121 GB unified** memory, 916 GB NVMe, 470 GB free. Operator ruling +2026-09-09: **experimental box, primarily for training, no serving seat.** Its +GPU is idle. + +Training stack already present and current at `/home/infra-ops/ml/.venv`: + +| | | +|---|---| +| torch | `2.14.0+cu130`, `torch.cuda.get_device_capability() == (12, 1)` | +| transformers | 5.16.1 (loads `Qwen3_5Config` natively) | +| peft / trl / accelerate | 0.20.0 / 1.12.0 / 1.14.0 | +| datasets / bitsandbytes | 5.0.1 / 0.50.2 | +| **absent** | `mamba_ssm`, `causal_conv1d`, `fla`, `flash_attn`, `kernels` (triton 3.8.0 present) | + +Staged on local NVMe under `/home/infra-ops/carriers/` (pulled 2026-09-09, +existence API-verified against the HF registry first, with a phantom repo run as +the negative control): + + Qwen3.5-0.8B-Base 1.7G Qwen3-0.6B-Base 1.2G + Qwen3.5-2B-Base 4.3G Qwen3-1.7B-Base 3.3G + Qwen3.5-4B-Base 8.8G + +Probes live at `scripts/training-probes/{probe_carrier.py,bench_lora_step.py}` +with raw output in `bench-lora-step-gx10-2026-09-09.jsonl`, so every number below +can be re-derived rather than taken on faith. + +## 3. What the carriers actually are — measured, not read off the model card + +R49 H02 names the `Qwen3.5` trio. Probing the checkpoints rather than the config +found three things worth knowing before writing a recipe. + +**They ship a vision tower and an MTP head.** `model.visual.*` is 153 tensors on +the 0.8B and 297 on the 2B — `attn.qkv`, `attn.proj`, `mlp.linear_fc1/2`, all +`nn.Linear` leaves that `target_modules="all-linear"` would attach LoRA to and +then train on pure text. This is the *same* defect the ERP harness audit caught +on gemma-4. **Mitigation is free:** loading through `AutoModelForCausalLM` +returns `Qwen3_5ForCausalLM` with the vision tower and MTP head dropped +entirely — 0.752 B of text model, module paths `model.layers.N.*`. + +⚠ **That mitigation creates a serving trap.** vLLM will load the full +`Qwen3_5ForConditionalGeneration`, where the same weights live at +`model.language_model.layers.N.*`. An adapter trained against the CausalLM +prefix may not bind. **Pre-flight:** load the finished adapter in the serving +path and confirm a sampled target tensor actually changed — the same silent-no-op +check the ERP merge step already uses. + +**Three quarters of the layers are not attention.** `layer_types` is 3× +`linear_attention` + 1× `full_attention` repeating: 18 SSM / 6 attention at +0.8B and 2B, 24 / 8 at 4B. The SSM blocks carry `conv1d`, `A_log`, `dt_bias` and +five Linear projections; `mamba_ssm_dtype` is `float32`. + +| region | 0.8B | 2B | 4B | LoRA-able leaves | +|---|---|---|---|---| +| MLP | 35.1% | 48.1% | 53.9% | `gate_proj`, `up_proj`, `down_proj` | +| embeddings (tied) | 33.8% | 27.0% | 15.1% | `lm_head` — exclude | +| linear-attn (SSM) | 25.2% | 20.1% | 24.0% | `in_proj_{qkv,a,b,z}`, `out_proj` | +| full attention | 5.9% | 4.7% | 7.0% | `q_proj`, `k_proj`, `v_proj`, `o_proj` | +| **total** | **0.752 B** | **1.882 B** | **4.206 B** | | + +The conventional `q,k,v,o` + MLP recipe therefore covers **41%** of the 0.8B and +leaves the SSM stack untouched in 18 of 24 layers. Adding the SSM Linears takes +coverage to 66%. At this scale that is a cheap ablation, not an agonising choice. + +⚠ **Packing across document boundaries is unsafe on this architecture.** An SSM +layer carries recurrent state along the sequence and an attention mask does not +reset it, so two renamed copies packed into one 8k window can bleed in 18 of 24 +layers — which is precisely the per-copy name-consistency invariant the design +doc calls sacred. Either one document per sequence, or prove the trainer's +sequence-boundary signal is honoured by the linear-attn path. Under a dense +carrier this problem does not exist. + +**The tied embedding is a third of the small carrier.** vocab 248,320 × hidden +1024 = 254 M of the 0.752 B. The transformer body being tested at the small end +is ~0.50 B, which matters when reporting "the carrier floor". + +## 4. Throughput — and the newest carrier is the slow one + +One forward+backward+AdamW microbatch, LoRA r=32/α=64 on `q,k,v,o` + MLP, +bf16, `sdpa`, gradient checkpointing on, seq 4096, on gx10's GB10. n=10 +measured after 3 warmup steps; median reported with the full spread. + +| carrier | architecture | params | s/step | tok/s | peak | spread | +|---|---|---|---|---|---|---| +| `Qwen3.5-0.8B-Base` | hybrid, 18 SSM / 6 attn | 0.765 B | 7.581 | **540** | 15.1 GiB | 2.6% | +| `Qwen3.5-0.8B-Base` (no grad-ckpt) | " | 0.765 B | 6.364 | 644 | 38.9 GiB | 1.5% | +| `Qwen3-0.6B-Base` | dense | 0.616 B | 1.707 | **2,399** | 9.8 GiB | 0.6% | +| `Qwen3-1.7B-Base` | dense | 1.755 B | 2.895 | **1,415** | 12.2 GiB | 0.8% | +| `Qwen3-1.7B-Base`, batch 4 | dense | 1.755 B | 11.387 | 1,439 | 38.0 GiB | 0.6% | +| `Qwen3.5-0.8B-Base`, batch 4 | hybrid | 0.765 B | 30.030 | 546 | 55.5 GiB | 0.7% | + +**The dense 1.755 B carrier trains 2.6× faster than the hybrid 0.765 B one** — on +2.3× the parameters, with *more* LoRA modules adapted (196 vs 96, because dense +has real attention in every layer). Per parameter the dense path is ~6× more +efficient. Spreads of 0.6–2.6% across n=10 put the instrument's noise an order of +magnitude below the effect, so this is not variance. + +The cause is almost certainly that **no fused linear-attention kernel is +installed** (§2) so the SSM path runs a reference implementation. Grad +checkpointing is *not* the culprit — turning it off recovers only 19% and costs +2.6× the memory, so leave it on. Batching is not the lever for either family: 1,415 → 1,439 +tok/s dense and 540 → 546 tok/s hybrid from batch 1 to 4. **Both architectures +are already at this box's roofline at batch 1**, which is a bandwidth story +(GB10's unified LPDDR5X against an RTX PRO 6000's ~6.6× higher figure) — and it +means the 2.6× gap is the kernel path, not a batching artefact. + +**What that does to the regime's premise.** Projecting a Brontë-scale corpus +(~1 M words ≈ 1.3 M tokens, × 6 rename copies, 3 epochs ≈ 23 M tokens): + +| carrier | projected wall-clock per voice | +|---|---| +| `Qwen3-0.6B-Base` dense | **2.7 h** | +| `Qwen3-1.7B-Base` dense | **4.6 h** | +| `Qwen3.5-0.8B-Base` hybrid | **12 h** | + +The hybrid carrier would make a per-voice run *longer than the 7 h 26B-A4B tune +it exists to replace.* At R49 H03's hoped-for corpus floor (~300 k words) the +dense 1.7B lands near **1.4 h** — a voice per afternoon, which is the regime the +operator asked for. ⚠ These are projections from a synthetic-token throughput +harness, not from a completed run; treat them as sizing, and re-measure on the +first real corpus. + +## 5. Prep remaining, in order + +1. **Carrier family decision** (§6) — everything downstream keys on it. +2. **Corpus D1** — Gutenberg Brontë (Jane Eyre, Villette, Shirley, The + Professor), boilerplate stripped, chapter-segmented, typography normalised, + character inventory recorded. Public domain, clean under any disposition. +3. **Re-point the R49 deterministic machinery at Brontë.** The entity detector + (corpus-level capitalised-vs-lowercase ratio), identity linking, gender + resolution and the 23,398-name dictionary were all built and hardened against + a *Yarros* sample. Per-work re-derivation needed: entity map, alphabet, and + the `UNRESOLVED_BLOCKING` human pass (~20–40 entities per work). +4. **Beat annotation (D4)** via `gen` inverse-prompting, using F02's hardened + prompt (banned meta-language, three PD worked examples, ≤20-word gate). +5. **Trainer.** `erp_sft_harness` is chat-shaped and carries ERP-specific + eligibility machinery; the author-voice job is plain continuation with a + masked prefix. Decision: a small purpose-built trainer that *keeps* the + harness's §4 disciplines (provenance pin, order manifest, truncation report, + cache key that sees semantic changes, recorded attention backend) rather than + a fork of its corpus logic. +6. **Pre-set the decision threshold before collecting data**, wider than the + measured seed-to-seed spread, per the R49 charter — and run the positive + control the R49 journal's own lesson demands: confirm the stylometric + instrument separates real Brontë from unadapted base output *before* it is + asked to judge an adapter. +7. **Launcher** modelled on `launch-run-07.sh` — its guards were each bought with + a past failure (GPU-clear assertion, pidfile not `pgrep -f`, refuse an + existing log, free-space floor, `setsid` detach). + +Deliberately out of scope here, per R49: the Director/critic loop, style +arithmetic and the Pelican test, multi-LoRA arsenal serving, the Skaldsong +integration contract, the modern in-copyright arsenal, inference latency. + +## 6. Open for the operator + +**(a) Carrier family — recommend the dense `Qwen3` line.** R49 H02 pins +`Qwen3.5-{0.8,2,4}B-Base`. The measurements say that family costs 2.6–6× the +wall-clock on this box, brings a vision tower and an MTP head to a text job, and +makes cross-document packing unsafe in 18 of 24 layers. `Qwen3-{0.6,1.7,4}B-Base` +— the design doc's own original pin — is plain dense, has none of those, and is +the better instrument for a probe whose whole point is isolating one variable. +The cost is one model generation of base quality. Reversible: the Qwen3.5 +checkpoints stay staged, and a fused-kernel install (`fla` is pure Triton and +would plausibly work on aarch64) could revive them later as a follow-up rather +than a blocker. + +**(b) 45 GB of intermediate checkpoints on gx10, and 354 GB on ana-ml2.** +`run-03c/04/05/06/checkpoints` total 45 GB on gx10; `/tank/erp-tune/serve` is +354 GB of superseded merged models on ana-ml2. The final adapters are safe (§7) +and run 6 is the standing seat. Purging is the operator's call — 470 GB free on +gx10 means it does not block this regime. + +## 7. Adapter disposition — settled, and made real + +Operator, 2026-09-09: **keep the adapter.** As of 22:30 PT all five +gx10-resident ERP adapters are mirrored to `ana-ml2:/tank/erp-tune/run-/adapter`, +matching the layout runs 01–03 already use there, byte-total identical on both +sides and `sha256` matching on every `adapter_model.safetensors`: + + run-03c run-04 run-05 run-06 run-07 315 MB each, 8 files each + +`/tank/*` is deliberately **excluded** from ana-ml2's restic sources — terabytes +of regenerable model weights. A trained adapter is the one thing under there +upstream cannot hand back, so `configs/restic/ana-ml2/profiles.yaml` now carries +a single documented carve-out, `/tank/erp-tune/run-*/adapter`, verified by +`resticprofile --dry-run` to expand to exactly those eight paths and nothing +else. The nightly 01:00 run picks them up. diff --git a/persistent-memory.md b/persistent-memory.md index f842df3..41f02d9 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-09 22:05 PT (**Pfish-6** = run-6 NVFP4 is the standing seat, ana-ml2 :8021 ONLY; run 7 PURGED ~139 GiB; pfi-gx10 is an experimental/TRAINING box and carries no serving seat; no new run planned)_ +_Last updated: 2026-09-09 22:50 PT (**Pfish-6** = run-6 NVFP4 is the standing seat, ana-ml2 :8021 ONLY; run 7 PURGED ~139 GiB; pfi-gx10 is an experimental/TRAINING box and carries no serving seat; all five ERP adapters now MIRRORED to ana-ml2 and inside restic; **BabyBronte / R49 author-voice regime is in PREP on gx10** — carrier-family decision open)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under an hour old, read it (it carries the in-flight @@ -125,9 +125,47 @@ preserved verbatim in `archival-memory.md` § Superseded in-flight snapshots._ `run-07/checkpoints` 9.2 GiB, `serve/pfish6-nvfp4a16` 16 GiB; ana-ml2 `erp-tune-v7-bf16` 49 GiB, `erp-tune-v7-nvfp4a16` 16 GiB, `erp-tune-v7-quant-work`. **~139 GiB reclaimed** (gx10 53%→47%). ⚠ **KEPT deliberately: `~/erp-tune/run-07/adapter` 315 MB + provenance + `loss-series-r7.json`** — - the only non-reproducible piece (14 h of training), and it costs nothing. `rm -rf - /home/infra-ops/erp-tune/run-07` finishes the job if wanted; everything else run-7 is already gone. -- **NO NEW TRAINING RUN PLANNED.** The opening-split idea is not being re-tested; run 6 stands. + the only non-reproducible piece (14 h of training), and it costs nothing. Everything else run-7 is + already gone; do NOT `rm -rf /home/infra-ops/erp-tune/run-07` — operator ruled **keep the adapter** + 2026-09-09. +- **NO NEW *ERP* TRAINING RUN PLANNED.** The opening-split idea is not being re-tested; run 6 stands. +- **✅ ALL FIVE gx10 ERP ADAPTERS ARE NOW TWO-COPY AND BACKED UP** (2026-09-09 22:30 PT, operator: + *"keep the adapter"*). `run-03c/04/05/06/07` mirrored to `ana-ml2:/tank/erp-tune/run-/adapter` + (the layout runs 01–03 already used), byte-totals identical both sides + `sha256` match on every + `adapter_model.safetensors`. ⚠ `/tank/*` is deliberately OUT of ana-ml2's restic sources (TB of + regenerable weights), so `configs/restic/ana-ml2/profiles.yaml` gained ONE documented carve-out — + `/tank/erp-tune/run-*/adapter` — verified by `resticprofile --dry-run` to expand to exactly those 8 + paths and nothing else. Live file is byte-identical to the repo canonical; `.bak-20260909` beside it. +- **🖋 BabyBronte / R49 author-voice LoRA regime — IN PREP on pfi-gx10, nothing training.** Plan + + every measured number: [`docs/pfi/author-voice-lora-regime.md`](docs/pfi/author-voice-lora-regime.md). + Research target is **brokkr-smithy R49** (`research/R49-author-voice-adapters/`) — brokkr owns + hypotheses/instruments/adjudication, infra-ops owns box+corpus staging+trainer+wall-clock, same split + as ERP runs 3c–7. Five carriers staged on gx10 `~/carriers/` (Qwen3.5-0.8/2/4B-Base + + Qwen3-0.6/1.7B-Base); probes committed at `scripts/training-probes/{probe_carrier,bench_lora_step}.py`. + ⚠⚠ **THE HEADLINE, AND IT INVERTS R49's PIN: the newest carrier is the SLOW one.** Measured on gx10 + (n=10, spread 0.6–2.6%, seq 4096, r=32 attn+mlp, bf16 sdpa, grad-ckpt on): dense `Qwen3-1.7B-Base` + **1,415 tok/s** vs hybrid `Qwen3.5-0.8B-Base` **540 tok/s** — the dense model is 2.6x faster on 2.3x + the parameters (~6x per-param), because Qwen3.5 is **18 SSM / 6 attention layers** and NO fused + linear-attention kernel is installed (`mamba_ssm`/`causal_conv1d`/`fla` all absent; triton 3.8 is + there). Projected per voice: dense 0.6B **2.7 h**, dense 1.7B **4.6 h**, hybrid 0.8B **12 h** — the + hybrid would take LONGER than the 7 h 26B-A4B tune it exists to replace, killing the regime's whole + premise. Grad-ckpt is not the cause (19%, and saves 2.6x memory — keep it); batching is not the lever + (dense 1,415→1,439 and hybrid 540→546 tok/s at batch 4 — BOTH already at this box's roofline at batch 1, a bandwidth story, so the 2.6x gap is the kernel path not a batching artefact). + ⚠ Two more Qwen3.5 landmines, both measured: it **ships a vision tower** (153/297 `model.visual.*` + Linear tensors that `all-linear` would train on text — the same defect the gemma-4 audit caught; + `AutoModelForCausalLM` drops it and the MTP head for free, BUT that renames modules + `model.layers.N.*` vs vLLM's `model.language_model.layers.N.*`, so an adapter may not bind → run the + sampled-target-changed check in the SERVING path); and **cross-document packing is unsafe** because + SSM state ignores the attention mask, which breaks the per-copy name-consistency invariant the design + doc calls sacred. None of these exist on the dense line. + **⭐ OPEN FOR THE OPERATOR: carrier family — recommend the dense `Qwen3-{0.6,1.7,4}B-Base` line** + (the design doc's own original pin) over R49 H02's `Qwen3.5` trio. Reversible: Qwen3.5 stays staged + and an `fla` install (pure Triton, plausibly fine on aarch64) could revive it as a follow-up. +- **📌 forseti shipped althing 3.6.1 (2026-09-09) — every box's herald needs a restart to pick up the + new poke text.** `uv tool install --force --reinstall ` + `systemctl --user restart + althing-po-herald`, expect `3.6.1`; the plugin also went to 0.1.6 (inbox.md no longer calls the + result reply "optional"). NOT DONE — deferred, nothing is blocked on it, and it is a fleet-wide + multi-box pass with its own verification burden. - **⏳ ana-ml2 storage follow-ups, operator's call** (the three actions themselves LANDED 09-09 02:02): (a) **ZFS pool-health ALERTING** — `tank` sat DEGRADED 04-23→09-05 with nvme7 physically absent and nobody knew (ZED mails `root`, no MTA on the box); (b) nvme7 / slot 0-5 keep-vs-replace — diff --git a/scripts/training-probes/bench-lora-step-gx10-2026-09-09.jsonl b/scripts/training-probes/bench-lora-step-gx10-2026-09-09.jsonl new file mode 100644 index 0000000..ae5247e --- /dev/null +++ b/scripts/training-probes/bench-lora-step-gx10-2026-09-09.jsonl @@ -0,0 +1,6 @@ +{"model": "Qwen3.5-0.8B-Base", "total_params_B": 0.765, "lora_rank": 32, "targets": "attn_mlp", "adapted_modules": 96, "trainable_params_M": 12.78, "trainable_pct": 1.67, "batch": 1, "seq": 4096, "tokens_per_microbatch": 4096, "grad_checkpointing": true, "attn_impl": "sdpa", "dtype": "bfloat16", "device": "NVIDIA GB10", "torch": "2.14.0+cu130", "warmup": 3, "n": 10, "s_per_step_median": 7.5814, "s_per_step_min": 7.4864, "s_per_step_max": 7.6843, "s_per_step_spread_pct": 2.6, "tok_per_s_median": 540.3, "peak_mem_GiB": 15.1} +{"model": "Qwen3.5-0.8B-Base", "total_params_B": 0.765, "lora_rank": 32, "targets": "attn_mlp", "adapted_modules": 96, "trainable_params_M": 12.78, "trainable_pct": 1.67, "batch": 1, "seq": 4096, "tokens_per_microbatch": 4096, "grad_checkpointing": false, "attn_impl": "sdpa", "dtype": "bfloat16", "device": "NVIDIA GB10", "torch": "2.14.0+cu130", "warmup": 3, "n": 10, "s_per_step_median": 6.3639, "s_per_step_min": 6.3549, "s_per_step_max": 6.4533, "s_per_step_spread_pct": 1.5, "tok_per_s_median": 643.6, "peak_mem_GiB": 38.91} +{"model": "Qwen3-0.6B-Base", "total_params_B": 0.616, "lora_rank": 32, "targets": "attn_mlp", "adapted_modules": 196, "trainable_params_M": 20.19, "trainable_pct": 3.276, "batch": 1, "seq": 4096, "tokens_per_microbatch": 4096, "grad_checkpointing": true, "attn_impl": "sdpa", "dtype": "bfloat16", "device": "NVIDIA GB10", "torch": "2.14.0+cu130", "warmup": 3, "n": 10, "s_per_step_median": 1.7072, "s_per_step_min": 1.6993, "s_per_step_max": 1.7097, "s_per_step_spread_pct": 0.6, "tok_per_s_median": 2399.2, "peak_mem_GiB": 9.75} +{"model": "Qwen3-1.7B-Base", "total_params_B": 1.755, "lora_rank": 32, "targets": "attn_mlp", "adapted_modules": 196, "trainable_params_M": 34.87, "trainable_pct": 1.986, "batch": 1, "seq": 4096, "tokens_per_microbatch": 4096, "grad_checkpointing": true, "attn_impl": "sdpa", "dtype": "bfloat16", "device": "NVIDIA GB10", "torch": "2.14.0+cu130", "warmup": 3, "n": 10, "s_per_step_median": 2.8946, "s_per_step_min": 2.8782, "s_per_step_max": 2.901, "s_per_step_spread_pct": 0.8, "tok_per_s_median": 1415.0, "peak_mem_GiB": 12.24} +{"model": "Qwen3-1.7B-Base", "total_params_B": 1.755, "lora_rank": 32, "targets": "attn_mlp", "adapted_modules": 196, "trainable_params_M": 34.87, "trainable_pct": 1.986, "batch": 4, "seq": 4096, "tokens_per_microbatch": 16384, "grad_checkpointing": true, "attn_impl": "sdpa", "dtype": "bfloat16", "device": "NVIDIA GB10", "torch": "2.14.0+cu130", "warmup": 3, "n": 10, "s_per_step_median": 11.3872, "s_per_step_min": 11.3401, "s_per_step_max": 11.4128, "s_per_step_spread_pct": 0.6, "tok_per_s_median": 1438.8, "peak_mem_GiB": 37.99} +{"model": "Qwen3.5-0.8B-Base", "total_params_B": 0.765, "lora_rank": 32, "targets": "attn_mlp", "adapted_modules": 96, "trainable_params_M": 12.78, "trainable_pct": 1.67, "batch": 4, "seq": 4096, "tokens_per_microbatch": 16384, "grad_checkpointing": true, "attn_impl": "sdpa", "dtype": "bfloat16", "device": "NVIDIA GB10", "torch": "2.14.0+cu130", "warmup": 3, "n": 10, "s_per_step_median": 30.0301, "s_per_step_min": 29.9379, "s_per_step_max": 30.1444, "s_per_step_spread_pct": 0.7, "tok_per_s_median": 545.6, "peak_mem_GiB": 55.49} diff --git a/scripts/training-probes/bench_lora_step.py b/scripts/training-probes/bench_lora_step.py new file mode 100644 index 0000000..a33e07b --- /dev/null +++ b/scripts/training-probes/bench_lora_step.py @@ -0,0 +1,93 @@ +"""Throughput floor for an R49 author-voice LoRA step on pfi-gx10 (GB10, sm_121). + +Measures the cost of ONE forward+backward+optimizer microbatch on synthetic +tokens, so a full-corpus wall-clock can be projected before any corpus exists. + +Deliberately synthetic: random token ids exercise the same kernels at the same +shapes as real text, and this is a THROUGHPUT harness only -- it says nothing +about loss, quality, or voice transfer. The harness is part of the number, so +every knob is printed with the result. + + python bench_lora_step.py --seq 4096 --targets attn_mlp|all_linear_text +""" +import argparse, json, statistics, time, os +import torch +from transformers import AutoModelForCausalLM, AutoConfig +from peft import LoraConfig, get_peft_model + +ATTN_MLP = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"] +PLUS_SSM = ATTN_MLP + ["in_proj_qkv", "in_proj_a", "in_proj_b", "in_proj_z", "out_proj"] + +ap = argparse.ArgumentParser() +ap.add_argument("model") +ap.add_argument("--seq", type=int, default=4096) +ap.add_argument("--batch", type=int, default=1) +ap.add_argument("--rank", type=int, default=32) +ap.add_argument("--targets", choices=["attn_mlp", "plus_ssm"], default="attn_mlp") +ap.add_argument("--warmup", type=int, default=3) +ap.add_argument("--steps", type=int, default=10) +ap.add_argument("--no-grad-ckpt", action="store_true") +ap.add_argument("--attn", default="sdpa") +a = ap.parse_args() + +torch.manual_seed(0) +cfg = AutoConfig.from_pretrained(a.model) +vocab = getattr(getattr(cfg, "text_config", cfg), "vocab_size") + +model = AutoModelForCausalLM.from_pretrained( + a.model, dtype=torch.bfloat16, attn_implementation=a.attn, +).to("cuda") +targets = ATTN_MLP if a.targets == "attn_mlp" else PLUS_SSM +peft_cfg = LoraConfig( + r=a.rank, lora_alpha=2 * a.rank, lora_dropout=0.0, bias="none", + task_type="CAUSAL_LM", target_modules=targets, +) +model = get_peft_model(model, peft_cfg) +if not a.no_grad_ckpt: + model.gradient_checkpointing_enable() + model.enable_input_require_grads() +model.train() + +trainable = sum(p.numel() for p in model.parameters() if p.requires_grad) +total = sum(p.numel() for p in model.parameters()) +n_adapted = sum(1 for n, _ in model.named_modules() if n.endswith("lora_A.default")) + +opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=1e-4) +ids = torch.randint(0, vocab, (a.batch, a.seq), device="cuda") + +def step(): + opt.zero_grad(set_to_none=True) + out = model(input_ids=ids, labels=ids) + out.loss.backward() + opt.step() + return float(out.loss) + +for _ in range(a.warmup): + step() +torch.cuda.synchronize() + +lat = [] +for _ in range(a.steps): + t0 = time.perf_counter() + step() + torch.cuda.synchronize() + lat.append(time.perf_counter() - t0) + +tok = a.batch * a.seq +res = dict( + model=os.path.basename(a.model.rstrip("/")), + total_params_B=round(total / 1e9, 3), + lora_rank=a.rank, targets=a.targets, adapted_modules=n_adapted, + trainable_params_M=round(trainable / 1e6, 2), + trainable_pct=round(100 * trainable / total, 3), + batch=a.batch, seq=a.seq, tokens_per_microbatch=tok, + grad_checkpointing=not a.no_grad_ckpt, attn_impl=a.attn, + dtype="bfloat16", device=torch.cuda.get_device_name(0), + torch=torch.__version__, warmup=a.warmup, n=a.steps, + s_per_step_median=round(statistics.median(lat), 4), + s_per_step_min=round(min(lat), 4), s_per_step_max=round(max(lat), 4), + s_per_step_spread_pct=round(100 * (max(lat) - min(lat)) / statistics.median(lat), 1), + tok_per_s_median=round(tok / statistics.median(lat), 1), + peak_mem_GiB=round(torch.cuda.max_memory_allocated() / 2**30, 2), +) +print(json.dumps(res)) diff --git a/scripts/training-probes/probe_carrier.py b/scripts/training-probes/probe_carrier.py new file mode 100644 index 0000000..cd7d59e --- /dev/null +++ b/scripts/training-probes/probe_carrier.py @@ -0,0 +1,69 @@ +"""Read-only structural probe of an R49 candidate carrier. + +Answers, by measurement rather than by reading the config: + * does transformers on this box load the checkpoint at all, + * which module paths are nn.Linear (the only LoRA-attachable leaves), + * how the parameter budget splits across text body / vision tower / MTP / + embeddings, so "0.8B carrier" can be reported honestly, + * which attention implementations the class accepts. + +Loads on CPU in bf16. No training, no GPU, nothing written but stdout. +""" +import json, sys, collections, re +import torch +from transformers import AutoConfig, AutoModelForCausalLM + +path = sys.argv[1] +print(f"== {path}") +cfg = AutoConfig.from_pretrained(path, trust_remote_code=False) +print(" config class :", type(cfg).__name__) +print(" architectures :", getattr(cfg, "architectures", None)) +tc = getattr(cfg, "text_config", None) +if tc is not None: + lt = getattr(tc, "layer_types", None) or [] + print(" text layers :", getattr(tc, "num_hidden_layers", "?"), + "| full_attention:", lt.count("full_attention"), + "| linear_attention:", lt.count("linear_attention")) + print(" hidden/inter :", getattr(tc, "hidden_size", "?"), "/", getattr(tc, "intermediate_size", "?")) + print(" vocab :", getattr(tc, "vocab_size", "?"), "| tied:", getattr(tc, "tie_word_embeddings", "?")) + +try: + model = AutoModelForCausalLM.from_pretrained( + path, dtype=torch.bfloat16, device_map="cpu", + attn_implementation="sdpa", + ) +except Exception as e: + print(" LOAD FAILED:", type(e).__name__, str(e)[:400]) + raise SystemExit(1) +print(" model class :", type(model).__name__) +print(" attn impl :", getattr(model.config, "_attn_implementation", "?")) + +# --- parameter budget ------------------------------------------------------ +buckets = collections.Counter() +def bucket(name): + if ".visual." in name or name.startswith("visual."): return "vision_tower" + if name.startswith("mtp.") or ".mtp." in name: return "mtp_head" + if "embed_tokens" in name or name.endswith("lm_head.weight"): return "embeddings" + if "linear_attn" in name: return "text_linear_attn" + if "self_attn" in name: return "text_full_attn" + if ".mlp." in name: return "text_mlp" + return "text_other" +for n, p in model.named_parameters(): + buckets[bucket(n)] += p.numel() +total = sum(buckets.values()) +print(f" TOTAL params : {total/1e9:.3f} B") +for k, v in sorted(buckets.items(), key=lambda kv: -kv[1]): + print(f" {k:<18} {v/1e6:9.1f} M ({100*v/total:5.1f}%)") + +# --- LoRA-attachable leaves ------------------------------------------------ +lin = collections.defaultdict(list) +for name, mod in model.named_modules(): + if isinstance(mod, torch.nn.Linear): + lin[bucket(name + ".weight")].append(name) +print(" nn.Linear leaves by region:") +for region in sorted(lin): + names = lin[region] + tmpl = sorted({re.sub(r"\.\d+\.", ".N.", n) for n in names}) + print(f" {region:<18} {len(names):4d} modules, {len(tmpl)} distinct shapes") + for t in tmpl: + print(f" {t}")