2 Commits

Include renames

Author	SHA1	Message	Date
vh	36f1b70a88	chore(erp-tune): purge intermediate checkpoints (~74 GB); R49 carrier settled on dense Qwen3 ... Two operator rulings, 2026-09-09. "purge intermediate checkpoints" -- seven checkpoints/ directories removed with literal paths, one rm per line, after confirming none was a symlink and that every run's final adapter/ is an independent real directory: pfi-gx10 run-03c 11G run-04 16G run-05 9.2G run-06 9.2G = 45 GB ana-ml2 run-01 12G run-02 12G run-03 5.9G = 29 GB gx10 419G->374G used, 496 GB free. /tank/erp-tune 392G->363G with zfs list -t snapshot empty, so the space is genuinely returned rather than snapshot-held. All eight adapters re-verified by sha256 after the deletion, matching the values recorded during the mirror. Merged artifacts deliberately untouched -- they are not checkpoints, and the ~550 GB of superseded merges stays a separate call. "use dense qwen3" -- the R49 H02 carrier sweep becomes Qwen3-{0.6,1.7,4}B-Base, which overrides the Qwen3.5 arms H02 names; brokkr-smithy owns that file and was told directly. Qwen3-4B-Base staged and benched to complete the family: Qwen3-0.6B-Base 0.616 B 1.707 s/step 2,399 tok/s spread 0.6% Qwen3-1.7B-Base 1.755 B 2.895 s/step 1,415 tok/s spread 0.8% Qwen3-4B-Base 4.089 B 5.714 s/step 717 tok/s spread 0.3% The dense 4.089 B carrier still trains 33% faster than the hybrid 0.765 B one. Projected per voice 2.7 / 4.6 / 9.1 h; the three-arm sweep at two seeds is ~33 h of GPU, ~10 h if H03's corpus floor holds. The three Qwen3.5 checkpoints stay staged so the decision is reversible behind an fla install. Also recorded: verified at 22:45-22:48 PT that nothing is training on gx10, ana-ml2, nh3-dev or irv-ml1, and that brokkr's own run07-gate close states "Nothing is owed. No battery to run." Run 7 has no servable artifact left. And a correction to a standing lesson -- the bracketed-class trick does not defeat a wrapper's argv, since the invoking shell's command line carries the literal pattern; observe the artifact instead.	2026-09-09 22:55:34 -07:00
vh	7db6c44bcd	feat(r49-prep): author-voice LoRA regime prep on gx10 — carriers staged, throughput measured, adapters secured ... Prep for the BabyBronte / brokkr-smithy R49 author-voice adapter regime, plus the operator's "keep the adapter" ruling made durable. Measured on pfi-gx10 (GB10, sm_121), n=10 per arm after 3 warmup steps, seq 4096, LoRA r=32 on q/k/v/o + MLP, bf16, sdpa, grad-checkpointing on: Qwen3-0.6B-Base dense 0.616 B 1.707 s/step 2,399 tok/s Qwen3-1.7B-Base dense 1.755 B 2.895 s/step 1,415 tok/s Qwen3.5-0.8B-Base hybrid 0.765 B 7.581 s/step 540 tok/s The dense 1.755 B carrier trains 2.6x faster than the hybrid 0.765 B one on 2.3x the parameters (~6x per parameter), with more LoRA modules adapted (196 vs 96). Spreads of 0.6-2.6% put instrument noise an order of magnitude below the effect. Cause: Qwen3.5 is 18 linear-attention (SSM) layers to 6 attention, and no fused linear-attention kernel is installed on the box. Grad checkpointing is not the culprit (19%, and saves 2.6x memory). Batching is not the lever for either family -- both sit at this box's roofline at batch 1. Projected per voice on a Brontë-scale corpus: dense 0.6B 2.7 h, dense 1.7B 4.6 h, hybrid 0.8B 12 h. The hybrid would take longer than the 7 h 26B-A4B tune the regime exists to replace, so the carrier family is now an open decision with a recommendation for the dense Qwen3 line -- the design doc's original pin. Two further Qwen3.5 findings, both measured rather than read off the config: the Base checkpoints ship a vision tower (153/297 model.visual.* Linear tensors that target_modules="all-linear" would train on text) and an MTP head, both dropped for free by loading through AutoModelForCausalLM -- which renames modules relative to the vLLM serving path, so adapter binding needs the sampled-target-changed check on the serving side; and cross-document packing is unsafe because SSM state ignores the attention mask, breaking the per-copy name-consistency invariant the design doc calls sacred. Neither exists on dense. Adapter disposition, per the operator's ruling: all five gx10-resident ERP adapters (run-03c/04/05/06/07) mirrored to ana-ml2:/tank/erp-tune/run-<N>/adapter matching the layout runs 01-03 already used, byte-totals identical both sides and sha256 matching on every adapter_model.safetensors. /tank/* is deliberately excluded from ana-ml2's restic sources, so the profile gains one documented carve-out for /tank/erp-tune/run-*/adapter, verified by resticprofile --dry-run to expand to exactly those eight paths. Nothing is training and nothing is queued.	2026-09-09 22:41:47 -07:00