vh
36f1b70a88
Two operator rulings, 2026-09-09.
"purge intermediate checkpoints" -- seven checkpoints/ directories removed with
literal paths, one rm per line, after confirming none was a symlink and that
every run's final adapter/ is an independent real directory:
pfi-gx10 run-03c 11G run-04 16G run-05 9.2G run-06 9.2G = 45 GB
ana-ml2 run-01 12G run-02 12G run-03 5.9G = 29 GB
gx10 419G->374G used, 496 GB free. /tank/erp-tune 392G->363G with zfs list -t
snapshot empty, so the space is genuinely returned rather than snapshot-held. All
eight adapters re-verified by sha256 after the deletion, matching the values
recorded during the mirror. Merged artifacts deliberately untouched -- they are
not checkpoints, and the ~550 GB of superseded merges stays a separate call.
"use dense qwen3" -- the R49 H02 carrier sweep becomes Qwen3-{0.6,1.7,4}B-Base,
which overrides the Qwen3.5 arms H02 names; brokkr-smithy owns that file and was
told directly. Qwen3-4B-Base staged and benched to complete the family:
Qwen3-0.6B-Base 0.616 B 1.707 s/step 2,399 tok/s spread 0.6%
Qwen3-1.7B-Base 1.755 B 2.895 s/step 1,415 tok/s spread 0.8%
Qwen3-4B-Base 4.089 B 5.714 s/step 717 tok/s spread 0.3%
The dense 4.089 B carrier still trains 33% faster than the hybrid 0.765 B one.
Projected per voice 2.7 / 4.6 / 9.1 h; the three-arm sweep at two seeds is ~33 h
of GPU, ~10 h if H03's corpus floor holds. The three Qwen3.5 checkpoints stay
staged so the decision is reversible behind an fla install.
Also recorded: verified at 22:45-22:48 PT that nothing is training on gx10,
ana-ml2, nh3-dev or irv-ml1, and that brokkr's own run07-gate close states
"Nothing is owed. No battery to run." Run 7 has no servable artifact left. And a
correction to a standing lesson -- the bracketed-class trick does not defeat a
wrapper's argv, since the invoking shell's command line carries the literal
pattern; observe the artifact instead.
Training throughput probes
Instruments for finding where a training step's time actually went. Written
2026-08-24 during the Gemma-4 26B-A4B ERP/RP tune investigation; the lessons
they produced live in
docs/pfi/training-throughput-playbook.md.
These are diagnostic instruments, not production code. They hard-code paths for that run. Adapt the constants at the top; keep the measurement design.
The probes
| script | settles | GPU | runtime |
|---|---|---|---|
step0_mask.py |
mask band structure; which layers keep the is_causal fast path |
no | ~30 s |
step2_padding.py |
padding waste, length distribution, CE chunk sizing | no | ~2 min |
step_bucket.py |
bucketing gain, bucket-size sweep, source diversity | no | ~3 min |
step1_profile.py |
scaling fit, padding penalty, CE wall clock, kernel table | yes | ~15 min |
Run in that order. Only the last needs the real checkpoint, and it wants an
idle card — it loads ~48 GiB and peaks near 77 GiB at 2 × 16,384.
Design rules worth preserving when you adapt these
step1_profile.py reuses the harness's own discover_target_modules and
replicates its compute_loss byte-for-byte rather than re-implementing the
step. A probe that reimplements the training step measures the probe. If you
port this, keep the import from the real harness.
step0_mask.py needs no weights and no GPU — SDPA backend selection and
mask construction depend on shapes, dtype and mask presence, not on weight
values. That is what makes the correctness assertion cheap enough to run before
every job.
The scaling test takes three points, not two. Two points over three plausible terms (quadratic, linear, fixed-per-batch) is underdetermined; see playbook §1.1 for the hour that cost.
step_bucket.py sweeps bucket size deliberately. The first version
re-sorted within each bucket, which silently collapsed every bucket size to a
full global sort and made the sweep a no-op. If you change the pairing logic,
check that the sweep still varies something.
Raw evidence
step1-profile-output-2026-08-24.txt is the unedited output of the run the
playbook's numbers come from — scaling points, padding penalty, CE timing, and
the full key_averages() kernel table. Kept so the claims can be re-derived
rather than taken on faith.
⚠ That table double-counts: key_averages() lists both the ATen op and the
CUDA kernel it launched, each carrying the same self device time. Sum device
kernel rows only. See playbook §3.4.