vh
7b5fd91d3c
Run-01 was killed at step 19 by operator instruction to root-cause before spending a ~13.9h window. Two independent methods now agree on where the step time went, and neither was the hypothesis the consult panel converged on. Scaling fit (3 points, 2 params, residuals <3ms over an 8x range): A = 6.87e-4 s/token, B = 8.85e-8 s/token^2 quadratic share 20.9% @ w=2048 -> 67.8% @ w=16384 No fixed term was needed, which refutes launch-bound outright. Profiler kernel table (device rows only): attention 22,835.8 ms 65.2% fmha_cutlass*_sm80 dense GEMM 2,774.0 ms 7.9% other 5,739.0 ms 16.4% The attention kernels are sm80 — Ampere-generation CUTLASS running on an sm_120 Blackwell card, with the forward on the gmem fallback tier. That is the mechanism behind 100% SM utilisation at 27 of 304 available TFLOPS. Correctness cleared separately: the sliding mask asserts at max 1024 allowed/row, so the 25 windowed layers were genuinely windowed. The same probe found that right-padding is what pins the 5 global layers to an explicit 4D mask and off the is_causal fast path — measured at 9.4% slower for 24% less loss work at fixed width. The largest available win is not the attention kernel. The corpus is 29.9% padding, and bucket-to-pair + shuffle-to-mix takes it to 0.0% for >=35.5% wall clock, no new dependency, unchanged peak memory. Bucket size turned out not to be a diversity knob — roots per accumulation window are flat across a 256x range, so the global micro-batch shuffle does that work alone and the bucket should be tight. Adds docs/pfi/training-throughput-playbook.md as the durable model-agnostic home (sibling to the quantization playbook), the four probes under scripts/training-probes/ with raw output kept for re-derivation, and a §6 to the sizing doc carrying the Gemma-4-specific numbers and round-2 restart parameters. Measured negatives recorded so they are not re-chased: grouped_mm (0.9% slower, and MoE is only 7.9% of the step), CUDA graphs / torch.compile over the expert loop (no fixed cost to amortise), liger fused CE (~1-3% lever), FA4 on sm_120. Round-1 state preserved: 609MB encode cache, order manifest, truncation report, resume script. No checkpoints — it died at step 19 and the first was due at 100, so the lora_B inert-adapter gate never ran and moves to the restart.
113 lines
4.4 KiB
Python
113 lines
4.4 KiB
Python
"""Measure bucket-to-pair / shuffle-to-mix against the REAL encode cache.
|
|
|
|
Brokkr's design, validated on measured record lengths rather than a calibrated
|
|
length model:
|
|
|
|
1. sort records by length
|
|
2. cut into buckets of BUCKET records
|
|
3. form micro-batches of 2 WITHIN each bucket (adjacent after sort)
|
|
4. shuffle the resulting MICRO-BATCHES globally, seeded
|
|
|
|
Padding efficiency is a property of the pairing only, so step 4 costs nothing
|
|
and restores root-mixing inside each accumulation window.
|
|
|
|
Also applies the fitted cost model from the replica scaling test to convert
|
|
token savings into predicted wall clock.
|
|
"""
|
|
import json
|
|
import random
|
|
from collections import Counter
|
|
|
|
CACHE = "/tank/erp-tune/run-01/encode-cache/encoded-a4b0796de1260930.jsonl"
|
|
IGNORE_INDEX = -100
|
|
MB = 2
|
|
ACCUM = 8
|
|
SEED = 20260824
|
|
|
|
# fitted on the replica: t(w) = A*w + B*w^2 for a batch of 2 sequences of len w
|
|
A = 6.8715e-04
|
|
B = 8.8509e-08
|
|
|
|
rows = []
|
|
with open(CACHE) as fh:
|
|
for line in fh:
|
|
r = json.loads(line)
|
|
rows.append((len(r["input_ids"]), r.get("dataset_id", "?"),
|
|
r.get("sample_kind", "?")))
|
|
n = len(rows)
|
|
print("records %d" % n)
|
|
print()
|
|
|
|
|
|
def evaluate(order, label, show_roots=False):
|
|
real = padded = 0
|
|
widths = []
|
|
batches = []
|
|
for i in range(0, n - n % MB, MB):
|
|
grp = [rows[j] for j in order[i:i + MB]]
|
|
w = max(g[0] for g in grp)
|
|
real += sum(g[0] for g in grp)
|
|
padded += w * MB
|
|
widths.append(w)
|
|
batches.append([g[1] for g in grp])
|
|
nb = len(widths)
|
|
Ew = sum(widths) / nb
|
|
Ew2 = sum(w * w for w in widths) / nb
|
|
t_mb = A * Ew + B * Ew2
|
|
srt = sorted(widths)
|
|
print("--- %s ---" % label)
|
|
print(" padded tokens %s" % f"{padded:,}")
|
|
print(" waste %.1f%%" % (100 * (1 - real / padded)))
|
|
print(" E[w] (per-seq) %.0f" % Ew)
|
|
print(" E[w^2] %.3e" % Ew2)
|
|
print(" width p50/p90/p99 %d / %d / %d" % (
|
|
srt[nb // 2], srt[int(nb * .9)], srt[int(nb * .99)]))
|
|
print(" predicted micro-batch %.3f s (lin %.3f + quad %.3f, quad %.0f%%)" % (
|
|
t_mb, A * Ew, B * Ew2, 100 * B * Ew2 / t_mb))
|
|
print(" predicted step (x%d) %.1f s -> %.2f h over 1312 steps" % (
|
|
ACCUM, t_mb * ACCUM, t_mb * ACCUM * 1312 / 3600))
|
|
# unpadded micro-batches take the is_causal fast path on the 5 global layers
|
|
exact = sum(1 for i in range(0, n - n % MB, MB)
|
|
if len(set(rows[j][0] for j in order[i:i + MB])) == 1)
|
|
print(" ZERO-PAD micro-batches %d / %d (%.1f%%) <- global layers on is_causal" % (
|
|
exact, nb, 100 * exact / nb))
|
|
if show_roots:
|
|
# root diversity inside an accumulation window
|
|
div = []
|
|
for i in range(0, nb - nb % ACCUM, ACCUM):
|
|
win = [d for b in batches[i:i + ACCUM] for d in b]
|
|
div.append(len(set(win)))
|
|
print(" roots per accum window mean %.2f min %d (of %d roots)" % (
|
|
sum(div) / len(div), min(div), len({r[1] for r in rows})))
|
|
print()
|
|
return padded, t_mb
|
|
|
|
|
|
# --- current: encode-cache order, SequentialSampler ---
|
|
cur_padded, cur_t = evaluate(list(range(n)), "CURRENT (SequentialSampler)", True)
|
|
|
|
# --- bucket-to-pair + shuffle-to-mix ---
|
|
# BUCKET controls the efficiency-vs-diversity trade: records are globally
|
|
# sorted, cut into buckets of BUCKET, SHUFFLED WITHIN the bucket (not
|
|
# re-sorted), then paired adjacently. BUCKET=2 is a perfect global sort
|
|
# (0% waste, worst root mixing); larger buckets admit more length spread
|
|
# inside a pair but draw partners from a wider slice of the corpus.
|
|
for BUCKET in (2, 8, 32, 128, 512):
|
|
by_len = sorted(range(n), key=lambda i: rows[i][0])
|
|
rng = random.Random(SEED)
|
|
micro = []
|
|
for s in range(0, n, BUCKET):
|
|
chunk = by_len[s:s + BUCKET]
|
|
rng.shuffle(chunk) # mix WITHIN the length bucket
|
|
for k in range(0, len(chunk) - len(chunk) % MB, MB):
|
|
micro.append(chunk[k:k + MB])
|
|
rng.shuffle(micro) # shuffle-to-mix across buckets
|
|
order = [i for b in micro for i in b]
|
|
placed = set(order)
|
|
order += [i for i in by_len if i not in placed]
|
|
p, t = evaluate(order, "BUCKET=%d, shuffle within + global micro-batch shuffle" % BUCKET,
|
|
True)
|
|
print(" >>> vs current: %.1f%% fewer padded tokens, %.1f%% less wall clock" % (
|
|
100 * (1 - p / cur_padded), 100 * (1 - t / cur_t)))
|
|
print()
|