vh
7b5fd91d3c
Run-01 was killed at step 19 by operator instruction to root-cause before spending a ~13.9h window. Two independent methods now agree on where the step time went, and neither was the hypothesis the consult panel converged on. Scaling fit (3 points, 2 params, residuals <3ms over an 8x range): A = 6.87e-4 s/token, B = 8.85e-8 s/token^2 quadratic share 20.9% @ w=2048 -> 67.8% @ w=16384 No fixed term was needed, which refutes launch-bound outright. Profiler kernel table (device rows only): attention 22,835.8 ms 65.2% fmha_cutlass*_sm80 dense GEMM 2,774.0 ms 7.9% other 5,739.0 ms 16.4% The attention kernels are sm80 — Ampere-generation CUTLASS running on an sm_120 Blackwell card, with the forward on the gmem fallback tier. That is the mechanism behind 100% SM utilisation at 27 of 304 available TFLOPS. Correctness cleared separately: the sliding mask asserts at max 1024 allowed/row, so the 25 windowed layers were genuinely windowed. The same probe found that right-padding is what pins the 5 global layers to an explicit 4D mask and off the is_causal fast path — measured at 9.4% slower for 24% less loss work at fixed width. The largest available win is not the attention kernel. The corpus is 29.9% padding, and bucket-to-pair + shuffle-to-mix takes it to 0.0% for >=35.5% wall clock, no new dependency, unchanged peak memory. Bucket size turned out not to be a diversity knob — roots per accumulation window are flat across a 256x range, so the global micro-batch shuffle does that work alone and the bucket should be tight. Adds docs/pfi/training-throughput-playbook.md as the durable model-agnostic home (sibling to the quantization playbook), the four probes under scripts/training-probes/ with raw output kept for re-derivation, and a §6 to the sizing doc carrying the Gemma-4-specific numbers and round-2 restart parameters. Measured negatives recorded so they are not re-chased: grouped_mm (0.9% slower, and MoE is only 7.9% of the step), CUDA graphs / torch.compile over the expert loop (no fixed cost to amortise), liger fused CE (~1-3% lever), FA4 on sm_120. Round-1 state preserved: 609MB encode cache, order manifest, truncation report, resume script. No checkpoints — it died at step 19 and the first was due at 100, so the lora_B inert-adapter gate never ran and moves to the restart.
78 lines
3.1 KiB
Python
78 lines
3.1 KiB
Python
"""Step 2 — padding ratio. Data-side, no GPU, no model.
|
|
|
|
Replicates the exact batching the trainer used: SequentialSampler over the
|
|
encode-cache order, per_device_batch_size=2, collate_mixed right-padding to
|
|
the pair max. Reports real vs padded token counts and the loss-target count
|
|
that sizes the chunked CE.
|
|
"""
|
|
import json, sys
|
|
from collections import Counter
|
|
|
|
CACHE = "/tank/erp-tune/run-01/encode-cache/encoded-a4b0796de1260930.jsonl"
|
|
IGNORE_INDEX = -100
|
|
MB = 2 # per_device_batch_size
|
|
ACCUM = 8 # gradient_accumulation_steps
|
|
|
|
lens, kept_counts, kinds = [], [], []
|
|
with open(CACHE) as fh:
|
|
for line in fh:
|
|
row = json.loads(line)
|
|
ids = row["input_ids"]
|
|
labels = row["labels"]
|
|
lens.append(len(ids))
|
|
kept_counts.append(sum(1 for x in labels if x != IGNORE_INDEX))
|
|
kinds.append(row.get("sample_kind", "?"))
|
|
|
|
n = len(lens)
|
|
print(f"records {n:,}")
|
|
print(f"sample_kind mix {dict(Counter(kinds))}")
|
|
print()
|
|
print(f"seq len min/mean/max {min(lens)} / {sum(lens)/n:.0f} / {max(lens)}")
|
|
print(f"loss targets min/mean/max {min(kept_counts)} / {sum(kept_counts)/n:.0f} / {max(kept_counts)}")
|
|
print()
|
|
|
|
# --- micro-batch padding, exactly as collate_mixed builds it ---
|
|
real = padded = 0
|
|
mb_widths, mb_waste, mb_kept = [], [], []
|
|
for i in range(0, n - n % MB, MB):
|
|
group = lens[i:i + MB]
|
|
width = max(group)
|
|
r = sum(group)
|
|
p = width * MB
|
|
real += r
|
|
padded += p
|
|
mb_widths.append(width)
|
|
mb_waste.append(1 - r / p)
|
|
mb_kept.append(sum(kept_counts[i:i + MB]))
|
|
|
|
nb = len(mb_widths)
|
|
print(f"micro-batches (mb={MB}) {nb:,}")
|
|
print(f"real tokens {real:,}")
|
|
print(f"padded tokens {padded:,}")
|
|
print(f"PADDING WASTE {100 * (1 - real / padded):.1f}% ({padded - real:,} pad tokens)")
|
|
print()
|
|
print(f"mb width min/mean/max {min(mb_widths)} / {sum(mb_widths)/nb:.0f} / {max(mb_widths)}")
|
|
srt = sorted(mb_widths)
|
|
for q in (50, 75, 90, 95, 99):
|
|
print(f" p{q} width {srt[int(nb*q/100)]}")
|
|
print(f"mb at max_seq_len 16384 {sum(1 for w in mb_widths if w >= 16384):,} ({100*sum(1 for w in mb_widths if w>=16384)/nb:.1f}%)")
|
|
print()
|
|
srtw = sorted(mb_waste)
|
|
print(f"per-mb waste p50/p90/max {100*srtw[nb//2]:.1f}% / {100*srtw[int(nb*0.9)]:.1f}% / {100*max(mb_waste):.1f}%")
|
|
print()
|
|
print(f"loss targets per mb min/mean/max {min(mb_kept)} / {sum(mb_kept)/nb:.0f} / {max(mb_kept)}")
|
|
print(f" -> CE chunks per mb (1024) min/mean/max {min(mb_kept)//1024+1} / {sum(mb_kept)/nb/1024:.1f} / {max(mb_kept)//1024+1}")
|
|
print()
|
|
|
|
# --- what length-bucketing would recover (sort by length, then batch) ---
|
|
order = sorted(range(n), key=lambda i: lens[i])
|
|
b_real = b_padded = 0
|
|
for i in range(0, n - n % MB, MB):
|
|
group = [lens[j] for j in order[i:i + MB]]
|
|
b_real += sum(group)
|
|
b_padded += max(group) * MB
|
|
print("--- counterfactual: length-bucketed sampler ---")
|
|
print(f"bucketed padded tokens {b_padded:,}")
|
|
print(f"bucketed waste {100 * (1 - b_real / b_padded):.1f}%")
|
|
print(f"TOKEN REDUCTION vs current {100 * (1 - b_padded / padded):.1f}%")
|