memory: concurrent training on pfi-gx10 is net slower than serial
VRAM said yes and the box said no. Two 4B LoRA jobs fit easily -- 36 of 121 GiB with 98 free -- but both dropped from a 37.10 s/it solo baseline to ~85 s/it, 2.29x each, so combined throughput fell below serial. The GB10 is past its memory-bandwidth roofline and pays a contention penalty on top of the split. Worth keeping because the intuition it corrects is a common one: free GPU memory is the number everyone reaches for, and on this box it does not predict whether a second job is affordable. The control is what makes it a finding rather than an anecdote -- killing the second job returned the first to 37 s/it on the next step.
This commit is contained in:
@@ -167,6 +167,7 @@ _As of 2026-09-11 09:00 PT._
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-09-11]` ⭐ **MEASURED: two concurrent training jobs on pfi-gx10 are 13% NET SLOWER than running them back to back — VRAM is not the constraint and never was.** Operator asked to run the two BabyYarros arms in parallel if VRAM allowed. It does, comfortably: **18.4 GiB per 4B LoRA job, 36 of 121 GiB with both up, 98 GiB free.** But the GB10 is a *capacity* box, not a throughput box, and the binding constraint is memory bandwidth. Solo baseline **37.10 s/it (n=6, 0.05% spread)**; with a second job both arms settled at **~85 s/it — 2.29x each**, so combined throughput 0.0235 vs 0.0270 steps/s solo. Not a clean 2x split: the box is *past* its roofline and pays a contention penalty on top. ⭐ **Control: killing the second job returned the first to 37 s/it on the very next step**, so the slowdown tracked contention and reversed with it. Chaining finished both arms ~43 min earlier than concurrency would have. **General form: on this box, `nvidia-smi` free memory tells you nothing about whether a second job is affordable.** Decision rule was pre-registered before the numbers were read (<55 s/it keep both, ≥2x chain). `scripts/yarros-corpus/{launch-yarros-4b-base,chain-yarros-4b-base}.sh`; the shared-GPU bypass is an explicit argument, never a default.
|
||||
- `[2026-09-11]` ⭐⭐ **BabyYarros UNBLOCKED and TRAINING: the leak gate passes at 0 of 325 entities and 0 of 91 phrases, and closing it turned up three defects nobody was looking for.** The gate itself is the first artifact — there was no committed instrument for "does any of the author's proper nouns survive", so Brontë's 0-of-203 was a hand count. `scripts/r49-corpus/leak_gate.py` now runs the same scan over the UNRENAMED source as a **positive control** plus a nonce **negative control** every time, because a detector that only ever sees renamed text cannot tell *absent* from *blind*. Its first reading was **212 surviving, not 86** — it scans the whole corpus rather than per work, and counts the sub-threshold entities rename never looked at. Training launched 10:06 PT on pfi-gx10: Qwen3-4B-**Instruct**, 1 epoch, seed 4919, 178 steps / 5,824,512 tokens, corpus sha `e85f69f1e49d57c9`. → `persistent-memory.d/2026-09-11-babyyarros-leak-gate-passes.md`
|
||||
- `[2026-09-11]` ⭐⭐ **A SECOND corpus typography defect, and the D1 note that "no unwrap was needed" was right about the wrong thing.** Kvasir's cleaner does emit flowing paragraphs, so Brontë's hard-wrap defect genuinely does not exist here. A *different* one does: the Empyrean books set chapter epigraphs in small caps and the extractor rendered the run as uppercase while leaving the large initial its own token — `— M AJOR A FENDRA'S G UIDE TO THE R IDERS Q UADRANT (U NAUTHORIZED E DITION)`, **106 lines / ~700 splits**, plus **52 drop caps** (`T he flight field`, `X aden.`, 51 of 52 in iron-flame). That is the entire source of the entities called `IDERS`, `UADRANT`, `NAUTHORIZED`, `DITION` and seventeen bare single letters. ⭐ **The restore is exact, not approximate**: a split initial next to an uppercased run recovers the original mixed case, because a word WITH a split initial was capitalised in the source and an all-caps word WITHOUT one was lowercase. ⚠ Guards that matter: only lines with **≥2** splits are treated as a run (one split is a sentence next to an acronym), and `I`/`A`/`O` are excluded from the drop-cap join or `A slow smile` becomes `Aslow`. `scripts/yarros-corpus/repair_typography.py`.
|
||||
- `[2026-09-11]` ⚠⚠ **Back matter was inside the prose of all five works — 4,555 words naming the author's agent, editors and children.** The builder splits on chapter headings and nothing follows the last one, so acknowledgments, newsletter pitches and cover-artist credits rode inside the final chapter. Found by the gate's phrase audit surfacing `Louise Fury` (Yarros's literary agent), not by reading. ⚠ **iron-flame's marker is `ACKNOWLEDGMENTS` in all caps** and a case-sensitive scan missed it — the strip is case-insensitive and last-chapter-only, with an acceptance check that refuses if it would remove more than 2% of the corpus.
|
||||
|
||||
Reference in New Issue
Block a user