Unwrap the Brontë corpus and launch the 1.7B rung
Operator: "start the 1.7b training." The 0.6B adapter learned the Gutenberg transcription's ~70-character line breaks along with the prose -- its output wrapped at a mid-length-line ratio of 0.85 against the base model's 0.00. That is typography rather than style, and every further rung would have inherited it, so the corpus is reflowed before rung 2 rather than after the sweep. The reflow joins 57,430 of 85,380 paragraph blocks and keeps 27,950. Verse is the hazard a blind join would destroy, so the decision is per block by median line length: blocks whose lines cluster near the wrap width are flowed prose, blocks of consistently short lines keep their breaks. Every kept multi-line block in the sample was genuinely verse with its lineation intact. No line ended in a lone hyphen, so the space-join could not split a word across lines. The acceptance check is content identity -- " ".join(text.split()) byte-identical before and after -- and it passed on all 852 records, proving only whitespace changed. Concrete cost of the old defect: 5.7% of the training budget was newline tokens. The same words pack to 5,210,112 tokens unwrapped against 5,525,504 wrapped. The 1.7B run is live at 159 steps and roughly 18.7 s/it. Everything but the carrier and the corpus is held from the 0.6B run: seed 4919, rank 32, lr 1e-4, seq 4096, batch 1 by accum 8, one epoch, eval and save every 25 steps so the minimum is located rather than assumed. That corpus change is a second variable and it is named as one. A 0.6B-vs-1.7B comparison is descriptive, not attributable, until the chained 0.6B rerun on the same unwrapped corpus lands behind it -- gated on the 1.7B actually producing an adapter, because a chain that fires on failure turns one lost run into two. "Did sense come back at 1.7B" is a within-arm reading and survives the confound; any between-rung delta does not. The original wrapped corpus is untouched, so the 0.6B run's pinned corpus sha 3959036cf851bf62 stays reproducible.
This commit is contained in:
@@ -206,6 +206,8 @@ _As of 2026-09-10 10:25 PT._
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-09-10]` **R49 rung 2 LAUNCHED: Qwen3-1.7B-Base, 1 epoch, seed 4919, on an UNWRAPPED corpus.** Operator: *"start the 1.7b training."* Live at `gx10:~/r49-runs/h02-1p7b-1ep/`, 159 steps at ~18.7 s/it (~50 min), corpus sha **`77f37057b2782e49`**. A 0.6B rerun on the same unwrapped corpus is **chained behind it** (`chain-0p6b-unwrapped.sh`, gated on the 1.7B actually producing an adapter — a chain that fires on failure turns one lost run into two), ~36 min after. ⚠⚠ **THE CORPUS CHANGED, SO 0.6B-vs-1.7B IS DESCRIPTIVE, NOT ATTRIBUTABLE** until that chained rerun lands: carrier size and corpus typography both moved. *"Did sense come back at 1.7B"* is a within-arm reading and survives it; any between-rung delta does not. **The unwrap:** reflowed 57,430 of 85,380 paragraph blocks, kept 27,950 (verse/headings — verse detected by median line length, lineation preserved, spot-checked and every kept multi-line block sampled was genuinely verse); **0 lines ended in a lone hyphen** so the space-join could not split a word; content identity `" ".join(text.split())` verified byte-identical on all **852 records**, i.e. whitespace-only. Mid-length-line ratio **0.94 → 0.25** (the residual is the preserved verse). ⚠ Concrete cost of the old defect: **5.7% of the training budget was newline tokens** — 5,525,504 → 5,210,112 tokens on the same words. Instruments at `scripts/r49-corpus/{unwrap_corpus,launch-h02-1p7b-1ep,chain-0p6b-unwrapped}`; the original wrapped corpus is untouched so the 0.6B run's pinned sha `3959036cf851bf62` stays reproducible.
|
||||
|
||||
- `[2026-09-10]` **BabyBronte H02 adapter: the VOICE transferred, the SENSE did not — operator's read, "it's all nonsense, but it sounds like Brontë's nonsense."** Eyeball A/B (NOT the adjudication; nothing here feeds the frozen rule), 9 arbitrary prompts on a deliberate difficulty gradient × 2 arms × 2 seeds, booth at `http://10.100.10.50:8090/b/babybronte-voice/`. Measured across the 18 pairs: **curly quotes 1/18 base → 18/18 tuned**, **math/worksheet collapse 3/18 base → 0/18 tuned**. Given *"The self-checkout machine refused her coupon"* the base 0.6B produced a **quadratic-formula worksheet**; the tuned arm wrote a clerk refusing a customer in Victorian retrospective first person. This is the expected and informative result for the smallest rung — **voice is separable from coherence at 0.6B**, which is the premise the whole lightweight-adapter regime rests on, and the 1.7B/4B rungs are where sense should return. The 1-epoch loss was still descending at step 169 (undertrained, not overfit), so the incoherence is carrier capacity, not training. ⚠ **Corpus-prep defect found: the tuned output is hard-wrapped at ~70 chars** (median mid-length-line ratio 0.85 vs base 0.00) — the Gutenberg source kept its original line breaks and the adapter learned the typography along with the voice. Unwrap to flowing paragraphs before any real use or the next rung learns it too.
|
||||
|
||||
- `[2026-09-10]` **GPU0 SETTLED after THREE crashes and two failed attempts: mog-sec now runs `MOG_GPU_MEM_UTIL=0.50` + **`MOG_KV_CACHE_MEMORY=17697765376` (KV PINNED IN BYTES)** + `MOG_MAX_MODEL_LEN=327680` (was 0.52 / ratio-inferred KV / 420000).** ⚠ **I got the severity call wrong first**: off a `RestartCount=1` I said "rare, not chronic, do nothing"; the operator pushed back and it crashed twice more (20:20:30Z, 20:30:02Z at the old settings, then 21:46:35Z after the first fix). ⚠ **Dropping utilization alone is NOT the fix and fails in a worse way** — at 0.50 with 420k the seat crash-loops at STARTUP, because one 420k sequence needs 17.88 GiB of KV and the 0.50 pool is 17.4–17.5. ⚠⚠ **THE REAL FINDING: `--gpu-memory-utilization` does NOT bound actual usage.** It sizes the KV calculation, but peak activation is measured at profiling time and real long-context work exceeds the profile. vLLM's own budget line proved mog-sec was running **0.9 GiB OVER** its 47.48 GiB reservation (26.44 consumed + 3.53 peak act + 0.89 CUDAGraph + 17.52 KV = 48.38), and **`gen` was over by 0.33 on the same card** — that overage came out of the shared slack and is what kept OOMing. Fix is the one vLLM printed itself: pin KV in bytes at its recommended 16.48 GiB, same discipline `erp-seat` already uses. **The KV pin and the context length are COUPLED** — 16.48 GiB yields ~383,730 tokens, so 393,216 falls under the 1.0x floor and pinning KV while keeping 384k was never available. Verified after: `reserved 16.48 GiB ... as specified by kv_cache_memory_bytes config and skipped memory profiling`, KV 375,901 tokens, **concurrency 1.15x** (was 1.03x), GPU0 90,561 MiB (~750 MiB reclaimed), both aliases 200 through the gateway. Backups `.env.bak-util052-20260910` and `.env.bak-384k-20260910`; full reasoning in `stacks/mog-sec/compose.yaml`'s header. Cost: 320k context instead of 420k, on a seat whose crashes happened at ~151k.
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
"""Render the base-vs-tuned voice A/B into a booth page.
|
||||
|
||||
Layout is the argument. A flat gallery would let you read one arm at a time, which
|
||||
is exactly how you talk yourself into seeing a difference. So every prompt is one
|
||||
row, the arms are side by side, and BOTH seeds of each arm sit in the same cell --
|
||||
so the within-arm variation is visible in the same glance as the between-arm
|
||||
variation. If the two base samples differ from each other as much as base differs
|
||||
from tuned, there is nothing here, and the layout should make that obvious rather
|
||||
than hide it.
|
||||
|
||||
Prompts are ordered by tier, hardest first: modern/mundane, then period-neutral,
|
||||
then Victorian-adjacent. The modern tier is the one that matters -- Brontë showing
|
||||
up there is the adapter's doing, whereas Brontë showing up in the period tier could
|
||||
just be the prompt.
|
||||
"""
|
||||
import html
|
||||
import json
|
||||
import sys
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
base_f, tuned_f, out_dir = sys.argv[1], sys.argv[2], Path(sys.argv[3])
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def load(p):
|
||||
d = defaultdict(dict)
|
||||
for line in Path(p).read_text(encoding="utf-8").splitlines():
|
||||
if not line.strip():
|
||||
continue
|
||||
r = json.loads(line)
|
||||
d[r["id"]][r["seed"]] = r
|
||||
return d
|
||||
|
||||
base, tuned = load(base_f), load(tuned_f)
|
||||
ids = [i for i in base if i in tuned]
|
||||
TIER = {"modern": ("Tier A — modern / mundane",
|
||||
"Nothing here invites Victorian prose. Brontë in this tier is the adapter's doing."),
|
||||
"neutral": ("Tier B — period-neutral",
|
||||
"Could be any century. A voice shift shows cleanly without the prompt supplying it."),
|
||||
"period": ("Tier C — Victorian-adjacent, plainly worded",
|
||||
"The setting leans period but the diction does not. Easiest tier; weakest evidence.")}
|
||||
order = {"modern": 0, "neutral": 1, "period": 2}
|
||||
ids.sort(key=lambda i: (order.get(base[i][list(base[i])[0]]["tier"], 9), i))
|
||||
|
||||
def cell(rec_by_seed):
|
||||
parts = []
|
||||
for seed in sorted(rec_by_seed):
|
||||
t = (rec_by_seed[seed]["continuation"] or "").strip()
|
||||
parts.append(f'<div class="s"><span class="seed">seed {seed}</span>'
|
||||
f'<p>{html.escape(t) or "<em>(empty)</em>"}</p></div>')
|
||||
return "".join(parts)
|
||||
|
||||
rows, seen = [], set()
|
||||
for i in ids:
|
||||
any_rec = base[i][list(base[i])[0]]
|
||||
tier = any_rec["tier"]
|
||||
if tier not in seen:
|
||||
seen.add(tier)
|
||||
title, sub = TIER.get(tier, (tier, ""))
|
||||
rows.append(f'<h2>{html.escape(title)}</h2><p class="tsub">{html.escape(sub)}</p>')
|
||||
rows.append(f"""
|
||||
<section class="row">
|
||||
<div class="prompt"><span class="pid">{html.escape(i)}</span>{html.escape(any_rec["prompt"])}</div>
|
||||
<div class="arms">
|
||||
<div class="arm"><h3>Base <small>Qwen3-0.6B-Base, no adapter</small></h3>{cell(base[i])}</div>
|
||||
<div class="arm tuned"><h3>Tuned <small>+ H02 LoRA, 1 epoch, seed 4919</small></h3>{cell(tuned[i])}</div>
|
||||
</div>
|
||||
</section>""")
|
||||
|
||||
page = f"""<!doctype html><meta charset="utf-8"><title>BabyBronte — voice A/B</title>
|
||||
<style>
|
||||
:root{{--bg:#faf8f5;--fg:#1c1a17;--mut:#6b6560;--line:#e0dad2;--acc:#7a3b2e;--tint:#fdfbf7}}
|
||||
*{{box-sizing:border-box}}
|
||||
body{{margin:0;background:var(--bg);color:var(--fg);font:16px/1.6 Georgia,"Iowan Old Style",serif;padding:2.5rem 1.5rem 5rem}}
|
||||
.wrap{{max-width:1180px;margin:0 auto}}
|
||||
h1{{font-size:1.9rem;margin:0 0 .3rem}}
|
||||
.lede{{color:var(--mut);max-width:70ch;margin:0 0 .9rem}}
|
||||
.warn{{border-left:3px solid var(--acc);background:#fff;padding:.8rem 1rem;margin:1.2rem 0 2rem;max-width:80ch;font-size:.93rem}}
|
||||
h2{{font-size:1.15rem;margin:2.8rem 0 .2rem;padding-top:1rem;border-top:1px solid var(--line)}}
|
||||
.tsub{{color:var(--mut);font-size:.9rem;margin:0 0 1.2rem;font-style:italic}}
|
||||
.row{{margin:0 0 2.2rem}}
|
||||
.prompt{{background:#fff;border:1px solid var(--line);border-left:3px solid var(--acc);padding:.7rem .9rem;font-size:1.02rem;margin-bottom:.7rem}}
|
||||
.pid{{display:inline-block;font:600 .72rem/1 ui-monospace,monospace;color:var(--mut);background:var(--bg);border:1px solid var(--line);padding:.22rem .4rem;margin-right:.6rem;vertical-align:1px}}
|
||||
.arms{{display:grid;grid-template-columns:1fr 1fr;gap:1rem}}
|
||||
@media(max-width:820px){{.arms{{grid-template-columns:1fr}}}}
|
||||
.arm{{background:#fff;border:1px solid var(--line);padding:.9rem 1rem}}
|
||||
.arm.tuned{{background:var(--tint);border-color:#d8ccbe}}
|
||||
.arm h3{{margin:0 0 .6rem;font-size:.9rem;letter-spacing:.04em;text-transform:uppercase;color:var(--acc)}}
|
||||
.arm h3 small{{display:block;text-transform:none;letter-spacing:0;color:var(--mut);font-weight:400;font-size:.82rem;margin-top:.15rem}}
|
||||
.s{{border-top:1px dotted var(--line);padding-top:.6rem;margin-top:.6rem}}
|
||||
.arm .s:first-of-type{{border-top:0;padding-top:0;margin-top:0}}
|
||||
.seed{{display:block;font:600 .7rem/1 ui-monospace,monospace;color:var(--mut);margin-bottom:.25rem}}
|
||||
.s p{{margin:0;white-space:pre-wrap;font-size:.95rem}}
|
||||
footer{{margin-top:3rem;padding-top:1rem;border-top:1px solid var(--line);color:var(--mut);font-size:.85rem;max-width:80ch}}
|
||||
</style>
|
||||
<div class="wrap">
|
||||
<h1>BabyBronte — did the voice move?</h1>
|
||||
<p class="lede">Same prompts, same sampler, same box, same seeds. The only difference between the
|
||||
columns is the H02 LoRA adapter (Charlotte Brontë, 680k words, 1 epoch, seed 4919).</p>
|
||||
|
||||
<div class="warn"><strong>Read this as an eyeball test, not a result.</strong> Two samples per arm
|
||||
per prompt is enough to see whether the gap between the columns is bigger than the gap between the
|
||||
two seeds <em>inside</em> a column — and not enough for anything else. No scoring, no statistics.
|
||||
The frozen adjudication rule and the Burrows's-Delta instrument are untouched by this page and
|
||||
nothing here feeds them.<br><br>
|
||||
Both arms are <strong>Qwen3-0.6B-Base doing continuation</strong>, not instruction-following. The
|
||||
adapter was trained as pure continuation, so each prompt is an opening line the model carries on
|
||||
from — asking a base model to "rewrite this in Brontë's voice" would test instruction-following
|
||||
instead of voice.</div>
|
||||
|
||||
{''.join(rows)}
|
||||
|
||||
<footer>Generated on pfi-gx10 (GB10), bf16, sdpa. Sampler pinned identical across arms:
|
||||
temperature 0.9, top_p 0.95, 400 new tokens, seeds 1234 and 5678. Adapter binding proven at
|
||||
generation time (196/196 lora_B tensors non-zero) — a silent no-op looks exactly like a tune that
|
||||
changed nothing.</footer>
|
||||
</div>"""
|
||||
|
||||
(out_dir / "index.html").write_text(page, encoding="utf-8")
|
||||
print(f"wrote {out_dir/'index.html'} ({len(ids)} prompts x 2 arms x 2 seeds)")
|
||||
Executable
+48
@@ -0,0 +1,48 @@
|
||||
#!/usr/bin/env bash
|
||||
# Chained after the 1.7B rung: re-run the 0.6B carrier on the SAME unwrapped corpus.
|
||||
#
|
||||
# The 1.7B run moved two variables at once -- carrier size AND corpus typography --
|
||||
# because the unwrap could not wait if every further rung was to avoid inheriting the
|
||||
# Gutenberg line breaks. That makes a 0.6B-vs-1.7B comparison descriptive rather than
|
||||
# attributable, which is fine for "did sense come back" (a within-arm reading) and not
|
||||
# fine for anything quantitative between rungs.
|
||||
#
|
||||
# This closes that hole for the price of ~36 minutes on an idle experimental box:
|
||||
# 0.6B on the unwrapped corpus, seed 4919, everything else held. Then carrier size is
|
||||
# the ONLY difference between this and h02-1p7b-1ep, and the sweep is single-variable
|
||||
# again.
|
||||
#
|
||||
# ⚠ Gated on the 1.7B run having actually produced an adapter. If that run died, this
|
||||
# must not quietly start and consume the box; a chain that fires on failure turns one
|
||||
# lost run into two.
|
||||
set -uo pipefail
|
||||
PREV=/home/infra-ops/r49-runs/h02-1p7b-1ep
|
||||
OUT=/home/infra-ops/r49-runs/h02-0p6b-1ep-unwrapped
|
||||
LOG=$OUT/train.log
|
||||
CHAINLOG=/home/infra-ops/r49-runs/chain-0p6b-unwrapped.log
|
||||
|
||||
exec >> "$CHAINLOG" 2>&1
|
||||
echo "=== $(date -Is) chain armed, waiting on $PREV"
|
||||
while [ -f "$PREV/run.pid" ] && kill -0 "$(cat "$PREV/run.pid")" 2>/dev/null; do sleep 60; done
|
||||
echo "=== $(date -Is) 1.7B run finished"
|
||||
|
||||
if [ ! -f "$PREV/adapter/adapter_model.safetensors" ]; then
|
||||
echo "=== REFUSING to chain: $PREV produced no adapter -- the 1.7B run did not succeed"
|
||||
exit 1
|
||||
fi
|
||||
apps=$(nvidia-smi --query-compute-apps=pid --format=csv,noheader | tr -d '[:space:]')
|
||||
if [ -n "$apps" ]; then
|
||||
echo "=== REFUSING to chain: GPU not clear"
|
||||
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mkdir -p "$OUT"
|
||||
echo "# launched $(date -Is) Qwen3-0.6B-Base, 1 epoch, seed 4919, UNWRAPPED corpus (single-variable partner to h02-1p7b-1ep)" > "$LOG"
|
||||
setsid nohup /home/infra-ops/ml/.venv/bin/python /home/infra-ops/r49-prep/train_voice_lora.py \
|
||||
--corpus /home/infra-ops/r49-corpus-renamed-unwrapped \
|
||||
--base /home/infra-ops/carriers/Qwen3-0.6B-Base \
|
||||
--seed 4919 --epochs 1 --eval-steps 25 --save-steps 25 \
|
||||
--out "$OUT" >> "$LOG" 2>&1 < /dev/null &
|
||||
echo $! > "$OUT/run.pid"
|
||||
echo "=== $(date -Is) chained 0.6B launched pid $(cat "$OUT/run.pid") -> $LOG"
|
||||
@@ -0,0 +1,82 @@
|
||||
"""Eyeball A/B: does the H02 adapter pull arbitrary prose toward Charlotte Brontë?
|
||||
|
||||
NOT the adjudication. The frozen rule, the Burrows's-Delta instrument and the
|
||||
held-out chapters are untouched by this; nothing here feeds them. This exists
|
||||
because the operator asked to *see* whether the voice moved.
|
||||
|
||||
Two design choices that decide whether the test says anything:
|
||||
|
||||
* The prompts are deliberately NOT Brontë-ish. Feed a base model "the moors lay
|
||||
dark under a bruised sky" and both arms come back Victorian, because the prompt
|
||||
did the work. So the set runs a difficulty gradient -- modern/mundane, then
|
||||
period-neutral, then Victorian-adjacent-but-plainly-worded -- and any Brontë in
|
||||
the modern tier is attributable to the adapter rather than to the setup.
|
||||
* Two seeds per prompt per arm, which is nearly free on a 0.6B and is the only
|
||||
thing that makes the comparison readable. One sample per arm cannot tell "the
|
||||
adapter changed the voice" from "sampling is noisy"; a reader with two samples
|
||||
of each arm can at least see whether the between-arm gap exceeds the
|
||||
within-arm gap. That is an eyeball noise floor, not a measurement, and it is
|
||||
not offered as one.
|
||||
|
||||
Same harness for both arms -- same box, same sampler, same prompts, same lengths --
|
||||
because a cross-comparison whose harness differs is invalid rather than noisy.
|
||||
Sampler matches the pinned adjudication sampler (temp 0.9 / top_p 0.95 / 400 new
|
||||
tokens) so what is on screen is the same shape of output the real arms produced.
|
||||
|
||||
⚠ This is a BASE model doing CONTINUATION, and the adapter was trained as pure
|
||||
continuation (H02 has no beat annotation by design). It will not follow a "rewrite
|
||||
this in Brontë's voice" instruction, and asking it to would test instruction-
|
||||
following rather than voice. So each prompt is an opening line the model continues.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import argparse, json, time
|
||||
from pathlib import Path
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--base", required=True)
|
||||
ap.add_argument("--adapter", default=None)
|
||||
ap.add_argument("--arm", required=True)
|
||||
ap.add_argument("--prompts", required=True)
|
||||
ap.add_argument("--out", required=True)
|
||||
ap.add_argument("--seeds", type=int, nargs="+", default=[1234, 5678])
|
||||
ap.add_argument("--max-new-tokens", type=int, default=400)
|
||||
ap.add_argument("--temperature", type=float, default=0.9)
|
||||
ap.add_argument("--top-p", type=float, default=0.95)
|
||||
a = ap.parse_args()
|
||||
|
||||
prompts = json.loads(Path(a.prompts).read_text())
|
||||
tok = AutoTokenizer.from_pretrained(a.base)
|
||||
model = AutoModelForCausalLM.from_pretrained(a.base, dtype=torch.bfloat16,
|
||||
attn_implementation="sdpa").to("cuda")
|
||||
if a.adapter:
|
||||
from peft import PeftModel
|
||||
model = PeftModel.from_pretrained(model, a.adapter)
|
||||
# ⚠ Prove the adapter actually BOUND. A silent no-op looks exactly like a tune
|
||||
# that changed nothing -- which is the very thing this test is trying to see.
|
||||
deltas = [float(m.lora_B["default"].weight.abs().sum())
|
||||
for m in model.modules() if hasattr(m, "lora_B")]
|
||||
nonzero = sum(1 for d in deltas if d > 0)
|
||||
print(f"[gen] adapter bound: {nonzero}/{len(deltas)} lora_B tensors non-zero", flush=True)
|
||||
if nonzero == 0:
|
||||
raise SystemExit("REFUSING: adapter applied but every lora_B is zero -- it did not bind")
|
||||
model.eval()
|
||||
|
||||
out = Path(a.out); out.parent.mkdir(parents=True, exist_ok=True)
|
||||
t0 = time.time()
|
||||
with out.open("w", encoding="utf-8") as fh:
|
||||
for p in prompts:
|
||||
for seed in a.seeds:
|
||||
torch.manual_seed(seed) # per-sample, so seed N is comparable across arms
|
||||
ids = tok(p["prompt"], return_tensors="pt").to("cuda")
|
||||
with torch.no_grad():
|
||||
g = model.generate(**ids, do_sample=True, temperature=a.temperature,
|
||||
top_p=a.top_p, max_new_tokens=a.max_new_tokens,
|
||||
pad_token_id=tok.eos_token_id)
|
||||
cont = tok.decode(g[0][ids["input_ids"].shape[1]:], skip_special_tokens=True)
|
||||
fh.write(json.dumps({"arm": a.arm, "id": p["id"], "tier": p["tier"],
|
||||
"prompt": p["prompt"], "seed": seed,
|
||||
"continuation": cont}) + "\n")
|
||||
print(f" {a.arm} {p['id']} seed={seed} {len(cont.split())}w", flush=True)
|
||||
print(f"[gen] {a.arm} -> {out} in {time.time()-t0:.0f}s", flush=True)
|
||||
Executable
+44
@@ -0,0 +1,44 @@
|
||||
#!/usr/bin/env bash
|
||||
# R49 H02 — rung 2 of the carrier sweep: Qwen3-1.7B-Base, 1 epoch.
|
||||
#
|
||||
# WHY THIS RUN. The 0.6B rung answered the narrow question and the operator's read
|
||||
# was the finding: "it's all nonsense, but it sounds like Brontë's nonsense."
|
||||
# Voice transferred (curly quotes 1/18 -> 18/18, worksheet collapse 3/18 -> 0/18)
|
||||
# while coherence did not. That separation is the premise the whole lightweight
|
||||
# author-voice regime rests on, so the live question is which carrier size brings
|
||||
# sense back while the voice stays. 1.7B is the next rung; 4B is after it.
|
||||
#
|
||||
# ⚠ THE CORPUS CHANGED, DELIBERATELY, AND IT IS A SECOND VARIABLE.
|
||||
# The 0.6B adapter learned the Gutenberg transcription's ~70-char line breaks along
|
||||
# with the prose -- its output wrapped at a mid-length-line ratio of 0.85 against
|
||||
# the base model's 0.00. That is typography, not style, and every further rung
|
||||
# would inherit it. So this trains on r49-corpus-renamed-unwrapped: same words,
|
||||
# reflowed into paragraphs, verified whitespace-only on all 852 records, with verse
|
||||
# blocks detected by median line length and their lineation preserved (0 lines
|
||||
# ended in a lone hyphen, so the space-join could not split a word).
|
||||
# CONSEQUENCE: a 0.6B-vs-1.7B comparison is now DESCRIPTIVE, not attributable --
|
||||
# carrier size and corpus typography both moved. "Did sense come back at 1.7B" is a
|
||||
# within-arm reading and survives that; any between-rung delta does not. The 0.6B
|
||||
# rerun on this same corpus is chained after this run so the clean single-variable
|
||||
# comparison exists too.
|
||||
#
|
||||
# Everything else is held from the 0.6B run: seed 4919, rank 32, lr 1e-4, seq 4096,
|
||||
# batch 1 x accum 8, 1 epoch, eval+save every 25 steps so the minimum is LOCATED
|
||||
# rather than assumed (the 3-epoch run overfit with save_strategy="no" and left
|
||||
# nothing to fall back to).
|
||||
set -euo pipefail
|
||||
OUT=/home/infra-ops/r49-runs/h02-1p7b-1ep
|
||||
LOG=$OUT/train.log
|
||||
apps=$(nvidia-smi --query-compute-apps=pid --format=csv,noheader | tr -d '[:space:]')
|
||||
[ -n "$apps" ] && { echo "REFUSING: GPU not clear" >&2; nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv >&2; exit 1; }
|
||||
[ -f "$OUT/run.pid" ] && kill -0 "$(cat "$OUT/run.pid")" 2>/dev/null && { echo "REFUSING: live pid" >&2; exit 1; }
|
||||
[ -e "$LOG" ] && { echo "REFUSING: $LOG exists" >&2; exit 1; }
|
||||
mkdir -p "$OUT"
|
||||
echo "# launched $(date -Is) Qwen3-1.7B-Base, 1 epoch, seed 4919, UNWRAPPED corpus, eval+save every 25" > "$LOG"
|
||||
setsid nohup /home/infra-ops/ml/.venv/bin/python /home/infra-ops/r49-prep/train_voice_lora.py \
|
||||
--corpus /home/infra-ops/r49-corpus-renamed-unwrapped \
|
||||
--base /home/infra-ops/carriers/Qwen3-1.7B-Base \
|
||||
--seed 4919 --epochs 1 --eval-steps 25 --save-steps 25 \
|
||||
--out "$OUT" >> "$LOG" 2>&1 < /dev/null &
|
||||
echo $! > "$OUT/run.pid"
|
||||
echo "launched pid $(cat "$OUT/run.pid") -> $LOG"
|
||||
@@ -0,0 +1,99 @@
|
||||
"""Reflow the hard-wrapped renamed corpus into flowing paragraphs.
|
||||
|
||||
The 0.6B adapter learned the Gutenberg transcription's ~70-character line breaks
|
||||
along with Charlotte Brontë's voice: its output wraps at a median mid-length-line
|
||||
ratio of 0.85 against the base model's 0.00. That is typography, not style, and
|
||||
every further rung would inherit it.
|
||||
|
||||
⚠ VERSE IS THE HAZARD. These novels contain poems, and a blind join would smear
|
||||
them into prose. So the join is decided per paragraph block by MEDIAN LINE LENGTH:
|
||||
a block whose lines sit near the wrap width is flowed prose and gets joined; a
|
||||
block of consistently short lines is verse (or a heading, or an address) and keeps
|
||||
its breaks. Both counts are reported, because a rule that silently reclassified
|
||||
half the poetry would look exactly like a rule that worked.
|
||||
|
||||
The acceptance check is content identity: `" ".join(text.split())` must be
|
||||
byte-identical before and after, for every record. That proves ONLY whitespace
|
||||
changed -- no word gained, lost, or altered. A reflow that quietly dropped a line
|
||||
would otherwise be invisible.
|
||||
|
||||
Writes to a NEW directory. The original stays exactly as the 0.6B run's provenance
|
||||
pins it (corpus_sha256_16 3959036cf851bf62), so that run remains reproducible.
|
||||
"""
|
||||
import json
|
||||
import statistics
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
SRC = Path(sys.argv[1])
|
||||
DST = Path(sys.argv[2])
|
||||
PROSE_MEDIAN = 55 # a wrapped-prose block's lines cluster near the wrap width
|
||||
|
||||
stats = {"records": 0, "blocks": 0, "joined": 0, "kept": 0, "hyphen_ends": 0}
|
||||
|
||||
|
||||
def reflow(text: str) -> str:
|
||||
out_blocks = []
|
||||
for block in text.split("\n\n"):
|
||||
lines = block.split("\n")
|
||||
body = [l for l in lines if l.strip()]
|
||||
if not body:
|
||||
out_blocks.append(block)
|
||||
continue
|
||||
stats["blocks"] += 1
|
||||
if len(body) == 1:
|
||||
out_blocks.append(block)
|
||||
stats["kept"] += 1
|
||||
continue
|
||||
med = statistics.median(len(l.rstrip()) for l in body[:-1] or body)
|
||||
if med >= PROSE_MEDIAN:
|
||||
stats["joined"] += 1
|
||||
for l in body[:-1]:
|
||||
if l.rstrip().endswith("-") and not l.rstrip().endswith("--"):
|
||||
stats["hyphen_ends"] += 1
|
||||
out_blocks.append(" ".join(l.strip() for l in body))
|
||||
else:
|
||||
stats["kept"] += 1
|
||||
out_blocks.append(block)
|
||||
return "\n\n".join(out_blocks)
|
||||
|
||||
|
||||
DST.mkdir(parents=True, exist_ok=True)
|
||||
(DST / "copies").mkdir(exist_ok=True)
|
||||
for f in sorted((SRC / "copies").glob("*.jsonl")):
|
||||
rows_out = []
|
||||
for line in f.read_text(encoding="utf-8").splitlines():
|
||||
if not line.strip():
|
||||
continue
|
||||
r = json.loads(line)
|
||||
before = r["text"]
|
||||
after = reflow(before)
|
||||
# ⚠ the acceptance check: content identical, whitespace only
|
||||
assert " ".join(before.split()) == " ".join(after.split()), \
|
||||
f"CONTENT CHANGED in {f.name} {r.get('work')}/{r.get('chapter')}"
|
||||
r["text"] = after
|
||||
rows_out.append(json.dumps(r, ensure_ascii=False))
|
||||
stats["records"] += 1
|
||||
(DST / "copies" / f.name).write_text("\n".join(rows_out) + "\n", encoding="utf-8")
|
||||
|
||||
for extra in ("rename_stats.json", "manifest.json"):
|
||||
if (SRC / extra).exists():
|
||||
(DST / extra).write_text((SRC / extra).read_text(encoding="utf-8"), encoding="utf-8")
|
||||
|
||||
# What the wrap ratio actually became -- the number the defect was measured with.
|
||||
def wrap_ratio(root):
|
||||
rs = []
|
||||
for f in sorted((root / "copies").glob("*.jsonl")):
|
||||
for line in f.read_text(encoding="utf-8").splitlines():
|
||||
if not line.strip():
|
||||
continue
|
||||
lines = [l for l in json.loads(line)["text"].split("\n") if l.strip()]
|
||||
rs.append(sum(1 for l in lines if 20 < len(l) < 78) / max(1, len(lines)))
|
||||
return statistics.median(rs)
|
||||
|
||||
print(f" records {stats['records']} blocks {stats['blocks']} "
|
||||
f"joined {stats['joined']} kept-as-is {stats['kept']} (verse/headings/single-line)")
|
||||
print(f" lines ending in a lone hyphen inside joined blocks: {stats['hyphen_ends']} "
|
||||
f"(a nonzero count means words were split across lines and a space-join would break them)")
|
||||
print(f" mid-length-line ratio before {wrap_ratio(SRC):.2f} -> after {wrap_ratio(DST):.2f}")
|
||||
print(f" content identity: PASSED on all {stats['records']} records (whitespace-only change)")
|
||||
@@ -0,0 +1,11 @@
|
||||
[
|
||||
{"id":"a1","tier":"modern","prompt":"The self-checkout machine refused her coupon for the third time."},
|
||||
{"id":"a2","tier":"modern","prompt":"He parked the car and sat listening to the engine tick as it cooled."},
|
||||
{"id":"a3","tier":"modern","prompt":"The office kitchen smelled of burnt coffee and somebody's reheated fish."},
|
||||
{"id":"b1","tier":"neutral","prompt":"She had not slept, and the morning found her at the window."},
|
||||
{"id":"b2","tier":"neutral","prompt":"There was a letter on the table, and no one would say who had brought it."},
|
||||
{"id":"b3","tier":"neutral","prompt":"The child would not speak, though she had been asked three times."},
|
||||
{"id":"c1","tier":"period","prompt":"The new governess arrived on a wet Tuesday and nobody came to meet her."},
|
||||
{"id":"c2","tier":"period","prompt":"Rain came on hard after dark, and the road up to the house turned to mud."},
|
||||
{"id":"c3","tier":"period","prompt":"He was the sort of man who took up all the air in a small room."}
|
||||
]
|
||||
Reference in New Issue
Block a user