Unwrap the Brontë corpus and launch the 1.7B rung
Operator: "start the 1.7b training." The 0.6B adapter learned the Gutenberg transcription's ~70-character line breaks along with the prose -- its output wrapped at a mid-length-line ratio of 0.85 against the base model's 0.00. That is typography rather than style, and every further rung would have inherited it, so the corpus is reflowed before rung 2 rather than after the sweep. The reflow joins 57,430 of 85,380 paragraph blocks and keeps 27,950. Verse is the hazard a blind join would destroy, so the decision is per block by median line length: blocks whose lines cluster near the wrap width are flowed prose, blocks of consistently short lines keep their breaks. Every kept multi-line block in the sample was genuinely verse with its lineation intact. No line ended in a lone hyphen, so the space-join could not split a word across lines. The acceptance check is content identity -- " ".join(text.split()) byte-identical before and after -- and it passed on all 852 records, proving only whitespace changed. Concrete cost of the old defect: 5.7% of the training budget was newline tokens. The same words pack to 5,210,112 tokens unwrapped against 5,525,504 wrapped. The 1.7B run is live at 159 steps and roughly 18.7 s/it. Everything but the carrier and the corpus is held from the 0.6B run: seed 4919, rank 32, lr 1e-4, seq 4096, batch 1 by accum 8, one epoch, eval and save every 25 steps so the minimum is located rather than assumed. That corpus change is a second variable and it is named as one. A 0.6B-vs-1.7B comparison is descriptive, not attributable, until the chained 0.6B rerun on the same unwrapped corpus lands behind it -- gated on the 1.7B actually producing an adapter, because a chain that fires on failure turns one lost run into two. "Did sense come back at 1.7B" is a within-arm reading and survives the confound; any between-rung delta does not. The original wrapped corpus is untouched, so the 0.6B run's pinned corpus sha 3959036cf851bf62 stays reproducible.
This commit is contained in:
@@ -0,0 +1,120 @@
|
||||
"""Render the base-vs-tuned voice A/B into a booth page.
|
||||
|
||||
Layout is the argument. A flat gallery would let you read one arm at a time, which
|
||||
is exactly how you talk yourself into seeing a difference. So every prompt is one
|
||||
row, the arms are side by side, and BOTH seeds of each arm sit in the same cell --
|
||||
so the within-arm variation is visible in the same glance as the between-arm
|
||||
variation. If the two base samples differ from each other as much as base differs
|
||||
from tuned, there is nothing here, and the layout should make that obvious rather
|
||||
than hide it.
|
||||
|
||||
Prompts are ordered by tier, hardest first: modern/mundane, then period-neutral,
|
||||
then Victorian-adjacent. The modern tier is the one that matters -- Brontë showing
|
||||
up there is the adapter's doing, whereas Brontë showing up in the period tier could
|
||||
just be the prompt.
|
||||
"""
|
||||
import html
|
||||
import json
|
||||
import sys
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
base_f, tuned_f, out_dir = sys.argv[1], sys.argv[2], Path(sys.argv[3])
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def load(p):
|
||||
d = defaultdict(dict)
|
||||
for line in Path(p).read_text(encoding="utf-8").splitlines():
|
||||
if not line.strip():
|
||||
continue
|
||||
r = json.loads(line)
|
||||
d[r["id"]][r["seed"]] = r
|
||||
return d
|
||||
|
||||
base, tuned = load(base_f), load(tuned_f)
|
||||
ids = [i for i in base if i in tuned]
|
||||
TIER = {"modern": ("Tier A — modern / mundane",
|
||||
"Nothing here invites Victorian prose. Brontë in this tier is the adapter's doing."),
|
||||
"neutral": ("Tier B — period-neutral",
|
||||
"Could be any century. A voice shift shows cleanly without the prompt supplying it."),
|
||||
"period": ("Tier C — Victorian-adjacent, plainly worded",
|
||||
"The setting leans period but the diction does not. Easiest tier; weakest evidence.")}
|
||||
order = {"modern": 0, "neutral": 1, "period": 2}
|
||||
ids.sort(key=lambda i: (order.get(base[i][list(base[i])[0]]["tier"], 9), i))
|
||||
|
||||
def cell(rec_by_seed):
|
||||
parts = []
|
||||
for seed in sorted(rec_by_seed):
|
||||
t = (rec_by_seed[seed]["continuation"] or "").strip()
|
||||
parts.append(f'<div class="s"><span class="seed">seed {seed}</span>'
|
||||
f'<p>{html.escape(t) or "<em>(empty)</em>"}</p></div>')
|
||||
return "".join(parts)
|
||||
|
||||
rows, seen = [], set()
|
||||
for i in ids:
|
||||
any_rec = base[i][list(base[i])[0]]
|
||||
tier = any_rec["tier"]
|
||||
if tier not in seen:
|
||||
seen.add(tier)
|
||||
title, sub = TIER.get(tier, (tier, ""))
|
||||
rows.append(f'<h2>{html.escape(title)}</h2><p class="tsub">{html.escape(sub)}</p>')
|
||||
rows.append(f"""
|
||||
<section class="row">
|
||||
<div class="prompt"><span class="pid">{html.escape(i)}</span>{html.escape(any_rec["prompt"])}</div>
|
||||
<div class="arms">
|
||||
<div class="arm"><h3>Base <small>Qwen3-0.6B-Base, no adapter</small></h3>{cell(base[i])}</div>
|
||||
<div class="arm tuned"><h3>Tuned <small>+ H02 LoRA, 1 epoch, seed 4919</small></h3>{cell(tuned[i])}</div>
|
||||
</div>
|
||||
</section>""")
|
||||
|
||||
page = f"""<!doctype html><meta charset="utf-8"><title>BabyBronte — voice A/B</title>
|
||||
<style>
|
||||
:root{{--bg:#faf8f5;--fg:#1c1a17;--mut:#6b6560;--line:#e0dad2;--acc:#7a3b2e;--tint:#fdfbf7}}
|
||||
*{{box-sizing:border-box}}
|
||||
body{{margin:0;background:var(--bg);color:var(--fg);font:16px/1.6 Georgia,"Iowan Old Style",serif;padding:2.5rem 1.5rem 5rem}}
|
||||
.wrap{{max-width:1180px;margin:0 auto}}
|
||||
h1{{font-size:1.9rem;margin:0 0 .3rem}}
|
||||
.lede{{color:var(--mut);max-width:70ch;margin:0 0 .9rem}}
|
||||
.warn{{border-left:3px solid var(--acc);background:#fff;padding:.8rem 1rem;margin:1.2rem 0 2rem;max-width:80ch;font-size:.93rem}}
|
||||
h2{{font-size:1.15rem;margin:2.8rem 0 .2rem;padding-top:1rem;border-top:1px solid var(--line)}}
|
||||
.tsub{{color:var(--mut);font-size:.9rem;margin:0 0 1.2rem;font-style:italic}}
|
||||
.row{{margin:0 0 2.2rem}}
|
||||
.prompt{{background:#fff;border:1px solid var(--line);border-left:3px solid var(--acc);padding:.7rem .9rem;font-size:1.02rem;margin-bottom:.7rem}}
|
||||
.pid{{display:inline-block;font:600 .72rem/1 ui-monospace,monospace;color:var(--mut);background:var(--bg);border:1px solid var(--line);padding:.22rem .4rem;margin-right:.6rem;vertical-align:1px}}
|
||||
.arms{{display:grid;grid-template-columns:1fr 1fr;gap:1rem}}
|
||||
@media(max-width:820px){{.arms{{grid-template-columns:1fr}}}}
|
||||
.arm{{background:#fff;border:1px solid var(--line);padding:.9rem 1rem}}
|
||||
.arm.tuned{{background:var(--tint);border-color:#d8ccbe}}
|
||||
.arm h3{{margin:0 0 .6rem;font-size:.9rem;letter-spacing:.04em;text-transform:uppercase;color:var(--acc)}}
|
||||
.arm h3 small{{display:block;text-transform:none;letter-spacing:0;color:var(--mut);font-weight:400;font-size:.82rem;margin-top:.15rem}}
|
||||
.s{{border-top:1px dotted var(--line);padding-top:.6rem;margin-top:.6rem}}
|
||||
.arm .s:first-of-type{{border-top:0;padding-top:0;margin-top:0}}
|
||||
.seed{{display:block;font:600 .7rem/1 ui-monospace,monospace;color:var(--mut);margin-bottom:.25rem}}
|
||||
.s p{{margin:0;white-space:pre-wrap;font-size:.95rem}}
|
||||
footer{{margin-top:3rem;padding-top:1rem;border-top:1px solid var(--line);color:var(--mut);font-size:.85rem;max-width:80ch}}
|
||||
</style>
|
||||
<div class="wrap">
|
||||
<h1>BabyBronte — did the voice move?</h1>
|
||||
<p class="lede">Same prompts, same sampler, same box, same seeds. The only difference between the
|
||||
columns is the H02 LoRA adapter (Charlotte Brontë, 680k words, 1 epoch, seed 4919).</p>
|
||||
|
||||
<div class="warn"><strong>Read this as an eyeball test, not a result.</strong> Two samples per arm
|
||||
per prompt is enough to see whether the gap between the columns is bigger than the gap between the
|
||||
two seeds <em>inside</em> a column — and not enough for anything else. No scoring, no statistics.
|
||||
The frozen adjudication rule and the Burrows's-Delta instrument are untouched by this page and
|
||||
nothing here feeds them.<br><br>
|
||||
Both arms are <strong>Qwen3-0.6B-Base doing continuation</strong>, not instruction-following. The
|
||||
adapter was trained as pure continuation, so each prompt is an opening line the model carries on
|
||||
from — asking a base model to "rewrite this in Brontë's voice" would test instruction-following
|
||||
instead of voice.</div>
|
||||
|
||||
{''.join(rows)}
|
||||
|
||||
<footer>Generated on pfi-gx10 (GB10), bf16, sdpa. Sampler pinned identical across arms:
|
||||
temperature 0.9, top_p 0.95, 400 new tokens, seeds 1234 and 5678. Adapter binding proven at
|
||||
generation time (196/196 lora_B tensors non-zero) — a silent no-op looks exactly like a tune that
|
||||
changed nothing.</footer>
|
||||
</div>"""
|
||||
|
||||
(out_dir / "index.html").write_text(page, encoding="utf-8")
|
||||
print(f"wrote {out_dir/'index.html'} ({len(ids)} prompts x 2 arms x 2 seeds)")
|
||||
Executable
+48
@@ -0,0 +1,48 @@
|
||||
#!/usr/bin/env bash
|
||||
# Chained after the 1.7B rung: re-run the 0.6B carrier on the SAME unwrapped corpus.
|
||||
#
|
||||
# The 1.7B run moved two variables at once -- carrier size AND corpus typography --
|
||||
# because the unwrap could not wait if every further rung was to avoid inheriting the
|
||||
# Gutenberg line breaks. That makes a 0.6B-vs-1.7B comparison descriptive rather than
|
||||
# attributable, which is fine for "did sense come back" (a within-arm reading) and not
|
||||
# fine for anything quantitative between rungs.
|
||||
#
|
||||
# This closes that hole for the price of ~36 minutes on an idle experimental box:
|
||||
# 0.6B on the unwrapped corpus, seed 4919, everything else held. Then carrier size is
|
||||
# the ONLY difference between this and h02-1p7b-1ep, and the sweep is single-variable
|
||||
# again.
|
||||
#
|
||||
# ⚠ Gated on the 1.7B run having actually produced an adapter. If that run died, this
|
||||
# must not quietly start and consume the box; a chain that fires on failure turns one
|
||||
# lost run into two.
|
||||
set -uo pipefail
|
||||
PREV=/home/infra-ops/r49-runs/h02-1p7b-1ep
|
||||
OUT=/home/infra-ops/r49-runs/h02-0p6b-1ep-unwrapped
|
||||
LOG=$OUT/train.log
|
||||
CHAINLOG=/home/infra-ops/r49-runs/chain-0p6b-unwrapped.log
|
||||
|
||||
exec >> "$CHAINLOG" 2>&1
|
||||
echo "=== $(date -Is) chain armed, waiting on $PREV"
|
||||
while [ -f "$PREV/run.pid" ] && kill -0 "$(cat "$PREV/run.pid")" 2>/dev/null; do sleep 60; done
|
||||
echo "=== $(date -Is) 1.7B run finished"
|
||||
|
||||
if [ ! -f "$PREV/adapter/adapter_model.safetensors" ]; then
|
||||
echo "=== REFUSING to chain: $PREV produced no adapter -- the 1.7B run did not succeed"
|
||||
exit 1
|
||||
fi
|
||||
apps=$(nvidia-smi --query-compute-apps=pid --format=csv,noheader | tr -d '[:space:]')
|
||||
if [ -n "$apps" ]; then
|
||||
echo "=== REFUSING to chain: GPU not clear"
|
||||
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mkdir -p "$OUT"
|
||||
echo "# launched $(date -Is) Qwen3-0.6B-Base, 1 epoch, seed 4919, UNWRAPPED corpus (single-variable partner to h02-1p7b-1ep)" > "$LOG"
|
||||
setsid nohup /home/infra-ops/ml/.venv/bin/python /home/infra-ops/r49-prep/train_voice_lora.py \
|
||||
--corpus /home/infra-ops/r49-corpus-renamed-unwrapped \
|
||||
--base /home/infra-ops/carriers/Qwen3-0.6B-Base \
|
||||
--seed 4919 --epochs 1 --eval-steps 25 --save-steps 25 \
|
||||
--out "$OUT" >> "$LOG" 2>&1 < /dev/null &
|
||||
echo $! > "$OUT/run.pid"
|
||||
echo "=== $(date -Is) chained 0.6B launched pid $(cat "$OUT/run.pid") -> $LOG"
|
||||
@@ -0,0 +1,82 @@
|
||||
"""Eyeball A/B: does the H02 adapter pull arbitrary prose toward Charlotte Brontë?
|
||||
|
||||
NOT the adjudication. The frozen rule, the Burrows's-Delta instrument and the
|
||||
held-out chapters are untouched by this; nothing here feeds them. This exists
|
||||
because the operator asked to *see* whether the voice moved.
|
||||
|
||||
Two design choices that decide whether the test says anything:
|
||||
|
||||
* The prompts are deliberately NOT Brontë-ish. Feed a base model "the moors lay
|
||||
dark under a bruised sky" and both arms come back Victorian, because the prompt
|
||||
did the work. So the set runs a difficulty gradient -- modern/mundane, then
|
||||
period-neutral, then Victorian-adjacent-but-plainly-worded -- and any Brontë in
|
||||
the modern tier is attributable to the adapter rather than to the setup.
|
||||
* Two seeds per prompt per arm, which is nearly free on a 0.6B and is the only
|
||||
thing that makes the comparison readable. One sample per arm cannot tell "the
|
||||
adapter changed the voice" from "sampling is noisy"; a reader with two samples
|
||||
of each arm can at least see whether the between-arm gap exceeds the
|
||||
within-arm gap. That is an eyeball noise floor, not a measurement, and it is
|
||||
not offered as one.
|
||||
|
||||
Same harness for both arms -- same box, same sampler, same prompts, same lengths --
|
||||
because a cross-comparison whose harness differs is invalid rather than noisy.
|
||||
Sampler matches the pinned adjudication sampler (temp 0.9 / top_p 0.95 / 400 new
|
||||
tokens) so what is on screen is the same shape of output the real arms produced.
|
||||
|
||||
⚠ This is a BASE model doing CONTINUATION, and the adapter was trained as pure
|
||||
continuation (H02 has no beat annotation by design). It will not follow a "rewrite
|
||||
this in Brontë's voice" instruction, and asking it to would test instruction-
|
||||
following rather than voice. So each prompt is an opening line the model continues.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import argparse, json, time
|
||||
from pathlib import Path
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--base", required=True)
|
||||
ap.add_argument("--adapter", default=None)
|
||||
ap.add_argument("--arm", required=True)
|
||||
ap.add_argument("--prompts", required=True)
|
||||
ap.add_argument("--out", required=True)
|
||||
ap.add_argument("--seeds", type=int, nargs="+", default=[1234, 5678])
|
||||
ap.add_argument("--max-new-tokens", type=int, default=400)
|
||||
ap.add_argument("--temperature", type=float, default=0.9)
|
||||
ap.add_argument("--top-p", type=float, default=0.95)
|
||||
a = ap.parse_args()
|
||||
|
||||
prompts = json.loads(Path(a.prompts).read_text())
|
||||
tok = AutoTokenizer.from_pretrained(a.base)
|
||||
model = AutoModelForCausalLM.from_pretrained(a.base, dtype=torch.bfloat16,
|
||||
attn_implementation="sdpa").to("cuda")
|
||||
if a.adapter:
|
||||
from peft import PeftModel
|
||||
model = PeftModel.from_pretrained(model, a.adapter)
|
||||
# ⚠ Prove the adapter actually BOUND. A silent no-op looks exactly like a tune
|
||||
# that changed nothing -- which is the very thing this test is trying to see.
|
||||
deltas = [float(m.lora_B["default"].weight.abs().sum())
|
||||
for m in model.modules() if hasattr(m, "lora_B")]
|
||||
nonzero = sum(1 for d in deltas if d > 0)
|
||||
print(f"[gen] adapter bound: {nonzero}/{len(deltas)} lora_B tensors non-zero", flush=True)
|
||||
if nonzero == 0:
|
||||
raise SystemExit("REFUSING: adapter applied but every lora_B is zero -- it did not bind")
|
||||
model.eval()
|
||||
|
||||
out = Path(a.out); out.parent.mkdir(parents=True, exist_ok=True)
|
||||
t0 = time.time()
|
||||
with out.open("w", encoding="utf-8") as fh:
|
||||
for p in prompts:
|
||||
for seed in a.seeds:
|
||||
torch.manual_seed(seed) # per-sample, so seed N is comparable across arms
|
||||
ids = tok(p["prompt"], return_tensors="pt").to("cuda")
|
||||
with torch.no_grad():
|
||||
g = model.generate(**ids, do_sample=True, temperature=a.temperature,
|
||||
top_p=a.top_p, max_new_tokens=a.max_new_tokens,
|
||||
pad_token_id=tok.eos_token_id)
|
||||
cont = tok.decode(g[0][ids["input_ids"].shape[1]:], skip_special_tokens=True)
|
||||
fh.write(json.dumps({"arm": a.arm, "id": p["id"], "tier": p["tier"],
|
||||
"prompt": p["prompt"], "seed": seed,
|
||||
"continuation": cont}) + "\n")
|
||||
print(f" {a.arm} {p['id']} seed={seed} {len(cont.split())}w", flush=True)
|
||||
print(f"[gen] {a.arm} -> {out} in {time.time()-t0:.0f}s", flush=True)
|
||||
Executable
+44
@@ -0,0 +1,44 @@
|
||||
#!/usr/bin/env bash
|
||||
# R49 H02 — rung 2 of the carrier sweep: Qwen3-1.7B-Base, 1 epoch.
|
||||
#
|
||||
# WHY THIS RUN. The 0.6B rung answered the narrow question and the operator's read
|
||||
# was the finding: "it's all nonsense, but it sounds like Brontë's nonsense."
|
||||
# Voice transferred (curly quotes 1/18 -> 18/18, worksheet collapse 3/18 -> 0/18)
|
||||
# while coherence did not. That separation is the premise the whole lightweight
|
||||
# author-voice regime rests on, so the live question is which carrier size brings
|
||||
# sense back while the voice stays. 1.7B is the next rung; 4B is after it.
|
||||
#
|
||||
# ⚠ THE CORPUS CHANGED, DELIBERATELY, AND IT IS A SECOND VARIABLE.
|
||||
# The 0.6B adapter learned the Gutenberg transcription's ~70-char line breaks along
|
||||
# with the prose -- its output wrapped at a mid-length-line ratio of 0.85 against
|
||||
# the base model's 0.00. That is typography, not style, and every further rung
|
||||
# would inherit it. So this trains on r49-corpus-renamed-unwrapped: same words,
|
||||
# reflowed into paragraphs, verified whitespace-only on all 852 records, with verse
|
||||
# blocks detected by median line length and their lineation preserved (0 lines
|
||||
# ended in a lone hyphen, so the space-join could not split a word).
|
||||
# CONSEQUENCE: a 0.6B-vs-1.7B comparison is now DESCRIPTIVE, not attributable --
|
||||
# carrier size and corpus typography both moved. "Did sense come back at 1.7B" is a
|
||||
# within-arm reading and survives that; any between-rung delta does not. The 0.6B
|
||||
# rerun on this same corpus is chained after this run so the clean single-variable
|
||||
# comparison exists too.
|
||||
#
|
||||
# Everything else is held from the 0.6B run: seed 4919, rank 32, lr 1e-4, seq 4096,
|
||||
# batch 1 x accum 8, 1 epoch, eval+save every 25 steps so the minimum is LOCATED
|
||||
# rather than assumed (the 3-epoch run overfit with save_strategy="no" and left
|
||||
# nothing to fall back to).
|
||||
set -euo pipefail
|
||||
OUT=/home/infra-ops/r49-runs/h02-1p7b-1ep
|
||||
LOG=$OUT/train.log
|
||||
apps=$(nvidia-smi --query-compute-apps=pid --format=csv,noheader | tr -d '[:space:]')
|
||||
[ -n "$apps" ] && { echo "REFUSING: GPU not clear" >&2; nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv >&2; exit 1; }
|
||||
[ -f "$OUT/run.pid" ] && kill -0 "$(cat "$OUT/run.pid")" 2>/dev/null && { echo "REFUSING: live pid" >&2; exit 1; }
|
||||
[ -e "$LOG" ] && { echo "REFUSING: $LOG exists" >&2; exit 1; }
|
||||
mkdir -p "$OUT"
|
||||
echo "# launched $(date -Is) Qwen3-1.7B-Base, 1 epoch, seed 4919, UNWRAPPED corpus, eval+save every 25" > "$LOG"
|
||||
setsid nohup /home/infra-ops/ml/.venv/bin/python /home/infra-ops/r49-prep/train_voice_lora.py \
|
||||
--corpus /home/infra-ops/r49-corpus-renamed-unwrapped \
|
||||
--base /home/infra-ops/carriers/Qwen3-1.7B-Base \
|
||||
--seed 4919 --epochs 1 --eval-steps 25 --save-steps 25 \
|
||||
--out "$OUT" >> "$LOG" 2>&1 < /dev/null &
|
||||
echo $! > "$OUT/run.pid"
|
||||
echo "launched pid $(cat "$OUT/run.pid") -> $LOG"
|
||||
@@ -0,0 +1,99 @@
|
||||
"""Reflow the hard-wrapped renamed corpus into flowing paragraphs.
|
||||
|
||||
The 0.6B adapter learned the Gutenberg transcription's ~70-character line breaks
|
||||
along with Charlotte Brontë's voice: its output wraps at a median mid-length-line
|
||||
ratio of 0.85 against the base model's 0.00. That is typography, not style, and
|
||||
every further rung would inherit it.
|
||||
|
||||
⚠ VERSE IS THE HAZARD. These novels contain poems, and a blind join would smear
|
||||
them into prose. So the join is decided per paragraph block by MEDIAN LINE LENGTH:
|
||||
a block whose lines sit near the wrap width is flowed prose and gets joined; a
|
||||
block of consistently short lines is verse (or a heading, or an address) and keeps
|
||||
its breaks. Both counts are reported, because a rule that silently reclassified
|
||||
half the poetry would look exactly like a rule that worked.
|
||||
|
||||
The acceptance check is content identity: `" ".join(text.split())` must be
|
||||
byte-identical before and after, for every record. That proves ONLY whitespace
|
||||
changed -- no word gained, lost, or altered. A reflow that quietly dropped a line
|
||||
would otherwise be invisible.
|
||||
|
||||
Writes to a NEW directory. The original stays exactly as the 0.6B run's provenance
|
||||
pins it (corpus_sha256_16 3959036cf851bf62), so that run remains reproducible.
|
||||
"""
|
||||
import json
|
||||
import statistics
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
SRC = Path(sys.argv[1])
|
||||
DST = Path(sys.argv[2])
|
||||
PROSE_MEDIAN = 55 # a wrapped-prose block's lines cluster near the wrap width
|
||||
|
||||
stats = {"records": 0, "blocks": 0, "joined": 0, "kept": 0, "hyphen_ends": 0}
|
||||
|
||||
|
||||
def reflow(text: str) -> str:
|
||||
out_blocks = []
|
||||
for block in text.split("\n\n"):
|
||||
lines = block.split("\n")
|
||||
body = [l for l in lines if l.strip()]
|
||||
if not body:
|
||||
out_blocks.append(block)
|
||||
continue
|
||||
stats["blocks"] += 1
|
||||
if len(body) == 1:
|
||||
out_blocks.append(block)
|
||||
stats["kept"] += 1
|
||||
continue
|
||||
med = statistics.median(len(l.rstrip()) for l in body[:-1] or body)
|
||||
if med >= PROSE_MEDIAN:
|
||||
stats["joined"] += 1
|
||||
for l in body[:-1]:
|
||||
if l.rstrip().endswith("-") and not l.rstrip().endswith("--"):
|
||||
stats["hyphen_ends"] += 1
|
||||
out_blocks.append(" ".join(l.strip() for l in body))
|
||||
else:
|
||||
stats["kept"] += 1
|
||||
out_blocks.append(block)
|
||||
return "\n\n".join(out_blocks)
|
||||
|
||||
|
||||
DST.mkdir(parents=True, exist_ok=True)
|
||||
(DST / "copies").mkdir(exist_ok=True)
|
||||
for f in sorted((SRC / "copies").glob("*.jsonl")):
|
||||
rows_out = []
|
||||
for line in f.read_text(encoding="utf-8").splitlines():
|
||||
if not line.strip():
|
||||
continue
|
||||
r = json.loads(line)
|
||||
before = r["text"]
|
||||
after = reflow(before)
|
||||
# ⚠ the acceptance check: content identical, whitespace only
|
||||
assert " ".join(before.split()) == " ".join(after.split()), \
|
||||
f"CONTENT CHANGED in {f.name} {r.get('work')}/{r.get('chapter')}"
|
||||
r["text"] = after
|
||||
rows_out.append(json.dumps(r, ensure_ascii=False))
|
||||
stats["records"] += 1
|
||||
(DST / "copies" / f.name).write_text("\n".join(rows_out) + "\n", encoding="utf-8")
|
||||
|
||||
for extra in ("rename_stats.json", "manifest.json"):
|
||||
if (SRC / extra).exists():
|
||||
(DST / extra).write_text((SRC / extra).read_text(encoding="utf-8"), encoding="utf-8")
|
||||
|
||||
# What the wrap ratio actually became -- the number the defect was measured with.
|
||||
def wrap_ratio(root):
|
||||
rs = []
|
||||
for f in sorted((root / "copies").glob("*.jsonl")):
|
||||
for line in f.read_text(encoding="utf-8").splitlines():
|
||||
if not line.strip():
|
||||
continue
|
||||
lines = [l for l in json.loads(line)["text"].split("\n") if l.strip()]
|
||||
rs.append(sum(1 for l in lines if 20 < len(l) < 78) / max(1, len(lines)))
|
||||
return statistics.median(rs)
|
||||
|
||||
print(f" records {stats['records']} blocks {stats['blocks']} "
|
||||
f"joined {stats['joined']} kept-as-is {stats['kept']} (verse/headings/single-line)")
|
||||
print(f" lines ending in a lone hyphen inside joined blocks: {stats['hyphen_ends']} "
|
||||
f"(a nonzero count means words were split across lines and a space-join would break them)")
|
||||
print(f" mid-length-line ratio before {wrap_ratio(SRC):.2f} -> after {wrap_ratio(DST):.2f}")
|
||||
print(f" content identity: PASSED on all {stats['records']} records (whitespace-only change)")
|
||||
@@ -0,0 +1,11 @@
|
||||
[
|
||||
{"id":"a1","tier":"modern","prompt":"The self-checkout machine refused her coupon for the third time."},
|
||||
{"id":"a2","tier":"modern","prompt":"He parked the car and sat listening to the engine tick as it cooled."},
|
||||
{"id":"a3","tier":"modern","prompt":"The office kitchen smelled of burnt coffee and somebody's reheated fish."},
|
||||
{"id":"b1","tier":"neutral","prompt":"She had not slept, and the morning found her at the window."},
|
||||
{"id":"b2","tier":"neutral","prompt":"There was a letter on the table, and no one would say who had brought it."},
|
||||
{"id":"b3","tier":"neutral","prompt":"The child would not speak, though she had been asked three times."},
|
||||
{"id":"c1","tier":"period","prompt":"The new governess arrived on a wet Tuesday and nobody came to meet her."},
|
||||
{"id":"c2","tier":"period","prompt":"Rain came on hard after dark, and the road up to the house turned to mud."},
|
||||
{"id":"c3","tier":"period","prompt":"He was the sort of man who took up all the air in a small room."}
|
||||
]
|
||||
Reference in New Issue
Block a user