vh
71e44176e9
Run 2 is finished, gated FAIL, and serving on the gateway at operator request. Run 3's corpus was built to brokkr's first recipe and held before any GPU spend: creative-writing-multiturn is a DECLARED MEGAMIX containing bluemoon, PIPPA, LimaRP and stheno, and the remix promoted creative-writing AND bluemoon -- the two roots that overlap, at median jaccard 0.873. Containment, not overlap. Dedup direction reversed so the primary source survives rather than the copy inside the bag: bluemoon 67 -> 126 conversations and 38.6% of loss signal, the largest contributor. Wholly-human share up, megamix share down, total context unchanged at 12.49M so the operator's settled mix arithmetic survived. Two structural findings recorded because they outlive this recipe: F1 'excise PIPPA' removes the ROOT and not the MATERIAL (F2's 250-word floor does that work, since PIPPA turns cannot exceed 123 words wherever they live), and LimaRP and stheno remain unchecked against any other root. Also records the correction I published wrong twice: run 2 was never unstable. All 46 flags were too_short, the collapse guards fired zero times, and it is the left tail of a length distribution -- not new to run 2 either, so it is a property of the recipe and a further base swap will not fix it.
6.2 KiB
[2026-08-26] Run 3 corpus built, then held — and the megamix containment defect it exposed
Run 3's corpus was built to brokkr's SFT-RECIPE-run3.md (f6c0589), a blocker
was found before any GPU spend, and brokkr is now redoing the recipe. Nothing
launched. The build artifacts survive and are re-parameterisable.
⚠ THE FINDING: creative-writing-multiturn is a MEGAMIX containing our other roots
Its own Hoard card says so — hoard/datasets/creative-writing-multiturn/dataset.yaml,
archive_provenance_note:
"Dampfinchen/Creative_Writing_Multiturn — 2026 curated ~4.6k high-turn creative-writing/RP megamix (bluemoon, LimaRP/PIPPA, stheno, gutenberg-dpo chosen, VellumK2 subset)"
It is not a sibling root. It is a bag holding several of our other roots.
Found because my corpus build would not reproduce brokkr's numbers: he computed on the raw roots, I computed on run 1's deduped survivor list. bluemoon came out 67 against his 126. Run 1's dedup record explains it:
DROPPED -> KEPT n median jaccard
bluemoon -> creative-writing-multiturn 59 1.000
creative-writing-multiturn -> pippa 30 1.000
c2-logs-32k-llama3 -> creative-writing-multiturn 23 0.945
brokkr then measured it directly against the raw corpora, and it is worse than the dedup threshold showed — the MEDIAN bluemoon conversation is 87% similar to something in creative-writing (≥0.99: 28.3%, ≥0.90: 49.6%, max 0.999). That is containment, not overlap.
Why it mattered here specifically: the remix elevates creative-writing to 43.2% and bluemoon to 31.4% of loss signal — the two roots that overlap. Building on raw roots would have double-counted bluemoon inside creative-writing's share, in a run whose entire justification is the composition change.
Three consequences, two of them structural
1. "bluemoon is the only human-authored root" was overstated. It is the only wholly human-authored root; creative-writing is partly bluemoon. brokkr restated rather than withdrew.
2. ⚠ F1 does not do what the recipe says. "Excise PIPPA" removes the ROOT, not the MATERIAL — PIPPA persists inside the megamix, which the remix promotes to largest contributor. F2's 250-word floor is what actually removes PIPPA material, because PIPPA turns cannot exceed 123 words wherever they live. The outcome was right; the stated mechanism was wrong. Had the operator chosen a floor below 123, F1 would have been nearly cosmetic.
3. ⚠ The dedup kept the wrong side of the pair, 59 times. When a primary source and a megamix containing it collide, the survivor should be the primary source, not the copy inside the bag — the megamix version may have been reprocessed by its curator. We kept the derivative and discarded the origin.
Reversed for run 3 (brokkr's call, ~3 minutes): keep bluemoon, drop the megamix copy. Result is better than either prior version —
root run 2 recipe DEDUPED+REVERSED
bluemoon 6.9% 31.4% 38.6%
creative-writing 18.0% 43.2% 30.4%
c2-logs 40.2% 25.4% 31.0%
pippa 34.9% 0% 0%
The wholly-human share went UP and the megamix share went DOWN — strictly the composition the recipe argued for, now honestly counted. Total context unchanged at 12.49M, so the operator's settled mix arithmetic survived untouched.
⚠ The catalog-level problem, unsolved
A megamix root silently contains other catalog roots and nothing in the mix arithmetic knows. Any share computed over roots including creative-writing double-counts bluemoon, PIPPA, LimaRP and stheno by an unknown amount. Run 2's mix had this defect and nobody noticed, because run 2 did not promote the overlapping roots.
LimaRP and stheno are still unchecked against anything. Wants a
contains_datasets: field on the Hoard card plus a preflight assertion that no
recipe includes both a megamix and a root it contains. brokkr is raising it as
its own item; not a run-3 blocker.
What is staged on ana-ml2
/tank/erp-tune/recipe-r3/recipe-erp-seat-sft-r3.json
/tank/erp-tune/recipe-r3/dialogue-survivors.jsonl 610 conversations
/tank/erp-tune/recipe-r3/lowquality-mask-v1.jsonl 5,867 mask rows
/tank/erp-tune/build_r3.py the filter build; floors are constants at the top
/tank/erp-tune/pippa_clip.py length-distribution measurement
/tank/erp-tune/clip_share.py turn-share vs word-share
⚠ /mnt/smithy is READ-ONLY from ana-ml2, so recipe artifacts live under
/tank/erp-tune/recipe-r3/ rather than beside the r1 recipe. The harness takes
absolute paths, so this is fine — but do not expect to write to the r1 recipe dir.
F2/F3/F4 are implemented as a LOSS MASK reusing the impersonation-mask format,
so load_impersonation_mask reads it and _impersonation_flags sha-asserts every
masked turn with no new harness code. F4 is masked rather than stripped:
stripping mutates content, which invalidates the shas, and the read-only mount
forecloses a mutated derived root anyway. 28 turns of 6,209 (0.45%).
Re-parameterising for a new recipe is one command and ~90 seconds; the dedup reversal and F4-as-mask handling are baked in and survive it.
Decisions I took
No conversation admission threshold (brokkr delegated it, then endorsed).
81% of surviving bot turns are context-only, but masked context costs the
transformer body and not the LM head — the harness drops IGNORE_INDEX
before the 262k-vocab head, so the waste is bounded well below what the 81%
suggests. A 20% context trim buys ~35 minutes against a 3-hour run, and it would
be a fifth filter moving composition inside a run whose purpose is testing the
filters alone. Buying 35 minutes by confounding a 3-hour experiment is a bad
trade. brokkr: "your reason 1 is decisive."
Left erp-tune-v2 UP on the operator's discretion grant. The card is not
needed until a recipe is settled, and it is his evaluation surface plus a live
gateway route. Take it down when run 3 is actually ready, not before.