vh
71e44176e9
Run 2 is finished, gated FAIL, and serving on the gateway at operator request. Run 3's corpus was built to brokkr's first recipe and held before any GPU spend: creative-writing-multiturn is a DECLARED MEGAMIX containing bluemoon, PIPPA, LimaRP and stheno, and the remix promoted creative-writing AND bluemoon -- the two roots that overlap, at median jaccard 0.873. Containment, not overlap. Dedup direction reversed so the primary source survives rather than the copy inside the bag: bluemoon 67 -> 126 conversations and 38.6% of loss signal, the largest contributor. Wholly-human share up, megamix share down, total context unchanged at 12.49M so the operator's settled mix arithmetic survived. Two structural findings recorded because they outlive this recipe: F1 'excise PIPPA' removes the ROOT and not the MATERIAL (F2's 250-word floor does that work, since PIPPA turns cannot exceed 123 words wherever they live), and LimaRP and stheno remain unchecked against any other root. Also records the correction I published wrong twice: run 2 was never unstable. All 46 flags were too_short, the collapse guards fired zero times, and it is the left tail of a length distribution -- not new to run 2 either, so it is a property of the recipe and a further base swap will not fix it.
117 lines
6.2 KiB
Markdown
117 lines
6.2 KiB
Markdown
# `[2026-08-26]` Run 3 corpus built, then held — and the megamix containment defect it exposed
|
|
|
|
Run 3's corpus was built to brokkr's `SFT-RECIPE-run3.md` (`f6c0589`), a blocker
|
|
was found before any GPU spend, and **brokkr is now redoing the recipe**. Nothing
|
|
launched. The build artifacts survive and are re-parameterisable.
|
|
|
|
## ⚠ THE FINDING: `creative-writing-multiturn` is a MEGAMIX containing our other roots
|
|
|
|
Its own Hoard card says so — `hoard/datasets/creative-writing-multiturn/dataset.yaml`,
|
|
`archive_provenance_note`:
|
|
|
|
> *"Dampfinchen/Creative_Writing_Multiturn — 2026 curated ~4.6k high-turn
|
|
> creative-writing/RP **megamix (bluemoon, LimaRP/PIPPA, stheno, gutenberg-dpo
|
|
> chosen, VellumK2 subset)**"*
|
|
|
|
**It is not a sibling root. It is a bag holding several of our other roots.**
|
|
|
|
Found because my corpus build would not reproduce brokkr's numbers: he computed
|
|
on the raw roots, I computed on run 1's deduped survivor list. bluemoon came out
|
|
67 against his 126. Run 1's dedup record explains it:
|
|
|
|
DROPPED -> KEPT n median jaccard
|
|
bluemoon -> creative-writing-multiturn 59 1.000
|
|
creative-writing-multiturn -> pippa 30 1.000
|
|
c2-logs-32k-llama3 -> creative-writing-multiturn 23 0.945
|
|
|
|
brokkr then measured it directly against the raw corpora, and it is worse than
|
|
the dedup threshold showed — **the MEDIAN bluemoon conversation is 87% similar to
|
|
something in creative-writing** (≥0.99: 28.3%, ≥0.90: 49.6%, max 0.999). That is
|
|
containment, not overlap.
|
|
|
|
**Why it mattered here specifically:** the remix elevates creative-writing to
|
|
43.2% and bluemoon to 31.4% of loss signal — *the two roots that overlap*.
|
|
Building on raw roots would have double-counted bluemoon inside creative-writing's
|
|
share, in a run whose entire justification is the composition change.
|
|
|
|
## Three consequences, two of them structural
|
|
|
|
**1. "bluemoon is the only human-authored root" was overstated.** It is the only
|
|
*wholly* human-authored root; creative-writing is *partly bluemoon*. brokkr
|
|
restated rather than withdrew.
|
|
|
|
**2. ⚠ F1 does not do what the recipe says.** "Excise PIPPA" removes the ROOT, not
|
|
the MATERIAL — PIPPA persists inside the megamix, which the remix promotes to
|
|
largest contributor. **F2's 250-word floor is what actually removes PIPPA
|
|
material**, because PIPPA turns cannot exceed 123 words wherever they live. The
|
|
outcome was right; the stated mechanism was wrong. **Had the operator chosen a
|
|
floor below 123, F1 would have been nearly cosmetic.**
|
|
|
|
**3. ⚠ The dedup kept the wrong side of the pair, 59 times.** When a primary
|
|
source and a megamix containing it collide, the survivor should be the primary
|
|
source, not the copy inside the bag — the megamix version may have been
|
|
reprocessed by its curator. We kept the derivative and discarded the origin.
|
|
|
|
**Reversed for run 3** (brokkr's call, ~3 minutes): keep bluemoon, drop the
|
|
megamix copy. Result is better than either prior version —
|
|
|
|
root run 2 recipe DEDUPED+REVERSED
|
|
bluemoon 6.9% 31.4% 38.6%
|
|
creative-writing 18.0% 43.2% 30.4%
|
|
c2-logs 40.2% 25.4% 31.0%
|
|
pippa 34.9% 0% 0%
|
|
|
|
**The wholly-human share went UP and the megamix share went DOWN** — strictly the
|
|
composition the recipe argued for, now honestly counted. Total context unchanged
|
|
at 12.49M, so the operator's settled mix arithmetic survived untouched.
|
|
|
|
## ⚠ The catalog-level problem, unsolved
|
|
|
|
**A megamix root silently contains other catalog roots and nothing in the mix
|
|
arithmetic knows.** Any share computed over roots including creative-writing
|
|
double-counts bluemoon, PIPPA, LimaRP and stheno by an unknown amount. Run 2's
|
|
mix had this defect and nobody noticed, because run 2 did not promote the
|
|
overlapping roots.
|
|
|
|
LimaRP and stheno are **still unchecked against anything.** Wants a
|
|
`contains_datasets:` field on the Hoard card plus a preflight assertion that no
|
|
recipe includes both a megamix and a root it contains. brokkr is raising it as
|
|
its own item; not a run-3 blocker.
|
|
|
|
## What is staged on ana-ml2
|
|
|
|
/tank/erp-tune/recipe-r3/recipe-erp-seat-sft-r3.json
|
|
/tank/erp-tune/recipe-r3/dialogue-survivors.jsonl 610 conversations
|
|
/tank/erp-tune/recipe-r3/lowquality-mask-v1.jsonl 5,867 mask rows
|
|
/tank/erp-tune/build_r3.py the filter build; floors are constants at the top
|
|
/tank/erp-tune/pippa_clip.py length-distribution measurement
|
|
/tank/erp-tune/clip_share.py turn-share vs word-share
|
|
|
|
⚠ **`/mnt/smithy` is READ-ONLY from ana-ml2**, so recipe artifacts live under
|
|
`/tank/erp-tune/recipe-r3/` rather than beside the r1 recipe. The harness takes
|
|
absolute paths, so this is fine — but do not expect to write to the r1 recipe dir.
|
|
|
|
**F2/F3/F4 are implemented as a LOSS MASK reusing the impersonation-mask format**,
|
|
so `load_impersonation_mask` reads it and `_impersonation_flags` sha-asserts every
|
|
masked turn with no new harness code. F4 is masked rather than stripped:
|
|
stripping mutates content, which invalidates the shas, and the read-only mount
|
|
forecloses a mutated derived root anyway. 28 turns of 6,209 (0.45%).
|
|
|
|
Re-parameterising for a new recipe is one command and ~90 seconds; the dedup
|
|
reversal and F4-as-mask handling are baked in and survive it.
|
|
|
|
## Decisions I took
|
|
|
|
**No conversation admission threshold** (brokkr delegated it, then endorsed).
|
|
81% of surviving bot turns are context-only, but masked context costs the
|
|
transformer body and **not** the LM head — the harness drops `IGNORE_INDEX`
|
|
before the 262k-vocab head, so the waste is bounded well below what the 81%
|
|
suggests. A 20% context trim buys ~35 minutes against a 3-hour run, and it would
|
|
be a fifth filter moving composition inside a run whose purpose is testing the
|
|
filters alone. **Buying 35 minutes by confounding a 3-hour experiment is a bad
|
|
trade.** brokkr: "your reason 1 is decisive."
|
|
|
|
**Left `erp-tune-v2` UP** on the operator's discretion grant. The card is not
|
|
needed until a recipe is settled, and it is his evaluation surface plus a live
|
|
gateway route. Take it down when run 3 is actually ready, not before.
|