# ERP tune — DPO stage readiness

_Written 2026-08-26 while run 2 (SFT, stock instruct base) finished. This is a
**readiness survey and a decision surface**, not a plan. The plan needs the
operator and brokkr-smithy-dev; what follows is what is on disk, what is
missing, and what has to be decided before anyone writes a config._

## Why there is a DPO stage at all

Run 2 is an SFT on `google/gemma-4-26B-A4B-it` — the **official instruct**
release, not an abliteration. Nothing in it attempted to remove refusals, so:

> **Run 2 will refuse at near-stock rates (~98-100/100 on the harmful-behaviors
> probe) and that is the design, not a regression.** Run 1's 29 came from its
> abliterated base, which run 2 deliberately does not use.

Targeted DPO is the stage where refusals get pruned — **on axes we choose**,
keeping the general-harm ones. That is the whole reason the stock base was
picked over a third-party abliteration: it replaces "inherit a stranger's
judgement about what to strip" with "strip what we decide to, deliberately."
Trading one training stage for that control was the operator's call on
2026-08-25.

## What is ready

| piece | state |
|---|---|
| **Base** | run 2's merged bf16 tune, `/tank/erp-tune/serve/merged-run02` |
| **SFT adapter** | `/tank/erp-tune/run-02/adapter/` — the DPO starting point |
| **Compute** | ana-ml2 GPU0, free once run 2's eval seat comes down |
| **Harness** | `eitri-smithy/erp_sft_harness` — corpus verification, encode cache, provenance, order manifest, resume. All reusable; none of it is SFT-specific except the loss |
| **Prose preference data** | two sets already on disk, unvetted: `/mnt/smithy/datasets/raw/jondurbin__gutenberg-dpo-v0.1` and `nbeerbower__gutenberg2-dpo` |
| **Reward judge** | LitBench-RM `Llama8B-CreativeWritingVerifier`, irv-ml1 `:8202/classify` — on-demand, ~90 s to respin |

## What is missing, in order of how much it will hurt

### 1. ⭐⭐⭐ Preference data for the refusal axes — the long pole

**Nothing on disk targets this.** The Gutenberg sets are prose-quality
preferences; they say nothing about refusal behaviour. Refusal-axis pairs have
to be generated: prompt the SFT tune, collect the refusal as `rejected`, and
produce a `chosen` completion that engages.

Three sub-problems, none of them mechanical:

- **Where does `chosen` come from?** A model that will write it, a human, or a
  template. Each has a different failure mode, and a templated `chosen` teaches
  the template.
- **How many pairs, and how balanced?** Too few and nothing moves; too many on
  one axis and it generalises past the axis.
- **What is the held-out set?** DPO can be evaluated by preference accuracy on
  held-out pairs, and that has to be split before generation, not after.

### 2. ⭐⭐⭐ The axis list itself — OPERATOR DECISION, blocks everything

Which refusals get pruned, and which are kept. Nobody should generate a single
pair before this is written down, because the axis list *is* the data spec.

R19's track-2 map is the existing artifact that names candidate axes. The
domain-compliance probe scaffold at
[`scripts/training-probes/counted_classifier.py`](../../scripts/training-probes/counted_classifier.py)
measures the RIGHT axis — explicit fiction — as opposed to the
`harmful_behaviors` probe that run 1 measured, which was the wrong axis and is
why run 1's refusal number was never the number anyone wanted.

⚠ **Measure the axis before pruning it.** Run the domain-compliance probe
against run 2 first: pruning an axis the tune already complies on is wasted
work, and the pre-number is the only baseline that will ever exist.

### 3. ⭐⭐ A DPO trainer — not present

`trl` is **not installed** in `/tank/erp-tune/venv`. Two routes:

- **Install `trl`, use `DPOTrainer`.** Fast, conventional, and drags a
  dependency into a venv whose torch/transformers/peft versions are load-bearing
  and known-good. ⚠ Pin and verify: `warmup_ratio` (playbook 4.2) is exactly
  what a version bump does to you.
- **Implement the DPO loss in the existing harness.** More work, but keeps the
  corpus verification, encode cache, order manifest, provenance and resume that
  the SFT harness already has and TRL does not know about. The harness is not
  SFT-specific except for the loss function.

⚠ **Whichever route, do the install AFTER a run finishes, never during one.** A
dependency resolution that upgrades `transformers` under a live process can
break its save path, which is the most expensive moment to break.

### 4. ⭐ Provenance for a preference run

The manifest gains fields the SFT one has no concept of: preference-set sha,
pair count, per-axis balance, beta, and the reference-model identity. Same rule
as everything else — see playbook 4.3's audit table. **A DPO run that does not
record which axes it pruned is unauditable**, and this stage is the one where
that matters most.

## ⚠ Measured 2026-08-26 — a length artifact the DPO pairs would inherit

The run-2 gate found tuned rp turns at **88.5 / 90.9 words against the base's
137.1 / 145.4** — a 36% reduction, consistent across blocks. brokkr-smithy-dev
hypothesised the mix was teaching PIPPA's product clip. The corpus side is now
measured and it is confirmed, with a mechanism sharper than "PIPPA is in the
mix":

    root                          bot turns   p50   p90   p99   max   <=123   124-130
    pippa                            79,414    67    95   108   123  100.0%     0.00%
    bluemoon                          1,170   563   908  1432  1924    0.0%     0.00%
    c2-logs-32k-llama3               23,704   234   394   686  2000    6.6%     1.44%
    creative-writing-multiturn        8,621   204   485   858  1837   24.0%     2.16%

**PIPPA's max is 123 exactly, 100% at or under it, 0.00% in the 124-130 band.**
That is a product clip (2023 Character.AI), not a length preference — every
other root crosses its own p99 smoothly.

The exposure asymmetry is the finding:

    root                        turn share    word share    mean len
    pippa                            70.3%         37.5%          67
    c2-logs-32k-llama3               21.0%         42.5%         256
    creative-writing-multiturn        7.6%         15.1%         251
    bluemoon                          1.0%          4.9%         601

**PIPPA is 70.3% of bot TURNS but 37.5% of bot WORDS — and length is learned per
turn, not per token.** Each of those 79,414 turns is a separate demonstration of
where a bot reply ends. By loss tokens PIPPA looks like a third of the dialogue
signal; by end-of-turn demonstrations it is seven in ten, from a source that
cannot exceed 123 words.

⚠ **Generalises past PIPPA: a length-clipped root is over-represented in the
length signal by exactly the ratio its clipping creates.** The shorter the clip,
the more turns per token of budget, the harder it teaches the clip. Check the
turn-share/word-share split of every root before assuming a mix is balanced.

### The output-side test ran. The clip hypothesis is FALSIFIED — and the truth is more interesting

Measured on the live tuned seat, n=96 rp generations (brokkr-smithy-dev,
2026-08-26). **There is no shoulder at 123:**

       0-19    18  ██████████████████
      20-39    22  ██████████████████████   <- MODE
      40-59     5  █████                    <- trough
      60-79    15  ███████████████          <- PIPPA median 67
      80-99    10  ██████████
     100-119    9  █████████
     120-139     3  ███                     <- PIPPA CAP 123. Three. A trough.
     140-159     5  █████
     160-179     4  ████
       300+      3  ███                      max 505

    <=123  79/96 = 82.3%   (PIPPA itself: 100.0%)
    median 62 · mean 77.9 · p90 152 · p99 367

**The clip-as-boundary reading is dead.** Mass would pile *under* a learned wall;
the 120-139 bin holds three of ninety-six and is a trough, and **17.7% of
generations cross a cap PIPPA can never cross.**

⚠ **A correction to this document's earlier read.** It compared the tuned *mean*
(88.5) to PIPPA's *median* (67) and p90 (95) and concluded "central tendency,
comfortably inside the upper body." **Median to median it is 62 against 67** — at
or slightly below PIPPA's centre. Mixing statistics across a comparison produced
a more reassuring answer than the data supports; the 88.5 was a mean dragged up
by the tail.

**What the data shows instead: the distribution is BIMODAL.** A mode at 20-39, a
trough at 40-59, a second mode at 60-79 astride PIPPA's centre, then a long tail
to 505. The base arm has no such shape (tuned mean/median ratio 1.45 and 1.23
against the base's 1.09 and 1.14).

So the tune did not change rp length so much as change its **shape** — replacing
one symmetric distribution with a mixture. That is what you would expect from
roots whose length distributions do not overlap (PIPPA 67, c2-logs 256, bluemoon
601) being learned as **distinct modes rather than blended into an average**.

⚠ **And the skew is rp-ONLY.** Story sits at mean/median 0.98 / 0.95 tuned
against 0.98 / 1.00 base. The shape change appears exactly in the family the
clipped root lives in — the strongest support the turn-share mechanism gets from
the output side.

**What this means for pair generation:** `chosen`/`rejected` sampled from a
bimodal generator inherit the mixture, not a mean. Sampling naively will
over-draw from the dominant short mode. If length is not the axis being tuned,
control for it explicitly rather than assuming a representative sample.

### ⚠ The degeneracy rate is NOT yet a usable baseline

Same arm, same seed block, re-run:

    scored       162/192 -> 163/192
    degenerate      7    ->   11
    truncated      23    ->   18
    rp VOID        no    ->  YES

Distance moved 0.003, well inside the 0.0125 floor, so the gate result is not
threatened. But **degeneracy straddles the 10% budget**, so the same arm on the
same seed VOIDs or does not depending on the run. **A guard whose trip point sits
at the noise boundary flips**, and it produces disagreement between honest
observers rather than silence — a distinct defect from a gate that cannot fail.

The honest statement is that **the tuned arm's rp degeneracy is AT the budget**,
not clearly over or under it. Replicates are being run to pin it. **Do not build
DPO pairs on the single-sample figure** — the degeneracy rate decides how much of
the tune's own output is usable as `chosen`, so it is load-bearing for pair
generation and much cheaper to pin now than to discover after pairs exist.

**Why this belongs in the DPO prep and not just the gate record:** preference
pairs generated *from this tune* inherit its length distribution in both
`chosen` and `rejected`. If short-rp is an artifact rather than a preference,
DPO on those pairs trains it in as an explicit objective rather than leaving it
as an incidental SFT bias. Settle the length question before generating pairs.

Scripts: `/tank/erp-tune/pippa_clip.py`, `/tank/erp-tune/clip_share.py`.

## The decision that gates the rest

**Which refusal axes are in scope, and which are explicitly kept?**

Everything downstream — data generation, pair counts, held-out split, the probe
that measures success — is a function of that list. It is not a technical call
and it should not be made by whoever writes the config.

## Pre-flight, when the time comes

Run [`training-throughput-playbook.md`](training-throughput-playbook.md) 4.8
before the window opens. Two items bite harder here than they did for SFT:

- **cache key covers the change** — a preference set is exactly the kind of
  auxiliary file whose *path* is stable while its *content* changes.
- **an observed consequence beside every configured value** — the axis list is a
  configured value, and its observed pair is the per-axis probe delta.