# ERP tune — DPO stage readiness

_Written 2026-08-26 while run 2 (SFT, stock instruct base) finished. This is a
**readiness survey and a decision surface**, not a plan. The plan needs the
operator and brokkr-smithy-dev; what follows is what is on disk, what is
missing, and what has to be decided before anyone writes a config._

## Why there is a DPO stage at all

Run 2 is an SFT on `google/gemma-4-26B-A4B-it` — the **official instruct**
release, not an abliteration. Nothing in it attempted to remove refusals, so:

> **Run 2 will refuse at near-stock rates (~98-100/100 on the harmful-behaviors
> probe) and that is the design, not a regression.** Run 1's 29 came from its
> abliterated base, which run 2 deliberately does not use.

Targeted DPO is the stage where refusals get pruned — **on axes we choose**,
keeping the general-harm ones. That is the whole reason the stock base was
picked over a third-party abliteration: it replaces "inherit a stranger's
judgement about what to strip" with "strip what we decide to, deliberately."
Trading one training stage for that control was the operator's call on
2026-08-25.

## What is ready

| piece | state |
|---|---|
| **Base** | run 2's merged bf16 tune, `/tank/erp-tune/serve/merged-run02` |
| **SFT adapter** | `/tank/erp-tune/run-02/adapter/` — the DPO starting point |
| **Compute** | ana-ml2 GPU0, free once run 2's eval seat comes down |
| **Harness** | `eitri-smithy/erp_sft_harness` — corpus verification, encode cache, provenance, order manifest, resume. All reusable; none of it is SFT-specific except the loss |
| **Prose preference data** | two sets already on disk, unvetted: `/mnt/smithy/datasets/raw/jondurbin__gutenberg-dpo-v0.1` and `nbeerbower__gutenberg2-dpo` |
| **Reward judge** | LitBench-RM `Llama8B-CreativeWritingVerifier`, irv-ml1 `:8202/classify` — on-demand, ~90 s to respin |

## What is missing, in order of how much it will hurt

### 1. ⭐⭐⭐ Preference data for the refusal axes — the long pole

**Nothing on disk targets this.** The Gutenberg sets are prose-quality
preferences; they say nothing about refusal behaviour. Refusal-axis pairs have
to be generated: prompt the SFT tune, collect the refusal as `rejected`, and
produce a `chosen` completion that engages.

Three sub-problems, none of them mechanical:

- **Where does `chosen` come from?** A model that will write it, a human, or a
  template. Each has a different failure mode, and a templated `chosen` teaches
  the template.
- **How many pairs, and how balanced?** Too few and nothing moves; too many on
  one axis and it generalises past the axis.
- **What is the held-out set?** DPO can be evaluated by preference accuracy on
  held-out pairs, and that has to be split before generation, not after.

### 2. ⭐⭐⭐ The axis list itself — OPERATOR DECISION, blocks everything

Which refusals get pruned, and which are kept. Nobody should generate a single
pair before this is written down, because the axis list *is* the data spec.

R19's track-2 map is the existing artifact that names candidate axes. The
domain-compliance probe scaffold at
[`scripts/training-probes/counted_classifier.py`](../../scripts/training-probes/counted_classifier.py)
measures the RIGHT axis — explicit fiction — as opposed to the
`harmful_behaviors` probe that run 1 measured, which was the wrong axis and is
why run 1's refusal number was never the number anyone wanted.

⚠ **Measure the axis before pruning it.** Run the domain-compliance probe
against run 2 first: pruning an axis the tune already complies on is wasted
work, and the pre-number is the only baseline that will ever exist.

### 3. ⭐⭐ A DPO trainer — not present

`trl` is **not installed** in `/tank/erp-tune/venv`. Two routes:

- **Install `trl`, use `DPOTrainer`.** Fast, conventional, and drags a
  dependency into a venv whose torch/transformers/peft versions are load-bearing
  and known-good. ⚠ Pin and verify: `warmup_ratio` (playbook 4.2) is exactly
  what a version bump does to you.
- **Implement the DPO loss in the existing harness.** More work, but keeps the
  corpus verification, encode cache, order manifest, provenance and resume that
  the SFT harness already has and TRL does not know about. The harness is not
  SFT-specific except for the loss function.

⚠ **Whichever route, do the install AFTER a run finishes, never during one.** A
dependency resolution that upgrades `transformers` under a live process can
break its save path, which is the most expensive moment to break.

### 4. ⭐ Provenance for a preference run

The manifest gains fields the SFT one has no concept of: preference-set sha,
pair count, per-axis balance, beta, and the reference-model identity. Same rule
as everything else — see playbook 4.3's audit table. **A DPO run that does not
record which axes it pruned is unauditable**, and this stage is the one where
that matters most.

## ⚠ Measured 2026-08-26 — a length artifact the DPO pairs would inherit

The run-2 gate found tuned rp turns at **88.5 / 90.9 words against the base's
137.1 / 145.4** — a 36% reduction, consistent across blocks. brokkr-smithy-dev
hypothesised the mix was teaching PIPPA's product clip. The corpus side is now
measured and it is confirmed, with a mechanism sharper than "PIPPA is in the
mix":

    root                          bot turns   p50   p90   p99   max   <=123   124-130
    pippa                            79,414    67    95   108   123  100.0%     0.00%
    bluemoon                          1,170   563   908  1432  1924    0.0%     0.00%
    c2-logs-32k-llama3               23,704   234   394   686  2000    6.6%     1.44%
    creative-writing-multiturn        8,621   204   485   858  1837   24.0%     2.16%

**PIPPA's max is 123 exactly, 100% at or under it, 0.00% in the 124-130 band.**
That is a product clip (2023 Character.AI), not a length preference — every
other root crosses its own p99 smoothly.

The exposure asymmetry is the finding:

    root                        turn share    word share    mean len
    pippa                            70.3%         37.5%          67
    c2-logs-32k-llama3               21.0%         42.5%         256
    creative-writing-multiturn        7.6%         15.1%         251
    bluemoon                          1.0%          4.9%         601

**PIPPA is 70.3% of bot TURNS but 37.5% of bot WORDS — and length is learned per
turn, not per token.** Each of those 79,414 turns is a separate demonstration of
where a bot reply ends. By loss tokens PIPPA looks like a third of the dialogue
signal; by end-of-turn demonstrations it is seven in ten, from a source that
cannot exceed 123 words.

⚠ **Generalises past PIPPA: a length-clipped root is over-represented in the
length signal by exactly the ratio its clipping creates.** The shorter the clip,
the more turns per token of budget, the harder it teaches the clip. Check the
turn-share/word-share split of every root before assuming a mix is balanced.

⚠ **The tune landed near PIPPA's MEDIAN (67), not its CAP (123).** If it were
learning the clip as a boundary you would expect mass piling toward 123. It
looks more like central tendency, which is a weaker and less alarming claim than
"it learned the limit". Not demonstrated either way — the closing test is the
tuned arm's own rp length distribution against PIPPA's and bluemoon's, looking
for a shoulder near 123.

**Why this belongs in the DPO prep and not just the gate record:** preference
pairs generated *from this tune* inherit its length distribution in both
`chosen` and `rejected`. If short-rp is an artifact rather than a preference,
DPO on those pairs trains it in as an explicit objective rather than leaving it
as an incidental SFT bias. Settle the length question before generating pairs.

Scripts: `/tank/erp-tune/pippa_clip.py`, `/tank/erp-tune/clip_share.py`.

## The decision that gates the rest

**Which refusal axes are in scope, and which are explicitly kept?**

Everything downstream — data generation, pair counts, held-out split, the probe
that measures success — is a function of that list. It is not a technical call
and it should not be made by whoever writes the config.

## Pre-flight, when the time comes

Run [`training-throughput-playbook.md`](training-throughput-playbook.md) 4.8
before the window opens. Two items bite harder here than they did for SFT:

- **cache key covers the change** — a preference set is exactly the kind of
  auxiliary file whose *path* is stable while its *content* changes.
- **an observed consequence beside every configured value** — the axis list is a
  configured value, and its observed pair is the per-axis probe delta.
