esh-pfi-infrastructure/docs/pfi/erp-dpo-stage-prep.md

ERP tune — DPO stage readiness

Written 2026-08-26 while run 2 (SFT, stock instruct base) finished. This is a readiness survey and a decision surface, not a plan. The plan needs the operator and brokkr-smithy-dev; what follows is what is on disk, what is missing, and what has to be decided before anyone writes a config.

Why there is a DPO stage at all

Run 2 is an SFT on google/gemma-4-26B-A4B-it — the official instruct release, not an abliteration. Nothing in it attempted to remove refusals, so:

Run 2 will refuse at near-stock rates (~98-100/100 on the harmful-behaviors probe) and that is the design, not a regression. Run 1's 29 came from its abliterated base, which run 2 deliberately does not use.

Targeted DPO is the stage where refusals get pruned — on axes we choose, keeping the general-harm ones. That is the whole reason the stock base was picked over a third-party abliteration: it replaces "inherit a stranger's judgement about what to strip" with "strip what we decide to, deliberately." Trading one training stage for that control was the operator's call on 2026-08-25.

What is ready

piece	state
Base	run 2's merged bf16 tune, /tank/erp-tune/serve/merged-run02
SFT adapter	/tank/erp-tune/run-02/adapter/ — the DPO starting point
Compute	ana-ml2 GPU0, free once run 2's eval seat comes down
Harness	eitri-smithy/erp_sft_harness — corpus verification, encode cache, provenance, order manifest, resume. All reusable; none of it is SFT-specific except the loss
Prose preference data	two sets already on disk, unvetted: /mnt/smithy/datasets/raw/jondurbin__gutenberg-dpo-v0.1 and nbeerbower__gutenberg2-dpo
Reward judge	LitBench-RM Llama8B-CreativeWritingVerifier, irv-ml1 :8202/classify — on-demand, ~90 s to respin

What is missing, in order of how much it will hurt

1. ⭐⭐⭐ Preference data for the refusal axes — the long pole

Nothing on disk targets this. The Gutenberg sets are prose-quality preferences; they say nothing about refusal behaviour. Refusal-axis pairs have to be generated: prompt the SFT tune, collect the refusal as rejected, and produce a chosen completion that engages.

Three sub-problems, none of them mechanical:

• Where does chosen come from? A model that will write it, a human, or a template. Each has a different failure mode, and a templated chosen teaches the template.
• How many pairs, and how balanced? Too few and nothing moves; too many on one axis and it generalises past the axis.
• What is the held-out set? DPO can be evaluated by preference accuracy on held-out pairs, and that has to be split before generation, not after.

2. ⭐⭐⭐ The axis list itself — OPERATOR DECISION, blocks everything

Which refusals get pruned, and which are kept. Nobody should generate a single pair before this is written down, because the axis list is the data spec.

R19's track-2 map is the existing artifact that names candidate axes. The domain-compliance probe scaffold at scripts/training-probes/counted_classifier.py measures the RIGHT axis — explicit fiction — as opposed to the harmful_behaviors probe that run 1 measured, which was the wrong axis and is why run 1's refusal number was never the number anyone wanted.

⚠ Measure the axis before pruning it. Run the domain-compliance probe against run 2 first: pruning an axis the tune already complies on is wasted work, and the pre-number is the only baseline that will ever exist.

3. ⭐⭐ A DPO trainer — not present

trl is not installed in /tank/erp-tune/venv. Two routes:

• Install trl, use DPOTrainer. Fast, conventional, and drags a dependency into a venv whose torch/transformers/peft versions are load-bearing and known-good. ⚠ Pin and verify: warmup_ratio (playbook 4.2) is exactly what a version bump does to you.
• Implement the DPO loss in the existing harness. More work, but keeps the corpus verification, encode cache, order manifest, provenance and resume that the SFT harness already has and TRL does not know about. The harness is not SFT-specific except for the loss function.

⚠ Whichever route, do the install AFTER a run finishes, never during one. A dependency resolution that upgrades transformers under a live process can break its save path, which is the most expensive moment to break.

4. ⭐ Provenance for a preference run

The manifest gains fields the SFT one has no concept of: preference-set sha, pair count, per-axis balance, beta, and the reference-model identity. Same rule as everything else — see playbook 4.3's audit table. A DPO run that does not record which axes it pruned is unauditable, and this stage is the one where that matters most.

⚠ Measured 2026-08-26 — a length artifact the DPO pairs would inherit

The run-2 gate found tuned rp turns at 88.5 / 90.9 words against the base's 137.1 / 145.4 — a 36% reduction, consistent across blocks. brokkr-smithy-dev hypothesised the mix was teaching PIPPA's product clip. The corpus side is now measured and it is confirmed, with a mechanism sharper than "PIPPA is in the mix":

root                          bot turns   p50   p90   p99   max   <=123   124-130
pippa                            79,414    67    95   108   123  100.0%     0.00%
bluemoon                          1,170   563   908  1432  1924    0.0%     0.00%
c2-logs-32k-llama3               23,704   234   394   686  2000    6.6%     1.44%
creative-writing-multiturn        8,621   204   485   858  1837   24.0%     2.16%

PIPPA's max is 123 exactly, 100% at or under it, 0.00% in the 124-130 band. That is a product clip (2023 Character.AI), not a length preference — every other root crosses its own p99 smoothly.

The exposure asymmetry is the finding:

root                        turn share    word share    mean len
pippa                            70.3%         37.5%          67
c2-logs-32k-llama3               21.0%         42.5%         256
creative-writing-multiturn        7.6%         15.1%         251
bluemoon                          1.0%          4.9%         601

PIPPA is 70.3% of bot TURNS but 37.5% of bot WORDS — and length is learned per turn, not per token. Each of those 79,414 turns is a separate demonstration of where a bot reply ends. By loss tokens PIPPA looks like a third of the dialogue signal; by end-of-turn demonstrations it is seven in ten, from a source that cannot exceed 123 words.

⚠ Generalises past PIPPA: a length-clipped root is over-represented in the length signal by exactly the ratio its clipping creates. The shorter the clip, the more turns per token of budget, the harder it teaches the clip. Check the turn-share/word-share split of every root before assuming a mix is balanced.

The output-side test ran. The clip hypothesis is FALSIFIED — and the truth is more interesting

Measured on the live tuned seat, n=96 rp generations (brokkr-smithy-dev, 2026-08-26). There is no shoulder at 123:

   0-19    18  ██████████████████
  20-39    22  ██████████████████████   <- MODE
  40-59     5  █████                    <- trough
  60-79    15  ███████████████          <- PIPPA median 67
  80-99    10  ██████████
 100-119    9  █████████
 120-139     3  ███                     <- PIPPA CAP 123. Three. A trough.
 140-159     5  █████
 160-179     4  ████
   300+      3  ███                      max 505

<=123  79/96 = 82.3%   (PIPPA itself: 100.0%)
median 62 · mean 77.9 · p90 152 · p99 367

The clip-as-boundary reading is dead. Mass would pile under a learned wall; the 120-139 bin holds three of ninety-six and is a trough, and 17.7% of generations cross a cap PIPPA can never cross.

⚠ A correction to this document's earlier read. It compared the tuned mean (88.5) to PIPPA's median (67) and p90 (95) and concluded "central tendency, comfortably inside the upper body." Median to median it is 62 against 67 — at or slightly below PIPPA's centre. Mixing statistics across a comparison produced a more reassuring answer than the data supports; the 88.5 was a mean dragged up by the tail.

What the data shows instead: the distribution is BIMODAL. A mode at 20-39, a trough at 40-59, a second mode at 60-79 astride PIPPA's centre, then a long tail to 505. The base arm has no such shape (tuned mean/median ratio 1.45 and 1.23 against the base's 1.09 and 1.14).

So the tune did not change rp length so much as change its shape — replacing one symmetric distribution with a mixture. That is what you would expect from roots whose length distributions do not overlap (PIPPA 67, c2-logs 256, bluemoon 601) being learned as distinct modes rather than blended into an average.

⚠ And the skew is rp-ONLY. Story sits at mean/median 0.98 / 0.95 tuned against 0.98 / 1.00 base. The shape change appears exactly in the family the clipped root lives in — the strongest support the turn-share mechanism gets from the output side.

What this means for pair generation: chosen/rejected sampled from a bimodal generator inherit the mixture, not a mean. Sampling naively will over-draw from the dominant short mode. If length is not the axis being tuned, control for it explicitly rather than assuming a representative sample.

⚠ The degeneracy rate is NOT yet a usable baseline

Same arm, same seed block, re-run:

scored       162/192 -> 163/192
degenerate      7    ->   11
truncated      23    ->   18
rp VOID        no    ->  YES

Distance moved 0.003, well inside the 0.0125 floor, so the gate result is not threatened. But degeneracy straddles the 10% budget, so the same arm on the same seed VOIDs or does not depending on the run. A guard whose trip point sits at the noise boundary flips, and it produces disagreement between honest observers rather than silence — a distinct defect from a gate that cannot fail.

The honest statement is that the tuned arm's rp degeneracy is AT the budget, not clearly over or under it. Replicates are being run to pin it. Do not build DPO pairs on the single-sample figure — the degeneracy rate decides how much of the tune's own output is usable as chosen, so it is load-bearing for pair generation and much cheaper to pin now than to discover after pairs exist.

Why this belongs in the DPO prep and not just the gate record: preference pairs generated from this tune inherit its length distribution in both chosen and rejected. If short-rp is an artifact rather than a preference, DPO on those pairs trains it in as an explicit objective rather than leaving it as an incidental SFT bias. Settle the length question before generating pairs.

Scripts: /tank/erp-tune/pippa_clip.py, /tank/erp-tune/clip_share.py.

The decision that gates the rest

Which refusal axes are in scope, and which are explicitly kept?

Everything downstream — data generation, pair counts, held-out split, the probe that measures success — is a function of that list. It is not a technical call and it should not be made by whoever writes the config.

Pre-flight, when the time comes

Run training-throughput-playbook.md 4.8 before the window opens. Two items bite harder here than they did for SFT:

• cache key covers the change — a preference set is exactly the kind of auxiliary file whose path is stable while its content changes.
• an observed consequence beside every configured value — the axis list is a configured value, and its observed pair is the per-axis probe delta.