vh
bcf63db527
Run 2 is an SFT on the official instruct base, so it will refuse at near-stock rates by design; targeted DPO is where refusals get pruned on chosen axes. That was the trade accepted when the stock base was picked over a third-party abliteration. Surveys what is on disk against what the stage needs. Ready: the merged tune, the SFT adapter, GPU0 once the eval seat comes down, the whole non-loss half of the SFT harness, two unvetted Gutenberg preference sets, and the LitBench-RM judge. Missing, in order of pain: preference data for the refusal axes (nothing on disk targets it -- the Gutenberg sets are prose-quality), the axis list itself, and a DPO trainer (trl is not installed). The gating item is not technical: WHICH refusal axes are in scope and which are explicitly kept. Data generation, pair counts, the held-out split and the success probe are all functions of that list, so nobody should generate a pair before it is written down. Flags that the domain-compliance probe should measure run 2 BEFORE pruning, since the pre-number is the only baseline that will ever exist. Also records the operational trap: do the trl install AFTER a run finishes, never during one -- a resolution that upgrades transformers under a live process can break its save path.
5.6 KiB
ERP tune — DPO stage readiness
Written 2026-08-26 while run 2 (SFT, stock instruct base) finished. This is a readiness survey and a decision surface, not a plan. The plan needs the operator and brokkr-smithy-dev; what follows is what is on disk, what is missing, and what has to be decided before anyone writes a config.
Why there is a DPO stage at all
Run 2 is an SFT on google/gemma-4-26B-A4B-it — the official instruct
release, not an abliteration. Nothing in it attempted to remove refusals, so:
Run 2 will refuse at near-stock rates (~98-100/100 on the harmful-behaviors probe) and that is the design, not a regression. Run 1's 29 came from its abliterated base, which run 2 deliberately does not use.
Targeted DPO is the stage where refusals get pruned — on axes we choose, keeping the general-harm ones. That is the whole reason the stock base was picked over a third-party abliteration: it replaces "inherit a stranger's judgement about what to strip" with "strip what we decide to, deliberately." Trading one training stage for that control was the operator's call on 2026-08-25.
What is ready
| piece | state |
|---|---|
| Base | run 2's merged bf16 tune, /tank/erp-tune/serve/merged-run02 |
| SFT adapter | /tank/erp-tune/run-02/adapter/ — the DPO starting point |
| Compute | ana-ml2 GPU0, free once run 2's eval seat comes down |
| Harness | eitri-smithy/erp_sft_harness — corpus verification, encode cache, provenance, order manifest, resume. All reusable; none of it is SFT-specific except the loss |
| Prose preference data | two sets already on disk, unvetted: /mnt/smithy/datasets/raw/jondurbin__gutenberg-dpo-v0.1 and nbeerbower__gutenberg2-dpo |
| Reward judge | LitBench-RM Llama8B-CreativeWritingVerifier, irv-ml1 :8202/classify — on-demand, ~90 s to respin |
What is missing, in order of how much it will hurt
1. ⭐⭐⭐ Preference data for the refusal axes — the long pole
Nothing on disk targets this. The Gutenberg sets are prose-quality
preferences; they say nothing about refusal behaviour. Refusal-axis pairs have
to be generated: prompt the SFT tune, collect the refusal as rejected, and
produce a chosen completion that engages.
Three sub-problems, none of them mechanical:
- Where does
chosencome from? A model that will write it, a human, or a template. Each has a different failure mode, and a templatedchosenteaches the template. - How many pairs, and how balanced? Too few and nothing moves; too many on one axis and it generalises past the axis.
- What is the held-out set? DPO can be evaluated by preference accuracy on held-out pairs, and that has to be split before generation, not after.
2. ⭐⭐⭐ The axis list itself — OPERATOR DECISION, blocks everything
Which refusals get pruned, and which are kept. Nobody should generate a single pair before this is written down, because the axis list is the data spec.
R19's track-2 map is the existing artifact that names candidate axes. The
domain-compliance probe scaffold at
scripts/training-probes/counted_classifier.py
measures the RIGHT axis — explicit fiction — as opposed to the
harmful_behaviors probe that run 1 measured, which was the wrong axis and is
why run 1's refusal number was never the number anyone wanted.
⚠ Measure the axis before pruning it. Run the domain-compliance probe against run 2 first: pruning an axis the tune already complies on is wasted work, and the pre-number is the only baseline that will ever exist.
3. ⭐⭐ A DPO trainer — not present
trl is not installed in /tank/erp-tune/venv. Two routes:
- Install
trl, useDPOTrainer. Fast, conventional, and drags a dependency into a venv whose torch/transformers/peft versions are load-bearing and known-good. ⚠ Pin and verify:warmup_ratio(playbook 4.2) is exactly what a version bump does to you. - Implement the DPO loss in the existing harness. More work, but keeps the corpus verification, encode cache, order manifest, provenance and resume that the SFT harness already has and TRL does not know about. The harness is not SFT-specific except for the loss function.
⚠ Whichever route, do the install AFTER a run finishes, never during one. A
dependency resolution that upgrades transformers under a live process can
break its save path, which is the most expensive moment to break.
4. ⭐ Provenance for a preference run
The manifest gains fields the SFT one has no concept of: preference-set sha, pair count, per-axis balance, beta, and the reference-model identity. Same rule as everything else — see playbook 4.3's audit table. A DPO run that does not record which axes it pruned is unauditable, and this stage is the one where that matters most.
The decision that gates the rest
Which refusal axes are in scope, and which are explicitly kept?
Everything downstream — data generation, pair counts, held-out split, the probe that measures success — is a function of that list. It is not a technical call and it should not be made by whoever writes the config.
Pre-flight, when the time comes
Run training-throughput-playbook.md 4.8
before the window opens. Two items bite harder here than they did for SFT:
- cache key covers the change — a preference set is exactly the kind of auxiliary file whose path is stable while its content changes.
- an observed consequence beside every configured value — the axis list is a configured value, and its observed pair is the per-axis probe delta.