docs(erp-dpo): readiness survey for the DPO stage
Run 2 is an SFT on the official instruct base, so it will refuse at near-stock rates by design; targeted DPO is where refusals get pruned on chosen axes. That was the trade accepted when the stock base was picked over a third-party abliteration. Surveys what is on disk against what the stage needs. Ready: the merged tune, the SFT adapter, GPU0 once the eval seat comes down, the whole non-loss half of the SFT harness, two unvetted Gutenberg preference sets, and the LitBench-RM judge. Missing, in order of pain: preference data for the refusal axes (nothing on disk targets it -- the Gutenberg sets are prose-quality), the axis list itself, and a DPO trainer (trl is not installed). The gating item is not technical: WHICH refusal axes are in scope and which are explicitly kept. Data generation, pair counts, the held-out split and the success probe are all functions of that list, so nobody should generate a pair before it is written down. Flags that the domain-compliance probe should measure run 2 BEFORE pruning, since the pre-number is the only baseline that will ever exist. Also records the operational trap: do the trl install AFTER a run finishes, never during one -- a resolution that upgrades transformers under a live process can break its save path.
This commit is contained in:
@@ -0,0 +1,111 @@
|
||||
# ERP tune — DPO stage readiness
|
||||
|
||||
_Written 2026-08-26 while run 2 (SFT, stock instruct base) finished. This is a
|
||||
**readiness survey and a decision surface**, not a plan. The plan needs the
|
||||
operator and brokkr-smithy-dev; what follows is what is on disk, what is
|
||||
missing, and what has to be decided before anyone writes a config._
|
||||
|
||||
## Why there is a DPO stage at all
|
||||
|
||||
Run 2 is an SFT on `google/gemma-4-26B-A4B-it` — the **official instruct**
|
||||
release, not an abliteration. Nothing in it attempted to remove refusals, so:
|
||||
|
||||
> **Run 2 will refuse at near-stock rates (~98-100/100 on the harmful-behaviors
|
||||
> probe) and that is the design, not a regression.** Run 1's 29 came from its
|
||||
> abliterated base, which run 2 deliberately does not use.
|
||||
|
||||
Targeted DPO is the stage where refusals get pruned — **on axes we choose**,
|
||||
keeping the general-harm ones. That is the whole reason the stock base was
|
||||
picked over a third-party abliteration: it replaces "inherit a stranger's
|
||||
judgement about what to strip" with "strip what we decide to, deliberately."
|
||||
Trading one training stage for that control was the operator's call on
|
||||
2026-08-25.
|
||||
|
||||
## What is ready
|
||||
|
||||
| piece | state |
|
||||
|---|---|
|
||||
| **Base** | run 2's merged bf16 tune, `/tank/erp-tune/serve/merged-run02` |
|
||||
| **SFT adapter** | `/tank/erp-tune/run-02/adapter/` — the DPO starting point |
|
||||
| **Compute** | ana-ml2 GPU0, free once run 2's eval seat comes down |
|
||||
| **Harness** | `eitri-smithy/erp_sft_harness` — corpus verification, encode cache, provenance, order manifest, resume. All reusable; none of it is SFT-specific except the loss |
|
||||
| **Prose preference data** | two sets already on disk, unvetted: `/mnt/smithy/datasets/raw/jondurbin__gutenberg-dpo-v0.1` and `nbeerbower__gutenberg2-dpo` |
|
||||
| **Reward judge** | LitBench-RM `Llama8B-CreativeWritingVerifier`, irv-ml1 `:8202/classify` — on-demand, ~90 s to respin |
|
||||
|
||||
## What is missing, in order of how much it will hurt
|
||||
|
||||
### 1. ⭐⭐⭐ Preference data for the refusal axes — the long pole
|
||||
|
||||
**Nothing on disk targets this.** The Gutenberg sets are prose-quality
|
||||
preferences; they say nothing about refusal behaviour. Refusal-axis pairs have
|
||||
to be generated: prompt the SFT tune, collect the refusal as `rejected`, and
|
||||
produce a `chosen` completion that engages.
|
||||
|
||||
Three sub-problems, none of them mechanical:
|
||||
|
||||
- **Where does `chosen` come from?** A model that will write it, a human, or a
|
||||
template. Each has a different failure mode, and a templated `chosen` teaches
|
||||
the template.
|
||||
- **How many pairs, and how balanced?** Too few and nothing moves; too many on
|
||||
one axis and it generalises past the axis.
|
||||
- **What is the held-out set?** DPO can be evaluated by preference accuracy on
|
||||
held-out pairs, and that has to be split before generation, not after.
|
||||
|
||||
### 2. ⭐⭐⭐ The axis list itself — OPERATOR DECISION, blocks everything
|
||||
|
||||
Which refusals get pruned, and which are kept. Nobody should generate a single
|
||||
pair before this is written down, because the axis list *is* the data spec.
|
||||
|
||||
R19's track-2 map is the existing artifact that names candidate axes. The
|
||||
domain-compliance probe scaffold at
|
||||
[`scripts/training-probes/counted_classifier.py`](../../scripts/training-probes/counted_classifier.py)
|
||||
measures the RIGHT axis — explicit fiction — as opposed to the
|
||||
`harmful_behaviors` probe that run 1 measured, which was the wrong axis and is
|
||||
why run 1's refusal number was never the number anyone wanted.
|
||||
|
||||
⚠ **Measure the axis before pruning it.** Run the domain-compliance probe
|
||||
against run 2 first: pruning an axis the tune already complies on is wasted
|
||||
work, and the pre-number is the only baseline that will ever exist.
|
||||
|
||||
### 3. ⭐⭐ A DPO trainer — not present
|
||||
|
||||
`trl` is **not installed** in `/tank/erp-tune/venv`. Two routes:
|
||||
|
||||
- **Install `trl`, use `DPOTrainer`.** Fast, conventional, and drags a
|
||||
dependency into a venv whose torch/transformers/peft versions are load-bearing
|
||||
and known-good. ⚠ Pin and verify: `warmup_ratio` (playbook 4.2) is exactly
|
||||
what a version bump does to you.
|
||||
- **Implement the DPO loss in the existing harness.** More work, but keeps the
|
||||
corpus verification, encode cache, order manifest, provenance and resume that
|
||||
the SFT harness already has and TRL does not know about. The harness is not
|
||||
SFT-specific except for the loss function.
|
||||
|
||||
⚠ **Whichever route, do the install AFTER a run finishes, never during one.** A
|
||||
dependency resolution that upgrades `transformers` under a live process can
|
||||
break its save path, which is the most expensive moment to break.
|
||||
|
||||
### 4. ⭐ Provenance for a preference run
|
||||
|
||||
The manifest gains fields the SFT one has no concept of: preference-set sha,
|
||||
pair count, per-axis balance, beta, and the reference-model identity. Same rule
|
||||
as everything else — see playbook 4.3's audit table. **A DPO run that does not
|
||||
record which axes it pruned is unauditable**, and this stage is the one where
|
||||
that matters most.
|
||||
|
||||
## The decision that gates the rest
|
||||
|
||||
**Which refusal axes are in scope, and which are explicitly kept?**
|
||||
|
||||
Everything downstream — data generation, pair counts, held-out split, the probe
|
||||
that measures success — is a function of that list. It is not a technical call
|
||||
and it should not be made by whoever writes the config.
|
||||
|
||||
## Pre-flight, when the time comes
|
||||
|
||||
Run [`training-throughput-playbook.md`](training-throughput-playbook.md) 4.8
|
||||
before the window opens. Two items bite harder here than they did for SFT:
|
||||
|
||||
- **cache key covers the change** — a preference set is exactly the kind of
|
||||
auxiliary file whose *path* is stable while its *content* changes.
|
||||
- **an observed consequence beside every configured value** — the axis list is a
|
||||
configured value, and its observed pair is the per-axis probe delta.
|
||||
Reference in New Issue
Block a user