From bcf63db5272d2cf4a4cdacfe218ef6cecf29d574 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Wed, 26 Aug 2026 01:18:41 -0700 Subject: [PATCH] docs(erp-dpo): readiness survey for the DPO stage Run 2 is an SFT on the official instruct base, so it will refuse at near-stock rates by design; targeted DPO is where refusals get pruned on chosen axes. That was the trade accepted when the stock base was picked over a third-party abliteration. Surveys what is on disk against what the stage needs. Ready: the merged tune, the SFT adapter, GPU0 once the eval seat comes down, the whole non-loss half of the SFT harness, two unvetted Gutenberg preference sets, and the LitBench-RM judge. Missing, in order of pain: preference data for the refusal axes (nothing on disk targets it -- the Gutenberg sets are prose-quality), the axis list itself, and a DPO trainer (trl is not installed). The gating item is not technical: WHICH refusal axes are in scope and which are explicitly kept. Data generation, pair counts, the held-out split and the success probe are all functions of that list, so nobody should generate a pair before it is written down. Flags that the domain-compliance probe should measure run 2 BEFORE pruning, since the pre-number is the only baseline that will ever exist. Also records the operational trap: do the trl install AFTER a run finishes, never during one -- a resolution that upgrades transformers under a live process can break its save path. --- docs/pfi/erp-dpo-stage-prep.md | 111 +++++++++++++++++++++++++++++++++ 1 file changed, 111 insertions(+) create mode 100644 docs/pfi/erp-dpo-stage-prep.md diff --git a/docs/pfi/erp-dpo-stage-prep.md b/docs/pfi/erp-dpo-stage-prep.md new file mode 100644 index 0000000..56a0d5f --- /dev/null +++ b/docs/pfi/erp-dpo-stage-prep.md @@ -0,0 +1,111 @@ +# ERP tune — DPO stage readiness + +_Written 2026-08-26 while run 2 (SFT, stock instruct base) finished. This is a +**readiness survey and a decision surface**, not a plan. The plan needs the +operator and brokkr-smithy-dev; what follows is what is on disk, what is +missing, and what has to be decided before anyone writes a config._ + +## Why there is a DPO stage at all + +Run 2 is an SFT on `google/gemma-4-26B-A4B-it` — the **official instruct** +release, not an abliteration. Nothing in it attempted to remove refusals, so: + +> **Run 2 will refuse at near-stock rates (~98-100/100 on the harmful-behaviors +> probe) and that is the design, not a regression.** Run 1's 29 came from its +> abliterated base, which run 2 deliberately does not use. + +Targeted DPO is the stage where refusals get pruned — **on axes we choose**, +keeping the general-harm ones. That is the whole reason the stock base was +picked over a third-party abliteration: it replaces "inherit a stranger's +judgement about what to strip" with "strip what we decide to, deliberately." +Trading one training stage for that control was the operator's call on +2026-08-25. + +## What is ready + +| piece | state | +|---|---| +| **Base** | run 2's merged bf16 tune, `/tank/erp-tune/serve/merged-run02` | +| **SFT adapter** | `/tank/erp-tune/run-02/adapter/` — the DPO starting point | +| **Compute** | ana-ml2 GPU0, free once run 2's eval seat comes down | +| **Harness** | `eitri-smithy/erp_sft_harness` — corpus verification, encode cache, provenance, order manifest, resume. All reusable; none of it is SFT-specific except the loss | +| **Prose preference data** | two sets already on disk, unvetted: `/mnt/smithy/datasets/raw/jondurbin__gutenberg-dpo-v0.1` and `nbeerbower__gutenberg2-dpo` | +| **Reward judge** | LitBench-RM `Llama8B-CreativeWritingVerifier`, irv-ml1 `:8202/classify` — on-demand, ~90 s to respin | + +## What is missing, in order of how much it will hurt + +### 1. ⭐⭐⭐ Preference data for the refusal axes — the long pole + +**Nothing on disk targets this.** The Gutenberg sets are prose-quality +preferences; they say nothing about refusal behaviour. Refusal-axis pairs have +to be generated: prompt the SFT tune, collect the refusal as `rejected`, and +produce a `chosen` completion that engages. + +Three sub-problems, none of them mechanical: + +- **Where does `chosen` come from?** A model that will write it, a human, or a + template. Each has a different failure mode, and a templated `chosen` teaches + the template. +- **How many pairs, and how balanced?** Too few and nothing moves; too many on + one axis and it generalises past the axis. +- **What is the held-out set?** DPO can be evaluated by preference accuracy on + held-out pairs, and that has to be split before generation, not after. + +### 2. ⭐⭐⭐ The axis list itself — OPERATOR DECISION, blocks everything + +Which refusals get pruned, and which are kept. Nobody should generate a single +pair before this is written down, because the axis list *is* the data spec. + +R19's track-2 map is the existing artifact that names candidate axes. The +domain-compliance probe scaffold at +[`scripts/training-probes/counted_classifier.py`](../../scripts/training-probes/counted_classifier.py) +measures the RIGHT axis — explicit fiction — as opposed to the +`harmful_behaviors` probe that run 1 measured, which was the wrong axis and is +why run 1's refusal number was never the number anyone wanted. + +⚠ **Measure the axis before pruning it.** Run the domain-compliance probe +against run 2 first: pruning an axis the tune already complies on is wasted +work, and the pre-number is the only baseline that will ever exist. + +### 3. ⭐⭐ A DPO trainer — not present + +`trl` is **not installed** in `/tank/erp-tune/venv`. Two routes: + +- **Install `trl`, use `DPOTrainer`.** Fast, conventional, and drags a + dependency into a venv whose torch/transformers/peft versions are load-bearing + and known-good. ⚠ Pin and verify: `warmup_ratio` (playbook 4.2) is exactly + what a version bump does to you. +- **Implement the DPO loss in the existing harness.** More work, but keeps the + corpus verification, encode cache, order manifest, provenance and resume that + the SFT harness already has and TRL does not know about. The harness is not + SFT-specific except for the loss function. + +⚠ **Whichever route, do the install AFTER a run finishes, never during one.** A +dependency resolution that upgrades `transformers` under a live process can +break its save path, which is the most expensive moment to break. + +### 4. ⭐ Provenance for a preference run + +The manifest gains fields the SFT one has no concept of: preference-set sha, +pair count, per-axis balance, beta, and the reference-model identity. Same rule +as everything else — see playbook 4.3's audit table. **A DPO run that does not +record which axes it pruned is unauditable**, and this stage is the one where +that matters most. + +## The decision that gates the rest + +**Which refusal axes are in scope, and which are explicitly kept?** + +Everything downstream — data generation, pair counts, held-out split, the probe +that measures success — is a function of that list. It is not a technical call +and it should not be made by whoever writes the config. + +## Pre-flight, when the time comes + +Run [`training-throughput-playbook.md`](training-throughput-playbook.md) 4.8 +before the window opens. Two items bite harder here than they did for SFT: + +- **cache key covers the change** — a preference set is exactly the kind of + auxiliary file whose *path* is stable while its *content* changes. +- **an observed consequence beside every configured value** — the axis list is a + configured value, and its observed pair is the per-axis probe delta.