49 lines
3.5 KiB
Markdown
49 lines
3.5 KiB
Markdown
# [2026-07-17] P06-driver Referential-frame A/B/C probe (operator-ratified, deploy-gated)
|
||
|
||
**QUEUED, non-urgent — waits on WT b122 landing on :8081.** worldtree-dev thread
|
||
`01KXQK7A0RHXV2YSXK5F5ETKJW`; brokkr's design in his thread msg `01KXQJDWJH`; rubric pin
|
||
`01KXQKB7ZQ`. This is ratatoskr's P06-driver role (I drove the R34/R35 P06 eval — the harness
|
||
is `scratchpad/p06_driver.py` + `p06_bind.py` + `manifest_memory.jsonl` + `eval_profiles_WIRE_READY.md`).
|
||
|
||
## Hypothesis (Vuong, ratified)
|
||
The char-self leak (Sindra's own scripted behavior — e.g. `75aa3110` "prefers to establish clear
|
||
parameters" — mis-slotted to the USER partition, caught this session) is a subject-taxonomy
|
||
**REFERENTIAL-FRAME artifact**: the extractor speaks first-person AS the character but must emit the
|
||
third-person token "character". brokkr's 3-arm A/B/C tests whether swapping the vocab fixes it.
|
||
|
||
## What WT built (v1.0.0b122, zero config delta — existing probe key + :8081 route work as-is)
|
||
- `POST /admin/producer-probe` takes optional `subject_vocab_override: {"user","character","both","other"}→token`.
|
||
- Arm A: omit. Arm B: `{"user":"user","character":"yourself","both":"both of you","other":"other"}`.
|
||
- Arm C: `{"user":"user","character":"me","both":"both of us","other":"other"}`. Malformed → 422 pre-spend.
|
||
- Each response item carries `subject_raw` (token actually emitted). Both prompt_paths carry the swap.
|
||
|
||
## Run shape (brokkr)
|
||
Identical fixture exchanges × 3 arms per seat — Deckard/Mag-class primary + ≥1 strong contrast seat.
|
||
Paired per-item deltas B-A / C-A. Primary metric = char-self recall. Guards = user→self over-attribution
|
||
+ subject-field omission rate.
|
||
|
||
## PINNED SCORING RUBRIC (brokkr, binding)
|
||
1. **Score = map `subject_raw` → canonical class via the ARM'S OWN vocab** (response meta echoes it),
|
||
compare to gold. **NEVER score the mapped `subject` field** (it fail-safes non-canonical tokens to "self").
|
||
2. Three emitted-token cases, pre-specified:
|
||
- ARM token → map via arm vocab.
|
||
- CANONICAL token under B/C (model REVERTED to "character"/"both", the swap didn't take) → map
|
||
canonical→canonical AND **count it** (informative).
|
||
- OFF-MENU token ("self"/"myself"/…) → best-effort canonical map, else an **UNSCORABLE-ERROR row**
|
||
in the n_actual accounting — never silently →self.
|
||
3. **MANIPULATION CHECK (load-bearing): report VOCAB-ADHERENCE rate (offered-token vs reverted vs
|
||
off-menu) per arm per seat ALONGSIDE every accuracy number. Interpret recall deltas ONLY where
|
||
adherence is adequate** — a low-adherence arm's delta is uninterpretable (flag it, don't report a delta).
|
||
4. Char-self fixtures carry per-item **construction-provenance** (what makes each char-self — e.g.
|
||
"verbatim from the character's system-prompt behavior clause, phrased user-neutrally").
|
||
|
||
## Fixture ask (deferrable pre-work, before any run — brokkr reviews the gold)
|
||
From the real P06 battery exchanges, pull candidate items per seat for 4 classes: (a) char-self leak
|
||
(character's own scripted behavior phrased USER-NEUTRALLY), (b) genuine-user, (c) both/commitment,
|
||
(d) other/third-party. ~15–25 char-self + balancing per seat; construction-known gold where possible;
|
||
WT hand-labels ambiguous; brokkr adjudicates the gold. Assembly + gold adjudication proceed pre-deploy;
|
||
the 3-arm run fires when b122 lands. Queued behind the R39 quartet-mapping.
|
||
|
||
Related: the char-self leak's origin is the same 75aa3110 finding folded into WT #364 as a
|
||
subject-attribution gate — see [[2026-07-16-wt364-r39-name-recall]].
|