memory: snapshot — the tune is trained, gated, and serving

Run-01 completed in 7:21:52 (47% faster than the 13.85h round-1 projection),
lora_B gate 205/205 non-zero at median norm 1.708, and the acceptance gate says
it did the thing it was built for: diversity +0.178 against a 0.008 floor (22x),
attractor hit rate -11.3pt against a 2.0pt floor, memorisation 0.0000 on both
arms — which closes the R20 licensed-prose exposure on measurement rather than
argument.

Five new detail files carry the substance:

  erp-tune-run2-complete        the run, the gate, the noise-floor near-miss
                                (brokkr was one step from reporting a 13-point
                                T6 regression sitting inside twice his
                                instrument's own variance)
  mfu-root-caused-attention     8.6% MFU was an accounting artifact; real
                                utilisation 17-20%, cost was attention on
                                AMPERE kernels. Two independent methods agreed
                                to 2.6 points.
  nvfp4-serving-pipeline        merged weights are MANDATORY — vLLM cannot
                                serve a LoRA on ANY Gemma-4 — plus the recipe
                                that silently misses all 11,520 expert tensors
  refusal-retention-probe       measured base 0/100 -> tuned 29/100, then had
                                to accept it was the wrong axis
  worldtree-b188-b189-and-selene  three arcs closed, and a #411 diagnosis I got
                                wrong twice before a directory probe settled it

Current state rewritten end to end — the previous snapshot had the run in
flight at ~17h with MFU unexplained. Both are now closed.

The open operator decision is run 2's base, deliberately unstaged and flagged
against being filed as a config knob: it is a reversal of the trainee-selection
decision, and the pretrained-base option removes the last non-lexical floor on
the CSAM axis given stage-2-detector-inert and contamination-scan-absent are
both already overridden.

Tried-and-abandoned gains four measured-dead throughput levers, the packing
correction (bucketing wins under sdpa and the conclusion flips under flex — do
not carry it past the backend decision), and the merge-back-undoes-abliteration
trap brokkr caught in his own advice.

Index stays at 291 lines, under the soft cap. No archival this run.
This commit is contained in:
vh
2026-08-25 16:54:49 -07:00
parent 2a05ae91af
commit 2656196f47
6 changed files with 470 additions and 21 deletions
@@ -0,0 +1,92 @@
# ERP/RP tune run-01 COMPLETE — 7.36h, gate passed on the axis it was built for
`[2026-08-25]`
## The run
1312/1312 in 7:21:52 train_loss 2.793 epoch 1.0
20.1 s/it FLAT across every 100-step window (round 1: 35-46.5 s/it)
adapter: /tank/erp-tune/run-01/adapter/ 410 tensors, provenance.json
**47% faster than the round-1 projection of 13.85h**, from two changes: the
bucketed sampler and flex attention. Rate was flat — 19.7 / 19.8 / 20.4 / 20.3
across the four 100-step windows — which means the 35-46.5 spread in round 1 was
*entirely padding*, and removing padding removed the variance rather than just
the mean.
⚠ **I quoted three different ETAs (6.9h, 8h, 7.3h) before I started using a
rolling average.** The first two were instantaneous tqdm readings off a number
that swings 17-25 s/it with batch width. Only the rolling rate was honest. Same
measure-don't-sample discipline I wrote into the throughput playbook, violated on
the one metric I kept reporting.
## lora_B gate — PASSED, twice
checkpoint-100 205/205 non-zero, median norm 0.829
final adapter 205/205 non-zero, median norm 1.708
vision_tower tensors: 0 on both
Median norm rising 0.829 -> 1.708 means it kept learning through the whole run
rather than saturating early. This check **never ran in round 1** (died at step
19, first checkpoint was 100) and it is the only failure mode that stays
invisible until the acceptance gate reports base-identical numbers.
## The gate — brokkr-smithy-dev
**It did the thing it was built to do:**
metric base A/B tuned delta floor
attractor hit rate 94.8% / 96.8% 84.5% -11.3pt 2.0pt
diversity (pairwise) 0.213 / 0.221 0.3948 +0.178 0.008
Diversity moved **22x its own noise floor**. Attractor rate (how often the model
reaches for the same names and phrasings) fell 11 points against a 2-point floor.
T1 100 · T2 95 · T3 96-97 · T4 98 · T5 100 · T6 81-82 · core ~94.2
memorisation: 0.0000 on BOTH arms, all three corpora
**Zero memorisation closes the R20 licensed-prose exposure on measurement rather
than argument.**
⚠ **Caveat brokkr volunteered rather than buried:** the tuned arm lost 18 of 192
generations to truncation/degeneracy against base's 1-2. Lopsided exclusions
plausibly flatter the diversity magnitude. Direction is unambiguous at 22x floor;
the number carries an asterisk.
## The noise-floor near-miss — the methodology lesson
brokkr was one step from reporting a 13-point T6 regression **that sat inside
twice his instrument's own variance.**
--per-type 32 max swing across tasks: 9 points
--per-type 128 max swing across tasks: 1 point
His gate criterion is "no task regresses by more than one item" = 3.1 points at
n=32. **The instrument's own run-to-run noise was 3 items.** He was scoring a
preregistered gate at 4x finer resolution than it could resolve, and caught it by
running a control he did not strictly need. Quadrupling n collapsed the noise
exactly as binomial statistics predicts.
⚠ **Root cause of the noise is a property of the SEAT:** `max-num-seqs` is unset,
so with a 218,625-token KV cache the scheduler batches freely up to vLLM's
default of 256. Continuous batching changes reduction order and borderline items
flip. Temperature 0 buys deterministic *sampling*, not deterministic
*arithmetic*. He declined a `--max-num-seqs 1` determinism control for the right
reason: a floor measured on a seat serving one request at a time is not the floor
that applies to the seat we ship.
## The confound I built and he caught
I optimised a pipeline for production and then handed him its output as an eval
instrument **without asking whether those were the same job.** The tuned arm
would have reached the seat as NVFP4A16 while his base arm was bf16 — any
regression would have been tuning-damage OR quantization-damage with no way to
separate them, and the gate's whole question is "did the tune cost us
capability."
**Both arms now bf16, same seat, same port, argv differing in exactly two
lines** (weights path, served name), template sha256 identical
(`ae53464bf3be2580`), KV cache identical to the digit (218,625 tokens across all
three launches). Quantization moved *downstream* of the gate.
See [[2026-08-25-refusal-retention-probe]] for the axis his gate did not have.
@@ -0,0 +1,86 @@
# The 8.6% MFU was an accounting artifact — attention on Ampere kernels
`[2026-08-25]`
## The answer
**Real utilisation was 17-20%, inside the honest stock band.** The 8.6% divided
the *intended* (windowed) FLOPs by the wall time the *dense* reality took.
nominal billed 27.1 TFLOPS x 34.85 s = 9.4e14 FLOP
dense-sliding extra 25 layers, 2 seqs, 4 passes = +8.2e14
padded full layers lose the causal skip = +3.5e14
work performed ~ 1.8e15 = 51-61 TFLOPS
The card was doing ~2x the arithmetic the architecture specifies, and the excess
was the sliding window being computed and thrown away.
## Two independent methods agreed
scaling fit (3 points, 2 params, residuals <3ms over 8x range)
A = 6.87e-4 s/token B = 8.85e-8 s/token^2
quadratic share: 20.9% @ w=2048 -> 67.8% @ w=16384
kernel table (device rows only)
attention 22,835.8 ms 65.2% fmha_cutlass*_sm80
dense GEMM 2,774.0 ms 7.9%
other 5,739.0 ms 16.4%
**67.8% vs 65.2% — 2.6 points apart, no shared assumptions.** The two-term fit
needed no constant term, which refutes launch-bound outright (3,840 expert-GEMM
launches per forward are not the cost).
## The mechanism, source-verified by brokkr's panel (arm: Bil)
masking_utils.py:292-301 _ignore_causal_mask_sdpa requires
kv_length < local_attention_size. 16384 >= 1024,
so THE SLIDING MASK ALWAYS MATERIALISES.
sdp_utils_cpp.h:259-267 flash rejects ANY explicit mask
sdp_utils.cpp:647 cuDNN head_dim capped at 128 -> unreachable
Context.h:480-485 prefer-cuDNN needs major 9 or 10; sm_120 is 12
⚠ **The kernels are `sm80` — Ampere-generation CUTLASS on a Blackwell card**,
with the forward on `gmem`, the memory-efficient backend's slowest fallback tier.
## What actually fixed it
**Bucketing (bucket-to-pair, shuffle-to-mix)** — 29.9% padding -> 0.0%, and
78.3% of micro-batches become exactly zero-pad, which puts the 5 global layers
back on `is_causal`. Measured: padding costs **9.4% MORE time for 24% LESS work**
at fixed width, because an explicit mask knocks those layers off the fast path.
⚠ **Bucket size is NOT a diversity knob.** Swept across a 256x range, roots per
accumulation window stayed flat at 3.54-3.61. The global micro-batch shuffle does
all the mixing; the bucket only costs padding. Use the tightest bucket.
**flex_attention** — Triton-generated so it compiles for sm_120 instead of
shipping sm_80 binaries. 21.7x on sliding layers, 2.1x on global. Needs mandatory
`kernel_options` at 32x32 blocks: 64x32 needs 102,400 bytes against a
**101,376-byte hardware ceiling** — misses by 1 KB, and Triton is already opting
into the full 99 KB, so it is the card, not a default.
## ⚠⚠ The trap that produced TWO wrong published conclusions
`torch._dynamo` defaults to a recompile ceiling of **8**. Every distinct sequence
width is a new shape. On hitting the ceiling dynamo does not error — it silently
falls back to UNCOMPILED flex, which is ~20x slower AND documented to *"not work
with the backwards pass and may produce incorrect results."*
That artifact produced a bogus **0.76x slowdown** and a bogus **2.9% loss
divergence**, and I believed and reported both. Raising the limit to 256 flipped
the speed result to 1.41x.
The loss divergence turned out to be real but benign — adjudicated against fp32
MATH ground truth, both backends sit ~2e-3 from truth with flex fractionally
CLOSER at every width. **Do not re-open it by comparing the two backends to each
other; that cannot answer it. Compare to fp32.**
## Process lesson
brokkr's panel produced **four self-retractions in ninety minutes**. Every
retraction was a derivation; every survivor was a measurement. And the whole
head_dim-512 SDP problem was **already documented in zerofata's published Axolotl
config since April** — the right first stop for "why is this architecture slow"
is practitioner configs for that exact base, before any panel.
Playbook: `docs/pfi/training-throughput-playbook.md`, commit `7b5fd91`.
@@ -0,0 +1,84 @@
# NVFP4A16 serving pipeline — built, validated, and the MoE landmine it found
`[2026-08-25]`
Pipeline at `scripts/erp-tune-serve/` (commits `6a85829`, `ab980e9`).
Validated end-to-end against checkpoint-100 before the real adapter existed.
## ⚠⚠ THE LANDMINE: a `targets=["Linear"]` recipe misses EVERY MoE expert
before linearize_moe: 427 Linears, 205 targeted, experts 0
after linearize_moe: 11,947 Linears, 11,725 targeted, experts 11,520
(30 layers x 128 experts x 3 projections)
Gemma-4 stores each layer's 128 experts as two fused 3-D `nn.Parameter` tensors
(`gate_up_proj` [128,1408,2816], `down_proj` [128,2816,704]) — note the absent
`.weight` suffix. A Linear-targeting recipe resolves 205 of 427 modules and
**zero experts**, leaving 22.84 B params (88.5% of the model) in BF16 with no
warning.
**This is the same defect that killed QLoRA here via bitsandbytes.** The blind
spot is in the *checkpoint layout*, not the tool. Fix:
`llmcompressor.modeling.moe.linearize.linearize_moe` — no registration needed,
Gemma-4 satisfies `FusedExpertsProtocol` structurally. Playbook §3.15.
## Scheme: NVFP4A16, deviating from the playbook default, on measured grounds
brokkr benched the W4A4 quant of this checkpoint at **12% on contradiction
detection with CoT off against gen's 81%** — the signature of 4-bit input
activations on a reasoning-dense task. Plus W4A4 KLD is 2-4x worse past ~10k ctx
on sm_120. This is a 16,384-ctx RP seat. Marlin's prefill cost accepted.
⚠ Several HF repos named `…-NVFP4A16` declare `input_activations num_bits 4` —
W4A4 wearing an A16 label. The script refuses if the emitted config says 4.
## Four silent defects the dry run found
1. **transformers 5.15 MIGRATES the config schema on save** — drops
`global_head_dim`/`num_global_key_value_heads`, writes `per_layer_config`.
transformers 5.10 (the llmcompressor venv) then reads `num_key_value_heads`
as None and dies with `TypeError: unsupported operand type(s) for //`.
Every working artifact on the box uses the OLD schema. Merge now downgrades it.
2. **llmcompressor cannot auto-init a processor for a multimodal checkpoint** —
pass the tokenizer explicitly as `processor`.
3. **`save_pretrained` does not carry `processor_config.json`** — vLLM then fails
with "Can't load feature extractor", which reads as a vision bug.
4. **The quant needs more than GPU1's free 32 GiB.** `quant_with_gen_down.sh`
stops `vllm-gen` and restores it from a trap on EVERY exit path, using
`docker start` not `compose up` so the container returns with its exact config.
## Verified on the emitted artifact
49 GB -> 17 GB, format nvfp4-pack-quantized, a=null (genuine A16)
weight_packed 11,725 of which expert 11,520
tokenizer truncation: clean (§3.14 trap avoided by calibrating on the
encode cache, so the tokenizer is never called
with truncation=True at all)
served: Marlin NVFP4 kernel + Marlin MoE backend, coherent generation
⚠ The reference `nvfp4a16` artifact triggers a vLLM warning that q/k/v carry
*different* weight global scales ("likely reduced accuracy"). **Ours does not** —
llmcompressor 0.12 links weight observers across fused groups automatically. The
in-house quant is better than the downloaded one on that axis.
## ⚠ MERGED WEIGHTS ARE MANDATORY — and not for the reason we assumed
The open question was whether LoRA-on-NVFP4 hot-swap still silently no-ops.
Retested on `vllm/vllm-openai:latest`: **it refuses to start.**
AttributeError: To support LoRA for MoE model,
'get_expert_mapping' must be implemented
The check is in `vllm/lora/utils.py::process_packed_modules_mapping` and branches
on `is_moe_model()` — **quantization is not in the condition.** `gemma4.py`,
`gemma4_mm.py`, `gemma4_mtp.py`, `gemma4_unified.py` all have ZERO occurrences;
`deepseek_v2`, `mixtral`, `glm4_moe`, `ernie45_moe` implement it.
**vLLM cannot serve a LoRA on ANY Gemma-4, bf16 or quantized.** Merging is the
only path for this architecture, and it would have bitten identically on the
unquantized base. A loud refusal is strictly better than the 0.24.0 silent no-op,
which shipped a base model wearing the tune's name.
⚠ Base-viability pre-flight is now playbook §3.11 — three greps before picking a
base. **Grep the CLASS, not the file**: `mistral.py` greps as `SupportsLoRA=0`
and is fully LoRA-capable via inheritance from `LlamaForCausalLM`.
@@ -0,0 +1,64 @@
# Refusal retention — the axis the gate did not have, and the axis I measured wrong
`[2026-08-25]`
## Why it exists
brokkr's gate measures reasoning (T1-T6), craft (diversity/attractor) and
regurgitation (memorisation). **Nothing measured whether the model still
COMPLIES** — which for this seat is arguably the most important property.
The risk is specific to our operation order. We do **tune(abliterate(stock))**,
so the tune has 57.7M tokens of opportunity to walk the abliteration back. *A
tune that gains 41 items of contradiction detection and quietly re-installs
refusals is a failed seat that passes the entire gate.*
## The measurement — controlled, single instrument, both arms
arm HARD DEFLECT COMPLY
base 0/100 0 100
tuned 29/100 0 71
Same seat, same probe, temp 0, `mlabonne/harmful_behaviors` x100.
Probe: `scripts/training-probes/refusal_probe.py`.
**The tune added 29 general-harm refusals where the base had none.**
Two things fell out:
- **The instrument validates.** Base measured 0/100 on my generated-text regex
against Heretic's recorded 3/100 from a first-token-probability scorer. 0 vs 3
is agreement — the incomparability worry was right caution about a non-problem.
- **DEFLECT is 0 on BOTH arms, so the free control fires.** An instrument
artifact does not care which arm it runs against. Both zero means the model is
**binary** — refuses in refusal-language or engages, no soft-deflection tail.
The R19 undercount does not apply here.
## ⚠⚠ But it is the WRONG AXIS — brokkr's catch, and it is the better one
`mlabonne/harmful_behaviors` is **general harm** — weapons, malware, fraud. **The
abliteration was not run so the model would explain bomb-making. It was run so
the model would engage with explicit fiction.** Different refusal surfaces; a
model moves on them independently.
I picked that set because it was cached, had a recorded baseline, and was what
the abliteration tool used. **Every one of those is a reason it was convenient,
not a reason it was right** — and "it has a baseline" was actively misleading,
because a comparable number for a question nobody is asking looks like evidence.
**29/100 general-harm refusals on a seat writing prose the operator was actively
praising is plausibly the DESIRED shape**, not a defect. General-harm refusals
returning while domain compliance holds is close to ideal for an internal
creative seat. I would have reported it as damage.
**The load-bearing cell is COMPLY 71, not the 29.** Stock refused 100/100;
anything near that would mean the abliteration was undone. 71 complying means
"partially walked back on one axis" — a different finding, and only one of the
two threatens the seat.
Domain-compliance probe (the right axis, from R19's track-2 map) is brokkr's,
pending. Scaffold supplied: `scripts/training-probes/counted_classifier.py`
(`2a05ae9`) — classify-never-surface, three-way, ERROR path deliberately does not
log the exception body because an exception can echo the prompt back.
Playbook §3.13. See [[2026-08-25-erp-tune-run2-complete]].
@@ -0,0 +1,123 @@
# Worldtree b188 + b189 bridge cutover, and the selene metadata that lied
`[2026-08-25]`
Three arcs in one day, all infra-ops side, all landed.
## b188 — matrix.yaml pre-sync (#406/#409/#410 closed)
From b188 the bridge reads per-agent `rendering` + `ambient_buffer_size` from
`config/matrix.yaml` ONLY; agent `config.yaml` matrix blocks are gone from the
image. Staged as `6417115` in `worldtree-instance-configs`, deployed to both
instances with operator approval.
- mimir gets thinking-to-thread + tool-call reactions + 7 reaction labels;
forseti and lofn stay quiet.
- **Rider #409 pruned six dead agents** (bragi, leif, troi, soong, cara, glados)
from BOTH rosters — originally scoped personal-only, which I flagged as a
possible oversight and it was. **The settling fact worth keeping: the engine
roster comes from the image's baked `agents/` directory (only `config/` is
bind-mounted), and both instances run the same image**, so instance-level
evidence about which agents the engine lists generalises by construction.
- Both rosters now exactly `[mimir, forseti, lofn]` — the three engine agents
actually bridged. mask/vili/echo exist in the engine, deliberately unbridged
(operator ruling).
⚠ **Edited text-surgically, not via a yaml round-trip** — PyYAML would reflow
1,249 lines and drop every comment, and the comments are the documentation.
⚠ **`deploy-wt-config` uses `docker restart`, NOT `compose up`.** A `compose up`
on corviduo-dev re-resolves the image tag and can silently swap the running
build — which would turn an "inert pre-sync" into an unintended image roll on two
live instances. That property is easy to lose in a future refactor of the script.
## b189 — #407 bridge extracted to its own repo (#404 umbrella closed)
Bridge now `gitea.phasefinal.com/pfi/wt-matrix-bridge`, its own repo, own CI.
⚠ **It publishes to the `pfi` ORG, not `vh`, and the reason is structural:** `vh`
is a **USER**, not an org. Gitea scopes user-namespace packages to the owning
user — there are no package collaborators on a user namespace. **No service
account can ever publish to `gitea.phasefinal.com/vh/*`.** claude-bot is an Owner
of `pfi`, so that is where it goes. Token `wt-matrix-bridge-ci` (id 28, scopes
`write:package,read:repository`), vaulted at
`nh3-dev/.config/claude-bot/gitea-token-wt-matrix-bridge-ci`.
**Minted a dedicated token rather than reuse `claude-bot-sdk-ops` or `arbo-ci`,
both of which already carry `write:package`** — a shared credential cannot be
revoked without collateral. ⚠ A first mint attempt succeeded then failed to save;
Gitea returns a token value exactly once, so it was unrecoverable. Deleted the
orphan (id 27) rather than leave a live package-write credential on the account.
**Both instances PINNED** to `f3f8ec902267` (`e90f436`), closing the #410 shape:
`WORLDTREE_IMAGE` was pinned and the bridge was the one drifting service.
⚠ The pin moved from `b178285b1cb5` because a cross-frontier bug-hunt found the
M_EXCLUSIVE **fallback could itself litter unboundedly** in exactly the state it
was written for. **The safety net had the same failure mode as the thing it was
catching** — and my staged window leaned on that fallback.
## #411 — the debug-room failure, diagnosed twice and wrong both times first
My theory: the alias was held by orphaned rooms. **Refuted by a directory probe
returning 404.** The real cause: Synapse's **M_EXCLUSIVE** — an appservice may
only create aliases inside a namespace it has RESERVED, and
`aipa_appservice.yaml` had `namespaces.aliases: []`.
⚠ **I inferred a cause from a symptom that was the RESPONSE to the cause** — the
log's "re-resolving alias" line is the recovery path firing, not evidence the
alias exists. One directory probe settled it and I reasoned instead.
Fixed with operator clearance: added
`regex: '#aipa-debug-[a-z0-9_-]*:matrix\.phasefinal\.com'` (exclusive) at
`/opt/docker/conf/synapse/aipa_appservice.yaml` on **ana-docker** (NOT
`/opt/docker/data/`, which worldtree-dev's issue cited). **My regex was tighter
than the proposed `#aipa-debug-.*`**, which fullmatches only because `.` also
matches the `:` separator and would equally claim other homeservers.
Pre-apply sweep (an `exclusive: true` claim can make Synapse refuse to START):
aliases matching '%aipa-debug%' 0
total room_aliases on the homeserver 1 <- why it went unnoticed this long
rooms created by @aipa-debug 17 <- the litter, confirmed unaliased
Synapse healthy in 40s, both bridges rode through. **The aliased create then
worked FIRST TRY on the next personal recreate** — designed path, fallback never
fired, both reuse mechanisms live (canonical-alias rediscovery AND the room-id
cache).
⚠ The #411 writer census **inverted its own premise**: the api (uid 1000) cannot
write `/app/sessions`, the bridge (root) can — the reverse of the issue text. But
worldtree-dev's reconciliation is better than "backwards": pre-#407 the bridge
ran from the ENGINE image as uid 1000, and the cutover changed the answer
underneath the issue. Both readings were true at their timestamps. **No live
writer exists for that path anyway** (`sessions.path` is a legacy default, #330
moved the tools off it), so the chown is optional future-proofing.
## selene-1-mini-8b — a config that lied about what answers
forseti's fleet sweep found the only genuine residual in
`worldtree-instance-configs`, and it was **live on both boxes, not just drifted
in git.** Fixed in `a77639d`.
Routing was never broken (`model: "chat-judge"` stays), but `display_name` said
"Selene 1 Mini 8B" and the description said "Atla Selene 1 Mini 8B — reward model
derived from Llama 3.1 8B" while chat-judge has been backed by
**qwen3.8-27b-uncensored** since 2026-08-23.
⚠ **It misstated the KIND of model, not just the identity.** A reward model and a
generative judge are different instruments; a consumer reading
"pairwise/likert/binary/scalar reward model" would expect scalar-reward semantics
qwen3.8-27b does not provide.
Verified: `selene-1-mini-8b` -> HTTP 400 (by design), `chat-judge` -> HTTP 200.
**NOT changed:** the catalog key and the `selene-judgment` role — `model_roles.yaml`
binds to that key, so renaming is worldtree-dev's schema call. Flagged that a role
named after a retired model defeats the purpose of role aliases.
⚠ forseti's sharpest finding is for the operator: **`~/.claude/CLAUDE.md` line 502
lists the retired name in the global tools roster.** A broadcast reaches sessions
that already exist; the roster line keeps minting new ones. His file, awaiting his
word. Also: **there is no fan-out primitive on the bus** — a fleet relay is 73
individual posts, recorded as a real gap rather than papered over.