memory: snapshot — the tune is trained, gated, and serving
Run-01 completed in 7:21:52 (47% faster than the 13.85h round-1 projection),
lora_B gate 205/205 non-zero at median norm 1.708, and the acceptance gate says
it did the thing it was built for: diversity +0.178 against a 0.008 floor (22x),
attractor hit rate -11.3pt against a 2.0pt floor, memorisation 0.0000 on both
arms — which closes the R20 licensed-prose exposure on measurement rather than
argument.
Five new detail files carry the substance:
erp-tune-run2-complete the run, the gate, the noise-floor near-miss
(brokkr was one step from reporting a 13-point
T6 regression sitting inside twice his
instrument's own variance)
mfu-root-caused-attention 8.6% MFU was an accounting artifact; real
utilisation 17-20%, cost was attention on
AMPERE kernels. Two independent methods agreed
to 2.6 points.
nvfp4-serving-pipeline merged weights are MANDATORY — vLLM cannot
serve a LoRA on ANY Gemma-4 — plus the recipe
that silently misses all 11,520 expert tensors
refusal-retention-probe measured base 0/100 -> tuned 29/100, then had
to accept it was the wrong axis
worldtree-b188-b189-and-selene three arcs closed, and a #411 diagnosis I got
wrong twice before a directory probe settled it
Current state rewritten end to end — the previous snapshot had the run in
flight at ~17h with MFU unexplained. Both are now closed.
The open operator decision is run 2's base, deliberately unstaged and flagged
against being filed as a config knob: it is a reversal of the trainee-selection
decision, and the pretrained-base option removes the last non-lexical floor on
the CSAM axis given stage-2-detector-inert and contamination-scan-absent are
both already overridden.
Tried-and-abandoned gains four measured-dead throughput levers, the packing
correction (bucketing wins under sdpa and the conclusion flips under flex — do
not carry it past the backend decision), and the merge-back-undoes-abliteration
trap brokkr caught in his own advice.
Index stays at 291 lines, under the soft cap. No archival this run.
This commit is contained in:
@@ -0,0 +1,92 @@
|
||||
# ERP/RP tune run-01 COMPLETE — 7.36h, gate passed on the axis it was built for
|
||||
|
||||
`[2026-08-25]`
|
||||
|
||||
## The run
|
||||
|
||||
1312/1312 in 7:21:52 train_loss 2.793 epoch 1.0
|
||||
20.1 s/it FLAT across every 100-step window (round 1: 35-46.5 s/it)
|
||||
adapter: /tank/erp-tune/run-01/adapter/ 410 tensors, provenance.json
|
||||
|
||||
**47% faster than the round-1 projection of 13.85h**, from two changes: the
|
||||
bucketed sampler and flex attention. Rate was flat — 19.7 / 19.8 / 20.4 / 20.3
|
||||
across the four 100-step windows — which means the 35-46.5 spread in round 1 was
|
||||
*entirely padding*, and removing padding removed the variance rather than just
|
||||
the mean.
|
||||
|
||||
⚠ **I quoted three different ETAs (6.9h, 8h, 7.3h) before I started using a
|
||||
rolling average.** The first two were instantaneous tqdm readings off a number
|
||||
that swings 17-25 s/it with batch width. Only the rolling rate was honest. Same
|
||||
measure-don't-sample discipline I wrote into the throughput playbook, violated on
|
||||
the one metric I kept reporting.
|
||||
|
||||
## lora_B gate — PASSED, twice
|
||||
|
||||
checkpoint-100 205/205 non-zero, median norm 0.829
|
||||
final adapter 205/205 non-zero, median norm 1.708
|
||||
vision_tower tensors: 0 on both
|
||||
|
||||
Median norm rising 0.829 -> 1.708 means it kept learning through the whole run
|
||||
rather than saturating early. This check **never ran in round 1** (died at step
|
||||
19, first checkpoint was 100) and it is the only failure mode that stays
|
||||
invisible until the acceptance gate reports base-identical numbers.
|
||||
|
||||
## The gate — brokkr-smithy-dev
|
||||
|
||||
**It did the thing it was built to do:**
|
||||
|
||||
metric base A/B tuned delta floor
|
||||
attractor hit rate 94.8% / 96.8% 84.5% -11.3pt 2.0pt
|
||||
diversity (pairwise) 0.213 / 0.221 0.3948 +0.178 0.008
|
||||
|
||||
Diversity moved **22x its own noise floor**. Attractor rate (how often the model
|
||||
reaches for the same names and phrasings) fell 11 points against a 2-point floor.
|
||||
|
||||
T1 100 · T2 95 · T3 96-97 · T4 98 · T5 100 · T6 81-82 · core ~94.2
|
||||
memorisation: 0.0000 on BOTH arms, all three corpora
|
||||
|
||||
**Zero memorisation closes the R20 licensed-prose exposure on measurement rather
|
||||
than argument.**
|
||||
|
||||
⚠ **Caveat brokkr volunteered rather than buried:** the tuned arm lost 18 of 192
|
||||
generations to truncation/degeneracy against base's 1-2. Lopsided exclusions
|
||||
plausibly flatter the diversity magnitude. Direction is unambiguous at 22x floor;
|
||||
the number carries an asterisk.
|
||||
|
||||
## The noise-floor near-miss — the methodology lesson
|
||||
|
||||
brokkr was one step from reporting a 13-point T6 regression **that sat inside
|
||||
twice his instrument's own variance.**
|
||||
|
||||
--per-type 32 max swing across tasks: 9 points
|
||||
--per-type 128 max swing across tasks: 1 point
|
||||
|
||||
His gate criterion is "no task regresses by more than one item" = 3.1 points at
|
||||
n=32. **The instrument's own run-to-run noise was 3 items.** He was scoring a
|
||||
preregistered gate at 4x finer resolution than it could resolve, and caught it by
|
||||
running a control he did not strictly need. Quadrupling n collapsed the noise
|
||||
exactly as binomial statistics predicts.
|
||||
|
||||
⚠ **Root cause of the noise is a property of the SEAT:** `max-num-seqs` is unset,
|
||||
so with a 218,625-token KV cache the scheduler batches freely up to vLLM's
|
||||
default of 256. Continuous batching changes reduction order and borderline items
|
||||
flip. Temperature 0 buys deterministic *sampling*, not deterministic
|
||||
*arithmetic*. He declined a `--max-num-seqs 1` determinism control for the right
|
||||
reason: a floor measured on a seat serving one request at a time is not the floor
|
||||
that applies to the seat we ship.
|
||||
|
||||
## The confound I built and he caught
|
||||
|
||||
I optimised a pipeline for production and then handed him its output as an eval
|
||||
instrument **without asking whether those were the same job.** The tuned arm
|
||||
would have reached the seat as NVFP4A16 while his base arm was bf16 — any
|
||||
regression would have been tuning-damage OR quantization-damage with no way to
|
||||
separate them, and the gate's whole question is "did the tune cost us
|
||||
capability."
|
||||
|
||||
**Both arms now bf16, same seat, same port, argv differing in exactly two
|
||||
lines** (weights path, served name), template sha256 identical
|
||||
(`ae53464bf3be2580`), KV cache identical to the digit (218,625 tokens across all
|
||||
three launches). Quantization moved *downstream* of the gate.
|
||||
|
||||
See [[2026-08-25-refusal-retention-probe]] for the axis his gate did not have.
|
||||
@@ -0,0 +1,86 @@
|
||||
# The 8.6% MFU was an accounting artifact — attention on Ampere kernels
|
||||
|
||||
`[2026-08-25]`
|
||||
|
||||
## The answer
|
||||
|
||||
**Real utilisation was 17-20%, inside the honest stock band.** The 8.6% divided
|
||||
the *intended* (windowed) FLOPs by the wall time the *dense* reality took.
|
||||
|
||||
nominal billed 27.1 TFLOPS x 34.85 s = 9.4e14 FLOP
|
||||
dense-sliding extra 25 layers, 2 seqs, 4 passes = +8.2e14
|
||||
padded full layers lose the causal skip = +3.5e14
|
||||
work performed ~ 1.8e15 = 51-61 TFLOPS
|
||||
|
||||
The card was doing ~2x the arithmetic the architecture specifies, and the excess
|
||||
was the sliding window being computed and thrown away.
|
||||
|
||||
## Two independent methods agreed
|
||||
|
||||
scaling fit (3 points, 2 params, residuals <3ms over 8x range)
|
||||
A = 6.87e-4 s/token B = 8.85e-8 s/token^2
|
||||
quadratic share: 20.9% @ w=2048 -> 67.8% @ w=16384
|
||||
|
||||
kernel table (device rows only)
|
||||
attention 22,835.8 ms 65.2% fmha_cutlass*_sm80
|
||||
dense GEMM 2,774.0 ms 7.9%
|
||||
other 5,739.0 ms 16.4%
|
||||
|
||||
**67.8% vs 65.2% — 2.6 points apart, no shared assumptions.** The two-term fit
|
||||
needed no constant term, which refutes launch-bound outright (3,840 expert-GEMM
|
||||
launches per forward are not the cost).
|
||||
|
||||
## The mechanism, source-verified by brokkr's panel (arm: Bil)
|
||||
|
||||
masking_utils.py:292-301 _ignore_causal_mask_sdpa requires
|
||||
kv_length < local_attention_size. 16384 >= 1024,
|
||||
so THE SLIDING MASK ALWAYS MATERIALISES.
|
||||
sdp_utils_cpp.h:259-267 flash rejects ANY explicit mask
|
||||
sdp_utils.cpp:647 cuDNN head_dim capped at 128 -> unreachable
|
||||
Context.h:480-485 prefer-cuDNN needs major 9 or 10; sm_120 is 12
|
||||
|
||||
⚠ **The kernels are `sm80` — Ampere-generation CUTLASS on a Blackwell card**,
|
||||
with the forward on `gmem`, the memory-efficient backend's slowest fallback tier.
|
||||
|
||||
## What actually fixed it
|
||||
|
||||
**Bucketing (bucket-to-pair, shuffle-to-mix)** — 29.9% padding -> 0.0%, and
|
||||
78.3% of micro-batches become exactly zero-pad, which puts the 5 global layers
|
||||
back on `is_causal`. Measured: padding costs **9.4% MORE time for 24% LESS work**
|
||||
at fixed width, because an explicit mask knocks those layers off the fast path.
|
||||
|
||||
⚠ **Bucket size is NOT a diversity knob.** Swept across a 256x range, roots per
|
||||
accumulation window stayed flat at 3.54-3.61. The global micro-batch shuffle does
|
||||
all the mixing; the bucket only costs padding. Use the tightest bucket.
|
||||
|
||||
**flex_attention** — Triton-generated so it compiles for sm_120 instead of
|
||||
shipping sm_80 binaries. 21.7x on sliding layers, 2.1x on global. Needs mandatory
|
||||
`kernel_options` at 32x32 blocks: 64x32 needs 102,400 bytes against a
|
||||
**101,376-byte hardware ceiling** — misses by 1 KB, and Triton is already opting
|
||||
into the full 99 KB, so it is the card, not a default.
|
||||
|
||||
## ⚠⚠ The trap that produced TWO wrong published conclusions
|
||||
|
||||
`torch._dynamo` defaults to a recompile ceiling of **8**. Every distinct sequence
|
||||
width is a new shape. On hitting the ceiling dynamo does not error — it silently
|
||||
falls back to UNCOMPILED flex, which is ~20x slower AND documented to *"not work
|
||||
with the backwards pass and may produce incorrect results."*
|
||||
|
||||
That artifact produced a bogus **0.76x slowdown** and a bogus **2.9% loss
|
||||
divergence**, and I believed and reported both. Raising the limit to 256 flipped
|
||||
the speed result to 1.41x.
|
||||
|
||||
The loss divergence turned out to be real but benign — adjudicated against fp32
|
||||
MATH ground truth, both backends sit ~2e-3 from truth with flex fractionally
|
||||
CLOSER at every width. **Do not re-open it by comparing the two backends to each
|
||||
other; that cannot answer it. Compare to fp32.**
|
||||
|
||||
## Process lesson
|
||||
|
||||
brokkr's panel produced **four self-retractions in ninety minutes**. Every
|
||||
retraction was a derivation; every survivor was a measurement. And the whole
|
||||
head_dim-512 SDP problem was **already documented in zerofata's published Axolotl
|
||||
config since April** — the right first stop for "why is this architecture slow"
|
||||
is practitioner configs for that exact base, before any panel.
|
||||
|
||||
Playbook: `docs/pfi/training-throughput-playbook.md`, commit `7b5fd91`.
|
||||
@@ -0,0 +1,84 @@
|
||||
# NVFP4A16 serving pipeline — built, validated, and the MoE landmine it found
|
||||
|
||||
`[2026-08-25]`
|
||||
|
||||
Pipeline at `scripts/erp-tune-serve/` (commits `6a85829`, `ab980e9`).
|
||||
Validated end-to-end against checkpoint-100 before the real adapter existed.
|
||||
|
||||
## ⚠⚠ THE LANDMINE: a `targets=["Linear"]` recipe misses EVERY MoE expert
|
||||
|
||||
before linearize_moe: 427 Linears, 205 targeted, experts 0
|
||||
after linearize_moe: 11,947 Linears, 11,725 targeted, experts 11,520
|
||||
(30 layers x 128 experts x 3 projections)
|
||||
|
||||
Gemma-4 stores each layer's 128 experts as two fused 3-D `nn.Parameter` tensors
|
||||
(`gate_up_proj` [128,1408,2816], `down_proj` [128,2816,704]) — note the absent
|
||||
`.weight` suffix. A Linear-targeting recipe resolves 205 of 427 modules and
|
||||
**zero experts**, leaving 22.84 B params (88.5% of the model) in BF16 with no
|
||||
warning.
|
||||
|
||||
**This is the same defect that killed QLoRA here via bitsandbytes.** The blind
|
||||
spot is in the *checkpoint layout*, not the tool. Fix:
|
||||
`llmcompressor.modeling.moe.linearize.linearize_moe` — no registration needed,
|
||||
Gemma-4 satisfies `FusedExpertsProtocol` structurally. Playbook §3.15.
|
||||
|
||||
## Scheme: NVFP4A16, deviating from the playbook default, on measured grounds
|
||||
|
||||
brokkr benched the W4A4 quant of this checkpoint at **12% on contradiction
|
||||
detection with CoT off against gen's 81%** — the signature of 4-bit input
|
||||
activations on a reasoning-dense task. Plus W4A4 KLD is 2-4x worse past ~10k ctx
|
||||
on sm_120. This is a 16,384-ctx RP seat. Marlin's prefill cost accepted.
|
||||
|
||||
⚠ Several HF repos named `…-NVFP4A16` declare `input_activations num_bits 4` —
|
||||
W4A4 wearing an A16 label. The script refuses if the emitted config says 4.
|
||||
|
||||
## Four silent defects the dry run found
|
||||
|
||||
1. **transformers 5.15 MIGRATES the config schema on save** — drops
|
||||
`global_head_dim`/`num_global_key_value_heads`, writes `per_layer_config`.
|
||||
transformers 5.10 (the llmcompressor venv) then reads `num_key_value_heads`
|
||||
as None and dies with `TypeError: unsupported operand type(s) for //`.
|
||||
Every working artifact on the box uses the OLD schema. Merge now downgrades it.
|
||||
2. **llmcompressor cannot auto-init a processor for a multimodal checkpoint** —
|
||||
pass the tokenizer explicitly as `processor`.
|
||||
3. **`save_pretrained` does not carry `processor_config.json`** — vLLM then fails
|
||||
with "Can't load feature extractor", which reads as a vision bug.
|
||||
4. **The quant needs more than GPU1's free 32 GiB.** `quant_with_gen_down.sh`
|
||||
stops `vllm-gen` and restores it from a trap on EVERY exit path, using
|
||||
`docker start` not `compose up` so the container returns with its exact config.
|
||||
|
||||
## Verified on the emitted artifact
|
||||
|
||||
49 GB -> 17 GB, format nvfp4-pack-quantized, a=null (genuine A16)
|
||||
weight_packed 11,725 of which expert 11,520
|
||||
tokenizer truncation: clean (§3.14 trap avoided by calibrating on the
|
||||
encode cache, so the tokenizer is never called
|
||||
with truncation=True at all)
|
||||
served: Marlin NVFP4 kernel + Marlin MoE backend, coherent generation
|
||||
|
||||
⚠ The reference `nvfp4a16` artifact triggers a vLLM warning that q/k/v carry
|
||||
*different* weight global scales ("likely reduced accuracy"). **Ours does not** —
|
||||
llmcompressor 0.12 links weight observers across fused groups automatically. The
|
||||
in-house quant is better than the downloaded one on that axis.
|
||||
|
||||
## ⚠ MERGED WEIGHTS ARE MANDATORY — and not for the reason we assumed
|
||||
|
||||
The open question was whether LoRA-on-NVFP4 hot-swap still silently no-ops.
|
||||
Retested on `vllm/vllm-openai:latest`: **it refuses to start.**
|
||||
|
||||
AttributeError: To support LoRA for MoE model,
|
||||
'get_expert_mapping' must be implemented
|
||||
|
||||
The check is in `vllm/lora/utils.py::process_packed_modules_mapping` and branches
|
||||
on `is_moe_model()` — **quantization is not in the condition.** `gemma4.py`,
|
||||
`gemma4_mm.py`, `gemma4_mtp.py`, `gemma4_unified.py` all have ZERO occurrences;
|
||||
`deepseek_v2`, `mixtral`, `glm4_moe`, `ernie45_moe` implement it.
|
||||
|
||||
**vLLM cannot serve a LoRA on ANY Gemma-4, bf16 or quantized.** Merging is the
|
||||
only path for this architecture, and it would have bitten identically on the
|
||||
unquantized base. A loud refusal is strictly better than the 0.24.0 silent no-op,
|
||||
which shipped a base model wearing the tune's name.
|
||||
|
||||
⚠ Base-viability pre-flight is now playbook §3.11 — three greps before picking a
|
||||
base. **Grep the CLASS, not the file**: `mistral.py` greps as `SupportsLoRA=0`
|
||||
and is fully LoRA-capable via inheritance from `LlamaForCausalLM`.
|
||||
@@ -0,0 +1,64 @@
|
||||
# Refusal retention — the axis the gate did not have, and the axis I measured wrong
|
||||
|
||||
`[2026-08-25]`
|
||||
|
||||
## Why it exists
|
||||
|
||||
brokkr's gate measures reasoning (T1-T6), craft (diversity/attractor) and
|
||||
regurgitation (memorisation). **Nothing measured whether the model still
|
||||
COMPLIES** — which for this seat is arguably the most important property.
|
||||
|
||||
The risk is specific to our operation order. We do **tune(abliterate(stock))**,
|
||||
so the tune has 57.7M tokens of opportunity to walk the abliteration back. *A
|
||||
tune that gains 41 items of contradiction detection and quietly re-installs
|
||||
refusals is a failed seat that passes the entire gate.*
|
||||
|
||||
## The measurement — controlled, single instrument, both arms
|
||||
|
||||
arm HARD DEFLECT COMPLY
|
||||
base 0/100 0 100
|
||||
tuned 29/100 0 71
|
||||
|
||||
Same seat, same probe, temp 0, `mlabonne/harmful_behaviors` x100.
|
||||
Probe: `scripts/training-probes/refusal_probe.py`.
|
||||
|
||||
**The tune added 29 general-harm refusals where the base had none.**
|
||||
|
||||
Two things fell out:
|
||||
|
||||
- **The instrument validates.** Base measured 0/100 on my generated-text regex
|
||||
against Heretic's recorded 3/100 from a first-token-probability scorer. 0 vs 3
|
||||
is agreement — the incomparability worry was right caution about a non-problem.
|
||||
- **DEFLECT is 0 on BOTH arms, so the free control fires.** An instrument
|
||||
artifact does not care which arm it runs against. Both zero means the model is
|
||||
**binary** — refuses in refusal-language or engages, no soft-deflection tail.
|
||||
The R19 undercount does not apply here.
|
||||
|
||||
## ⚠⚠ But it is the WRONG AXIS — brokkr's catch, and it is the better one
|
||||
|
||||
`mlabonne/harmful_behaviors` is **general harm** — weapons, malware, fraud. **The
|
||||
abliteration was not run so the model would explain bomb-making. It was run so
|
||||
the model would engage with explicit fiction.** Different refusal surfaces; a
|
||||
model moves on them independently.
|
||||
|
||||
I picked that set because it was cached, had a recorded baseline, and was what
|
||||
the abliteration tool used. **Every one of those is a reason it was convenient,
|
||||
not a reason it was right** — and "it has a baseline" was actively misleading,
|
||||
because a comparable number for a question nobody is asking looks like evidence.
|
||||
|
||||
**29/100 general-harm refusals on a seat writing prose the operator was actively
|
||||
praising is plausibly the DESIRED shape**, not a defect. General-harm refusals
|
||||
returning while domain compliance holds is close to ideal for an internal
|
||||
creative seat. I would have reported it as damage.
|
||||
|
||||
**The load-bearing cell is COMPLY 71, not the 29.** Stock refused 100/100;
|
||||
anything near that would mean the abliteration was undone. 71 complying means
|
||||
"partially walked back on one axis" — a different finding, and only one of the
|
||||
two threatens the seat.
|
||||
|
||||
Domain-compliance probe (the right axis, from R19's track-2 map) is brokkr's,
|
||||
pending. Scaffold supplied: `scripts/training-probes/counted_classifier.py`
|
||||
(`2a05ae9`) — classify-never-surface, three-way, ERROR path deliberately does not
|
||||
log the exception body because an exception can echo the prompt back.
|
||||
|
||||
Playbook §3.13. See [[2026-08-25-erp-tune-run2-complete]].
|
||||
@@ -0,0 +1,123 @@
|
||||
# Worldtree b188 + b189 bridge cutover, and the selene metadata that lied
|
||||
|
||||
`[2026-08-25]`
|
||||
|
||||
Three arcs in one day, all infra-ops side, all landed.
|
||||
|
||||
## b188 — matrix.yaml pre-sync (#406/#409/#410 closed)
|
||||
|
||||
From b188 the bridge reads per-agent `rendering` + `ambient_buffer_size` from
|
||||
`config/matrix.yaml` ONLY; agent `config.yaml` matrix blocks are gone from the
|
||||
image. Staged as `6417115` in `worldtree-instance-configs`, deployed to both
|
||||
instances with operator approval.
|
||||
|
||||
- mimir gets thinking-to-thread + tool-call reactions + 7 reaction labels;
|
||||
forseti and lofn stay quiet.
|
||||
- **Rider #409 pruned six dead agents** (bragi, leif, troi, soong, cara, glados)
|
||||
from BOTH rosters — originally scoped personal-only, which I flagged as a
|
||||
possible oversight and it was. **The settling fact worth keeping: the engine
|
||||
roster comes from the image's baked `agents/` directory (only `config/` is
|
||||
bind-mounted), and both instances run the same image**, so instance-level
|
||||
evidence about which agents the engine lists generalises by construction.
|
||||
- Both rosters now exactly `[mimir, forseti, lofn]` — the three engine agents
|
||||
actually bridged. mask/vili/echo exist in the engine, deliberately unbridged
|
||||
(operator ruling).
|
||||
|
||||
⚠ **Edited text-surgically, not via a yaml round-trip** — PyYAML would reflow
|
||||
1,249 lines and drop every comment, and the comments are the documentation.
|
||||
|
||||
⚠ **`deploy-wt-config` uses `docker restart`, NOT `compose up`.** A `compose up`
|
||||
on corviduo-dev re-resolves the image tag and can silently swap the running
|
||||
build — which would turn an "inert pre-sync" into an unintended image roll on two
|
||||
live instances. That property is easy to lose in a future refactor of the script.
|
||||
|
||||
## b189 — #407 bridge extracted to its own repo (#404 umbrella closed)
|
||||
|
||||
Bridge now `gitea.phasefinal.com/pfi/wt-matrix-bridge`, its own repo, own CI.
|
||||
|
||||
⚠ **It publishes to the `pfi` ORG, not `vh`, and the reason is structural:** `vh`
|
||||
is a **USER**, not an org. Gitea scopes user-namespace packages to the owning
|
||||
user — there are no package collaborators on a user namespace. **No service
|
||||
account can ever publish to `gitea.phasefinal.com/vh/*`.** claude-bot is an Owner
|
||||
of `pfi`, so that is where it goes. Token `wt-matrix-bridge-ci` (id 28, scopes
|
||||
`write:package,read:repository`), vaulted at
|
||||
`nh3-dev/.config/claude-bot/gitea-token-wt-matrix-bridge-ci`.
|
||||
|
||||
**Minted a dedicated token rather than reuse `claude-bot-sdk-ops` or `arbo-ci`,
|
||||
both of which already carry `write:package`** — a shared credential cannot be
|
||||
revoked without collateral. ⚠ A first mint attempt succeeded then failed to save;
|
||||
Gitea returns a token value exactly once, so it was unrecoverable. Deleted the
|
||||
orphan (id 27) rather than leave a live package-write credential on the account.
|
||||
|
||||
**Both instances PINNED** to `f3f8ec902267` (`e90f436`), closing the #410 shape:
|
||||
`WORLDTREE_IMAGE` was pinned and the bridge was the one drifting service.
|
||||
|
||||
⚠ The pin moved from `b178285b1cb5` because a cross-frontier bug-hunt found the
|
||||
M_EXCLUSIVE **fallback could itself litter unboundedly** in exactly the state it
|
||||
was written for. **The safety net had the same failure mode as the thing it was
|
||||
catching** — and my staged window leaned on that fallback.
|
||||
|
||||
## #411 — the debug-room failure, diagnosed twice and wrong both times first
|
||||
|
||||
My theory: the alias was held by orphaned rooms. **Refuted by a directory probe
|
||||
returning 404.** The real cause: Synapse's **M_EXCLUSIVE** — an appservice may
|
||||
only create aliases inside a namespace it has RESERVED, and
|
||||
`aipa_appservice.yaml` had `namespaces.aliases: []`.
|
||||
|
||||
⚠ **I inferred a cause from a symptom that was the RESPONSE to the cause** — the
|
||||
log's "re-resolving alias" line is the recovery path firing, not evidence the
|
||||
alias exists. One directory probe settled it and I reasoned instead.
|
||||
|
||||
Fixed with operator clearance: added
|
||||
`regex: '#aipa-debug-[a-z0-9_-]*:matrix\.phasefinal\.com'` (exclusive) at
|
||||
`/opt/docker/conf/synapse/aipa_appservice.yaml` on **ana-docker** (NOT
|
||||
`/opt/docker/data/`, which worldtree-dev's issue cited). **My regex was tighter
|
||||
than the proposed `#aipa-debug-.*`**, which fullmatches only because `.` also
|
||||
matches the `:` separator and would equally claim other homeservers.
|
||||
|
||||
Pre-apply sweep (an `exclusive: true` claim can make Synapse refuse to START):
|
||||
|
||||
aliases matching '%aipa-debug%' 0
|
||||
total room_aliases on the homeserver 1 <- why it went unnoticed this long
|
||||
rooms created by @aipa-debug 17 <- the litter, confirmed unaliased
|
||||
|
||||
Synapse healthy in 40s, both bridges rode through. **The aliased create then
|
||||
worked FIRST TRY on the next personal recreate** — designed path, fallback never
|
||||
fired, both reuse mechanisms live (canonical-alias rediscovery AND the room-id
|
||||
cache).
|
||||
|
||||
⚠ The #411 writer census **inverted its own premise**: the api (uid 1000) cannot
|
||||
write `/app/sessions`, the bridge (root) can — the reverse of the issue text. But
|
||||
worldtree-dev's reconciliation is better than "backwards": pre-#407 the bridge
|
||||
ran from the ENGINE image as uid 1000, and the cutover changed the answer
|
||||
underneath the issue. Both readings were true at their timestamps. **No live
|
||||
writer exists for that path anyway** (`sessions.path` is a legacy default, #330
|
||||
moved the tools off it), so the chown is optional future-proofing.
|
||||
|
||||
## selene-1-mini-8b — a config that lied about what answers
|
||||
|
||||
forseti's fleet sweep found the only genuine residual in
|
||||
`worldtree-instance-configs`, and it was **live on both boxes, not just drifted
|
||||
in git.** Fixed in `a77639d`.
|
||||
|
||||
Routing was never broken (`model: "chat-judge"` stays), but `display_name` said
|
||||
"Selene 1 Mini 8B" and the description said "Atla Selene 1 Mini 8B — reward model
|
||||
derived from Llama 3.1 8B" while chat-judge has been backed by
|
||||
**qwen3.8-27b-uncensored** since 2026-08-23.
|
||||
|
||||
⚠ **It misstated the KIND of model, not just the identity.** A reward model and a
|
||||
generative judge are different instruments; a consumer reading
|
||||
"pairwise/likert/binary/scalar reward model" would expect scalar-reward semantics
|
||||
qwen3.8-27b does not provide.
|
||||
|
||||
Verified: `selene-1-mini-8b` -> HTTP 400 (by design), `chat-judge` -> HTTP 200.
|
||||
|
||||
**NOT changed:** the catalog key and the `selene-judgment` role — `model_roles.yaml`
|
||||
binds to that key, so renaming is worldtree-dev's schema call. Flagged that a role
|
||||
named after a retired model defeats the purpose of role aliases.
|
||||
|
||||
⚠ forseti's sharpest finding is for the operator: **`~/.claude/CLAUDE.md` line 502
|
||||
lists the retired name in the global tools roster.** A broadcast reaches sessions
|
||||
that already exist; the roster line keeps minting new ones. His file, awaiting his
|
||||
word. Also: **there is no fan-out primitive on the bus** — a fleet relay is 73
|
||||
individual posts, recorded as a real gap rather than papered over.
|
||||
Reference in New Issue
Block a user