The scheme has existed since August as a single line of persistent
memory, which a snapshot then deleted. It is a naming convention
rather than temporal state, so it now lives in docs/pfi as a proper
document, and the memory entry is reduced to a pointer at it. The
document carries the full table, the address structure, the reasoning
about which slots can and cannot hold a name, and the recipe for
applying one to a host.
It also corrects the conclusion the original note ended on. That note
held that these names could never appear on the wire, which is true
of everything UniFi is able to assign but not of what a host can
assign to itself, and the distinction is the whole difference between
a joke and an address.
AdGuard on esh-docker-vm now holds the esh-server name, at
2607:73c0:402:1d02:4411:b105:50:45, where the segment identity and the
IPv4 address are both legible. It is applied by an if-up.d hook that
derives the prefix at runtime rather than hardcoding it, backgrounds
itself with a retry so it cannot stall interface bring-up, and adds
nothing to the existing interface configuration.
This is load-bearing rather than decorative. The gateway advertises an
IPv6 resolver to clients, macOS prefers it over the IPv4 one, and it
previously pointed at an address derived from that host's MAC.
Benchmarked selene against gen on selene's own job: 24 designed judge items
with checkable ground truth, pairwise + absolute modes, 3 repeats, on BOTH a
neutral JSON prompt and Selene's native Atla template. 288 calls, all free
local.
neutral JSON selene 20/24 (83%) gen 23/24 (96%)
native Atla selene 21/24 (88%) gen 22/24 (92%)
gen won on both templates and selene's BEST sat below gen's WORST. Selene was
given its own fine-tuned template as a fairness check; it gained one point,
not the three it needed.
Decisive defect: selene cannot emit "tie" -- 0/2 on both templates, forcing a
winner on every equivalent pair. For eval work that is the case that matters
most. gen returned tie correctly on the JSON template. Selene also compressed
the 1-5 scale (clustered at 2s and 4s) where gen used it fully. Selene's only
win was ~3x latency, unexercised at ~60 calls/day with zero queueing.
TWO NAMES, TWO DIFFERENT TREATMENTS, deliberately:
- chat-judge -> repointed to gen. It is a ROLE alias and ADR-0012 says
consumers bind the capability, not a concrete model. Sampler profile copied
from image-judge (temp 0, top_p 1.0, top_k 1, thinking off) so the served
config matches the benchmarked condition.
- selene-1-mini-8b -> REMOVED. It 404s. It was NOT aliased to gen. A
served-name is a contract about what the model IS; answering it with a
different model hides a material change behind a stable string. Operator
ruling: "never repoint a named model at a different model's endpoint --
that is intentionally misleading." Verified: the gateway now returns
HTTP 400 "Invalid model name" for it.
Reclaimed 17.2 GiB on ana-ml2 GPU 1 (free 1,818 -> 19,450 MiB) on a card that
had under 2 GiB of headroom. gen already runs on GPU 0, so the judge role
moved onto an existing seat rather than allocating anything new.
Canonical litellm config synced from the host; ana-ml2 README and
recommended-model-settings updated. compose.yaml kept for reference, not
deployed.
Measured 2026-08-22 on one target with one instrument: raising MTP
num_speculative_tokens from 3 to 7 improved accepted length from 2.753 to
3.041 per forward pass while throughput fell from 114.9 to 74.0 tok/s.
Reporting acceptance alone would have recommended a 36% regression.
The cause is architectural rather than model-specific. A single-module MTP
head has no depth of its own, so vLLM runs it autoregressively and k draft
tokens cost k sequential forward passes. Past a shallow depth the drafting
cost exceeds what the extra accepted tokens save.
Records the comparison rule that follows: match k when comparing two
speculative methods, or the measurement is of depth rather than method. A
parallel-drafting drafter at k=7 against an autoregressive MTP at k=3 is not
a method comparison. In the case that produced this, the depth control
showed most of the apparent acceptance advantage was depth, while the
throughput advantage was real and came from parallel drafting -- our MTP was
better at position 0 and still lost overall.
Only the measured, model-agnostic result is recorded here. The
DFlash2-specific findings, the hypotheses that remain unproven, and the
wrong turns taken along the way live in
persistent-memory.d/2026-08-22-dflash2-spec-decode.md with explicit
epistemic labels, deliberately kept out of the playbook.
load_calib tokenizes with tok(..., truncation=True, max_length=seqlen). For a
fast tokenizer that mutates the Rust backend's truncation state in place, and
the subsequent tok.save_pretrained() persisted it, so every mixed-NVFP4 build
shipped a tokenizer.json carrying
"truncation": {"direction": "Right", "max_length": 2048, ...}
against a source whose value is null. Every prompt was clamped at the
calibration length, permanently.
It hid because older transformers does not enforce the text-vs-ids count
check. On a newer one the seat dies at startup with a message that names
images and never mentions tokenizers:
ValueError: Mismatch in `image` token count between text and `input_ids`.
Got ids=[2047] and text=[16384].
The cap also silently limited image resolution well before it killed
anything -- at 2048 the largest servable image is about 1448x1448, since
(edge/patch)^2 / merge^2 image tokens have to fit under it.
Fix saves a pristine tokenizer re-read from the source rather than the
mutated calibration object, and then asserts truncation is null so the
defect fails the build instead of shipping again.
Playbook gains section 3.14 with the symptom, the cause, the audit one-liner
and a table of which builds were affected, plus a fourth mandatory post-step.
The transferable lesson is called out: this is the third case of an artifact
carrying config authored against an older transformers that a newer one
begins enforcing, so an image bump is a config-compatibility event rather
than just a version change.
Autonomous overnight run under the operator's full-autonomy grant. End state:
fleet up, gen seat untouched, a new verified pen-test seat serving where fable was.
PPL on the orcarouter gen seat (fable downed to free GPU1 for a nospec probe,
probe torn down after): mean 7.07 / median 5.76, within noise of heresy 6.910 /
5.625 and identical to our recipe's usual 7.059. The gen-seat search is settled.
M.O.G.-SEC: chose Blackfrost-Research/M.O.G.-SEC-27B-1M-CTX-BF16 (rev deede677)
over the pre-made ModelOpt NVFP4, which was disqualified on W4A4 4-bit activations
(the AEON degradation mode, catastrophic on a 1M-context model), zero MTP tensors,
and ModelOpt format. Pulled, format-screened (P(<think>) 1.11e-05, clean), quanted
in-house to mixed NVFP4+FP8 (23.4 GB, MTP + vision preserved), and served in the
retired fable slot.
stacks/mog-sec ana-ml2 GPU1 :8019, KV 418,218 tok / 1.60x @ 262K
aliases mog-sec (non-thinking), mog-sec-reasoning (thinking)
gates surface 6/6, MTP 55.3%, format 0/15 leak, vision 7/3/1,
capability 4/4 (delivers offensive-security content)
Served at native 262K, NOT the card's 1M -- the 1M needs YaRN (absent from the
weights' config) plus the SGLang/DFlash2 path the repo ships a deployment kit for,
neither of which is our vLLM surface. A real 1M seat is a separate SGLang project.
Retired char-rp-reasoning + char-rp-fable (zero traffic, pointed at the downed
fable :8019; now 404 cleanly, not repointed -- a security model is not an RP model).
char-rp (meromero) untouched. Vision preprocessor built from the model's own
image_processor block, same trick as the MeroMero seat.
GPU0 seats (gen, meromero) were untouched and healthy throughout. The quant ran in
GPU1 free space with no production seat stopped except fable, which was replaced.
preetpatel/Qwen3.8-27B-Uncensored-NVFP4 is disqualified on two independent hard
failures, both read directly off the artifacts via HTTP Range requests against the
safetensors header (about a megabyte, not a 20 GB download):
- ZERO mtp tensors. The author's recipe.yaml asks to ignore re:.*mtp.*, but the
written config.json has no mtp ignore entry while re:.*visual.* expanded to 110
explicit ones. That asymmetry is llm-compressor pruning a pattern that matched
nothing, i.e. the MTP head was never loaded. Costs roughly half our decode.
- NVFP4 W4A4, 4-bit activations. Precisely the AEON failure mode: the fidelity
gradient is W4A4 < W4+FP8 < W4+bf16, W4A4 drove ~15-20% stochastic degeneration,
and it collapses past ~30k context. The gen seat serves 262K.
orcarouter/Qwen3.8-27B-Uncensored checks out as a quant source: stock-Qwen base
rather than a reasoning-compression finetune, Arditi-style single-direction
abliteration, 15 mtp and 333 visual tensors verified present, chat template
byte-identical to the heresy build we are serving, and the gate is already accepted
on our token.
Also records the author's FP8 release as a noted-but-not-recommended third option:
far more traction, but 30.9 GB against NVFP4's 22 GB, and on a zero-sum GPU0 that
+9 GB comes out of the KV pool and breaks 262K context.
And states the imatrix constraint plainly. Our recipe has always requested
imatrix_mse and always silently fallen back to uniform MSE; playbook 3.13 warns
against assuming an imatrix would help before verifying llm-compressor can consume
external importance data at all. The W4A16 portions are data-free by construction
and cannot use it regardless.
Ran Heretic v1.4.0's 300-trial TPE search on Cold-Fusion. Best trial scores
8/100 refusals at KL 0.0136 against a 98/100 base, versus absolute-heresy at
29/100 and our hand-tuned Robinson L35 at 72/100 / KL 0.0116 — i.e. 64 fewer
refusals for the same damage. Hand-verified coherent: correct arithmetic with
shown working, clean code, 66-167 word prose across nine probes.
Durable findings:
- direction_scope=0 (single shared direction) is decisive on this merged base:
n=129, best 8/100. Per-layer directions n=131 never beat 52/100 despite a
better median. Points against the multi-direction intuition for a diffuse
direction (our two-template |cos| is 0.62 vs Robinson's 0.99 on stock).
- Aggression is not the lever. r(KL, refusals) = -0.561 over 261 trials; the
KL<0.02 band contains both the worst results (median 87/100) and the single
best. A KL 0.3554 trial scored worse than one at 0.0193.
- PR #317 confirmed: Heretic silently drops the MTP head on save. Source 1199
tensors -> export 1184, all 15 mtp.* gone, vision 333/333 intact, exit 0, no
warning. This is also why absolute-heresy ships a byte-identical MTP head —
a bug, not a design choice. Always diff tensor keys after a Heretic export.
- Heretic's recovered direction carries 6.18% of its energy in sink dim 3994,
versus 0.094% for our L35 and 1.97% for the L39 we rejected as brick-inducing.
It survives that only because of magnitude-preserving ablation
(row_normalization=FULL); our plain projection has no such protection, so the
sink screen correctly refused the in-band MTP graft. Same direction, different
operation. MPOA is the prerequisite for in-band MTP on a Heretic trunk.
- Heretic's edit is recoverable from weights: delta is rank-1 (s2/s1 ~ 0.010),
SVD gives the direction, norms give per-layer weights (1.08 -> 1.34, i.e.
over-projection). Cross-layer |cos| agreement 0.9903 independently confirms
the single-direction result.
New tooling in services/coldfusion-abliteration/:
kl_divergence.py first-token KL, class-split, zero noise floor
catatonia_gate.py 12 probes x 220 tokens, prints every completion
heretic_export.py PTY driver; selects by measured value, never by menu
position — Heretic's resume prompt puts "delete the
checkpoint and all results" one arrow-key from the target
graft_mtp.py recovers the trunk direction by SVD; --pristine for the
safe path when the sink screen refuses
Also adds quant playbook 3.13: the NVFP4 recipe sets observer="imatrix_mse" but
llm-compressor has always silently fallen back to uniform MSE for want of
importance data — on this build and on the incumbent. Existing A/B comparisons
stay valid since every build shares the fallback. Parked as id 42.
Guardrail note: this build has lost the self-harm guardrail that the Robinson
L35 build retained. Restoration is the operator's own work item.
Adds `kl_divergence.py`: first-token KL(stock || abliterated) over the full
248,320-token vocabulary, bf16 vs bf16, scored separately for held-out harmless
and reserved-harmful prompts.
Result (L35, 256 harmless / 104 harmful, answer mode):
harmless median 0.0211 mean 0.0364 top-1 agreement 89.8%
harmful median 0.5996 mean 0.6992 top-1 agreement 55.8%
selectivity 28.4x (72.8x in think mode)
Self-KL noise floor is exactly 0.0, and all 720 per-prompt values are
bit-identical between a single-process and a two-process run, so the figures are
signal rather than bf16 jitter. Reverse KL on harmful/answer is 1.43 vs forward
0.70 — the mass-where-stock-had-none asymmetry expected of a refusal-direction
removal. Against the Heretic reference figures (0.1191 prior seat, 0.0759 the
live absolute-heresy seat) this is materially gentler, but those are the other
tool's optimizer output on a different base with its own harmless set and
template — order-of-magnitude, not head-to-head. KL remains a fidelity number;
the viability gate is still MTP acceptance (59.1%).
Method notes:
- Prompt classes are reported separately by design. A single averaged KL over a
mixed corpus is close to meaningless, since the metric is meant to be large on
harmful prompts and small on benign ones; the ratio carries the information.
- The harmless evaluation set is drawn from the alpaca pool minus calibration's
own draw, reconstructed by replaying that draw rather than remembered, and
asserted disjoint on text. The harmful set is the reserved test split.
- `render` is imported from abliterate.py rather than copied, so the measurement
cannot drift from the rendering the direction was captured against.
- Batch size 1 with logits_to_keep=1: no padding semantics, ~0.6 MB of logits.
Three corrections to the runbook, each of which cost time:
- "bf16 is 50 GB, only gen must go" was 50.10 GiB mislabelled. Text-only weights
are 51,300 MiB; freeing either GPU0 seat alone leaves ~50,933 MiB. Both must
stop. VRAM is now sized from the safetensors headers at run time.
- A 27B model cannot be released in-process: `del` + gc + empty_cache left free
VRAM at 45,287 MiB, and so did confining the model to an inner frame that
exits. Only process exit returned the card (96,689 MiB). The first run
completed only because the allocator hit OOM, collected, and retried. Each
model now gets its own process, handing log-probs to disk between stages.
- The residency gate read hf_device_map, which transformers leaves empty when the
model fits on one device — it reported "(unsharded)" whether or not anything
was wrong, so it could never fail. It now reads parameter devices directly.
Model-agnostic lessons promoted to the quant playbook (new 3.12).
The abliterated model works. A/B vs stock on a matched greedy battery: explicit
sexual + graphic torture (the measured stock refusal surface) go from refused to
complied/engaged, held-out AdvBench prompts loosen, the self-harm guardrail
survives, coherence intact — the Robinson design point exactly. Output at
/tank/aimodels/qwen38-27b-coldfusion-abliterated-L35-bf16, verified bitwise:
131/131 targets changed, 333/333 vision byte-identical (delta 0.0), 735/735
others untouched.
Getting there corrected three diagnoses the prior session had backwards.
1. The layer-selection metric was wrong, and that was the whole ballgame. The
recipe picks the abliteration layer by peak two-template |cos| agreement. On
this heavily-merged base that metric is anti-correlated with efficacy: its
argmax (layer 18) is the WORST-separating layer in the window (Cohen's d 5.51
vs 9.89 at the peak), and abliterating there was a measured behavioral no-op —
stock and "abliterated" refused all six probes identically. Cause: the two
renderings end in different generative modes (</think> vs <think>), so |cos|
scores answer-vs-reason mode, not refusal, and on a merge the mode term
dominates. Replaced selection with harmful/harmless SEPARATION (Cohen's d /
AUC of the direction's projection), gated on the sink screen since separation
and sink-energy both climb with depth. Picks layer 35 (d 9.35, AUC 0.9997,
sink 0.094%). Agreement is kept as a printed diagnostic.
2. The "bf16 NaNs, use fp32" rule was a misdiagnosis. The NaN was never
precision — it was multi-GPU sharding (the residual stream zeroes two layers
past the GPU0->GPU1 boundary; the first capture's layer 22 happened to sit in
the healthy region, which is why it looked fine) plus
PYTORCH_CUDA_ALLOC_CONF=expandable_segments (corrupts retained tensors; the
corruption MOVED between bit-identical forwards, the tell that it was memory
not math). On one GPU with a plain allocator, bf16 full-64-layer is exactly
deterministic and coherent, at 50 GB and 4.3x the throughput of the 111 GB
fp32 it replaced. Both defects are now hard gates (residency exit 8, allocator
exit 9); capture pins CUDA_VISIBLE_DEVICES=0.
3. The corpus-size hypothesis was falsified. 52x more calibration data (8->416,
mlabonne/harmful_behaviors = the recipe's actual AdvBench split, already on the
box) moved agreement 0.594->0.624 — nothing. Kept the 416/416 corpus anyway
(calibration.py); it gives the clean separation signal. The held-out 104-prompt
test split is reserved and asserted disjoint.
Also: the --out write is now shard-level surgery (reads/writes the 18 safetensors
directly, no model object, no GPU). This is correctness, not thrift —
AutoModelForCausalLM resolves to the TEXT model, so save_pretrained would drop all
333 vision tensors AND skip the MTP head (the in-band MTP edit is the entire point
of the Robinson formula). Neither failure raises. Shard surgery makes vision and
the other 1068 tensors byte-identical by construction.
Batched capture with a dtype-aware equivalence gate; hidden states captured via
forward pre-hook (reading output_hidden_states off the returned object is unsafe
here — buffers get recycled). Sharding/allocator lessons promoted to the
quantization playbook (model-agnostic, sections 3.9-3.11 + superseded table); the
selection-metric lesson added to the recipe doc.
The dead layer-18 no-op checkpoint was removed (52 GB, confirmed identical to
stock). Incumbent gen seat untouched. Full canonical refusal-probe re-profile and
MTP-acceptance-on-quant still owed before this becomes a gen-seat candidate.
Reference recipe (not a deployed artifact) for MTP-aware, vision-preserving
single-direction abliteration of Qwen3.8-27B -- the base family the gen seat
runs. Captures the two things this recipe gets right that naive abliterations
of this architecture miss:
- The MTP head is abliterated in-band (its two residual-write matrices, glue
left alone), so speculative acceptance does not collapse on the prompts
abliteration exists to fix -- directly relevant to the gen seat's MTP>=40%
gate.
- The vision tower is preserved byte-identical (333 tensors, max delta 0).
Plus the two calibration traps specific to this base: the twice-captured
refusal direction (layer 26, |cos| 0.99) and the attention-sink dimension 3994
that bricks the model if orthogonalized out. Documents the coverage gate
(o_proj 16 + linear_out 48 == 64 layers) that catches a half-abliterated
model before it writes a byte, and the foot-gun that the GGUF imatrix does not
cover the MTP block. Links into model-quantization-playbook.md for the quant
half of the pipeline.
Operator's framing, and it is a better argument than the terminology
correction that preceded it. Under v4, exposing a host needed two
affirmative acts -- a DNAT and an accept rule -- so missing either left
the host dark. There is no v4 misconfiguration that exposes an internal
host by accident. NAT was load-bearing security whether or not anyone
designed it that way.
v6 removes the first control entirely. The path exists inherently, so
the firewall is the only thing left, and the failure mode inverts from
fail-closed to fail-open. Rule-ordering slips, rulesets that silently
match only one address family, new VLANs added without policy, and
re-delegated prefixes unmatching address-literal rules all become
exposure events rather than no-ops.
Records the practical consequences: key rules on interface/zone rather
than address literals, treat enabling v6 on a segment as requiring
policy to exist first, and verify default-deny from off-net rather than
by reading the ruleset -- which is lesson 3's assert-the-effective-value
discipline applied to firewall policy.
Also corrects my own claim from the previous commit that the pending
firewall pass was 'smaller' than I had implied. It is not smaller, it is
different in kind.
The link died when ESH lost its public IP during the fiber cutover. Two
independent causes, and the second would have defeated the obvious fix:
- phase1 ana-to-eshudm was type static, pinned to 70.181.90.232, an
address that no longer exists.
- nattraversal was disable, so ESP could not have crossed NAT even with
the peer IP corrected. pfi-ana-nh3 shares that setting and survives
only because NH3 is publicly addressed, which is why the two tunnels
diverged.
FortiOS refuses `set type dynamic` on an existing tunnel -- "Cannot
change tunnel type once configured" -- and rolled back cleanly, so the
fix could not be an edit. Rather than delete and recreate, which
cascades into the phase2, two static routes and ten policies, the
replacement was built alongside: new phase1+phase2 ana-eshudm-dyn
(type dynamic, ikev2, aes256-sha1, dh14, NAT-T on, PSK read from the ESH
UDM API so neither side needed a new key), static route id 10 at
distance 20, and two consolidated multi-zone policies 73/74. The old
tunnel is left in place, dead and harmless, as rollback.
Verified up: ana-eshudm-dyn_0 97.170.236.56:4500 selectors 1/1 -- the _0
suffix is a dialup child, :4500 is NAT-T, and the address is the
carrier's, which is precisely what could never have been pinned. ESH
reaches all four colo hosts at 40-56ms, the colo reaches all three ESH
hosts, and traceroute drops from eight hops leaking into the carrier
network to three hops fully encapsulated.
Config was backed up before any write (1.17MB, 36903 lines, off-box).
Residual fragility recorded: the UDM's ipsec_local_ip demands a literal
address -- empty is rejected as api.err.InvalidPayload -- so it still
needs updating when the fiber changes ESH's WAN address. The gateway end
is now address-agnostic; the UniFi end is not.
Correcting an over-generalisation from earlier today. Proving that NAT
does not break Site Magic, I wrote it up as "no addressing outcome
threatens the inter-site tunnel." That is wrong: the fleet has two
inter-site links with opposite NAT behaviour.
- NH3<->ESH is Site Magic, i.e. WireGuard. It survives arbitrary NAT,
proven live on RFC1918 double-NAT (192.168.200.111) with nh3-dev and
nh3-docker reachable at ~40ms. It dials out to NH3's public edge and
never needs inbound reachability.
- colo<->ESH is IPsec on the ana-gw FortiGate, and it is broken right
now under those same conditions. ana-docker, pfi-pve and pbs-ana all
fail from esh-pve-nas, and traceroute shows packets for 10.250.x
leaving the UDM to the 5G modem and then wandering the carrier network
before dying -- not encapsulated at all, so no SA is up and the
traffic falls through to the default route. Site-to-site IPsec pins a
peer IP and ESH no longer has a routable one.
So the IPv6 work keeps its justification, but on the IPsec link
specifically rather than on the tunnels generally. Operator caught the
over-generalisation.
Adds lesson 8 -- a result proven for one protocol does not transfer to
another -- and corrects the superseded-claims row rather than replacing
it, since the original claim was half right and the halves are the
point. Also records my own over-broad claim as its own superseded row.
Seeded by the Site Magic / CGNAT premise, which justified a body of IPv6
work and turned out to be false the first time anything actually tested
it. The mechanism was discoverable in advance: Site Magic is WireGuard
and the far side has a public endpoint, so the NAT'd side dials out and
never needs inbound reachability. NAT breaks inbound; it does not break
outbound-initiated tunnels with keepalives.
Also fills the first row of the superseded-claims table, which is what
that table exists for -- the claim is corrected with a date rather than
quietly deleted, so older references to it resolve instead of misleading.
Sibling to model-quantization-playbook.md, and it exists for the same
reason that one does: hard-won lessons were dying inside per-host
runbooks where nobody finds them until after repeating the mistake.
Six entries seeded from the esh-pve-nas migration, all of which would
bite identically on any other host:
1. mount --rbind into a chroot needs --make-rslave, and losing cgroup2
impersonates failing root-disk I/O closely enough that it was
misdiagnosed as exactly that.
2. A reboot is not confirmed until the host is observed DOWN; "never
rebooted" and "rebooted fast" are indistinguishable otherwise.
3. Assert the effective value, not the presence of a substring. Grep
proves presence; only evaluation proves effect.
4. Ask the server who its clients are -- documented dependent lists rot.
Plus the corollary that an idle hard NFS mount blocks and resumes, so
quiescing means stopping consumers, not always unmounting.
5. The scoped-looking command can be the dangerous one; setting a ZFS
cachefile on one pool of three would have stopped the other two from
importing at boot.
6. Long uptime hides breakage, and a forced look is worth more than it
appears -- one migration surfaced an 82-day-dead pvestatd, a 126-day
hung vzdump, a VM in prelaunch for four months, and an undocumented
cluster, none of them caused by the work.
Carries a superseded-claims table so corrections are dated rather than
silently edited, same discipline as the quantization playbook. The ESH
runbook now links here so the general rules are reachable from the
specific story and vice versa.
Operator correction to the prior 3.8 framing (d28a371), which over-blamed
AEON and dismissed the vLLM bug as a mere amplifier. Both were real and
compounded:
- Cause 1 (real, upstream): the qwen3_5_mtp x GDN partial-accept bug
(#51113), architectural across vLLM/SGLang/llama.cpp, genuinely improved
by the nightly fix -- not just an amplifier.
- Cause 2 (real, quant): AEON is FULL W4A4 (A4 activations on attention),
the bottom of the KNOWN activation-precision gradient already in 1
(W4A4 < W4+FP8 < W4+bf16) -- mildly subpar, not 'defective'. On top of
Cause 1 it degenerated ~15-20% of real multi-turn generations.
The mixed FP8-attention build sits a rung up that gradient and is coherent;
a W4+bf16 build would be higher still at a prefill cost. Process lessons
retained (two causes mask each other; stochastic degeneration is invisible
to n=1 probes; isolate weights in parallel with serving flags -- but the
weight swap alone would NOT have found the real vLLM bug).
Root cause of the multi-day degeneration hunt, operator-confirmed: the AEON
NVFP4 W4A4 quant (sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4,
full W4A4 incl. attention) went degenerate ~15-20% of generations in real
multi-turn use and forced regenerates. MTP, prefix-caching, and the gateway
all merely AMPLIFIED it, which is why MTP-off, APC-off, and the vLLM #51113
fix each 'helped' a synthetic probe without fixing it -- three plausible
false root-causes, each passing one clean run then failing in real use.
The fix was the WEIGHTS: the in-house JonathanColetti/Heretic mixed
NVFP4+FP8 build (qwen38-27b-uncensored-nvfp4-mixed, FP8 attention not W4A4,
same base, same MTP) is coherent through long multi-turn with MTP ON. W4A4
*attention* was the defect; FP8 attention is not.
This commit:
- GEN_MODEL -> the mixed FP8-attn build (primary gen until DavidAU 3.8 lands)
- GEN_IMAGE pinned to vllm/vllm-openai:nightly-311b3513... (v0.27.2rc1.dev150,
carries #51113; pinned by sha so it does not drift on the next pull)
- AEON weights PURGED from /tank (no-good), safety-checked not-in-use first
- playbook 3.8: the stochastic-W4A4-degeneration lesson + isolate-weights-early
+ do-not-declare-a-fix-from-one-probe (it validated three non-fixes)
AEON is re-pullable from HF if ever needed, but the operator ruled it no-good.
The qwen3_5_mtp corruption (playbook 3.7) is gated on MTP x prefix-caching
TOGETHER (vllm#43559 / #47194), per both cross-frontier peers. Disabling
prefix caching (--no-enable-prefix-caching; vLLM V1 defaults it ON, so the
explicit --no- form is required) forces the GDN cache into a mode where the
partial-accept align-path bug is inert, so MTP can stay on.
Verified on our stack (AEON W4A4): MTP on + prefix-caching off -> the 7-turn
varied series stays coherent through 3.9k tokens, zero cross-turn bleed, at
104.6 tok/s / 53.6% acceptance -- the FULL MTP speedup restored (vs ~half
with MTP off), losing only prefix-cache reuse. All 7 aliases route.
Ruled out on the way: num_speculative_tokens=1 (corruption is
depth-independent, n=1 and n=2 both corrupt); switching to SGLang (vLLM /
SGLang / llama.cpp mainline all share the architectural GDN-rollback bug).
Proper upstream fix (#51113) is in main / v0.27.2rc0 only, not stable, so we
hold at APC-off rather than jump the fleet gateway to an RC.
Supersedes the MTP-off config from 7bd38b3.
The single hardest bug of the night, and invisible to the existing
acceptance gate: a LOADED, healthy-accepting MTP head still corrupts
Qwen3.8-27B multi-turn output past ~2k cumulative tokens (length collapse +
cross-turn content bleed), while single-turn is perfect. Model-independent
across all three of our Qwen3.8 quants; Qwen3.6 on the same qwen3_5_mtp
method is clean; disabling MTP fixes it. New rule: gate MTP on a multi-turn
coherence probe, not just single-shot acceptance.
Quants are hard-fought and we keep re-paying for the same lessons. A survey
found quant knowledge scattered across 18 files in four trees, with three
documents having independently discovered and recorded overlapping
"landmines" sections — and one of them now actively misleading.
Adds docs/pfi/model-quantization-playbook.md as the single home for the
TRANSFERABLE lessons, with per-model artifacts demoted to worked examples
that link up to it. Contents:
- scheme decision table, incl. that a literal "W4A8" NVFP4 checkpoint is
unservable on vLLM (two legal activation settings, FP8 is not one)
- the reference mixed-precision recipe and the three parts of it that are
load-bearing and easy to drop
- the recurring landmines, ordered by cost: the loader-class trap
(rediscovered THREE times), the three separate ways to lose the MTP head,
toolchain deadlocks, vision configs, memory/device placement
- pipeline shape: prove targets before spending GPU time; mandatory
post-steps that verify rather than assume
- the acceptance gate, and the three ways measurement has lied to us —
prefix caching faking both speed metrics, prompt_logprobs going uniform
under speculative decoding, and a 0600 .env making compose silently no-op
- hardware/co-residency, including that a SMALLER model can starve its
neighbour because gpu-memory-utilization is a fraction of the whole card
- a superseded-claims table, and measured negatives not to re-chase
The superseded table earns its place immediately: the heretic2 runbook tells
readers to use modelopt because "compressed-tensors can't load the BF16 MTP
head, 0% acceptance". That symptom was real but the cause was not the format
-- it was the missing re:^mtp.* ignore entry. compressed-tensors gives
47.7-83.2% acceptance in production. A fresh session following that doc would
be sent down the modelopt path that current memory calls dependency hell, so
the runbook now carries a stale-warning header pointing here.
Wires discovery: an orientation.md "Where to look for what" row, pointers
from the gen-seat / heretic2 / mistral artifacts, and a CLAUDE.md maintenance
rule so the playbook gets fed instead of going stale -- model-agnostic
lessons land in the playbook, model-specific ones stay put, and a wrong
claim earns a dated superseded row rather than a silent edit.
Motivated by Qwen3.8 having just released: the next model swap will need a
requant, and this is what that session should read first.
56KB deep-research report on thinking-capable eRP finetunes 15-30B, weighted for
spatial/state coherence, targeting RTX PRO 6000 Blackwell (sm_120) NVFP4. Preserved
from an ephemeral Booth (gecko-65) into durable reference for the quant decision.
Brokkr independent verify clean (maxdiff 0.000000, no split). R42 v13
acceptance gate PASSES first time in its history: main+kb 56/90->90/90,
evictions 33->0. A2 control torn down. A3 throughput characterized at
~34 req/s (graceful queueing), with A4/util-bump/replica as levers.
A3 now backs the prod reranker alias but was launched --restart no;
docker update to unless-stopped so an ana-ml2 reboot can't silently
break the alias. Full compose-service promotion tracked as a follow-up
in the selection ledger.
The incumbent Qwen3-Reranker-0.6B was measured actively harming 80/90
fleet queries on main+knowledge_base (and inverting the bare-name region
behind Worldtree #389) — no-reranker beat it 89/90 vs 56/90. Brokkr's R43
bake-off selected BAAI/bge-reranker-v2-m3 (A3): 90/90 top-10, mean rank
0.19, multilingual (XLM-R), ~1.2 GB lighter than the incumbent.
Control arm (A2 = same Qwen weights, seq-cls head) scored identical to the
incumbent, proving the fault is a training prior, not the serving head —
which cancelled the expensive Qwen3-4B arm before it cost a GPU seat.
Cutover boundary 2026-08-06T17:37:48Z. The qwen3-reranker alias and the
:8002 backend are retained for one-edit rollback. Adds the process audit
trail at docs/pfi/reranker-selection-ledger.md.
The live gateway config has served char-rp-reasoning as deckard-pkd-27b (:8018)
since the 2026-07-08 A/B; the standalone doc had frozen on QwQ-RpR-v4. Corrects
seat 4 (backend + samplers + server-side DRY/reasoning-budget notes).
Also snapshots session state in persistent-memory.md: phantom-qwen verified
already-clean, ana-docker docker log-cap (logrotate copytruncate, no bounce),
and the granite→gen memory_extractor bind live on demo+personal.
Add §9 "PFI LiteLLM Gateway — Deployed Sampling Defaults": the live fleet
sampling table (granite/qwen/judges/GLM) with provenance, overrideable-default
semantics, the GLM API-accepted-subset caveat, and the research-confirmed temp-0
rationale for granite + image-judge. Accepts the dvalin-smithy-dev recommendations
as deployed. §§1-8 vendor reference left intact.
Replaced the 9-month-stale 'Last Updated: 2025-07-14' line with a
status disclaimer — this is KB-derived advisory material, not
something we maintain in lockstep with code. Tracer-date claim was
implying a freshness contract we don't fulfill. Found via
/tend-docs.
Same anti-pattern as the deleted model-list.md — a hardcoded snapshot
of mutable config that's guaranteed to drift. Replaced the 15-row
table with a one-liner pointing at stacks/llama-swap/conf/config.yaml
(the canonical source) and the live /v1/models HTTP endpoint. Found
via /tend-docs.
Self-reported "Synchronized with llama-swap config.yaml on 2025-07-18"
— 9 months stale. Canonical model truth is stacks/llama-swap/conf/config.yaml;
the static snapshot was guaranteed to drift. No historical value (not
an ADR, post-mortem, or migration runbook).
Found via /tend-docs.
Second pass on docs/ cleanup (item #15 in STATUS.md):
- pfi/chromadb-setup.md: deleted. References configs/pfi-ana/... and
scripts/setup-chromadb.sh, neither of which exist in this repo
(artifacts of an earlier project layout). ChromaDB is already live
per docker-stack.md; the operational truth lives there.
- pfi/docker-stack.md: removed the cross-link to the deleted file and
pulled the bit of useful content from it (auth-token generation +
client Settings example) into the inline ChromaDB section.
- pfi/vm-102-matrix-{synapse,appservice}.md: stripped the YAML
frontmatter. The `path:` values pointed at docs/pfi-ana/... which
doesn't exist in this repo; no toolchain consumed the metadata.
- README.md: tree updated to reflect the deletion.
VM-102 docs kept separate by design — each is right-sized; merging
would push past the ~500-line guideline.
Network probes (no web admin, Debian SSH banner, only file-server ports
open) show 10.250.50.50 is vanilla Debian 12 with hand-configured NFS/SMB,
reprovisioned from the original TrueNAS SCALE install. Update stack
comments, README storage notes, Backrest description, proxmox-vms.md
entry for VM 100, and the restic configs intro to match.
Cross-site sync plan simplifies to plain rsync over SSH on both sides —
no appliance-specific tooling needed.
Captures the full workspace state built up to this point:
- CLAUDE.md + README.md describing conventions and the four-host fleet
(ana-ml2, ana-docker, nh3-docker, esh-docker-vm).
- Per-host notes under servers/<host>/ with ssh-target fallback files
and latest system-details snapshots (two in-compose credential leaks
scrubbed; the upstream compose files still need to move those to .env).
- scripts/: server_inspect.sh (read-only remote diagnostic),
refresh-server-info.sh (dir-driven discovery + snapshot capture with
validation warnings), add-host.sh, sync-stacks.sh (pull
compose/conf trees), deploy-stack.sh (push with per-file diff + prompt).
- stacks/: canonical compose for backrest, beszel, dozzle, llama-swap,
rest-server-ana, rest-server-nh3, vllm-qwen3, plus the retired
infinity reference. All use the .env-driven + traefik-net + homepage
label pattern.
- configs/restic/ana-docker/: first resticprofile config + pre-backup
hook (Synapse pg_dump, Seafile mysqldump, Vaultwarden SQLite); templates
for the other three hosts to come.
- docs/pfi/: general infrastructure reference carried over.
- .gitignore excludes .env, stacks-mirror/, and assorted secret/state
filenames to prevent re-leaks on later commits.
vh