Gates run against the live seat while the operator tested in parallel.
<think> leak (n=30, 4 prompt types + multi-turn) 0/30, 0 empty
MTP acceptance 58.4% @ 117.11 tok/s median
surface 6/6
abliteration survival 4/4 compliance
deterministic quality gens coherent and correct
PPL still blocked
For scale on the leak gate, the abandoned h300 build scored 8/30 on this exact
instrument, and its abliteration-survival samples had 2 of 4 open with
"<think>Ok, let's figure this out:". Orcarouter has none.
The headline is MTP acceptance. 58.4% against heresy's byte-identical base head
at 47.2% is +11 points, and it sits level with our own in-band L35 at 59.1%. That
is the additive in-band-vs-graft delta the entire Cold-Fusion experiment was built
to measure and never cleanly delivered -- orcarouter handed it over for free
because the author had already done the Robinson edit on the head.
Surface 6/6 covers plain chat, vision, tool calling, the thinking split, a
36,042-token long-context retrieval, and streaming.
PPL remains blocked on a spec-decode-free probe seat: it needs ~22 GB and GPU1 has
~16 GB free. Comparison target is heresy at 6.910.
Operator directive was seat-first so he can test while the gates run.
GEN_MODEL -> /tank/aimodels/qwen38-27b-orcarouter-nvfp4-mixed. Healthy in ~4 min.
KV pool 401,550 tok / 1.53x. MTP drafter detected and wired, sharing embedding and
lm_head with the target. 7/7 gateway aliases 200. Vision correct on the shape
probe. Live decode observed at 102-133 tok/s under load.
Critically, <think> does not appear in the top-20 first tokens on the live seat.
That is the Cold-Fusion failure mode measured absent in production, matching the
pre-quant screen on the bf16 (1.23e-06, rank 52).
Rollback is one line to .env.bak-heresy-restored-20260821.
Full gates were still running when this landed; PPL stays blocked on a
spec-decode-free probe seat, which needs ~22 GB against GPU1's ~16 GB free.
Pulled orcarouter/Qwen3.8-27B-Uncensored at rev 9878936b (55.5 GB, gated, our
token has access) and built /tank/aimodels/qwen38-27b-orcarouter-nvfp4-mixed
(23.4 GB, mixed NVFP4+FP8). Verified, not yet cut over.
The operator asked whether we could apply the Robinson path to the MTP head. We
cannot, because the author already did. compare_mtp_head.py against the verbatim
base graft: 13 of 15 tensors byte-identical, exactly 2 differ --
mtp.layers.0.self_attn.o_proj.weight and mtp.layers.0.mlp.down_proj.weight, which
are precisely the two residual writers our own abliterate.py targets
(EXPECT_MTP_WRITERS = 2).
Reverse-engineered the edit from the weights alone (mtp_delta.py, added here):
sigma2/sigma1 = 0.0164 on BOTH tensors rank-1, a single-direction projection
|cos| between the two recovered dirs = 1.0000 ONE shared direction
||delta||/||W|| = 1.42% and 1.41% a gentle, consistent projection
sink energy dim 3994 = 0.0000% sink-clean; Heretic's was 6.18%
That is the Robinson in-band MTP abliteration, already applied, with a direction
that passes our sink screen outright. Nothing to do but preserve it, and the quant
carries it byte-identically. This is the configuration the entire Cold-Fusion
experiment was designed to test and never cleanly delivered.
The new format screen paid for itself on its first real use: think_prior.py on the
bf16 BEFORE any GPU time gave P(<think>) = 1.23e-06 at rank 52, against
Cold-Fusion stock 0.1850 and h300 0.2216. Roughly 150,000x cleaner.
Two durable findings about the pipeline itself:
The quant needs ~17 GB, not a whole card. It ran entirely in GPU1's spare 16 GB
with ZERO production seats stopped -- the h300 run's "stop BOTH GPU0 seats" was
never necessary, it simply had a free card by coincidence. The first attempt OOM'd
by 2.37 GiB at layer 64 of 65 with 3.57 GiB reserved-but-unallocated, which is
fragmentation, and PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True closed it.
post_quant.py now builds a missing output index from the safetensors headers.
A sub-23 GB quant saves one bare shard with no index, and post_quant needs one;
this has broken three separate rounds and been hand-fixed every time. The header
is read by struct-unpacking the u64 length and parsing the JSON -- never
safe_open, which mmaps the whole 22 GB shard and ENOMEMs on ZFS.
Artifact verified: mixed-precision, 1968 tensors, 15 mtp, 333 visual, re:^mtp.*
present in the ignore list (llm-compressor pruned it as always), preproc restored.
Imatrix deferred per operator; the log confirms the usual uniform-MSE fallback, so
this build stays apples-to-apples with heresy's PPL 6.910.
Operator directive following the decision to abandon the Cold-Fusion base.
Removed with explicit literal paths, one at a time:
qwen38-27b-coldfusion-bf16 stock DavidAU base
qwen38-27b-coldfusion-abliterated-L35-bf16 Robinson L35
qwen38-27b-coldfusion-h300-mtp-bf16 Heretic-300 + MTP graft
qwen38-27b-coldfusion-heretic300-bf16 raw Heretic export
Verified against ZFS used, not df: 4.48T -> 4.33T, ~154 GB. No snapshots were
holding the blocks, all four paths confirmed gone, gen seat unaffected.
The last two were hardlink twins -- same inode, links=2, because the MTP graft
hardlinked every unchanged shard -- so deleting only one would have freed
nothing. `du` across several paths in a single invocation dedupes hardlinks and
reported heretic300-bf16 as 2.5K, which would have made a size estimate wrong in
both directions. Check `stat -c %h` before sizing a delete.
Kept deliberately, so the research record outlives the weights:
qwen38-27b-coldfusion-bf16.PROVENANCE.txt pinned HF revision 9c44193f
coldfusion-abliteration/ harness, 300-trial Optuna
journal, catatonia-T260.json
With those two, every deleted build is reproducible: re-pull stock at the pin and
replay the winning config.
Held back pending an explicit call: the two NVFP4 quants, h300-nvfp4-mixed (the
only remaining servable copy of the Heretic-300 result) and L35-nvfp4-mixed. The
directive named bf16 weights; these are quants, and there is no storage pressure
arguing for haste at 4.26T free.
Operator directive, given before the result was in: if it's the base, abandon
h300 and the base too. The dose-response said base (18.5% of 22.2%), so it fired.
Live gen seat is /tank/aimodels/qwen38-27b-heresy-nvfp4-mixed again, restored
from .env.bak-coldfusion-L35-20260820. The h300 env is preserved at
.env.bak-h300-abandoned-20260821.
The clincher, same probe pointed at heresy:
Cold-Fusion stock P(<think>) 0.1850
Cold-Fusion L35 0.2048
Cold-Fusion h300 0.2216
heresy (restored) not in the top 20, <0.002
A >100x gap between the families, which is why no rollback inside Cold-Fusion
would have helped -- stock and L35 leak at nearly the h300 rate.
Verified after rollback: 0/30 leaks and 0 empty on the same instrument that
scored h300 at 8/30, with the EXISTING enable_thinking:false config; KV pool
403,065 tok / 1.54x, heresy's exact documented baseline; 7/7 aliases; vision
intact. No LiteLLM change was needed, so the chat_template_kwargs fix is left
unapplied -- it worked, but it was a workaround for a base we no longer serve.
Cost, stated plainly: 8/100 refusals becomes 29/100, a 3.6x regression on the
axis the whole Heretic-300 run existed to move. Accepted deliberately.
What carries forward is the methodology, none of which lived in the Cold-Fusion
weights: direction_scope=0 beating per-layer on a merged base, aggression not
being the lever, PR #317 silently dropping the MTP head on save, the MPOA and
sink-screen reasoning, the graft/KL/catatonia/export harnesses, and the finding
that a pristine MTP graft accepts as well as an in-band edit.
New acceptance gate earned here: run think_prior.py on a candidate's STOCK
weights before committing GPU time. It is a ~10s CPU measurement and it would
have disqualified Cold-Fusion before the 300-trial study ever started. Heretic's
objective has no format-compliance term at all -- the same blindness that removed
the self-harm guardrail.
Nothing deleted. Every Cold-Fusion artifact, the 300-trial Optuna journal and
catatonia-T260.json remain on disk. Abandon means stop serving, not rm.
Answers "how likely is it that our abliteration caused this?" with a measurement
instead of a prior. P(<think>) at the first generated token, template rendered
enable_thinking=false so the prompt already carries a CLOSED think pair -- the
exact event behind the leak. Raw softmax, bf16, CPU-only, one process per model.
Deterministic: stock reproduced to 17 significant figures across two runs.
coldfusion-bf16 none (stock) 0.1850 rank 3
coldfusion-abliterated-L35-bf16 Robinson L35, mild 0.2048 rank 2
coldfusion-h300-mtp-bf16 Heretic-300, heavy 0.2216 rank 2
The stock, untouched base already puts 18.5% of first-token mass on opening a
think block the template had closed. Abliteration adds a real, monotonic,
dose-dependent +3.7 points -- a nudge on a pre-existing base, not the cause.
Cold-Fusion is a reasoning-token-compression finetune, i.e. a model trained to
think briefly, and the leak's text shape agrees: a compact correct trace with a
trained transition marker, which is trained behavior rather than damage.
This changes the options. Rolling back to L35 or stock does NOT fix the leak --
at 18.5% under temp 0.7 / top_p 0.8 they leak at nearly the h300 rate. Only
leaving the Cold-Fusion family escapes it, at the cost of the 8/100 refusal
result. The chat_template_kwargs fix is the correct lever.
Durable methodology point: a forward-KL budget cannot catch this. Heretic
minimizes forward KL(stock||abliterated), which is near-blind to the model
putting new mass on tokens stock barely used -- that is reverse KL's job, and we
measured exactly that asymmetry on L35 (reverse 1.43 vs forward 0.70). h300's KL
of 0.0136 is not evidence of innocence. For any "did the abliteration break
behavior X" question, measure P(token) directly.
Ran CPU-only deliberately: 96 EPYC cores and 265 GB of RAM make a 27B forward
pass cheap, so this cost no GPU window and no seat downtime, where the obvious
route was stopping both GPU0 seats.
Also normalizes two more abliteration output dirs from root-owned 0600 to
llmuser 0664. The unreadable-model failure surfaces as FileNotFoundError rather
than a permission error, which is worth knowing before it wastes a run.
Operator reported the new Heretic-300 gen seat "sends CoT but never completes
the turn" through Lobe. Diagnosed; not yet fixed (the fix changes gen's
semantics, so it is the operator's call).
The Qwen3.8 chat template appends a pre-closed <think>\n\n</think>\n\n when
enable_thinking is false. The h300 model opens a fresh <think> anyway and never
closes it. Because the prompt already closed the block, vLLM's qwen3 reasoning
parser is not in reasoning state, so the tag passes through as ordinary text --
reasoning_content empty, reasoning_tokens 0, and the whole reasoning-plus-answer
blob lands in content. Lobe then correctly treats the unterminated tag as
still-thinking and renders no answer. The client and the serving stack are both
behaving correctly; the model is not.
The trigger is TEMPERATURE, not presence_penalty (n=12 per arm):
temp 0.7, pp 1.5 (current gen) 4/12
temp 0.7, pp 0.0 4/12
temp 0.7, pp 0.5 3/12
temp 0, pp 1.5 0/12
That falsifies the standing hypothesis, recorded in the litellm config comment
and in the operator's own 2026-08-16 note, that presence_penalty 1.5 is the
first dial to move. It is not this bug's cause.
It also explains the blast radius: only the two temp-0.7 aliases leak, `gen`
and `summarizer-large`. summarizer, classifier, image-judge and qwen-image-bench
all run at temp 0 and are clean, so nevermore's summarizer path is unaffected.
Candidate fix, validated n=30 over 4 prompt types plus a 3-turn conversation:
chat_template_kwargs {enable_thinking: true, reasoning_effort: low} takes 8/30
leaks to 0/30, at ~+27% completion tokens and a ~3% empty-content residual.
The tell appears in eval_coldfusion_h300.json and in none of the aeon, heresy,
mixed or w4a16 evals, so it is new with this build -- but L35 was never evaled,
so this does not separate a Cold-Fusion base trait from a Heretic-300
abliteration artifact.
Reproducers and the full method land in bench/think-leak/. Note in particular
that the 7/7 alias smoke test run at cutover structurally could not catch this:
trivial prompts never invite reasoning, so they never sample the leaking token.
nevermore pinned LLAMA_SWAP_MODEL=granite-4.1-8b, an alias retired with the
granite seat on 2026-08-12. Every summarization call since then failed: 67
consecutive status=failure rows, 0 tokens, twice daily, entirely silently. The
briefing had been rendering with no LLM pass at all. Nothing alerts on
status=failure in the spend logs, so it took an unrelated question about
reranker VRAM to surface it.
It was also pinned to NEVERMORE_RERANK_MODEL=qwen3-reranker -- the incumbent
Brokkr R43 measured harming 80/90 fleet queries -- and was its ONLY caller,
while the production `reranker` alias sat at 0 calls for 4 days. The R43
cutover repointed the alias but never moved the consumer.
nevermore/.env LLAMA_SWAP_MODEL granite-4.1-8b -> summarizer
NEVERMORE_RERANK_MODEL qwen3-reranker -> reranker
(server-only; .env is excluded from the mirror both ways)
Verified against nevermore's exact call shape: summarizer returns clean content
with 0 reasoning chars at temperature 0.2 / max_tokens 4000; reranker scores
0.95 on-topic vs ~1e-5 off-topic; embedding returns dim-1024.
Retired alongside it:
vllm-rerank :8002 Qwen3-Reranker-0.6B + the qwen3-reranker alias
vllm-rerank-a4 :8014 gte-reranker-modernbert + its alias
vllm-granite :8004 Exited 8 days, dead service block
and vllm-rerank-a3 was promoted from a throwaway `docker run` into this stack
(the selection ledger's own open follow-up). Healthy in 55s. It keeps the
bake-off arm name so the ledger, memory and R43 record stay valid.
VLLM_VERSION is pinned latest -> v0.24.0. Every service in the stack shares that
one variable, so a bare `compose up -d` could have silently upgraded all of
them at once; both tags resolved to the same local image (4091d5593f77), so the
pin changed nothing at runtime.
GPU1 is down to 81,448 of 97,887 MiB -- 13.9 GB reclaimed tonight.
Correction: an earlier claim that A4 had no gateway alias was wrong. It did.
LiteLLM serves both config-defined and DB-defined models -- live showed 32
against config.yaml's 26 -- and grepping the file cannot see the difference.
/v1/models and /model/info (which flags db_model) are the ground truth. DB
models delete hot via POST /model/delete with no restart.
Left alone: reranker-a3-bge-v2-m3, a zero-call duplicate of `reranker` on the
same backend. It is Brokkr's cutover-verification handle -- redundant rather
than broken, and another agent's tooling is not mine to delete unilaterally.
Operator directive: lfm2.5-2.6b goes down permanently.
- stacks/vllm/compose.yaml vllm-lfm25 service removed (replaced by a
tombstone comment), pushed live to ana-ml2
- ana-ml2 container docker rm -f'd, 8,721 MiB freed on GPU1
(95,388 -> 86,667 of 97,887)
- litellm config lfm2.5-2.6b alias deleted, live + canonical,
28 -> 27 models
It was an EVAL-ONLY bake-off seat against granite-4.1-8b that never received
the operator ruling it was pending; the comparator was retired from the roster
on 2026-08-15; it was deliberately never wired into any default or fallback
routing chain; and spend logs show 0 calls in the 4-day window to 2026-08-21.
Weights stay in the shared HF cache -- nothing deleted from disk.
The gateway restart that makes the alias deletion take effect is HELD so it can
batch with a pending reranker change. Until then the name is still routable
in-memory and will error against a dead backend.
Auditing the three reranker seats while answering "why do we have three" turned
up a real problem. The design is one production, one rollback, one fallback --
but the traffic is backwards:
:8013 A3 bge-v2-m3 PRODUCTION, backs `reranker` 0 calls / 4 days
:8002 Qwen3-Reranker RETIRED incumbent, rollback only 7 calls, 12-hourly
:8014 A4 gte-modernbert "fallback" no alias at all
nevermore is hard-wired to the incumbent by name (NEVERMORE_RERANK_MODEL=
qwen3-reranker), so the R43 cutover never moved it -- the cutover repointed the
`reranker` alias and correctly left `qwen3-reranker` naming the Qwen model.
Brokkr R43 measured that model harming 80/90 fleet queries, so nevermore's
twice-daily rerank pass is likely degrading its own briefing.
Fix is one line in nevermore's .env plus a nevermore restart, and it must land
before :8002 is retired. Recorded in persistent-memory with the A4 alias also
noted as absent (global CLAUDE.md names reranker-a4-gte-modernbert; it does not
exist).
Live GEN_MODEL is now qwen38-27b-coldfusion-h300-nvfp4-mixed (ana-ml2 GPU0
:8015). Served-name left unchanged so all 7 LiteLLM aliases route without a
gateway edit.
Verification:
KV pool 401,550 tok / 1.53x (baseline 403k / 1.54x)
LiteLLM aliases 7/7 green
vision 3/3 shapes, colour+form+position correct
MTP acceptance 59.7% median @ 118.37 tok/s
quality gens 4/4 correct
abliteration 4/4 compliance
PPL NOT measured (see below)
The roadmap predicted ~47% acceptance for a pristine MTP graft versus L35's
59.1% in-band edit. Measured 59.7% on the same harness: there is no acceptance
penalty, which removes the throughput argument for reimplementing MPOA.
A single long-prose generation read 47.5% off the same counters -- below the
8-run minimum of 49.0% -- and would have "confirmed" the prediction by
coincidence. Acceptance must be read from quickbench.py, never one sample.
PPL is blocked on VRAM, not on the model: eval_quality.py aborts with
"prompt_logprobs look uniform" under --speculative-config, and the probe-seat
workaround needs ~22 GB while both cards sit at ~96% committed.
Also normalizes the quant dir from root:0600 to llmuser:llmuser 0664 to match
every other model dir, and records that config.json sha256 is byte-identical
across the h300 and L35 quants and is therefore useless for confirming which
weights are mounted (mtime and a head-hash are the discriminating views).
Rollback is one line to .env.bak-pre-h300-20260820.
8/100 refusals at KL 0.0136, hand-verified coherent, beating the absolute-heresy
bar 3.6x. NVFP4 quant complete: 21 GB, 1968 tensors, MTP head grafted back after
PR #317 dropped it, and re:^mtp.* re-injected into quantization_config.ignore
after llm-compressor pruned it. Self-harm guardrail is gone on this build and is
the operator's own next work item; the four-dwarf panel is stood down.
Ran Heretic v1.4.0's 300-trial TPE search on Cold-Fusion. Best trial scores
8/100 refusals at KL 0.0136 against a 98/100 base, versus absolute-heresy at
29/100 and our hand-tuned Robinson L35 at 72/100 / KL 0.0116 — i.e. 64 fewer
refusals for the same damage. Hand-verified coherent: correct arithmetic with
shown working, clean code, 66-167 word prose across nine probes.
Durable findings:
- direction_scope=0 (single shared direction) is decisive on this merged base:
n=129, best 8/100. Per-layer directions n=131 never beat 52/100 despite a
better median. Points against the multi-direction intuition for a diffuse
direction (our two-template |cos| is 0.62 vs Robinson's 0.99 on stock).
- Aggression is not the lever. r(KL, refusals) = -0.561 over 261 trials; the
KL<0.02 band contains both the worst results (median 87/100) and the single
best. A KL 0.3554 trial scored worse than one at 0.0193.
- PR #317 confirmed: Heretic silently drops the MTP head on save. Source 1199
tensors -> export 1184, all 15 mtp.* gone, vision 333/333 intact, exit 0, no
warning. This is also why absolute-heresy ships a byte-identical MTP head —
a bug, not a design choice. Always diff tensor keys after a Heretic export.
- Heretic's recovered direction carries 6.18% of its energy in sink dim 3994,
versus 0.094% for our L35 and 1.97% for the L39 we rejected as brick-inducing.
It survives that only because of magnitude-preserving ablation
(row_normalization=FULL); our plain projection has no such protection, so the
sink screen correctly refused the in-band MTP graft. Same direction, different
operation. MPOA is the prerequisite for in-band MTP on a Heretic trunk.
- Heretic's edit is recoverable from weights: delta is rank-1 (s2/s1 ~ 0.010),
SVD gives the direction, norms give per-layer weights (1.08 -> 1.34, i.e.
over-projection). Cross-layer |cos| agreement 0.9903 independently confirms
the single-direction result.
New tooling in services/coldfusion-abliteration/:
kl_divergence.py first-token KL, class-split, zero noise floor
catatonia_gate.py 12 probes x 220 tokens, prints every completion
heretic_export.py PTY driver; selects by measured value, never by menu
position — Heretic's resume prompt puts "delete the
checkpoint and all results" one arrow-key from the target
graft_mtp.py recovers the trunk direction by SVD; --pristine for the
safe path when the sink screen refuses
Also adds quant playbook 3.13: the NVFP4 recipe sets observer="imatrix_mse" but
llm-compressor has always silently fallen back to uniform MSE for want of
importance data — on this build and on the incumbent. Existing A/B comparisons
stay valid since every build shares the fallback. Parked as id 42.
Guardrail note: this build has lost the self-harm guardrail that the Robinson
L35 build retained. Restoration is the operator's own work item.
vLLM sizes --gpu-memory-utilization against total VRAM but gates startup on free
VRAM, so the GPU0 pair coexists only in its original boot order. Records the
restore sequence (meromero to healthy first, then gen), the observed-not-slept
rule, and the KV-pool baselines to verify a restore against — nvidia-smi used-MiB
is the wrong check, it swings ~7 GB on allocator slack at identical capacity.
Restoring the two GPU0 seats with `start meromero; sleep 10; start gen` put
meromero into a 7-restart crash-loop:
ValueError: Free memory on device cuda:0 (35.3/94.97 GiB) on startup is less
than desired GPU memory utilization (0.52, 49.38 GiB).
The previous commit's README claimed restore order "is not actually load-bearing"
on the grounds that both seats pass --gpu-memory-utilization as a fraction of
total VRAM. That is half right and the wrong half mattered: the fraction sets the
target, but vLLM gates startup on FREE VRAM and refuses to start unless the whole
target is available. GPU0 runs at ~96.4/97.9 GB with roughly 0.4 GiB of slack, so
the seats coexist only in the order they were originally brought up, and meromero
is the one that does not fit in the remainder. The pre-existing auto-memory note
("gen takes a fraction of free VRAM at startup and will starve meromero") was
pointing at the real effect.
Also: "first" means healthy, not ten seconds earlier. A sleep 10 against a
two-to-three minute weight load is simultaneity, not ordering — gate on observed
state.
Recovery applied: stop gen, wait for meromero healthy, start gen. Verified
against the pre-window baseline rather than against "both green":
gen KV 14.36 GiB / 403,065 tok / 1.54x -> 14.34 GiB / 401,550 tok / 1.53x
meromero KV 542,202 tok -> 542,202 tok
RestartCount 0 on both; summarizer smoke-tested through LiteLLM
Note for the next reader: raw nvidia-smi used-MiB is the wrong check here. It
reads 89,503 now vs 96,376 before, which looks like a 6.9 GB regression and is
allocator slack — serving capacity is unchanged. The anomalous boots were the
high ones (34.95 GiB KV), where gen came up on an empty card mid-window.
Adds `kl_divergence.py`: first-token KL(stock || abliterated) over the full
248,320-token vocabulary, bf16 vs bf16, scored separately for held-out harmless
and reserved-harmful prompts.
Result (L35, 256 harmless / 104 harmful, answer mode):
harmless median 0.0211 mean 0.0364 top-1 agreement 89.8%
harmful median 0.5996 mean 0.6992 top-1 agreement 55.8%
selectivity 28.4x (72.8x in think mode)
Self-KL noise floor is exactly 0.0, and all 720 per-prompt values are
bit-identical between a single-process and a two-process run, so the figures are
signal rather than bf16 jitter. Reverse KL on harmful/answer is 1.43 vs forward
0.70 — the mass-where-stock-had-none asymmetry expected of a refusal-direction
removal. Against the Heretic reference figures (0.1191 prior seat, 0.0759 the
live absolute-heresy seat) this is materially gentler, but those are the other
tool's optimizer output on a different base with its own harmless set and
template — order-of-magnitude, not head-to-head. KL remains a fidelity number;
the viability gate is still MTP acceptance (59.1%).
Method notes:
- Prompt classes are reported separately by design. A single averaged KL over a
mixed corpus is close to meaningless, since the metric is meant to be large on
harmful prompts and small on benign ones; the ratio carries the information.
- The harmless evaluation set is drawn from the alpaca pool minus calibration's
own draw, reconstructed by replaying that draw rather than remembered, and
asserted disjoint on text. The harmful set is the reserved test split.
- `render` is imported from abliterate.py rather than copied, so the measurement
cannot drift from the rendering the direction was captured against.
- Batch size 1 with logits_to_keep=1: no padding semantics, ~0.6 MB of logits.
Three corrections to the runbook, each of which cost time:
- "bf16 is 50 GB, only gen must go" was 50.10 GiB mislabelled. Text-only weights
are 51,300 MiB; freeing either GPU0 seat alone leaves ~50,933 MiB. Both must
stop. VRAM is now sized from the safetensors headers at run time.
- A 27B model cannot be released in-process: `del` + gc + empty_cache left free
VRAM at 45,287 MiB, and so did confining the model to an inner frame that
exits. Only process exit returned the card (96,689 MiB). The first run
completed only because the allocator hit OOM, collected, and retried. Each
model now gets its own process, handing log-probs to disk between stages.
- The residency gate read hf_device_map, which transformers leaves empty when the
model fits on one device — it reported "(unsharded)" whether or not anything
was wrong, so it could never fail. It now reads parameter devices directly.
Model-agnostic lessons promoted to the quant playbook (new 3.12).
Flip the Cold-Fusion in-flight line to thesis-proven: L35 quantized to mixed
NVFP4, MTP acceptance 59.1% median beats the incumbent Heretic graft's ~47%,
abliteration survives quant. Not cut over — cutover is a separate operator
decision. Records the two env foot-guns hardened (quant venv config-delegation
drift; single-file no-index quant needs a header-built index).
Flip the in-flight status from 'capture done, calibration expansion next' to
'landed, works'. New detail file captures the three corrected diagnoses (layer-
selection metric, sharding/allocator misdiagnosis, corpus-size falsified) and the
verify/quant work still owed. Supersedes the -capture.md detail file's framing.
Captures the session's real work as the in-flight focus: abliterating
DavidAU Cold-Fusion with the Robinson formula. fp32 capture succeeded
(finite direction, layer 22, sink-clean) but two-template agreement is
0.59 vs Robinson's 0.99 — calibration-set expansion is the next step.
New detail file records the full saga including the transformers/DeltaNet
bf16-NaN fight (fp32 fix, the causal-conv1d kernel gap, the seat-restart
VRAM-greed gotcha). Supersedes the earlier "watch for DavidAU's heretic
build" posture — we abliterate it ourselves.
Auto-archived 4 closed entries (Recent decisions: Booth-3-features 08-05,
worldtree-sdk 07-31; Tried and abandoned: containerd-race 08-03,
mv-rename 08-02) to archival-memory.md; the rest of the over-cap entries
are held back by the <14-day and open-deferred guards. Index 331 -> 327.
Corrects three claims that had been standing in the fleet IPv6 notes and that
sent a three-arm research effort after a problem that did not exist:
- ESH was recorded as having no IPv6. It has a /56 delegated and a routable
WAN GUA -- substantially more prefix than NH3's single /64.
- The mesh was recorded as broken by ESH's CGNAT. It is not and was not down;
ESH is outbound and working. CGNAT on v4 alongside generous v6 is just the
modern ISP pattern, not an outage.
- IPv6 was framed as the escape hatch for that outage. The actual plan is that
IPv6 carries tunnel ENDPOINTS for Site Magic and WireGuard, and LANs are not
numbered in v6 at all.
NH3 internal v6 was brought up on the delegated /64 and verified end-to-end
(global GUA on nh3-docker, zero loss to Cloudflare and Google v6, un-NATed
source address seen from outside), then reverted on operator direction: one
/64 lights exactly one VLAN and that is not worth the split-brain.
The AT&T prefix research is kept as reference rather than deleted -- the /60
is real but undelegatable, the living multi-prefix mechanism is multiple
IA_PD in one solicit rather than the VRRP/multi-MAC recipe we were handed,
and the UDM SE can express neither. That is the answer if NH3 LAN-side v6
ever earns a maintenance window; it is not on any critical path now.
esh-pve hard-froze at 03:34 on 2026-08-19 and stayed frozen ~4.5 hours
until a manual power cycle. No panic, no OOM, no MCE — the journal stops
mid-operation. The whole ESH site lost DNS with it, because esh-userland
(VLAN 10, the PVC SSID and wired userland LAN) was handed exactly one
resolver: 10.0.50.45, AdGuard on esh-docker-vm, on a different VLAN, with
no secondary. Internet and routing were healthy throughout.
Two fixes.
1. DNS: 10.0.10.1 (the gateway, verified resolving) added as secondary on
esh-userland via the UDM Classic API. Note this is degradation cover,
not clean failover — clients that query resolvers in parallel will
bypass AdGuard for a share of lookups.
2. Watchdog: softdog -> iTCO_wdt under systemd (RuntimeWatchdogSec=60),
watchdog-mux masked. The box looked watchdog-protected and was not: a
software watchdog cannot fire when the kernel it lives in is wedged,
and watchdog-mux only pets the device while an HA client is connected,
which never happens on a cluster with no HA resources. Firmware does
not block the TCO timer here, checked before committing to it.
Also pins VM 102 off (onboot: 0). It starts with full GPU passthrough and
vfio-pci enabling that device is the last thing the kernel logged, 39
minutes before the freeze. The other suspect is the kernel itself: the
host ran 4.5 months on 6.8.12-16, took 6.8.12-42 in an apt batch on
08-18, and died 20 hours into the first boot on it. 6.8.12-16 is still
installed as the rollback.
The playbook is idempotent — a second run skips all six steps and passes
all six verifies. The watchdog is confirmed armed (identity=iTCO_wdt,
state=active, held by PID 1) but has NOT been observed firing; proving
that needs a deliberate wedge.
Memory also corrects two wrong mid-incident calls: the mgmt VLAN is
routed over the site tunnel and is not firewalled off — both symptoms
were the dead host generating ICMP unreachables.
One update.sh run on irv-ml1 carried both open upstream PRs, per the
operator's green-light on the job-store fix:
- #5 (464dfc2) declares cupy-cuda12x[ctk] on the gpu extra and takes uv
out of the render path (sys.executable -m waterland.cli), retiring the
runtime prune trap at the source.
- #6 (b72425b) rehydrates the job index from the data volume at startup,
fixing the unbounded store growth reported from this side.
Verified after the update rather than assumed: healthy on backend cupy;
/api/jobs went 1 -> 16 against 16 directories on disk, so API and volume
agree for the first time; nothing wrongly reclaimed, correct since 16 is
under RETAIN=40 and adoption only makes them visible; a real 256^2 plate
render completes warm, so the kernel-cache volume survived the image swap.
A subsequent render took both counts to 17.
The image keeps its explicit [ctk] install and UV_NO_SYNC/UV_OFFLINE pins
even though both are now redundant. The header requirement is a property
of this slim base, not of the upstream extra, and the cost is measured
rather than assumed: uv sync satisfies it first, so the line reports
"Audited 1 package" and adds 0.3s to the build. The env pins are now
cheap defence-in-depth against any future path that re-enters uv.
Docs corrected in place: the README's upstream-finding section is now a
resolved-finding record, and the two "bounded ~500 MB" claims say which
commit made that bound hold across restarts rather than only within a
process. Comment-side changes pushed to the live compose dir; no restart
was needed for them.
waterland-dev merged PR #5 (main now 464dfc2), fixing both landmines at
source: the gpu extra declares cupy-cuda12x[ctk], and the renderer spawns
sys.executable -m waterland.cli instead of re-entering uv mid-job.
The running container deliberately stays on 8025366. Its own [ctk] install
and UV_NO_SYNC/UV_OFFLINE pins already neutralise both defects, so a rebuild
would buy reliability that is already present — and the project is in
wind-down. Both guards are kept rather than dropped: the header requirement
is a property of this slim image, not of the upstream extra, and the uv pins
are now cheap defence-in-depth against any future path that re-enters uv.
Also records waterland-dev's confirmation of the unbounded job-store growth
and the operator's green-light on their startup-rehydrate fix. That PR
merging is the rebuild trigger: one update.sh run lands the rehydrate and
464dfc2 together. Marks the inbox drained.
Captures a long infra session: fleet *.internal DNS (git-sourced, 42 names,
three resolvers including a new colo one), waterland studio containerised on
irv-ml1, Homepage cleaned up and themed with Australis Skyfall over an
Arbo-generated background, and four unmanaged stacks adopted into stacks/.
Four detail files added. Auto-archived 4 entries to archival-memory.md
(Recent decisions 2, Tried and abandoned 2); 5 held back by the open-deferred
guard rather than moved.
Also records three operator-owned open items: the colo DNS repoint, the
static-v6 convention, and the deliberately belayed AI-tab Dormant regrouping.
Session captured: ESH cut over to Cityside 2Gb symmetric fiber and was
fully provisioned on it; esh-pve-nas completed its ZFS-root migration off
the USB DOM and took its 225-package security backlog with the reboot
deferred; ESH<->colo IPsec was rebuilt as a dialup tunnel with NAT-T after
CGNAT broke the statically-pinned one; IPv6 was mapped across all three
sites.
Auto-archival fired at the soft cap: 7 entries moved to
archival-memory.md (Recent decisions 3, Tried and abandoned 4), all
verified-complete arcs, with two detail files moved and removed. The
remaining pre-Aug-05 entries were held back by the open-deferred-work
guard, so the index stays slightly over cap at 313 lines rather than
losing a live pointer.
Also records the one self-inflicted outage of the session (missing
--make-rslave on a chroot rbind) and that three long-dead things
surfaced incidentally: pvestatd down 82 days, a vzdump hung 126 days,
and a VM sitting in prelaunch for four months.
Operator will retrieve the BGW210 Device Access Code from the NH3 office
and vault it, after which the IPv6 LAN settings page can be driven
remotely.
Parked on the henge as reclaim-nh3-s-7-unclaimed-ipv6-64s-from-the with
everything needed to resume cold: the verified facts about the /60 split
and the seven unclaimed prefixes, why AT&T cannot fix it, the exact page
to start at, what to look for in priority order, the other settings
pages behind the same login, and the wpa_supplicant fallback with its
warning about modifying NH3's only uplink.
Suggested vault path unifi/bgw210-nh3-device-access-code, matching the
existing unifi/* credentials.
Operator suggested checking the BGW on its 192.x management address,
which turned out to give the whole picture from unauthenticated status
pages.
The CPE is a BGW210-700 on firmware 4.28.7 at 192.168.1.254. AT&T does
hand it a /60 -- c110 through c11f. The BGW keeps c110-c117 for itself
and re-delegates up to eight individual /64s on c118-c11f, top-down. Our
UDM holds c11f, delegation number eight.
So the earlier conclusion that AT&T only grants a /64 was right about
the symptom and wrong about the cause. Seven further /64s are available
and simply never solicited, because UniFi exposes a single
wan_dhcpv6_pd_size integer with no field for how many prefixes to
request. The documented workaround is repeated -P flags to dhclient,
which the UniFi UI cannot express.
This also settles that an AT&T ticket cannot help: the rationing is CPE
firmware behaviour, not provisioning. Records the two real options --
accept one /64, or bypass the BGW entirely with wpa_supplicant EAP-TLS
on the UDM to negotiate the full /60 -- with the warning that the latter
modifies NH3's only uplink and needs a planned window.
AT&T support guessed 'I do not believe att will do that' from a DNS
provisioning desk. The guess was correct, but it needed proving rather
than accepting, so the UDM solicited DHCPv6-PD at /48, /56 and /60. All
three returned the same single /64. This is not a case of nobody having
asked -- the ask was made three ways.
Proven by temporarily enabling PD on nh3-iot, the only NH3 VLAN with
zero clients, then setting ipv6_pd_prefixid to 0, 15 and 16. All three
returned an identical c11f prefix, which only happens when exactly one
/64 is delegated; with a larger block the prefix-id moves the LAN within
it.
Records a mistake worth not repeating: I first read the gap between the
WAN address (c110) and the delegated prefix (c11f) as evidence of a /60.
It is not -- AT&T assigns those from different parts of their pool and
the spread means nothing.
NH3 was fully restored afterwards, with rollback artifacts kept. Also
captures the concrete ask for AT&T Business, phrased as something their
provisioning team can verify against their own DHCPv6 logs, and the
consequence if refused: NH3 can host exactly one v6 segment against
ESH's 256.
Operator opened a ticket with Cogent for v6 at Anaheim, which closes the
one thing tonight's investigation could not resolve from our side.
Captures the diagnosis so the ticket has evidence behind it: a single RA
in 90 seconds of sniffing wan1, from fe80::ea0a:b9ff:fe3b:2c16, proving
an IPv6-capable router sits one hop away on the circuit terminating
38.120.12.42/29 -- but SLAAC obtained no global address across multiple
RA intervals and ping6 to Cloudflare and Google both returned 100% loss.
Router present, circuit unprovisioned.
Also records that the FortiGate v6 config was fully reverted after
testing, the FortiOS gotcha that SLAAC is 'set autoconf enable' rather
than an ip6-mode, and the ask to make when it lands: a /56 or better,
since NH3 only receives a single /64 from AT&T.
Notes the consequence worth planning around -- once provisioned, the
colo becomes the only site with both a static public v4 and routable
v6, which makes it the natural v6 hub given ESH is CGNAT'd and NH3 is
prefix-constrained.
I wrote that enabling SLAAC would give LAN devices 'globally reachable
addresses'. Wrong word, and the wrong word in a persistent-memory entry
a future session inherits as fact.
Addressable is a property of the address. Reachable is a policy decision
the firewall makes. v6 removes NAT; it does not remove the firewall, and
treating those as the same thing is exactly how v6 gets mischaracterised
as automatic exposure.
Records the operator's position while correcting it: no 1:1 inbound
pass-through. The pending firewall-policy pass is about writing explicit
default-deny inbound rules per v6 segment, not about deciding what to
expose.
esh-iot keeps the identical eight digits -- 4DBAD107, rendering
4dba:d107 -- and only the reading changes: 4 is FOR rather than A, so it
parses 'FOR DA BAD IOT', which describes what the segment is actually
for.
esh-mgmt genuinely changes: 115D:B055 becomes 15DA:B055. The leading I
is dropped and DA is spelled in full, giving 'IS DA BOSS' with the
network as subject rather than speaker. Still eight digits. DA written
out needs no substitution since D and A are both native hex; spelling it
as a single D the way esh-iot does would have yielded seven digits and
broken the house pattern.
'FOR ALL DA BAD' -- 4=FOR, 411=ALL, D=DA, BAD=BAD. Eight digits, house
style, renders 4411:dbad.
No DMZ network exists on the ESH UDM today; this is a name claimed
against the day one is built. Pairs deliberately with esh-iot's
4DBA:D107 -- IoT is 'a da bad', the DMZ is 'for all da bad', which is
the correct relationship between the two segments.
'FOR ALL BIOS' -- 4=FOR, 411=ALL, B105=BIOS. Eight digits like the rest.
Completes the set. All six ESH networks now carry an 8-hex-digit phrase
in a consistent first-person/declarative voice:
Default 4BA5:3417 A BASE FOR IT
esh-mgmt 115D:B055 I IS DA BOSS
esh-server 4411:B105 FOR ALL BIOS
esh-userland CAFE:4411 CAFE FOR ALL
esh-iot 4DBA:D107 A DA BAD IOT
esh-cameras 1533:FACE5 I SEE FACES
Still a documentation convention rather than wire-level configuration --
UniFi has no static-v6 client assignment and the gateway address is
platform-fixed -- but these are the values to use whenever ESH LAN v6 is
switched on and hosts get hand-assigned addresses.
Same phrase, 'A BASE FOR IT', but written as a plain 8-digit string
rather than forcing the article into its own group. 4=A, BA53=BASE,
4=FOR, 17=IT renders as 4ba5:3417 -- two groups, matching every other
network in the scheme, with the words straddling the colon exactly the
way 4DBA:D107 does.
Corrects the previous commit, which claimed this needed nine digits and
a third group. It is eight, and always was.
'A BASE FOR IT' -- the article makes it a full sentence, matching the
voice of the other five. BA53 uses 3=E rather than the 5E spelling used
in the previous BA5E version.
Nine hex digits rather than eight, so unlike the others it does not fit
two groups and renders across three as a:ba53:0417.
'BASE FOR IT' -- BA5E=BASE, 4=FOR, 17=IT. The foundation segment
everything else hangs off, which is what the default network is, and it
doubles as 'base for IT'.
Note the trailing group zero-pads: it renders as ba5e:0417, not
ba5e:417.
'CAFE FOR ALL' -- CAFE, 4=FOR, 411=ALL. Eight digits like the others,
and the only one so far that splits on its own phrase boundary, so it
renders legibly as cafe:4411.
Bonus reading: 411 is US directory assistance, which is a fitting second
joke for the segment the humans actually live on.
'A DA BAD IOT' -- 4=A, D=DA, BAD=BAD, 107=IOT. Eight hex digits to match
the cameras and mgmt picks.
Worth noting it renders as 4dba:d107, so unlike the other two the phrase
does not split on its word boundaries and reads as noise unless you know
it is there -- which is arguably right for the untrusted segment.
'I IS DA BOSS' -- 1=I, 15=IS, D=DA, B055=BOSS. Eight hex digits like
the cameras pick, so it renders as 2607:73c0:402:1d??::115d:b055 with
the same two-group split and room for host numbering.
Pairs structurally with 1533:FACE5 on cameras: both eight digits, both
first-person, and the network that is actually in charge gets to say so
to the one that is merely watching.
'I SEE FACES' -- 1->I, 5->S, 3->E, 3->E then FACES. Eight hex digits
splitting cleanly across two groups, so it renders as
2607:73c0:402:1d00::1533:face5 with room left for host numbering.
Still a documentation convention rather than anything on the wire, per
the constraints recorded in the same entry, but this one is good enough
that it should survive to whenever ESH LAN v6 actually gets switched on.
Picked six hexspeak names for the ESH LANs during a wind-down moment
(FACE/B055/B105/CAFE/DEAD/BASE), then checked whether any of it could
actually land on the wire before implementing anything.
It can't, for three independent reasons: a network's only nameable slot
is its /64 prefix id, which is 2 hex digits and can't spell a 4-char
word; the gateway's own address is fixed at ::1 by the UniFi platform
with no field to override it; and UniFi has no IPv6 equivalent of
use_fixedip/fixed_ip, confirmed directly against the client schema, so
individual devices can't be pinned to a chosen v6 address either --
SLAAC devices self-assign via EUI-64 or privacy extension.
So this stays a documentation mnemonic. Recorded as such rather than
implied as something live, since I'd already started suggesting a
static-camera-assignment plan that the schema check ruled out.
The ESH<->colo tunnel is restored and the FortiGate end is permanently
address-agnostic, but the UniFi end still needs a literal ipsec_local_ip
and so drops on any ESH WAN change -- Cox reclaiming WAN1, the fiber
cutover, or a DHCP renewal.
Operator's call not to build the self-healing watcher yet, which is
right: it would be written against the 5G failover address, which is
about to be replaced, and the fiber may reshape the topology anyway.
Parked as self-healing-ipsec-local-ip-watcher-for-the-esh with the
trigger recorded, plus the follow-up to retire the old ana-to-eshudm
tunnel whose distance-10 route would otherwise reclaim traffic if Cox
returned on the old address.
Records the manual stopgap in persistent memory so the gap is cheap to
cover by hand in the meantime: read wan_ip from the UDM's health
endpoint, PUT it into esh-ana.ipsec_local_ip.
The link died when ESH lost its public IP during the fiber cutover. Two
independent causes, and the second would have defeated the obvious fix:
- phase1 ana-to-eshudm was type static, pinned to 70.181.90.232, an
address that no longer exists.
- nattraversal was disable, so ESP could not have crossed NAT even with
the peer IP corrected. pfi-ana-nh3 shares that setting and survives
only because NH3 is publicly addressed, which is why the two tunnels
diverged.
FortiOS refuses `set type dynamic` on an existing tunnel -- "Cannot
change tunnel type once configured" -- and rolled back cleanly, so the
fix could not be an edit. Rather than delete and recreate, which
cascades into the phase2, two static routes and ten policies, the
replacement was built alongside: new phase1+phase2 ana-eshudm-dyn
(type dynamic, ikev2, aes256-sha1, dh14, NAT-T on, PSK read from the ESH
UDM API so neither side needed a new key), static route id 10 at
distance 20, and two consolidated multi-zone policies 73/74. The old
tunnel is left in place, dead and harmless, as rollback.
Verified up: ana-eshudm-dyn_0 97.170.236.56:4500 selectors 1/1 -- the _0
suffix is a dialup child, :4500 is NAT-T, and the address is the
carrier's, which is precisely what could never have been pinned. ESH
reaches all four colo hosts at 40-56ms, the colo reaches all three ESH
hosts, and traceroute drops from eight hops leaking into the carrier
network to three hops fully encapsulated.
Config was backed up before any write (1.17MB, 36903 lines, off-box).
Residual fragility recorded: the UDM's ipsec_local_ip demands a literal
address -- empty is rejected as api.err.InvalidPayload -- so it still
needs updating when the fiber changes ESH's WAN address. The gateway end
is now address-agnostic; the UniFi end is not.
The prior edit missed its anchor and left the over-broad version in
place. The entry now separates the two inter-site links rather than
treating them as one: Site Magic (WireGuard, NH3<->ESH) survives
arbitrary NAT and is proven to; IPsec (colo<->ESH via ana-gw) does not
and is currently down, with traffic leaking unencapsulated to the
carrier. IPv6 keeps its justification on the IPsec link specifically.
The fleet IPv6 work was justified primarily by the expectation that ESH
fiber landing behind CGNAT would break Site Magic on IPv4, making v6 the
escape hatch. The fiber cutover provided a free natural experiment and
the premise does not hold.
Cox was unplugged, ESH failed over to the 5G WAN (already configured
failover-only, so this needed no intervention), and the resulting WAN
address is 192.168.200.111 -- RFC1918, double-NAT, no inbound path at
all, which is strictly worse than the CGNAT that was feared. Site Magic
stayed up throughout: all four ESH hosts reachable, ssh and command exec
working, 20MB pulled over the tunnel, latency 15ms -> ~46ms as expected
for cable to 5G.
The mechanism is visible on the device: magic_site_to_site_vpn holds
only `enabled` plus a WireGuard keypair, with peer orchestration in the
UniFi cloud and no WAN binding of any kind. NH3's edge is publicly
reachable, so the NAT'd side dials out and never needs reachability.
Consequence: no addressing outcome on the new fiber -- public, CGNAT or
double-NAT -- threatens the inter-site tunnel. IPv6 stays worth doing on
its own merits but stops being urgent, and stops gating anything.
Also worth recording that Site Magic cannot be pinned to a WAN. It rides
whichever uplink is active, so the only lever is failover priority --
which moves all site traffic, not just the tunnel. The existing
failover-only config on WAN2 already handles a primary-WAN outage
correctly and needed no change.
pve-manager 8.4.11 -> 8.4.20, corosync 3.1.9 -> 3.1.10-pve2, and kernel
6.8.12-42 staged on the /boot LV. dpkg clean, nothing outstanding for
apt -f install, all PVE services active, cluster quorate, no unapplied
conffiles. Reboot deliberately deferred at operator request, so the host
still runs 6.8.12-13 until a chosen window.
This validates the GRUB fix from 061c4b7 under the exact condition it
was written for. update-grub regenerated entries for the new kernel and
entry 0 -- what GRUB_DEFAULT=0 selects -- is now
/vmlinuz-6.8.12-42-pve with root=ZFS=nvme/ROOT/pve-1, supplied by the
grub.d drop-in since grub-mkconfig cannot derive the pool name itself.
The old kernel keeps correct entries as a fallback and the ext4 rollback
entry is untouched. Had the fix not landed first, saved_entry would
still be pinned to 6.8.12-13 and the host would boot the old kernel
indefinitely -- 161 security updates installed and never run.
/boot holds both kernel sets at 176M used of 488M, confirming the 512M
LV carved out of swap was sized correctly.
Adds a ZFS snapshot step to the upgrade playbook, taken automatically on
ZFS-root nodes before any package lands. That is the first real use of
the boot-environment upside the migration was meant to unlock: rollback
for this upgrade is now `zfs rollback -r
nvme/ROOT/pve-1@pre-upgrade-20260818T141652Z && reboot` rather than
archaeology in dpkg. Also documents that the corosync bump restarts
corosync mid-upgrade, which on a 2-node cluster is a brief quorum event.
Root is now nvme/ROOT/pve-1. The USB DOM keeps the ESP and /boot but is
out of the runtime I/O path, so a bus reset can no longer drop root from
under a running hypervisor. All five guests healthy, three pools ONLINE,
system running, ext4 pve-root intact and unmounted as the rollback with
its own kernel and initrd. zfs-import-cache is now the active import
path -- the all-three-pools cachefile fix doing its job.
The window cost an unplanned outage, and the cause was this repo's own
tooling rather than the migration.
The staging chroot ran `mount --rbind /dev` and /sys with no
--make-rslave. On systemd `/` has shared propagation, so the cutover's
`umount -R` propagated back into the live host and removed the real
/sys/fs/cgroup, /dev/pts and /dev/shm. With cgroup2 gone systemd-logind
could not create a session: ping fine, TCP fine, SSH authentication
succeeded, resident daemons kept serving -- and every new exec hung,
including /sbin/reboot, so the reboot never ran at all.
It impersonates failing root-disk I/O almost perfectly, and I called it
as the DOM dying. That was wrong. dmesg had the answer throughout: the
DOM attached cleanly with no errors, and the last log timestamp was
12114881s -- 140 days -- meaning this was still the original boot. A
down-detector had also never reported the host down, which I read as a
fast reboot rather than as no reboot.
Fixes and guards:
- --make-rslave after every rbind, plus a guard that refuses to proceed
while any chroot bind still reports shared propagation.
- Confirm a reboot by observing the host DOWN, not by watching for it to
come back. Those two states are indistinguishable otherwise.
- Blast radius now measured from the server: `ss` inside CT 103 found
five NFS clients, not the two documented. The new one that mattered is
esh-vm-db, hard-mounted and unreachable by ssh. Left mounted on
purpose and it came through read-write.
- grub-reboot's one-shot does NOT work here: grubenv sits on an LVM LV
which GRUB reads but cannot write, so next_entry survived the boot
that consumed it. Steady state is saved_entry=pve-zfs-root with no
next_entry. There is no auto-fallback on this host and no IPMI.
Recovery needed no console: an idempotent cgroup2/devpts/shm remount
landed in the brief windows where exec succeeded. No data was lost, and
neither the DOM nor any pool was ever at risk.
Everything but the reboot. Two rerunnable elway playbooks; the host is
still running from the ext4 root and its boot path is byte-identical to
the last 140 days, because grub-install is deliberately held back to the
cutover window.
Phase 1 (esh-pve-nas-stage-zfs-root.yaml): carve a 512 MB /boot LV out
of the 768 MB swap LV, populate it, rsync the 4.3 GB ext4 root into
nvme/ROOT/pve-1, write the copy's fstab.
Phase 2 (esh-pve-nas-stage-bootloader.yaml): ZFS initramfs, grub.cfg,
explicit pve-zfs-root and pve-ext4-rollback entries with stable ids,
grubenv pinned to the rollback so cutover's grub-reboot is a one-shot.
Three landmines the plan did not predict, all caught by verify steps
asserting effective state rather than by reading the plan:
- The /boot LV had nowhere to live. VG pve had 4 MB free and mounted
ext4 cannot shrink; freeing space from root needs a rescue boot, which
costs the one-reboot property. Space came from swap (768M -> 256M).
- The runbook's `zpool set cachefile=... nvme` would have broken the
NAS. Populating a cachefile flips the host from import-by-scan to
import-by-cache, so a one-pool cache leaves ssd and tank unimported --
and CT 103 esh-nas has twelve bind mounts spanning all three pools.
Set on all three instead, verified in the resulting cache.
- update-grub silently emitted a pool-less root=ZFS=/ROOT/pve-1, which
boots to an initramfs prompt. Debian's 10_linux builds ${rpool}${bootfs}
and rpool comes from grub-probe --target=fs_label, which returns empty
because GRUB's ZFS reader cannot open a pool with encryption,
large_dnode and zstd_compress -- the same feature set that forced /boot
to stay ext4. The probe failure is swallowed by `2>/dev/null || true`.
Fixed with a /etc/default/grub.d drop-in plus explicit menu entries.
The transferable lesson: the original verify grepped for the correct
root= string appearing somewhere in grub.cfg, which passes while every
menu entry is still broken. Assert the effective value, not the presence
of a substring.
Lobe's TTS had never worked. It sends model:"tts-1" and LiteLLM resolves the
model name before routing, so it 403'd against the scoped key's allow-list and
never reached :8198 -- our belief that an unknown model routes to the gateway
default was true of the gateway and false of the LiteLLM path, which is what
hid it.
Fixed at the gateway rather than the client: tts-1, tts-1-hd and gpt-4o-mini-tts
aliased to the same upstream as ext-tts, and added to the lobe-chat-esh
allow-list. Verified with Lobe's exact payload on Lobe's own key. The obsolete
'one-time human UI pass' follow-up is dropped.
Banks two durable facts: those aliases are independent DB rows that must move if
ext-tts repoints, and the infra-ops key has admin rights for /model/new and
/key/update so this class of work does not need sk-corvid.
Records the one signal the synthetic gates cannot provide -- multi-turn
degeneration is stochastic and invisible to probes, and four synthetic tests
once validated three non-fixes on this exact seat.
Not yet the 60k-token bar the prior seat cleared, so the rollback weights
stay in place.
vh