Live GEN_MODEL is now qwen38-27b-coldfusion-h300-nvfp4-mixed (ana-ml2 GPU0
:8015). Served-name left unchanged so all 7 LiteLLM aliases route without a
gateway edit.
Verification:
KV pool 401,550 tok / 1.53x (baseline 403k / 1.54x)
LiteLLM aliases 7/7 green
vision 3/3 shapes, colour+form+position correct
MTP acceptance 59.7% median @ 118.37 tok/s
quality gens 4/4 correct
abliteration 4/4 compliance
PPL NOT measured (see below)
The roadmap predicted ~47% acceptance for a pristine MTP graft versus L35's
59.1% in-band edit. Measured 59.7% on the same harness: there is no acceptance
penalty, which removes the throughput argument for reimplementing MPOA.
A single long-prose generation read 47.5% off the same counters -- below the
8-run minimum of 49.0% -- and would have "confirmed" the prediction by
coincidence. Acceptance must be read from quickbench.py, never one sample.
PPL is blocked on VRAM, not on the model: eval_quality.py aborts with
"prompt_logprobs look uniform" under --speculative-config, and the probe-seat
workaround needs ~22 GB while both cards sit at ~96% committed.
Also normalizes the quant dir from root:0600 to llmuser:llmuser 0664 to match
every other model dir, and records that config.json sha256 is byte-identical
across the h300 and L35 quants and is therefore useless for confirming which
weights are mounted (mtime and a head-hash are the discriminating views).
Rollback is one line to .env.bak-pre-h300-20260820.
8/100 refusals at KL 0.0136, hand-verified coherent, beating the absolute-heresy
bar 3.6x. NVFP4 quant complete: 21 GB, 1968 tensors, MTP head grafted back after
PR #317 dropped it, and re:^mtp.* re-injected into quantization_config.ignore
after llm-compressor pruned it. Self-harm guardrail is gone on this build and is
the operator's own next work item; the four-dwarf panel is stood down.
Ran Heretic v1.4.0's 300-trial TPE search on Cold-Fusion. Best trial scores
8/100 refusals at KL 0.0136 against a 98/100 base, versus absolute-heresy at
29/100 and our hand-tuned Robinson L35 at 72/100 / KL 0.0116 — i.e. 64 fewer
refusals for the same damage. Hand-verified coherent: correct arithmetic with
shown working, clean code, 66-167 word prose across nine probes.
Durable findings:
- direction_scope=0 (single shared direction) is decisive on this merged base:
n=129, best 8/100. Per-layer directions n=131 never beat 52/100 despite a
better median. Points against the multi-direction intuition for a diffuse
direction (our two-template |cos| is 0.62 vs Robinson's 0.99 on stock).
- Aggression is not the lever. r(KL, refusals) = -0.561 over 261 trials; the
KL<0.02 band contains both the worst results (median 87/100) and the single
best. A KL 0.3554 trial scored worse than one at 0.0193.
- PR #317 confirmed: Heretic silently drops the MTP head on save. Source 1199
tensors -> export 1184, all 15 mtp.* gone, vision 333/333 intact, exit 0, no
warning. This is also why absolute-heresy ships a byte-identical MTP head —
a bug, not a design choice. Always diff tensor keys after a Heretic export.
- Heretic's recovered direction carries 6.18% of its energy in sink dim 3994,
versus 0.094% for our L35 and 1.97% for the L39 we rejected as brick-inducing.
It survives that only because of magnitude-preserving ablation
(row_normalization=FULL); our plain projection has no such protection, so the
sink screen correctly refused the in-band MTP graft. Same direction, different
operation. MPOA is the prerequisite for in-band MTP on a Heretic trunk.
- Heretic's edit is recoverable from weights: delta is rank-1 (s2/s1 ~ 0.010),
SVD gives the direction, norms give per-layer weights (1.08 -> 1.34, i.e.
over-projection). Cross-layer |cos| agreement 0.9903 independently confirms
the single-direction result.
New tooling in services/coldfusion-abliteration/:
kl_divergence.py first-token KL, class-split, zero noise floor
catatonia_gate.py 12 probes x 220 tokens, prints every completion
heretic_export.py PTY driver; selects by measured value, never by menu
position — Heretic's resume prompt puts "delete the
checkpoint and all results" one arrow-key from the target
graft_mtp.py recovers the trunk direction by SVD; --pristine for the
safe path when the sink screen refuses
Also adds quant playbook 3.13: the NVFP4 recipe sets observer="imatrix_mse" but
llm-compressor has always silently fallen back to uniform MSE for want of
importance data — on this build and on the incumbent. Existing A/B comparisons
stay valid since every build shares the fallback. Parked as id 42.
Guardrail note: this build has lost the self-harm guardrail that the Robinson
L35 build retained. Restoration is the operator's own work item.
vLLM sizes --gpu-memory-utilization against total VRAM but gates startup on free
VRAM, so the GPU0 pair coexists only in its original boot order. Records the
restore sequence (meromero to healthy first, then gen), the observed-not-slept
rule, and the KV-pool baselines to verify a restore against — nvidia-smi used-MiB
is the wrong check, it swings ~7 GB on allocator slack at identical capacity.
Restoring the two GPU0 seats with `start meromero; sleep 10; start gen` put
meromero into a 7-restart crash-loop:
ValueError: Free memory on device cuda:0 (35.3/94.97 GiB) on startup is less
than desired GPU memory utilization (0.52, 49.38 GiB).
The previous commit's README claimed restore order "is not actually load-bearing"
on the grounds that both seats pass --gpu-memory-utilization as a fraction of
total VRAM. That is half right and the wrong half mattered: the fraction sets the
target, but vLLM gates startup on FREE VRAM and refuses to start unless the whole
target is available. GPU0 runs at ~96.4/97.9 GB with roughly 0.4 GiB of slack, so
the seats coexist only in the order they were originally brought up, and meromero
is the one that does not fit in the remainder. The pre-existing auto-memory note
("gen takes a fraction of free VRAM at startup and will starve meromero") was
pointing at the real effect.
Also: "first" means healthy, not ten seconds earlier. A sleep 10 against a
two-to-three minute weight load is simultaneity, not ordering — gate on observed
state.
Recovery applied: stop gen, wait for meromero healthy, start gen. Verified
against the pre-window baseline rather than against "both green":
gen KV 14.36 GiB / 403,065 tok / 1.54x -> 14.34 GiB / 401,550 tok / 1.53x
meromero KV 542,202 tok -> 542,202 tok
RestartCount 0 on both; summarizer smoke-tested through LiteLLM
Note for the next reader: raw nvidia-smi used-MiB is the wrong check here. It
reads 89,503 now vs 96,376 before, which looks like a 6.9 GB regression and is
allocator slack — serving capacity is unchanged. The anomalous boots were the
high ones (34.95 GiB KV), where gen came up on an empty card mid-window.
Adds `kl_divergence.py`: first-token KL(stock || abliterated) over the full
248,320-token vocabulary, bf16 vs bf16, scored separately for held-out harmless
and reserved-harmful prompts.
Result (L35, 256 harmless / 104 harmful, answer mode):
harmless median 0.0211 mean 0.0364 top-1 agreement 89.8%
harmful median 0.5996 mean 0.6992 top-1 agreement 55.8%
selectivity 28.4x (72.8x in think mode)
Self-KL noise floor is exactly 0.0, and all 720 per-prompt values are
bit-identical between a single-process and a two-process run, so the figures are
signal rather than bf16 jitter. Reverse KL on harmful/answer is 1.43 vs forward
0.70 — the mass-where-stock-had-none asymmetry expected of a refusal-direction
removal. Against the Heretic reference figures (0.1191 prior seat, 0.0759 the
live absolute-heresy seat) this is materially gentler, but those are the other
tool's optimizer output on a different base with its own harmless set and
template — order-of-magnitude, not head-to-head. KL remains a fidelity number;
the viability gate is still MTP acceptance (59.1%).
Method notes:
- Prompt classes are reported separately by design. A single averaged KL over a
mixed corpus is close to meaningless, since the metric is meant to be large on
harmful prompts and small on benign ones; the ratio carries the information.
- The harmless evaluation set is drawn from the alpaca pool minus calibration's
own draw, reconstructed by replaying that draw rather than remembered, and
asserted disjoint on text. The harmful set is the reserved test split.
- `render` is imported from abliterate.py rather than copied, so the measurement
cannot drift from the rendering the direction was captured against.
- Batch size 1 with logits_to_keep=1: no padding semantics, ~0.6 MB of logits.
Three corrections to the runbook, each of which cost time:
- "bf16 is 50 GB, only gen must go" was 50.10 GiB mislabelled. Text-only weights
are 51,300 MiB; freeing either GPU0 seat alone leaves ~50,933 MiB. Both must
stop. VRAM is now sized from the safetensors headers at run time.
- A 27B model cannot be released in-process: `del` + gc + empty_cache left free
VRAM at 45,287 MiB, and so did confining the model to an inner frame that
exits. Only process exit returned the card (96,689 MiB). The first run
completed only because the allocator hit OOM, collected, and retried. Each
model now gets its own process, handing log-probs to disk between stages.
- The residency gate read hf_device_map, which transformers leaves empty when the
model fits on one device — it reported "(unsharded)" whether or not anything
was wrong, so it could never fail. It now reads parameter devices directly.
Model-agnostic lessons promoted to the quant playbook (new 3.12).
Flip the Cold-Fusion in-flight line to thesis-proven: L35 quantized to mixed
NVFP4, MTP acceptance 59.1% median beats the incumbent Heretic graft's ~47%,
abliteration survives quant. Not cut over — cutover is a separate operator
decision. Records the two env foot-guns hardened (quant venv config-delegation
drift; single-file no-index quant needs a header-built index).
Quantized the L35 abliterated model to mixed NVFP4 and measured MTP acceptance
end to end. The experiment's whole premise: Heretic (the incumbent gen seat)
leaves the MTP head a byte-identical base graft its wrapper never loads, whereas
Robinson abliterates the MTP head in-band — the question was whether that in-band
edit survives well enough to spec-decode. It does, better than the graft:
MTP acceptance 59.1% median (51-65%, 8 cache-busted topics) vs incumbent ~47%
decode 118.7 tok/s median (faster; image-confounded, read as not-worse)
abliteration survives quant (creative refusals drop, self-harm guardrail
intact, coherent)
Output at /tank/aimodels/qwen38-27b-coldfusion-L35-nvfp4-mixed (22.5 GB). Result
JSON in bench/. NOT cut over — the incumbent seat is untouched; making L35 the gen
seat is a separate decision needing the full Stage-3 gate + real multi-turn hold.
Two env foot-guns hardened along the way:
- quant_mixed_nvfp4.py now promotes text_config attention fields
(num_attention_heads etc.) to the top-level config for the oneshot, then
restores. transformers 5.10 / llmcompressor 0.12 (this venv moved under us
since the Aug-15 heresy quant) no longer delegate the top-level lookup, so
oneshot raised "Cannot determine num_attention_heads". Same "the fight is the
environment" pattern as the abliteration capture.
- a sub-~23GB quant saves as a single model.safetensors with no index, so the
post_quant MTP graft needed an index built first — from the safetensors header,
not safe_open (which mmaps the whole shard and ENOMEMs on ZFS).
post_quant grafted the abliterated MTP (15 tensors, 849 MB) and re-injected
re:^mtp.* into quantization_config.ignore (llm-compressor pruned it again — the
two-rounds-lost 0%-MTP bug, fired and repaired as designed). Probe served on the
pinned nightly (#51113 qwen3_5_mtp fix) to match the live seat's vLLM.
Flip the in-flight status from 'capture done, calibration expansion next' to
'landed, works'. New detail file captures the three corrected diagnoses (layer-
selection metric, sharding/allocator misdiagnosis, corpus-size falsified) and the
verify/quant work still owed. Supersedes the -capture.md detail file's framing.
The abliterated model works. A/B vs stock on a matched greedy battery: explicit
sexual + graphic torture (the measured stock refusal surface) go from refused to
complied/engaged, held-out AdvBench prompts loosen, the self-harm guardrail
survives, coherence intact — the Robinson design point exactly. Output at
/tank/aimodels/qwen38-27b-coldfusion-abliterated-L35-bf16, verified bitwise:
131/131 targets changed, 333/333 vision byte-identical (delta 0.0), 735/735
others untouched.
Getting there corrected three diagnoses the prior session had backwards.
1. The layer-selection metric was wrong, and that was the whole ballgame. The
recipe picks the abliteration layer by peak two-template |cos| agreement. On
this heavily-merged base that metric is anti-correlated with efficacy: its
argmax (layer 18) is the WORST-separating layer in the window (Cohen's d 5.51
vs 9.89 at the peak), and abliterating there was a measured behavioral no-op —
stock and "abliterated" refused all six probes identically. Cause: the two
renderings end in different generative modes (</think> vs <think>), so |cos|
scores answer-vs-reason mode, not refusal, and on a merge the mode term
dominates. Replaced selection with harmful/harmless SEPARATION (Cohen's d /
AUC of the direction's projection), gated on the sink screen since separation
and sink-energy both climb with depth. Picks layer 35 (d 9.35, AUC 0.9997,
sink 0.094%). Agreement is kept as a printed diagnostic.
2. The "bf16 NaNs, use fp32" rule was a misdiagnosis. The NaN was never
precision — it was multi-GPU sharding (the residual stream zeroes two layers
past the GPU0->GPU1 boundary; the first capture's layer 22 happened to sit in
the healthy region, which is why it looked fine) plus
PYTORCH_CUDA_ALLOC_CONF=expandable_segments (corrupts retained tensors; the
corruption MOVED between bit-identical forwards, the tell that it was memory
not math). On one GPU with a plain allocator, bf16 full-64-layer is exactly
deterministic and coherent, at 50 GB and 4.3x the throughput of the 111 GB
fp32 it replaced. Both defects are now hard gates (residency exit 8, allocator
exit 9); capture pins CUDA_VISIBLE_DEVICES=0.
3. The corpus-size hypothesis was falsified. 52x more calibration data (8->416,
mlabonne/harmful_behaviors = the recipe's actual AdvBench split, already on the
box) moved agreement 0.594->0.624 — nothing. Kept the 416/416 corpus anyway
(calibration.py); it gives the clean separation signal. The held-out 104-prompt
test split is reserved and asserted disjoint.
Also: the --out write is now shard-level surgery (reads/writes the 18 safetensors
directly, no model object, no GPU). This is correctness, not thrift —
AutoModelForCausalLM resolves to the TEXT model, so save_pretrained would drop all
333 vision tensors AND skip the MTP head (the in-band MTP edit is the entire point
of the Robinson formula). Neither failure raises. Shard surgery makes vision and
the other 1068 tensors byte-identical by construction.
Batched capture with a dtype-aware equivalence gate; hidden states captured via
forward pre-hook (reading output_hidden_states off the returned object is unsafe
here — buffers get recycled). Sharding/allocator lessons promoted to the
quantization playbook (model-agnostic, sections 3.9-3.11 + superseded table); the
selection-metric lesson added to the recipe doc.
The dead layer-18 no-op checkpoint was removed (52 GB, confirmed identical to
stock). Incumbent gen seat untouched. Full canonical refusal-probe re-profile and
MTP-acceptance-on-quant still owed before this becomes a gen-seat candidate.
The 8/8 calibration set gave |cos| agreement 0.594 against the recipe's 0.9925.
This wires in the corpus the recipe actually used and makes a capture at that
scale affordable.
Corpus (calibration.py, new). The recipe's "held-out train/test split of 416/104
with overlap 0" names mlabonne/harmful_behaviors exactly — 416 train / 104 test,
AdvBench-derived — and it plus harmless_alpaca were already staged in ana-ml2's
HF dataset cache. Read via pyarrow, no datasets dependency, no hub access.
Harmful is order-deterministic (no seed), so a re-capture is reproducible from
the flags alone. The 104-prompt test split is reserved as the held-out
generalization probe and asserted disjoint, so the post-write re-profile cannot
silently become in-distribution. --calib builtin reproduces the legacy run.
Batched capture. 832 prompts x 2 templates = 1664 forwards. Padding is on the
RIGHT: in a causal stack nothing after position t reaches position t, so
trailing pads cannot touch the token read, whereas left padding feeds pads into
the DeltaNet recurrence ahead of the prompt — the path whose torch fallback
already NaN'd once here. Means accumulate in float64; the direction is a
difference of means, which is where cancellation lives on this model.
Gates added, both protecting numbers rather than tensors:
- batch-equivalence: proves padded-batch == single-prompt (rel 1e-3) before
spending the capture window.
- surgery pre-check: aborts if any of the 131 targets is absent or on the meta
device. orthogonalize_ edits in place, and an in-place write to an
accelerate-offloaded tensor is a silent no-op — that ships a half-abliterated
model past a smoke test.
Fixed a reporting bug: the agreement line printed the global agree.max() beside
the window's argmax layer, so the first capture read as 0.8538 when the real
in-window number was 0.5944. The global peak sits in the early layers where the
dim-3994 massive activation inflates agreement for reasons unrelated to refusal.
Now prints window max, a top-5, and labels the global figure informational.
--max-layer truncates the decoder for capture. Exact, not approximate: a causal
stack's layer-N state cannot depend on layers above N, so any value above the
window top leaves the direction bit-identical while cutting fp32 residency and
forward cost. 46 drops 18 of 64 layers and is what keeps fp32 off CPU offload.
Refused on the write path, where it would emit a truncated checkpoint.
Verified on ana-ml2 without the GPU: dry-run still 1:1 (131 tensors, all
coverage gates), calibration loads 416/416 deterministically with its guards
firing, both --max-layer guards exit as designed. Also confirmed against
chat_template.jinja that enable_thinking=True does resolve reasoning_effort to
xhigh, so the two renderings are the recipe's — template selection was not the
cause of the low agreement.
The re-capture itself is unrun: it needs the fp32 VRAM window and therefore
production seat downtime.
Captures the session's real work as the in-flight focus: abliterating
DavidAU Cold-Fusion with the Robinson formula. fp32 capture succeeded
(finite direction, layer 22, sink-clean) but two-template agreement is
0.59 vs Robinson's 0.99 — calibration-set expansion is the next step.
New detail file records the full saga including the transformers/DeltaNet
bf16-NaN fight (fp32 fix, the causal-conv1d kernel gap, the seat-restart
VRAM-greed gotcha). Supersedes the earlier "watch for DavidAU's heretic
build" posture — we abliterate it ourselves.
Auto-archived 4 closed entries (Recent decisions: Booth-3-features 08-05,
worldtree-sdk 07-31; Tried and abandoned: containerd-race 08-03,
mv-rename 08-02) to archival-memory.md; the rest of the over-cap entries
are held back by the <14-day and open-deferred guards. Index 331 -> 327.
The --capture forward NaN'd repeatedly. Root cause: transformers' Qwen3.5
DeltaNet linear-attention needs the causal-conv1d fast-path kernel, which
can't be built here (no nvcc, no prebuilt wheel). Its torch fallback produces
nondeterministic all-NaN hidden states in bf16 -- same 11-token input finite
on one forward, NaN at layer 4 on the next. bf16 and fp32 share exponent
range, so it's precision-driven catastrophic cancellation, not overflow, and
fp32 resolves it.
Fixes:
- --capture now loads fp32 (the write/surgery path stays bf16 -- no forward,
no NaN). attn_implementation=sdpa pinned.
- A finite-gate aborts on a non-finite direction. The sink screen alone can't
catch this: nan > threshold is False, so a NaN direction "passed" it and got
saved silently on the first run.
Capture result (fp32, full GPU): refusal direction finite, unit-normed, layer
22, sink energy 0.0008% in dim 3994 -- clean, not sink-dominated. Saved.
Caveat recorded: two-template |cos| agreement is 0.59 at layer 22 vs Robinson's
0.99, almost certainly the small 8/8 calibration set vs their 416/104. Valid
but noisier than ideal; the README flags expanding the sets before the write.
README documents the three environment gotchas (fp32-for-capture, the seats
that must be stopped for the 110GB fp32 VRAM and how to restore them, and the
fla side-dir PYTHONPATH) so the next run doesn't rediscover them.
Dry-run against the fully-staged bf16 confirms the Robinson recipe transfers
onto the DavidAU Cold-Fusion checkpoint with no name drift: 1199 tensors, 333
vision preserved, down_proj=64/o_proj=16/linear_out=48/mtp=2/embed=1, coverage
gate 6/6, exactly 131 tensors to orthogonalize. Harness verified-ready; the
destructive write still gates on operator go.
Harness to abliterate DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1 using the
MTP-aware, vision-preserving recipe in docs/pfi/abliteration-recipe-qwen38.md.
Motivation is measured, not assumed: the stock model's refusal profile
(probed 2026-08-19, hand-verified) is ~33% on creative content, concentrated
on explicit-sexual and graphic-torture, with 4/5 hard-harm refused, self-harm
guardrails intact, and zero benign over-refusal. So there is a real
creative-content refusal surface. The Robinson formula is chosen specifically
because it abliterates the MTP head IN-BAND -- which the current gen seat's
Heretic pass does not (its MTP head is a byte-identical base graft the
Qwen3_5 wrapper never loads). That in-band MTP edit is the additive delta.
The script refuses to brick the model. Two hard gates from the recipe halt
before any write: the coverage identity o_proj(16)+linear_out(48)==64 (catches
a tensor-name mismatch that would ship a half-abliterated model), and the
attention-sink screen on dim 3994 (orthogonalizing a direction living there
produces a model that loads, runs, and emits garbage). The direction is
captured from two chat templates and the layer auto-picked by peak |cos|
agreement in [18,45]. Classification is suffix-based and name-agnostic so it
survives minor drift; the coverage gate is the backstop.
Modes: --dry-run (enumerate + gate, no forward, no write), --capture
(direction + sink screen, no write), default (write to --out). The README
sequences dry-run -> capture -> write -> verify, and names the post-checks
(vision byte-identical, refusal re-profile via services/refusal-probe/, MTP
acceptance on the quant, PPL/coherence).
bf16 staged to ana-ml2:/tank/aimodels/qwen38-27b-coldfusion-bf16 (pinned
9c44193, provenance recorded). The destructive run is NOT executed here --
dry-run verification and operator go gate it.
Reference recipe (not a deployed artifact) for MTP-aware, vision-preserving
single-direction abliteration of Qwen3.8-27B -- the base family the gen seat
runs. Captures the two things this recipe gets right that naive abliterations
of this architecture miss:
- The MTP head is abliterated in-band (its two residual-write matrices, glue
left alone), so speculative acceptance does not collapse on the prompts
abliteration exists to fix -- directly relevant to the gen seat's MTP>=40%
gate.
- The vision tower is preserved byte-identical (333 tensors, max delta 0).
Plus the two calibration traps specific to this base: the twice-captured
refusal direction (layer 26, |cos| 0.99) and the attention-sink dimension 3994
that bricks the model if orthogonalized out. Documents the coverage gate
(o_proj 16 + linear_out 48 == 64 layers) that catches a half-abliterated
model before it writes a byte, and the foot-gun that the GGUF imatrix does not
cover the MTP block. Links into model-quantization-playbook.md for the quant
half of the pipeline.
Docs used to render as a clumsy link that navigated to a separate page. They
now render in place: build_gallery pre-renders each doc (markdown -> HTML,
plain text raw) and the gallery shows it inside a native <details open>
disclosure that spans the full grid width so prose has a readable measure.
The doc bar carries: a collapse chevron (the whole <details> summary toggles,
works with JS off), a full-page link (still reaches the standalone viewer), a
download link, and a session-close ✕. The ✕ needed stopPropagation +
preventDefault because it lives inside <summary> — otherwise its click would
toggle the disclosure instead of hiding the item. Close is JS (progressive
enhancement); collapse is native.
Two design points:
- Plain text is returned RAW from build_gallery and escaped by the template
inside <pre>. Pre-escaping in Python plus Jinja autoescape would
double-encode angle brackets; a test pins the single-escape.
- Inlining is bounded by DOC_MAX_BYTES. A doc over the limit keeps the old
link-out behaviour rather than being rendered into every index load; a test
covers the fallback.
The shared .markdown-body / .textview typography moved from doc.html's scoped
<style> into base.html so the inline body and the full-page view render
identically; doc.html keeps only its page-layout wrapper.
Updated the pre-existing test_gallery_links_docs_to_view: it asserted the old
link-out behaviour the operator asked to change, so it now asserts the inline
render plus the surviving full-page and download affordances. 61 pass.
Verified live: markdown renders with headings/table/blockquote/code, txt
preserves whitespace and single-escapes, collapse and ✕-close both work.
Corrects three claims that had been standing in the fleet IPv6 notes and that
sent a three-arm research effort after a problem that did not exist:
- ESH was recorded as having no IPv6. It has a /56 delegated and a routable
WAN GUA -- substantially more prefix than NH3's single /64.
- The mesh was recorded as broken by ESH's CGNAT. It is not and was not down;
ESH is outbound and working. CGNAT on v4 alongside generous v6 is just the
modern ISP pattern, not an outage.
- IPv6 was framed as the escape hatch for that outage. The actual plan is that
IPv6 carries tunnel ENDPOINTS for Site Magic and WireGuard, and LANs are not
numbered in v6 at all.
NH3 internal v6 was brought up on the delegated /64 and verified end-to-end
(global GUA on nh3-docker, zero loss to Cloudflare and Google v6, un-NATed
source address seen from outside), then reverted on operator direction: one
/64 lights exactly one VLAN and that is not worth the split-brain.
The AT&T prefix research is kept as reference rather than deleted -- the /60
is real but undelegatable, the living multi-prefix mechanism is multiple
IA_PD in one solicit rather than the VRRP/multi-MAC recipe we were handed,
and the UDM SE can express neither. That is the answer if NH3 LAN-side v6
ever earns a maintenance window; it is not on any critical path now.
Agent sessions hand the operator URLs and they drown in terminal
scrollback. The Booth is the right home for them — it already has the one
property that decides adoption, which is that a session can publish with
mkdir and cp, no API key, no schema, no deploy — but everything in it dies
in 24h.
So: a booth containing `.forever` is never swept, and renders in its own
Kept lane at the top of the index. Opt-in per booth, so the ephemeral
default is untouched and nobody inherits a cleanup chore. `rm` the
sentinel and the board rejoins the sweep; the CLI verbs are sugar over
exactly that, which keeps the filesystem-is-the-state model honest.
The pin is deliberately NOT wired into is_expired(). That stays a pure age
question feeding the `expires_in` countdown; only sweep_once() honours the
sentinel. Keeping expiry arithmetic and reaper policy apart means they
cannot drift into each other.
Kept cards are visually separated per Australis: a 2px top edge in aurora
blue, the one accent border the system sanctions. They show "kept" instead
of a countdown, and they deliberately lose the one-click wipe button — a ×
next to the durable stuff is a footgun, so removing a kept board is a
two-step act.
`booth link <url> [description]` appends to the standing `links` board,
creating and keeping it on first use. Entries carry provenance (handle or
hostname, plus a timestamp) because a bare URL is unreadable three days
later. The append is one printf of one line to an O_APPEND fd — atomic
under PIPE_BUF on POSIX — which matters because many agents post to one
board and interleaved half-lines would be the obvious failure mode.
Seven tests cover the sentinel: detection, survival of a sweep that wipes
its neighbour, the deliberate is_expired/sweep_once split, the listing
flag, the sentinel not inflating item counts, and both lane-rendering
directions. Two of them originally asserted on the bare strings "Kept" and
"kept-grid", which passed for the wrong reason — those also appear in the
inlined stylesheet served on every page — so they now assert the full
class attribute. 55 pass.
Also corrects the Homepage card's description, which advertised a flat 24h
TTL that is no longer the whole story.
Removing the `background:` block from settings.yaml was not sufficient.
Homepage server-renders the wallpaper as an INLINE style on
`<div id="background">` and Next.js caches the rendered page, so the aurora
survived both the config removal and a container restart. Only a full
recreate clears that cache, and recreating this container costs an hour of
missing tab bar and i18n before it heals itself.
Adding `#background` to the canvas reset is deterministic and immediate,
and it also keeps the canvas correct if the setting is ever re-added by
accident. The existing selector missed it: the DOM is
body > div#__next > div#background, so `body > div` matched the Next.js
root, not the wallpaper layer.
Verified live rather than locally: the served page now reports no
background image, with all three canonical faces loaded and the group
eyebrows rendering as JetBrains Mono in Australis cyan.
The predecessor theme was ugly for two structural reasons, not one.
It did not use the design system's colours. It built a parallel OKLCH
palette "derived from the Australis philosophy" and swapped the canonical
typeface for Supreme -- a fork, not a theme. Every hex here is now copied
verbatim from ~/.claude/skills/australis-design/colors_and_type.css, and
build.py re-checks all 19 against that file at build time and warns on
drift so it cannot quietly fork again. Type is the canonical stack: Space
Grotesk / Inter / JetBrains Mono, vendored as latin-subset VARIABLE woff2
(one file per family, 102 KB total against 56 KB for three static Supreme
cuts, and no Google Fonts request at page load).
It also carried a generated full-bleed aurora image behind the entire
dashboard. Canon forbids exactly that -- "solid fills only on chrome, no
full-bleed photography, no decorative gradients", and the aurora motif
"never as a background fill behind text". The predecessor knew, said so in
its own header, and dialled the opacity down rather than dropping it. The
image is gone; the aurora survives as a 1px accent edge under the tab bar,
which is where canon sanctions it. The asset stays in images/ in case it
is ever revisited.
Direction is instrument panel. Group headings become the Australis mono
eyebrow with a hairline to the right edge -- canon calls the eyebrow a
system signature, and it turns the groups into register bands instead of
headings floating over a grid. Status stops shouting: the filled emerald
chips read louder than the service names they annotated, so they are now a
semantic dot plus a mono micro-label at tertiary contrast. Cards are
bordered and opaque, per canon's border-over-shadow rule for chrome.
Alignment, per operator feedback that pills and cards did not line up:
- The status cluster is centred on the service name's line rather than
parked in the card's top-right corner, where Homepage's `absolute top-0`
left it floating ~7px above the title's optical centre. The offsets
reconstruct the title line box and are documented as moving together.
- Descriptions get a two-line minimum, so the common one-line/two-line mix
bottom-aligns across a row. This is what made the grid look ragged.
useEqualHeights stays false: it inflated short cards to match a widget
card twice their height, which was the worse failure.
- The status dot is flex-centred rather than nudged with vertical-align,
so it stays centred if the type scale changes.
Retires the Skyfall sources and the Supreme faces; theme/ now has one
source of truth.
esh-pve hard-froze at 03:34 on 2026-08-19 and stayed frozen ~4.5 hours
until a manual power cycle. No panic, no OOM, no MCE — the journal stops
mid-operation. The whole ESH site lost DNS with it, because esh-userland
(VLAN 10, the PVC SSID and wired userland LAN) was handed exactly one
resolver: 10.0.50.45, AdGuard on esh-docker-vm, on a different VLAN, with
no secondary. Internet and routing were healthy throughout.
Two fixes.
1. DNS: 10.0.10.1 (the gateway, verified resolving) added as secondary on
esh-userland via the UDM Classic API. Note this is degradation cover,
not clean failover — clients that query resolvers in parallel will
bypass AdGuard for a share of lookups.
2. Watchdog: softdog -> iTCO_wdt under systemd (RuntimeWatchdogSec=60),
watchdog-mux masked. The box looked watchdog-protected and was not: a
software watchdog cannot fire when the kernel it lives in is wedged,
and watchdog-mux only pets the device while an HA client is connected,
which never happens on a cluster with no HA resources. Firmware does
not block the TCO timer here, checked before committing to it.
Also pins VM 102 off (onboot: 0). It starts with full GPU passthrough and
vfio-pci enabling that device is the last thing the kernel logged, 39
minutes before the freeze. The other suspect is the kernel itself: the
host ran 4.5 months on 6.8.12-16, took 6.8.12-42 in an apt batch on
08-18, and died 20 hours into the first boot on it. 6.8.12-16 is still
installed as the rollback.
The playbook is idempotent — a second run skips all six steps and passes
all six verifies. The watchdog is confirmed armed (identity=iTCO_wdt,
state=active, held by PID 1) but has NOT been observed firing; proving
that needs a deliberate wedge.
Memory also corrects two wrong mid-incident calls: the mgmt VLAN is
routed over the site tunnel and is not firewalled off — both symptoms
were the dead host generating ICMP unreachables.
waterland-dev confirmed the mechanism: adoption marks a job failed on a
sidecar saying running/queued, or on a directory with no plate.png. The
pre-header-fix renders died 1.7s in with a source and no plate, so they
land in the second branch. Recorded so nobody investigates adopted
history as a live fault.
One update.sh run on irv-ml1 carried both open upstream PRs, per the
operator's green-light on the job-store fix:
- #5 (464dfc2) declares cupy-cuda12x[ctk] on the gpu extra and takes uv
out of the render path (sys.executable -m waterland.cli), retiring the
runtime prune trap at the source.
- #6 (b72425b) rehydrates the job index from the data volume at startup,
fixing the unbounded store growth reported from this side.
Verified after the update rather than assumed: healthy on backend cupy;
/api/jobs went 1 -> 16 against 16 directories on disk, so API and volume
agree for the first time; nothing wrongly reclaimed, correct since 16 is
under RETAIN=40 and adoption only makes them visible; a real 256^2 plate
render completes warm, so the kernel-cache volume survived the image swap.
A subsequent render took both counts to 17.
The image keeps its explicit [ctk] install and UV_NO_SYNC/UV_OFFLINE pins
even though both are now redundant. The header requirement is a property
of this slim base, not of the upstream extra, and the cost is measured
rather than assumed: uv sync satisfies it first, so the line reports
"Audited 1 package" and adds 0.3s to the build. The env pins are now
cheap defence-in-depth against any future path that re-enters uv.
Docs corrected in place: the README's upstream-finding section is now a
resolved-finding record, and the two "bounded ~500 MB" claims say which
commit made that bound hold across restarts rather than only within a
process. Comment-side changes pushed to the live compose dir; no restart
was needed for them.
waterland-dev merged PR #5 (main now 464dfc2), fixing both landmines at
source: the gpu extra declares cupy-cuda12x[ctk], and the renderer spawns
sys.executable -m waterland.cli instead of re-entering uv mid-job.
The running container deliberately stays on 8025366. Its own [ctk] install
and UV_NO_SYNC/UV_OFFLINE pins already neutralise both defects, so a rebuild
would buy reliability that is already present — and the project is in
wind-down. Both guards are kept rather than dropped: the header requirement
is a property of this slim image, not of the upstream extra, and the uv pins
are now cheap defence-in-depth against any future path that re-enters uv.
Also records waterland-dev's confirmation of the unbounded job-store growth
and the operator's green-light on their startup-rehydrate fix. That PR
merging is the rebuild trigger: one update.sh run lands the rehydrate and
464dfc2 together. Marks the inbox drained.
Captures a long infra session: fleet *.internal DNS (git-sourced, 42 names,
three resolvers including a new colo one), waterland studio containerised on
irv-ml1, Homepage cleaned up and themed with Australis Skyfall over an
Arbo-generated background, and four unmanaged stacks adopted into stacks/.
Four detail files added. Auto-archived 4 entries to archival-memory.md
(Recent decisions 2, Tried and abandoned 2); 5 held back by the open-deferred
guard rather than moved.
Also records three operator-owned open items: the colo DNS repoint, the
static-v6 convention, and the deliberately belayed AI-tab Dormant regrouping.
Names for fleet hosts so addresses stop needing to be memorised. Built because
IPv6 makes that hopeless — and, more to the point, because v6 addresses are
derived rather than assigned, so they cannot reliably be written down once and
trusted either.
dns/internal.yaml source of truth: 38 hosts + 4 service aliases
scripts/dns-sync.py reconciles AdGuard resolvers against it
stacks/adguard-ana/ the colo's resolver, which did not exist
Naming is <host>.<site>.internal with sites ana/esh/nh3 (operator's call).
.internal is ICANN-reserved for this; .local is reserved for mDNS, which is
why searxng.pfi.local was a collision that merely happened to work.
Same posture as deploy-stack.sh: file is intent, resolvers are derived state,
you see a diff before anything changes. Every name is published to every
resolver, so the site label says where a host IS, not who knows about it.
Two properties that matter:
- Authority is scoped to the ZONE, not the resolver. ESH carries hand-made
esteban.net rewrites predating this; they are read, ignored and preserved.
Resolver-wide authority would have silently deleted them.
- Within .internal it IS authoritative, so UI-added names get removed. That is
the point — one place to look.
Colo gap closed: ana-docker had no resolver at all (hosts went straight to
1.1.1.1). Its AdGuard runs API on 8053 because 8080/3000 were taken, so the
port is carried per-site in the yaml rather than assumed by the script. It
ships with no blocklists — a false positive on a server network breaks
service-to-service calls for no upside.
Auth is a dedicated infra-ops AdGuard user, not the operator's account,
password vaulted at nh3-dev/adguard-infra-ops-password. Pre-change configs
backed up on each resolver. Both resolvers stayed answering across the restart.
searxng.pfi.local -> searxng.ana.internal, with the old Host() kept alongside
so nothing breaks mid-migration. matrix.pfi.local deliberately NOT migrated: a
Matrix server_name is baked into every user id, room id and signing key, so
renaming it rebuilds the homeserver's identity rather than changing a DNS name.
The v6 column is empty and correct — no fleet host has a global v6 address
yet. The file documents why addresses must be pinned statically before they go
in, since a record that silently stops matching is worse than no record.
Operator granted claude-bot read on vh/waterland; verified scoped correctly
(admin false, push false, pull true). Token is on irv-ml1 at
/root/.config/waterland-studio/git-credentials, 0600 root-owned, wired as a
REPO-SCOPED credential helper rather than a global one, and .git/config holds
no token so the remote stays clean in any diff or backup. The vh site-admin
token was used only for the initial clone and the grant itself and was never
written to disk on that host.
update.sh now runs end to end: fetch, rebuild, recreate, health. Verified the
kernel-cache volume survives a recreate (warm 256^2+anim render 6.6s straight
after) and the job store survives with all 16 directories intact.
Records an upstream finding surfaced by that check: JobStore._jobs is
memory-only and nothing scans the data dir at startup, so after a restart the
API lists only new jobs while old ones persist on disk — cosmetic — but the
RETAIN=40 eviction only sees in-memory jobs, so restart-orphaned directories
are never reclaimed. The handover's ~500 MB bound holds per process lifetime,
not across restarts. Reported to waterland-dev; upstream's call to fix.
Replaces a bare nohup on irv-ml1:8410 that would not have survived a reboot,
handed over by waterland-dev. Tracks vh/waterland @ main (PR #4 merged; main
HEAD is exactly the pinned 8025366).
Build context is a checkout at /opt/waterland-studio/src, deliberately OUTSIDE
the compose dir — deploy-stack.sh rsyncs stacks/<stack>/ with --delete and
would otherwise eat it. The Dockerfile is passed out-of-context.
Three landmines, all measured:
1. Both uv extras are load-bearing at build AND run. jobs.py shells the
renderer out as a literal with no --extra flags, so uv
would re-sync at runtime and prune cupy — silently dropping to the numpy
path at ~21x wall time. UV_NO_SYNC pins it; UV_OFFLINE makes any failure
loud instead of quietly slow.
2. cupy needs CUDA HEADERS for its NVRTC compile, not just the driver and the
wheel's runtime libs. The host has a system CUDA toolkit so the nohup
process found them by accident; a slim image does not, and every render
died 1.7s in with 'Failed to find CUDA headers' printed through argparse's
usage banner — which reads like a CLI bug, not a missing toolkit. Fixed
with cupy-cuda12x[ctk] (hundreds of MB, vs ~6 GB for a -devel base).
3. The A6000 is host device 1 but container device 0, since compose exposes
exactly one GPU. CUDA_VISIBLE_DEVICES_TARGET=0 inside; copying the host's
value selects a device that does not exist.
/root/.cupy is a volume because the NVRTC compile costs ~17s: verified at
23.3s cold vs 6.1s warm, and re-verified across a restart (23.2s on a fresh
cache volume, 6.0s once populated). Warm 256^2+anim beats the 7.4s recorded
against bare metal, so containerising cost nothing.
Job store seeded with the 4 jobs from the displaced instance. Serial by design
(one replica, one card) and unauthenticated, so it stays LAN/WireGuard-only.
Two operator corrections in one pass.
Stat values overshot: the previous commit took them from font-thin 13px to
bold 22px in heading white, which went from whisper to shout. A stat only has
to out-rank its own label, not the service name above it — now --text-md at
medium weight in cyan, which clears the label but sits below the card title
where it belongs.
Colour lift, staying inside the system rather than around it: Skyfall names
Aurora (blue, cyan, green) the PRIMARY families, 'used generously, in that
order', while Dawn (amber, red, violet) is semantic-only. So group markers now
cycle blue -> cyan -> green down the page — icons at full strength, names at
0.72 — service icons take a single cool wash, header resource icons go cyan,
and latency tags move to the info family so 'how fast' stops looking like
'is it alive'. No Dawn colour is used decoratively anywhere.
Also fixes selectors that never bound: Homepage emits docker-status-<state>,
not status-<state>, so the green pills up to now were stock colouring rather
than this file. Both forms are matched and the trap is commented.
README records the iteration loop that would have caught the overshoot: CSS is
served per-request, so it needs a reload, not a recreate and not the layout
warm-up — and candidate CSS can be injected into the live page for a
seconds-long feedback loop instead of a 10-minute one.
Stock Homepage builds each stat as a font-thin (weight 100) 13px value above a
font-bold 12px uppercase label, so the number you actually came to read is the
quietest thing in the card while its label shouts. Skyfall's rule is that
hierarchy comes emphatically from weight AND size, and that numbers are data.
Value now renders at --text-xl bold in tabular mono at --text-heading; label
drops to a --text-2xs tracked eyebrow at --text-faint. The well itself moves to
--surface-input, one step DOWN from the card it sits on, so stats read as inset
data rather than as another floating surface — a recess, so it takes the
hairline without the shadow.
Also drops .service-block from the generic .service-tag rule, which was what
pinned every number to --text-2xs in the first place.
Visible on Plex, Jellyfin, PaperlessNGX and Uptime Kuma.
The char-rp-reasoning seat on ana-ml2 GPU0 — NEO-CODE Heretic2 27B at modelopt
NVFP4 with a grafted BF16 MTP head, ~77 tok/s via qwen3_5_mtp spec-decode,
replacing the retired GGUF seat. It had been running untracked.
Includes conf/mtp-workaround/sitecustomize.py, which is not optional: vLLM
0.24.0 does not propagate modelopt exclude_modules to the spec-decode DRAFT
model, so the BF16 MTP head gets quantized and the engine dies at load. The
shim force-skips mtp.* in is_layer_skipped. Both the mount and PYTHONPATH are
load-bearing.
Adds the two files house convention expects and the directory lacked: a
.env.example naming every knob (all values are the compose defaults; the live
host overrides only the three VRAM ones) and a README that points at
docs/runbooks/heretic2-nvfp4-mtp-seat.md rather than duplicating it.
No secrets: API_KEY is empty by default and the real .env stays on the host.
Replaces the previous theme attempt, which was built on a misread: the ask was
to use Arbo as an IMAGE-GEN ENGINE for the background, with the operator's
Australis Skyfall design system supplying the palette.
theme/ holds the source — colors/layout/typography vendored verbatim from the
Skyfall handoff bundle, Supreme 400/500/700 woff2, the Homepage bindings in
skyfall.css.in, and build.py which inlines fonts + tokens into
conf/custom.css. custom.css is GENERATED; edit the .in file and rebuild.
The build exists because Homepage serves only custom.css and custom.js out of
its config dir, so a @font-face pointing at a vendored woff2 would 404 — the
face has to arrive as a data: URI. The background image takes the other route:
/app/public/images is a real static route, so compose.yaml now mounts
images/ there read-only and settings.yaml points at /images/.
Bindings map Skyfall's semantic layer onto Homepage's DOM: Sea surfaces, the
depth recipe (hairline AND two-layer shadow, never one alone), uppercase
eyebrow group headers, the sanctioned accent-rail on the active tab rather
than a glow, and semantic status colour so a green pill means the service is
actually serving.
Background generated by Arbo (irv-ml1:8201) workflow t2i-ui-background, job
13f0891f4e42, seed 26, flux2-klein-9b, 2048x1152 — abstract, no subject,
cool-temperature aurora. 1.6 MB PNG -> 22 KB WebP.
Two deviations are documented rather than hidden: Skyfall forbids imagery
behind body text (held at opacity 30 as mitigation), and service icons stay
full-colour vendor logos.
NOT DEPLOYED — live still runs the old theme. Prototype on :5199.
Ports Arbo's design tokens (irv-ml1:8201) into conf/custom.css — flat raven
ink #021425, card surface #112333 on #1B2E3D borders, Manrope, and mint
#2FFC89 reserved for signal so a green pill means the service is actually
serving. Values read off Arbo's running :root custom properties rather than
sampled from a screenshot. CSS rather than settings.yaml because Homepage's
color: setting only accepts built-in Tailwind ramps.
NOT YET DEPLOYED — live still runs the stock theme pending an A/B decision.
Prototype is at 10.0.50.45:5199; shots in ~/booth-data/homepage-cleanup/.
Promoting it also means dropping the background: block from settings.yaml.
Also corrects the previous commit's tab-bar claim. It is not a fixed few
minutes of warm-up: a fresh container was still tab-less at 4m30s twice, and
recovered on its own about an hour later. Cause remains unpinned; the README
now records the measured timing and the four ruled-out causes.
searxng: the healthcheck passed '--tries' and '--spider' as separate argv
entries, so wget consumed '--spider' as the value of '--tries'. Spider mode
never engaged and every 30s probe downloaded the response to disk; the
container's working directory had accumulated 295,287 healthz.N files since
April, and the directory scan to pick the next free filename is what
intermittently blew the 10s timeout and flapped the dashboard card to
UNHEALTHY. Restored '--tries=1'. The junk was in the writable layer, so the
recreate cleared it. Now healthy, fails=0, 200 in 0.16s.
seafile: none of the three services declared a restart policy, so Docker
defaulted them to 'no'. The daemon stopped all three within 200ms on
2026-05-06 and nothing brought them back — a three-month outage whose only
trace was an EXITED card. Exit 255 is what a container ignoring SIGTERM
reports when the daemon stops it, not a crash. Added restart: unless-stopped.
Stack is back up; mysql gates on its healthcheck as designed and seahub
started without the race. 302 -> login page.
Both stacks were running unmanaged on ana-docker and are now tracked here.
homepage: AI tab reordered by clickability per operator — chat frontends,
ComfyUI and the control plane on top; vLLM /docs seats and TTS endpoints
below. Corrects the previous commit's UNRESOLVED tab-bar section: it was
warm-up time after a recreate, not a defect.
UltraSeedbox had no layout: entry, and Homepage renders an untabbed group on
every tab — eight full-width bookmark bars repeated four times. Pinned to Main
with a row layout.
Uptime Kuma rendered twice: a manual services.yaml block under Monitoring plus
homepage.group=Apps on the container. Dropped the manual block, moved the
label to Monitoring, added homepage.siteMonitor. Adopted the previously
unmanaged uptimekuma stack into stacks/ so the label is version-controlled.
Column counts declared more columns than groups had members, leaving the last
row of several groups mostly empty. Columns now track member counts.
Also records an UNRESOLVED regression: since the container was recreated the
client render has lost its tab bar, wallpaper and i18n. Ruled out the config
changes (committed pre-cleanup config reproduces it) and v2.0.0 (v1.13.2
reproduces it). Server HTML still carries the tab markup, so the loss is
client-side. Details in the stack README.
Session captured: ESH cut over to Cityside 2Gb symmetric fiber and was
fully provisioned on it; esh-pve-nas completed its ZFS-root migration off
the USB DOM and took its 225-package security backlog with the reboot
deferred; ESH<->colo IPsec was rebuilt as a dialup tunnel with NAT-T after
CGNAT broke the statically-pinned one; IPv6 was mapped across all three
sites.
Auto-archival fired at the soft cap: 7 entries moved to
archival-memory.md (Recent decisions 3, Tried and abandoned 4), all
verified-complete arcs, with two detail files moved and removed. The
remaining pre-Aug-05 entries were held back by the open-deferred-work
guard, so the index stays slightly over cap at 313 lines rather than
losing a live pointer.
Also records the one self-inflicted outage of the session (missing
--make-rslave on a chroot rbind) and that three long-dead things
surfaced incidentally: pvestatd down 82 days, a vzdump hung 126 days,
and a VM sitting in prelaunch for four months.
Operator will retrieve the BGW210 Device Access Code from the NH3 office
and vault it, after which the IPv6 LAN settings page can be driven
remotely.
Parked on the henge as reclaim-nh3-s-7-unclaimed-ipv6-64s-from-the with
everything needed to resume cold: the verified facts about the /60 split
and the seven unclaimed prefixes, why AT&T cannot fix it, the exact page
to start at, what to look for in priority order, the other settings
pages behind the same login, and the wpa_supplicant fallback with its
warning about modifying NH3's only uplink.
Suggested vault path unifi/bgw210-nh3-device-access-code, matching the
existing unifi/* credentials.
Operator suggested checking the BGW on its 192.x management address,
which turned out to give the whole picture from unauthenticated status
pages.
The CPE is a BGW210-700 on firmware 4.28.7 at 192.168.1.254. AT&T does
hand it a /60 -- c110 through c11f. The BGW keeps c110-c117 for itself
and re-delegates up to eight individual /64s on c118-c11f, top-down. Our
UDM holds c11f, delegation number eight.
So the earlier conclusion that AT&T only grants a /64 was right about
the symptom and wrong about the cause. Seven further /64s are available
and simply never solicited, because UniFi exposes a single
wan_dhcpv6_pd_size integer with no field for how many prefixes to
request. The documented workaround is repeated -P flags to dhclient,
which the UniFi UI cannot express.
This also settles that an AT&T ticket cannot help: the rationing is CPE
firmware behaviour, not provisioning. Records the two real options --
accept one /64, or bypass the BGW entirely with wpa_supplicant EAP-TLS
on the UDM to negotiate the full /60 -- with the warning that the latter
modifies NH3's only uplink and needs a planned window.
AT&T support guessed 'I do not believe att will do that' from a DNS
provisioning desk. The guess was correct, but it needed proving rather
than accepting, so the UDM solicited DHCPv6-PD at /48, /56 and /60. All
three returned the same single /64. This is not a case of nobody having
asked -- the ask was made three ways.
Proven by temporarily enabling PD on nh3-iot, the only NH3 VLAN with
zero clients, then setting ipv6_pd_prefixid to 0, 15 and 16. All three
returned an identical c11f prefix, which only happens when exactly one
/64 is delegated; with a larger block the prefix-id moves the LAN within
it.
Records a mistake worth not repeating: I first read the gap between the
WAN address (c110) and the delegated prefix (c11f) as evidence of a /60.
It is not -- AT&T assigns those from different parts of their pool and
the spread means nothing.
NH3 was fully restored afterwards, with rollback artifacts kept. Also
captures the concrete ask for AT&T Business, phrased as something their
provisioning team can verify against their own DHCPv6 logs, and the
consequence if refused: NH3 can host exactly one v6 segment against
ESH's 256.
Operator opened a ticket with Cogent for v6 at Anaheim, which closes the
one thing tonight's investigation could not resolve from our side.
Captures the diagnosis so the ticket has evidence behind it: a single RA
in 90 seconds of sniffing wan1, from fe80::ea0a:b9ff:fe3b:2c16, proving
an IPv6-capable router sits one hop away on the circuit terminating
38.120.12.42/29 -- but SLAAC obtained no global address across multiple
RA intervals and ping6 to Cloudflare and Google both returned 100% loss.
Router present, circuit unprovisioned.
Also records that the FortiGate v6 config was fully reverted after
testing, the FortiOS gotcha that SLAAC is 'set autoconf enable' rather
than an ip6-mode, and the ask to make when it lands: a /56 or better,
since NH3 only receives a single /64 from AT&T.
Notes the consequence worth planning around -- once provisioned, the
colo becomes the only site with both a static public v4 and routable
v6, which makes it the natural v6 hub given ESH is CGNAT'd and NH3 is
prefix-constrained.
Operator's framing, and it is a better argument than the terminology
correction that preceded it. Under v4, exposing a host needed two
affirmative acts -- a DNAT and an accept rule -- so missing either left
the host dark. There is no v4 misconfiguration that exposes an internal
host by accident. NAT was load-bearing security whether or not anyone
designed it that way.
v6 removes the first control entirely. The path exists inherently, so
the firewall is the only thing left, and the failure mode inverts from
fail-closed to fail-open. Rule-ordering slips, rulesets that silently
match only one address family, new VLANs added without policy, and
re-delegated prefixes unmatching address-literal rules all become
exposure events rather than no-ops.
Records the practical consequences: key rules on interface/zone rather
than address literals, treat enabling v6 on a segment as requiring
policy to exist first, and verify default-deny from off-net rather than
by reading the ruleset -- which is lesson 3's assert-the-effective-value
discipline applied to firewall policy.
Also corrects my own claim from the previous commit that the pending
firewall pass was 'smaller' than I had implied. It is not smaller, it is
different in kind.
I wrote that enabling SLAAC would give LAN devices 'globally reachable
addresses'. Wrong word, and the wrong word in a persistent-memory entry
a future session inherits as fact.
Addressable is a property of the address. Reachable is a policy decision
the firewall makes. v6 removes NAT; it does not remove the firewall, and
treating those as the same thing is exactly how v6 gets mischaracterised
as automatic exposure.
Records the operator's position while correcting it: no 1:1 inbound
pass-through. The pending firewall-policy pass is about writing explicit
default-deny inbound rules per v6 segment, not about deciding what to
expose.
esh-iot keeps the identical eight digits -- 4DBAD107, rendering
4dba:d107 -- and only the reading changes: 4 is FOR rather than A, so it
parses 'FOR DA BAD IOT', which describes what the segment is actually
for.
esh-mgmt genuinely changes: 115D:B055 becomes 15DA:B055. The leading I
is dropped and DA is spelled in full, giving 'IS DA BOSS' with the
network as subject rather than speaker. Still eight digits. DA written
out needs no substitution since D and A are both native hex; spelling it
as a single D the way esh-iot does would have yielded seven digits and
broken the house pattern.
'FOR ALL DA BAD' -- 4=FOR, 411=ALL, D=DA, BAD=BAD. Eight digits, house
style, renders 4411:dbad.
No DMZ network exists on the ESH UDM today; this is a name claimed
against the day one is built. Pairs deliberately with esh-iot's
4DBA:D107 -- IoT is 'a da bad', the DMZ is 'for all da bad', which is
the correct relationship between the two segments.
'FOR ALL BIOS' -- 4=FOR, 411=ALL, B105=BIOS. Eight digits like the rest.
Completes the set. All six ESH networks now carry an 8-hex-digit phrase
in a consistent first-person/declarative voice:
Default 4BA5:3417 A BASE FOR IT
esh-mgmt 115D:B055 I IS DA BOSS
esh-server 4411:B105 FOR ALL BIOS
esh-userland CAFE:4411 CAFE FOR ALL
esh-iot 4DBA:D107 A DA BAD IOT
esh-cameras 1533:FACE5 I SEE FACES
Still a documentation convention rather than wire-level configuration --
UniFi has no static-v6 client assignment and the gateway address is
platform-fixed -- but these are the values to use whenever ESH LAN v6 is
switched on and hosts get hand-assigned addresses.
Same phrase, 'A BASE FOR IT', but written as a plain 8-digit string
rather than forcing the article into its own group. 4=A, BA53=BASE,
4=FOR, 17=IT renders as 4ba5:3417 -- two groups, matching every other
network in the scheme, with the words straddling the colon exactly the
way 4DBA:D107 does.
Corrects the previous commit, which claimed this needed nine digits and
a third group. It is eight, and always was.
'A BASE FOR IT' -- the article makes it a full sentence, matching the
voice of the other five. BA53 uses 3=E rather than the 5E spelling used
in the previous BA5E version.
Nine hex digits rather than eight, so unlike the others it does not fit
two groups and renders across three as a:ba53:0417.
'BASE FOR IT' -- BA5E=BASE, 4=FOR, 17=IT. The foundation segment
everything else hangs off, which is what the default network is, and it
doubles as 'base for IT'.
Note the trailing group zero-pads: it renders as ba5e:0417, not
ba5e:417.
'CAFE FOR ALL' -- CAFE, 4=FOR, 411=ALL. Eight digits like the others,
and the only one so far that splits on its own phrase boundary, so it
renders legibly as cafe:4411.
Bonus reading: 411 is US directory assistance, which is a fitting second
joke for the segment the humans actually live on.
vh