Full seat rebalance across GPU0/GPU1 (flash on GPU2 and empty GPU3 untouched),
operator-directed. Every target seat now serves native 262,144 context with
concurrency in the requested 1.2-2.5x band, verified from live boot logs:
cyberprev (sec) 262144 @ 1.37x depth-probed CLEAN to 259,722 tokens
flash-next (gen) 262144 @ 1.31x (untouched, already in band)
gen-small (NEW) 262144 @ 2.56x MTP k=3 measured 69.6% accept / 3.09 len
char-rp 262144 @ 1.22x (was 1.02x; util 0.40->0.52)
char-rp-fast 262144 @ 2.04x (util cap 0.30->0.24, pinned KV unchanged)
- gen-small: NEW seat, Qwen3.6-35B-A3B (3B active MoE), llmfan46 Heretic
(MPOA) NVFP4 experts-only, already on disk at qwen36-35b-a3b-heretic-nvfp4.
There is no general Qwen3.8 A3B (3.8 MoEs are Flash-Next and the 2.4T), so
this is the 3.6 fallback the operator specified. GPU0, :8026, MTP k=3, coherent
and MTP-verified before wiring. gen-small / gen-small-reasoning gateway aliases.
- coder: 8192 @ 13.75x -> 16384 @ 4.70x (util 0.06->0.055). Context doubled,
waste cut. Not the exact 2-3x target: the 1.5B weight+overhead floor (~4.2 GiB)
sits just under the util knob's resolution, so hitting <=3x reliably needs a
--kv-cache-memory byte pin (compose change) rather than the util fraction.
- cyberprev raised 163840 -> 262144: depth-probed with non-repeating prompts to
259,722 tokens, clean (no OOM, memory flat). Unlike mog-sec (same base arch,
capped at 163840 for depth crashes), this checkpoint holds native depth.
- Gateway (operator calls): summarizer + classifier -> gen-small; new
classifier-large -> gen-large (flash) for the accuracy tier; summarizer-large
stays on flash. All verified end-to-end.
- GPU1 hit its ceiling raising char-rp; resolved by trimming char-rp-fast's
reservation cap (its KV is pinned, so concurrency held at 2.04x) rather than
moving a utility seat -- the shared GPU_ID on reward/embed/rerank made a
single-seat move messier than the in-GPU rebalance.
Seat inventory regenerated from the live containers.
Operator instruction: take down the existing sec seat (mog-sec) and promote
hotdogs (cyberprev) into the sec and sec-reasoning gateway seats.
- mog-sec container (vllm-mog-sec, :8019, fv-ml1 GPU0) taken down; ~48 GB freed
on GPU0 (cyberprev, already co-resident there, is now the sole GPU0 chat seat).
- Gateway sec -> hosted_vllm/cyberprev-27b @ :8025; sec-reasoning ->
hosted_vllm/cyberprev-27b-thinking @ :8025. sec/sec-reasoning are ROLE aliases,
so this is a promotion, not silent substitution (samplers were already identical
between the sec blocks and cyberprev, so only model+api_base changed).
- Removed the standalone cyberprev-27b / cyberprev-reasoning gateway aliases added
in the prior commit -- now redundant with sec/sec-reasoning, and the fleet
convention is a role alias on the gateway with the model's served-name only at
the vLLM layer (as mog-sec had). cyberprev's vLLM served-names are unchanged.
- Verified e2e through the gateway: sec answers (nmap -sV version detection),
sec-reasoning answers with a thinking split (127 reasoning tokens); retired
mog-sec-27b now 400s.
Note: mog-sec was the fleet's only offense+defense/blue-team seat; the sec role
is now offense-only (cyberprev tool-calling). Operator-directed after reviewing
the capability comparison. mog-sec stack files retained for a future restore.
The sentinel-r3 header and two memory notes described mog-sec (Blackfrost
M.O.G.-SEC / Qwentium) as 'a persona system prompt on stock weights'. Its card
is explicit that it is NOT: base_model_relation: finetune on Qwen/Qwen3.8-27B,
a refusal-free offense+defense cybersecurity SFT with YaRN 1M context ('not a
system-prompt sticker on a stock Qwen'). So all three sec-seat candidates are
Qwen3.8-27B SFT finetunes and differ in training focus, not in kind:
mog-sec = broad offense+defense SFT; sentinel-r3 = pentest agent-trajectory SFT;
cyberprev = cyber tool-calling LoRA SFT on an abliterated base.
Earlier commits and the compose header called cyberprev "an abliteration of
Qwen3.8-27B (refusal-direction weight edit, no finetune)". That was wrong. Per
the hotdogs model card (base_model_relation: finetune), cyberprev is an
offensive-security agentic tool-calling LoRA, merged at scale 1.0, trained on
hotdogs/cyber-sft-agent-qwen38 (8,400 rows across 22 pentest tools: nmap,
sqlmap, metasploit, hydra, crackmapexec, ...). The abliteration is INHERITED
from its base (hotdogs/Qwen3.8-27B-abliterated); the new capability in this
repo is the cyber SFT itself.
So the sec-seat lineage is: mog-sec = persona on stock weights; sentinel-r3 =
pentest-trajectory SFT on stock weights; cyberprev = cyber tool-calling SFT on
an abliterated base. All three are SFT-or-prompt variants; the earlier framing
of cyberprev as "not a finetune" was backwards.
Corrects the compose header lineage block, the homepage dashboard description,
and the gateway alias comment. No functional change; the running seat and its
weights are unchanged (label refreshes on next container recreate).
The first cyberprev quant served pure gibberish despite passing every structural
check. Root cause: the hotdogs/Qwen3.8-27B-abliterated-cyber-preview checkpoint
ships broken -- 850 of 1199 tensors (the whole transformer body) carry a TRIPLED
`language_model.` prefix, confirmed present in the upstream HF index (an unsloth
export bug, not our download). transformers silently loaded the unmapped tensors
as random-init, the pipeline quantized noise, and structural verification passed
because it inspects the output config, never that input names mapped. A plain
greedy coherence smoke caught it; nothing structural could.
Fix: rewrite tensor names against a known-good reference of the same architecture
(qwen38-27b-uncensored-bf16), verified the repaired name-set == reference name-set
exactly (1199 tensors, 0 collisions), and re-quantized from the corrected source
(cyberprev-bf16-fixed). v2 is coherence-verified (spec off AND live with dflash):
Paris/Berlin/Rome, correct arithmetic precedence, primes, clean cyber-domain chat
with a proper thinking split. dflash k=7 measured at 2.77 mean acceptance length.
Seat serves cyberprev-27b / cyberprev-27b-thinking on :8025/GPU0 with dflash k=7,
displacing the retired sentinel-r3. Gateway: removed the retired sentinel-r3 /
sentinel-r3-reasoning aliases and added cyberprev-27b / cyberprev-reasoning -- NOT
repointed, since serving cyberprev weights under sentinel's name is silent
substitution. Verified end-to-end through the gateway; sentinel-r3 now 400s.
Playbook gains a superseded-claims row: structural verification does not imply a
good quant; a coherence smoke is the load-bearing gate, and third-party sources
must have their tensor names diffed against a reference before quantizing.
Broken v1 quant dir removed.
Third sec-seat candidate: hotdogs/Qwen3.8-27B-abliterated-cyber-preview, an
abliteration (refusal-direction weight edit) of Qwen3.8-27B aimed at the
cyber-offense refusal surface -- distinct from mog-sec (persona on stock
weights) and sentinel-r3 (SFT finetune). Operator instruction: quant it, take
sentinel down, serve it with mtp or dflash.
Quantized to the house mixed recipe via services/gen-seat-mixed-quant/ (NVFP4
W4A4 on MLP layers 0-55 + FP8 W8A8 on attn/linear_attn/lm_head/MLP 56-63, FP8
KV). The prior attempt (2026-09-11/14) died with "Cannot determine
num_attention_heads" because it ran from a bare .venv whose newer
compressed-tensors reads that field at top level; quant_mixed_nvfp4.py already
promotes text_config attention fields for exactly this reason, and the run
through the canonical vLLM-image + llmcompressor 0.13.0 / compressed-tensors
0.18.0 path (versions recorded from the container) completed clean.
post_quant.py did its job: grafted the 15 MTP tensors verbatim (BF16), and
re-injected re:^mtp.* into the ignore list after llm-compressor pruned it for
matching no module at quant time -- without which vLLM loads the grafted head
uninitialised and speculative decoding runs at 0% acceptance. It also caught a
missing preprocessor_config.json (absent from the abliterated source AND its
hotdogs upstream); restored from Qwen/Qwen3.8-27B, verified byte-identical to
the working sentinel-r3 build, so the vision tower preprocesses.
Verified from the tensors, not the config: NVFP4 covers MLP 0-55, FP8 covers
56-63, no overlap, 168 weight_packed tensors (56x3), 15 BF16 MTP, 333 BF16
vision. 51.0 GiB bf16 -> 21.0 GiB.
Served under its own name (cyberprev-27b / -thinking), NOT sentinel-r3 --
serving different weights under a retired name is silent substitution. Takes
over :8025 and GPU 0 (co-resident with mog-sec). dflash k=7 is the default,
carried from the sentinel measurement, but is re-measured against MTP on this
ABLITERATED body before cutover, since abliteration is exactly what can desync
an MTP head. Drafter (qwen38-27b-dflash2-drafter) shared with mog-sec.
Context capped at 163840 not native 262K, inherited from mog-sec's hard-won
depth lesson; raise only after a non-repeating deep-prefill probe on this
checkpoint.
Bisected context depth on the orcarouter checkpoint with non-repeating prompts
(unique random hex per probe, so prefix caching cannot short-circuit the
prefill). Six depths from 31,978 to 258,517 tokens, all served.
The load-bearing evidence is the engine allocator log: zero OOM, CUBLAS, or
illegal-memory entries across the run. That is the same detector that caught
the dealignai near-miss at 155K on the previous checkpoint, where it did fire.
The probe also ran under real concurrent operator load, making it a stricter
test than a solo run rather than a weaker one.
Positive control passed: a mis-sized first attempt produced a ~265K-token
prompt and got a clean 400 naming the limit instead of killing the engine, so
the probe could detect the failure mode it was looking for. Calibration for
re-runs: random hex words tokenize at 7.9 tokens/word here.
vLLM #54919 (long prefill starving decode for 3-7 minutes) did not reproduce:
258K prefilled in 28.9 s, roughly 8,900 tok/s, scaling near-linearly from 32K.
Records that the probe's memory-headroom half was BLIND and must not be reused.
It reported an identical 95,460 MiB used / 2,427 MiB free on every row across an
8x range of depths, which is the tell. Two causes: --kv-cache-memory pins the
pool and the engine logs "skipped memory profiling", so GPU usage is flat with
respect to depth; and the actual risk is a transient activation spike during
prefill, which before/after nvidia-smi bracketing structurally cannot observe.
Peak-activation headroom therefore remains unmeasured; the pass/fail result
rests on the allocator log alone.
Also qualifies the earlier 167.5 tok/s decode figure as a possibly-contended
lower bound, and records the operator's independent 140 tok/s average measured
in real use while this probe was loading the same card.
Operator instruction: delete the displaced dealignai checkpoint. 125 GiB
reclaimed from /tank (59% -> 57% used). Verified before removing: not mounted
by any running or exited container, no symlinks, no inodes shared with the
converted orcarouter directory.
Every "rollback is two .env keys" statement across the stack README, the
.env.example, persistent-memory and its detail file was true when written and
is false now -- the .env backup still names paths that no longer exist.
Corrected in place rather than left as false reassurance, since a stale
rollback instruction is discovered precisely when it is needed.
Reverting this seat now costs a 126 GiB re-download. The quality A/B against
dealignai is likewise no longer runnable locally: its reference arm is gone.
The pristine 170 GiB orcarouter download is retained deliberately -- it is what
makes the PLE bf16->FP8 conversion reproducible without re-fetching -- and that
is now recorded so a future session does not reclaim it as an obvious duplicate.
Also notes that ~75 GiB of non-PLE shards are duplicated between the pristine
and converted orca directories (the convert's hardlinks hit EXDEV across two
container bind mounts); both now sit directly on /tank, so relinking would
reclaim it if /tank ever tightens.
Swaps gen-large from the dealignai ModelOpt W4A4 build to
orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4, which is weight-only on both
axes (W8 float attn, W4 float experts, input_activations: null) and so avoids
the 4-bit-activation long-context degradation mode.
The checkpoint was previously recorded as unloadable on any mainline vLLM,
requiring a from-source PLE-loader patch. That conclusion was wrong on cost.
Qwen4ExpPLEEmbeddingMethod.from_quant_config checks ple_embedding_dtype as
branch 1, before any quant-config type check, and its NotImplementedError for
CompressedTensorsConfig is scoped to the PLE path only -- experts and dense
load through the ordinary compressed-tensors paths. Verified by instantiating
the real config and calling the selector both ways before doing any work.
orcarouter ships a bf16 PLE, so the fix was to make the declaration true:
convert the 51.2B-param table to FP8 and declare it. Its 128 PLE tensors sit
in one shard file with nothing else in it. Global amax 0.0894, per-shard
outlier ratio 1.66x, scale chosen exactly representable in bf16 so no
scale-rounding error stacks on quantization; amax maps to 446.17/448, no
clipping. Round-trip 2.655% RMS relative, 0.002% underflow, 0 saturation --
the same FP8-PLE treatment dealignai already shipped. MTP head (31 tensors)
and vision tower carried through untouched.
A second, independent blocker followed: orcarouter labels its 12 QSA layers
qwen_sparse_attention, which vLLM rejects; it accepts full_attention and
selects QSA via indexer_n_heads. Confirmed indexer_n_heads == 4 in both this
and the dealignai checkpoint before renaming -- without that check the rename
silently selects plain attention and serves a subtly wrong model that still
passes a healthcheck.
Measured on the live seat: healthy, coherent, KV 344,155 tokens @ 262,144 ctx,
MTP k=3 at 60.4% acceptance / 2.81 mean acceptance length, warm decode median
167.5 tok/s at conc=1 (n=5, spread 12.2%). The reorg note's dealignai figure
came from a different harness, so this is not claimed as a win over it; what
it does establish is that weight-only experts did not cost decode speed.
Still open: controlled quality A/B vs dealignai, and a deep-prefill probe at
262K. Rollback is two .env keys; dealignai remains on disk.
Also corrects the README's MTP-is-off section, stale since k=3 was deployed,
and adds a superseded-claims row to the quantization playbook.
FV colo recovered 2026-09-13 midday (chassis on PDU, firewall on the Eaton 5P1000,
GPU caps 275W/card). All-night fv-ml1 seat reorganization:
- flash-next gained MTP k=3 (campaign measured it a win here, +52% at conc=1),
inverting vLLM's 4xH100 recipe; KV 14->10 GiB.
- gen consolidated onto flash-next (all 8 gen/summarizer/classifier/judge aliases
repointed); 27B dense gen seat retired, 38 GB freed on GPU0.
- char-rp restored to the in-house MeroMero-v2-31B dense heretic (was serving a
leftover-test RedHatAI 26B); char-rp-fast is the deliberate speed tier.
- Sentinel-R3 (SFT pentest finetune) served for an A/B vs mog-sec, then dflash k=7
cut over after measuring it beat MTP (2.40 vs 2.18 acceptance, ~121 tok/s warm).
gen-large is intentionally DOWN: the orcarouter weight-only NVFP4 build downloaded
(170 GB, verified) but no mainline vLLM loads its compressed-tensors qwen4_exp PLE;
the third-party backport was vetted and is unfit (old-hardware fork, no Blackwell
image). Runtime decision pending -- this is the resume point.
Also this session: vh/infra-reference repo, scripts/seat-inventory.py + daily drift
alarm, OPNsense API reference vendored, secrets shed from a prior scratchpad.
Leaves the fv-to-ana-nat files (another session's) and graphify-out untouched.
Read from docker inspect, nvidia-smi and each model's own config.json rather
than from compose files or the gateway config, so the document records what is
deployed rather than what was intended.
Findings worth naming:
- char-rp drift. The LiteLLM config documents char-rp as G4-MeroMero-v2-31B on
ana-ml2 GPU 0; :8016 actually serves gemma4-26b-a4b-it-nvfp4, a 26B 30-layer
128-expert stock Gemma-4, on fv-ml1 GPU 1. Host renaming explains part of it,
but 31B MeroMero-v2 and 26B stock gemma4 are different models and the comment
block carries sampler defaults keyed to the model it names.
- Two gateway aliases are dead: erp-tune-v2 (:8098) and gemma4-26b-a4b-it-base
(:8099). Nothing is listening on either; callers get a connection failure
rather than a clear retirement error.
- Three different speculative-decoding methods coexist -- dflash k=7 with a
separate drafter on mog-sec, qwen3_5_mtp k=3 on gen, mtp k=3 on flash-next.
The method string is architecture-specific; copying a --speculative-config
between seats without changing it will not work.
- Three seats run the floating :latest tag, so what they would pull on recreate
is not what is running and was never recorded.
- Seven aliases resolve to vllm-gen on :8015, which is why GPU 0 shows draw
during ordinary OpenWebUI use -- its auto-titling fires summarizer per turn.
Also commits the flash-next-seat MTP k=3 change made today: compose gains the
speculative-config with its measured rationale, and .env.example documents the
KV-cache reduction MTP requires (14 GiB OOMs, 10 GiB verified in production).
Operator-directed: raise context to the model's native maximum and take as much KV
as the card safely allows, and expose the seat through LiteLLM as `gen-large`.
max_model_len 131,072 -> 262,144
KV cache 8.76 -> 14.00 GiB (332,721 -> 560,654 tokens)
concurrency 2.54x@128K -> 2.14x@262K
⚠ 16.00 GiB WAS TRIED FIRST AND IS TOO AGGRESSIVE. A 155,497-token non-repeating
prefill drove GPU 2 to 97,074 of 97,887 MiB and the caching allocator logged "OOM on
device 0 while trying to allocate 488636416 bytes (free: 422117376)" -- 466 MiB wanted
against 403 MiB free. The request completed, so nothing failed visibly; that is one
step before the shape that crashed stacks/mog-sec twice on 2026-09-10 (~1.04 GiB
wanted, ~600 MB free). Backed off to 14.00 GiB, which re-probes clean: zero allocator
warnings, a 155,557-token prefill in 14.2 s, and 2,085 MiB still free at peak.
The reason the first estimate was wrong is worth keeping, because it is not obvious
and it inverts the usual advice: --kv-cache-memory makes vLLM SKIP MEMORY PROFILING
ENTIRELY and ignore --gpu-memory-utilization. The profiler was the thing accounting
for deep-prefill activation, so pinning bytes switched off the protection that the
pin was supposed to formalise. vLLM's own "--kv-cache-memory=18745235968 (17.46 GiB)
to fully utilize gpu memory" line is computed from a profile measured at
max-num-batched-tokens depth and sits 3.5 GiB above what a 150K-token request
survives; open #54764 compounds it, since PLE short-conv prefill pads every request
in a batch to the batch-MAX query length.
max-num-batched-tokens stays at 8192 -- it is what bounds the activation peak, and
doubling max_model_len left the profiled peak unchanged at 1.65 GiB precisely because
the peak tracks chunk size, not context length.
Gateway: `gen-large` added to the LiteLLM model_list, pointing at fv-ml1:8022. One
alias on purpose -- a single alias cannot trip the shared-config enable_thinking
mutation footgun, which needs two over the same (model, api_base). Sampling is the
checkpoint's own declared set (temp 1.0 / top_p 0.95 / top_k 20); presence_penalty,
min_p and repetition_penalty are left unset because the checkpoint declares no
canonical value for them. Verified registered for both the infra-ops admin key and
the shared all-agents key, since a new model behind a scoped allowlist 403s silently.
Also adds services/flash-next-mtp-bench/ -- the MTP measurement campaign and its
rationale. MTP stays off, but on "not yet measured here" rather than on vLLM's
4xH100 recipe number, which is a cross-harness comparison and not evidence about a
TP=1 Blackwell seat.
Every fv-ml1 link on the Homepage dashboard was broken. Measured against the
live dashboard API before the fix: 16 entries pointing at the dead 10.250.50.54
and zero at the live 10.251.50.54, covering gen, M.O.G.-SEC, Scriberr, Embed,
Rerank, Reward, Coder, Dockge and six dormant seats.
The miss was structural, not careless. fv-ml1-rename-sweep.sh works from an
allowlist assembled from files that mention the HOST, and a homepage.href label
mentions only an IP -- so every stack whose sole stale reference was a label
fell outside it. The allowlist now covers those 24 files, and records how to
derive the list next time (grep the old address, subtract history) rather than
enumerating from memory.
History is still untouched, and the exclusions are now written down with the
reason each one keeps the old address: recorded benchmark results, whose
base_url is part of a measurement's provenance; the one LiteLLM comment
preserving a retired hand-test endpoint; and the cutover runbooks, where the old
address is the subject matter.
Two bugs found while applying it, both fixed here:
- deploy-stack.sh rejected any stack name containing a dot, so qwen3.5-122b,
qwopus3.5-122b and mistral-medium-3.5 could not be deployed by the script at
all. The check exists to stop path traversal, which means rejecting ".." and
"/" -- not every dot. Traversal is now rejected explicitly and tested.
- stacks/scriberr/.env.example allowed CORS only from the dead IP and from
scriberr.ana.internal, which no longer resolves; the box is at the fv site
and DNS already carries scriberr.fv.internal. The live .env had both stale
origins, i.e. an allowlist with nothing reachable in it.
Host side, applied separately: canonical pushed for the 16 stacks whose only
difference from the host was this renumber, and an in-place address-only fix for
the nine whose host copy has genuinely drifted or has no canonical copy, so that
drift survives for a deliberate reconciliation instead of being clobbered. Every
compose.yaml on fv-ml1 now reads 10.251.50.54. The labels themselves only take
effect at container creation, so the running containers still need recreating.
First seat whose weights do not fit its card. The model is 176B total -- a 125B
main model plus a 51B n-gram (PLE) lookup table -- at ~6B active per token. The
table is a pure embedding lookup, so it lives in pinned host RAM and the GPU
reads rows directly over CUDA UVA: ~78 GiB resident on a 95.6 GiB card, 47.7 GiB
pinned of 566 GB. GPU 2 and GPU 3 were both idle, so this displaced nothing.
Checkpoint dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4 @ be794b99, pinned by
revision: NVFP4 W4A4 routed experts, FP8 PLE table, everything else at source
precision. Chosen over better-liked builds because its provenance states
protocols and repeat counts -- AIME26 pass@1 98.75% over 30x8 repeats with a
stated SEM, full-set GSM8K, and a byte-equality audit covering all 31 MTP
tensors -- and because it declares text_config.ple_embedding_dtype, which is the
field vLLM reads first when selecting the PLE weight format. Builds that ship an
FP8 table without that declaration resolve to the unquantized path and fail on
load; the README records the check.
Requires vLLM #54371 (UVA PLE-offload, merged 2026-09-09T14:32Z), verified by
ancestry: the pinned nightly is +150 commits / behind_by 0 from the merge commit.
Not in v0.29.0, cut six hours earlier. The older worker-based offload (#53899) is
paused upstream and is not the path here -- its deadlocks, ptrace gate and
stale-output-under-graphs bugs all came from the separate worker process that UVA
does not have.
Five deliberate departures from the other seats on this box, each from a
measurement rather than a preference, all annotated in place:
- no MTP: the vLLM recipe measured it worse at every concurrency on 4xH100
(8-36% less throughput, 32-173% more latency, ~36% acceptance)
- modelopt_fp4, not compressed-tensors: only the ModelOpt reader honours the
ignore list keeping attention, shared experts, PLE and MTP out of W4A4
- KV left at auto: fp8 KV on this model's QSA path is an unmerged RFC (#54426)
- mamba-cache-mode stated explicitly: the model raises on mode "all"
- 128K context and 8192 batched tokens, not the native 262K: #54764 and #54919
make depth the risky axis, and sizing to the KV pool has never fixed a
depth-driven crash on this hardware
Nothing is wired into LiteLLM. Pointing an alias at this seat changes what
existing callers receive and is a separate decision.
A zsh glob failure in the staging step of 91bda3c meant the five renamed
playbooks went in as pure renames with their bodies still saying ana-ml2 and
10.250.50.54, and the homepage docker.yaml direct-connection edit from 17586ab
did not stage either. Same content, just the half that got dropped.
stacks/ana-ml2-proxy existed for one reason, stated in its own README: ESH
could not reach the box's Docker API cross-site because of a return-route gap,
so ana-docker relayed :2354 -> :2375 from the same subnet. The README named
its own retirement condition -- "once it has a working return route to ESH,
point Homepage back at it directly and compose down this stack".
That condition is met. FV is on the mesh, vb-gateway accepts routes, and the
NH3 UDM, ESH UDM and Anaheim FortiGate all carry 10.251.0.0/16 statics toward
their local scale nodes. Verified before removing anything: esh-docker-vm
opens a TCP connection to 10.251.50.54:2375 directly.
Homepage now points at 10.251.50.54:2375 and discovers 112 services across 20
groups, so nothing regressed. The socat container is stopped and removed from
ana-docker; the compose file and README are deleted here rather than renamed,
because renaming would preserve a moving part that no longer earns its place.
The box is physically at Fountain Valley, renamed, renumbered onto 10.251/16,
and serving inference again. This lands the repo half of that.
Host: hostname ana-ml2 -> fv-ml1, pinned to 10.251.50.54 by a dnsmasq
reservation so the address the runbook, DNS and LiteLLM all assume is the
address it actually has. Its headscale node is renamed too.
The sweep ran from scripts/fv-ml1-rename-sweep.sh, whose allowlist is the
reason this diff touches current-state files and not the record. Dated
persistent-memory entries, archival-memory and incident notes still say
ana-ml2 in 31 and 62 places respectively, because that is what the box was
when those things happened. Rewriting them would make the history lie.
LiteLLM was the load-bearing piece and needed more than the api_base sed the
runbook describes. Twenty api_base entries repointed, but a grep-and-verify
pass also caught a LIVE pass_through_endpoints target for the scalar-judge
reward route still on the old address -- an api_base-only substitution would
have left it dead. Four prose references describing current state were
repointed as well; one historical note recording where a hand-test was run
is deliberately left pointing at 10.250.50.54.
Two facts in the server tables were wrong and are corrected here. The site is
Fountain Valley, not Anaheim. And the box has FOUR RTX PRO 6000 Blackwell
Max-Q, not two -- verified by nvidia-smi -L and independently by PCI
enumeration of four GB202GL devices. That is 391 GB of VRAM rather than 196,
which changes what fits on it.
DNS: fv-ml1, fv-ml1-bmc and fv-gw added under the fv site via the piggyback
approach, scriberr re-homed, and the ana-ml2 records removed. Applied to all
three resolvers. The BMC record carries a warning that its 802.1q VLAN tag
must stay disabled -- it shipped tagging VLAN 250 into an untagged port,
which made it invisible to every network-side diagnostic and is the reason
it appeared dead through several cable changes.
Verified end to end: summarizer and sec both answer through the Anaheim
gateway across the mesh to FV seats on different ports.
Beszel agents are installed and verified across the fleet but the artifacts that
produced them were never committed, so the deployment existed only on the hosts.
Adds the per-host agent environment files (PORT, NICS, EXTRA_FILESYSTEMS and the
hub's PUBLIC key), the systemd unit, the guest install script, the Synology
compose, and the elway playbooks for native, guest-stage, guest-install and
Synology paths. The two dated memory detail files covering the priority-1 and
priority-2 waves ship alongside, per the convention that memory lands with the
work it describes.
No credentials here. The KEY= value in every host env is the Beszel hub's public
ed25519 key, identical across all nine and public by design; the agent README
says so explicitly. The nh3-nas sudo password referenced in the runbook prose
lives in Vaultwarden and the helper scripts named there never contained it.
⚠ Overlapping VMIDs across hypervisors are a standing trap and are recorded in
the priority-2 notes: pfi-pve 105=postgres and 100=pbs-ana, nh3-pve 105=pbs-nh3.
⚠ PBS-NH3's export was ~75.5% used at capture; resource checks are not job
success monitoring and should not be read as such.
Operator: "loading up the context killed sec again." That reproducer is what
finally made the failure legible, and it showed the previous four fixes had all
been aimed at the wrong quantity.
What the KV pool can hold and what the card can process at depth are different
numbers. Cutting context 420k to 384k to 320k, pinning the KV in bytes, and
dropping the prefill chunk from 16384 to 4096 all sized the pool. The crashes were
governed by the transient needed to process a prefill chunk against a quarter
million tokens of resident KV, which scales with depth and not with pool size.
Each change helped and none fixed it.
Bisected against the real reproducer, with a non-repeating prompt because prefix
caching would let a repeated one hash to cached blocks and never prefill deep:
113,247 prompt tokens SURVIVED (27 s)
200,088 prompt tokens SURVIVED (174 s)
~285,000 prompt tokens ENGINE DIED, HTTP 500, container restarted
The sustainable ceiling therefore sits between 200k and 285k with gen idle, and
gen shares the card with its load uncontrolled, so 163,840 takes about 20% margin
under the proven-good depth rather than sitting at the measured edge.
The ceiling's purpose is the refusal. Verified after the change: a 149,073-token
request serves in 41 s, and requests at both 200k and the ~285k depth that killed
the engine now return a clean 400 naming the limit in under a second with the seat
untouched. A seat that refuses what it cannot serve is strictly better than one
that dies trying.
Concurrency went 1.03x to 2.09x. The compose header's "served at native 262K" was
never actually deliverable on a shared card; it had simply not been exercised at
depth until today.
The probe is committed rather than described, so the ceiling can be re-measured
when the card's tenancy changes.
Operator: "yes, pin the kv and take it to 320k."
The real finding, which took three crashes and two failed attempts to reach:
--gpu-memory-utilization does not bound actual usage. It sizes the KV
calculation, but peak activation is measured at profiling time and real
long-context work exceeds the profile. vLLM's own budget line showed mog-sec
running 0.9 GiB over its 47.48 GiB reservation -- 26.44 consumed plus 3.53 peak
activation plus 0.89 CUDAGraph plus 17.52 KV equals 48.38 -- and gen was over by
0.33 on the same card. That overage came out of the shared card's slack, which is
what kept OOMing after the utilization drop.
The fix is the one vLLM printed itself: --kv-cache-memory=17697765376, its own
recommended figure to fit inside the requested budget. Same discipline erp-seat
already uses, and for the same stated reason -- an explicit figure is
reproducible where a ratio silently yields a different cache depending on what
else is resident at start time.
The KV pin and the context length are coupled. 16.48 GiB yields about 383,730
tokens, so a 393,216 max_model_len falls under the 1.0x floor and vLLM refuses to
start rather than crashing later; pinning the KV while keeping 384k was never an
available combination. 327,680 leaves 1.15x, up from 1.03x.
Verified: the engine now logs "reserved 16.48 GiB memory for KV Cache as
specified by kv_cache_memory_bytes config and skipped memory profiling", KV
375,901 tokens, GPU0 down to 90,561 MiB from 91,313, RestartCount 0, and both sec
and sec-reasoning return 200 through the gateway.
Also records the BabyBronte eyeball A/B, whose result is the operator's own: the
voice transferred and the sense did not. Curly quotes went 1 of 18 to 18 of 18
and worksheet collapse 3 of 18 to 0 of 18 between arms. That voice is separable
from coherence at 0.6B is the premise the lightweight-adapter regime rests on, so
this is the informative outcome rather than a disappointing one. A corpus-prep
defect surfaced with it: the tuned output is hard-wrapped at about 70 characters
because the Gutenberg source kept its line breaks and the adapter learned the
typography too.
Cost: 320k of context instead of 420k, on a seat whose crashes happened at 151k.
sec/sec-reasoning crash-bounced twice in ten minutes, not once in thirteen days.
My earlier read of "rare, not chronic" came off a RestartCount of 1 and was
wrong; the operator pushed back and the second and third failures arrived while
that recommendation was still on screen. The memory entry making that call is
replaced rather than left standing.
Cause is unchanged from the first diagnosis: mog-sec at 0.52 plus gen at 0.38
reserve 0.90 of the card, leaving about 4.6 GiB, and vLLM's utilization figure
covers weights and the KV pool but not transient activation memory. A request
about 151,700 tokens deep scheduling a further 15,700-token chunk asked for 1.04
GiB with roughly 600 MB free.
Dropping utilization alone does not work, and fails in a worse way: a single
420,000-token sequence needs 17.88 GiB of KV, and at 0.50 the pool is 17.4 to
17.5 GiB, so vLLM refuses to start at all and the seat crash-loops during startup
instead of during a request. The context length and the crash were directly
coupled -- 420k was only reachable at the utilization that left no transient
headroom. So both moved: 0.50 and 393,216.
384k rather than vLLM's suggested maximum, deliberately. It estimated 406,352 on
one boot and 409,840 on the next, because the available-KV figure drifts about
0.1 GiB boot to boot; pinning the edge value fails to start on an unlucky boot.
393,216 sits 3% under the lower estimate and leaves roughly 0.7 GiB of the pool
unspent, which is the transient headroom the change exists to buy.
Verified after: KV 405,612 tokens, concurrency 1.03x at 393,216, and both sec and
sec-reasoning return 200 through the gateway.
num_speculative_tokens is documented as NOT the lever. The crash window logged
17.6% draft acceptance with positions 5 through 7 at 1.5 to 4.9 percent, which
reads as an obvious cut from 7 to 3; across 180 samples the median acceptance
length is 3.12 of 7 and median draft acceptance is 30.4%, so the crash window sat
near the minimum and cutting would cap the workloads accepting nearly the full
draft.
Cost: 384k of context instead of 420k, an 8.5% reduction on a seat whose crashes
were happening at 151k.
Operator: "replace that a4b moe over pfish-6 -- remove the pfish-6 alias and
create an alias for char-rp-fast."
G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16 is live on ana-ml2 :8021 under
its own served name, behind the new gateway alias char-rp-fast. Pfish-6 is gone
from the gateway and now returns an explicit 400 rather than a substitution; 0 of
17 LiteLLM keys scoped it, so nothing was orphaned. The compose project name stays
erp-seat because asset-engine derives seat liveness from it.
The first quant of that A4B served NaN and passed its healthcheck doing it. It was
built with the dense v2-31B recipe, whose ignore list has no router regex, so all
30 MoE routers were quantized to 4 bits -- and a 4-bit router changes which experts
run rather than degrading them. Quant rc=0, healthcheck green, correct KV pool,
correct served name, and every completion returned finish_reason=length with the
full token count and content: null. The model was emitting a full budget of tokens
that decoded to the empty string. Raw /v1/completions was empty too, ruling out the
chat template and the reasoning parser. The signal that named it was logprobs:
vLLM refused to serialize the response, "Out of range float values are not JSON
compliant: nan".
The lesson is about the control rather than the router. That tree had already been
structurally diffed and passed -- against a verified-good DENSE quant of the same
Gemma-4 family. A dense model has no routers, so the single thing that was wrong
was the single thing the control could not distinguish. Diffing instead against
Pfish-6, a known-good quant of the same 26B-A4B MoE, gave it in one line: 222
ignore entries against 252, the 30 missing being layers.N.router.proj. A positive
control is only worth what it can distinguish, and "same family" is not "same
architecture class".
Re-quantized with the MoE recipe, whose dry-run asserts 11,520 expert Linears and
refuses a router in the quantize set before any GPU time. The live seat then passed
prose with no channel-prefix leak, a solid-colour image read correctly, an auto
tool call parsed, finite logprobs, and KV 534,649 tokens / 2.04x carried over from
Pfish-6 unchanged. The broken tree is parked on ana-ml2 as
...-NVFP4A16.BROKEN-routers-quantized-20260910.
Section 4.4's temp port was not reachable: 15.9 GiB of weights plus KV plus
multimodal encoder-cache profiling does not fit in the ~19 GiB free beside GPU1's
six other tenants -- 0.20 utilization refused admission, 0.185 OOM'd in encoder
profiling. The substitute was reversibility and ordering: named .env backup, prove
the seat on its real port while no alias points at it, move the alias last. That is
why a NaN-serving seat never reached a consumer. The seat was down about 16 minutes
across two attempts; no consumer saw a broken alias.
Playbook gains the router-quant failure signature and the control-class rule in
3.15, and a logprobs check in 4.4. seat_verify.py carries that check as check 6.
Quality is NOT established: no RP eval, no long-context check, no A/B against
Pfish-6 or char-rp. Samplers are the author's card values, untuned here.
Ships the two pending code changes alongside the memory that describes them:
train_voice_lora.py gains --eval-steps/--save-steps (the 3-epoch pilot overfit
with per-epoch eval and save_strategy="no", so the minimum was neither visible
nor recoverable), and the althing post-office compose is pinned to 3.6.2.
Index rewritten: in-flight compressed from 176 lines to the live state, three
detail files added for R49 D1-D3, the MeroMero four-failure chain, and the
althing rollout. Seven closed pre-08-27 entries archived; the guards held back
the rest because they carry open deferred pointers.
Operator ruling: run 6 is declared Pfish-6 and is the standing seat; run 7 is
retired (its gate failure was the detector bug fixed in cc42d76, but the run was
independently poor).
- served under its TRUE name Pfish-6; LiteLLM alias trial -> Pfish-6
- ana-ml2 :8021 and pfi-gx10 :8098, both at 262,144 ctx, same artifact
- SPEED: moe_backend=flashinfer_cutedsl PROVEN unusable here (engine init fails,
'kernel does not support current device'); Marlin is correct for a weight-only
scheme, and vLLM's 'no native FP4' warning is about the scheme not the card
- the real lever was max-num-seqs 8 -> 32: n=8 1269 tok/s and n=16 2170 tok/s,
3.2x the old ceiling, with single-stream latency unchanged
- head-to-head measured: ana-ml2 is 4.1x FASTER than the GX10, not slower
Alias-present-with-backend-down is a different and worse state than
alias-removed: it re-arms silently the moment anything is served on
10.250.50.54:8021 again, and the next thing served there inherits a
fleet-reachable name without anyone deciding it should. Flagged by
brokkr-smithy-dev; same writer/reader-contract family as ops-lessons §11.
Commented rather than deleted so restoring is uncommenting, not reconstructing.
Verified: /v1/models returns 33 entries, trial absent, a caller gets a clean
400 'Invalid model name' instead of a connection error to a port that could
re-arm. Disposition of the artifact and the alias is the operator's.
Config-file deployment (the trial model lives in config.yaml, so /model/update refuses it):
block rewritten to run-6 reality — abliterated base, NVFP4A16 artifact, stacks/erp-seat,
no gate on this artifact by operator ruling, same-site hop. Deployed to ana-docker and the
gateway restarted; trial -> erp-tune-v6-nvfp4a16 verified end to end (3/3).
- services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py: linearize_moe first (playbook §3.15),
asserts the expert Linear count, routers/vision/audio/norms/lm_head ignored, W4A16 for RP
long-session fidelity, post-steps restore processor configs + template and reset the
tokenizer truncation cap (§3.14); --dry-run proves targets before GPU time
- services/erp-seat-quant/run_quant_erp_v6.sh: detached container on GPU1 (vllm-llmcompressor)
- stacks/erp-seat: serve recipe copied from gemma4-charrp, true served name only, port 8021
Operator decision post-RESCUED verdict: point the LiteLLM trial alias at the
run-5 tuned seat (erp-tune-v5 on gx10:8098) for hand-testing prosody/feel in
Open WebUI. Rewrote the trial config block to run-5 reality, including the
now-MEASURED refusal-erosion note (k=25 both arms, de-gated + stated) replacing
the run-3c 'safety axes unmeasured' warning. Verified end-to-end: trial ->
erp-tune-v5 returns coherent output through the gateway. Container restarted to
re-read the bind-mounted config (up -d alone is a no-op for a config-only change).
ana-ml2 is up but not reachable cross-site from ESH (return-route gap on ana-ml2,
no SSH access to fix it). New stacks/ana-ml2-proxy: a pinned alpine/socat relay on
ana-docker (host net) exposing 10.250.50.70:2354 -> ana-ml2 10.250.50.54:2375.
Homepage docker.yaml ana-ml2-docker re-enabled pointed at the proxy. Verified:
esh-docker-vm reaches ana-ml2's docker (29.3.1) through it; /api/services returns
in 0s with 111 services; ana-ml2's inference seats (gen, coder, embed, rerank,
reward, image-bench, dockge) render. Reversible: point back at 10.250.50.54:2375
and compose-down the proxy once ana-ml2 routes to ESH.
The 2026-09-06 headscale cutover retired irv-ml1's wg0 tunnel IP 10.100.79.3
(now 10.6.110.50). Repointed all LIVE canonical refs to the DNS NAME so the next
move can't re-break them: homepage.href/siteMonitor labels across 25 stack
composes, load-bearing env defaults (asset-engine INFERENCE_HOST, open-webui
AUDIO_TTS_OPENAI_API_BASE_URL, skaldsong SKALDSONG_TTS_BASE_URL, zonos-gateway
ZONOS_URL, dia), homepage services.yaml manual cards (Voice Design Studio,
IRV-ML1), and servers/irv-ml1/ssh-target. Updated the stale 'WG tunnel' comment
to the mesh reality.
Left as-is: README curl-examples and .env.example comments (docs), and historical
mentions in CLAUDE.md/persistent-memory. NOTE: applying the label repoints to the
RUNNING irv-ml1 containers needs a recreate per service (labels read at creation);
deployed .env values are separate from these canonical defaults.
WhereTF (operator-directed, brokkr-smithy-dev; nh3-dev:8093, user-systemd Bun
service): manual Homepage card under Apps (can't auto-discover) + DNS alias
wherethef.nh3.internal -> nh3-dev.
While adding the card, found Homepage's /api/services was hung (dashboard
degraded since the 2026-09-06 irv-ml1 move): its docker.yaml still pointed
irv-ml1 at the dead 10.100.79.3, and ana-ml2 (10.250.50.54:2375) is unreachable
from esh-docker-vm — either dead host hangs the whole aggregation on its TCP
connect timeout. Repointed irv-ml1 to 10.6.110.50 (verified reachable via mesh)
and disabled ana-ml2 (reversible, noted). /api/services now returns in 0s with
96 services; WhereTF renders.
ext-tts / tts-1 / tts-1-hd / gpt-4o-mini-tts (all openai/zonos) were pinned to
irv-ml1's pre-move address 10.100.79.3:8198, dead since the 2026-09-06 headscale
cutover to 10.6.110.50 — so ext-tts through the gateway hung. Reported by
tts-dev/svos-dev.
The 4 aliases are DB-backed (store_model_in_db), so their api_base was updated
via the admin API to the DNS name http://irv-ml1.nh3.internal:8198/v1 (not a
fresh IP — that just re-arms the trap on the next move). The container cannot
resolve *.nh3.internal via split-horizon DNS, so this adds an extra_hosts entry
mapping the name to 10.6.110.50. Verified: container resolves the name and a
live ext-tts/sindra call returns 200 + valid MP3.
The tools had been on 3.6.0 while the container sat on 3.0.0 for seven days, so
the operator verbs failed with "no tool named delete_handle" — they live in the
post office, not the client.
Image built on nh3-dev from a clean tree at 4d26226 and pushed under the
claude-bot namespace; the compose pin moves to the new digest rather than
floating on the tag, since this container is the fleet's whole message bus.
The backup procedure this file documents earned itself again: at stop time the
database was 23.8 MB with a 5.9 MB WAL beside it, so a plain copy would have
produced a database that opens cleanly, smokes green, and is missing the day's
mail. Stop, explicit checkpoint to a zero-byte WAL, copy, then verify counts on
both sides — 76 handles, 995 messages, 1022 recipients, integrity ok.
Post-deploy the same counts came back with handles.retired_at present, and the
memory cap and OOM guard were confirmed by `docker inspect` rather than by
reading the yaml, which is what that file asks for and the only check that can
tell a working cap from a decorative one.
Bus down about four minutes.
Operator asked for it explicitly so he can test in Open WebUI in parallel with
brokkr's batteries. Named `trial` rather than erp-tune-v3c: the run-3 note in
this file records why a tier name arriving before its evidence is a mistake,
and 3c has no more evidence than run 3 had.
Points at the pfi-gx10 seat (http://10.100.50.60:8098/v1, vLLM 0.28.0 on
GB10/sm_121), so this is a cross-site route -- NH3 seat, Anaheim gateway.
⚠ Recorded in the entry, not just here: 3c's safety axes are unmeasured, and
run 3 -- same recipe, differing only in learning rate -- was withdrawn from
this gateway 40 minutes after being added for selfharm/methods -44.0pp and
meta/sysprompt -28.0pp against its own base. This gateway is the shared-key
surface. When brokkr's batteries read out, the entry should be re-decided on
evidence rather than left standing by default.
Applied with a restart of the litellm service only (~60 s of gateway
downtime); the compose .env is root-owned so the restart needs sudo.
The ana-docker instance was returning zero results for every query while
reporting healthy — 4.5 months stale (2026.4.17 against a current 2026.9.3),
its engine scrapers rotted against sites that had changed. /healthz proves
the web app answers and says nothing about whether search works, so seven
days of green sat on top of a search box that found nothing.
Moved to nh3-docker rather than updated in place, because the colo egress is
the other half of the problem: 38.120.12.42 is a datacenter address that
DuckDuckGo and Startpage CAPTCHA, while nh3-docker egresses residentially at
70.230.226.88. Same reasoning as the fleet's residential proxy for yt-dlp,
applied at the source instead of around it.
Config corrected along the way: base_url said searxng.pfi.local, a name
retired on 2026-08-19, while the environment said something else — the env
won so nothing broke and the file quietly lied. The karmasearch.videos
removal key never matched, because the engine's real name has a space.
scripts/searxng-health.sh asserts results > 0 across three unrelated
queries. That is the check that would have caught this, and the only kind
that can: the mechanism was healthy throughout.
services/searxng-mcp exposes it as `web_search` at user scope, so every
Claude Code session has it. Zero results raise rather than returning an
empty list — an empty list is indistinguishable from a broken aggregator,
which is precisely how this hid.
Old instance stopped and removed; DNS alias repointed to searxng.nh3.internal.
The gen-reasoning seat accepts only xhigh/medium/low and 400s on anything
else — including `high`, which is the default of several clients, so the
seat presented as broken rather than as one enum value out of step. The
DeepSeek Harness failed every request on its default setting, and the only
working client value was `low`: the seat's WEAKEST reasoning tier, while
its own default is xhigh.
conf/reasoning_effort_map.py is a pre-call hook in the same shape as the
existing strip_empty_tools hook. It is scoped to one model group, measured
rather than assumed: gen-reasoning rejects `high`; gen, sec,
char-rp-reasoning and summarizer all accept it and are left alone. Paid
passthroughs were not probed, because probing them spends vendor credits,
and are not mapped.
Verified after deploy: high and max now succeed on gen-reasoning, low and
xhigh still work, a request with no effort param still works, `gen` with
`high` still passes through unmapped, and the harness completes a real
file-edit task at full reasoning.
Needed a compose change as well as a conf push — callbacks are bind-mounted
per file, so the volume only attaches on container create. Recreated the
litellm service by name so the DB was not bounced with it.
I wrote "one-line fallback to pfi-fleet costs you nothing" to a peer,
meaning a manual edit; it was read as scheduler behaviour, and
`runs-on: [vastblue, pfi-fleet]` would have shipped into a contract. A
label list requires ALL the labels, so that form matches no runner and
queues forever instead of failing — the worst outcome of the three,
because a queued job looks busy rather than broken.
Standard Actions semantics rather than a gitea quirk; the ambiguity was
in my prose. Documented where the label guidance already lives so the
next reader does not have to have been in the conversation.
The README's capability table is a measurement with a date on it, and a
runner upgrade or a gitea bump can invalidate it without anything saying
so. Committing the workflow that produced it means the next person can
re-derive the claims instead of trusting them, which is the same posture
the training probes take.
Carries the three job shapes plus the traps each one cost to find: job
images need a node binary or the first `uses:` fails, the Postgres service
needed ~6s and a pg_isready wait, and a run must be polled at
/actions/runs because /actions/tasks reports it as empty on 1.26.1.
Requested by vastblue-dev as a reference for their first smoke.
Answering a CI-posture question from vastblue-dev meant measuring three
things rather than recalling them. Two came back the opposite of the way
the config reads:
- `container.valid_volumes: []` does NOT keep the docker daemon out of
jobs. act_runner mounts /var/run/docker.sock on its own, so every job
on the shared runner is uid 0 with `docker ps` over all 49 containers
on ana-docker — gitea, synapse, phasefinal-web, adguardhome included.
It is also load-bearing: four repos drive buildx through it, so the
fix is isolation onto a dedicated runner, not tightening this one.
- A full-URL `uses: https://gitea.phasefinal.com/actions/checkout@v4`
resolves from the local mirrors today. That is github-independence per
workflow without the DEFAULT_ACTIONS_URL flip that has been parked on
act_runner's action-fetch auth since 2026-08-05.
Also recorded: `services:` containers work (Postgres 16 on the service
name), job images need a node binary for JS actions, and `/actions/runs`
lists runs that `/actions/tasks` reports as empty on gitea 1.26.1.
Measured on a throwaway repo under the claude-bot account, since deleted.
config.yaml change is comment-only and deliberately not deployed — it
would bounce the runner for no runtime effect.
Synapse mounts its admin API on the same vhost as the client API, so publishing
matrix.phasefinal.com published the admin surface too -- it answered 200 from
the open internet. HMAC-protected, so not an open door, but Synapse's own
guidance is to keep it off the public listener.
A higher-priority router (explicit priority 100, not relying on Traefik's
rule-length tie-break) scopes PathPrefix(/_synapse/admin) behind an ipallowlist.
Verified from a genuinely external vantage rather than from a fleet host, since
nh3-dev sits inside the allowed range and would have proved nothing: via the NH3
residential egress proxy the admin path returns 403 while the client API returns
200 and Element is unaffected.
The 10.0.0.0/8 entry matches nothing today and the comment says so rather than
implying fleet access exists. matrix.phasefinal.com resolves publicly, so fleet
hosts hairpin out their own WAN -- a request from nh3-dev arrived as
70.230.226.88. The rule is effectively deny-all through Traefik, which is the
intended posture: admin work goes through docker exec to localhost:8008 and
never traverses Traefik. Allow-listing the sites' WAN addresses was considered
and rejected as a maintenance trap on dynamic addresses.
Also brings the stack under stacks/ with the Postgres password replaced by a
required .env variable. The tracked copy and the live file have therefore
DIVERGED and deploy-stack.sh must not be used until the live file reads from a
.env; the README says so.
Cache rule on www.phasefinal.com with edge and browser TTL both respect_origin,
so cache policy stays declared once in nginx.conf rather than split between the
repo and the dashboard. Always Online enabled, which is what actually survives
an origin outage; a 300s document TTL alone would only mask five minutes.
Verified: document and assets both reach cf-cache-status HIT, apex 301s to www,
edge email obfuscation active.
The healthcheck used http://localhost/, which resolves to ::1 in nginx:alpine
while nginx listens on IPv4 only — so it never passed, the container stayed
unhealthy, and Traefik silently declined to create a router for it. That
presents as a broken docker provider: correct labels, right network, no route,
no error. Target 127.0.0.1 explicitly and add a start_period.
Adds the apex router (301 phasefinal.com -> www) and drops the file-provider
workaround, which was mitigating the wrong diagnosis.
Single static page (nginx) fronted by Traefik at www.phasefinal.com, built
from the design brief. Site markup/CSS checked in verbatim from the design
session; fonts self-hosted (SIL OFL) with the @font-face block uncommented,
which every fresh export re-comments.
Routed via a Traefik file-provider config rather than the container labels:
the docker provider on ana-docker was not registering newly-created
containers, so the file router avoids restarting shared ingress. Labels are
retained in compose so the file can be dropped once that is fixed.
Labels the container into `Toolchain`, an existing group under the
existing Toolchain tab -- "the plumbing", which is where a message bus
belongs. Confirmed live: Homepage's API now returns it.
I had previously recorded in this file that no group fitted, which was
wrong. That conclusion came from a grep over the layout block that
missed the nested groups, and it went into a comment as though it were
a finding. The group was there the whole time.
Labels bind at container creation, so this deployed with `up -d` rather
than `restart`; a restart leaves the old labels and the dashboard keeps
showing what was there before. nh3-docker is already a discovered host
in homepage's docker.yaml as `nh3-pfi-docker`, so the label alone is
enough -- adding a services.yaml entry as well would render the card
twice.
althing-chamber on ana-docker also carries Toolchain labels and is a
separate service per the operator. Left alone.
## Registry
The image moved by `docker save | ssh | docker load`, so a rebuild meant
repeating that by hand. It is now published and the compose pulls a
digest-pinned reference, so a redeploy is `compose up -d` on any host
that has logged in.
Pinned by digest rather than by tag: `:3.0.0` is a mutable pointer on a
registry anyone can re-push, and this container is the fleet's whole
message bus. The tag rides alongside so a human can read what it is.
Namespace is claude-bot, not vh. claude-bot's token carries
write:package and `docker login` succeeds, but package namespaces are
owned -- pushing to vh/ returns "unauthorized: authentication required"
after a successful login, which reads like a credential fault and is
actually an ownership one. Publishing under claude-bot's own namespace
also satisfies the standing directive to stop reusing the operator's
personal credentials for infra work, so the constraint and the policy
point the same way. Recorded in the compose header so the next person
does not read that error as a broken token.
Pull path proven rather than assumed: the running container was
recreated from the registry reference and its data verified afterwards.
## Statusline
Brought under version control because the v3 cutover broke it invisibly.
The segment gated on `command -v althing-cli`, a binary the cutover
deleted, so the unread badge and the armed bell silently vanished for
every session on the box. With 71 of 73 handles pull-only, that badge is
the only out-of-band signal telling a session with no armed waiter that
it has mail -- a dead statusline made a working bus look like an empty
one.
Canonical here, live at ~/.claude/statusline-command.sh, copies rather
than symlinks per the same rule as stacks/.
Operator directive, and a standing goal: the bus belongs on the docker
host. The flag-day deployment put it on nh3-dev because the herald lives
there -- but the herald is the piece that must be host-local, and the
post office is explicitly the piece that is not.
nh3-dev was wrong on three counts. Our own server table calls it "not a
Docker-stack host". It has had three OOM events in fourteen days with
the interval halving, and the confirmed hog is Claude Code sessions at
5-18 GB, which is that box's actual job. And mem_limit protects the
fleet from the post office while doing nothing in the other direction:
oom_score_adj was 0, an ordinary kill candidate, on a box whose last
sweep took althing-herald and uvicorn. The new deployment sets
oom_score_adj=-500.
The compose is now version-controlled here as a normal stack rather than
living only in the althing repo's deploy dir.
## docker stop does not checkpoint the WAL
The database was 155 KB with a 4.1 MB write-ahead log, and every recent
message was in the log. A clean container stop left it untouched -- an
explicit PRAGMA wal_checkpoint(TRUNCATE) was required.
A docker cp of the .db alone would have produced a database that opens
cleanly, passes integrity_check, serves the full 73-handle roster, and
is missing the day's mail, with nothing raising an error. Row counts
were verified at source, in the staged copy, and after seeding, because
the count is the only thing that separates those two outcomes.
The old volume is left in place. Not a rollback path, which the operator
ruled out -- just not deleting the only other copy on the day of a move.
## Follow-up left open
The image has no registry push and moves by save/ssh/load, so a rebuild
means repeating that by hand. It should join the gitea registry pattern
the other stacks use.
vh