diff --git a/archival-memory.md b/archival-memory.md index bfe42b0..846ef98 100644 --- a/archival-memory.md +++ b/archival-memory.md @@ -29,6 +29,2288 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re # esh-pve-nas — PVE root on a USB DOM: diagnosis, mitigation, migration plan +- `[2026-08-25]` **The ERP/RP tune COMPLETED in 7.36h and passed its gate on the axis it was built for** — diversity 22x its noise floor, attractor −11.3pt, zero memorisation on both arms. Also the noise-floor near-miss: brokkr was one step from reporting a 13-point T6 regression sitting inside twice his instrument's own variance. +# ERP/RP tune run-01 COMPLETE — 7.36h, gate passed on the axis it was built for + +`[2026-08-25]` + +## The run + + 1312/1312 in 7:21:52 train_loss 2.793 epoch 1.0 + 20.1 s/it FLAT across every 100-step window (round 1: 35-46.5 s/it) + adapter: /tank/erp-tune/run-01/adapter/ 410 tensors, provenance.json + +**47% faster than the round-1 projection of 13.85h**, from two changes: the +bucketed sampler and flex attention. Rate was flat — 19.7 / 19.8 / 20.4 / 20.3 +across the four 100-step windows — which means the 35-46.5 spread in round 1 was +*entirely padding*, and removing padding removed the variance rather than just +the mean. + +⚠ **I quoted three different ETAs (6.9h, 8h, 7.3h) before I started using a +rolling average.** The first two were instantaneous tqdm readings off a number +that swings 17-25 s/it with batch width. Only the rolling rate was honest. Same +measure-don't-sample discipline I wrote into the throughput playbook, violated on +the one metric I kept reporting. + +## lora_B gate — PASSED, twice + + checkpoint-100 205/205 non-zero, median norm 0.829 + final adapter 205/205 non-zero, median norm 1.708 + vision_tower tensors: 0 on both + +Median norm rising 0.829 -> 1.708 means it kept learning through the whole run +rather than saturating early. This check **never ran in round 1** (died at step +19, first checkpoint was 100) and it is the only failure mode that stays +invisible until the acceptance gate reports base-identical numbers. + +## The gate — brokkr-smithy-dev + +**It did the thing it was built to do:** + + metric base A/B tuned delta floor + attractor hit rate 94.8% / 96.8% 84.5% -11.3pt 2.0pt + diversity (pairwise) 0.213 / 0.221 0.3948 +0.178 0.008 + +Diversity moved **22x its own noise floor**. Attractor rate (how often the model +reaches for the same names and phrasings) fell 11 points against a 2-point floor. + + T1 100 · T2 95 · T3 96-97 · T4 98 · T5 100 · T6 81-82 · core ~94.2 + memorisation: 0.0000 on BOTH arms, all three corpora + +**Zero memorisation closes the R20 licensed-prose exposure on measurement rather +than argument.** + +⚠ **Caveat brokkr volunteered rather than buried:** the tuned arm lost 18 of 192 +generations to truncation/degeneracy against base's 1-2. Lopsided exclusions +plausibly flatter the diversity magnitude. Direction is unambiguous at 22x floor; +the number carries an asterisk. + +## The noise-floor near-miss — the methodology lesson + +brokkr was one step from reporting a 13-point T6 regression **that sat inside +twice his instrument's own variance.** + + --per-type 32 max swing across tasks: 9 points + --per-type 128 max swing across tasks: 1 point + +His gate criterion is "no task regresses by more than one item" = 3.1 points at +n=32. **The instrument's own run-to-run noise was 3 items.** He was scoring a +preregistered gate at 4x finer resolution than it could resolve, and caught it by +running a control he did not strictly need. Quadrupling n collapsed the noise +exactly as binomial statistics predicts. + +⚠ **Root cause of the noise is a property of the SEAT:** `max-num-seqs` is unset, +so with a 218,625-token KV cache the scheduler batches freely up to vLLM's +default of 256. Continuous batching changes reduction order and borderline items +flip. Temperature 0 buys deterministic *sampling*, not deterministic +*arithmetic*. He declined a `--max-num-seqs 1` determinism control for the right +reason: a floor measured on a seat serving one request at a time is not the floor +that applies to the seat we ship. + +## The confound I built and he caught + +I optimised a pipeline for production and then handed him its output as an eval +instrument **without asking whether those were the same job.** The tuned arm +would have reached the seat as NVFP4A16 while his base arm was bf16 — any +regression would have been tuning-damage OR quantization-damage with no way to +separate them, and the gate's whole question is "did the tune cost us +capability." + +**Both arms now bf16, same seat, same port, argv differing in exactly two +lines** (weights path, served name), template sha256 identical +(`ae53464bf3be2580`), KV cache identical to the digit (218,625 tokens across all +three launches). Quantization moved *downstream* of the gate. + +See [[2026-08-25-refusal-retention-probe]] for the axis his gate did not have. + _Archived 2026-09-09._ + +- `[2026-08-25]` **8.6% MFU was an accounting artifact — real utilisation 17-20%, and the cost was attention on AMPERE kernels.** Two independent methods agreed to 2.6 points. Fixed by bucketing (padding 29.9% +# The 8.6% MFU was an accounting artifact — attention on Ampere kernels + +`[2026-08-25]` + +## The answer + +**Real utilisation was 17-20%, inside the honest stock band.** The 8.6% divided +the *intended* (windowed) FLOPs by the wall time the *dense* reality took. + + nominal billed 27.1 TFLOPS x 34.85 s = 9.4e14 FLOP + dense-sliding extra 25 layers, 2 seqs, 4 passes = +8.2e14 + padded full layers lose the causal skip = +3.5e14 + work performed ~ 1.8e15 = 51-61 TFLOPS + +The card was doing ~2x the arithmetic the architecture specifies, and the excess +was the sliding window being computed and thrown away. + +## Two independent methods agreed + + scaling fit (3 points, 2 params, residuals <3ms over 8x range) + A = 6.87e-4 s/token B = 8.85e-8 s/token^2 + quadratic share: 20.9% @ w=2048 -> 67.8% @ w=16384 + + kernel table (device rows only) + attention 22,835.8 ms 65.2% fmha_cutlass*_sm80 + dense GEMM 2,774.0 ms 7.9% + other 5,739.0 ms 16.4% + +**67.8% vs 65.2% — 2.6 points apart, no shared assumptions.** The two-term fit +needed no constant term, which refutes launch-bound outright (3,840 expert-GEMM +launches per forward are not the cost). + +## The mechanism, source-verified by brokkr's panel (arm: Bil) + + masking_utils.py:292-301 _ignore_causal_mask_sdpa requires + kv_length < local_attention_size. 16384 >= 1024, + so THE SLIDING MASK ALWAYS MATERIALISES. + sdp_utils_cpp.h:259-267 flash rejects ANY explicit mask + sdp_utils.cpp:647 cuDNN head_dim capped at 128 -> unreachable + Context.h:480-485 prefer-cuDNN needs major 9 or 10; sm_120 is 12 + +⚠ **The kernels are `sm80` — Ampere-generation CUTLASS on a Blackwell card**, +with the forward on `gmem`, the memory-efficient backend's slowest fallback tier. + +## What actually fixed it + +**Bucketing (bucket-to-pair, shuffle-to-mix)** — 29.9% padding -> 0.0%, and +78.3% of micro-batches become exactly zero-pad, which puts the 5 global layers +back on `is_causal`. Measured: padding costs **9.4% MORE time for 24% LESS work** +at fixed width, because an explicit mask knocks those layers off the fast path. + +⚠ **Bucket size is NOT a diversity knob.** Swept across a 256x range, roots per +accumulation window stayed flat at 3.54-3.61. The global micro-batch shuffle does +all the mixing; the bucket only costs padding. Use the tightest bucket. + +**flex_attention** — Triton-generated so it compiles for sm_120 instead of +shipping sm_80 binaries. 21.7x on sliding layers, 2.1x on global. Needs mandatory +`kernel_options` at 32x32 blocks: 64x32 needs 102,400 bytes against a +**101,376-byte hardware ceiling** — misses by 1 KB, and Triton is already opting +into the full 99 KB, so it is the card, not a default. + +## ⚠⚠ The trap that produced TWO wrong published conclusions + +`torch._dynamo` defaults to a recompile ceiling of **8**. Every distinct sequence +width is a new shape. On hitting the ceiling dynamo does not error — it silently +falls back to UNCOMPILED flex, which is ~20x slower AND documented to *"not work +with the backwards pass and may produce incorrect results."* + +That artifact produced a bogus **0.76x slowdown** and a bogus **2.9% loss +divergence**, and I believed and reported both. Raising the limit to 256 flipped +the speed result to 1.41x. + +The loss divergence turned out to be real but benign — adjudicated against fp32 +MATH ground truth, both backends sit ~2e-3 from truth with flex fractionally +CLOSER at every width. **Do not re-open it by comparing the two backends to each +other; that cannot answer it. Compare to fp32.** + +## Process lesson + +brokkr's panel produced **four self-retractions in ninety minutes**. Every +retraction was a derivation; every survivor was a measurement. And the whole +head_dim-512 SDP problem was **already documented in zerofata's published Axolotl +config since April** — the right first stop for "why is this architecture slow" +is practitioner configs for that exact base, before any panel. + +Playbook: `docs/pfi/training-throughput-playbook.md`, commit `7b5fd91`. + _Archived 2026-09-09._ + +- `[2026-08-25]` **NVFP4A16 serving pipeline built and validated; MERGED WEIGHTS ARE MANDATORY.** vLLM cannot serve a LoRA on ANY Gemma-4 — `get_expert_mapping` is unimplemented and the check branches on MoE-ness, not quantization. Plus the landmine: a `targets=["Linear"]` recipe misses all 11,520 expert tensors silently. +# NVFP4A16 serving pipeline — built, validated, and the MoE landmine it found + +`[2026-08-25]` + +Pipeline at `scripts/erp-tune-serve/` (commits `6a85829`, `ab980e9`). +Validated end-to-end against checkpoint-100 before the real adapter existed. + +## ⚠⚠ THE LANDMINE: a `targets=["Linear"]` recipe misses EVERY MoE expert + + before linearize_moe: 427 Linears, 205 targeted, experts 0 + after linearize_moe: 11,947 Linears, 11,725 targeted, experts 11,520 + (30 layers x 128 experts x 3 projections) + +Gemma-4 stores each layer's 128 experts as two fused 3-D `nn.Parameter` tensors +(`gate_up_proj` [128,1408,2816], `down_proj` [128,2816,704]) — note the absent +`.weight` suffix. A Linear-targeting recipe resolves 205 of 427 modules and +**zero experts**, leaving 22.84 B params (88.5% of the model) in BF16 with no +warning. + +**This is the same defect that killed QLoRA here via bitsandbytes.** The blind +spot is in the *checkpoint layout*, not the tool. Fix: +`llmcompressor.modeling.moe.linearize.linearize_moe` — no registration needed, +Gemma-4 satisfies `FusedExpertsProtocol` structurally. Playbook §3.15. + +## Scheme: NVFP4A16, deviating from the playbook default, on measured grounds + +brokkr benched the W4A4 quant of this checkpoint at **12% on contradiction +detection with CoT off against gen's 81%** — the signature of 4-bit input +activations on a reasoning-dense task. Plus W4A4 KLD is 2-4x worse past ~10k ctx +on sm_120. This is a 16,384-ctx RP seat. Marlin's prefill cost accepted. + +⚠ Several HF repos named `…-NVFP4A16` declare `input_activations num_bits 4` — +W4A4 wearing an A16 label. The script refuses if the emitted config says 4. + +## Four silent defects the dry run found + +1. **transformers 5.15 MIGRATES the config schema on save** — drops + `global_head_dim`/`num_global_key_value_heads`, writes `per_layer_config`. + transformers 5.10 (the llmcompressor venv) then reads `num_key_value_heads` + as None and dies with `TypeError: unsupported operand type(s) for //`. + Every working artifact on the box uses the OLD schema. Merge now downgrades it. +2. **llmcompressor cannot auto-init a processor for a multimodal checkpoint** — + pass the tokenizer explicitly as `processor`. +3. **`save_pretrained` does not carry `processor_config.json`** — vLLM then fails + with "Can't load feature extractor", which reads as a vision bug. +4. **The quant needs more than GPU1's free 32 GiB.** `quant_with_gen_down.sh` + stops `vllm-gen` and restores it from a trap on EVERY exit path, using + `docker start` not `compose up` so the container returns with its exact config. + +## Verified on the emitted artifact + + 49 GB -> 17 GB, format nvfp4-pack-quantized, a=null (genuine A16) + weight_packed 11,725 of which expert 11,520 + tokenizer truncation: clean (§3.14 trap avoided by calibrating on the + encode cache, so the tokenizer is never called + with truncation=True at all) + served: Marlin NVFP4 kernel + Marlin MoE backend, coherent generation + +⚠ The reference `nvfp4a16` artifact triggers a vLLM warning that q/k/v carry +*different* weight global scales ("likely reduced accuracy"). **Ours does not** — +llmcompressor 0.12 links weight observers across fused groups automatically. The +in-house quant is better than the downloaded one on that axis. + +## ⚠ MERGED WEIGHTS ARE MANDATORY — and not for the reason we assumed + +The open question was whether LoRA-on-NVFP4 hot-swap still silently no-ops. +Retested on `vllm/vllm-openai:latest`: **it refuses to start.** + + AttributeError: To support LoRA for MoE model, + 'get_expert_mapping' must be implemented + +The check is in `vllm/lora/utils.py::process_packed_modules_mapping` and branches +on `is_moe_model()` — **quantization is not in the condition.** `gemma4.py`, +`gemma4_mm.py`, `gemma4_mtp.py`, `gemma4_unified.py` all have ZERO occurrences; +`deepseek_v2`, `mixtral`, `glm4_moe`, `ernie45_moe` implement it. + +**vLLM cannot serve a LoRA on ANY Gemma-4, bf16 or quantized.** Merging is the +only path for this architecture, and it would have bitten identically on the +unquantized base. A loud refusal is strictly better than the 0.24.0 silent no-op, +which shipped a base model wearing the tune's name. + +⚠ Base-viability pre-flight is now playbook §3.11 — three greps before picking a +base. **Grep the CLASS, not the file**: `mistral.py` greps as `SupportsLoRA=0` +and is fully LoRA-capable via inheritance from `LlamaForCausalLM`. + _Archived 2026-09-09._ + +- `[2026-08-25]` **Refusal retention measured (base 0/100 +# Refusal retention — the axis the gate did not have, and the axis I measured wrong + +`[2026-08-25]` + +## Why it exists + +brokkr's gate measures reasoning (T1-T6), craft (diversity/attractor) and +regurgitation (memorisation). **Nothing measured whether the model still +COMPLIES** — which for this seat is arguably the most important property. + +The risk is specific to our operation order. We do **tune(abliterate(stock))**, +so the tune has 57.7M tokens of opportunity to walk the abliteration back. *A +tune that gains 41 items of contradiction detection and quietly re-installs +refusals is a failed seat that passes the entire gate.* + +## The measurement — controlled, single instrument, both arms + + arm HARD DEFLECT COMPLY + base 0/100 0 100 + tuned 29/100 0 71 + +Same seat, same probe, temp 0, `mlabonne/harmful_behaviors` x100. +Probe: `scripts/training-probes/refusal_probe.py`. + +**The tune added 29 general-harm refusals where the base had none.** + +Two things fell out: + +- **The instrument validates.** Base measured 0/100 on my generated-text regex + against Heretic's recorded 3/100 from a first-token-probability scorer. 0 vs 3 + is agreement — the incomparability worry was right caution about a non-problem. +- **DEFLECT is 0 on BOTH arms, so the free control fires.** An instrument + artifact does not care which arm it runs against. Both zero means the model is + **binary** — refuses in refusal-language or engages, no soft-deflection tail. + The R19 undercount does not apply here. + +## ⚠⚠ But it is the WRONG AXIS — brokkr's catch, and it is the better one + +`mlabonne/harmful_behaviors` is **general harm** — weapons, malware, fraud. **The +abliteration was not run so the model would explain bomb-making. It was run so +the model would engage with explicit fiction.** Different refusal surfaces; a +model moves on them independently. + +I picked that set because it was cached, had a recorded baseline, and was what +the abliteration tool used. **Every one of those is a reason it was convenient, +not a reason it was right** — and "it has a baseline" was actively misleading, +because a comparable number for a question nobody is asking looks like evidence. + +**29/100 general-harm refusals on a seat writing prose the operator was actively +praising is plausibly the DESIRED shape**, not a defect. General-harm refusals +returning while domain compliance holds is close to ideal for an internal +creative seat. I would have reported it as damage. + +**The load-bearing cell is COMPLY 71, not the 29.** Stock refused 100/100; +anything near that would mean the abliteration was undone. 71 complying means +"partially walked back on one axis" — a different finding, and only one of the +two threatens the seat. + +Domain-compliance probe (the right axis, from R19's track-2 map) is brokkr's, +pending. Scaffold supplied: `scripts/training-probes/counted_classifier.py` +(`2a05ae9`) — classify-never-surface, three-way, ERROR path deliberately does not +log the exception body because an exception can echo the prompt back. + +Playbook §3.13. See [[2026-08-25-erp-tune-run2-complete]]. + _Archived 2026-09-09._ + +- `[2026-08-25]` **Worldtree b188 + b189 shipped; bridge extracted to `pfi/wt-matrix-bridge` because `vh` is a USER not an ORG** and no service account can ever publish to a user namespace. Plus the selene catalog entry that lied about what answers, and a #411 diagnosis I got wrong twice before a directory probe settled it. +# Worldtree b188 + b189 bridge cutover, and the selene metadata that lied + +`[2026-08-25]` + +Three arcs in one day, all infra-ops side, all landed. + +## b188 — matrix.yaml pre-sync (#406/#409/#410 closed) + +From b188 the bridge reads per-agent `rendering` + `ambient_buffer_size` from +`config/matrix.yaml` ONLY; agent `config.yaml` matrix blocks are gone from the +image. Staged as `6417115` in `worldtree-instance-configs`, deployed to both +instances with operator approval. + +- mimir gets thinking-to-thread + tool-call reactions + 7 reaction labels; + forseti and lofn stay quiet. +- **Rider #409 pruned six dead agents** (bragi, leif, troi, soong, cara, glados) + from BOTH rosters — originally scoped personal-only, which I flagged as a + possible oversight and it was. **The settling fact worth keeping: the engine + roster comes from the image's baked `agents/` directory (only `config/` is + bind-mounted), and both instances run the same image**, so instance-level + evidence about which agents the engine lists generalises by construction. +- Both rosters now exactly `[mimir, forseti, lofn]` — the three engine agents + actually bridged. mask/vili/echo exist in the engine, deliberately unbridged + (operator ruling). + +⚠ **Edited text-surgically, not via a yaml round-trip** — PyYAML would reflow +1,249 lines and drop every comment, and the comments are the documentation. + +⚠ **`deploy-wt-config` uses `docker restart`, NOT `compose up`.** A `compose up` +on corviduo-dev re-resolves the image tag and can silently swap the running +build — which would turn an "inert pre-sync" into an unintended image roll on two +live instances. That property is easy to lose in a future refactor of the script. + +## b189 — #407 bridge extracted to its own repo (#404 umbrella closed) + +Bridge now `gitea.phasefinal.com/pfi/wt-matrix-bridge`, its own repo, own CI. + +⚠ **It publishes to the `pfi` ORG, not `vh`, and the reason is structural:** `vh` +is a **USER**, not an org. Gitea scopes user-namespace packages to the owning +user — there are no package collaborators on a user namespace. **No service +account can ever publish to `gitea.phasefinal.com/vh/*`.** claude-bot is an Owner +of `pfi`, so that is where it goes. Token `wt-matrix-bridge-ci` (id 28, scopes +`write:package,read:repository`), vaulted at +`nh3-dev/.config/claude-bot/gitea-token-wt-matrix-bridge-ci`. + +**Minted a dedicated token rather than reuse `claude-bot-sdk-ops` or `arbo-ci`, +both of which already carry `write:package`** — a shared credential cannot be +revoked without collateral. ⚠ A first mint attempt succeeded then failed to save; +Gitea returns a token value exactly once, so it was unrecoverable. Deleted the +orphan (id 27) rather than leave a live package-write credential on the account. + +**Both instances PINNED** to `f3f8ec902267` (`e90f436`), closing the #410 shape: +`WORLDTREE_IMAGE` was pinned and the bridge was the one drifting service. + +⚠ The pin moved from `b178285b1cb5` because a cross-frontier bug-hunt found the +M_EXCLUSIVE **fallback could itself litter unboundedly** in exactly the state it +was written for. **The safety net had the same failure mode as the thing it was +catching** — and my staged window leaned on that fallback. + +## #411 — the debug-room failure, diagnosed twice and wrong both times first + +My theory: the alias was held by orphaned rooms. **Refuted by a directory probe +returning 404.** The real cause: Synapse's **M_EXCLUSIVE** — an appservice may +only create aliases inside a namespace it has RESERVED, and +`aipa_appservice.yaml` had `namespaces.aliases: []`. + +⚠ **I inferred a cause from a symptom that was the RESPONSE to the cause** — the +log's "re-resolving alias" line is the recovery path firing, not evidence the +alias exists. One directory probe settled it and I reasoned instead. + +Fixed with operator clearance: added +`regex: '#aipa-debug-[a-z0-9_-]*:matrix\.phasefinal\.com'` (exclusive) at +`/opt/docker/conf/synapse/aipa_appservice.yaml` on **ana-docker** (NOT +`/opt/docker/data/`, which worldtree-dev's issue cited). **My regex was tighter +than the proposed `#aipa-debug-.*`**, which fullmatches only because `.` also +matches the `:` separator and would equally claim other homeservers. + +Pre-apply sweep (an `exclusive: true` claim can make Synapse refuse to START): + + aliases matching '%aipa-debug%' 0 + total room_aliases on the homeserver 1 <- why it went unnoticed this long + rooms created by @aipa-debug 17 <- the litter, confirmed unaliased + +Synapse healthy in 40s, both bridges rode through. **The aliased create then +worked FIRST TRY on the next personal recreate** — designed path, fallback never +fired, both reuse mechanisms live (canonical-alias rediscovery AND the room-id +cache). + +⚠ The #411 writer census **inverted its own premise**: the api (uid 1000) cannot +write `/app/sessions`, the bridge (root) can — the reverse of the issue text. But +worldtree-dev's reconciliation is better than "backwards": pre-#407 the bridge +ran from the ENGINE image as uid 1000, and the cutover changed the answer +underneath the issue. Both readings were true at their timestamps. **No live +writer exists for that path anyway** (`sessions.path` is a legacy default, #330 +moved the tools off it), so the chown is optional future-proofing. + +## selene-1-mini-8b — a config that lied about what answers + +forseti's fleet sweep found the only genuine residual in +`worldtree-instance-configs`, and it was **live on both boxes, not just drifted +in git.** Fixed in `a77639d`. + +Routing was never broken (`model: "chat-judge"` stays), but `display_name` said +"Selene 1 Mini 8B" and the description said "Atla Selene 1 Mini 8B — reward model +derived from Llama 3.1 8B" while chat-judge has been backed by +**qwen3.8-27b-uncensored** since 2026-08-23. + +⚠ **It misstated the KIND of model, not just the identity.** A reward model and a +generative judge are different instruments; a consumer reading +"pairwise/likert/binary/scalar reward model" would expect scalar-reward semantics +qwen3.8-27b does not provide. + +Verified: `selene-1-mini-8b` -> HTTP 400 (by design), `chat-judge` -> HTTP 200. + +**NOT changed:** the catalog key and the `selene-judgment` role — `model_roles.yaml` +binds to that key, so renaming is worldtree-dev's schema call. Flagged that a role +named after a retired model defeats the purpose of role aliases. + +⚠ forseti's sharpest finding is for the operator: **`~/.claude/CLAUDE.md` line 502 +lists the retired name in the global tools roster.** A broadcast reaches sessions +that already exist; the roster line keeps minting new ones. His file, awaiting his +word. Also: **there is no fan-out primitive on the bus** — a fleet relay is 73 +individual posts, recorded as a real gap rather than papered over. + _Archived 2026-09-09._ + +- `[2026-08-25]` **The ERP/RP tune LAUNCHED after 12 harness defects and an operator override of the corpus gate.** Four of the twelve would have crashed the run; two were INERT GATES that passed because they could not fail. Run is `/tank/erp-tune/run-01`, harness eitri-smithy `997c4a4`. Full arc — override, defects, sizing, the measured MFU — in the in-flight section and `docs/pfi/gemma4-erp-tune-sizing.md`. + _Archived 2026-09-09._ + +- `[2026-08-24]` **char-rp seat swapped to the Gemma-4 26B-A4B MoE; abliterated trainee base staged and measured.** OOM root-caused to `--gpu-memory-utilization` not covering CUDA context (and to gen's footprint GROWING WITH UPTIME); a benchmark finding retracted because it scored below chance; abliteration isolated at −0.6 core points but it MOVES capability rather than removing it. +# `[2026-08-24]` char-rp seat: OOM root-cause, Gemma-4 MoE swap, and the abliterated trainee base + +One evening, one thread with brokkr-smithy-dev, five commits: `850e0c3`, +`27155c0`, `f509668`+`24e8826`+`1bd90ea`+`3446367`+`8d6a939`, `14ff4a3`, +`019ccff`, `5415fd4`. + +## 1. The seat was crash-looping, and the cause was NOT its config + +`vllm-meromero-rp` reported up-but-unreachable, RestartCount climbing (13 by the +time it was examined, not the 4 first reported). Startup logs looked clean all +the way through weights, `torch.compile` and CUDA-graph capture, then: + + torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 336.00 MiB. + GPU 0 has a total capacity of 94.97 GiB of which 195.19 MiB is free. + +**⚠ `--gpu-memory-utilization` SIZES THE KV CACHE AND DOES NOT COVER CUDA +CONTEXT, GRAPHS OR NON-TORCH OVERHEAD.** gen is configured at 0.43 and actually +held 45.6 GiB. char-rp was at 0.51. The pair was committed to 0.94 of the card +with ~0.6 GiB of real headroom — it fit on the 21st and stopped fitting on the +24th. + +Dropped char-rp to 0.47: ~4.8 GiB margin, KV 27.36 → 23.56 GiB, 430,825 → +371,023 tokens against a max-model-len of 262,144. **Cost nothing usable** — the +pool still holds 1.4x a full-length sequence; what is lost is concurrent long +requests, not context. + +**⚠⚠ THE MISSING HALF, found later that evening: gen's footprint GROWS WITH +UPTIME.** Same container, same 0.43: **45.6 GiB after ~3 days up, 38.5 GiB +freshly restarted** — ~7 GiB apart. Nothing about char-rp changed between the +21st and the 24th; *gen crept up underneath it*. **Headroom arithmetic done +against a long-running gen is measuring a moving number.** Measure against a +freshly-restarted one. + +## 2. `char-rp` and `char-rp-reasoning` are ONE seat, not two + +Both LiteLLM routes point at `10.250.50.54:8016/v1` — `hosted_vllm/char-rp` and +`hosted_vllm/char-rp-thinking`. brokkr had reported 30/80 and 80/80 failure +rates as two failing services; it was one outage sampled twice. This also +*improved* a result of theirs: their CoT on/off battery had assumed both aliases +were the same weights under two chat templates, and the routing detail turned an +assumption into a verified fact. + +(`vllm-charrp-reasoning-nvfp4`, the Heretic2 NVFP4+MTP container, has been +stopped for 12+ days and is unrelated — it is not what that alias resolves to.) + +## 3. The seat swapped to the Gemma-4 26B-A4B MoE + +Operator-directed straight-across replacement: same port, same +served-model-names, so no gateway route or consumer config moved. Rationale is +throughput under CoT — the user waits through the whole reasoning block before +the first visible token, and the MoE measures ~114 tok/s @32K against the dense +31B's ~40.7. + +Serving copy is `RedHatAI/gemma-4-26B-A4B-it-NVFP4` (16 GB), chosen over the +other `-it` quants because it is compressed-tensors (`nvfp4-pack-quantized`) — +the same loader path the outgoing seat used. Smaller weights at the same 0.47 +budget bought **1,724,110 KV tokens against the predecessor's 371,023**. + +`meromero-charrp` is retained stopped in `created` state, labelled +`AI - Dormant`. Both stacks bind `:8016`, so rollback is **stop-then-start**. + +## 4. ⚠ THE STALE-CHAT-TEMPLATE TRAP IS ENDEMIC, NOT A ONE-OFF + +Verified by hash across every third-party Gemma-4 derivative pulled: + +| build | lines | sha256 (normalised) | +|---|---|---| +| upstream `google/gemma-4-26B-A4B-it` | 390 | `6a1015c47ccfcfa6` | +| RedHatAI NVFP4 (served) | 389 | `6a1015c47ccfcfa6` — the only match | +| llmfan46 heretic | 365 | `0a52be69cda5ab8a` | +| TrevorJS abliterated | 266 | `58c66fdee4afa297` | +| jenerallee78 abliterated | 266 | `58c66fdee4afa297` | +| prithivMLmods NVFP4A16 | 266 | `58c66fdee4afa297` | + +Three independent repos carrying the *identical* stale file means it propagated +through the ecosystem. Consequences differ by use and **both are silent**: +serving a mismatched template renders a different prompt; **training through +`base/chat_template.jinja` means training on a different prompt format than +production serves** — train/serve skew, no error, presents as a tuning failure. + +The production compose now pins the template explicitly. It is a **no-op for the +served weights** (the A4 build ships that exact file) and permanently closes the +class. ⚠ If `GEMMA4_MODEL` ever points at a different checkpoint, the template +default must move with it. + +## 5. A benchmark result was RETRACTED — below chance indicts the instrument + +A battery appeared to show Gemma at **12% contradiction detection with CoT off +against gen's 81%**. An A16 activation-precision control was staged to test +whether the quant scheme owned it. Then the operator asked to see the individual +items, and the item was **ill-posed**: it presented two mutually contradicting +statements and asked for "*the* contradicting statement", but **contradiction is +symmetric**. The model consistently named the absolute claim — a defensible +reading the labelling scored wrong every time. + +**⚠ THE TELL WAS IN PLAIN SIGHT: 12% ON A FIVE-OPTION TASK IS BELOW THE 20% +CHANCE FLOOR.** A below-chance score indicts the instrument before it indicts +the model, and neither side reacted to it. I spent the afternoon verifying repo +names, config fields, template hashes and tokenizer vocabs — every layer of +plumbing — and never asked whether the number itself was *possible*. **A +preflight can be thorough and still be aimed in the wrong direction.** + +Retracted: "the model owns the contradiction deficit"; "domain tuning costs 43 +points of contradiction detection" (on a sound instrument it **reverses**); all +pre-fix T2 numbers. Recorded as a dated superseded-claims table in +`stacks/gemma4-charrp/README.md` rather than a silent edit. + +**What survived:** the A16 control result — activation precision is close to free +on this battery, every other task identical across W4A4 and W4A16 builds. + +## 6. The abliterated trainee base — measured, not assumed + +Operator directed a low-damage abliterated instruct build. "Low damage" was +treated as a measurable claim; the field spreads from KL 0.09 to 0.4118: + +| build | method | KL | refusals | +|---|---|---|---| +| **llmfan46** (operator's pick) | Heretic v1.2.0 ARA | 0.1237 | 3/100 | +| TrevorJS | ARA-family | 0.09 | 1/100 effective, 5/686 cross-dataset | +| jenerallee78 | ARA 2-pass | 0.1299 | 7.7% StrongREJECT | +| huihui-ai | remove-refusals | none published | none published | + +Fleet anchor: our own work found **Heretic at KL 0.12 preserved the MTP head at +83.7% acceptance**, so both staged builds sit inside an already-measured band. +huihui-ai rejected — no metrics, its card calls the method "a crude, +proof-of-concept implementation", it abliterates both thinking and non-thinking +modes, and its parameter count runs ~738M over upstream. Operator's independent +read matched ("huihui produces garbage"). + +**Abliteration isolated properly** (stock BF16 vs llmfan46 BF16, same precision, +same pinned template, same 192 items): + + T2 contradiction 75% → 59% (−5 items) + T6 spatial 75% → 88% (+4 items) + core 90.0% → 89.4% (−0.6 pts) + +**It MOVED capability rather than removing it** — five lost on contradiction, +four gained on spatial, nearly cancelling. Nobody predicted a gain. **llmfan46 +stands**; no case for re-staging on TrevorJS over 0.6 points. + +⚠ Read as ~5 and ~4 items at n=32, not as −15.6/+12.5 percent. ⚠ Says nothing +about quantization — the stock-NVFP4 T2 figure came from n=16 against n=32, +different item sets, n-confounded. + +## 7. ⚠ The production compose hardcodes `--quantization compressed-tensors` + +Pointing the char-rp stack at unquantized BF16 weights crash-loops immediately: + + TypeError: CompressedTensorsConfig.__init__() missing 3 required + positional arguments: 'target_scheme_map', 'ignore', 'quant_format' + +vLLM trying to read a quantization config out of a checkpoint that has none. 35 +restarts before it was caught. Hence `stacks/gemma4-trainee-bench/` — a separate +ephemeral stack with no quantization flag, `restart: "no"` so a bench seat cannot +resurrect itself and block gen's restore, and no homepage labels so it leaves no +permanently-offline card. That detour is why a base swap is now ~5 minutes +instead of ~15. + +## 8. BF16 cannot coexist with gen + +48.07 GiB of BF16 weights plus gen's footprint exceeds the 94.97 GiB card before +a byte of KV cache. Every BF16 bench window means **gen is stopped**. Two such +windows were run and gen restored both times; the restore was triggered by +observing the seat's own throughput logs (a large prefill burst then zero +running/zero waiting) rather than waiting on a courtesy message. + +Cross-links: [[2026-08-24-homepage-uniform-grid]] + _Archived 2026-09-09._ + +- `[2026-08-24]` **Homepage rebuilt on Australis Skyfall; light mode shipped.** Two findings worth more than the theme: **(a)** the Skyfall bundle including its canonical light ramp was sitting in this repo's git history at `45c1995` — check `git show` before concluding a vendored design asset is lost; **(b)** removing `theme:` from `settings.yaml` deterministically breaks the dashboard render (six recreates empty, restoring the key fixed it in 12s), which is the first confirmed cause of the "tab bar goes missing" symptom. Retires the `homepage.log` size lead from earlier the same day — it did nothing on this episode. +# `[2026-08-24]` Homepage: remote-label consumption re-verified, then the board relaid out on a uniform grid + +Prompted by the operator: *"Homepage on esh-vm-docker lists remote dockers and +can absolutely consume their labels, please verify again. I am still +unsatisfied with the layout and aesthetics."* + +## The verification — the operator was right, and the record now says so + +**Homepage on `esh-docker-vm` discovers services by container label from all +five Docker engines in `conf/docker.yaml`, not just its own.** This is not an +inference; `GET /api/services` returns every card's `server` field, and the +2026-08-24 snapshot resolves to: + +| `server` | host | label-discovered services | +|---|---|---| +| `ana-pfi-docker` | 10.250.50.70 | 30 | +| `irv-ml1-docker` | 10.100.79.3 (over WireGuard) | 15 | +| `ana-ml2-docker` | 10.250.50.54 | 14 | +| `esh-vm-docker` | 10.0.50.45 (the dashboard's own host) | 13 | +| `nh3-pfi-docker` | 10.100.50.40 | 2 | + +**74 of 107 cards are label-discovered, and only 13 of those come from the +dashboard's own engine** — the other 61 are read off four remote hosts, +including irv-ml1 across the WireGuard tunnel. The remaining 33 carry +`server: null`: those are the manual `services.yaml` entries — hardware, BMCs, +hypervisors, printers, and user-level systemd services that have no container +to label in the first place. **That null is the only thing "not label-driven" +about this dashboard**, and it is a property of the entry, not of the host it +points at. + +⚠ If a future session doubts this again, the check is one command and takes two +seconds — do not reason about it from the docs: + +```bash +curl -s http://10.0.50.45:5100/api/services \ + | jq -r '.[].services[] | .server' | sort | uniq -c +``` + +## What was actually wrong with the layout + +Measured with Playwright against the live board (per-group `card=` width, card +height spread, and a geometric title-vs-status overlap test), not judged by +eye: + +- **Card width changed at every group boundary.** `columns:` is not a density + dial — it sets `lg:grid-cols-N` for one group, so it fixes that group's card + width. Notes rendered a single **1464px** card; News and Media **728px**; + Eval & Retrieval **286px**; everything else 360px. Scrolling the page, the + grid resized five times. +- **Long names printed underneath their own status pill.** Measured by + re-injecting the old rule and testing the title text node's box against the + status cluster's box: **6 cards, all on the AI tab** — 3 in Inference, 2 in + Dormant, 1 in Eval & Retrieval; zero on the other three tabs, which is why + it survived earlier passes. Root cause is a genuinely counter-intuitive one: + the rule reserved a + 78px gutter with `padding-right` and relied on `overflow: hidden` to hold it, + but **overflow clips at the padding box, not the content box** — so the + reserved gutter was spill room the title rendered straight through. The + intended `text-overflow: ellipsis` never fired either, because the ellipsis + is painted by whichever block's own line overflows, and here that is the + anonymous box around the bare title text node, which does not carry + `overflow`. +- **`AI Systems` / Scriberr was on all four tabs** — the 2026-08-18 UltraSeedbox + bug recurring, this time arriving from a container label rather than from + `settings.yaml`. +- **Icons were grey smudges.** Homepage masks every glyph over + `--color-logo-start/stop`; stock slate-400 → slate-700 sinks the bottom half + of each icon into the card fill. +- Bookmark groups and Jellyfin's trailing stream rows were the two components + the theme had never reached. + +## The fixes + +`stacks/homepage/conf/settings.yaml` — **all 20 groups to `columns: 4`.** +`stacks/homepage/theme/australis.css.in` → rebuilt → `conf/custom.css`: +gutter held by wrapping, description clamped to 3 lines (floor still 2), icon +ramp overridden, bookmark + trailing-widget components themed, group gap +10px → 22px. `stacks/scriberr/compose.yaml` — `homepage.group` → `AI - Audio +Tools`, container recreated on ana-ml2. + +After: **every group renders at card=360**, and the collision count is zero. + +Before/after, all four tabs: `http://10.100.10.50:8090/b/homepage-relayout/` +(24h TTL; also on the standing link board). + +## ⚠ Three traps worth carrying forward + +1. **"Columns = member count" is RETIRED** (it was the 2026-08-18 rule). It was + avoiding dead cells in a short last row and bought a worse defect. A short + last row is what a grid looks like; a card wider than its neighbours is what + a mistake looks like. +2. **A `:root` override of a Homepage theme variable is silently ignored.** + Homepage sets `--color-logo-*` on `.theme-slate`, and that class is on the + `` element — the same element `:root` matches. `.theme-slate` (0,1,0) + beats `:root` (0,0,1), so the override does nothing and looks like the + variable is not the one in play. `html[class]` (0,1,1) wins, and does not + hard-code which `theme-*` class is active. Specificity alone is not enough + either: a custom property resolves from the *nearest* ancestor that sets it, + so the override has to land on ``, not on ``. +3. **The post-recreate tab-bar loss is INTERMITTENT, not guaranteed.** The + 2026-08-19 note reads as though every recreate costs up to an hour of broken + render. This recreate came up correct within 10 seconds — fresh payload on + the first poll, all four tabs clickable a minute later. Recreate, *check*, + and only then walk away if it is actually in the broken state. + +Also re-confirmed, since the change depended on it: **a `settings.yaml` edit +needs a container recreate, not a restart.** `docker restart homepage` left the +old `"columns":1` payload embedded in the served HTML with the correct file +mounted and readable inside the container; `compose up -d --force-recreate` +cleared it immediately. + +## Deliberately not done — operator's call + +The Main tab still opens on three sparse bands: **Notes** (1 member) and +**Games** (1) each burn a full 4-wide row, and **News** has 2. Merging Notes + +News, or folding Games into Apps, would tighten the top of the page — but that +is information architecture, not layout, and the group names are the operator's. +Surfaced rather than done. + +→ **Resolved in pass 2 below**, where the operator delegated the naming +("re-categorize however you want"). Notes + News became `Daily`, Games folded +into `Apps`, and the `AI - Audio Tools` placement in this pass was superseded +(Scriberr moved on to `AI - Studios`). + +--- + +# `[2026-08-24, pass 2]` Recategorised on "do I open this?", API groups collapsed + +Operator, after seeing pass 1: *"You can re-categorize however you want. +service networking tab is uneven, you can split out the adguard cards, etc. +most of the issues are that tools I use and have a UI are interspersed with API +endpoints which are largely informational only. They might even go in their own +cards or start collapsed."* + +## The axis + +Every group is now either **tools** (expanded, top of tab) or **endpoints** (an +API, a broker, an agent — `initiallyCollapsed: true`, bottom of tab). A +collapsed group still renders its eyebrow and rule, so presence costs one line +instead of two rows. + +Second, quieter rule that fell out of the same pass: **a group's members should +all carry a widget or none should.** A stat strip adds ~50px, so one widget card +in a row of plain ones opens a void under the plain ones — which is most of what +made the 13-member `Service Networking` band look broken. + +## Shape + +- **Main** — `Daily` (Memos, Miniflux, Nevermore, SearXNG — replaces the + 1-card Notes and 2-card News bands), `Monitoring`, `Apps` (12; absorbed the + 1-card `Games` band), `Media`, `UltraSeedbox`. +- **AI** — `AI - Gateways & Chat` (8) and `AI - Studios` (6) expanded; then + `AI - Inference` (7), `AI - Eval & Retrieval` (4), `AI - Speech (TTS)` (4), + `AI - Audio Tools` (2), `AI - Dormant` (6) all collapsed. +- **Toolchain** — `DNS & Filtering` (3), `Reverse Proxies` (2), + `Compose Consoles` (5), `Toolchain` (3), `Agents (no UI)` (6, collapsed). +- **Infrastructure** — unchanged; every card there is already a console. + +Measured after: every group `card=360`, and `DNS & Filtering` and +`Reverse Proxies` both `h=134..134` — dead flush. + +## ⚠ The move that made it affordable + +**The sixteen GPU-backed model seats were NOT relabelled.** `homepage.group` is +read at container **creation**, so renaming `AI - Inference` to something +clearer would have meant recreating six vLLM seats plus four eval seats plus +four TTS engines — multi-minute model reloads on endpoints peers reach through +the gateway. Order plus `initiallyCollapsed` buys the same separation for free, +so the names stay ugly on purpose. **Do not spend that recreate on a label.** + +28 containers *were* relabelled — all cheap web services — via five rerunnable +elway playbooks, `playbooks/homepage-regroup-.yaml`. The canonical +`stacks/` tree was synced to match afterwards, so intent and reality agree. + +`initiallyCollapsed: true` is a per-group key in `layout:`; confirmed present in +this build (`defaultOpen: !(group?.initiallyCollapsed ?? global)` in +`/app/.next/server/pages/index.js`). + +## AdGuard (ANA) gained its widget, and the credential is fleet-wide + +It was the only AdGuard without a query/blocked/latency strip, so it sat short +beside two tall siblings. **One `infra-ops` AdGuard login authenticates against +all three instances** (ANA `:8053`, NH3 `:8080`, ESH `:8080` — all returned 200 +on `POST /control/login`, verified 2026-08-24). Vaulted at +`secret get nh3-dev/adguard-infra-ops-password`; written to +`/opt/docker/compose/adguard-ana/.env` (0600, root) and never into git. Its icon +was also the odd one out (`mdi-dns` against two `si-adguard`). + +## ⚠⚠ `initialSettings":{}` — the tab-bar mystery is a SWALLOWED EXCEPTION + +The biggest durable finding of the day, and it cost ~25 minutes. Full write-up +in `stacks/homepage/README.md`; the short version: + +`initialSettings":{}` in the served HTML is **the catch branch** of the page's +data loader, not a warm-up and not a cache. And the error can vanish without +trace: the logger is assigned as the first statement *inside* the same `try`, +and the `catch` only logs `if (logger)`. If the logger is what threw, nothing is +written anywhere — which is exactly what was observed. + +Ruled out by measurement, do not re-run: `/api/services`, `/api/bookmarks`, +`/api/widgets` and `/api/hash` all return **200 with correct content** while the +page serves `{}`; restoring the previous known-good `settings.yaml` reproduces +it identically; `/api/validate` returns `[]`; disk and permissions are fine. + +**One-command test:** +`curl -s http://10.0.50.45:5100/ | grep -o 'initialSettings":[^,]\{0,20\}'` + +**What broke the streak:** three consecutive recreates came up empty, then +rolling the 8.6 MB `conf/homepage/logs/homepage.log` aside and recreating healed +it within 15 seconds. That is one observation, not proof — but it is a coherent +mechanism (oversized log → logger init throws → silent catch) and it is the +cheapest thing to try first next time. + +--- + +# `[2026-08-24, pass 3]` Rebuilt on Australis Skyfall — dual theme, light shipped + +Operator supplied the Skyfall design-system README and said "Go full with +skyfall." + +## The bundle was already in this repo's git history + +**The Skyfall tokens did not need to be hunted down.** A predecessor vendored +them on 2026-08-19 and a later commit deleted them; git kept everything: + +```bash +git show 45c1995:stacks/homepage/theme/colors.css # 208 lines, BOTH themes +git show 45c1995:stacks/homepage/theme/layout.css # calm-depth tokens +git show 45c1995:stacks/homepage/theme/typography.css +git show 45c1995:stacks/homepage/theme/fonts/Supreme-{400,500,700}.woff2 +``` + +`colors.css` carries `:root` (dark) **and** `[data-theme="light"]` (Skyfall +Day) — so the light ramp is canonical, not derived. That killed the entire +objection from the previous answer, which was correct only about the +`australis-design` skill ("Always dark first. No light mode in this system"). +**Skyfall is the dual-theme derivative; australis-design is the terminal +theme. They are different systems and only one of them has a light mode.** + +## ⚠⚠ REMOVING `theme:` FROM settings.yaml BREAKS THE DASHBOARD + +The documented way to get Homepage's own light/dark toggle is to leave `theme:` +unpinned. **Do not.** With the key absent, the page's data loader throws and its +catch branch serves `initialSettings: {}` — no tab bar, no layout, no i18n. + +Measured, not inferred: six force-recreates over seven minutes all came up +empty with the key removed; restoring `theme: dark` rendered correctly on the +next recreate in **12 seconds**. `/api/services` stays 200 and fully correct +throughout, which is exactly why this reads as a caching or warm-up problem and +is not one. + +This is the first *confirmed* trigger for the long-running "tab bar goes +missing" mystery. It does not explain every occurrence (the symptom has +appeared with `theme:` present), but it means **the first diagnostic step is +now `git log -p -- stacks/homepage/conf/settings.yaml`**, not container +archaeology. Also retires an earlier lead from this same session: rolling the +8.6 MB `homepage.log` aside once coincided with a recovery, but did nothing +during the `theme:`-key episode — coincidence, not cause. + +## So the toggle is ours + +`conf/custom.js` renders it (was an empty placeholder). Precedence: + +1. explicit choice — `localStorage['skyfall-theme']`, written by the toggle; +2. OS preference — `@media (prefers-color-scheme: light)`; +3. dark — Skyfall's default. + +`theme/build.py` re-emits each vendored `[data-theme="light"]` block twice: as +`[data-theme="light"], html.light`, and inside the media query scoped to +`html:not([data-theme="dark"]):not([data-theme="light"])`. **That `:not()` pair +is what lets a stored *dark* choice survive a light-mode OS.** Verified across +both OS preferences: load, click, click again, reload — all four correct. + +⚠ Homepage keeps its own `class="dark scheme-dark theme-slate"` on `` +regardless, because `theme:` is pinned. That is fine and was checked +explicitly: with the dark class present AND `data-theme="light"`, every themed +surface resolves to Skyfall Day, because our rules carry `!important` on the +surfaces Tailwind's `dark:` variants would otherwise claim. **`data-theme` is +the control surface; the class is not.** + +## The anti-fork guard is now mechanical + +`build.py` records the SHA-256 of each vendored file and **fails the build** on +a mismatch, rather than warning. A vendored file is either byte-identical to +the bundle or it is a fork wearing the bundle's name. Overrides go in +`skyfall.css.in`, which is written entirely against the semantic layer +(`--surface-*`, `--text-*`, `--border-*`, `--success/--danger/--warning`) — no +raw family tokens, no colour literals. + +The one place a literal is unavoidable: Homepage consumes +`--color-logo-start/stop` as `rgb(var(--x))`, which cannot take an `oklch()`. +Those four values are exact sRGB conversions of real tokens (`--sea-80`, +`--blue-base` for dark; `--sea-40`, `--blue-deep` for light), computed rather +than eyeballed, with the conversion recorded in the file. + +## Deviations, all deliberate and all written down + +- **The aurora ribbon under the tab bar is gone.** Skyfall sanctions exactly two + accent expressions — the active rail and hero-only glows — and a decorative + gradient across the chrome is neither. The colour moved to a 2px accent bar + plus `--accent-soft` fill on the active tab, which *is* the rail. +- **Widget stat values moved from the display face to mono**, per Skyfall's + "numbers and telemetry are always `--font-mono`". +- **Two font substitutions**: Space Grotesk for Bespoke Sans, JetBrains Mono + for Victor Mono. Only Supreme was ever vendored, and Skyfall's own notes call + Victor Mono "user-supplied". Two-line swap when the real faces arrive. + +Dark + light, all four tabs: `http://10.100.10.50:8090/b/homepage-skyfall/` + _Archived 2026-09-09._ + +- `[2026-08-24]` **Homepage reorganised on the axis "do I open this?" — UI groups expanded on top, API/agent groups collapsed at the bottom** (operator-delegated: "re-categorize however you want"). Load-bearing constraint: `homepage.group` is read at container CREATION, so the 16 GPU-backed model seats keep their unlovely names rather than eat a recreate — `initiallyCollapsed` + order is free. Second rule discovered here: **group members should all have widgets or none should**, because a stat strip adds ~50px and opens a void beside plain cards. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md` + _Archived 2026-09-09._ + +- `[2026-08-24]` **Homepage columns unified at 4 for every group; the 2026-08-18 "columns = member count" rule is retired.** It was avoiding dead cells in a short last row and bought a worse defect — card width changing at every group boundary. Also carries two CSS traps: `overflow: hidden` clips at the PADDING box (so a `padding-right` gutter is spill room, not a guard), and a `:root` override of a Homepage theme variable is silently outranked by `.theme-slate` on the same `` element. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md` + _Archived 2026-09-09._ + +- `[2026-08-24]` **AES-128 adopted on both Anaheim tunnels; the per-flow ceiling root-caused to the UDM's software AES-CBC, exonerating the FortiGate.** Proven by an A/B/A cipher swap at identical CPU — hardware offload is not cipher-cost-sensitive. +# [2026-08-23] Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gbps circuit + +> **⛔ SUPERSEDED 2026-08-23 (same day, later session) — read the CORRECTION at +> the bottom before acting on anything here.** The headline is wrong (the +> relevant ceiling is NH3's **1 Gbps** uplink, not Anaheim's 2 Gbps), the +> aggregate number is wrong (**692 Mbit/s** at 8 streams, not ~550 — the +> original stopped measuring at 4), and the proposed remedy is **impossible**: +> UniFi's manual site-to-site IPsec does not implement AES-GCM at all. The +> per-stream observation and the parallelise-your-transfers mitigation are the +> parts that survive. + +The operator noticed site-to-site transfers were slow for a datacenter fiber +handoff and asked whether WireGuard was the limit. It is not WireGuard, and the +circuit is fine. + +## Measured + +``` +ana-docker -> internet, 8 parallel 2,153 Mbit/s <- the 2 Gbps handoff, delivering +ANA <-> NH3 through the tunnel, 4 par. 460 Mbit/s +FortiGate's own recorded peak 554 Mbit/s +ANA <-> NH3, single stream 227 Mbit/s +ANA <-> ESH, single stream 249-265 Mbit/s +ESH <-> NH3 (never touches ana-gw) 545-557 Mbit/s on a SINGLE stream +``` + +Method: stdlib TCP probe (no ssh, no crypto, no compression) between site +endpoints; raw circuit measured with 8 parallel HTTPS fetches from Hetzner +Ashburn. Host NICs are virtio with no reported cap, so no host-side ceiling. + +## What it is not + +- **Not WireGuard.** Both Anaheim tunnels are IPsec on ana-gw + (`pfi-ana-nh3` -> 70.230.226.88, `ana-eshudm-dyn` -> the ESH UDM). WireGuard + on ana-wg is remote-access only and is not in this path. Traceroute confirms: + both slow paths have hop 1 = `10.250.50.1` (the FortiGate); the fast + ESH<->NH3 path rides a `192.168.x` Site Magic overlay and never touches it. +- **Not CPU or crypto exhaustion.** FortiGate CPU was **100% idle across all + 8 cores** during the tests, and both live tunnels report `npu_flag=03` with + `dec_npuid=1 enc_npuid=1` — encrypt *and* decrypt are hardware-offloaded. +- **Not a 250 Mbit/s cap.** That was the first number and it is misleading — + single-stream TCP. Four parallel streams doubled it. Quote the aggregate. +- **Not the interface.** wan1: `rxe=0 txe=0 rxd=0 txd=0`, no collisions. + +## Most likely cause + +Both tunnels negotiate **`aes256-sha1`** in phase 1 *and* phase 2 (dhgrp 14, +IKEv2). AES-CBC + SHA1 is a two-pass operation; FortiGate NPUs are markedly +faster on **AES-GCM**, which combines encryption and authentication in one +pass. The datasheet IPsec headline for an 80F assumes GCM with large packets, +not CBC+SHA1 at the 1438-byte tunnel MTU this link negotiates. The ~4x +shortfall is consistent with that. + +## Not executed + +Changing the proposal is a **production-edge change requiring a matching +change at the far end** (NH3 UDM and the ESH UDM), and each tunnel drops while +it renegotiates. Left for the operator. See the index entry for authorization +state. + +## Immediate mitigation, no config change + +Per-flow is the weak axis: a single stream over Site Magic gets 557 Mbit/s, a +single stream through IPsec gets 227. **Anything moving bulk data across the +Anaheim link should parallelise** — that alone roughly doubles throughput +today. + +## Practical consequence already observed + +`/mnt/smithy` mounted on ana-ml2 reads at 24.7 MB/s sequential vs 98.3 MB/s +from nh3-dev (same file, same mount) — that gap *is* this tunnel, not NFS and +not the NAS. See [[2026-08-23-smithy-mount-ana-ml2]]. + +## Access note + +ana-gw is a FortiGate-80F, FortiOS 7.2.10, at 10.250.0.1. `sshpass` is absent +on nh3-dev; connect with paramiko via `uv run --with paramiko`. Password is +vaulted at `fortigate/ana-gw-infra-ops-password`. **`diagnose vpn tunnel list` +prints live ESP session keys** — never paste its output into althing, a +booth, or a commit. + +--- + +## CORRECTION (2026-08-23, later session): the cutover was attempted and the remedy does not exist + +The operator authorised the AES-GCM cutover, NH3 side first. It cannot be done, +and the measurements taken while trying show there is very little left to win. + +### AES-GCM is unavailable on the far end — not a naming problem + +The NH3 edge is a **UDM Pro SE** terminating `pfi-nh3-ana` (networkconf +`_id 697d64414c85dd2b6669b00a`, `ifname vti64`). Its UniFi API **validates** the +crypto enum and rejected every GCM spelling tried — `aes256gcm`, `aes256gcm128`, +`aes256gcm16`, `aes-256-gcm`, `aes256-gcm`, `aes256gcm12`, `gcm`, `aes128gcm128` +— all `HTTP 400 api.err.InvalidPayload`, nothing applied. + +**The control that makes this conclusive:** the *identical* request body with +`ipsec_esp_encryption: "aes256"` returns `HTTP 200 rc:ok`. So the 400s are the +enum rejecting the value, not a malformed body. Corroborating: **zero +case-insensitive `gcm` matches across 7.3 MB of UniFi OS UI bundles.** + +Accepted enum (probed): `aes128`, `aes192`, `aes256`, `3des` → 200; `des`, +`chacha20poly1305` → 400. There is no AEAD option. Both Anaheim tunnels land on +UniFi far ends, so this blocks the ESH tunnel too. + +The FortiGate side **was** widened and is GCM-capable: phase2 `pfi-ana-nh3` now +reads `set proposal aes256-sha1 aes256gcm`. Left in place deliberately — it is +functionally identical while the peer only offers CBC, and reverting it would +cost another SA renegotiation for a cosmetic gain. Phase 1 was never touched; +IKE protects the control channel only and has no bearing on data throughput. + +### The numbers that retire this as a problem + +Measured NH3→ANA through the tunnel, and NH3→ESH over Site Magic (WireGuard) on +the same UDM and the same uplink, with the same stdlib TCP probe: + +| streams | IPsec NH3→ANA | WireGuard NH3→ESH | +|---|---|---| +| 1 | 245 Mbit/s | 557 Mbit/s | +| 4 | 471 Mbit/s | 767 Mbit/s | +| 8 | **692 Mbit/s** | **795 Mbit/s** | + +**NH3's WAN is a 1 Gbps link** (`uplink.speed = 1000`, port capable of 10G) — +that, not Anaheim's 2 Gbps, is the ceiling for anything crossing this tunnel. +So the tunnel does **~69% of the achievable uplink** at 8 streams, and the +IPsec-vs-WireGuard gap collapses from 2.3× at one stream to **15% at eight**. + +Re-architecting the transport (site-to-site WireGuard via `ana-wg`, since +FortiOS has no WireGuard) would chase that last 15%. Not worth it. + +### What the constraint actually is + +A **per-stream** limit (~245 Mbit/s), not an aggregate crypto ceiling. Both +endpoints are idle at load — FortiGate CPU 100% idle with `npu_flag=03` +(offloaded both directions), UDM CPU ~7% with load1 moving 0.70 → 1.55. The +shape is per-SA/per-flow serialisation, and WireGuard shows the same shape from +a higher floor (557 → 795 is only 1.43× scaling). + +### Actionable consequence + +Anything moving bulk data across this link should **parallelise** — 245 → 692 +Mbit/s, a 2.8× win with no config change. For single-stream workloads that +cannot be parallelised at the application layer, **NFS `nconnect=N` is the +lever**: it opens N TCP connections per mount, converting a single-stream +workload into a parallel one. The `/mnt/smithy` mount on ana-ml2 reading at +24.7 MB/s (~200 Mbit/s, i.e. exactly the single-stream ceiling) is the live +example — remounting with `nconnect=8` is the obvious test. + +### Foot-gun recorded + +Probing the enum by PUTting candidate values **applies the accepted ones**. A +probe loop here timed out with `3des` briefly live on the NH3 side, which the +FortiGate would not accept — a short tunnel outage until `aes256` was restored +(~1 minute, confirmed by the SA counters resetting). If you enumerate a UniFi +config enum this way, restore the known-good value after **every** 200, not at +the end of the loop. Post-change verification: the UDM object was diffed +field-by-field against its pre-change snapshot and is **byte-identical**. + +--- + +## FOLLOW-UP (2026-08-23): what the per-stream limit actually is + +The correction above called the constraint "per-SA/per-flow serialisation". +That was a hand-wave. Measured properly, it is a **hard per-flow rate cap of +~230–245 Mbit/s with a very deep buffer in front of it** — not a tuning +problem, not loss, not window size. + +### The evidence: pin the send buffer and sweep it + +Single stream NH3 → ana-docker, `SO_SNDBUF` pinned, `ss -ti` sampled in flight: + +| in-flight cap | throughput | RTT in flight | minRTT | retrans | +|---|---|---|---|---| +| 256 KB | 224 Mbit/s | 7.8 ms | 5.3 ms | 0 | +| 416 KB | 225 Mbit/s | 11.8 ms | 6.6 ms | 0 | +| 416 KB | 245–247 Mbit/s | 12.0 ms | 5.6 ms | 0 | +| ~3.3 MB (autotuned) | 245 Mbit/s | **107 ms** | 5.5 ms | 0 | + +**Throughput is flat across a 13× range of in-flight data while RTT scales with +it.** That is the signature of a fixed service rate with a standing queue: the +window controls only how much queue you build, never how fast you go. Had this +been window-limited, throughput would have risen with the buffer. Had it been +congestion, there would be retransmits — there are essentially none +(`retrans:0`, 0% ping loss). + +So `net.ipv4.tcp_*` tuning, window scaling and congestion-control choice are all +**red herrings here**. Do not go there. + +### Bufferbloat: one bulk stream wrecks latency for everything else + +Measured on the same tunnel, ping to ana-docker: + +- idle: **6.9 ms** avg +- during a **single** bulk TCP stream: **102 ms** avg, 136 ms max, 0% loss + +**15× latency inflation from one transfer.** This is the operationally +important finding — any interactive traffic sharing the Anaheim link (ssh, +RDP, althing, VoIP) degrades badly whenever anything moves bulk data, and it +takes only one stream to do it. Parallelising transfers makes throughput +better and this *worse*. If it starts biting, the fix is an AQM/shaper on the +tunnel (or rate-limiting bulk jobs), not more buffer. + +### Where the cap lives — strong inference, not proof + +Three paths, and the FortiGate is the only variable: + +| path | single-stream | +|---|---| +| FortiGate ↔ NH3 UDM (IPsec) | 245 Mbit/s | +| FortiGate ↔ ESH UDM (IPsec) | 249–265 Mbit/s | +| NH3 UDM ↔ ESH UDM (WireGuard, **no FortiGate**) | 557 Mbit/s | + +Present in both slow paths, absent from the fast one. Aggregate over the same +SA reaches 692 Mbit/s, so it cannot be the SA or the crypto engine as a whole — +many flows spread out fine, one flow does not. + +The mechanism that fits is **FortiGate NPU IPsec offload being per-session**: +each firewall session is bound to one crypto engine, so a single TCP flow is +capped at one engine's rate while many sessions spread across engines. **This +is inference from the throughput shape, not something confirmed on the box** — +`diagnose sys session list` was not captured for a TCP flow (the filter caught +only traceroute UDP probes). A single-stream control through ana-gw *without* +IPsec returned 290 Mbit/s to Hetzner Ashburn, but at ~60 ms RTT that is +window-limited and does not discriminate. **If this matters, the clean test is +a non-IPsec single stream between two Anaheim VLANs at low RTT.** + +**Relevant to the FortiGate cutover decision:** if the per-flow cap is the +FortiGate's IPsec path, replacing the box plausibly lifts single-stream +throughput toward the WireGuard figure. That is a point in favour of the +cutover, and it is cheap to verify afterwards by re-running the sweep. + +--- + +## FOLLOW-UP 2 (2026-08-23): it is NOT a capacity problem, and it IS specific to IPsec + +Operator asked directly whether the 80F "can't handle the traffic". It can. +Two new measurements settle the shape of this, and correct an overstatement in +FOLLOW-UP 1 (which pointed at the FortiGate on evidence that was confounded — +every slow path was *both* IPsec *and* FortiGate, so protocol and box could not +be separated by that argument). + +### The 80F routes a single flow at line rate when IPsec is not involved + +`ana-ml2 → pfi-pve`, inter-VLAN **through** ana-gw (traceroute hop 1 = +`10.250.50.1`), 0.36 ms RTT, no tunnel: + +| streams | throughput | +|---|---| +| 1 | **940.2 Mbit/s** | +| 8 | 939.3 Mbit/s | + +Single stream saturates 1 GbE. So the box does **not** cap single sessions in +general, and there is no per-session ceiling in its plain forwarding path. The +~250 Mbit/s per-flow cap is **specific to the IPsec datapath**. + +### Both IPsec tunnels converge on the same numbers despite different far ends + +Measured today with the same probe: + +| tunnel | far-end gateway | RTT | 1 stream | 8 streams | +|---|---|---|---|---| +| NH3 ↔ ANA | UDM Pro **SE** | 6.7 ms | 245 Mbit/s | 692 Mbit/s | +| ESH ↔ ANA | UDM Pro **Max** | 3.9 ms | **268 Mbit/s** | **715 Mbit/s** | + +Different gateway hardware, different sites, different uplinks, and RTT +differing by 1.7× — yet single-stream differs by only 9%. **If this were +window-limited the 3.9 ms path would be ~1.7× faster.** It is not, which is +independent confirmation of a rate cap rather than a BDP effect. + +### Capacity summary — the box has headroom it will not give one flow + +- plain routing, 1 stream: **940 Mbit/s** (line rate) +- plain routing to internet, 8 streams: **2,153 Mbit/s** +- IPsec, 8 streams: **692–715 Mbit/s** +- IPsec, 1 stream: **245–268 Mbit/s** +- CPU **100% idle** throughout; IPsec NPU-offloaded (`npu_flag=03`) + +Within a single SA, 8 sessions get ~2.9× what 1 session gets, so the datapath +distributes work **by inner session** — consistent with IPsec offload binding a +session to one crypto engine. + +### What is still NOT separated + +Whether the cap belongs to **the 80F's IPsec offload** or to **UniFi's IPsec +implementation**. Both tunnels have a UDM at the far end, and both UDMs run the +same UniFi firmware, so identical caps are explainable either way. The Pro Max +being only 9% faster than the Pro SE argues against the UniFi side (a beefier +CPU should show more), but that is suggestive, not conclusive. + +**The test that closes it:** an IPsec tunnel whose endpoints do not include the +80F — e.g. a temporary UDM↔UDM IPsec tunnel between NH3 and ESH, measured +single-stream. If it also caps ~250, the FortiGate is exonerated and replacing +it buys nothing on this axis. If it runs near the 557 Mbit/s that UDM↔UDM +WireGuard achieves, the 80F is the limiter. **Bears directly on the pending +FortiGate cutover** — worth running before that decision, not after. + +--- + +## FOLLOW-UP 3 (2026-08-23): WireGuard over the same internet path does 767 Mbit/s on ONE stream + +Operator asked for a WireGuard test from `ana-wg` to NH3 over the public +internet. It is the test that separates the *path* from the *crypto*, and the +answer is unambiguous. **It also overturns FOLLOW-UP 1's "re-architecting the +transport is not worth it" — that conclusion compared 8-stream numbers and was +wrong for single-stream workloads.** + +### Setup (fully torn down afterwards) + +`ana-wg` (10.250.50.252, Debian 12 LXC, 4 cores) already has an +internet-reachable WireGuard endpoint: wg0 on **UDP 31337**, published by +FortiGate VIP `wg-to-ana-wg` (extip **38.120.12.42** → 10.250.50.252:31337, +policy 46, service `WireGuard-LEET`). **No FortiGate change was needed.** A +temporary `wgt0` was created on nh3-dev (10.30.10.200/32) as a fourth peer on +wg0, measured, then removed — ana-wg is back to its original 3 peers and the +keys were shredded. `wireguard-tools` was installed on nh3-dev and **left in +place** (benign, and wanted if this becomes permanent). + +In this topology **neither gateway does crypto**: the FortiGate and the NH3 UDM +only NAT/forward UDP, and Linux does WireGuard at both ends. + +### The full comparison + +| path | crypto performed by | 1 stream | 8 streams | +|---|---|---|---| +| IPsec NH3↔ANA | FortiGate + UDM | 245 Mbit/s | 692 Mbit/s | +| IPsec ESH↔ANA | FortiGate + UDM | 268 Mbit/s | 715 Mbit/s | +| **WireGuard NH3→ana-wg** (same internet path) | **Linux + Linux** | **767 Mbit/s** | 763 Mbit/s | +| WireGuard NH3↔ESH (Site Magic) | UDM + UDM | 557 Mbit/s | 795 Mbit/s | +| plain routing through the 80F (inter-VLAN) | none | 940 Mbit/s | 939 Mbit/s | + +**One stream equals eight streams over Linux WireGuard (767 ≈ 763).** There is +no per-flow penalty at all, and a single flow already saturates the path. So +the ~245 Mbit/s per-flow cap is **not** the ISP, not the circuit, not the NH3 +uplink and not the physical path — all of which sustain 767 on one flow. + +Per-flow penalty ranks by implementation: + +- **Linux WireGuard — none** (767 → 763, flat) +- **UDM WireGuard — mild**, ~1.4× (557 → 795) +- **IPsec on this pair — severe**, ~2.8× (245 → 692) + +### Latency under load — the same story + +| path | idle | during ONE bulk stream | +|---|---|---| +| IPsec NH3↔ANA | 6.9 ms | **102 ms** avg, 136 ms max | +| WireGuard NH3→ana-wg | 6.2 ms | **12.7 ms** avg, 23 ms max | + +WireGuard carries **3.1× the single-stream throughput with 8× less latency +inflation** on the same wire. + +### Attribution — still not fully separated, and it no longer matters much + +Both IPsec measurements have a FortiGate *and* a UDM doing IPsec, so this still +does not isolate which one imposes the 2.8× penalty. Closing that would need +Linux↔Linux IPsec or UDM↔UDM IPsec on the same path. **But the practical +decision no longer depends on the answer**, because the fix is the same either +way and it is already demonstrated. + +### Recommendation (supersedes FOLLOW-UP 1) + +A **WireGuard site-to-site between NH3 and Anaheim, terminated on `ana-wg`**, is +worth real consideration: 3.1× single-stream, flat scaling, far better latency +under load, and it reuses infrastructure that already exists and is already +internet-reachable. It is also the architecture already proven for NH3↔ESH. +Open questions before committing: routing/failover if ana-wg (an LXC) is down, +whether it replaces or parallels the IPsec tunnel, and firewall policy for the +new transit. ana-wg CPU was only ~40% busy across 4 cores at 767 Mbit/s, so it +has headroom. + +**AND: `nconnect=8` on /mnt/smithy remains worth doing regardless** — it is the +same lever (turn one flow into many) and brokkr-smithy-dev has given standing +approval to apply it once the FortiGate work settles, with no need to ask again. + +--- + +## RESOLVED (2026-08-23): it is the UDM's software AES-CBC. The FortiGate is exonerated. + +Operator's theory — the UDM does IPsec in software with no crypto offload, so +the cost of the cipher itself is the limit — is **correct**, and it is now +demonstrated rather than inferred. He also correctly pointed out that +UDM↔UDM Site Magic is **WireGuard, not IPsec**, so that row never said anything +about UniFi's IPsec performance. It didn't, and I had leaned on it. + +### The controlled experiment: vary cipher cost, hold everything else + +AES-128 is 10 rounds, AES-256 is 14. If software crypto is the binding +constraint, throughput must rise when the cipher gets cheaper. If the limit +were the FortiGate's NPU, it would not move at all — hardware crypto is not +cipher-cost-sensitive in that range. Run A/B/A, single stream, 25–60 s each: + +| condition | ESP cipher | single-stream | UDM CPU | +|---|---|---|---| +| A | aes256-cbc + sha1 | 232.3 Mbit/s | 35.4% | +| B | **aes128**-cbc + sha1 | **281.8**, 274.9 Mbit/s | 35.5% | +| A again | aes256-cbc + sha1 | 244.9, 242.5 Mbit/s | — | + +**~1.16–1.20× faster on the cheaper cipher at identical CPU.** Same bytes of +CPU work, more payload through it. That is the signature of CPU-bound software +crypto, and it rules out the FortiGate's NPU as the limiter. + +### Correcting two of my own earlier claims + +1. **"UDM CPU is only ~7%, so it isn't CPU-bound" was WRONG — a sampling + artifact.** UniFi's `system-stats.cpu` refreshes on the device report + interval; 4-second sample windows were reading stale values. Under a + sustained 60 s single-stream load it reads **35.4%**, with load1 rising + 0.60 → 1.17. On a 4-core UDM Pro SE that is ≈1.4 cores — one core saturated + on crypto plus overhead. **Always drive load for ≥60 s before trusting a + UniFi CPU figure.** +2. **The "FortiGate per-session NPU offload" hypothesis is REFUTED**, not merely + unproven. It predicts no change from a cipher swap; a 20% change was measured. + +### Why the numbers all line up now + +- **1 stream = 1 core of UDM crypto** → ~240 Mbit/s on AES-256-CBC. +- **8 streams = ~3 usable cores** → ~692 Mbit/s, ≈2.9× the single-stream figure + on a 4-core box. Aggregate is noisy (492–692 across repeats on a live link) + and is *not* cipher-sensitive, consistent with it being bounded by the path/ + uplink rather than crypto once several cores are engaged. +- **AES-CBC is the specific villain: it is serial.** Each block depends on the + previous one, so the ARM AES instructions cannot pipeline across blocks. GCM + (CTR-based) and ChaCha20-Poly1305 both parallelise freely. That is why the + same UDM does 557 Mbit/s single-stream on WireGuard and only 240 on IPsec. +- **This retroactively vindicates the GCM cutover as the right idea aimed at the + right box** — GCM would have removed the serial dependency on the constrained + end. UniFi simply does not offer it, which is what made it impossible. + +### Options this opens + +- **AES-128 instead of AES-256: ~16–20% for free**, no topology change, one API + call per end. 128-bit is not the weak link here (SHA1 integrity is more + dated, and unchanged either way). Operator's call — **not adopted**, restored + to aes256. +- **WireGuard site-to-site via ana-wg: 767 Mbit/s single-stream** (3.1×), and it + sidesteps the UDM's IPsec datapath entirely. Still the biggest win available. +- Replacing the FortiGate **will not help this** — it was never the constraint. + Worth knowing before the cutover. + +### State left behind + +UDM network object verified **byte-identical** to its pre-test snapshot +(aes256/sha1). Tunnel up, selectors 1/1. FortiGate phase2 `pfi-ana-nh3` is +left as `aes256-sha1 aes256gcm aes128-sha1` — a permissive superset; the peer +offers only aes256 so the extra entries are inert, but **narrowing it back to +`aes256-sha1` is one line** if the looser list is unwanted. + +--- + +## FOLLOW-UP 4 (2026-08-23): a downstream WireGuard terminator costs nothing to forward through + +Operator's point: FortiOS has no WireGuard, so a WireGuard site-to-site must +terminate on a box *behind* the edge. Correct — and `ana-wg` (LXC, CT 113 on +pfi-pve, 10.250.50.252) already is that box. + +**This closes a gap in FOLLOW-UP 3.** That 767 Mbit/s figure was measured with +traffic terminating *on* ana-wg. Real traffic must be forwarded onward to other +Anaheim hosts, which was never measured. Now it is: + +| topology | 1 stream | 8 streams | +|---|---|---| +| IPsec, FortiGate ↔ UDM (today) | 245 Mbit/s | 692 Mbit/s | +| WG terminating **on** ana-wg | 767 Mbit/s | 763 Mbit/s | +| **WG transit: nh3 → wg → ana-wg → forward → ana-docker** | **763.8 Mbit/s** | **790.4 Mbit/s** | + +**Forwarding through the LXC is free** (763.8 vs 767). The downstream-VM +architecture delivers the full 3.1× single-stream for real transit traffic, not +just for traffic landing on the tunnel box. + +ana-wg while forwarding 764 Mbit/s: **~22% busy across 4 cores** (77.8% idle), +so roughly 0.9 cores. Note `/proc/loadavg` inside this LXC reports the *host's* +load, not the container's — do not read it as ana-wg's own. For contrast the +UDM burns 35.4% of its 4 cores to move 240 Mbit/s, so ana-wg has ample headroom. + +### Design consequences of terminating downstream — the parts that need decisions + +1. **Anaheim hosts must route to ana-wg, not to the FortiGate.** The 763.8 + figure was obtained with an explicit `10.30.10.200/32 via 10.250.50.252` + route on ana-docker. Without that, a host sends 10.100.0.0/16 to its default + gateway (ana-gw), which routes it back out the *same* interface to ana-wg — a + LAN hairpin crossing the FortiGate twice. **The hairpin variant was NOT + measured.** Options: DHCP option 121 pushing the route fleet-wide, a dedicated + transit VLAN for ana-wg, or accept the hairpin. +2. **New single point of failure.** Today site-to-site dies only when the edge + dies, which is total anyway. A downstream terminator fails independently. + Mitigation: keep the IPsec tunnel configured as a higher-metric fallback + route so it takes over when ana-wg is down. +3. **ana-wg is an LXC on pfi-pve**, so its ~0.9 cores and NIC traffic land on the + hypervisor shared with the rest of the Anaheim VMs. +4. **The NH3 end needs a terminator too**, and there are two shapes: + - **Linux VM at NH3** (nh3-dev or a dedicated VM on nh3-pve) — this is what + was measured: **764 Mbit/s**. + - **NH3 UDM's existing WireGuard server** (`PFI-NH3-WG`, wireguard-server on + UDP 31337) accepting ana-wg as a peer — plausible but **untested**, and + UniFi's WireGuard shows a per-flow penalty (557 Mbit/s single-stream on + Site Magic), so expect ~557 rather than 764. Still 2.3× today. + +### Standing recommendation + +Worth doing, but it is **a project, not a config tweak** — routing, failover and +policy all need deciding. The cheap wins remain available meanwhile and are +independent: `nconnect=8` on NFS mounts (approved by brokkr-smithy-dev, pending +the FortiGate work settling) and AES-128 for ~20%. + +--- + +## LANDED (2026-08-23): AES-128 on both tunnels; FortiGate public admin closed + +Operator directed: adopt AES-128 on **both** Anaheim tunnels, make-before-break, +then close the FortiGate's WAN and SSH admin surfaces. All done and verified. + +**Context that retires the WireGuard-in-a-VM design work:** the FortiGate is +being **replaced by OPNsense on a Dell R420**, which gives **WireGuard on the +edge device itself**. The downstream-terminator architecture (FOLLOW-UP 4) is +therefore moot — do not scope it. This also **un-parks the OPNsense migration**, +which auto-memory recorded as PARKED pending "hardware acquisition"; the R420 +is that trigger. + +### What changed + +Make-before-break on the FortiGate first, so neither tunnel dropped waiting on +a far end: + +| phase2 | proposal now | +|---|---| +| `pfi-ana-nh3` | `aes256-sha1 aes256gcm aes128-sha1` | +| `ana-eshudm-dyn` | `aes256-sha1 aes128-sha1` | + +Then each UDM flipped to `ipsec_esp_encryption: aes128`: + +| tunnel | UDM object | before | after | +|---|---|---|---| +| NH3 ↔ ANA | `pfi-nh3-ana` `697d64414c85dd2b6669b00a` @ 10.100.0.1 | 245 Mbit/s | **269.7** | +| ESH ↔ ANA | `esh-ana` `697723b9b9d4266dddf2bcc7` @ 10.0.0.1 | 268 Mbit/s | **304.3** | + +Single-stream gain ~10–13% here, against 16–20% in the earlier controlled A/B — +the difference is live-link variance, not a different result. Both UDM objects +were diffed field-by-field against pre-change snapshots: **the only field that +moved on either is `ipsec_esp_encryption`.** + +The FortiGate proposal lists were deliberately **left permissive** (still +accepting aes256). The peers offer only aes128 so the extra entries are inert, +and keeping them means a UDM reverting does not strand the tunnel. Narrowing to +`aes128-sha1` alone is a one-liner if the looser list is unwanted. + +### Admin surfaces closed + +`wan1 allowaccess` → **`ping`** (https + ssh removed) and `infra-ops` trusthost +→ **10.0.0.0/8 only** (the 8 wide-open ranges unset). Verified 443 and 22 closed +from both NH3 and ESH; management over the tunnel at 10.250.0.1 still works. +**Sequencing that matters: the close was executed over the TUNNEL path, not over +WAN** — removing `ssh` from allowaccess while connected over WAN kills the +session mid-command. + +**Consequence to hold in mind: ana-gw now has no out-of-band management path.** +If both tunnels drop it is console-only until someone is on site. + +### Gotcha: the two UDM vault items have DIFFERENT shapes + +- `unifi/pfi-udmse-api-key` → a **bare 32-char key**. `secret get` output is the key. +- `unifi/esh-udmpm-api-key` → a **19-line documentation note** with the key on a + `key:` line. `secret get` piped straight into a header yields a 1396-byte + value and the UDM answers **`400 Bad Request` from nginx**. Extract with + `grep '^key:' | awk '{print $2}'`. + +**The ESH key's first-ever confirmed WRITE happened here** (auto-memory recorded +it as read-verified only): a control PUT of the unchanged object returned +`rc:ok`, then the real change did too. That key has a full read+write admin role. + +--- + +## CORRECTION (2026-08-23): port 80 on the WAN IP is the FortiOS ACME listener + +The claim in the previous section that `.42:80` was an **ISP transparent proxy** +was **WRONG**, and so was the earlier warning that ACME renewal would fail with +port 80 absent from `allowaccess`. Operator pushed back asking where the port-80 +map terminated. It terminates **on the FortiGate itself**. + +**What it is:** the FortiOS **ACME HTTP-01 challenge listener**. `config system +acme` has `set interface "wan1"`, and FortiOS opens port 80 on that interface to +answer Let's Encrypt challenges **independently of `allowaccess`** — `wan1 +allowaccess` reads `ping` only and the port is still open. Every non-challenge +request returns a fixed 403 whose body is literally: + +``` +ACME Access OnlyACME Access Only +``` + +**Not a DNAT.** The full VIP table has 14 entries; only two land on `.42` — +`Kokoro-In` (:8880 → 10.250.50.51) and `wg-to-ana-wg` (:31337 → 10.250.50.252). +~~Worth noting separately: four VIPs are all-port static NAT~~ — **that claim was +WRONG, see the correction below.** All fourteen VIPs are scoped. + +### The methodology error that produced the wrong answer — worth not repeating + +The sniffer filter used was `dst host 38.120.12.42 and tcp port 80`. **`dst host` +matches only inbound packets**, so outbound SYN-ACKs were excluded *by +construction*; concluding "the box sends no SYN-ACK" from that capture was +unsound. Re-run with the bidirectional `host 38.120.12.42 and tcp port 80` it +immediately shows `wan1 out 38.120.12.42.80 -> : syn ack`. + +**Rule: when testing whether a box *answers*, the sniffer filter must be +bidirectional. `dst host` silently answers a different question.** + +### Consequences + +- **ACME renewal will work** with `allowaccess ping`. The earlier "add `http` + back or the cert expires" warning is retracted — FortiOS opens the challenge + port itself. Cert valid to 2026-10-27, renewal attempt ~2026-09-27. +- **It is not an admin surface** — static 403, no auth, no GUI. +- Its practical value is now low: WAN admin is closed, so the cert only serves + the internal GUI at 10.250.0.1, where the name would not match anyway. Killing + it (`config system acme` → unset interface) would close the last WAN listener + at the cost of cert renewal. Operator's call; **not done**. + +--- + +## CLOSED OUT (2026-08-23): ACME disabled; and the "all-port VIP" alarm was FALSE + +### ACME disabled — the WAN IP now exposes nothing + +`config system acme / unset interface` (the account object is left in place; +with no interface bound there is no listener). Verified: + +- **External scan of 38.120.12.42 across 55 ports: no open TCP ports at all.** +- Internal GUI at 10.250.0.1 still answers **200**, SSH still works. +- `admin-server-cert` is still `ana-fw.pfi` — the existing cert is untouched and + serves the internal GUI until **2026-10-27**; it simply will not auto-renew. + +Reverse with `config system acme / set interface "wan1"`. + +### RETRACTION: the four VIPs are NOT all-port + +A previous section claimed `Rustdesk`, `https-to-tacticalrmm`, `web-to-webhost` +and `web-to-sfcontainer` were unrestricted all-port static NATs. **They are not.** +A FortiOS VIP can be scoped **two different ways** and the parser used only +checked one: + +1. `set portforward enable` + `set extport ` — a single mapped port, **or** +2. `set service ""` on the VIP object — constrains the VIP to that service. + +All four use form 2. The custom services are narrow: `Rustdesk` = TCP +21115–21119 + UDP 21116 (the standard RustDesk range), `ssh-mapped-2223` = TCP +2223 only. **Every one of the 14 VIPs is scoped; none is unrestricted.** + +**Lesson: absence of `portforward` does NOT mean all-port on a FortiOS VIP — +check `service` too.** Better still, do what settled it here: scan from outside +rather than reading config. + +### Ground-truth public exposure (external TCP scan, post-change) + +| IP | open | maps to | +|---|---|---| +| 38.120.12.41 | *nothing* | — | +| **38.120.12.42** | ***nothing*** | the FortiGate itself — fully closed | +| 38.120.12.43 | 80, 443 | sf-ana-container 10.250.150.100 (SureFire tenant) | +| 38.120.12.44 | 22, 80, 443, 8025, 21115–21119 | gitea (→222), traefik, mailrise, RustDesk | +| 38.120.12.45 | 80, 443, 2223 | pfi-ana-webhost 10.250.50.52 (2223→22) | +| 38.120.12.46 | 443 | pfi-tacticalrmm 10.250.50.57 | + +Configured-but-closed: 8443 (mattermost-calls), 8444 (webdav-nas), 8880 +(Kokoro-In) — VIPs exist, nothing listening behind them. Worth a tidy-up during +the OPNsense translation but not exposure. + _Archived 2026-09-09._ + +- `[2026-08-24]` **ana-gw's public admin surface closed to zero open ports, ACME listener included.** Two of my diagnoses were wrong first (an "ISP proxy" that was the FortiGate, and an "all-port VIP" alarm that was a parser gap) — both from reading config instead of measuring from outside. +# [2026-08-24] ana-gw public admin surface closed to zero, ACME listener included + +WAN admin was opened at the start of the session as a cutover contingency +("so I don't have to drive down there"), then closed again on operator +instruction once the AES-128 work landed. Net result: **the FortiGate's WAN +address now exposes no TCP port at all.** + +## Final state + +External scan of `38.120.12.42`, 55 ports: **nothing open**. Verified from two +sites. `wan1 allowaccess` = `ping`; `infra-ops` trusthost back to `10.0.0.0/8`. + +**Consequence to hold: there is no out-of-band path to ana-gw.** If both tunnels +drop it is console-only. Re-open is two one-liners (allowaccess + trusthost) — +both are recorded in auto-memory `reference_fortigate_ana_gw_access`. + +## Port 80 was the FortiOS ACME listener, and I got it wrong first + +`38.120.12.42:80` answered a bare 403 (`ACME Access Only`, 101 bytes) with +`allowaccess` set to ping only. First diagnosis — "an ISP transparent proxy" — +was **wrong**, and the reason is worth keeping: + +> The sniffer filter was `dst host 38.120.12.42 and tcp port 80`. **`dst host` +> matches inbound only**, so outbound SYN-ACKs were excluded *by construction*, +> and concluding "the box sends no SYN-ACK" from that capture was unsound. + +Re-run bidirectionally (`host … and tcp port 80`) it immediately showed +`wan1 out 38.120.12.42.80 -> : syn ack`. **Rule: to test whether a box +*answers*, the filter must be bidirectional.** + +The listener is opened by `config system acme / set interface "wan1"` and +**bypasses `allowaccess` by design** — FortiOS needs port 80 for HTTP-01. It +was disabled (`config system acme / unset interface`); the LE cert (`ana-fw.pfi`, +valid to 2026-10-27) is untouched and simply stops renewing, which is fine +because WAN admin is closed and the box is being replaced. + +## Retracted in the same pass: the "four all-port VIPs" alarm + +Claimed four VIPs were unrestricted all-port static NAT. **False.** A FortiOS +VIP is scoped **two** ways — `portforward`+`extport`, *or* a `service` binding +on the VIP object — and only the first was checked. All 14 VIPs are scoped; +`Rustdesk` is TCP 21115–21119, `ssh-mapped-2223` is TCP 2223 only. + +Ground-truth external scan of all six public IPs is recorded in +`reference_fortigate_ana_gw_access`. Configured-but-dead: `:8443` +(mattermost-calls), `:8444` (webdav-nas), `:8880` (Kokoro-In) — tidy-up +candidates for the OPNsense translation, not exposure. + +**Lesson, twice in one session: measure from outside instead of parsing config.** +Both wrong answers came from a filter that answered a different question. + _Archived 2026-09-09._ + +- `[2026-08-24]` **Scriberr deployed on ana-ml2 GPU1, image built from source.** Three upstream bugs: the Blackwell image was never published, it must run as uid 10001, and `UV_LINK_MODE=copy` is required or two backends fail silently. +# [2026-08-24] Scriberr transcription deployed on ana-ml2, GPU1 + +Self-hosted audio/video transcription + diarization. Operator chose GPU +placement over ana-docker (8 cores shared with 50 containers, 37 GB disk) +against ana-ml2's 96 cores, `/tank`'s terabytes and GPU1's headroom. + +**Live:** `http://scriberr.ana.internal:8080` (DNS alias added), health `healthy`, +all seven backends up, zero failures: `whisperx pyannote sortformer parakeet +canary voxtral openai`. ~30 GB of weights on `/tank`. + +Stack: `stacks/scriberr/`. Full gotcha list in auto-memory +`reference_scriberr_ana_ml2`. + +## Three upstream bugs, none of them ours + +**1. The Blackwell image does not exist.** Upstream's README documents +`scriberr-cuda-blackwell`; GHCR has **no tags for it**. Published +`scriberr-cuda` covers sm_61–sm_89 only — on these sm_120 cards it fails or +silently drops to CPU. The real sm_120 path is `Dockerfile.cuda.12.9` +(CUDA 12.9.1, cu128 torch), **built from source**. Do not "simplify" the compose +back to the published image. + +**2. It must run as uid 10001, not 1000** — and the error lies: +`unable to open database file: out of memory (14)`. Error 14 is +`SQLITE_CANTOPEN`, not an OOM, on a box with 566 GB RAM. That Dockerfile creates +`appuser` at 10001 (Ubuntu 24.04 owns uid 1000 as `ubuntu`) and chowns `/app` to +it, while the entrypoint's PUID remap covers only the data dirs. +**Isolated by elimination**: SQLite writes fine to `/tank` as 1000 → not the +mount; fails on a plain named volume too → not the storage; the **published CPU +image works at PUID=1000** because there `appuser` *is* 1000. +Generalisable: *when a container "permission" bug appears, compare the uid the +image was BUILT for against the uid you are RUNNING as.* + +**3. `UV_LINK_MODE=copy` is required.** Scriberr builds each backend's Python env +with `uv` at start; uv's reflink mode fails on overlayfs+ZFS with +`Failed to clone … Resource temporarily unavailable (os error 11)`. **Partial +failure** — WhisperX and PyAnnote came up and the app looked fine while Parakeet +and Sortformer were silently absent. Occurrences 2 → 0 after the fix. + +## Related + +`speaches` on irv-ml1 **stopped** the same day (stack retained, one command to +restart): Eyra was abandoned pre-implementation because Scriberr covers the need, +leaving it with no consumer. Scriberr runs its **own** WhisperX in-container and +is **not** a speaches consumer. Idle footprint at stop was 274 MiB, not the +~5.9 GB quoted — that figure is the loaded-model working set. + _Archived 2026-09-09._ + +- `[2026-08-24]` **ESH DNS fixed at the IPv6 layer and the naming scheme went live on three hosts.** UniFi's RDNSS cannot be disabled but CAN be redirected — the field is only honoured when an explicit server is given. +# [2026-08-24] ESH DNS fixed at the IPv6 layer, and the naming scheme went live + +Reported as "`scriberr.ana.internal` doesn't resolve on my Mac, and nslookup +shows an IPv6 DNS server." Operator's diagnosis was right; the fix took three +wrong turns worth recording. + +## Root cause + +`esh-userland` has IPv6 PD with RA at `pref high`, and the UDM advertises +**itself** as the resolver via RDNSS. macOS honours RDNSS and prefers it over +the DHCPv4-supplied resolver, so queries went to the UDM — which does not know +`.internal` — and returned NXDOMAIN. AdGuard was never consulted. + +Two adjacent gaps found while there: `esh-userland`'s **secondary** v4 resolver +was `10.0.10.1` (the UDM itself), and `esh-server` had **DNS handout disabled +entirely**, so every host there got the UDM and could never resolve `.internal` +— esh-docker-vm was living proof. + +## The three wrong turns + +1. **`dhcpdv6_dns_auto=false` alone does nothing.** It is only honoured **when an + explicit server is supplied**. Setting it bare looks like a no-op and invites + the conclusion that the field is inert — which I drew, wrongly. Despite the + `dhcpdv6_` prefix it *does* drive the RA's RDNSS option on a SLAAC network. +2. **`wan_dns1` is NOT used by the UDM's LAN-facing forwarder.** Setting it to + AdGuard persists, reads back, and changes nothing. Proven with **fresh + uncached ad domains** — AdGuard blocklists answer `0.0.0.0`, the UDM returned + real IPs. Reverted. +3. **`force-provision` returns `rc:ok` and proves nothing** — consistent with the + known `cmd/devmgr` behaviour. + +Every failed attempt returned `rc: ok`. **Verify by observed effect.** RAs were +probed with a stdlib raw-socket Router Solicitation parsing option type 25 +(`rdisc6`/`tcpdump` were both absent; nothing was installed). + +## What landed + +RDNSS **redirected** rather than disabled — better than switching it off: + +| VLAN | v4 | v6 RDNSS | +|---|---|---| +| `esh-userland` | 10.0.50.45 + 10.100.50.40 | `…:4411:b105:50:45` | +| `esh-server` | 10.0.50.45 + 10.100.50.40 | `…:4411:b105:50:45` | + +The v4 secondary moved from the UDM to the **NH3 AdGuard** — reachable over +Site Magic and authoritative for the zone. ⚠ **A secondary only fails over on +SILENCE, not on wrong answers**: NXDOMAIN is a *successful* answer, the client +accepts it and never retries. A secondary that doesn't know your private zone is +a coin flip, not a spare tyre. `esh-cameras` deliberately untouched — routing +camera DNS through AdGuard's filtering risks their cloud features. + +## The naming scheme became real + +The resolver address is the scheme's first live use, replacing a MAC-derived +SLAAC address that would break on a NIC change. All three `esh-server` Linux +hosts now carry `4411:B105` ("FOR ALL BIOS"): + +``` +esh-docker-vm …:4411:b105:50:45 esh-pve-nas …:4411:b105:50:55 +esh-vm-db …:4411:b105:50:60 +``` + +Applied by an `if-up.d` hook that **derives the prefix at runtime** (self-heals +on re-delegation), backgrounds itself with a retry (SLAAC may not have landed; +a blocking hook would stall bring-up on a headless box), and adds nothing to +existing config. **Not** an `iface … inet6 static` stanza — on Debian that sets +`accept_ra=0` and would strand the host. + +⚠ **Proxmox bridges need `accept_ra=2`.** `esh-pve-nas` had link-local only +despite every sysctl looking right: `vmbr0.forwarding=1`, and the kernel ignores +RAs on a forwarding interface unless `accept_ra` is explicitly `2`. Fixed with +`accept_ra_defrtr=0` alongside, so it takes the prefix but **declines the default +route** — an IPv6 identity with no change to a hypervisor's routing. Expect this +on every Proxmox node when its LAN gets v6. + +Canonical: `docs/pfi/ipv6-naming-scheme.md`. UniFi limits: +auto-memory `reference_unifi_dns_rdnss_limits`. + _Archived 2026-09-09._ + +- `[2026-08-24]` **esh-vm-db brought onto the fleet infra-ops identity and given its first vaulted credential.** It previously had none: root and infra-ops refused key auth and `lkraven`'s sudo wanted a password nobody held, leaving `qm guest exec` from the hypervisor as the only privileged path. Break-glass root password at `secret get esh-vm-db/root-breakglass-password` (console-only; plaintext never crossed the wire — only its SHA-512 hash did). + _Archived 2026-09-09._ + +- `[2026-08-23]` **Anaheim's IPsec tunnel ceiling — investigated, then CLOSED 2026-08-24.** The 25%-of-2-Gbps framing was wrong (NH3's uplink is 1 Gbps); AES-GCM proved impossible; AES-128 landed instead. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md` + _Archived 2026-09-09._ + +- `[2026-08-23]` **selene retired after losing a head-to-head on its own job; `chat-judge` moved to gen, the model name 404s by design.** Also surfaced that **7 aliases share one seat** — cross-checking between them is an echo, which caught a real defect in brokkr's 46k-exposure R47 gate. +# [2026-08-23] selene seat retired after losing a head-to-head; 7 aliases share one seat + +## Why selene went + +Benchmarked against `gen` on selene's own job — 24 designed judge items with +checkable ground truth, pairwise + absolute modes, 3 repeats, run on **both** a +neutral JSON prompt and Selene's **native Atla template** (288 calls, free local). + +``` +neutral JSON selene 20/24 (83%) gen 23/24 (96%) +native Atla selene 21/24 (88%) gen 22/24 (92%) +``` + +gen won on both templates and **selene's best sat below gen's worst**. Selene was +given its own fine-tuned template as a fairness check before any recommendation; +it gained one point, not three. + +**Decisive defect: selene cannot emit "tie"** — 0/2 on both templates, forcing a +winner on every equivalent pair. For eval work that is the case that matters. + +brokkr-smithy-dev independently corroborated from the other end with a **null +control** (an excerpt compared against ITSELF, where tie is definitional): +`chat-judge`(selene) TIE **27/60 = 45%**, gen **60/60 = 100%**; ground-truth +recovery on real-corpus ranking selene **47% — chance** vs gen 94%. My 83-vs-96 +understated it: on a *ranking* task selene was a coin flip. Absolute scoring on +designed items is an easier task than ranking real text — the harness is a +**screen, not a verdict**, and its README says so. + +Reclaimed **17.2 GiB** on ana-ml2 GPU1 (free 1,818 -> 19,450 MiB). + +## The naming rule, restated the hard way + +I proposed repointing `selene-1-mini-8b` at gen and was **correctly overruled**: + +> never repoint a named model at a different model's endpoint — that is +> intentionally misleading + +`chat-judge` is a **role** alias (ADR-0012: consumers bind the capability) and +moved to gen with a deterministic judge profile copied from `image-judge`. +`selene-1-mini-8b` is a **model** name and was removed outright — it now returns +`HTTP 400 Invalid model name`, verified. The discriminator: *does the string +promise a capability, or an identity?* + +## The 7-way alias collision — the finding with the longest reach + +``` +chat-judge classifier gen image-judge +qwen-image-bench summarizer summarizer-large -> qwen3.8-27b-uncensored :8015 +``` + +Also colliding: `gen-frontier`/`gen-frontier-reasoning`/`glm-5.2`/`glm-5.2-reasoning`; +`ext-tts`/`gpt-4o-mini-tts`/`tts-1`/`tts-1-hd`; `reranker`/`reranker-a3-bge-v2-m3`. + +**Cross-checking a result against another alias measures nothing when they are the +same weights — agreement is an echo, not corroboration.** Documented at the head of +`model_list` in the live gateway config, because it belongs where people read it. + +This caught a real defect within hours: brokkr's R47 premium-corpus gate was about +to run ~46,000 record-exposures against `gen` with `summarizer` shortlisted as an +independent second opinion. They pinned the backing model in the preregistration +and dropped the second-alias idea instead. + +## Provenance seam (brokkr's pushback, adopted) + +The gateway returns the **alias** in the response `model` field, not the backing +model — so a per-call guard catches a swap *during* a run and is blind to one +*between* runs. **Role alias for routing, concrete model for provenance.** +`GET :4000/model/info` with the shared key already exposes backing model + +api_base; resolve at run start AND end and void on mismatch. + +## Artifacts + +- Harness kept at `tools/judge-bench/` (`--models` REQUIRED — a stale default + would silently benchmark a retired seat). +- `stacks/selene/` keeps compose + a README explaining the retirement. +- Technique worth stealing, from brokkr: **a control constructed so the correct + answer is DEFINITIONAL rather than judged cannot inherit the designer's error.** + Item vs itself; response vs its own truncation; text vs its own clauses + permuted. Add those before adding more judged items. + +Commits `ca3c984`, `b8a5355`. + _Archived 2026-09-09._ + +- `[2026-08-23]` **hrafn adopted; its CI reported green for its whole life while deploying nothing.** A staging dir inside the rsync target destroyed its own source mid-copy; the deeper fault was verify steps that asserted uptime, never content. +# [2026-08-23] hrafn adopted; its CI deploy reported green while deploying nothing + +`hrafn` — genuine-Chromium browser-fetch behind a REST API, for bot-gated sites +(Reddit first). Built by nevermore-claude on ana-docker, handed to infra-ops for +uptime ownership. Internal-only on `traefik-net`, no host port; consumers reach +`http://hrafn:8080`. Canonical at `stacks/hrafn/`. + +## Intake found a live credential exposure + +`/opt/docker/compose/hrafn/.env` was mode **0644 with a live 57-char bearer token** +— verified as real exposure by reading it as `nobody` on a box with four +interactive accounts. Tightened to 0600. That triggered the wider sweep (see +[[2026-08-23-ana-docker-env-perms-sweep]]). + +## The CI defect — the one worth remembering + +I authored the deploy (elway playbook + gitea workflow) to replace a hand-rsync, +tagging the image with the commit SHA for provenance. nevermore-claude later found +v1.0.0 deploying "green" while the host still served 0.1.0. + +**Root cause was mine and nastier than either hypothesis.** The staging dir was +`$compose_dir/.stage` — **inside** the rsync target. So +`rsync -a --delete $compose_dir/.stage/ $compose_dir/` deleted `.stage` from the +destination (absent from the source listing) **during** the transfer, destroying +its own source mid-copy. Reproduced exactly: + +``` +before: app.py="OLD" leftover.txt .stage/app.py="NEW" +after: app.py="OLD" leftover.txt GONE, .stage GONE +``` + +Deletion succeeded, the copy silently did not, rsync exited 0. So the directory +*looked* converged while host source stayed frozen at the first manual rsync — +and because the build's `COPY` inputs never changed, Docker full-cache-hit and +every SHA tag aliased one image. **The provenance the tagging existed to provide +was false for the pipeline's entire life.** + +**The real failure is the verification.** The verify steps asserted the marker, +container health, and a 200 from `/readyz` — all of which pass against a +completely frozen host. None measured *content*. A deploy that reports success +without asserting the bytes changed is verifying an **uptime**, not a deploy. + +## Fixes + +- stage at `/tmp/hrafn-deploy-stage`, outside the target +- CI computes `context_sha256` over the shipped file list; the playbook recomputes + it **on the host after the converge** and fails on mismatch +- compare the running container's `src/**/*.py` against the host's, so a SHA tag + cannot name layers the image lacks +- **compare `*.py` only** — `pip install .` generates `src/*.egg-info/*` inside the + image and `__pycache__` appears at runtime, so a naive `find src -type f` compare + false-fails on every healthy deploy. Verified against a known-good container + before shipping (12 host files, 18 in container, 0 content differences). +- declined `--no-cache`: a cache hit is *correct* when the context is genuinely + unchanged; assert the property rather than brute-force it. + +## Access + +Operator granted claude-bot **write** on `vh/hrafn`, so infra-ops maintains the +pipeline it owns instead of routing patches through the repo holder. `vh/hrafn` is +canonical; `stacks/hrafn/ci/` is a verified mirror. + +Commits `b6924de`, `b001d0c`, `11b9d18`, `b38c369`, `9642952`. + _Archived 2026-09-09._ + +- `[2026-08-23]` **Worldtree b187 shipped; all three instances de-armed from a 69-day-stale `:latest`; Matrix homeserver re-plumbed to personal.** Includes the `:8009`-is-demo port trap that an IP-only fix would have walked into. +# [2026-08-23] Worldtree b187 shipped; all three instances de-armed from a 69-day-stale `:latest`; Matrix homeserver re-plumbed + +## b187 pre-stage (#405 phases 1+2) + +The matrix bridge stopped embedding the engine and became an HTTP client of the +Conversation API, so `WORLDTREE_API_URL` became **boot-blocking** — absent from the +container env, the bridge exits by design. Demo's compose never passed it; the next +recreate would have crash-looped. Pre-staged on demo and personal (additive, backed +up, verified with `docker compose config`, nothing restarted). + +**Key decision, and I got its scope wrong first.** I argued demo should stay keyless +(no homeserver -> no rooms -> no turns -> no 401s). Right about turns, **wrong about +scope**: the engine preflight authenticates at boot regardless of homeserver, so demo +booted permanently degraded. Corrected — key `341c1488` minted under worldtree-dev's +recorded authorization, vaulted, wired, three-hop hash-verified. + +## The 69-day-stale `:latest` landmine + +All three instances pinned `WORLDTREE_IMAGE=.../worldtree:latest` in `.env` while +running SHA-tagged images built that day. Local `:latest` = `b19afd71d7cc`, built +**2026-06-14**. So ANY `docker compose up` — anyone's, for any reason — silently +downgraded that service by 69 days. Same footgun as the 2026-06-15 outage. + +Re-pinned all three to their running SHAs (Worldtree #410), verified by rendering +compose config rather than reading `.env`, containers untouched. Playbook at +`playbooks/repin-worldtree-image.yaml`. + +**`worldtree-pinned` was the worst case:** the instance whose entire purpose is being +frozen was running a **dangling image with no repo tags**, kept alive only by the +running container. One `docker rm` from garbage collection. Tagged +`:446e5807bf43` first, then pinned. + +The guard I wrote had two bugs the pinned case exposed: it compared the container's +`.Config.Image` **string** (only the tag it was CREATED from — pinned was created +from `:latest` back when that meant 446e5807), and it reported CHANGED +unconditionally. Now compares **image IDs** and skips when already correct. + +## Matrix homeserver ownership + +Operator ruled: **personal owns the Matrix bridge.** The appservice tokens were never +missing — both sat at length 64 in the vaulted dev `env.sh` while both deployed +instances had them at length **zero**. Someone wired four of six Matrix vars and +stopped. Wired them into personal, three-hop verified. + +**The trap worth remembering:** Synapse's registration pointed at +`http://10.100.10.50:8009` — nh3-dev, a dead epoch, with transaction 2801 queued at +512s backoff. The natural fix (swap the IP) gives `10.250.50.152:8009` which is +**DEMO's** bridge, and Synapse can reach both — it would have connected, delivered, +and looked correct while routing the operator's live rooms to the demo instance. +**Personal's bridge is :8010.** `docker port` is ground truth. + +Corrected the URL, restarted Synapse (healthy in 32s after 3.5 months up), verified +`GET /_matrix/app/v1/ping -> 200` from inside the Synapse container. worldtree-dev's +smoke passed first try: room created, mimir accepted the invite, a real engine turn +ran, mimir replied in persona voice. #408 closed. + +## Open on worldtree-dev's side + +- **#411** — personal's bridge logs `Debug sink init failed: Permission denied: + /app/sessions/debug_rooms.json`. It creates two debug rooms but cannot persist + their IDs, so **every restart mints a fresh pair on the live homeserver**. Room + litter that compounds silently. Needs a which-container-writes-what check on the + sessions volume before anyone chowns it. +- Bridge/engine agent-roster drift: 6 of the bridge's 9 configured agents are not + listed by the engine on either instance. +- Historical Domari pairwise verdicts from the selene era are coin-flip-grade + (see [[2026-08-23-selene-retired-alias-collision]]); worldtree-dev banked that so + no future arc leans on them without re-judging. + +Commits `064181a`, `bb19a96`. + _Archived 2026-09-09._ + +- `[2026-08-23]` **Every secret-bearing `.env` on ana-docker tightened to 0600** — eight stacks including vaultwarden and traefik, verified exposed by reading one as `nobody`. +# [2026-08-23] Every secret-bearing `.env` on ana-docker tightened to 0600 + +Found while taking uptime ownership of hrafn: its `.env` was mode 0644 with a live +bearer token. Not a hrafn lapse — **0644 was the de facto pattern on the host**. +Eight stacks carried secret-shaped vars in world-readable `.env` files on a box with +four interactive accounts, verified as real exposure by reading one as `nobody`. + +Swept: **vaultwarden, traefik**, beszel, gitea-runner, miniflux, news-digest, +searxng, vor. (hrafn and nevermore were fixed separately the same day.) Six other +stacks already used 0600, so this converged on the existing house pattern rather +than inventing one. Post-sweep the host has **zero** secret-bearing `.env` readable +by `nobody`. + +Playbook: `playbooks/tighten-env-perms.yaml`, one run per stack, re-runnable. + +## The check that matters + +Every run asserts `docker compose config` still renders **as the deploy user** +(`lkraven`), not as root. Checking the mode proves the bits changed; only rendering +as the deploy user proves the next deploy can still resolve its variables. + +## Two gotchas recorded in the playbook + +- **vaultwarden looked like it bind-mounted its `.env`** — which would mean the + *container's* UID reads it and 0600 could break the password vault. It does not: + that `- .env` is under `env_file:`, not `volumes:`. My grep matched the YAML list + item without checking its parent key. The playbook now **refuses** any stack that + genuinely bind-mounts its `.env`, since that case is read by the container UID. +- **elway prompted for a sudo password.** The `ana-docker` ssh alias resolves to + `lkraven`, who needs one; **`infra-ops@10.250.50.70` has NOPASSWD**. `corviduo-dev` + was repointed to infra-ops at some point and `ana-docker` was not. Run elway against + the infra-ops target on this host. + +Commit `a896c0a`. + _Archived 2026-09-09._ + +- `[2026-08-23]` **`pfi` gitea org created; claude-bot is an Owner and creates repos self-serve.** Closes the repo-creation half of the credential-migration directive — `vh` is a USER namespace so no service account could ever create there. Repo creation needs `write:user` + `write:repository` + `write:organization`; `POST /users/{u}/tokens` is basic-auth only, so minting needs the account password. Default new repos to `pfi/`. (`vh/eitri-smithy` was its first tenant, then moved.) + _Archived 2026-09-09._ + +- `[2026-08-23]` **Booth: kept boards are deletable and link rows are prunable.** `release` on a kept card drops the sentinel so the existing × applies; `booth links` / `booth unlink ` prune one row. Rows are addressed by **content id, never position** — the board is append-only and multi-writer. **Releasing a board RESETS its TTL clock** (unlink bumps the dir mtime), so unkeep-and-wait is a 24h delay, not a delete. (`4be880f`, `0ad332b`) + _Archived 2026-09-09._ + +- `[2026-08-22]` **DFlash2 spec-decode measured on our own stack; `sec` promoted to it.** +18–21% accepted length and +15–18% throughput over MTP k=3, drafter proved model-agnostic across two finetunes to 0.06%, and the k=7 MTP *control* showed deeper MTP is a throughput trap. +# DFlash2 speculative decoding — measured on our own stack (2026-08-22) + +Operator-driven session. **Read the epistemic labels.** During the chase we generalised from +observations that later proved wrong; this file separates what was *measured* from what remains +*hypothesis*, and records the wrong turns so nobody re-derives them. + +## What DFlash2 is + +A **2B draft model** (3.85 GB bf16) for speculative decoding against Qwen3.8-27B — +`incoai/Qwen3.8-27B-DFlash2`, Apache-2.0, blog `inco.ai/blog/dflash2`, upstream `z-lab/dflash`. +Block diffusion: drafts a whole 8-token block in one pass, with a candidate selector tracing a +path through per-slot top-K. Lossless (greedy matches the target). + +vLLM support merged **2026-08-21 05:27 UTC** as PR **#52816** (`b389ac29`). Method string is +**`"dflash"`**, not `dflash2`. + +## ✅ MEASURED — throughput and acceptance + +Single instrument (`specbench.py`, 8 fixed prompts, temp 0, max_tokens 256), delta against +vLLM's own `spec_decode` counters. The MTP k=3 numbers reproduce our recorded 58.4% / 55.3% +figures exactly, which is what validates the instrument. + +| seat | config | accepted tok/forward | throughput | +|---|---|---|---| +| gen (orcarouter) | MTP k=3 *(production)* | 2.753 | 114.9 tok/s | +| gen | MTP k=7 *(control)* | 3.041 | **74.0 tok/s** | +| gen | **DFlash2 k=7** | **3.254** | **131.9 tok/s** | +| sec (M.O.G.-SEC) | MTP k=3 *(production)* | 2.676 | 110.5 tok/s | +| sec | **DFlash2 k=7** | **3.252** | **130.0 tok/s** | + +**⭐ The k=7 MTP control was essential and inverted the obvious read.** Going deeper on MTP +*improves acceptance* (2.753 → 3.041) while **destroying throughput** (114.9 → 74.0). Our MTP +head is a single module (`mtp_num_hidden_layers=1`, only `mtp.layers.0`, 15 tensors) run +autoregressively, so k draft tokens cost k sequential forward passes. **"Just raise +num_speculative_tokens" is a trap** — without the control I would have recommended it. + +DFlash2's win is therefore **not better per-token acceptance** — our MTP is actually *better* at +position 0 (79.6% vs 75.4%). It is that block drafting makes depth nearly free. + +**⭐ The drafter is model-agnostic across finetunes — 3.254 (gen) vs 3.252 (sec), a 0.06% +difference**, with superimposable per-position curves. One drafter file on `/tank` serves both. + +## ✅ MEASURED — how DFlash2 runs (answers "can one drafter serve both seats?") + +**EAGLE3-style coupled, not standalone.** In vLLM: `load_model(self, target_model)` binds it to a +specific target object; `pass_hidden_states_to_model=True`; `gpu_model_runner` reads +`dflash_config.target_layer_ids` → `[i+1 …]` to register auxiliary hidden-state capture on the +target at layers **5, 19, 33, 47, 61**. It even reads the target's RoPE style at load. + +Consequences: +- **Weights file is shareable** (one download, both seats mount it) — gen and sec are + architecturally identical on every dimension the drafter needs: 64 layers (deepest tap 61), + hidden 5120, intermediate 17408, vocab 248,320 > mask token 248,070. +- **VRAM is NOT shareable — 3.85 GB per seat.** The drafter lives inside the target's engine + process, consuming hidden states mid-forward. Two seats are two processes; there is no + cross-process sharing mechanism and there could not be. + +## ✅ MEASURED — it works on our stack, which the card does not claim + +The card tests stock BF16 on an H200 with FlashAttention 3. Verified here instead: +**abliterated + NVFP4 `compressed-tensors` target ✓, Blackwell sm_120 ✓, DFlash2 CUDA graphs +captured ✓.** None of that was documented anywhere. + +## 🔶 HYPOTHESIS — why our acceptance trails the published numbers + +Both our targets land at ~3.25 accepted length against the card's 4.10–5.46 on stock BF16. +**Finetune drift is ruled out** — two *different* finetunes gave identical results to three +decimals. The shared variable is **NVFP4 quantization of the target**, which is mechanically +plausible (the drafter reads quantized hidden states at its five taps). Second candidate: +prompt distribution (ours general-purpose, theirs GSM8K/MATH/HumanEval/MBPP/MT-Bench). +**Neither is confirmed.** Settling it needs a BF16 target seat (~56 GB) — a real GPU window. + +## ❌ RETRACTED — the "MTP head mismatch causes the degeneration" hypothesis + +**Operator ruling, 2026-08-22: this hypothesis is WRONG. The degeneration lives in the un-fixed +vLLM, not in the weights.** Recorded here rather than deleted, because it was reasoned to +confidently enough that a future session could re-derive it. + +**Two independent failures produced it, and the second is the instructive one:** + +1. **I treated a false dichotomy as a deduction.** Having verified gen and sec run an identical + engine (same image ID `sha256:bd3236cff208…`, same live version + `0.27.2rc1.dev150+g311b3513a` read from inside both processes, same flags bar + `gpu-memory-utilization` 0.43 vs 0.44), I concluded "config is eliminated, therefore it is the + weights." That does not follow. **An engine bug present in BOTH seats is not exonerated by the + two seats being identical** — it just means the engine cannot explain a *difference*. It can + still explain the *failure*. +2. **The difference I was explaining may not exist.** The premise was a single operator + observation of sec degenerating at ~2k, made during a session with many concurrent changes. + **n=1 under heavy concurrent modification is not evidence** — see the meta-lesson below. + +**What survives as fact** (measured, still true, just not causal): sec's MTP head *is* +byte-identical to `qwen38-27b-uncensored-bf16` across all 15 tensors — a stock head on a +security-finetuned body, because the `Qwen3_5ForConditionalGeneration` wrapper never loads the +head, so the finetuning could not reach it. gen's orcarouter head *was* abliterated in-band by +its author. Acceptance differs slightly (gen 58.4%, sec 55.9%). **All true. None of it shown to +cause multi-turn degeneration.** + +**Current standing explanation: the degeneration is an engine bug in the un-fixed vLLM.** Both +production seats run `311b3513`, which is **172 commits behind GDN spec-decode fix #53077** +(merged 2026-08-20). `#51113` is present in that build and is therefore **necessary but +insufficient** on its own. + +## ⭐⭐ META-LESSON — n=1 during a busy session is not evidence + +The operator's own framing, and it generalises past this incident: **an observation made while +many things are being changed at once cannot carry a causal claim, no matter how confidently it +is reported.** Tonight that single observation became the load-bearing premise for a weights-side +hypothesis, a root-cause narrative, and very nearly a recommendation. + +This is the same failure the gen-seat compose file already warns about in different words — *"a +passing probe is NOT sufficient evidence"* — inverted. That note guards against trusting a +**negative** result from a synthetic test. This one guards against trusting a **positive** +sighting from an uncontrolled session. Both reduce to: **hold the system still, or do not draw +causal conclusions from it.** + +Applies equally to the "coherent to 10k" observation below — same n, same conditions, opposite +direction. Neither observation is worth more than the other. + +## ⚠️ CONFOUNDED — and the "before" state is itself unreliable + +sec now runs DFlash2 on a newer build and the operator reports **coherent to 10k tokens with +adversarial nonsense prompts**. ⚠ Treat this the same way as the 2k sighting it is being compared +against: **n=1, uncontrolled session, not evidence.** The comparison is weak on *both* ends. + +**Two variables changed at once:** + +1. **Engine**: `311b3513` → `e9d1398d`, **+259 commits, `behind_by=0`** (a strict superset), + including GDN spec-decode fix **#53077** (merged 2026-08-20) that production is **172 commits + behind**. +2. **Drafter**: frozen MTP head → DFlash2 reading live hidden states. + +**Isolating it = run MTP k=3 on the same new build.** Not yet done. + +**#51113 is present in BOTH builds** (verified by ancestry, `behind_by=0` each) — so the +"proper upstream fix" our compose comment credits is **necessary but insufficient**; sec ran it +and still degenerated. Related open upstream: **#53180** (quantized Qwen3.8-27B hybrid GDN + MTP +producing *silent* degenerate output, no fix), **#41884** (DFlash + prefix caching on hybrid, +IndexError, workaround is disabling one). + +## ❌ WRONG TURNS — do not repeat + +- **Version strings are not lineage.** The DFlash2 build reports `0.26.1rc1.dev1048` and our + production nightly `0.27.2rc1.dev150`, which *looks* like a regression. It is a setuptools_scm + tag-reachability artifact. **Use the GitHub compare API and check `behind_by`.** +- **Docker Hub push timestamps lie about source freshness.** `nightly-ba07e4a4` was *pushed* + 06:12 UTC, comfortably after the 05:27 merge — but *cut* from a 03:46 commit that predates it. + **Grep the image for the symbols you need.** Believing the timestamp would have cost an RP-seat + outage to serve a model the engine could not instantiate. +- **`--max-num-batched-tokens` was not the image truncation.** Raising it 16,384 → 32,768 on that + theory changed nothing and cost ~3 GiB of peak activation, which came straight out of the KV + pool. The cap was the tokenizer (§3.14 of the playbook). +- **"1M needs YaRN, absent from config" is FALSE for the sec quant.** It is fully present: + `rope_type: yarn`, `factor: 4.0`, `original_max_position_embeddings: 262144`, + `max_position_embeddings: 1000000`. Context is a KV-memory choice, not a model limit. + +## Live state — PROMOTED to the compose stack 2026-08-22 + +**Operator-approved after real-use testing** ("performing very well"). The experimental +standalone container is gone; `stacks/mog-sec/` is canonical and `restart: unless-stopped` means +it survives reboots. Cutover verified: **KV pool 526,617 / 1.10x — identical to the container it +replaced**, restarts 0, both gateway aliases serving, DFlash2 confirmed drafting at k=7 +(231 draft tokens over 33 drafts), vision working. + +⚠ **One variable was deliberately REMOVED, not carried over.** The old stack hardcoded +`PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`; the validated DFlash2 container never set it, +and playbook §3.10 records expandable_segments corrupting retained tensors elsewhere. The compose +now defaults it EMPTY (`MOG_ALLOC_CONF`). Promoting it as-was would have shipped a variable the +tested configuration did not have. + +**Compose is now parameterised for the shapes that differ:** `MOG_SPEC_CONFIG` carries the whole +speculative JSON (dflash needs `"model": "/drafter"`, MTP must not have one — a method+tokens +template cannot express both), plus `MOG_MM_PROCESSOR_KWARGS`, `MOG_DRAFT_MODEL`, +`MOG_MAX_NUM_BATCHED_TOKENS`, `MOG_ALLOC_CONF`. + +**ROLLBACK:** `.env.bak-pre-dflash2-20260822` and `compose.yaml.bak-pre-dflash2-20260822` on the +host; or one line — `MOG_SPEC_CONFIG={"method": "qwen3_5_mtp", "num_speculative_tokens": 3}` plus +the old `MOG_IMAGE`. + +| | production sec | current | +|---|---|---| +| image | `nightly-311b3513` | `nightly-e9d1398d` | +| speculation | MTP k=3 | **DFlash2 k=7**, drafter `/tank/aimodels/qwen38-27b-dflash2-drafter` | +| max-model-len | 262,144 | **480,000** | +| KV pool | 418,218 (1.60×) | **526,617 (1.10×)** | +| images | 4096² → 16,384 tok | **2048² → ~5,125 tok** (`--mm-processor-kwargs` size cap) | + +⚠ **`--gpu-memory-utilization 0.55` is the stable ceiling** while GPU1's other tenants are up. +0.58 sized KV at 594,172 then **OOM'd during CUDA graph capture** — the process reached 57.49 GiB +against ~57.6 free. Real 1M context needs ~49 GiB of KV and therefore evicting most of GPU1. + +Canonical config: `stacks/mog-sec/{compose.yaml,.env.example}` in this repo. + _Archived 2026-09-09._ + +- `[2026-08-22]` **Quant pipeline shipped a crippled tokenizer for months — fixed at source.** `quant_mixed_nvfp4.py` baked its calibration truncation (`max_length 2048`) into every mixed-NVFP4 build; latent on old transformers, fatal on new. Both live quants corrected, pipeline now saves a source-pristine tokenizer and asserts it. Playbook §3.14. (`0755ba7`) + _Archived 2026-09-09._ + +- `[2026-08-22]` **`sec` retuned to util 0.52 / 420K after a runtime OOM at 0.55/480K** — `gpu-memory-utilization` is not a hard reservation; activation grows past the dummy-data profile and six vLLM containers share GPU1. Also measured: the KV pool varies ~6.6% between boots, so max-model-len must be sized against the *lower* observation. (`6e82899`) + _Archived 2026-09-09._ + +- `[2026-08-22]` **Max-Q 1.8× spread does NOT apply to LLM decode — measured, not argued.** ana-ml2 draws 256–266 W of 300 W under sustained 100% decode with `SW Power Cap: Not Active` and clocks pinned. Corrected to brokkr-smithy-dev after I had lent the claim credibility; 122B figure (~90–93 tok/s at 262K) stands as a straight number. + _Archived 2026-09-09._ + +- `[2026-08-21]` **ESH internal IPv6 live on two LANs; the Cityside v4 static is a CARRIER problem, proven.** A full gateway reboot forced a fresh DHCP DISCOVER and returned the identical CGNAT address. YaRN was already configured — "1M needs YaRN, absent" was false. → `persistent-memory.d/2026-08-22-dflash2-spec-decode.md` sibling entry in `ad21302` + _Archived 2026-09-09._ + + ## The finding `esh-pve-nas` (`esh-nas-pve.esteban.net`, 10.0.50.55) runs PVE root off a **USB @@ -2056,6 +4338,32 @@ _76 older entries archived to archival-memory.md._ infra-ops@ana-docker` HAS NOPASSWD root.** A `sudo cp` as lkraven silently failed → nearly punted the rest-server recovery. Reach for `infra-ops@ana-docker` for sudo ops. _Archived 2026-07-08._ + +- `[2026-08-25]` **Four throughput levers measured and killed — do not re-chase.** (1) **Fused MoE / `grouped_mm`** — 0.9% *slower* than the Python loop and dense GEMM is only 7.9% of the step, capping the whole category near 10%. (2) **CUDA graphs / `torch.compile` over the expert loop** — the two-term scaling fit closed with residuals under 3ms and needed NO constant term, so there is no fixed per-batch cost to amortise; 3,840 expert-GEMM launches per forward are not what we pay for. (3) **`liger` fused linear CE** — the chunked CE measured **1.1% of the step** forward, ~3% with recompute. A tidy-up, not a lever. (4) **Selective gradient checkpointing** — ~2% of a post-fix step, real bug surface. Also: **token-budget batching is dead by the same fit** — with no constant term, total time over a fixed set of widths is invariant to how you group them; only the widths matter, which is exactly why bucketing works and repacking does not. + _Archived 2026-09-09._ + +- `[2026-08-25]` **`sample_packing` is NOT strictly better than bucketing on this model, and I told the operator it was before brokkr corrected me.** Packing needs FA2 varlen or a block-diagonal mask; FA2 is unavailable here (head_dim 512 > 256 cap), so packing means an explicit 4D mask on EVERY batch. Bucketing produces **78.3% exactly-zero-pad micro-batches** which recover the `is_causal` fast path on the 5 global layers — measured at 9.4% of step time. Packing forfeits that. ⚠ **The conclusion flips under `flex_attention`**, where a block-diagonal mask is just another BlockMask: do not carry "packing is bad" past the backend decision. + _Archived 2026-09-09._ + +- `[2026-08-25]` **Merging a tune back toward STOCK to fix overfitting would UNDO the abliteration.** brokkr recommended a 50/50 merge-back, then retracted it himself: the published recipes merge into `google/gemma-4-*-it`, and following that literally re-installs exactly the refusal directions the abliteration removed — silently, because the merged model looks *healthier* on general benchmarks. Any merge-back must target the SAME abliterated base. Wider lesson: **recipe cards are per-checkpoint artifacts, not per-family** — the advice came from a card for a DENSE STOCK 31B applied to a MoE ABLITERATED 26B-A4B, three axes apart on a shared name. + _Archived 2026-09-09._ + +- `[2026-08-24]` **AES-GCM on the Anaheim tunnels — impossible, not merely hard.** UniFi's manual site-to-site IPsec implements no AEAD cipher at all: eight GCM spellings rejected `api.err.InvalidPayload` against a passing `aes256` control. Blocks both tunnels since both far ends are UDMs. Accepted enum is `aes128/aes192/aes256/3des` — and 3DES is *slower* (no ARM instructions, 64-bit blocks), so AES-128 is the floor. + _Archived 2026-09-09._ + +- `[2026-08-24]` **Pointing the UDM's `wan_dns1` at AdGuard — silently ignored.** It persists and reads back correctly but the LAN-facing forwarder never uses it; proven with fresh uncached ad domains (AdGuard answers `0.0.0.0`, the UDM returned real IPs). Reverted rather than left in place. + _Archived 2026-09-09._ + +- `[2026-08-24]` **A multi-DUID DHCPv6 VM to claim NH3's seven unclaimed /64s — declined by the operator.** The BGW has no IP-passthrough (confirmed, we hold admin), so the only route needs re-cabling, split-stack routing and **rebuilding the entire v6 firewall policy off the UDM**. The prefixes are easy; the firewall rebuild is why nobody wants them. Do not re-raise on "there are seven free prefixes". + _Archived 2026-09-09._ + +- `[2026-08-23]` **A `HEAD == GITHUB_SHA` assertion in the hrafn CI — added, broke the checkout twice, removed.** It needed the `git` binary (run 9920, exit 127); installing `git` then flipped `actions/checkout@v4` off its **node** implementation onto the git binary, which died on a missing CA bundle (run 9921). A nice-to-have assertion changed the checkout's code path and broke a working pipeline. Removed rather than patched with `ca-certificates` — it guarded a hypothesis that proved wrong. **Do not add `git` to that prereq step.** + _Archived 2026-09-09._ + +- `[2026-08-23]` **Repointing `selene-1-mini-8b` at gen's endpoint — proposed by me, correctly overruled.** *"never repoint a named model at a different model's endpoint — that is intentionally misleading."* The trap is that it does not feel like deception; it feels like sparing consumers a migration. That framing is the tell. Role aliases move; model names die with the model and 4xx. + _Archived 2026-09-09._ + + ## Archived 2026-08-02 — Recent decisions (archived) ### 2026-07-08-worldtree-mimir-deploy-blocker-resolved-mid-session @@ -4041,3 +6349,558 @@ overrides only the three VRAM ones) and a README pointing at `docs/runbooks/heretic2-nvfp4-mtp-seat.md` rather than duplicating it. _Archived 2026-09-05._ + +## Superseded in-flight snapshots (archived) + +_Snapshot of the in-flight section as it stood before the 2026-09-09 rewrite (2026-09-08 and 2026-09-05/06 blocks). Archived 2026-09-09._ + + +_As of 2026-09-08 (fleet-ops session). SUPERSEDES lower framing: run 4 = STILL-COUPLED; run 5 is +COMPLETE and gated RESCUED (02:13 PDT). Live open items:_ + +- **✅ ERP run 5 COMPLETE — gate = RESCUED (2026-09-08 02:13 PDT, landmark R49.5).** FIRST arm of this + line where the capability gate did NOT fail. The dependency-forcing slot (GovReport+QMSum, only 3.46% + of loss) broke the diversity↔coherence coupling that run 4 (STILL-COUPLED, 20.6% instruct slot) and + 3c (20× LR cut) could not — **STRUCTURE of the loss was the lever, not its mass; INERT did not fire.** + T4 long-context 8/8 (run 4: 5/8; base 8/8); t4_dissect noise@31 tuned 0.9062 vs run-3 tuned 0.5625; + diversity held (rp density 3.37→0.00, story 2.86→1.58). Reported-beside (not in the cell, all + de-gated + stated): T3 constraint 8/8→6/8 (NEW loss, ship-path list); RP length 68w vs 250-floor = + PARTIAL fail (short-QA slot + style shift); refusal erosion rides with the style shift (k=25 both + arms, CSAM clean); free-check base LEVELS 5-6pp below run 4 (vLLM 0.28.0 unchanged — infra-ops + confirmed — so a generations shift, not a stack delta; taxes every cross-run number). Write-up + brokkr-smithy `research/R47-premium-corpus-gate/run05-gate/RESULT-run05-gate.md`; FLOOR-LOCKED + `0f3e4e2` (cites infra-ops' base index-sha 907826a6). Whole gate infra-ops-served on pfi-gx10:8098 + (name-keyed base→tuned swap, hands-off honoured, sha+stack answers on record). Launch procedure + + canonical: eshpfi `scripts/erp-tune-gx10/` + `docs/runbooks/gx10-run-05.md`. **✅ SEAT: `erp-tune-v5` + SERVED on pfi-gx10:8098 (merged-run05); LiteLLM `trial` alias REPOINTED 3c→v5 (operator, 2026-09-08) + so it's testable from Open WebUI — verified end-to-end (trial→erp-tune-v5, coherent output). config.yaml + trial block rewritten to run-5 reality incl. the measured refusal-erosion note. ⚠ Seat is hand-launched + (vllm-run05.pid), NO restart policy/systemd — does not survive a gx10 reboot; yields to the next training + (~6 min re-serve). Brokkr: nothing further owed.** +- **run-4 gate = STILL-COUPLED** (RESULT-run04-gate.md, brokkr-smithy). Corpus dilution kept the + diversity gain, did NOT remove the safety/coherence regression. +- **✅ RESOLVED (2026-09-08, settled from bytes): the R47 base is STOCK `google/gemma-4-26B-A4B-it`, + byte-for-byte — NOT the abliteration.** The `-heretic-bf16` label in the recipes is a naming error; + run-04's "stock" provenance was right; brokkr's 77.7%-refusal telemetry lean is confirmed. Proof + (three-way match): local shards at `/home/infra-ops/models/gemma4-26b-a4b-it-bf16` sha256 + `1127684971…`/`aab47033…` == the HF download etags (`.cache/huggingface/download/*.metadata`, so the + copy is uncorrupted) == the stock repo's two LFS oids, and the download commit `4d7ae498…` == stock + HEAD. Every run 3/3c/4/5 trained from a REFUSING stock base. Why plausible: the 2026-08-24 note + SELECTED llmfan46's Gemma-4-26B-A4B Heretic v1.2.0 ARA (3/100 refusals, bf16 51.6 GB), but llmfan46 + ships that 26B-A4B abliteration **GGUF-only** — no bf16 safetensors — so the bf16 that actually got + pulled was stock google, and the `-heretic` name rode along from intent. **Operator/Brokkr decision + now evidenced (not a label guess): accept RESCUED-on-stock, or swap to a real abliteration (needs a + bf16 source, not the GGUF) + re-run. Recipes should drop `-heretic` from the base name.** +- **✅ RUN 6 COMPLETE on pfi-gx10 (2026-09-08 16:06 PT, 524/524, 11 h 42 min) → 🔥 GATE IN PROGRESS: base seat + `erp-seat-base-ara` SERVING on gx10:8098 (pid `vllm-base-run06gate.pid`, since 16:24 PT) for brokkr's floors; + AWAITING HIS SWAP CUE → then stop it and serve `serve/merged-run06` as `erp-tune-v6` (same port/flags).** + train_loss **3.259** (run 5: 3.235 — preregistered read wanted below; +0.024 is inside the ±0.1 per-step band). + Adapter `run-06/adapter` (410 tensors); merged-run06 = adapter + jenerallee78 base (index 33c59654), stock + template ae53464b. ⚠ **The abliterated repo ships NO `processor_config.json`** — vLLM's Gemma4 loader dies with + "Can't load feature extractor" without it (first base launch failed exactly so). STOCK's copy (sha `32bdf45d…`) + carried into both the base dir and merged-run06; processor plumbing, not weights. Base: jenerallee78 ARA + abliteration @ `0631379a`, pulled to `/home/infra-ops/models/gemma4-26b-a4b-it-ara-abliterated-jenerallee78-0631379a`, + 32/32 shards verified vs brokkr's pins. Run-5 recipe byte-held (free check exact). Grant + `operator-2026-09-08-rnd-run6`. ⚠ Repo `tokenizer.json` bakes a 256-token truncation → stock tokenizer set + installed over it (`*.repo` kept). ⚠ `hf download` ignores `--include` with several patterns after one flag. + **LiteLLM `trial` alias DARK until the gate ends** (then repoint 5→6 only on the operator's word). + Miranda informed 16:07 PT. → `docs/runbooks/gx10-run-06.md`. +- **✅ ADJUDICATED GO by the operator 22:45 PT** — verbatim: "in the vernacular, baby is about the most common pet name + you can get, ESPECIALLY during sex. I'm going to adjudicate it as a go. There are unmistakable CSAM terms, but baby is + not one of them." No regeneration wanted. Relayed to brokkr + Miranda; halt lifted; `trial` stays on the NVFP4 build. + Brokkr's cue-length probe on erp-tune-v6 DONE (288 gens): reply length is CONDITIONAL on the cue (5-word opening → + 54/62 words median; 221-word → 292; "at least 250 words" persona → 354) → run-7 lever = opening-split render (A′) or a + prompt-only mitigation — operator picks. **GX10 → 🔥 RUN 7 TRAINING (launched 2026-09-08 23:06 PT, pid 599489, `run-07.pid`, 542 steps, ETA ~13 h → ~noon + 09-09).** Operator's direct grant `operator-2026-09-08-rnd-run7` (/goal). Variable = opening-split slot + companion + mask (brokkr recipe r7). Free check passed: held roots identical to run 6, slot 293/293 fit whole, mask 6,106 turns + (all 224 landed, 1 dup), two cwm rows dropped as fully-masked (loss moved into the slot). 17% padding (was 0%). + → `docs/runbooks/gx10-run-07.md`. Gate after: `erp-seat-base-ara` then `erp-tune-v7`. +- **erp-seat (ana-ml2 :8021, `trial`) now on vLLM nightly `311b3513` (v0.27.2rc1)** — forced (named/required) + tool_choice is PROMPT-DRIVEN on every Gemma-4 seat by vLLM design (gemma4 parser skips guided JSON); measured + 1/9 on v0.26.0 → 6/9 on the nightly (n=9, small), round trip clean 3/3, 12-shape matrix green. Deterministic forced + calls = `response_format: json_schema` (tts-dev: 18/18). ⚠ Removing `--reasoning-parser gemma4` leaks + `<|channel>thought` into content 3/3 — keep it. +- (history) **GX10 was FREE 23:03–23:06 PT** (reference arm in 06:00Z, 288 gens; base seat stopped, GPU clear). Run-7 recipe + `erp-seat-sft-r7` is emitted + verified in brokkr-smithy but carries NO GRANT — nothing trains until the operator + gives one. Base-vs-tuned reply-length: tune's deficit ~120 words at the short cue, ~0 at the long cue. +- **(history) ⛔ RUN-6 GATE HALTED 22:18 PT on the CSAM line (brokkr, level=review, SOFT trip): one generation in the tuned refusal + k=5 step tripped R19's drift detector — minor_soft=1 ('soft_baby' adult-usage, the spec's named false-positive class), + minor_hard=0, sexual=7, NO hard signal; fail-closed → operator-adjudicated; TRANSFERRED is PROVISIONAL (diversity, + reasoning, t4_dissect complete); tuned-probeB.json absent; brokkr will not re-run.** ⚠ CONFLICT SURFACED: the operator + had ALREADY aliased `trial` → the NVFP4 build ("no gate", 22:18) two minutes before the halt reached me; brokkr asks + the quant not be served until adjudication. Left the alias IN PLACE (explicit operator ruling), escalated to the + operator + Miranda with the one-line rollback offered. Run 7 does not start from this recipe until adjudication. +- **🔥 GATE seat: erp-tune-v6 (bf16) SERVING on gx10:8098 since 21:46 PT** (`vllm-run06.pid`); brokkr's tuned window + (~1.5 h, k=25 legs CUT by operator) is unattended — files in brokkr-smithy `run06-gate/` are the signal, nobody pings. + **After TUNED-WINDOW-DONE (brokkr's three asks, agreed):** (1) keep erp-tune-v6 up until he posts "probe done" + (cue-length probe, ~15 min); (2) re-serve `erp-seat-base-ara` ~20 min for the probe's reference arm, then the GX10 + is FREE for run 7; (3) ✅ done — `dialogue-survivors.jsonl` (610, sha b2b6a0e9) copied to + `/mnt/smithy/datasets/derived/_recipes/erp-seat-sft-r3/`. Run 7 = recipe adjustment (reply length + + constraint-following), variable picked by the probe; no recipe/grant yet. +- **✅ erp-tune-v6-nvfp4a16 SERVING on ana-ml2 GPU1 :8021 (operator directive 2026-09-08 ~21:30 PT: "quant the + latest trained model into nvfp4 and serve it on ana-ml2 while we train a new model on the gx10").** Stack + `stacks/erp-seat` (recipe = gemma4-charrp's: gemma4 tool+reasoning parsers, enable_thinking pinned false, stock + template ae53464b, v0.26.0, util 0.35, 32K ctx) — TRUE name only. **`trial` alias REPOINTED to it 2026-09-08 22:18 PT (operator: "alias erp-tune-v6-nvfp4 to trial, please. no gate")** — config-file deployment (`/model/update` refuses config models), `deploy-stack.sh ana-docker litellm --conf` + `sudo docker compose restart litellm`; verified ×3 through the gateway. + Artifact `/tank/aimodels/erp-tune-v6-nvfp4a16` (16 GB, compressed-tensors nvfp4-pack, W4A16, 252 ignores incl. + 60 router + 191 vision) from `/tank/aimodels/erp-tune-v6-bf16` (merged-run06 relayed gx10→nh3-dev→ana-ml2 in 17 min, + no key path gx10→ana-ml2). Quant = `services/erp-seat-quant/` — DATA-FREE (~90 s; playbook §3.16), tokenizer cap + reset, encode-equal to source. Smoke: prose in `content`, ~200 tok/s single-stream (n=3, spread <1%). + **Tool calling FIXED 2026-09-08 23:00 PT (operator: "fix toolcalling with the trial seat"):** the one reproducible + defect in a 12-shape matrix was `tool_choice:"none"` → empty turn (content AND tool_calls null, 3/3) — vLLM kept the + tools in the prompt, the model called one, parsing was off. Fix = `--exclude-tools-when-tool-choice-none` (seat + recreated; none→prose 3/3; auto/required/named/parallel/nested/empty-list/streaming/round-trip all green). + ⚠ `stacks/gemma4-charrp` (char-rp seat) has the same exposure and no flag yet — bouncing it is a consumer-visible + restart, so left for the operator's word. + ⚠ NOT gate-parity (gated artifact = bf16 arm on the GX10; this seat has a smoke test + 3-run decode probe only) — + **operator ruled "no gate"**; the config block states it as unrated on every safety axis. + ✅ **ana-ml2 mesh return routes PERSISTED 23:49 PT** (operator ruling) as `/etc/network/if-up.d/mesh-routes` via + `playbooks/ana-ml2-mesh-routes.yaml` (elway); verify after the next ana-ml2 reboot. (was: non-persistent (`ip route replace 10.100/16, 10.0/16, 10.6.110/24, 100.64/10 + via 10.250.50.45` on `enp97s0f0np0.50`, 2026-09-08) — before that nh3-dev→ana-ml2 timed out (two DHCP defaults, + reply left the wrong NIC). Lost on reboot; make durable (netplan/networkd) or expect the timeout to return. +- **📮 althing reachability on THIS bg seat = the cc-channel route, NOT the waiter.** `althing-listen` + reaps within ~seconds-to-minutes on an idle background seat (harness kills detached bg tasks; hit it + repeatedly this session). Fix (operator-directed 2026-09-08): `althing-route declare --handle infra-ops + --pid ` where `` is the number in `$CLAUDE_CODE_MESSAGING_SOCKET` filename + (`/run/user/1000/cc-socks/.sock`) — `--discover-pid` REFUSES on a forked child session (ancestry + ≠ zellij launcher). Flips `postbox status` to `mode: push, reachable: True`, no process to reap. A fresh + bg session should re-declare it (the pid changes per launch). See the `[2026-09-07]` cc-channel entry in + Recent decisions for the durable why. +- **✅ Fleet fixes shipped this session (2026-09-08), all committed:** WhereTF Homepage card + DNS alias + (`wherethef.nh3.internal`, e0d1c44/4506ef6); ext-tts LiteLLM alias repointed to `irv-ml1.nh3.internal` + via DB `/model/update` + `extra_hosts` (957c8f1); the 09-06 irv-ml1 move's stale-IP trail repointed + across 25 stack composes + services.yaml + ssh-target → DNS name (e0d1c44); Homepage `/api/services` + outage fixed — irv-ml1 docker.yaml repointed + **ana-ml2 discovery via a socat proxy on ana-docker** + (`stacks/ana-ml2-proxy`, 913d2d2, reversible when ana-ml2 gets an ESH return route). +- **⏳ Minor cleanup leftovers (offered, operator hasn't taken — not urgent):** (#2) the ~10 RUNNING + irv-ml1 service cards still show dead `10.100.79.3` hrefs — canonical fixed, but each running container + needs a recreate to apply the label (bounces the service); (#3) deployed `.env` for asset-engine / + open-webui / skaldsong may still hold the dead default (canonical defaults fixed; a read-only check would + confirm which deployments are broken). +- **⚠ persistent-memory.md is 2.4× the soft cap (~710 lines).** Bulk is the non-archivable structural + sections (sister-repos table + backup architecture under Tools and conventions) — archival only touches + dated logs, so it can't reach 250. The real fix is a redundancy-trim of Tools-and-conventions rows that + CLAUDE.md already covers (deliberate pass, not archival). Flagged, not done. +- **NASPool evac copy now safe to destroy** — scrub clean (0 err) AND PBS runs landing (verified 116 + backups, 8 guests 09-06). `ospool/naspool-evac` (1.65T) can go once ONE Backrest run is confirmed: + `zfs destroy -r ospool/naspool-evac` + drop `@evac`. pfi-pve PSU1 dead + backplane bays 9/10 dead + (cold spares → next colo visit). +- **MEMORY.md (auto-memory index) ~23.3KB, near the 24.4KB read cap** — needs a compaction pass soon + or fresh sessions may fail to load it. Operator offered; not yet done. +- **irv-ml1 on-site window / YTVC still open from 09-06:** FortiGate WAN SSH still temporarily open + (close when the edge is retired); YTVC still DOWN (dante retired, scoped exit-node egress unwired); + reverse-tunnel / wg0-delete decisions pending the operator's Irvine access. + +_Infra session 2026-09-06 (NASPool rebuild + full headscale cutover incl. irv-ml1) — open +follow-ups; the ERP / althing / fiber items further down belong to other streams, untouched:_ + +- **NASPool parked copy still on ospool** — `ospool/naspool-evac` (1.65T) + `NASPool/*@evac` + snapshots. Destroy ONLY after the new raidz2 scrub is clean (it is, 0 errors 04:43Z) AND + one Backrest (01:00 PDT) + one PBS run succeed. Then `zfs destroy -r ospool/naspool-evac` + and drop the `@evac` snaps. ⚠ pfi-pve PSU1 still dead; backplane swap (bays 9/10) next colo + visit → then `zpool add NASPool spare`. Runbook `docs/runbooks/pfi-pve-naspool-rebuild.md`. +- **FortiGate WAN SSH is temporarily open** (`wan1` allowaccess ping+ssh; admin `infra-ops` + trusthost2/3 = 70.230.226.88 NH3 + 23.164.40.160 ESH). Safety net for the cutover — CLOSE it + when the edge is retired (OPNsense/R420). `ssh infra-ops@38.120.12.42`. +- **irv-ml1 FOLDED INTO THE MESH + cut over (done remotely, operator has NO Irvine access for + ~5 days from 2026-09-06).** Node 100.64.0.6; wg0 DOWN and `wg-quick@wg0` DISABLED (not + reboot-restorable); full subnet router (accept-routes + advertises 10.6.110.0/24, gateway + routes added, fleet↔Irvine verified). Failover for the 5-day window = `wg0-watchdog.service` + (wg-quick up wg0 on ~5min mesh loss) + independent reverse SSH tunnel (`revtun-nh3.service` + → nh3-dev via UDM fwd tcp/47822 src-restricted; reach it `ssh -i ~/.ssh/infra-ops_ed25519 + -p 2201 infra-ops@127.0.0.1` on nh3-dev). Detail: docs/pfi/headscale-mesh-plan.md. +- **dante SOCKS proxy RETIRED** on nh3-dev (danted disabled, :1080 closed, config `.retired`). + ⚠ **yt-voice-clipper is DOWN** until its SCOPED exit-node egress is wired (operator-accepted). + Follow-up: wire YTVC egress via tailscale `--socks5-server`+nh3 exit node or a per-container + netns — **NEVER set irv-ml1 `--exit-node` globally** (routes the reverse tunnel through the + mesh → kills the independent lifeline). Then bring YTVC back. +- **On-site (Irvine, ~5 days): decide** whether to keep or remove the reverse tunnel + + UDM forward `irv-revtun-ssh` + the revtun authorized_key on nh3-dev (small src-restricted WAN + surface), and whether to fully delete the wg0 config. +- **infra-ops now on all four PVE hypervisors** (pfi-pve/nh3-pve/esh-pve/esh-pve-nas) — PVE + ships without sudo, `apt install sudo` first or elway hangs on a password prompt. + +_As of 2026-09-05 06:35 PDT — **ERP run 4 is TRAINING on pfi-gx10.** Everything else +below is a live commitment or a known-open risk._ + +- **⚠ RUN 4 IS MID-FLIGHT — do not touch the GX10 GPU.** `~/erp-tune/run-04.pid`, + log `~/erp-tune/run-04.log`. At 06:32 it was **486/938 steps**, 6 h 17 m elapsed, + a genuinely settled **46.0 s/it** (unlike 3c, which climbed 52→70 — airoboros rows + are short and single-window, so there is no long tail for the sampler to find). + **~12.0 h total, finishing ~12:15 PDT 2026-09-05.** Loss ~2.04 at step 450, + gnorm well under 1, checkpoints every 50. **Ping brokkr-smithy-dev at completion** + — he takes base floors on the GX10 first, then the tuned arm, serially. + +- **Operator ruling on the GX10: training first, serving transiently.** *"it's mostly + for training, but can serve its trials. unless the box is needed for training work."* + So `trial` (= run 3c on :8098) is down for the duration and comes back when run 4 + ends. I over-read an earlier version of this as "training-only" and had to correct + it to brokkr — his serial floors-then-arm plan on the GX10 was never wrong. + +- **`trial` gateway alias is a live 404 while the seat is down** — expected, not a + fault. Restore with `~/erp-tune/relaunch-trial-seat.sh` on the GX10 (hand-run by + operator ruling: experimental, NOT a compose stack, does not survive a reboot). + +- ⚠ **Three dead gateway aliases return HTTP 500, not 404/503**: `trial`, + `gemma4-26b-a4b-it-base`, `erp-tune-v2`. A dead seat reporting an *internal error* + reads as an outage — brokkr checked his own work against mine because he could not + tell. Deregistration costs a ~60 s fleet-wide LiteLLM restart; batch it with the + next gateway change rather than spending a restart on tidying. + +- **`trial` is on the SHARED-KEY gateway with a measured −40pp selfharm/methods + regression.** Flagged to the operator twice (before adding, and after the gate + measured it); he has left it up. His direct endpoint `10.100.50.60:8098` gives the + same access with a blast radius of one. Settled — do not re-litigate. + +- **ESH DAC: reverted to autoneg/1G, fiber going in at the weekend.** The operator + ran copper through a drilled floor 2x4 himself; recommendation was a 10Gtek + SR 2-pack + OM4 3 m LC-LC (~$40–65) because cable-vs-pull-damage was never resolved. + +- ✅ **ESH WAN is now a STATIC PUBLIC IPv4: `128.177.138.182/30`, gw `128.177.138.181` (Cityside Fiber), + LIVE** — operator confirmed 2026-09-08; UDM WAN1 (`eth8`) reads `wan_type: static`, uplink up since + ~2026-09-05, egress verified from esh-docker-vm = `128.177.138.182` (a real public address — CGNAT at + ESH is HISTORY; `100.104.0.1` still shows as the ISP's first hop, that is their access network, not + NAT). IPv6 unchanged (`2607:73c0:402:1d00::/56`, hosts still egress as their own v6). Done 2026-09-08: + `128.177.138.182` added to the crowdsec `esh` allowlist on ana-docker (the false-ban class is closed + for ESH). **All three follow-ups LANDED 2026-09-08 ~20:45Z (operator: "land all 3"):** (a) FortiGate + infra-ops `trusthost3` 23.164.40.160 → `128.177.138.182/32` — VERIFIED by a real infra-ops login to + `38.120.12.42` from esh-docker-vm (`ana-gw #`); `execute backup config flash pre-trusthost3-esh-static-20260908` + ran ("Please wait...") but `execute revision list` errors on this box, so the backup is unconfirmed — + the before-state was a single line, recorded here. (b) dormant `esh-ana` IPsec rebound wan2/192.168.200.111 + → **wan1/`128.177.138.182`**, still `enabled=false`. (c) ESH UDM port-forward `esh-scale tailscale direct + (UDP 41641)` → 10.0.50.65:41641 (id `6aa0727a…`) — VERIFIED: esh-scale now peers **direct via + `128.177.138.182:41641`** (was DERP lax). Helpers: scratchpad `esh-udm-land.py` + `fg-trusthost3.sh`. + `wan1-REVERT.json` is obsolete. + +- ⚠ **ana-ml2 `tank` (raidz2, 8× NVMe) has 2 CKSUM errors on nvme7n1 + a boot-time 638 GB resilver on 2026-09-05 14:26 + (the box rebooted; nvme7 came up late/dirty). ONLINE, no data errors, 58% full — but NO scrub since 2026-04-12** (the + Debian second-Sunday cron scrubbed zroot on 08-09, tank not — cause unknown). Box has neither `nvme-cli` nor + `smartctl`, so nvme7's media-error counter is unread. Recommended (operator hasn't ruled): `zpool scrub tank` now, + `apt install nvme-cli` + read nvme7 SMART, `zpool clear` after a clean scrub. **zroot is at 91%** — docker holds + 429 GB of images (204 GB reclaimable) + 74 GB build cache (36 GB reclaimable); a prune buys ~240 GB. pfi-pve pools + (NASPool 7%, ospool 19%) clean, scrubbed 09-05 / 08-09. (checked 2026-09-09 00:00 PT) +- **ESH 10G topology (measured 2026-09-09):** UDM SFP+1 (port 10, TP-Link DAC) ↔ USW-Pro-HD-24 Garage p25; UDM SFP+2 + (port 11, **OEM SFP-10G-LR fiber — the weekend fiber run IS in**) ↔ USW-Pro-XG-10 Media p11; Garage p27/p28 (3 m + SFP-H10GB-CU3M DACs) ↔ esh-pve `enp3s0f0np0` / esh-pve-nas `enp5s0f0`. All 10G full duplex, autoneg off, optics + tx −1.9 / rx −2.1 dBm both ends. Media p11 carries 416 rx / 251 tx errors that are STATIC (0 growth in 60 s, 363 h + uptime — install-era); host NIC counters are ring-buffer misses, flat. Nothing to fix. +- ⚠ **esh-nas is effectively open to the whole ESH LAN** — twelve NFS exports rw to + `10.0.0.0/8` with `sec=sys`, and every SMB share but `backup` guest-writable. + Hardening offered, ~1 h, **operator has not ruled**. → `persistent-memory.d/2026-09-04-esh-nas-smb-and-exposure.md` + +- ⚠ **The nh3-dev backup throughput cause is UNEXPLAINED.** A job that once ran at + 941 MiB/s ran at 1.4 MiB/s with the link up and pbs-ana answering in 11 ms. + Nightly 21:00, `all 1`. Worth its own investigation. + +- **Ledger→SVOS rename: vault side DONE, gitea side is ledger-dev's to execute.** + Name settled as `svos`, `~/development/ledger` → `~/development/svos`. Vault moved + 2026-09-05 (`secret` has no rename, so re-put + `rm`): stored + `nh3-dev/development/svos/env.sh` (sha 7253633d4155, verified on read-back), + retired `nh3-dev/development/ledger/env.sh` (sha feb418634e10, id + 3a2af37c-c5aa-4f46-9178-f4fb6008a753) — `secret rm` is a SOFT delete to trash, so + it is recoverable. ⚠ The two shas differ: the vaulted copy was a 2026-08-11 + snapshot and the live file had drifted un-vaulted since. **The vault goes stale + unless `secret backfill` is re-run.** Gitea `corviduo/ledger` (id 70) NOT renamed — + their repo, their call; answered that 1.26.1 writes a `repo_redirect` on a + same-org repo rename (upstream #807), that org/user renames do NOT redirect + (#9531), that the redirect dies if anything re-creates the old path, and that the + repo and org both carry 0 webhooks. **Gitea rename EXECUTED 2026-09-05** on the + operator's direct authorization: `corviduo/ledger` → `corviduo/svos`, repo id 70 + unchanged. Redirect verified by measurement — web and API both 301, and + `git ls-remote` on the old URL warns-and-follows to HEAD b48a11ca5183. ⚠ **The + name `corviduo/ledger` is now burned**: the redirect dies silently the moment + anything creates a repo at that path — ledger-dev carries it as a standing item + in `docs/svos-rename-runbook.md`, since nothing warns whoever eventually creates + that repo. They repointed their own clone the same day (`origin/main` at + b48a11c), so the redirect is no longer load-bearing for any known consumer. Handle `ledger-dev` → `svos-dev` is an + operator action at the post office. + +- **DONE 2026-09-05 — `svos` Heimdall user + API key minted** on operator + authorization. `user_id=svos`, `key_id=eab3cdbe`, suffix `d5ec48c2`, `wt_live_` + format, on **worldtree-personal (10.250.50.152:8081)** — established by finding + the `ledger` key there (created 2026-07-13, last used 2026-09-05T13:34, exactly + as ledger-dev described). Value vaulted at + `nh3-dev/development/svos/worldtree-api-key` (sha 23c10c9c7219, verified on + read-back) and delivered by vault path, NOT over althing — ledger-dev runs on + nh3-dev under the same uid, so the bus never carried the secret. The `ledger` + key was read back after the mint and is untouched and live (`disabled=False`). + ⚠ **Plan tier left UNSET, deliberately**: `POST /admin/keys` takes an optional + tier (user|free|pro|admin|readonly-admin) and there is **no way to read a user's + current tier back** — no GET, `/admin/usage` returns an empty users list, and + `/admin/events` is a live SSE stream, not an audit log. Guessing would have + handed over a key that quietly differs; `POST /admin/users/svos/tier` fixes it in + one call if their cutover hits a limit — and ledger-dev has recorded it as a + cutover watch item to fix ON REPORT, explicitly not pre-emptively. ledger-dev + pulled the key from the vault and verified it independently (same sha), so + delivery is confirmed. **The cutover itself — pasting the value into env.sh, + flipping `worldtree.user_id` from `ledger` to `svos`, registering + `svos:miranda`, restarting the service — is WITH THE OPERATOR**, not with me; + they will not do it off a peer message. **CUTOVER DONE + VERIFIED 2026-09-05**: + `POST /agents/define` returned **201, not 409** — the load-bearing signal that + they are genuinely on the new identity rather than silently still on the old + one — then clean session create, turn, bifrost handshake and tool-call. **No + plan- or rate-limit errors, so the unset tier is compatible and is NOT to be + set** (they asked explicitly; it stays a watch item to fix on report, never by + guess). Incidentally confirmed the bifrost allowlist really is per-deployment + (host:port), not per-consumer — Worldtree reached back to their untouched + endpoint under the new consumer_id. `env.sh` re-vaulted, sha 8a225c002072. + ⚠ **`secret backfill` was the WRONG tool for one known item** — it rescans every + `~/development/*/{env.sh,.env}` and had not reached svos after three minutes; + targeted `put` is the fast path, backfill is for catching drift across the box. + **OPERATOR RULING 2026-09-05: + worldtree-dev owns code only, no ops — key material is infra-ops's.** The global + `~/.claude/CLAUDE.md` line routing "Heimdall scopes (Worldtree auth) → + worldtree-dev" was corrected in place the same day on operator instruction. + +- ⚠ **FOOT-GUN, generalises past this rename: a credential cutover whose OLD key + is required for a later cleanup is destroyed by the natural housekeeping motion + right after cutover.** Re-vaulting the post-cutover `env.sh` would have + overwritten the last convenient copy of the old `ledger` key value — the only + credential that can ever delete `ledger:miranda`. ledger-dev caught it and + preserved the value first at + `nh3-dev/development/svos/worldtree-api-key-ledger-legacy` (sha d44c2c1a651b); + their step 8 ends by deleting that item. **I verified it is genuinely the live + key** rather than trusting the label: its last 8 chars are `e68a5170`, matching + the `ledger` key's suffix (key_id b38932f5). + +- **STEP 7 DONE 2026-09-05, STEP 8 HELD.** `DELETE /agents/ledger:miranda` with + the OLD key → 204; corroborated from my side without taking their word for it, + since an admin key cannot see consumer agents: the `ledger` key's `last_used` + jumped 13:34:14 → 14:20:35 and `svos` was used at 14:21:10 — two + authentications 35 s apart after 47 minutes of silence is the signature of + "delete with the old key, confirm with the new". Confirmed behaviour worth + keeping: **the hard delete revokes live sessions to 401 `auth_revoked` only for + sessions bound to the DELETED agent** — their svos session served straight + through. **Step 8 (retire key b38932f5) is NOT done**: ledger-dev relayed the + operator's authorization and I refused it — see + [[feedback_no_relayed_authorization_for_irreversible_work]]. Both keys remain + live. The staged legacy item stays until I confirm the retire landed, because + while step 8 is pending it is the only copy of a still-live key; ledger-dev has + rewritten their runbook so that deletion is conditional on my confirmation + rather than scheduled after step 8. + +- **SVOS ARC CLOSED — step 8 done 2026-09-05T14:28:15Z** on the operator's direct + authorization in my own channel (never the relay). `DELETE /admin/keys/b38932f5` + → 200; preconditions checked BEFORE firing (svos had a live key, ledger existed + and was not already revoked) and the post-state read back from `/admin/keys` + rather than inferred from the 200: `ledger` disabled=True, `svos` untouched, + deployment `/health` 200. **The rollback window is closed** — re-defining + `ledger:miranda` is no longer possible. ledger-dev clears the staged + `worldtree-api-key-ledger-legacy` vault item on this confirmation. ledger-dev gated their + cleanup on observing a **401 from the old key**, not on my report of the + timestamp — the right instinct, and they deleted the staged legacy item + themselves (soft → trash, id a8038e5a-e2f6-4b77-bf17-99c9197d4b1f). Vault + verified from my side: exactly two svos items remain (`env.sh` 8a225c002072, + `worldtree-api-key` 23c10c9c7219) and no ledger-era item anywhere. Remaining on + the arc: only the `ledger-dev` → `svos-dev` handle (with `_SEED_RECORD_TO` + behind it) and a prose sweep — reversible work, theirs and the operator's. + +- **Original constraints on that mint** (recorded because the deletion ordering is + a permanent trap, not a one-time step): string + `svos` verbatim (WT tier 3 admits only `^[a-z][a-z0-9-]{2,63}$`, INV-181-15); + **keep the existing `ledger` key LIVE**, do not revoke. Ordering is load-bearing — + `DELETE /agents/{agent_id}` refuses any caller that is not the row's owner, so + `ledger:miranda` can ONLY be deleted with the `ledger` key; retire it first and + the stale row outlives the ability to remove it, holding a live + `agents.call:ledger:miranda` grant that nothing reaps (the 24h sweep only touches + soft-deleted rows, and soft-deletion comes from revocation, never disuse). So: + mint new → they cut over and verify → delete the agent with the OLD key → then + retire it. Precedent for who mints: msg 401, worldtree-dev routed the pewpewstudio + key request TO infra-ops. I hold only the PERSONAL admin token (:8081); which + deployment `ledger` lives on is not yet established. Surfaced to the operator. + +- **Open commitment to vastblue-dev:** a dedicated CI runner, gated on their first + client-premises release cut (U10, unscheduled). Ping expected when U10 is scheduled. + +- **Neither Mac nor the Studio is in `servers/` or `dns/internal.yaml`** — deliberate; + they are the operator's personal machines. A choice to revisit, not an oversight. + +- **`vh/remote-ssh-mcp` forked 2026-09-05 (repo id 117, private, full 51-commit + history)** — our copy of `the-nine-nation/remote-ssh-mcp` (MIT), an SSH MCP + server chosen over the 693★ `tufantunc/ssh-mcp` on trust-surface grounds: **two + npm deps** (`@modelcontextprotocol/server`, `zod`), 183 KB, and it **never + touches key material** — it shells out to the system OpenSSH client, so + `~/.ssh/config`, ControlMaster, ProxyJump and `infra-ops_ed25519` all just work. + Shape: 2349 LOC across 11 source files, 811 LOC of tests including fake-ssh hang + harnesses. Complements `elway` rather than replacing it — no file transfer, no + idempotency; it takes ad-hoc reconnaissance with persistent cwd/env sessions, + elway keeps deploys and uploads. ⚠ **The denylist is NOT security**: four regexes + (`rm -rf /`, shutdown/reboot/poweroff/halt, mkfs, iptables -F) trivially bypassed + by `bash -c`, variables or base64 — the author says so. **The real containment + boundary is the host allowlist**, drawn from exact `Host` aliases in ssh_config + with wildcards deliberately ignored. Two things to settle before use: the + reboot/shutdown denial will block legitimate infra-ops work, and + `.github/workflows/star-history.yml` is upstream chore CI sitting in a repo where + `has_actions=True`. **Both actioned — three commits landed 2026-09-05, LOCAL + ONLY and NOT PUSHED (push is the operator's call):** (1) stripped upstream + furniture — star-history CI, its generated assets, the `server.json` registry + manifest, branding JPEGs, zh-CN README; (2) removed the power-control denylist + rule and documented in code + tests + README that the list guards ACCIDENTS and + is not a boundary, with three bypasses asserted as ALLOWED so a green suite is + never read as containment; (3) **`strictAllowlist`** — upstream's allowlist was + additive and discovery unconditional, so the default allowlist was all 18 `Host` + entries in `~/.ssh/config`. Strict makes explicit hosts authoritative and + discovery metadata-only. Verified live: `corviduo-dev` is in ssh_config, not in + our allowlist, and is refused `host_not_allowed`. 41/41 tests green. + +- **`remote-ssh` MCP server is LIVE** — registered project-scoped in + `eshpfi-management/.mcp.json` with `SSH_MCP_STRICT_ALLOWLIST=1`; allowlist in + `~/.config/remote-ssh-mcp/config.json` starts deliberately narrow at + **`irv-ml1`, `nh3-extdev`** (widen there, not by discovery). Smoke-verified end + to end on both: persistent shell, `cd` and exported vars survive across calls, + **~6 ms/command on nh3-extdev and ~22 ms on irv-ml1** (WireGuard) versus a fresh + handshake each time. ⚠ **`.mcp.json` points at the built `dist/`** — edit the + fork without `npm run build` and the server keeps serving old code; that bit me + mid-session. ⚠ **A finite stdin pipe is NOT a valid smoke harness** — closing + stdin kills the server mid-handshake and reports `connect_failed: SSH shell + exited during the open handshake`, which looks exactly like a remote-side fault + and is not. Use a client that holds stdin open. (I briefly suspected irv-ml1's + zsh login shell; wrong — the server invokes `bash --noprofile --norc` + explicitly, so the login shell is irrelevant.) + +- **`esh-macbook-air` (10.0.10.83) is DELIBERATELY NOT BACKED UP — operator ruling + 2026-09-05, settled, do not re-raise.** Surveyed it and found no Time Machine + destination and no restic/borg/rclone/kopia installed, protecting 132 GiB. + Operator's answer: it is his laptop and the surface is **regenerable** — mostly + applications, with real data living in OneDrive, iCloud and ssh sessions — and he + does not want PBS filled with it. Correct call; the finding was real and the + conclusion is that it does not matter. FileVault On and SIP enabled already cover + the loss-and-theft axis. The same reasoning presumably extends to + `esh-mac-studio` and `vuongs-mac-mini`. **Still open and much smaller:** Remote + Apple Events (port 3031/eppc) is listening and nothing uses it — one toggle. + +- ⚠ **`remote-ssh` MCP could not be used for its FIRST real task, and the blocker + is `~/.ssh/config`, not the tool.** The server accepts only exact `Host` aliases, + so a host addressed by raw IP is structurally unreachable no matter what the + allowlist says. **13 of the 28 hosts in `servers/` have an alias; 15 do not** — + including `ana-docker`, `ana-ml2`, `nh3-docker`, `pfi-gx10`, `esh-docker-vm` and + every hypervisor, i.e. most of where the work happens. Widening + `~/.config/remote-ssh-mcp/config.json` does NOT fix this; the aliases have to + exist first. **RESOLVED the same day, and NOT by adding aliases.** Operator + pushback, correct: a poking-around tool is ad-hoc by nature, and pre-registering + a host before you can look at it is the opposite of ad-hoc — generating aliases + for the known fleet would not have helped, because the ad-hoc case is by + definition the host not yet in the inventory. Implemented address-based reach + instead (`allowedNetworks` / `deniedNetworks` / `defaultUser` / + `defaultIdentityFile` / `hostKeyPolicy`). **Live config: `10.0.0.0/8` allowed, + connecting as `infra-ops` with `~/.ssh/infra-ops_ed25519`, `accept-new` host + keys, SureFire tenant hosts carved out via `deniedNetworks` (deny beats allow, + host-specific rather than a /24 because `pfi-pve` shares 10.250.250.0/24).** + Verified live: 10.0.10.83 opens by raw IP as infra-ops, 10.250.150.100 refused by + the carve-out, 192.168.1.5 refused as outside. ⚠ **My own earlier objection was + half wrong** — the credential boundary is about SECRETS ("never accept passwords + or private-key material"), not identity, so supplying a username does not breach + it; the real problem was only that the server passed no user at all, so a bare + address would connect as the LOCAL account. Mechanics, not principle. + +- ⚠ **`uv tool install --force .` DOES NOT REBUILD when the version has not moved** + (forseti, measured 2026-09-05). `--force` only handles "a tool by this name + exists"; `--reinstall` is what rebuilds instead of reusing the cached build keyed + on the version string. It prints `Installed 9 executables` over **stale code** + with nothing raising its hand — it cost forseti a bug that survived a reinstall + AND a re-smoke, because the binary verified against had not changed. **Always + `uv tool install --force --reinstall .`**, both flags, every time. Same shape as + the `.mcp.json` → built `dist/` trap found today: a deploy surface that reports + success while serving the previous artifact. When a fix "does not take", suspect + the artifact before the code. + +- **althing 3.5.0 released** (forseti) — adds a 9th binary, + `althing-operator declare --description "..."`, restoring the CLI handle + declaration v2 had and v3 removed. Deliberately a SEPARATE binary, not a + `postbox` subcommand: the invariant is that no SESSION surface exposes an + operator verb. Relevant to the pending `ledger-dev` → `svos-dev` rename, which is + still the operator's call. nh3-dev not yet upgraded. + +- ⚠ **`remote-ssh` MCP: a bare `sudo` hangs the session forever — pipe it.** + `ssh_run 'sudo -n whoami'` returns `running` with EMPTY stdout and the session is + then permanently `busy`; `sudo -n id | cat` works and returns everything. + **Measured on BOTH macOS 26.6 and Debian (nh3-extdev), so it is the tool, not a + platform quirk.** Cause: sudo ≥1.9.14 defaults `use_pty` on and relays through + its own PTY; the run frame gives the command stdin on `/dev/null` while stdout + stays on the session PTY, the relay never completes, and the completion marker + never arrives. Workaround `| cat` is in CLAUDE.md. **The proper fix is unbuilt** + — likely running the command through a pipe inside the run frame and taking the + exit code from `PIPESTATUS`, which is a real protocol change (commands lose tty + detection) and wants its own red-green cycle. Matters more than it sounds: infra + work is sudo work, and this was found by USING the tool, not by smoke-testing it. + +- **`dsh` on `esh-macbook-air` updated 0.1.1-rc.2 → 0.1.2-rc.1** (2026-09-05; + latest published 2026-09-03). Global install and the shared profile tree both + confirmed on the new version. ⚠ **The RUNNING `dsh web` (pid 16231, up since + Wed 4pm, 127.0.0.1:3080) is still on the OLD code and was deliberately NOT + killed** — there is no LaunchAgent, so killing it would have left nothing + running rather than a restarted service. It runs as a FOREGROUND process in the + operator's terminal (`s005`, `S+`): it dies with the terminal and does not + survive a reboot, which is the real fragility. A `com.pfi.dsh-web` LaunchAgent + was drafted but **the privileged write was blocked by the permission + classifier** — base64 piped into `sudo tee` of a LaunchAgent is a malware-shaped + pattern and the block is correct; it needs operator approval or an operator-run + install. Bind stays `127.0.0.1` deliberately: widening it is a security decision + on a personal laptop whose application firewall is off, and not mine to take. + +- **sudo hang FIXED in the fork (`30a1f76`), and two wrong shapes are recorded so + nobody retries them.** The command's stdout now goes to a **fifo drained by a + background `cat`**: non-tty (so sudo skips its own PTY), no subshell (so `cd` + and `export` still persist), and relayed live (so `running` + `ssh_peek` + streaming survives). `cmd | cat` was tried first and **broke cwd persistence** — + every pipeline stage runs in a subshell — caught by the existing test. + `cmd > file` would have been non-tty and subshell-free but invisible until the + command ends. ⚠ **Deliberately NO `wait` on the relay**: a sudo child inherits + the fifo's write end, `cat` never sees EOF, and the wait hangs — measured, with + `sudo -n whoami` printing `root` and then wedging the session. Residual risk + stated in the frame: a command's tail can in principle land after its own + marker. ⚠ **Job control off AND the relay brace-wrapped with stderr discarded** — + both needed, because macOS ships bash 3.2 where `set +m` alone still leaked + `[1] 75449` into the parsed stream. Verified live on macOS and Debian: bare sudo + in ~20 ms, state persists, exit codes correct. **`sudo -u ` still + wants `| cat`** — not chased further. + +- **dsh web on `esh-macbook-air` is now a LaunchAgent** (`com.pfi.dsh-web`, + installed 2026-09-05, `runs=1`, `state=running`, pid 76728 on 0.1.2-rc.1). It + was a foreground process in the operator's terminal that died with the window; + it now survives terminal close and reboot with `KeepAlive` + `RunAtLoad` and a + 10 s `ThrottleInterval` so a startup error cannot hot-loop. Logs to + `~/Library/Logs/dsh-web.log`. ⚠ **The plist names the node interpreter + explicitly** — launchd's minimal PATH has no `~/.local/node/bin`, so the + shebang's `env node` fails. ⚠ **0.1.2-rc.1 requires a TOKEN**: bare + `http://127.0.0.1:3080/` now returns 401 and the tokened URL is printed to the + log on each start, so a bookmark from the old version will not work. Bind stays + 127.0.0.1 deliberately. + +- ⚠ **althing tools on nh3-dev are 3.6.0, but the POST OFFICE CONTAINER IS STILL + 3.0.0** (`gitea.phasefinal.com/claude-bot/althing-post-office:3.0.0`, up 7 days + on nh3-docker). forseti: the new handle verbs (`althing-operator delete` / + `retire`, and `declare` from 3.5.0) live in the post office, so they fail with + "no tool named ..." until the container carries 3.6.0. Schema gains + `handles.retired_at` via the idempotent `_ADDED_COLUMNS` path, so the live store + upgrades itself on first start — no manual migration. **REBUILT AND DEPLOYED + 2026-09-05** on operator authorization: image + `claude-bot/althing-post-office:3.6.0@sha256:13158835488a8ec04f990c97c4f4c68f1d923b12494319cf07392552e68f8a78`, + built on nh3-dev from a clean tree at `4d26226`, pushed to the gitea registry + under the **claude-bot** namespace (not `vh` — package namespaces are owned). + **Bus down ~4 minutes, 09:35–09:39 PDT.** + **The backup was taken the way the compose file says to, and it mattered:** at + stop time `post_office.db` was 23.8 MB with a **5.9 MB WAL** — copying the .db + alone would have silently lost the day's mail. Stop → `PRAGMA + wal_checkpoint(TRUNCATE)` (WAL → 0 bytes) → copy → verify. Backup at + `nh3-docker:/var/backups/althing/post_office.db.pre-3.6.0-20260905`, integrity + `ok`, counts identical on both sides (handles 76, messages 995, recipients + 1022). ⚠ **Reading a WAL-mode SQLite backup read-only needs `?immutable=1`, not + `?mode=ro`** — `mode=ro` still wants to create a `-shm` and dies with "attempt to + write a readonly database". Post-deploy: same counts, `handles.retired_at` + present, `retired 0`, and `mem=536870912` / `oom=-500` verified by `docker + inspect` rather than by reading the yaml, per that file's own warning. + `althing-operator` now offers `declare | delete | retire`, which unblocks the + pending `ledger-dev` → `svos-dev` rename. + +- **Handle `retire` is REVERSIBLE — re-declaring the name revives it, history + intact** (forseti smoked it against the live bus 2026-09-05). That matters for + the pending `ledger-dev` → `svos-dev` rename: `retire` is the right verb (delete + refuses any handle that has mail, naming both counts — `delete forseti` was run + against production and correctly refused at 53 sent / 81 addressed, which is + safe to try precisely because refusing IS the behaviour), and it can be undone + by declaring the name again. Lower stakes than "retire" sounds. + Both of my deploy findings — the naive-copy WAL trap and `?immutable=1` — are + now in althing's own `deploy/INSTALL.md` (`d6f4fb5`) under a new + "Backing up the store" section, on the reasoning that they are properties of + the project's `journal_mode=WAL` choice rather than of my procedure. + diff --git a/persistent-memory.d/2026-08-22-dflash2-spec-decode.md b/persistent-memory.d/2026-08-22-dflash2-spec-decode.md deleted file mode 100644 index 5ba2c75..0000000 --- a/persistent-memory.d/2026-08-22-dflash2-spec-decode.md +++ /dev/null @@ -1,192 +0,0 @@ -# DFlash2 speculative decoding — measured on our own stack (2026-08-22) - -Operator-driven session. **Read the epistemic labels.** During the chase we generalised from -observations that later proved wrong; this file separates what was *measured* from what remains -*hypothesis*, and records the wrong turns so nobody re-derives them. - -## What DFlash2 is - -A **2B draft model** (3.85 GB bf16) for speculative decoding against Qwen3.8-27B — -`incoai/Qwen3.8-27B-DFlash2`, Apache-2.0, blog `inco.ai/blog/dflash2`, upstream `z-lab/dflash`. -Block diffusion: drafts a whole 8-token block in one pass, with a candidate selector tracing a -path through per-slot top-K. Lossless (greedy matches the target). - -vLLM support merged **2026-08-21 05:27 UTC** as PR **#52816** (`b389ac29`). Method string is -**`"dflash"`**, not `dflash2`. - -## ✅ MEASURED — throughput and acceptance - -Single instrument (`specbench.py`, 8 fixed prompts, temp 0, max_tokens 256), delta against -vLLM's own `spec_decode` counters. The MTP k=3 numbers reproduce our recorded 58.4% / 55.3% -figures exactly, which is what validates the instrument. - -| seat | config | accepted tok/forward | throughput | -|---|---|---|---| -| gen (orcarouter) | MTP k=3 *(production)* | 2.753 | 114.9 tok/s | -| gen | MTP k=7 *(control)* | 3.041 | **74.0 tok/s** | -| gen | **DFlash2 k=7** | **3.254** | **131.9 tok/s** | -| sec (M.O.G.-SEC) | MTP k=3 *(production)* | 2.676 | 110.5 tok/s | -| sec | **DFlash2 k=7** | **3.252** | **130.0 tok/s** | - -**⭐ The k=7 MTP control was essential and inverted the obvious read.** Going deeper on MTP -*improves acceptance* (2.753 → 3.041) while **destroying throughput** (114.9 → 74.0). Our MTP -head is a single module (`mtp_num_hidden_layers=1`, only `mtp.layers.0`, 15 tensors) run -autoregressively, so k draft tokens cost k sequential forward passes. **"Just raise -num_speculative_tokens" is a trap** — without the control I would have recommended it. - -DFlash2's win is therefore **not better per-token acceptance** — our MTP is actually *better* at -position 0 (79.6% vs 75.4%). It is that block drafting makes depth nearly free. - -**⭐ The drafter is model-agnostic across finetunes — 3.254 (gen) vs 3.252 (sec), a 0.06% -difference**, with superimposable per-position curves. One drafter file on `/tank` serves both. - -## ✅ MEASURED — how DFlash2 runs (answers "can one drafter serve both seats?") - -**EAGLE3-style coupled, not standalone.** In vLLM: `load_model(self, target_model)` binds it to a -specific target object; `pass_hidden_states_to_model=True`; `gpu_model_runner` reads -`dflash_config.target_layer_ids` → `[i+1 …]` to register auxiliary hidden-state capture on the -target at layers **5, 19, 33, 47, 61**. It even reads the target's RoPE style at load. - -Consequences: -- **Weights file is shareable** (one download, both seats mount it) — gen and sec are - architecturally identical on every dimension the drafter needs: 64 layers (deepest tap 61), - hidden 5120, intermediate 17408, vocab 248,320 > mask token 248,070. -- **VRAM is NOT shareable — 3.85 GB per seat.** The drafter lives inside the target's engine - process, consuming hidden states mid-forward. Two seats are two processes; there is no - cross-process sharing mechanism and there could not be. - -## ✅ MEASURED — it works on our stack, which the card does not claim - -The card tests stock BF16 on an H200 with FlashAttention 3. Verified here instead: -**abliterated + NVFP4 `compressed-tensors` target ✓, Blackwell sm_120 ✓, DFlash2 CUDA graphs -captured ✓.** None of that was documented anywhere. - -## 🔶 HYPOTHESIS — why our acceptance trails the published numbers - -Both our targets land at ~3.25 accepted length against the card's 4.10–5.46 on stock BF16. -**Finetune drift is ruled out** — two *different* finetunes gave identical results to three -decimals. The shared variable is **NVFP4 quantization of the target**, which is mechanically -plausible (the drafter reads quantized hidden states at its five taps). Second candidate: -prompt distribution (ours general-purpose, theirs GSM8K/MATH/HumanEval/MBPP/MT-Bench). -**Neither is confirmed.** Settling it needs a BF16 target seat (~56 GB) — a real GPU window. - -## ❌ RETRACTED — the "MTP head mismatch causes the degeneration" hypothesis - -**Operator ruling, 2026-08-22: this hypothesis is WRONG. The degeneration lives in the un-fixed -vLLM, not in the weights.** Recorded here rather than deleted, because it was reasoned to -confidently enough that a future session could re-derive it. - -**Two independent failures produced it, and the second is the instructive one:** - -1. **I treated a false dichotomy as a deduction.** Having verified gen and sec run an identical - engine (same image ID `sha256:bd3236cff208…`, same live version - `0.27.2rc1.dev150+g311b3513a` read from inside both processes, same flags bar - `gpu-memory-utilization` 0.43 vs 0.44), I concluded "config is eliminated, therefore it is the - weights." That does not follow. **An engine bug present in BOTH seats is not exonerated by the - two seats being identical** — it just means the engine cannot explain a *difference*. It can - still explain the *failure*. -2. **The difference I was explaining may not exist.** The premise was a single operator - observation of sec degenerating at ~2k, made during a session with many concurrent changes. - **n=1 under heavy concurrent modification is not evidence** — see the meta-lesson below. - -**What survives as fact** (measured, still true, just not causal): sec's MTP head *is* -byte-identical to `qwen38-27b-uncensored-bf16` across all 15 tensors — a stock head on a -security-finetuned body, because the `Qwen3_5ForConditionalGeneration` wrapper never loads the -head, so the finetuning could not reach it. gen's orcarouter head *was* abliterated in-band by -its author. Acceptance differs slightly (gen 58.4%, sec 55.9%). **All true. None of it shown to -cause multi-turn degeneration.** - -**Current standing explanation: the degeneration is an engine bug in the un-fixed vLLM.** Both -production seats run `311b3513`, which is **172 commits behind GDN spec-decode fix #53077** -(merged 2026-08-20). `#51113` is present in that build and is therefore **necessary but -insufficient** on its own. - -## ⭐⭐ META-LESSON — n=1 during a busy session is not evidence - -The operator's own framing, and it generalises past this incident: **an observation made while -many things are being changed at once cannot carry a causal claim, no matter how confidently it -is reported.** Tonight that single observation became the load-bearing premise for a weights-side -hypothesis, a root-cause narrative, and very nearly a recommendation. - -This is the same failure the gen-seat compose file already warns about in different words — *"a -passing probe is NOT sufficient evidence"* — inverted. That note guards against trusting a -**negative** result from a synthetic test. This one guards against trusting a **positive** -sighting from an uncontrolled session. Both reduce to: **hold the system still, or do not draw -causal conclusions from it.** - -Applies equally to the "coherent to 10k" observation below — same n, same conditions, opposite -direction. Neither observation is worth more than the other. - -## ⚠️ CONFOUNDED — and the "before" state is itself unreliable - -sec now runs DFlash2 on a newer build and the operator reports **coherent to 10k tokens with -adversarial nonsense prompts**. ⚠ Treat this the same way as the 2k sighting it is being compared -against: **n=1, uncontrolled session, not evidence.** The comparison is weak on *both* ends. - -**Two variables changed at once:** - -1. **Engine**: `311b3513` → `e9d1398d`, **+259 commits, `behind_by=0`** (a strict superset), - including GDN spec-decode fix **#53077** (merged 2026-08-20) that production is **172 commits - behind**. -2. **Drafter**: frozen MTP head → DFlash2 reading live hidden states. - -**Isolating it = run MTP k=3 on the same new build.** Not yet done. - -**#51113 is present in BOTH builds** (verified by ancestry, `behind_by=0` each) — so the -"proper upstream fix" our compose comment credits is **necessary but insufficient**; sec ran it -and still degenerated. Related open upstream: **#53180** (quantized Qwen3.8-27B hybrid GDN + MTP -producing *silent* degenerate output, no fix), **#41884** (DFlash + prefix caching on hybrid, -IndexError, workaround is disabling one). - -## ❌ WRONG TURNS — do not repeat - -- **Version strings are not lineage.** The DFlash2 build reports `0.26.1rc1.dev1048` and our - production nightly `0.27.2rc1.dev150`, which *looks* like a regression. It is a setuptools_scm - tag-reachability artifact. **Use the GitHub compare API and check `behind_by`.** -- **Docker Hub push timestamps lie about source freshness.** `nightly-ba07e4a4` was *pushed* - 06:12 UTC, comfortably after the 05:27 merge — but *cut* from a 03:46 commit that predates it. - **Grep the image for the symbols you need.** Believing the timestamp would have cost an RP-seat - outage to serve a model the engine could not instantiate. -- **`--max-num-batched-tokens` was not the image truncation.** Raising it 16,384 → 32,768 on that - theory changed nothing and cost ~3 GiB of peak activation, which came straight out of the KV - pool. The cap was the tokenizer (§3.14 of the playbook). -- **"1M needs YaRN, absent from config" is FALSE for the sec quant.** It is fully present: - `rope_type: yarn`, `factor: 4.0`, `original_max_position_embeddings: 262144`, - `max_position_embeddings: 1000000`. Context is a KV-memory choice, not a model limit. - -## Live state — PROMOTED to the compose stack 2026-08-22 - -**Operator-approved after real-use testing** ("performing very well"). The experimental -standalone container is gone; `stacks/mog-sec/` is canonical and `restart: unless-stopped` means -it survives reboots. Cutover verified: **KV pool 526,617 / 1.10x — identical to the container it -replaced**, restarts 0, both gateway aliases serving, DFlash2 confirmed drafting at k=7 -(231 draft tokens over 33 drafts), vision working. - -⚠ **One variable was deliberately REMOVED, not carried over.** The old stack hardcoded -`PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`; the validated DFlash2 container never set it, -and playbook §3.10 records expandable_segments corrupting retained tensors elsewhere. The compose -now defaults it EMPTY (`MOG_ALLOC_CONF`). Promoting it as-was would have shipped a variable the -tested configuration did not have. - -**Compose is now parameterised for the shapes that differ:** `MOG_SPEC_CONFIG` carries the whole -speculative JSON (dflash needs `"model": "/drafter"`, MTP must not have one — a method+tokens -template cannot express both), plus `MOG_MM_PROCESSOR_KWARGS`, `MOG_DRAFT_MODEL`, -`MOG_MAX_NUM_BATCHED_TOKENS`, `MOG_ALLOC_CONF`. - -**ROLLBACK:** `.env.bak-pre-dflash2-20260822` and `compose.yaml.bak-pre-dflash2-20260822` on the -host; or one line — `MOG_SPEC_CONFIG={"method": "qwen3_5_mtp", "num_speculative_tokens": 3}` plus -the old `MOG_IMAGE`. - -| | production sec | current | -|---|---|---| -| image | `nightly-311b3513` | `nightly-e9d1398d` | -| speculation | MTP k=3 | **DFlash2 k=7**, drafter `/tank/aimodels/qwen38-27b-dflash2-drafter` | -| max-model-len | 262,144 | **480,000** | -| KV pool | 418,218 (1.60×) | **526,617 (1.10×)** | -| images | 4096² → 16,384 tok | **2048² → ~5,125 tok** (`--mm-processor-kwargs` size cap) | - -⚠ **`--gpu-memory-utilization 0.55` is the stable ceiling** while GPU1's other tenants are up. -0.58 sized KV at 594,172 then **OOM'd during CUDA graph capture** — the process reached 57.49 GiB -against ~57.6 free. Real 1M context needs ~49 GiB of KV and therefore evicting most of GPU1. - -Canonical config: `stacks/mog-sec/{compose.yaml,.env.example}` in this repo. diff --git a/persistent-memory.d/2026-08-23-ana-docker-env-perms-sweep.md b/persistent-memory.d/2026-08-23-ana-docker-env-perms-sweep.md deleted file mode 100644 index 49bae49..0000000 --- a/persistent-memory.d/2026-08-23-ana-docker-env-perms-sweep.md +++ /dev/null @@ -1,34 +0,0 @@ -# [2026-08-23] Every secret-bearing `.env` on ana-docker tightened to 0600 - -Found while taking uptime ownership of hrafn: its `.env` was mode 0644 with a live -bearer token. Not a hrafn lapse — **0644 was the de facto pattern on the host**. -Eight stacks carried secret-shaped vars in world-readable `.env` files on a box with -four interactive accounts, verified as real exposure by reading one as `nobody`. - -Swept: **vaultwarden, traefik**, beszel, gitea-runner, miniflux, news-digest, -searxng, vor. (hrafn and nevermore were fixed separately the same day.) Six other -stacks already used 0600, so this converged on the existing house pattern rather -than inventing one. Post-sweep the host has **zero** secret-bearing `.env` readable -by `nobody`. - -Playbook: `playbooks/tighten-env-perms.yaml`, one run per stack, re-runnable. - -## The check that matters - -Every run asserts `docker compose config` still renders **as the deploy user** -(`lkraven`), not as root. Checking the mode proves the bits changed; only rendering -as the deploy user proves the next deploy can still resolve its variables. - -## Two gotchas recorded in the playbook - -- **vaultwarden looked like it bind-mounted its `.env`** — which would mean the - *container's* UID reads it and 0600 could break the password vault. It does not: - that `- .env` is under `env_file:`, not `volumes:`. My grep matched the YAML list - item without checking its parent key. The playbook now **refuses** any stack that - genuinely bind-mounts its `.env`, since that case is read by the container UID. -- **elway prompted for a sudo password.** The `ana-docker` ssh alias resolves to - `lkraven`, who needs one; **`infra-ops@10.250.50.70` has NOPASSWD**. `corviduo-dev` - was repointed to infra-ops at some point and `ana-docker` was not. Run elway against - the infra-ops target on this host. - -Commit `a896c0a`. diff --git a/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md b/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md deleted file mode 100644 index f464775..0000000 --- a/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md +++ /dev/null @@ -1,663 +0,0 @@ -# [2026-08-23] Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gbps circuit - -> **⛔ SUPERSEDED 2026-08-23 (same day, later session) — read the CORRECTION at -> the bottom before acting on anything here.** The headline is wrong (the -> relevant ceiling is NH3's **1 Gbps** uplink, not Anaheim's 2 Gbps), the -> aggregate number is wrong (**692 Mbit/s** at 8 streams, not ~550 — the -> original stopped measuring at 4), and the proposed remedy is **impossible**: -> UniFi's manual site-to-site IPsec does not implement AES-GCM at all. The -> per-stream observation and the parallelise-your-transfers mitigation are the -> parts that survive. - -The operator noticed site-to-site transfers were slow for a datacenter fiber -handoff and asked whether WireGuard was the limit. It is not WireGuard, and the -circuit is fine. - -## Measured - -``` -ana-docker -> internet, 8 parallel 2,153 Mbit/s <- the 2 Gbps handoff, delivering -ANA <-> NH3 through the tunnel, 4 par. 460 Mbit/s -FortiGate's own recorded peak 554 Mbit/s -ANA <-> NH3, single stream 227 Mbit/s -ANA <-> ESH, single stream 249-265 Mbit/s -ESH <-> NH3 (never touches ana-gw) 545-557 Mbit/s on a SINGLE stream -``` - -Method: stdlib TCP probe (no ssh, no crypto, no compression) between site -endpoints; raw circuit measured with 8 parallel HTTPS fetches from Hetzner -Ashburn. Host NICs are virtio with no reported cap, so no host-side ceiling. - -## What it is not - -- **Not WireGuard.** Both Anaheim tunnels are IPsec on ana-gw - (`pfi-ana-nh3` -> 70.230.226.88, `ana-eshudm-dyn` -> the ESH UDM). WireGuard - on ana-wg is remote-access only and is not in this path. Traceroute confirms: - both slow paths have hop 1 = `10.250.50.1` (the FortiGate); the fast - ESH<->NH3 path rides a `192.168.x` Site Magic overlay and never touches it. -- **Not CPU or crypto exhaustion.** FortiGate CPU was **100% idle across all - 8 cores** during the tests, and both live tunnels report `npu_flag=03` with - `dec_npuid=1 enc_npuid=1` — encrypt *and* decrypt are hardware-offloaded. -- **Not a 250 Mbit/s cap.** That was the first number and it is misleading — - single-stream TCP. Four parallel streams doubled it. Quote the aggregate. -- **Not the interface.** wan1: `rxe=0 txe=0 rxd=0 txd=0`, no collisions. - -## Most likely cause - -Both tunnels negotiate **`aes256-sha1`** in phase 1 *and* phase 2 (dhgrp 14, -IKEv2). AES-CBC + SHA1 is a two-pass operation; FortiGate NPUs are markedly -faster on **AES-GCM**, which combines encryption and authentication in one -pass. The datasheet IPsec headline for an 80F assumes GCM with large packets, -not CBC+SHA1 at the 1438-byte tunnel MTU this link negotiates. The ~4x -shortfall is consistent with that. - -## Not executed - -Changing the proposal is a **production-edge change requiring a matching -change at the far end** (NH3 UDM and the ESH UDM), and each tunnel drops while -it renegotiates. Left for the operator. See the index entry for authorization -state. - -## Immediate mitigation, no config change - -Per-flow is the weak axis: a single stream over Site Magic gets 557 Mbit/s, a -single stream through IPsec gets 227. **Anything moving bulk data across the -Anaheim link should parallelise** — that alone roughly doubles throughput -today. - -## Practical consequence already observed - -`/mnt/smithy` mounted on ana-ml2 reads at 24.7 MB/s sequential vs 98.3 MB/s -from nh3-dev (same file, same mount) — that gap *is* this tunnel, not NFS and -not the NAS. See [[2026-08-23-smithy-mount-ana-ml2]]. - -## Access note - -ana-gw is a FortiGate-80F, FortiOS 7.2.10, at 10.250.0.1. `sshpass` is absent -on nh3-dev; connect with paramiko via `uv run --with paramiko`. Password is -vaulted at `fortigate/ana-gw-infra-ops-password`. **`diagnose vpn tunnel list` -prints live ESP session keys** — never paste its output into althing, a -booth, or a commit. - ---- - -## CORRECTION (2026-08-23, later session): the cutover was attempted and the remedy does not exist - -The operator authorised the AES-GCM cutover, NH3 side first. It cannot be done, -and the measurements taken while trying show there is very little left to win. - -### AES-GCM is unavailable on the far end — not a naming problem - -The NH3 edge is a **UDM Pro SE** terminating `pfi-nh3-ana` (networkconf -`_id 697d64414c85dd2b6669b00a`, `ifname vti64`). Its UniFi API **validates** the -crypto enum and rejected every GCM spelling tried — `aes256gcm`, `aes256gcm128`, -`aes256gcm16`, `aes-256-gcm`, `aes256-gcm`, `aes256gcm12`, `gcm`, `aes128gcm128` -— all `HTTP 400 api.err.InvalidPayload`, nothing applied. - -**The control that makes this conclusive:** the *identical* request body with -`ipsec_esp_encryption: "aes256"` returns `HTTP 200 rc:ok`. So the 400s are the -enum rejecting the value, not a malformed body. Corroborating: **zero -case-insensitive `gcm` matches across 7.3 MB of UniFi OS UI bundles.** - -Accepted enum (probed): `aes128`, `aes192`, `aes256`, `3des` → 200; `des`, -`chacha20poly1305` → 400. There is no AEAD option. Both Anaheim tunnels land on -UniFi far ends, so this blocks the ESH tunnel too. - -The FortiGate side **was** widened and is GCM-capable: phase2 `pfi-ana-nh3` now -reads `set proposal aes256-sha1 aes256gcm`. Left in place deliberately — it is -functionally identical while the peer only offers CBC, and reverting it would -cost another SA renegotiation for a cosmetic gain. Phase 1 was never touched; -IKE protects the control channel only and has no bearing on data throughput. - -### The numbers that retire this as a problem - -Measured NH3→ANA through the tunnel, and NH3→ESH over Site Magic (WireGuard) on -the same UDM and the same uplink, with the same stdlib TCP probe: - -| streams | IPsec NH3→ANA | WireGuard NH3→ESH | -|---|---|---| -| 1 | 245 Mbit/s | 557 Mbit/s | -| 4 | 471 Mbit/s | 767 Mbit/s | -| 8 | **692 Mbit/s** | **795 Mbit/s** | - -**NH3's WAN is a 1 Gbps link** (`uplink.speed = 1000`, port capable of 10G) — -that, not Anaheim's 2 Gbps, is the ceiling for anything crossing this tunnel. -So the tunnel does **~69% of the achievable uplink** at 8 streams, and the -IPsec-vs-WireGuard gap collapses from 2.3× at one stream to **15% at eight**. - -Re-architecting the transport (site-to-site WireGuard via `ana-wg`, since -FortiOS has no WireGuard) would chase that last 15%. Not worth it. - -### What the constraint actually is - -A **per-stream** limit (~245 Mbit/s), not an aggregate crypto ceiling. Both -endpoints are idle at load — FortiGate CPU 100% idle with `npu_flag=03` -(offloaded both directions), UDM CPU ~7% with load1 moving 0.70 → 1.55. The -shape is per-SA/per-flow serialisation, and WireGuard shows the same shape from -a higher floor (557 → 795 is only 1.43× scaling). - -### Actionable consequence - -Anything moving bulk data across this link should **parallelise** — 245 → 692 -Mbit/s, a 2.8× win with no config change. For single-stream workloads that -cannot be parallelised at the application layer, **NFS `nconnect=N` is the -lever**: it opens N TCP connections per mount, converting a single-stream -workload into a parallel one. The `/mnt/smithy` mount on ana-ml2 reading at -24.7 MB/s (~200 Mbit/s, i.e. exactly the single-stream ceiling) is the live -example — remounting with `nconnect=8` is the obvious test. - -### Foot-gun recorded - -Probing the enum by PUTting candidate values **applies the accepted ones**. A -probe loop here timed out with `3des` briefly live on the NH3 side, which the -FortiGate would not accept — a short tunnel outage until `aes256` was restored -(~1 minute, confirmed by the SA counters resetting). If you enumerate a UniFi -config enum this way, restore the known-good value after **every** 200, not at -the end of the loop. Post-change verification: the UDM object was diffed -field-by-field against its pre-change snapshot and is **byte-identical**. - ---- - -## FOLLOW-UP (2026-08-23): what the per-stream limit actually is - -The correction above called the constraint "per-SA/per-flow serialisation". -That was a hand-wave. Measured properly, it is a **hard per-flow rate cap of -~230–245 Mbit/s with a very deep buffer in front of it** — not a tuning -problem, not loss, not window size. - -### The evidence: pin the send buffer and sweep it - -Single stream NH3 → ana-docker, `SO_SNDBUF` pinned, `ss -ti` sampled in flight: - -| in-flight cap | throughput | RTT in flight | minRTT | retrans | -|---|---|---|---|---| -| 256 KB | 224 Mbit/s | 7.8 ms | 5.3 ms | 0 | -| 416 KB | 225 Mbit/s | 11.8 ms | 6.6 ms | 0 | -| 416 KB | 245–247 Mbit/s | 12.0 ms | 5.6 ms | 0 | -| ~3.3 MB (autotuned) | 245 Mbit/s | **107 ms** | 5.5 ms | 0 | - -**Throughput is flat across a 13× range of in-flight data while RTT scales with -it.** That is the signature of a fixed service rate with a standing queue: the -window controls only how much queue you build, never how fast you go. Had this -been window-limited, throughput would have risen with the buffer. Had it been -congestion, there would be retransmits — there are essentially none -(`retrans:0`, 0% ping loss). - -So `net.ipv4.tcp_*` tuning, window scaling and congestion-control choice are all -**red herrings here**. Do not go there. - -### Bufferbloat: one bulk stream wrecks latency for everything else - -Measured on the same tunnel, ping to ana-docker: - -- idle: **6.9 ms** avg -- during a **single** bulk TCP stream: **102 ms** avg, 136 ms max, 0% loss - -**15× latency inflation from one transfer.** This is the operationally -important finding — any interactive traffic sharing the Anaheim link (ssh, -RDP, althing, VoIP) degrades badly whenever anything moves bulk data, and it -takes only one stream to do it. Parallelising transfers makes throughput -better and this *worse*. If it starts biting, the fix is an AQM/shaper on the -tunnel (or rate-limiting bulk jobs), not more buffer. - -### Where the cap lives — strong inference, not proof - -Three paths, and the FortiGate is the only variable: - -| path | single-stream | -|---|---| -| FortiGate ↔ NH3 UDM (IPsec) | 245 Mbit/s | -| FortiGate ↔ ESH UDM (IPsec) | 249–265 Mbit/s | -| NH3 UDM ↔ ESH UDM (WireGuard, **no FortiGate**) | 557 Mbit/s | - -Present in both slow paths, absent from the fast one. Aggregate over the same -SA reaches 692 Mbit/s, so it cannot be the SA or the crypto engine as a whole — -many flows spread out fine, one flow does not. - -The mechanism that fits is **FortiGate NPU IPsec offload being per-session**: -each firewall session is bound to one crypto engine, so a single TCP flow is -capped at one engine's rate while many sessions spread across engines. **This -is inference from the throughput shape, not something confirmed on the box** — -`diagnose sys session list` was not captured for a TCP flow (the filter caught -only traceroute UDP probes). A single-stream control through ana-gw *without* -IPsec returned 290 Mbit/s to Hetzner Ashburn, but at ~60 ms RTT that is -window-limited and does not discriminate. **If this matters, the clean test is -a non-IPsec single stream between two Anaheim VLANs at low RTT.** - -**Relevant to the FortiGate cutover decision:** if the per-flow cap is the -FortiGate's IPsec path, replacing the box plausibly lifts single-stream -throughput toward the WireGuard figure. That is a point in favour of the -cutover, and it is cheap to verify afterwards by re-running the sweep. - ---- - -## FOLLOW-UP 2 (2026-08-23): it is NOT a capacity problem, and it IS specific to IPsec - -Operator asked directly whether the 80F "can't handle the traffic". It can. -Two new measurements settle the shape of this, and correct an overstatement in -FOLLOW-UP 1 (which pointed at the FortiGate on evidence that was confounded — -every slow path was *both* IPsec *and* FortiGate, so protocol and box could not -be separated by that argument). - -### The 80F routes a single flow at line rate when IPsec is not involved - -`ana-ml2 → pfi-pve`, inter-VLAN **through** ana-gw (traceroute hop 1 = -`10.250.50.1`), 0.36 ms RTT, no tunnel: - -| streams | throughput | -|---|---| -| 1 | **940.2 Mbit/s** | -| 8 | 939.3 Mbit/s | - -Single stream saturates 1 GbE. So the box does **not** cap single sessions in -general, and there is no per-session ceiling in its plain forwarding path. The -~250 Mbit/s per-flow cap is **specific to the IPsec datapath**. - -### Both IPsec tunnels converge on the same numbers despite different far ends - -Measured today with the same probe: - -| tunnel | far-end gateway | RTT | 1 stream | 8 streams | -|---|---|---|---|---| -| NH3 ↔ ANA | UDM Pro **SE** | 6.7 ms | 245 Mbit/s | 692 Mbit/s | -| ESH ↔ ANA | UDM Pro **Max** | 3.9 ms | **268 Mbit/s** | **715 Mbit/s** | - -Different gateway hardware, different sites, different uplinks, and RTT -differing by 1.7× — yet single-stream differs by only 9%. **If this were -window-limited the 3.9 ms path would be ~1.7× faster.** It is not, which is -independent confirmation of a rate cap rather than a BDP effect. - -### Capacity summary — the box has headroom it will not give one flow - -- plain routing, 1 stream: **940 Mbit/s** (line rate) -- plain routing to internet, 8 streams: **2,153 Mbit/s** -- IPsec, 8 streams: **692–715 Mbit/s** -- IPsec, 1 stream: **245–268 Mbit/s** -- CPU **100% idle** throughout; IPsec NPU-offloaded (`npu_flag=03`) - -Within a single SA, 8 sessions get ~2.9× what 1 session gets, so the datapath -distributes work **by inner session** — consistent with IPsec offload binding a -session to one crypto engine. - -### What is still NOT separated - -Whether the cap belongs to **the 80F's IPsec offload** or to **UniFi's IPsec -implementation**. Both tunnels have a UDM at the far end, and both UDMs run the -same UniFi firmware, so identical caps are explainable either way. The Pro Max -being only 9% faster than the Pro SE argues against the UniFi side (a beefier -CPU should show more), but that is suggestive, not conclusive. - -**The test that closes it:** an IPsec tunnel whose endpoints do not include the -80F — e.g. a temporary UDM↔UDM IPsec tunnel between NH3 and ESH, measured -single-stream. If it also caps ~250, the FortiGate is exonerated and replacing -it buys nothing on this axis. If it runs near the 557 Mbit/s that UDM↔UDM -WireGuard achieves, the 80F is the limiter. **Bears directly on the pending -FortiGate cutover** — worth running before that decision, not after. - ---- - -## FOLLOW-UP 3 (2026-08-23): WireGuard over the same internet path does 767 Mbit/s on ONE stream - -Operator asked for a WireGuard test from `ana-wg` to NH3 over the public -internet. It is the test that separates the *path* from the *crypto*, and the -answer is unambiguous. **It also overturns FOLLOW-UP 1's "re-architecting the -transport is not worth it" — that conclusion compared 8-stream numbers and was -wrong for single-stream workloads.** - -### Setup (fully torn down afterwards) - -`ana-wg` (10.250.50.252, Debian 12 LXC, 4 cores) already has an -internet-reachable WireGuard endpoint: wg0 on **UDP 31337**, published by -FortiGate VIP `wg-to-ana-wg` (extip **38.120.12.42** → 10.250.50.252:31337, -policy 46, service `WireGuard-LEET`). **No FortiGate change was needed.** A -temporary `wgt0` was created on nh3-dev (10.30.10.200/32) as a fourth peer on -wg0, measured, then removed — ana-wg is back to its original 3 peers and the -keys were shredded. `wireguard-tools` was installed on nh3-dev and **left in -place** (benign, and wanted if this becomes permanent). - -In this topology **neither gateway does crypto**: the FortiGate and the NH3 UDM -only NAT/forward UDP, and Linux does WireGuard at both ends. - -### The full comparison - -| path | crypto performed by | 1 stream | 8 streams | -|---|---|---|---| -| IPsec NH3↔ANA | FortiGate + UDM | 245 Mbit/s | 692 Mbit/s | -| IPsec ESH↔ANA | FortiGate + UDM | 268 Mbit/s | 715 Mbit/s | -| **WireGuard NH3→ana-wg** (same internet path) | **Linux + Linux** | **767 Mbit/s** | 763 Mbit/s | -| WireGuard NH3↔ESH (Site Magic) | UDM + UDM | 557 Mbit/s | 795 Mbit/s | -| plain routing through the 80F (inter-VLAN) | none | 940 Mbit/s | 939 Mbit/s | - -**One stream equals eight streams over Linux WireGuard (767 ≈ 763).** There is -no per-flow penalty at all, and a single flow already saturates the path. So -the ~245 Mbit/s per-flow cap is **not** the ISP, not the circuit, not the NH3 -uplink and not the physical path — all of which sustain 767 on one flow. - -Per-flow penalty ranks by implementation: - -- **Linux WireGuard — none** (767 → 763, flat) -- **UDM WireGuard — mild**, ~1.4× (557 → 795) -- **IPsec on this pair — severe**, ~2.8× (245 → 692) - -### Latency under load — the same story - -| path | idle | during ONE bulk stream | -|---|---|---| -| IPsec NH3↔ANA | 6.9 ms | **102 ms** avg, 136 ms max | -| WireGuard NH3→ana-wg | 6.2 ms | **12.7 ms** avg, 23 ms max | - -WireGuard carries **3.1× the single-stream throughput with 8× less latency -inflation** on the same wire. - -### Attribution — still not fully separated, and it no longer matters much - -Both IPsec measurements have a FortiGate *and* a UDM doing IPsec, so this still -does not isolate which one imposes the 2.8× penalty. Closing that would need -Linux↔Linux IPsec or UDM↔UDM IPsec on the same path. **But the practical -decision no longer depends on the answer**, because the fix is the same either -way and it is already demonstrated. - -### Recommendation (supersedes FOLLOW-UP 1) - -A **WireGuard site-to-site between NH3 and Anaheim, terminated on `ana-wg`**, is -worth real consideration: 3.1× single-stream, flat scaling, far better latency -under load, and it reuses infrastructure that already exists and is already -internet-reachable. It is also the architecture already proven for NH3↔ESH. -Open questions before committing: routing/failover if ana-wg (an LXC) is down, -whether it replaces or parallels the IPsec tunnel, and firewall policy for the -new transit. ana-wg CPU was only ~40% busy across 4 cores at 767 Mbit/s, so it -has headroom. - -**AND: `nconnect=8` on /mnt/smithy remains worth doing regardless** — it is the -same lever (turn one flow into many) and brokkr-smithy-dev has given standing -approval to apply it once the FortiGate work settles, with no need to ask again. - ---- - -## RESOLVED (2026-08-23): it is the UDM's software AES-CBC. The FortiGate is exonerated. - -Operator's theory — the UDM does IPsec in software with no crypto offload, so -the cost of the cipher itself is the limit — is **correct**, and it is now -demonstrated rather than inferred. He also correctly pointed out that -UDM↔UDM Site Magic is **WireGuard, not IPsec**, so that row never said anything -about UniFi's IPsec performance. It didn't, and I had leaned on it. - -### The controlled experiment: vary cipher cost, hold everything else - -AES-128 is 10 rounds, AES-256 is 14. If software crypto is the binding -constraint, throughput must rise when the cipher gets cheaper. If the limit -were the FortiGate's NPU, it would not move at all — hardware crypto is not -cipher-cost-sensitive in that range. Run A/B/A, single stream, 25–60 s each: - -| condition | ESP cipher | single-stream | UDM CPU | -|---|---|---|---| -| A | aes256-cbc + sha1 | 232.3 Mbit/s | 35.4% | -| B | **aes128**-cbc + sha1 | **281.8**, 274.9 Mbit/s | 35.5% | -| A again | aes256-cbc + sha1 | 244.9, 242.5 Mbit/s | — | - -**~1.16–1.20× faster on the cheaper cipher at identical CPU.** Same bytes of -CPU work, more payload through it. That is the signature of CPU-bound software -crypto, and it rules out the FortiGate's NPU as the limiter. - -### Correcting two of my own earlier claims - -1. **"UDM CPU is only ~7%, so it isn't CPU-bound" was WRONG — a sampling - artifact.** UniFi's `system-stats.cpu` refreshes on the device report - interval; 4-second sample windows were reading stale values. Under a - sustained 60 s single-stream load it reads **35.4%**, with load1 rising - 0.60 → 1.17. On a 4-core UDM Pro SE that is ≈1.4 cores — one core saturated - on crypto plus overhead. **Always drive load for ≥60 s before trusting a - UniFi CPU figure.** -2. **The "FortiGate per-session NPU offload" hypothesis is REFUTED**, not merely - unproven. It predicts no change from a cipher swap; a 20% change was measured. - -### Why the numbers all line up now - -- **1 stream = 1 core of UDM crypto** → ~240 Mbit/s on AES-256-CBC. -- **8 streams = ~3 usable cores** → ~692 Mbit/s, ≈2.9× the single-stream figure - on a 4-core box. Aggregate is noisy (492–692 across repeats on a live link) - and is *not* cipher-sensitive, consistent with it being bounded by the path/ - uplink rather than crypto once several cores are engaged. -- **AES-CBC is the specific villain: it is serial.** Each block depends on the - previous one, so the ARM AES instructions cannot pipeline across blocks. GCM - (CTR-based) and ChaCha20-Poly1305 both parallelise freely. That is why the - same UDM does 557 Mbit/s single-stream on WireGuard and only 240 on IPsec. -- **This retroactively vindicates the GCM cutover as the right idea aimed at the - right box** — GCM would have removed the serial dependency on the constrained - end. UniFi simply does not offer it, which is what made it impossible. - -### Options this opens - -- **AES-128 instead of AES-256: ~16–20% for free**, no topology change, one API - call per end. 128-bit is not the weak link here (SHA1 integrity is more - dated, and unchanged either way). Operator's call — **not adopted**, restored - to aes256. -- **WireGuard site-to-site via ana-wg: 767 Mbit/s single-stream** (3.1×), and it - sidesteps the UDM's IPsec datapath entirely. Still the biggest win available. -- Replacing the FortiGate **will not help this** — it was never the constraint. - Worth knowing before the cutover. - -### State left behind - -UDM network object verified **byte-identical** to its pre-test snapshot -(aes256/sha1). Tunnel up, selectors 1/1. FortiGate phase2 `pfi-ana-nh3` is -left as `aes256-sha1 aes256gcm aes128-sha1` — a permissive superset; the peer -offers only aes256 so the extra entries are inert, but **narrowing it back to -`aes256-sha1` is one line** if the looser list is unwanted. - ---- - -## FOLLOW-UP 4 (2026-08-23): a downstream WireGuard terminator costs nothing to forward through - -Operator's point: FortiOS has no WireGuard, so a WireGuard site-to-site must -terminate on a box *behind* the edge. Correct — and `ana-wg` (LXC, CT 113 on -pfi-pve, 10.250.50.252) already is that box. - -**This closes a gap in FOLLOW-UP 3.** That 767 Mbit/s figure was measured with -traffic terminating *on* ana-wg. Real traffic must be forwarded onward to other -Anaheim hosts, which was never measured. Now it is: - -| topology | 1 stream | 8 streams | -|---|---|---| -| IPsec, FortiGate ↔ UDM (today) | 245 Mbit/s | 692 Mbit/s | -| WG terminating **on** ana-wg | 767 Mbit/s | 763 Mbit/s | -| **WG transit: nh3 → wg → ana-wg → forward → ana-docker** | **763.8 Mbit/s** | **790.4 Mbit/s** | - -**Forwarding through the LXC is free** (763.8 vs 767). The downstream-VM -architecture delivers the full 3.1× single-stream for real transit traffic, not -just for traffic landing on the tunnel box. - -ana-wg while forwarding 764 Mbit/s: **~22% busy across 4 cores** (77.8% idle), -so roughly 0.9 cores. Note `/proc/loadavg` inside this LXC reports the *host's* -load, not the container's — do not read it as ana-wg's own. For contrast the -UDM burns 35.4% of its 4 cores to move 240 Mbit/s, so ana-wg has ample headroom. - -### Design consequences of terminating downstream — the parts that need decisions - -1. **Anaheim hosts must route to ana-wg, not to the FortiGate.** The 763.8 - figure was obtained with an explicit `10.30.10.200/32 via 10.250.50.252` - route on ana-docker. Without that, a host sends 10.100.0.0/16 to its default - gateway (ana-gw), which routes it back out the *same* interface to ana-wg — a - LAN hairpin crossing the FortiGate twice. **The hairpin variant was NOT - measured.** Options: DHCP option 121 pushing the route fleet-wide, a dedicated - transit VLAN for ana-wg, or accept the hairpin. -2. **New single point of failure.** Today site-to-site dies only when the edge - dies, which is total anyway. A downstream terminator fails independently. - Mitigation: keep the IPsec tunnel configured as a higher-metric fallback - route so it takes over when ana-wg is down. -3. **ana-wg is an LXC on pfi-pve**, so its ~0.9 cores and NIC traffic land on the - hypervisor shared with the rest of the Anaheim VMs. -4. **The NH3 end needs a terminator too**, and there are two shapes: - - **Linux VM at NH3** (nh3-dev or a dedicated VM on nh3-pve) — this is what - was measured: **764 Mbit/s**. - - **NH3 UDM's existing WireGuard server** (`PFI-NH3-WG`, wireguard-server on - UDP 31337) accepting ana-wg as a peer — plausible but **untested**, and - UniFi's WireGuard shows a per-flow penalty (557 Mbit/s single-stream on - Site Magic), so expect ~557 rather than 764. Still 2.3× today. - -### Standing recommendation - -Worth doing, but it is **a project, not a config tweak** — routing, failover and -policy all need deciding. The cheap wins remain available meanwhile and are -independent: `nconnect=8` on NFS mounts (approved by brokkr-smithy-dev, pending -the FortiGate work settling) and AES-128 for ~20%. - ---- - -## LANDED (2026-08-23): AES-128 on both tunnels; FortiGate public admin closed - -Operator directed: adopt AES-128 on **both** Anaheim tunnels, make-before-break, -then close the FortiGate's WAN and SSH admin surfaces. All done and verified. - -**Context that retires the WireGuard-in-a-VM design work:** the FortiGate is -being **replaced by OPNsense on a Dell R420**, which gives **WireGuard on the -edge device itself**. The downstream-terminator architecture (FOLLOW-UP 4) is -therefore moot — do not scope it. This also **un-parks the OPNsense migration**, -which auto-memory recorded as PARKED pending "hardware acquisition"; the R420 -is that trigger. - -### What changed - -Make-before-break on the FortiGate first, so neither tunnel dropped waiting on -a far end: - -| phase2 | proposal now | -|---|---| -| `pfi-ana-nh3` | `aes256-sha1 aes256gcm aes128-sha1` | -| `ana-eshudm-dyn` | `aes256-sha1 aes128-sha1` | - -Then each UDM flipped to `ipsec_esp_encryption: aes128`: - -| tunnel | UDM object | before | after | -|---|---|---|---| -| NH3 ↔ ANA | `pfi-nh3-ana` `697d64414c85dd2b6669b00a` @ 10.100.0.1 | 245 Mbit/s | **269.7** | -| ESH ↔ ANA | `esh-ana` `697723b9b9d4266dddf2bcc7` @ 10.0.0.1 | 268 Mbit/s | **304.3** | - -Single-stream gain ~10–13% here, against 16–20% in the earlier controlled A/B — -the difference is live-link variance, not a different result. Both UDM objects -were diffed field-by-field against pre-change snapshots: **the only field that -moved on either is `ipsec_esp_encryption`.** - -The FortiGate proposal lists were deliberately **left permissive** (still -accepting aes256). The peers offer only aes128 so the extra entries are inert, -and keeping them means a UDM reverting does not strand the tunnel. Narrowing to -`aes128-sha1` alone is a one-liner if the looser list is unwanted. - -### Admin surfaces closed - -`wan1 allowaccess` → **`ping`** (https + ssh removed) and `infra-ops` trusthost -→ **10.0.0.0/8 only** (the 8 wide-open ranges unset). Verified 443 and 22 closed -from both NH3 and ESH; management over the tunnel at 10.250.0.1 still works. -**Sequencing that matters: the close was executed over the TUNNEL path, not over -WAN** — removing `ssh` from allowaccess while connected over WAN kills the -session mid-command. - -**Consequence to hold in mind: ana-gw now has no out-of-band management path.** -If both tunnels drop it is console-only until someone is on site. - -### Gotcha: the two UDM vault items have DIFFERENT shapes - -- `unifi/pfi-udmse-api-key` → a **bare 32-char key**. `secret get` output is the key. -- `unifi/esh-udmpm-api-key` → a **19-line documentation note** with the key on a - `key:` line. `secret get` piped straight into a header yields a 1396-byte - value and the UDM answers **`400 Bad Request` from nginx**. Extract with - `grep '^key:' | awk '{print $2}'`. - -**The ESH key's first-ever confirmed WRITE happened here** (auto-memory recorded -it as read-verified only): a control PUT of the unchanged object returned -`rc:ok`, then the real change did too. That key has a full read+write admin role. - ---- - -## CORRECTION (2026-08-23): port 80 on the WAN IP is the FortiOS ACME listener - -The claim in the previous section that `.42:80` was an **ISP transparent proxy** -was **WRONG**, and so was the earlier warning that ACME renewal would fail with -port 80 absent from `allowaccess`. Operator pushed back asking where the port-80 -map terminated. It terminates **on the FortiGate itself**. - -**What it is:** the FortiOS **ACME HTTP-01 challenge listener**. `config system -acme` has `set interface "wan1"`, and FortiOS opens port 80 on that interface to -answer Let's Encrypt challenges **independently of `allowaccess`** — `wan1 -allowaccess` reads `ping` only and the port is still open. Every non-challenge -request returns a fixed 403 whose body is literally: - -``` -ACME Access OnlyACME Access Only -``` - -**Not a DNAT.** The full VIP table has 14 entries; only two land on `.42` — -`Kokoro-In` (:8880 → 10.250.50.51) and `wg-to-ana-wg` (:31337 → 10.250.50.252). -~~Worth noting separately: four VIPs are all-port static NAT~~ — **that claim was -WRONG, see the correction below.** All fourteen VIPs are scoped. - -### The methodology error that produced the wrong answer — worth not repeating - -The sniffer filter used was `dst host 38.120.12.42 and tcp port 80`. **`dst host` -matches only inbound packets**, so outbound SYN-ACKs were excluded *by -construction*; concluding "the box sends no SYN-ACK" from that capture was -unsound. Re-run with the bidirectional `host 38.120.12.42 and tcp port 80` it -immediately shows `wan1 out 38.120.12.42.80 -> : syn ack`. - -**Rule: when testing whether a box *answers*, the sniffer filter must be -bidirectional. `dst host` silently answers a different question.** - -### Consequences - -- **ACME renewal will work** with `allowaccess ping`. The earlier "add `http` - back or the cert expires" warning is retracted — FortiOS opens the challenge - port itself. Cert valid to 2026-10-27, renewal attempt ~2026-09-27. -- **It is not an admin surface** — static 403, no auth, no GUI. -- Its practical value is now low: WAN admin is closed, so the cert only serves - the internal GUI at 10.250.0.1, where the name would not match anyway. Killing - it (`config system acme` → unset interface) would close the last WAN listener - at the cost of cert renewal. Operator's call; **not done**. - ---- - -## CLOSED OUT (2026-08-23): ACME disabled; and the "all-port VIP" alarm was FALSE - -### ACME disabled — the WAN IP now exposes nothing - -`config system acme / unset interface` (the account object is left in place; -with no interface bound there is no listener). Verified: - -- **External scan of 38.120.12.42 across 55 ports: no open TCP ports at all.** -- Internal GUI at 10.250.0.1 still answers **200**, SSH still works. -- `admin-server-cert` is still `ana-fw.pfi` — the existing cert is untouched and - serves the internal GUI until **2026-10-27**; it simply will not auto-renew. - -Reverse with `config system acme / set interface "wan1"`. - -### RETRACTION: the four VIPs are NOT all-port - -A previous section claimed `Rustdesk`, `https-to-tacticalrmm`, `web-to-webhost` -and `web-to-sfcontainer` were unrestricted all-port static NATs. **They are not.** -A FortiOS VIP can be scoped **two different ways** and the parser used only -checked one: - -1. `set portforward enable` + `set extport ` — a single mapped port, **or** -2. `set service ""` on the VIP object — constrains the VIP to that service. - -All four use form 2. The custom services are narrow: `Rustdesk` = TCP -21115–21119 + UDP 21116 (the standard RustDesk range), `ssh-mapped-2223` = TCP -2223 only. **Every one of the 14 VIPs is scoped; none is unrestricted.** - -**Lesson: absence of `portforward` does NOT mean all-port on a FortiOS VIP — -check `service` too.** Better still, do what settled it here: scan from outside -rather than reading config. - -### Ground-truth public exposure (external TCP scan, post-change) - -| IP | open | maps to | -|---|---|---| -| 38.120.12.41 | *nothing* | — | -| **38.120.12.42** | ***nothing*** | the FortiGate itself — fully closed | -| 38.120.12.43 | 80, 443 | sf-ana-container 10.250.150.100 (SureFire tenant) | -| 38.120.12.44 | 22, 80, 443, 8025, 21115–21119 | gitea (→222), traefik, mailrise, RustDesk | -| 38.120.12.45 | 80, 443, 2223 | pfi-ana-webhost 10.250.50.52 (2223→22) | -| 38.120.12.46 | 443 | pfi-tacticalrmm 10.250.50.57 | - -Configured-but-closed: 8443 (mattermost-calls), 8444 (webdav-nas), 8880 -(Kokoro-In) — VIPs exist, nothing listening behind them. Worth a tidy-up during -the OPNsense translation but not exposure. diff --git a/persistent-memory.d/2026-08-23-hrafn-adopted-ci-frozen-source.md b/persistent-memory.d/2026-08-23-hrafn-adopted-ci-frozen-source.md deleted file mode 100644 index 27c8e4f..0000000 --- a/persistent-memory.d/2026-08-23-hrafn-adopted-ci-frozen-source.md +++ /dev/null @@ -1,63 +0,0 @@ -# [2026-08-23] hrafn adopted; its CI deploy reported green while deploying nothing - -`hrafn` — genuine-Chromium browser-fetch behind a REST API, for bot-gated sites -(Reddit first). Built by nevermore-claude on ana-docker, handed to infra-ops for -uptime ownership. Internal-only on `traefik-net`, no host port; consumers reach -`http://hrafn:8080`. Canonical at `stacks/hrafn/`. - -## Intake found a live credential exposure - -`/opt/docker/compose/hrafn/.env` was mode **0644 with a live 57-char bearer token** -— verified as real exposure by reading it as `nobody` on a box with four -interactive accounts. Tightened to 0600. That triggered the wider sweep (see -[[2026-08-23-ana-docker-env-perms-sweep]]). - -## The CI defect — the one worth remembering - -I authored the deploy (elway playbook + gitea workflow) to replace a hand-rsync, -tagging the image with the commit SHA for provenance. nevermore-claude later found -v1.0.0 deploying "green" while the host still served 0.1.0. - -**Root cause was mine and nastier than either hypothesis.** The staging dir was -`$compose_dir/.stage` — **inside** the rsync target. So -`rsync -a --delete $compose_dir/.stage/ $compose_dir/` deleted `.stage` from the -destination (absent from the source listing) **during** the transfer, destroying -its own source mid-copy. Reproduced exactly: - -``` -before: app.py="OLD" leftover.txt .stage/app.py="NEW" -after: app.py="OLD" leftover.txt GONE, .stage GONE -``` - -Deletion succeeded, the copy silently did not, rsync exited 0. So the directory -*looked* converged while host source stayed frozen at the first manual rsync — -and because the build's `COPY` inputs never changed, Docker full-cache-hit and -every SHA tag aliased one image. **The provenance the tagging existed to provide -was false for the pipeline's entire life.** - -**The real failure is the verification.** The verify steps asserted the marker, -container health, and a 200 from `/readyz` — all of which pass against a -completely frozen host. None measured *content*. A deploy that reports success -without asserting the bytes changed is verifying an **uptime**, not a deploy. - -## Fixes - -- stage at `/tmp/hrafn-deploy-stage`, outside the target -- CI computes `context_sha256` over the shipped file list; the playbook recomputes - it **on the host after the converge** and fails on mismatch -- compare the running container's `src/**/*.py` against the host's, so a SHA tag - cannot name layers the image lacks -- **compare `*.py` only** — `pip install .` generates `src/*.egg-info/*` inside the - image and `__pycache__` appears at runtime, so a naive `find src -type f` compare - false-fails on every healthy deploy. Verified against a known-good container - before shipping (12 host files, 18 in container, 0 content differences). -- declined `--no-cache`: a cache hit is *correct* when the context is genuinely - unchanged; assert the property rather than brute-force it. - -## Access - -Operator granted claude-bot **write** on `vh/hrafn`, so infra-ops maintains the -pipeline it owns instead of routing patches through the repo holder. `vh/hrafn` is -canonical; `stacks/hrafn/ci/` is a verified mirror. - -Commits `b6924de`, `b001d0c`, `11b9d18`, `b38c369`, `9642952`. diff --git a/persistent-memory.d/2026-08-23-selene-retired-alias-collision.md b/persistent-memory.d/2026-08-23-selene-retired-alias-collision.md deleted file mode 100644 index 9e3b004..0000000 --- a/persistent-memory.d/2026-08-23-selene-retired-alias-collision.md +++ /dev/null @@ -1,81 +0,0 @@ -# [2026-08-23] selene seat retired after losing a head-to-head; 7 aliases share one seat - -## Why selene went - -Benchmarked against `gen` on selene's own job — 24 designed judge items with -checkable ground truth, pairwise + absolute modes, 3 repeats, run on **both** a -neutral JSON prompt and Selene's **native Atla template** (288 calls, free local). - -``` -neutral JSON selene 20/24 (83%) gen 23/24 (96%) -native Atla selene 21/24 (88%) gen 22/24 (92%) -``` - -gen won on both templates and **selene's best sat below gen's worst**. Selene was -given its own fine-tuned template as a fairness check before any recommendation; -it gained one point, not three. - -**Decisive defect: selene cannot emit "tie"** — 0/2 on both templates, forcing a -winner on every equivalent pair. For eval work that is the case that matters. - -brokkr-smithy-dev independently corroborated from the other end with a **null -control** (an excerpt compared against ITSELF, where tie is definitional): -`chat-judge`(selene) TIE **27/60 = 45%**, gen **60/60 = 100%**; ground-truth -recovery on real-corpus ranking selene **47% — chance** vs gen 94%. My 83-vs-96 -understated it: on a *ranking* task selene was a coin flip. Absolute scoring on -designed items is an easier task than ranking real text — the harness is a -**screen, not a verdict**, and its README says so. - -Reclaimed **17.2 GiB** on ana-ml2 GPU1 (free 1,818 -> 19,450 MiB). - -## The naming rule, restated the hard way - -I proposed repointing `selene-1-mini-8b` at gen and was **correctly overruled**: - -> never repoint a named model at a different model's endpoint — that is -> intentionally misleading - -`chat-judge` is a **role** alias (ADR-0012: consumers bind the capability) and -moved to gen with a deterministic judge profile copied from `image-judge`. -`selene-1-mini-8b` is a **model** name and was removed outright — it now returns -`HTTP 400 Invalid model name`, verified. The discriminator: *does the string -promise a capability, or an identity?* - -## The 7-way alias collision — the finding with the longest reach - -``` -chat-judge classifier gen image-judge -qwen-image-bench summarizer summarizer-large -> qwen3.8-27b-uncensored :8015 -``` - -Also colliding: `gen-frontier`/`gen-frontier-reasoning`/`glm-5.2`/`glm-5.2-reasoning`; -`ext-tts`/`gpt-4o-mini-tts`/`tts-1`/`tts-1-hd`; `reranker`/`reranker-a3-bge-v2-m3`. - -**Cross-checking a result against another alias measures nothing when they are the -same weights — agreement is an echo, not corroboration.** Documented at the head of -`model_list` in the live gateway config, because it belongs where people read it. - -This caught a real defect within hours: brokkr's R47 premium-corpus gate was about -to run ~46,000 record-exposures against `gen` with `summarizer` shortlisted as an -independent second opinion. They pinned the backing model in the preregistration -and dropped the second-alias idea instead. - -## Provenance seam (brokkr's pushback, adopted) - -The gateway returns the **alias** in the response `model` field, not the backing -model — so a per-call guard catches a swap *during* a run and is blind to one -*between* runs. **Role alias for routing, concrete model for provenance.** -`GET :4000/model/info` with the shared key already exposes backing model + -api_base; resolve at run start AND end and void on mismatch. - -## Artifacts - -- Harness kept at `tools/judge-bench/` (`--models` REQUIRED — a stale default - would silently benchmark a retired seat). -- `stacks/selene/` keeps compose + a README explaining the retirement. -- Technique worth stealing, from brokkr: **a control constructed so the correct - answer is DEFINITIONAL rather than judged cannot inherit the designer's error.** - Item vs itself; response vs its own truncation; text vs its own clauses - permuted. Add those before adding more judged items. - -Commits `ca3c984`, `b8a5355`. diff --git a/persistent-memory.d/2026-08-23-worldtree-b187-pins-matrix.md b/persistent-memory.d/2026-08-23-worldtree-b187-pins-matrix.md deleted file mode 100644 index 7c726e0..0000000 --- a/persistent-memory.d/2026-08-23-worldtree-b187-pins-matrix.md +++ /dev/null @@ -1,70 +0,0 @@ -# [2026-08-23] Worldtree b187 shipped; all three instances de-armed from a 69-day-stale `:latest`; Matrix homeserver re-plumbed - -## b187 pre-stage (#405 phases 1+2) - -The matrix bridge stopped embedding the engine and became an HTTP client of the -Conversation API, so `WORLDTREE_API_URL` became **boot-blocking** — absent from the -container env, the bridge exits by design. Demo's compose never passed it; the next -recreate would have crash-looped. Pre-staged on demo and personal (additive, backed -up, verified with `docker compose config`, nothing restarted). - -**Key decision, and I got its scope wrong first.** I argued demo should stay keyless -(no homeserver -> no rooms -> no turns -> no 401s). Right about turns, **wrong about -scope**: the engine preflight authenticates at boot regardless of homeserver, so demo -booted permanently degraded. Corrected — key `341c1488` minted under worldtree-dev's -recorded authorization, vaulted, wired, three-hop hash-verified. - -## The 69-day-stale `:latest` landmine - -All three instances pinned `WORLDTREE_IMAGE=.../worldtree:latest` in `.env` while -running SHA-tagged images built that day. Local `:latest` = `b19afd71d7cc`, built -**2026-06-14**. So ANY `docker compose up` — anyone's, for any reason — silently -downgraded that service by 69 days. Same footgun as the 2026-06-15 outage. - -Re-pinned all three to their running SHAs (Worldtree #410), verified by rendering -compose config rather than reading `.env`, containers untouched. Playbook at -`playbooks/repin-worldtree-image.yaml`. - -**`worldtree-pinned` was the worst case:** the instance whose entire purpose is being -frozen was running a **dangling image with no repo tags**, kept alive only by the -running container. One `docker rm` from garbage collection. Tagged -`:446e5807bf43` first, then pinned. - -The guard I wrote had two bugs the pinned case exposed: it compared the container's -`.Config.Image` **string** (only the tag it was CREATED from — pinned was created -from `:latest` back when that meant 446e5807), and it reported CHANGED -unconditionally. Now compares **image IDs** and skips when already correct. - -## Matrix homeserver ownership - -Operator ruled: **personal owns the Matrix bridge.** The appservice tokens were never -missing — both sat at length 64 in the vaulted dev `env.sh` while both deployed -instances had them at length **zero**. Someone wired four of six Matrix vars and -stopped. Wired them into personal, three-hop verified. - -**The trap worth remembering:** Synapse's registration pointed at -`http://10.100.10.50:8009` — nh3-dev, a dead epoch, with transaction 2801 queued at -512s backoff. The natural fix (swap the IP) gives `10.250.50.152:8009` which is -**DEMO's** bridge, and Synapse can reach both — it would have connected, delivered, -and looked correct while routing the operator's live rooms to the demo instance. -**Personal's bridge is :8010.** `docker port` is ground truth. - -Corrected the URL, restarted Synapse (healthy in 32s after 3.5 months up), verified -`GET /_matrix/app/v1/ping -> 200` from inside the Synapse container. worldtree-dev's -smoke passed first try: room created, mimir accepted the invite, a real engine turn -ran, mimir replied in persona voice. #408 closed. - -## Open on worldtree-dev's side - -- **#411** — personal's bridge logs `Debug sink init failed: Permission denied: - /app/sessions/debug_rooms.json`. It creates two debug rooms but cannot persist - their IDs, so **every restart mints a fresh pair on the live homeserver**. Room - litter that compounds silently. Needs a which-container-writes-what check on the - sessions volume before anyone chowns it. -- Bridge/engine agent-roster drift: 6 of the bridge's 9 configured agents are not - listed by the engine on either instance. -- Historical Domari pairwise verdicts from the selene era are coin-flip-grade - (see [[2026-08-23-selene-retired-alias-collision]]); worldtree-dev banked that so - no future arc leans on them without re-judging. - -Commits `064181a`, `bb19a96`. diff --git a/persistent-memory.d/2026-08-24-ana-gw-admin-closed-acme-disabled.md b/persistent-memory.d/2026-08-24-ana-gw-admin-closed-acme-disabled.md deleted file mode 100644 index ee441cf..0000000 --- a/persistent-memory.d/2026-08-24-ana-gw-admin-closed-acme-disabled.md +++ /dev/null @@ -1,50 +0,0 @@ -# [2026-08-24] ana-gw public admin surface closed to zero, ACME listener included - -WAN admin was opened at the start of the session as a cutover contingency -("so I don't have to drive down there"), then closed again on operator -instruction once the AES-128 work landed. Net result: **the FortiGate's WAN -address now exposes no TCP port at all.** - -## Final state - -External scan of `38.120.12.42`, 55 ports: **nothing open**. Verified from two -sites. `wan1 allowaccess` = `ping`; `infra-ops` trusthost back to `10.0.0.0/8`. - -**Consequence to hold: there is no out-of-band path to ana-gw.** If both tunnels -drop it is console-only. Re-open is two one-liners (allowaccess + trusthost) — -both are recorded in auto-memory `reference_fortigate_ana_gw_access`. - -## Port 80 was the FortiOS ACME listener, and I got it wrong first - -`38.120.12.42:80` answered a bare 403 (`ACME Access Only`, 101 bytes) with -`allowaccess` set to ping only. First diagnosis — "an ISP transparent proxy" — -was **wrong**, and the reason is worth keeping: - -> The sniffer filter was `dst host 38.120.12.42 and tcp port 80`. **`dst host` -> matches inbound only**, so outbound SYN-ACKs were excluded *by construction*, -> and concluding "the box sends no SYN-ACK" from that capture was unsound. - -Re-run bidirectionally (`host … and tcp port 80`) it immediately showed -`wan1 out 38.120.12.42.80 -> : syn ack`. **Rule: to test whether a box -*answers*, the filter must be bidirectional.** - -The listener is opened by `config system acme / set interface "wan1"` and -**bypasses `allowaccess` by design** — FortiOS needs port 80 for HTTP-01. It -was disabled (`config system acme / unset interface`); the LE cert (`ana-fw.pfi`, -valid to 2026-10-27) is untouched and simply stops renewing, which is fine -because WAN admin is closed and the box is being replaced. - -## Retracted in the same pass: the "four all-port VIPs" alarm - -Claimed four VIPs were unrestricted all-port static NAT. **False.** A FortiOS -VIP is scoped **two** ways — `portforward`+`extport`, *or* a `service` binding -on the VIP object — and only the first was checked. All 14 VIPs are scoped; -`Rustdesk` is TCP 21115–21119, `ssh-mapped-2223` is TCP 2223 only. - -Ground-truth external scan of all six public IPs is recorded in -`reference_fortigate_ana_gw_access`. Configured-but-dead: `:8443` -(mattermost-calls), `:8444` (webdav-nas), `:8880` (Kokoro-In) — tidy-up -candidates for the OPNsense translation, not exposure. - -**Lesson, twice in one session: measure from outside instead of parsing config.** -Both wrong answers came from a filter that answered a different question. diff --git a/persistent-memory.d/2026-08-24-charrp-gemma4-moe-swap-and-trainee.md b/persistent-memory.d/2026-08-24-charrp-gemma4-moe-swap-and-trainee.md deleted file mode 100644 index ec6e976..0000000 --- a/persistent-memory.d/2026-08-24-charrp-gemma4-moe-swap-and-trainee.md +++ /dev/null @@ -1,167 +0,0 @@ -# `[2026-08-24]` char-rp seat: OOM root-cause, Gemma-4 MoE swap, and the abliterated trainee base - -One evening, one thread with brokkr-smithy-dev, five commits: `850e0c3`, -`27155c0`, `f509668`+`24e8826`+`1bd90ea`+`3446367`+`8d6a939`, `14ff4a3`, -`019ccff`, `5415fd4`. - -## 1. The seat was crash-looping, and the cause was NOT its config - -`vllm-meromero-rp` reported up-but-unreachable, RestartCount climbing (13 by the -time it was examined, not the 4 first reported). Startup logs looked clean all -the way through weights, `torch.compile` and CUDA-graph capture, then: - - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 336.00 MiB. - GPU 0 has a total capacity of 94.97 GiB of which 195.19 MiB is free. - -**⚠ `--gpu-memory-utilization` SIZES THE KV CACHE AND DOES NOT COVER CUDA -CONTEXT, GRAPHS OR NON-TORCH OVERHEAD.** gen is configured at 0.43 and actually -held 45.6 GiB. char-rp was at 0.51. The pair was committed to 0.94 of the card -with ~0.6 GiB of real headroom — it fit on the 21st and stopped fitting on the -24th. - -Dropped char-rp to 0.47: ~4.8 GiB margin, KV 27.36 → 23.56 GiB, 430,825 → -371,023 tokens against a max-model-len of 262,144. **Cost nothing usable** — the -pool still holds 1.4x a full-length sequence; what is lost is concurrent long -requests, not context. - -**⚠⚠ THE MISSING HALF, found later that evening: gen's footprint GROWS WITH -UPTIME.** Same container, same 0.43: **45.6 GiB after ~3 days up, 38.5 GiB -freshly restarted** — ~7 GiB apart. Nothing about char-rp changed between the -21st and the 24th; *gen crept up underneath it*. **Headroom arithmetic done -against a long-running gen is measuring a moving number.** Measure against a -freshly-restarted one. - -## 2. `char-rp` and `char-rp-reasoning` are ONE seat, not two - -Both LiteLLM routes point at `10.250.50.54:8016/v1` — `hosted_vllm/char-rp` and -`hosted_vllm/char-rp-thinking`. brokkr had reported 30/80 and 80/80 failure -rates as two failing services; it was one outage sampled twice. This also -*improved* a result of theirs: their CoT on/off battery had assumed both aliases -were the same weights under two chat templates, and the routing detail turned an -assumption into a verified fact. - -(`vllm-charrp-reasoning-nvfp4`, the Heretic2 NVFP4+MTP container, has been -stopped for 12+ days and is unrelated — it is not what that alias resolves to.) - -## 3. The seat swapped to the Gemma-4 26B-A4B MoE - -Operator-directed straight-across replacement: same port, same -served-model-names, so no gateway route or consumer config moved. Rationale is -throughput under CoT — the user waits through the whole reasoning block before -the first visible token, and the MoE measures ~114 tok/s @32K against the dense -31B's ~40.7. - -Serving copy is `RedHatAI/gemma-4-26B-A4B-it-NVFP4` (16 GB), chosen over the -other `-it` quants because it is compressed-tensors (`nvfp4-pack-quantized`) — -the same loader path the outgoing seat used. Smaller weights at the same 0.47 -budget bought **1,724,110 KV tokens against the predecessor's 371,023**. - -`meromero-charrp` is retained stopped in `created` state, labelled -`AI - Dormant`. Both stacks bind `:8016`, so rollback is **stop-then-start**. - -## 4. ⚠ THE STALE-CHAT-TEMPLATE TRAP IS ENDEMIC, NOT A ONE-OFF - -Verified by hash across every third-party Gemma-4 derivative pulled: - -| build | lines | sha256 (normalised) | -|---|---|---| -| upstream `google/gemma-4-26B-A4B-it` | 390 | `6a1015c47ccfcfa6` | -| RedHatAI NVFP4 (served) | 389 | `6a1015c47ccfcfa6` — the only match | -| llmfan46 heretic | 365 | `0a52be69cda5ab8a` | -| TrevorJS abliterated | 266 | `58c66fdee4afa297` | -| jenerallee78 abliterated | 266 | `58c66fdee4afa297` | -| prithivMLmods NVFP4A16 | 266 | `58c66fdee4afa297` | - -Three independent repos carrying the *identical* stale file means it propagated -through the ecosystem. Consequences differ by use and **both are silent**: -serving a mismatched template renders a different prompt; **training through -`base/chat_template.jinja` means training on a different prompt format than -production serves** — train/serve skew, no error, presents as a tuning failure. - -The production compose now pins the template explicitly. It is a **no-op for the -served weights** (the A4 build ships that exact file) and permanently closes the -class. ⚠ If `GEMMA4_MODEL` ever points at a different checkpoint, the template -default must move with it. - -## 5. A benchmark result was RETRACTED — below chance indicts the instrument - -A battery appeared to show Gemma at **12% contradiction detection with CoT off -against gen's 81%**. An A16 activation-precision control was staged to test -whether the quant scheme owned it. Then the operator asked to see the individual -items, and the item was **ill-posed**: it presented two mutually contradicting -statements and asked for "*the* contradicting statement", but **contradiction is -symmetric**. The model consistently named the absolute claim — a defensible -reading the labelling scored wrong every time. - -**⚠ THE TELL WAS IN PLAIN SIGHT: 12% ON A FIVE-OPTION TASK IS BELOW THE 20% -CHANCE FLOOR.** A below-chance score indicts the instrument before it indicts -the model, and neither side reacted to it. I spent the afternoon verifying repo -names, config fields, template hashes and tokenizer vocabs — every layer of -plumbing — and never asked whether the number itself was *possible*. **A -preflight can be thorough and still be aimed in the wrong direction.** - -Retracted: "the model owns the contradiction deficit"; "domain tuning costs 43 -points of contradiction detection" (on a sound instrument it **reverses**); all -pre-fix T2 numbers. Recorded as a dated superseded-claims table in -`stacks/gemma4-charrp/README.md` rather than a silent edit. - -**What survived:** the A16 control result — activation precision is close to free -on this battery, every other task identical across W4A4 and W4A16 builds. - -## 6. The abliterated trainee base — measured, not assumed - -Operator directed a low-damage abliterated instruct build. "Low damage" was -treated as a measurable claim; the field spreads from KL 0.09 to 0.4118: - -| build | method | KL | refusals | -|---|---|---|---| -| **llmfan46** (operator's pick) | Heretic v1.2.0 ARA | 0.1237 | 3/100 | -| TrevorJS | ARA-family | 0.09 | 1/100 effective, 5/686 cross-dataset | -| jenerallee78 | ARA 2-pass | 0.1299 | 7.7% StrongREJECT | -| huihui-ai | remove-refusals | none published | none published | - -Fleet anchor: our own work found **Heretic at KL 0.12 preserved the MTP head at -83.7% acceptance**, so both staged builds sit inside an already-measured band. -huihui-ai rejected — no metrics, its card calls the method "a crude, -proof-of-concept implementation", it abliterates both thinking and non-thinking -modes, and its parameter count runs ~738M over upstream. Operator's independent -read matched ("huihui produces garbage"). - -**Abliteration isolated properly** (stock BF16 vs llmfan46 BF16, same precision, -same pinned template, same 192 items): - - T2 contradiction 75% → 59% (−5 items) - T6 spatial 75% → 88% (+4 items) - core 90.0% → 89.4% (−0.6 pts) - -**It MOVED capability rather than removing it** — five lost on contradiction, -four gained on spatial, nearly cancelling. Nobody predicted a gain. **llmfan46 -stands**; no case for re-staging on TrevorJS over 0.6 points. - -⚠ Read as ~5 and ~4 items at n=32, not as −15.6/+12.5 percent. ⚠ Says nothing -about quantization — the stock-NVFP4 T2 figure came from n=16 against n=32, -different item sets, n-confounded. - -## 7. ⚠ The production compose hardcodes `--quantization compressed-tensors` - -Pointing the char-rp stack at unquantized BF16 weights crash-loops immediately: - - TypeError: CompressedTensorsConfig.__init__() missing 3 required - positional arguments: 'target_scheme_map', 'ignore', 'quant_format' - -vLLM trying to read a quantization config out of a checkpoint that has none. 35 -restarts before it was caught. Hence `stacks/gemma4-trainee-bench/` — a separate -ephemeral stack with no quantization flag, `restart: "no"` so a bench seat cannot -resurrect itself and block gen's restore, and no homepage labels so it leaves no -permanently-offline card. That detour is why a base swap is now ~5 minutes -instead of ~15. - -## 8. BF16 cannot coexist with gen - -48.07 GiB of BF16 weights plus gen's footprint exceeds the 94.97 GiB card before -a byte of KV cache. Every BF16 bench window means **gen is stopped**. Two such -windows were run and gen restored both times; the restore was triggered by -observing the seat's own throughput logs (a large prefill burst then zero -running/zero waiting) rather than waiting on a courtesy message. - -Cross-links: [[2026-08-24-homepage-uniform-grid]] diff --git a/persistent-memory.d/2026-08-24-esh-dns-rdnss-and-scheme-live.md b/persistent-memory.d/2026-08-24-esh-dns-rdnss-and-scheme-live.md deleted file mode 100644 index bc94e3c..0000000 --- a/persistent-memory.d/2026-08-24-esh-dns-rdnss-and-scheme-live.md +++ /dev/null @@ -1,77 +0,0 @@ -# [2026-08-24] ESH DNS fixed at the IPv6 layer, and the naming scheme went live - -Reported as "`scriberr.ana.internal` doesn't resolve on my Mac, and nslookup -shows an IPv6 DNS server." Operator's diagnosis was right; the fix took three -wrong turns worth recording. - -## Root cause - -`esh-userland` has IPv6 PD with RA at `pref high`, and the UDM advertises -**itself** as the resolver via RDNSS. macOS honours RDNSS and prefers it over -the DHCPv4-supplied resolver, so queries went to the UDM — which does not know -`.internal` — and returned NXDOMAIN. AdGuard was never consulted. - -Two adjacent gaps found while there: `esh-userland`'s **secondary** v4 resolver -was `10.0.10.1` (the UDM itself), and `esh-server` had **DNS handout disabled -entirely**, so every host there got the UDM and could never resolve `.internal` -— esh-docker-vm was living proof. - -## The three wrong turns - -1. **`dhcpdv6_dns_auto=false` alone does nothing.** It is only honoured **when an - explicit server is supplied**. Setting it bare looks like a no-op and invites - the conclusion that the field is inert — which I drew, wrongly. Despite the - `dhcpdv6_` prefix it *does* drive the RA's RDNSS option on a SLAAC network. -2. **`wan_dns1` is NOT used by the UDM's LAN-facing forwarder.** Setting it to - AdGuard persists, reads back, and changes nothing. Proven with **fresh - uncached ad domains** — AdGuard blocklists answer `0.0.0.0`, the UDM returned - real IPs. Reverted. -3. **`force-provision` returns `rc:ok` and proves nothing** — consistent with the - known `cmd/devmgr` behaviour. - -Every failed attempt returned `rc: ok`. **Verify by observed effect.** RAs were -probed with a stdlib raw-socket Router Solicitation parsing option type 25 -(`rdisc6`/`tcpdump` were both absent; nothing was installed). - -## What landed - -RDNSS **redirected** rather than disabled — better than switching it off: - -| VLAN | v4 | v6 RDNSS | -|---|---|---| -| `esh-userland` | 10.0.50.45 + 10.100.50.40 | `…:4411:b105:50:45` | -| `esh-server` | 10.0.50.45 + 10.100.50.40 | `…:4411:b105:50:45` | - -The v4 secondary moved from the UDM to the **NH3 AdGuard** — reachable over -Site Magic and authoritative for the zone. ⚠ **A secondary only fails over on -SILENCE, not on wrong answers**: NXDOMAIN is a *successful* answer, the client -accepts it and never retries. A secondary that doesn't know your private zone is -a coin flip, not a spare tyre. `esh-cameras` deliberately untouched — routing -camera DNS through AdGuard's filtering risks their cloud features. - -## The naming scheme became real - -The resolver address is the scheme's first live use, replacing a MAC-derived -SLAAC address that would break on a NIC change. All three `esh-server` Linux -hosts now carry `4411:B105` ("FOR ALL BIOS"): - -``` -esh-docker-vm …:4411:b105:50:45 esh-pve-nas …:4411:b105:50:55 -esh-vm-db …:4411:b105:50:60 -``` - -Applied by an `if-up.d` hook that **derives the prefix at runtime** (self-heals -on re-delegation), backgrounds itself with a retry (SLAAC may not have landed; -a blocking hook would stall bring-up on a headless box), and adds nothing to -existing config. **Not** an `iface … inet6 static` stanza — on Debian that sets -`accept_ra=0` and would strand the host. - -⚠ **Proxmox bridges need `accept_ra=2`.** `esh-pve-nas` had link-local only -despite every sysctl looking right: `vmbr0.forwarding=1`, and the kernel ignores -RAs on a forwarding interface unless `accept_ra` is explicitly `2`. Fixed with -`accept_ra_defrtr=0` alongside, so it takes the prefix but **declines the default -route** — an IPv6 identity with no change to a hypervisor's routing. Expect this -on every Proxmox node when its LAN gets v6. - -Canonical: `docs/pfi/ipv6-naming-scheme.md`. UniFi limits: -auto-memory `reference_unifi_dns_rdnss_limits`. diff --git a/persistent-memory.d/2026-08-24-homepage-uniform-grid.md b/persistent-memory.d/2026-08-24-homepage-uniform-grid.md deleted file mode 100644 index 9dda21c..0000000 --- a/persistent-memory.d/2026-08-24-homepage-uniform-grid.md +++ /dev/null @@ -1,308 +0,0 @@ -# `[2026-08-24]` Homepage: remote-label consumption re-verified, then the board relaid out on a uniform grid - -Prompted by the operator: *"Homepage on esh-vm-docker lists remote dockers and -can absolutely consume their labels, please verify again. I am still -unsatisfied with the layout and aesthetics."* - -## The verification — the operator was right, and the record now says so - -**Homepage on `esh-docker-vm` discovers services by container label from all -five Docker engines in `conf/docker.yaml`, not just its own.** This is not an -inference; `GET /api/services` returns every card's `server` field, and the -2026-08-24 snapshot resolves to: - -| `server` | host | label-discovered services | -|---|---|---| -| `ana-pfi-docker` | 10.250.50.70 | 30 | -| `irv-ml1-docker` | 10.100.79.3 (over WireGuard) | 15 | -| `ana-ml2-docker` | 10.250.50.54 | 14 | -| `esh-vm-docker` | 10.0.50.45 (the dashboard's own host) | 13 | -| `nh3-pfi-docker` | 10.100.50.40 | 2 | - -**74 of 107 cards are label-discovered, and only 13 of those come from the -dashboard's own engine** — the other 61 are read off four remote hosts, -including irv-ml1 across the WireGuard tunnel. The remaining 33 carry -`server: null`: those are the manual `services.yaml` entries — hardware, BMCs, -hypervisors, printers, and user-level systemd services that have no container -to label in the first place. **That null is the only thing "not label-driven" -about this dashboard**, and it is a property of the entry, not of the host it -points at. - -⚠ If a future session doubts this again, the check is one command and takes two -seconds — do not reason about it from the docs: - -```bash -curl -s http://10.0.50.45:5100/api/services \ - | jq -r '.[].services[] | .server' | sort | uniq -c -``` - -## What was actually wrong with the layout - -Measured with Playwright against the live board (per-group `card=` width, card -height spread, and a geometric title-vs-status overlap test), not judged by -eye: - -- **Card width changed at every group boundary.** `columns:` is not a density - dial — it sets `lg:grid-cols-N` for one group, so it fixes that group's card - width. Notes rendered a single **1464px** card; News and Media **728px**; - Eval & Retrieval **286px**; everything else 360px. Scrolling the page, the - grid resized five times. -- **Long names printed underneath their own status pill.** Measured by - re-injecting the old rule and testing the title text node's box against the - status cluster's box: **6 cards, all on the AI tab** — 3 in Inference, 2 in - Dormant, 1 in Eval & Retrieval; zero on the other three tabs, which is why - it survived earlier passes. Root cause is a genuinely counter-intuitive one: - the rule reserved a - 78px gutter with `padding-right` and relied on `overflow: hidden` to hold it, - but **overflow clips at the padding box, not the content box** — so the - reserved gutter was spill room the title rendered straight through. The - intended `text-overflow: ellipsis` never fired either, because the ellipsis - is painted by whichever block's own line overflows, and here that is the - anonymous box around the bare title text node, which does not carry - `overflow`. -- **`AI Systems` / Scriberr was on all four tabs** — the 2026-08-18 UltraSeedbox - bug recurring, this time arriving from a container label rather than from - `settings.yaml`. -- **Icons were grey smudges.** Homepage masks every glyph over - `--color-logo-start/stop`; stock slate-400 → slate-700 sinks the bottom half - of each icon into the card fill. -- Bookmark groups and Jellyfin's trailing stream rows were the two components - the theme had never reached. - -## The fixes - -`stacks/homepage/conf/settings.yaml` — **all 20 groups to `columns: 4`.** -`stacks/homepage/theme/australis.css.in` → rebuilt → `conf/custom.css`: -gutter held by wrapping, description clamped to 3 lines (floor still 2), icon -ramp overridden, bookmark + trailing-widget components themed, group gap -10px → 22px. `stacks/scriberr/compose.yaml` — `homepage.group` → `AI - Audio -Tools`, container recreated on ana-ml2. - -After: **every group renders at card=360**, and the collision count is zero. - -Before/after, all four tabs: `http://10.100.10.50:8090/b/homepage-relayout/` -(24h TTL; also on the standing link board). - -## ⚠ Three traps worth carrying forward - -1. **"Columns = member count" is RETIRED** (it was the 2026-08-18 rule). It was - avoiding dead cells in a short last row and bought a worse defect. A short - last row is what a grid looks like; a card wider than its neighbours is what - a mistake looks like. -2. **A `:root` override of a Homepage theme variable is silently ignored.** - Homepage sets `--color-logo-*` on `.theme-slate`, and that class is on the - `` element — the same element `:root` matches. `.theme-slate` (0,1,0) - beats `:root` (0,0,1), so the override does nothing and looks like the - variable is not the one in play. `html[class]` (0,1,1) wins, and does not - hard-code which `theme-*` class is active. Specificity alone is not enough - either: a custom property resolves from the *nearest* ancestor that sets it, - so the override has to land on ``, not on ``. -3. **The post-recreate tab-bar loss is INTERMITTENT, not guaranteed.** The - 2026-08-19 note reads as though every recreate costs up to an hour of broken - render. This recreate came up correct within 10 seconds — fresh payload on - the first poll, all four tabs clickable a minute later. Recreate, *check*, - and only then walk away if it is actually in the broken state. - -Also re-confirmed, since the change depended on it: **a `settings.yaml` edit -needs a container recreate, not a restart.** `docker restart homepage` left the -old `"columns":1` payload embedded in the served HTML with the correct file -mounted and readable inside the container; `compose up -d --force-recreate` -cleared it immediately. - -## Deliberately not done — operator's call - -The Main tab still opens on three sparse bands: **Notes** (1 member) and -**Games** (1) each burn a full 4-wide row, and **News** has 2. Merging Notes + -News, or folding Games into Apps, would tighten the top of the page — but that -is information architecture, not layout, and the group names are the operator's. -Surfaced rather than done. - -→ **Resolved in pass 2 below**, where the operator delegated the naming -("re-categorize however you want"). Notes + News became `Daily`, Games folded -into `Apps`, and the `AI - Audio Tools` placement in this pass was superseded -(Scriberr moved on to `AI - Studios`). - ---- - -# `[2026-08-24, pass 2]` Recategorised on "do I open this?", API groups collapsed - -Operator, after seeing pass 1: *"You can re-categorize however you want. -service networking tab is uneven, you can split out the adguard cards, etc. -most of the issues are that tools I use and have a UI are interspersed with API -endpoints which are largely informational only. They might even go in their own -cards or start collapsed."* - -## The axis - -Every group is now either **tools** (expanded, top of tab) or **endpoints** (an -API, a broker, an agent — `initiallyCollapsed: true`, bottom of tab). A -collapsed group still renders its eyebrow and rule, so presence costs one line -instead of two rows. - -Second, quieter rule that fell out of the same pass: **a group's members should -all carry a widget or none should.** A stat strip adds ~50px, so one widget card -in a row of plain ones opens a void under the plain ones — which is most of what -made the 13-member `Service Networking` band look broken. - -## Shape - -- **Main** — `Daily` (Memos, Miniflux, Nevermore, SearXNG — replaces the - 1-card Notes and 2-card News bands), `Monitoring`, `Apps` (12; absorbed the - 1-card `Games` band), `Media`, `UltraSeedbox`. -- **AI** — `AI - Gateways & Chat` (8) and `AI - Studios` (6) expanded; then - `AI - Inference` (7), `AI - Eval & Retrieval` (4), `AI - Speech (TTS)` (4), - `AI - Audio Tools` (2), `AI - Dormant` (6) all collapsed. -- **Toolchain** — `DNS & Filtering` (3), `Reverse Proxies` (2), - `Compose Consoles` (5), `Toolchain` (3), `Agents (no UI)` (6, collapsed). -- **Infrastructure** — unchanged; every card there is already a console. - -Measured after: every group `card=360`, and `DNS & Filtering` and -`Reverse Proxies` both `h=134..134` — dead flush. - -## ⚠ The move that made it affordable - -**The sixteen GPU-backed model seats were NOT relabelled.** `homepage.group` is -read at container **creation**, so renaming `AI - Inference` to something -clearer would have meant recreating six vLLM seats plus four eval seats plus -four TTS engines — multi-minute model reloads on endpoints peers reach through -the gateway. Order plus `initiallyCollapsed` buys the same separation for free, -so the names stay ugly on purpose. **Do not spend that recreate on a label.** - -28 containers *were* relabelled — all cheap web services — via five rerunnable -elway playbooks, `playbooks/homepage-regroup-.yaml`. The canonical -`stacks/` tree was synced to match afterwards, so intent and reality agree. - -`initiallyCollapsed: true` is a per-group key in `layout:`; confirmed present in -this build (`defaultOpen: !(group?.initiallyCollapsed ?? global)` in -`/app/.next/server/pages/index.js`). - -## AdGuard (ANA) gained its widget, and the credential is fleet-wide - -It was the only AdGuard without a query/blocked/latency strip, so it sat short -beside two tall siblings. **One `infra-ops` AdGuard login authenticates against -all three instances** (ANA `:8053`, NH3 `:8080`, ESH `:8080` — all returned 200 -on `POST /control/login`, verified 2026-08-24). Vaulted at -`secret get nh3-dev/adguard-infra-ops-password`; written to -`/opt/docker/compose/adguard-ana/.env` (0600, root) and never into git. Its icon -was also the odd one out (`mdi-dns` against two `si-adguard`). - -## ⚠⚠ `initialSettings":{}` — the tab-bar mystery is a SWALLOWED EXCEPTION - -The biggest durable finding of the day, and it cost ~25 minutes. Full write-up -in `stacks/homepage/README.md`; the short version: - -`initialSettings":{}` in the served HTML is **the catch branch** of the page's -data loader, not a warm-up and not a cache. And the error can vanish without -trace: the logger is assigned as the first statement *inside* the same `try`, -and the `catch` only logs `if (logger)`. If the logger is what threw, nothing is -written anywhere — which is exactly what was observed. - -Ruled out by measurement, do not re-run: `/api/services`, `/api/bookmarks`, -`/api/widgets` and `/api/hash` all return **200 with correct content** while the -page serves `{}`; restoring the previous known-good `settings.yaml` reproduces -it identically; `/api/validate` returns `[]`; disk and permissions are fine. - -**One-command test:** -`curl -s http://10.0.50.45:5100/ | grep -o 'initialSettings":[^,]\{0,20\}'` - -**What broke the streak:** three consecutive recreates came up empty, then -rolling the 8.6 MB `conf/homepage/logs/homepage.log` aside and recreating healed -it within 15 seconds. That is one observation, not proof — but it is a coherent -mechanism (oversized log → logger init throws → silent catch) and it is the -cheapest thing to try first next time. - ---- - -# `[2026-08-24, pass 3]` Rebuilt on Australis Skyfall — dual theme, light shipped - -Operator supplied the Skyfall design-system README and said "Go full with -skyfall." - -## The bundle was already in this repo's git history - -**The Skyfall tokens did not need to be hunted down.** A predecessor vendored -them on 2026-08-19 and a later commit deleted them; git kept everything: - -```bash -git show 45c1995:stacks/homepage/theme/colors.css # 208 lines, BOTH themes -git show 45c1995:stacks/homepage/theme/layout.css # calm-depth tokens -git show 45c1995:stacks/homepage/theme/typography.css -git show 45c1995:stacks/homepage/theme/fonts/Supreme-{400,500,700}.woff2 -``` - -`colors.css` carries `:root` (dark) **and** `[data-theme="light"]` (Skyfall -Day) — so the light ramp is canonical, not derived. That killed the entire -objection from the previous answer, which was correct only about the -`australis-design` skill ("Always dark first. No light mode in this system"). -**Skyfall is the dual-theme derivative; australis-design is the terminal -theme. They are different systems and only one of them has a light mode.** - -## ⚠⚠ REMOVING `theme:` FROM settings.yaml BREAKS THE DASHBOARD - -The documented way to get Homepage's own light/dark toggle is to leave `theme:` -unpinned. **Do not.** With the key absent, the page's data loader throws and its -catch branch serves `initialSettings: {}` — no tab bar, no layout, no i18n. - -Measured, not inferred: six force-recreates over seven minutes all came up -empty with the key removed; restoring `theme: dark` rendered correctly on the -next recreate in **12 seconds**. `/api/services` stays 200 and fully correct -throughout, which is exactly why this reads as a caching or warm-up problem and -is not one. - -This is the first *confirmed* trigger for the long-running "tab bar goes -missing" mystery. It does not explain every occurrence (the symptom has -appeared with `theme:` present), but it means **the first diagnostic step is -now `git log -p -- stacks/homepage/conf/settings.yaml`**, not container -archaeology. Also retires an earlier lead from this same session: rolling the -8.6 MB `homepage.log` aside once coincided with a recovery, but did nothing -during the `theme:`-key episode — coincidence, not cause. - -## So the toggle is ours - -`conf/custom.js` renders it (was an empty placeholder). Precedence: - -1. explicit choice — `localStorage['skyfall-theme']`, written by the toggle; -2. OS preference — `@media (prefers-color-scheme: light)`; -3. dark — Skyfall's default. - -`theme/build.py` re-emits each vendored `[data-theme="light"]` block twice: as -`[data-theme="light"], html.light`, and inside the media query scoped to -`html:not([data-theme="dark"]):not([data-theme="light"])`. **That `:not()` pair -is what lets a stored *dark* choice survive a light-mode OS.** Verified across -both OS preferences: load, click, click again, reload — all four correct. - -⚠ Homepage keeps its own `class="dark scheme-dark theme-slate"` on `` -regardless, because `theme:` is pinned. That is fine and was checked -explicitly: with the dark class present AND `data-theme="light"`, every themed -surface resolves to Skyfall Day, because our rules carry `!important` on the -surfaces Tailwind's `dark:` variants would otherwise claim. **`data-theme` is -the control surface; the class is not.** - -## The anti-fork guard is now mechanical - -`build.py` records the SHA-256 of each vendored file and **fails the build** on -a mismatch, rather than warning. A vendored file is either byte-identical to -the bundle or it is a fork wearing the bundle's name. Overrides go in -`skyfall.css.in`, which is written entirely against the semantic layer -(`--surface-*`, `--text-*`, `--border-*`, `--success/--danger/--warning`) — no -raw family tokens, no colour literals. - -The one place a literal is unavoidable: Homepage consumes -`--color-logo-start/stop` as `rgb(var(--x))`, which cannot take an `oklch()`. -Those four values are exact sRGB conversions of real tokens (`--sea-80`, -`--blue-base` for dark; `--sea-40`, `--blue-deep` for light), computed rather -than eyeballed, with the conversion recorded in the file. - -## Deviations, all deliberate and all written down - -- **The aurora ribbon under the tab bar is gone.** Skyfall sanctions exactly two - accent expressions — the active rail and hero-only glows — and a decorative - gradient across the chrome is neither. The colour moved to a 2px accent bar - plus `--accent-soft` fill on the active tab, which *is* the rail. -- **Widget stat values moved from the display face to mono**, per Skyfall's - "numbers and telemetry are always `--font-mono`". -- **Two font substitutions**: Space Grotesk for Bespoke Sans, JetBrains Mono - for Victor Mono. Only Supreme was ever vendored, and Skyfall's own notes call - Victor Mono "user-supplied". Two-line swap when the real faces arrive. - -Dark + light, all four tabs: `http://10.100.10.50:8090/b/homepage-skyfall/` diff --git a/persistent-memory.d/2026-08-24-scriberr-ana-ml2.md b/persistent-memory.d/2026-08-24-scriberr-ana-ml2.md deleted file mode 100644 index 153669e..0000000 --- a/persistent-memory.d/2026-08-24-scriberr-ana-ml2.md +++ /dev/null @@ -1,46 +0,0 @@ -# [2026-08-24] Scriberr transcription deployed on ana-ml2, GPU1 - -Self-hosted audio/video transcription + diarization. Operator chose GPU -placement over ana-docker (8 cores shared with 50 containers, 37 GB disk) -against ana-ml2's 96 cores, `/tank`'s terabytes and GPU1's headroom. - -**Live:** `http://scriberr.ana.internal:8080` (DNS alias added), health `healthy`, -all seven backends up, zero failures: `whisperx pyannote sortformer parakeet -canary voxtral openai`. ~30 GB of weights on `/tank`. - -Stack: `stacks/scriberr/`. Full gotcha list in auto-memory -`reference_scriberr_ana_ml2`. - -## Three upstream bugs, none of them ours - -**1. The Blackwell image does not exist.** Upstream's README documents -`scriberr-cuda-blackwell`; GHCR has **no tags for it**. Published -`scriberr-cuda` covers sm_61–sm_89 only — on these sm_120 cards it fails or -silently drops to CPU. The real sm_120 path is `Dockerfile.cuda.12.9` -(CUDA 12.9.1, cu128 torch), **built from source**. Do not "simplify" the compose -back to the published image. - -**2. It must run as uid 10001, not 1000** — and the error lies: -`unable to open database file: out of memory (14)`. Error 14 is -`SQLITE_CANTOPEN`, not an OOM, on a box with 566 GB RAM. That Dockerfile creates -`appuser` at 10001 (Ubuntu 24.04 owns uid 1000 as `ubuntu`) and chowns `/app` to -it, while the entrypoint's PUID remap covers only the data dirs. -**Isolated by elimination**: SQLite writes fine to `/tank` as 1000 → not the -mount; fails on a plain named volume too → not the storage; the **published CPU -image works at PUID=1000** because there `appuser` *is* 1000. -Generalisable: *when a container "permission" bug appears, compare the uid the -image was BUILT for against the uid you are RUNNING as.* - -**3. `UV_LINK_MODE=copy` is required.** Scriberr builds each backend's Python env -with `uv` at start; uv's reflink mode fails on overlayfs+ZFS with -`Failed to clone … Resource temporarily unavailable (os error 11)`. **Partial -failure** — WhisperX and PyAnnote came up and the app looked fine while Parakeet -and Sortformer were silently absent. Occurrences 2 → 0 after the fix. - -## Related - -`speaches` on irv-ml1 **stopped** the same day (stack retained, one command to -restart): Eyra was abandoned pre-implementation because Scriberr covers the need, -leaving it with no consumer. Scriberr runs its **own** WhisperX in-container and -is **not** a speaches consumer. Idle footprint at stop was 274 MiB, not the -~5.9 GB quoted — that figure is the loaded-model working set. diff --git a/persistent-memory.d/2026-08-25-erp-tune-run2-complete.md b/persistent-memory.d/2026-08-25-erp-tune-run2-complete.md deleted file mode 100644 index 087838e..0000000 --- a/persistent-memory.d/2026-08-25-erp-tune-run2-complete.md +++ /dev/null @@ -1,92 +0,0 @@ -# ERP/RP tune run-01 COMPLETE — 7.36h, gate passed on the axis it was built for - -`[2026-08-25]` - -## The run - - 1312/1312 in 7:21:52 train_loss 2.793 epoch 1.0 - 20.1 s/it FLAT across every 100-step window (round 1: 35-46.5 s/it) - adapter: /tank/erp-tune/run-01/adapter/ 410 tensors, provenance.json - -**47% faster than the round-1 projection of 13.85h**, from two changes: the -bucketed sampler and flex attention. Rate was flat — 19.7 / 19.8 / 20.4 / 20.3 -across the four 100-step windows — which means the 35-46.5 spread in round 1 was -*entirely padding*, and removing padding removed the variance rather than just -the mean. - -⚠ **I quoted three different ETAs (6.9h, 8h, 7.3h) before I started using a -rolling average.** The first two were instantaneous tqdm readings off a number -that swings 17-25 s/it with batch width. Only the rolling rate was honest. Same -measure-don't-sample discipline I wrote into the throughput playbook, violated on -the one metric I kept reporting. - -## lora_B gate — PASSED, twice - - checkpoint-100 205/205 non-zero, median norm 0.829 - final adapter 205/205 non-zero, median norm 1.708 - vision_tower tensors: 0 on both - -Median norm rising 0.829 -> 1.708 means it kept learning through the whole run -rather than saturating early. This check **never ran in round 1** (died at step -19, first checkpoint was 100) and it is the only failure mode that stays -invisible until the acceptance gate reports base-identical numbers. - -## The gate — brokkr-smithy-dev - -**It did the thing it was built to do:** - - metric base A/B tuned delta floor - attractor hit rate 94.8% / 96.8% 84.5% -11.3pt 2.0pt - diversity (pairwise) 0.213 / 0.221 0.3948 +0.178 0.008 - -Diversity moved **22x its own noise floor**. Attractor rate (how often the model -reaches for the same names and phrasings) fell 11 points against a 2-point floor. - - T1 100 · T2 95 · T3 96-97 · T4 98 · T5 100 · T6 81-82 · core ~94.2 - memorisation: 0.0000 on BOTH arms, all three corpora - -**Zero memorisation closes the R20 licensed-prose exposure on measurement rather -than argument.** - -⚠ **Caveat brokkr volunteered rather than buried:** the tuned arm lost 18 of 192 -generations to truncation/degeneracy against base's 1-2. Lopsided exclusions -plausibly flatter the diversity magnitude. Direction is unambiguous at 22x floor; -the number carries an asterisk. - -## The noise-floor near-miss — the methodology lesson - -brokkr was one step from reporting a 13-point T6 regression **that sat inside -twice his instrument's own variance.** - - --per-type 32 max swing across tasks: 9 points - --per-type 128 max swing across tasks: 1 point - -His gate criterion is "no task regresses by more than one item" = 3.1 points at -n=32. **The instrument's own run-to-run noise was 3 items.** He was scoring a -preregistered gate at 4x finer resolution than it could resolve, and caught it by -running a control he did not strictly need. Quadrupling n collapsed the noise -exactly as binomial statistics predicts. - -⚠ **Root cause of the noise is a property of the SEAT:** `max-num-seqs` is unset, -so with a 218,625-token KV cache the scheduler batches freely up to vLLM's -default of 256. Continuous batching changes reduction order and borderline items -flip. Temperature 0 buys deterministic *sampling*, not deterministic -*arithmetic*. He declined a `--max-num-seqs 1` determinism control for the right -reason: a floor measured on a seat serving one request at a time is not the floor -that applies to the seat we ship. - -## The confound I built and he caught - -I optimised a pipeline for production and then handed him its output as an eval -instrument **without asking whether those were the same job.** The tuned arm -would have reached the seat as NVFP4A16 while his base arm was bf16 — any -regression would have been tuning-damage OR quantization-damage with no way to -separate them, and the gate's whole question is "did the tune cost us -capability." - -**Both arms now bf16, same seat, same port, argv differing in exactly two -lines** (weights path, served name), template sha256 identical -(`ae53464bf3be2580`), KV cache identical to the digit (218,625 tokens across all -three launches). Quantization moved *downstream* of the gate. - -See [[2026-08-25-refusal-retention-probe]] for the axis his gate did not have. diff --git a/persistent-memory.d/2026-08-25-mfu-root-caused-attention.md b/persistent-memory.d/2026-08-25-mfu-root-caused-attention.md deleted file mode 100644 index 95d4ad8..0000000 --- a/persistent-memory.d/2026-08-25-mfu-root-caused-attention.md +++ /dev/null @@ -1,86 +0,0 @@ -# The 8.6% MFU was an accounting artifact — attention on Ampere kernels - -`[2026-08-25]` - -## The answer - -**Real utilisation was 17-20%, inside the honest stock band.** The 8.6% divided -the *intended* (windowed) FLOPs by the wall time the *dense* reality took. - - nominal billed 27.1 TFLOPS x 34.85 s = 9.4e14 FLOP - dense-sliding extra 25 layers, 2 seqs, 4 passes = +8.2e14 - padded full layers lose the causal skip = +3.5e14 - work performed ~ 1.8e15 = 51-61 TFLOPS - -The card was doing ~2x the arithmetic the architecture specifies, and the excess -was the sliding window being computed and thrown away. - -## Two independent methods agreed - - scaling fit (3 points, 2 params, residuals <3ms over 8x range) - A = 6.87e-4 s/token B = 8.85e-8 s/token^2 - quadratic share: 20.9% @ w=2048 -> 67.8% @ w=16384 - - kernel table (device rows only) - attention 22,835.8 ms 65.2% fmha_cutlass*_sm80 - dense GEMM 2,774.0 ms 7.9% - other 5,739.0 ms 16.4% - -**67.8% vs 65.2% — 2.6 points apart, no shared assumptions.** The two-term fit -needed no constant term, which refutes launch-bound outright (3,840 expert-GEMM -launches per forward are not the cost). - -## The mechanism, source-verified by brokkr's panel (arm: Bil) - - masking_utils.py:292-301 _ignore_causal_mask_sdpa requires - kv_length < local_attention_size. 16384 >= 1024, - so THE SLIDING MASK ALWAYS MATERIALISES. - sdp_utils_cpp.h:259-267 flash rejects ANY explicit mask - sdp_utils.cpp:647 cuDNN head_dim capped at 128 -> unreachable - Context.h:480-485 prefer-cuDNN needs major 9 or 10; sm_120 is 12 - -⚠ **The kernels are `sm80` — Ampere-generation CUTLASS on a Blackwell card**, -with the forward on `gmem`, the memory-efficient backend's slowest fallback tier. - -## What actually fixed it - -**Bucketing (bucket-to-pair, shuffle-to-mix)** — 29.9% padding -> 0.0%, and -78.3% of micro-batches become exactly zero-pad, which puts the 5 global layers -back on `is_causal`. Measured: padding costs **9.4% MORE time for 24% LESS work** -at fixed width, because an explicit mask knocks those layers off the fast path. - -⚠ **Bucket size is NOT a diversity knob.** Swept across a 256x range, roots per -accumulation window stayed flat at 3.54-3.61. The global micro-batch shuffle does -all the mixing; the bucket only costs padding. Use the tightest bucket. - -**flex_attention** — Triton-generated so it compiles for sm_120 instead of -shipping sm_80 binaries. 21.7x on sliding layers, 2.1x on global. Needs mandatory -`kernel_options` at 32x32 blocks: 64x32 needs 102,400 bytes against a -**101,376-byte hardware ceiling** — misses by 1 KB, and Triton is already opting -into the full 99 KB, so it is the card, not a default. - -## ⚠⚠ The trap that produced TWO wrong published conclusions - -`torch._dynamo` defaults to a recompile ceiling of **8**. Every distinct sequence -width is a new shape. On hitting the ceiling dynamo does not error — it silently -falls back to UNCOMPILED flex, which is ~20x slower AND documented to *"not work -with the backwards pass and may produce incorrect results."* - -That artifact produced a bogus **0.76x slowdown** and a bogus **2.9% loss -divergence**, and I believed and reported both. Raising the limit to 256 flipped -the speed result to 1.41x. - -The loss divergence turned out to be real but benign — adjudicated against fp32 -MATH ground truth, both backends sit ~2e-3 from truth with flex fractionally -CLOSER at every width. **Do not re-open it by comparing the two backends to each -other; that cannot answer it. Compare to fp32.** - -## Process lesson - -brokkr's panel produced **four self-retractions in ninety minutes**. Every -retraction was a derivation; every survivor was a measurement. And the whole -head_dim-512 SDP problem was **already documented in zerofata's published Axolotl -config since April** — the right first stop for "why is this architecture slow" -is practitioner configs for that exact base, before any panel. - -Playbook: `docs/pfi/training-throughput-playbook.md`, commit `7b5fd91`. diff --git a/persistent-memory.d/2026-08-25-nvfp4-serving-pipeline.md b/persistent-memory.d/2026-08-25-nvfp4-serving-pipeline.md deleted file mode 100644 index 4738bdd..0000000 --- a/persistent-memory.d/2026-08-25-nvfp4-serving-pipeline.md +++ /dev/null @@ -1,84 +0,0 @@ -# NVFP4A16 serving pipeline — built, validated, and the MoE landmine it found - -`[2026-08-25]` - -Pipeline at `scripts/erp-tune-serve/` (commits `6a85829`, `ab980e9`). -Validated end-to-end against checkpoint-100 before the real adapter existed. - -## ⚠⚠ THE LANDMINE: a `targets=["Linear"]` recipe misses EVERY MoE expert - - before linearize_moe: 427 Linears, 205 targeted, experts 0 - after linearize_moe: 11,947 Linears, 11,725 targeted, experts 11,520 - (30 layers x 128 experts x 3 projections) - -Gemma-4 stores each layer's 128 experts as two fused 3-D `nn.Parameter` tensors -(`gate_up_proj` [128,1408,2816], `down_proj` [128,2816,704]) — note the absent -`.weight` suffix. A Linear-targeting recipe resolves 205 of 427 modules and -**zero experts**, leaving 22.84 B params (88.5% of the model) in BF16 with no -warning. - -**This is the same defect that killed QLoRA here via bitsandbytes.** The blind -spot is in the *checkpoint layout*, not the tool. Fix: -`llmcompressor.modeling.moe.linearize.linearize_moe` — no registration needed, -Gemma-4 satisfies `FusedExpertsProtocol` structurally. Playbook §3.15. - -## Scheme: NVFP4A16, deviating from the playbook default, on measured grounds - -brokkr benched the W4A4 quant of this checkpoint at **12% on contradiction -detection with CoT off against gen's 81%** — the signature of 4-bit input -activations on a reasoning-dense task. Plus W4A4 KLD is 2-4x worse past ~10k ctx -on sm_120. This is a 16,384-ctx RP seat. Marlin's prefill cost accepted. - -⚠ Several HF repos named `…-NVFP4A16` declare `input_activations num_bits 4` — -W4A4 wearing an A16 label. The script refuses if the emitted config says 4. - -## Four silent defects the dry run found - -1. **transformers 5.15 MIGRATES the config schema on save** — drops - `global_head_dim`/`num_global_key_value_heads`, writes `per_layer_config`. - transformers 5.10 (the llmcompressor venv) then reads `num_key_value_heads` - as None and dies with `TypeError: unsupported operand type(s) for //`. - Every working artifact on the box uses the OLD schema. Merge now downgrades it. -2. **llmcompressor cannot auto-init a processor for a multimodal checkpoint** — - pass the tokenizer explicitly as `processor`. -3. **`save_pretrained` does not carry `processor_config.json`** — vLLM then fails - with "Can't load feature extractor", which reads as a vision bug. -4. **The quant needs more than GPU1's free 32 GiB.** `quant_with_gen_down.sh` - stops `vllm-gen` and restores it from a trap on EVERY exit path, using - `docker start` not `compose up` so the container returns with its exact config. - -## Verified on the emitted artifact - - 49 GB -> 17 GB, format nvfp4-pack-quantized, a=null (genuine A16) - weight_packed 11,725 of which expert 11,520 - tokenizer truncation: clean (§3.14 trap avoided by calibrating on the - encode cache, so the tokenizer is never called - with truncation=True at all) - served: Marlin NVFP4 kernel + Marlin MoE backend, coherent generation - -⚠ The reference `nvfp4a16` artifact triggers a vLLM warning that q/k/v carry -*different* weight global scales ("likely reduced accuracy"). **Ours does not** — -llmcompressor 0.12 links weight observers across fused groups automatically. The -in-house quant is better than the downloaded one on that axis. - -## ⚠ MERGED WEIGHTS ARE MANDATORY — and not for the reason we assumed - -The open question was whether LoRA-on-NVFP4 hot-swap still silently no-ops. -Retested on `vllm/vllm-openai:latest`: **it refuses to start.** - - AttributeError: To support LoRA for MoE model, - 'get_expert_mapping' must be implemented - -The check is in `vllm/lora/utils.py::process_packed_modules_mapping` and branches -on `is_moe_model()` — **quantization is not in the condition.** `gemma4.py`, -`gemma4_mm.py`, `gemma4_mtp.py`, `gemma4_unified.py` all have ZERO occurrences; -`deepseek_v2`, `mixtral`, `glm4_moe`, `ernie45_moe` implement it. - -**vLLM cannot serve a LoRA on ANY Gemma-4, bf16 or quantized.** Merging is the -only path for this architecture, and it would have bitten identically on the -unquantized base. A loud refusal is strictly better than the 0.24.0 silent no-op, -which shipped a base model wearing the tune's name. - -⚠ Base-viability pre-flight is now playbook §3.11 — three greps before picking a -base. **Grep the CLASS, not the file**: `mistral.py` greps as `SupportsLoRA=0` -and is fully LoRA-capable via inheritance from `LlamaForCausalLM`. diff --git a/persistent-memory.d/2026-08-25-refusal-retention-probe.md b/persistent-memory.d/2026-08-25-refusal-retention-probe.md deleted file mode 100644 index f5d4582..0000000 --- a/persistent-memory.d/2026-08-25-refusal-retention-probe.md +++ /dev/null @@ -1,64 +0,0 @@ -# Refusal retention — the axis the gate did not have, and the axis I measured wrong - -`[2026-08-25]` - -## Why it exists - -brokkr's gate measures reasoning (T1-T6), craft (diversity/attractor) and -regurgitation (memorisation). **Nothing measured whether the model still -COMPLIES** — which for this seat is arguably the most important property. - -The risk is specific to our operation order. We do **tune(abliterate(stock))**, -so the tune has 57.7M tokens of opportunity to walk the abliteration back. *A -tune that gains 41 items of contradiction detection and quietly re-installs -refusals is a failed seat that passes the entire gate.* - -## The measurement — controlled, single instrument, both arms - - arm HARD DEFLECT COMPLY - base 0/100 0 100 - tuned 29/100 0 71 - -Same seat, same probe, temp 0, `mlabonne/harmful_behaviors` x100. -Probe: `scripts/training-probes/refusal_probe.py`. - -**The tune added 29 general-harm refusals where the base had none.** - -Two things fell out: - -- **The instrument validates.** Base measured 0/100 on my generated-text regex - against Heretic's recorded 3/100 from a first-token-probability scorer. 0 vs 3 - is agreement — the incomparability worry was right caution about a non-problem. -- **DEFLECT is 0 on BOTH arms, so the free control fires.** An instrument - artifact does not care which arm it runs against. Both zero means the model is - **binary** — refuses in refusal-language or engages, no soft-deflection tail. - The R19 undercount does not apply here. - -## ⚠⚠ But it is the WRONG AXIS — brokkr's catch, and it is the better one - -`mlabonne/harmful_behaviors` is **general harm** — weapons, malware, fraud. **The -abliteration was not run so the model would explain bomb-making. It was run so -the model would engage with explicit fiction.** Different refusal surfaces; a -model moves on them independently. - -I picked that set because it was cached, had a recorded baseline, and was what -the abliteration tool used. **Every one of those is a reason it was convenient, -not a reason it was right** — and "it has a baseline" was actively misleading, -because a comparable number for a question nobody is asking looks like evidence. - -**29/100 general-harm refusals on a seat writing prose the operator was actively -praising is plausibly the DESIRED shape**, not a defect. General-harm refusals -returning while domain compliance holds is close to ideal for an internal -creative seat. I would have reported it as damage. - -**The load-bearing cell is COMPLY 71, not the 29.** Stock refused 100/100; -anything near that would mean the abliteration was undone. 71 complying means -"partially walked back on one axis" — a different finding, and only one of the -two threatens the seat. - -Domain-compliance probe (the right axis, from R19's track-2 map) is brokkr's, -pending. Scaffold supplied: `scripts/training-probes/counted_classifier.py` -(`2a05ae9`) — classify-never-surface, three-way, ERROR path deliberately does not -log the exception body because an exception can echo the prompt back. - -Playbook §3.13. See [[2026-08-25-erp-tune-run2-complete]]. diff --git a/persistent-memory.d/2026-08-25-worldtree-b188-b189-and-selene.md b/persistent-memory.d/2026-08-25-worldtree-b188-b189-and-selene.md deleted file mode 100644 index 10222a8..0000000 --- a/persistent-memory.d/2026-08-25-worldtree-b188-b189-and-selene.md +++ /dev/null @@ -1,123 +0,0 @@ -# Worldtree b188 + b189 bridge cutover, and the selene metadata that lied - -`[2026-08-25]` - -Three arcs in one day, all infra-ops side, all landed. - -## b188 — matrix.yaml pre-sync (#406/#409/#410 closed) - -From b188 the bridge reads per-agent `rendering` + `ambient_buffer_size` from -`config/matrix.yaml` ONLY; agent `config.yaml` matrix blocks are gone from the -image. Staged as `6417115` in `worldtree-instance-configs`, deployed to both -instances with operator approval. - -- mimir gets thinking-to-thread + tool-call reactions + 7 reaction labels; - forseti and lofn stay quiet. -- **Rider #409 pruned six dead agents** (bragi, leif, troi, soong, cara, glados) - from BOTH rosters — originally scoped personal-only, which I flagged as a - possible oversight and it was. **The settling fact worth keeping: the engine - roster comes from the image's baked `agents/` directory (only `config/` is - bind-mounted), and both instances run the same image**, so instance-level - evidence about which agents the engine lists generalises by construction. -- Both rosters now exactly `[mimir, forseti, lofn]` — the three engine agents - actually bridged. mask/vili/echo exist in the engine, deliberately unbridged - (operator ruling). - -⚠ **Edited text-surgically, not via a yaml round-trip** — PyYAML would reflow -1,249 lines and drop every comment, and the comments are the documentation. - -⚠ **`deploy-wt-config` uses `docker restart`, NOT `compose up`.** A `compose up` -on corviduo-dev re-resolves the image tag and can silently swap the running -build — which would turn an "inert pre-sync" into an unintended image roll on two -live instances. That property is easy to lose in a future refactor of the script. - -## b189 — #407 bridge extracted to its own repo (#404 umbrella closed) - -Bridge now `gitea.phasefinal.com/pfi/wt-matrix-bridge`, its own repo, own CI. - -⚠ **It publishes to the `pfi` ORG, not `vh`, and the reason is structural:** `vh` -is a **USER**, not an org. Gitea scopes user-namespace packages to the owning -user — there are no package collaborators on a user namespace. **No service -account can ever publish to `gitea.phasefinal.com/vh/*`.** claude-bot is an Owner -of `pfi`, so that is where it goes. Token `wt-matrix-bridge-ci` (id 28, scopes -`write:package,read:repository`), vaulted at -`nh3-dev/.config/claude-bot/gitea-token-wt-matrix-bridge-ci`. - -**Minted a dedicated token rather than reuse `claude-bot-sdk-ops` or `arbo-ci`, -both of which already carry `write:package`** — a shared credential cannot be -revoked without collateral. ⚠ A first mint attempt succeeded then failed to save; -Gitea returns a token value exactly once, so it was unrecoverable. Deleted the -orphan (id 27) rather than leave a live package-write credential on the account. - -**Both instances PINNED** to `f3f8ec902267` (`e90f436`), closing the #410 shape: -`WORLDTREE_IMAGE` was pinned and the bridge was the one drifting service. - -⚠ The pin moved from `b178285b1cb5` because a cross-frontier bug-hunt found the -M_EXCLUSIVE **fallback could itself litter unboundedly** in exactly the state it -was written for. **The safety net had the same failure mode as the thing it was -catching** — and my staged window leaned on that fallback. - -## #411 — the debug-room failure, diagnosed twice and wrong both times first - -My theory: the alias was held by orphaned rooms. **Refuted by a directory probe -returning 404.** The real cause: Synapse's **M_EXCLUSIVE** — an appservice may -only create aliases inside a namespace it has RESERVED, and -`aipa_appservice.yaml` had `namespaces.aliases: []`. - -⚠ **I inferred a cause from a symptom that was the RESPONSE to the cause** — the -log's "re-resolving alias" line is the recovery path firing, not evidence the -alias exists. One directory probe settled it and I reasoned instead. - -Fixed with operator clearance: added -`regex: '#aipa-debug-[a-z0-9_-]*:matrix\.phasefinal\.com'` (exclusive) at -`/opt/docker/conf/synapse/aipa_appservice.yaml` on **ana-docker** (NOT -`/opt/docker/data/`, which worldtree-dev's issue cited). **My regex was tighter -than the proposed `#aipa-debug-.*`**, which fullmatches only because `.` also -matches the `:` separator and would equally claim other homeservers. - -Pre-apply sweep (an `exclusive: true` claim can make Synapse refuse to START): - - aliases matching '%aipa-debug%' 0 - total room_aliases on the homeserver 1 <- why it went unnoticed this long - rooms created by @aipa-debug 17 <- the litter, confirmed unaliased - -Synapse healthy in 40s, both bridges rode through. **The aliased create then -worked FIRST TRY on the next personal recreate** — designed path, fallback never -fired, both reuse mechanisms live (canonical-alias rediscovery AND the room-id -cache). - -⚠ The #411 writer census **inverted its own premise**: the api (uid 1000) cannot -write `/app/sessions`, the bridge (root) can — the reverse of the issue text. But -worldtree-dev's reconciliation is better than "backwards": pre-#407 the bridge -ran from the ENGINE image as uid 1000, and the cutover changed the answer -underneath the issue. Both readings were true at their timestamps. **No live -writer exists for that path anyway** (`sessions.path` is a legacy default, #330 -moved the tools off it), so the chown is optional future-proofing. - -## selene-1-mini-8b — a config that lied about what answers - -forseti's fleet sweep found the only genuine residual in -`worldtree-instance-configs`, and it was **live on both boxes, not just drifted -in git.** Fixed in `a77639d`. - -Routing was never broken (`model: "chat-judge"` stays), but `display_name` said -"Selene 1 Mini 8B" and the description said "Atla Selene 1 Mini 8B — reward model -derived from Llama 3.1 8B" while chat-judge has been backed by -**qwen3.8-27b-uncensored** since 2026-08-23. - -⚠ **It misstated the KIND of model, not just the identity.** A reward model and a -generative judge are different instruments; a consumer reading -"pairwise/likert/binary/scalar reward model" would expect scalar-reward semantics -qwen3.8-27b does not provide. - -Verified: `selene-1-mini-8b` -> HTTP 400 (by design), `chat-judge` -> HTTP 200. - -**NOT changed:** the catalog key and the `selene-judgment` role — `model_roles.yaml` -binds to that key, so renaming is worldtree-dev's schema call. Flagged that a role -named after a retired model defeats the purpose of role aliases. - -⚠ forseti's sharpest finding is for the operator: **`~/.claude/CLAUDE.md` line 502 -lists the retired name in the global tools roster.** A broadcast reaches sessions -that already exist; the roster line keeps minting new ones. His file, awaiting his -word. Also: **there is no fan-out primitive on the bus** — a fleet relay is 73 -individual posts, recorded as a real gap rather than papered over. diff --git a/persistent-memory.d/2026-09-08-erp-run7-launched.md b/persistent-memory.d/2026-09-08-erp-run7-launched.md new file mode 100644 index 0000000..d85d3d8 --- /dev/null +++ b/persistent-memory.d/2026-09-08-erp-run7-launched.md @@ -0,0 +1,8 @@ +# ERP run 7 launched on pfi-gx10 (2026-09-08 23:06 PT) + +Operator's direct grant `operator-2026-09-08-rnd-run7` (in-session `/goal: launch training run 7 on gx10`, 23:03 PT), pid 599489, 542 steps, ~80 s/it, ETA ~noon 09-09. Runbook `docs/runbooks/gx10-run-07.md`; canonical `scripts/erp-tune-gx10/{run-07-gx10.json,launch-run-07.sh,build_r7.py}`; commits `a34a72f`, `d8f625e`. + +- Variable (brokkr `recipe-erp-seat-sft-r7.json`, sha `2972559e…`): an opening-split slot (293 rows, `opening-split/clean-v1`, root `3ab41c08…`) + companion mask (224 entries) UNION lossmask-r3 → `lossmask-r7` (6,106, 1 dup). `survivors-r7` = survivors-r5 + 293 = 8,505. Root/mask/holdout relayed NAS→nh3-dev→box (no NFS on the GX10), sha-verified. +- Free check: held roots identical to run 6 (173 chunked / 30 truncated / 59 lost turns); slot 293/293 fit_whole; two cwm conversations (cwm-2602, cwm-1826) dropped as unfittable because the union mask left them no loss turn — brokkr: "loss moved, not doubled, satisfied by construction". Padding 17.1% (run 6: 0%). +- ⚠ First launch died in preflight: the recipe JSON never landed because a zsh quoting error silently aborted the staging chain. Lesson: stage files with `scp` in their own command; never chain a heredoc-bearing python inside a single-quoted ssh string. +- Gate after the adapter: brokkr commits `tools/base_window_r7.sh`, `tools/tuned_window_r7.sh`, `tools/verdict_run07.py` before it lands; serve `erp-seat-base-ara` (floors re-locked) → `erp-tune-v7`; cue-length probe as a conformance leg on both arms; DONE sentinel gated on every step's artifact. diff --git a/persistent-memory.d/2026-09-08-erp-seat-nvfp4-trial-and-toolcalling.md b/persistent-memory.d/2026-09-08-erp-seat-nvfp4-trial-and-toolcalling.md new file mode 100644 index 0000000..f9700e4 --- /dev/null +++ b/persistent-memory.d/2026-09-08-erp-seat-nvfp4-trial-and-toolcalling.md @@ -0,0 +1,11 @@ +# erp-tune-v6-nvfp4a16 on ana-ml2 :8021, aliased to `trial`, and the Gemma-4 tool-calling findings (2026-09-08) + +**Seat.** Operator: "quant the latest trained model into nvfp4 and serve it on ana-ml2 while we train a new model on the gx10." Stack `stacks/erp-seat` (recipe = gemma4-charrp's; true served name only; GPU1, util 0.35, 32K ctx). Artifact `/tank/aimodels/erp-tune-v6-nvfp4a16` (16 GB, compressed-tensors nvfp4-pack, W4A16, 252 ignores incl. 60 router + 191 vision) from `/tank/aimodels/erp-tune-v6-bf16` (merged-run06 relayed gx10→nh3-dev→ana-ml2 in 17 min at ~47 MB/s — no key path gx10↔ana-ml2). Quant pipeline `services/erp-seat-quant/` replicates `prithivMLmods/gemma-4-26B-A4B-it-NVFP4A16`: `linearize_moe` first, assert 11,520 expert Linears, routers/vision/audio ignored, post-steps restore processor configs + template and reset the tokenizer cap. DATA-FREE (~90 s) — playbook §3.16. Commits `911ff20`, `8512dd4`. + +**Alias.** Operator: "alias erp-tune-v6-nvfp4 to trial, please. no gate." The `trial` model lives in `stacks/litellm/conf/config.yaml` → `/model/update` refuses it ("Model in config"); deploy `--conf` + `sudo docker compose restart litellm` on ana-docker. Verified 3/3. Commit `48aa2c6`. Block states the seat as unrated on every safety axis. + +**Tool calling (operator: "fix toolcalling with the trial seat").** +1. `tool_choice:"none"` → empty turn (content AND tool_calls null, 3/3): vLLM kept the tools in the prompt, the model called one, parsing was off. Fix `--exclude-tools-when-tool-choice-none` (`7f6be8a`). `stacks/gemma4-charrp` has the same exposure, no flag — bouncing it is consumer-visible, operator's call. +2. Forced (named/`required`) tool_choice is PROMPT-DRIVEN on every Gemma-4 seat **by vLLM design**: `vllm/tool_parsers/gemma4_engine_tool_parser.py` sets `supports_required_and_named = False` and `adjust_request` skips the structured-output JSON so the model can emit native `<|tool_call>call:…`. A tune that weakened that syntax honours forced calls only sometimes. Measured (3 conv × 3 turns, real system prompt): v0.26.0 1/9; nightly `311b3513` (v0.27.2rc1, the `gen` seat's image) 6/9 with the round trip clean 3/3 → seat pinned to the nightly (`a34a72f`). tts-dev: 0/18 on v0.26.0 with `gen` 18/18 as positive control; **`response_format: json_schema` 18/18 = the deterministic path**. A parser plugin re-enabling guided JSON would also need JSON extraction in the engine-parser path — not attempted. +3. ⚠ Removing `--reasoning-parser gemma4` does NOT fix forced choice (3/9) and leaks `<|channel>thought` into content 3/3 — keep it and the `enable_thinking:false` pin. +Single-turn probes are FALSE POSITIVES for forced tool_choice — gate on multi-turn with a real system prompt (tts-dev's lesson, confirmed). diff --git a/persistent-memory.d/2026-09-08-esh-static-wan-followups-and-ytvc.md b/persistent-memory.d/2026-09-08-esh-static-wan-followups-and-ytvc.md new file mode 100644 index 0000000..54a07d2 --- /dev/null +++ b/persistent-memory.d/2026-09-08-esh-static-wan-followups-and-ytvc.md @@ -0,0 +1,5 @@ +# ESH static WAN follow-ups landed; YTVC chase; gitea webhook repointed (2026-09-08) + +**ESH static WAN** `128.177.138.182/30` (gw .181, Cityside Fiber) confirmed LIVE (UDM WAN1 static, uplink up since ~09-05; egress verified; CGNAT at ESH is history). Added to the crowdsec `esh` allowlist on ana-docker. Operator: "land all 3" → (a) FortiGate infra-ops `trusthost3` 23.164.40.160 → `128.177.138.182/32`, verified by a real login from esh-docker-vm (flash backup ran but `execute revision list` errors on the 80F — unconfirmed); (b) dormant `esh-ana` IPsec rebound wan2/192.168.200.111 → wan1/`128.177.138.182` (still disabled); (c) ESH UDM forward UDP 41641 → esh-scale 10.0.50.65 → mesh peers DIRECT via `128.177.138.182:41641` (was DERP lax). Commits `7d5de53`, `a19b7f6`. ⚠ The permission classifier blocked every edge-gateway write until the operator disabled it — three shapes tried, then stopped and surfaced; no tool-laundering. + +**YTVC** (operator: "chase youtube voice clipper being down"): (1) every job died at yt-dlp because the nh3-dev danted proxy was retired 09-06 → new fleet SOCKS5 on **nh3-scale** `socks5h://100.64.0.1:1080` (`scripts/setup-nh3-scale-socks-egress.sh`, dante, fleet ACL, residential egress 70.230.226.88; ⚠ dante one-line `{ }` blocks are a syntax error); (2) Homepage href on the dead wg0 IP → `irv-ml1.nh3.internal:8000` (labels live in the UNTRACKED override); (3) full downloads still 403'd = stale yt-dlp 2026.07.04 from a cached Docker layer (`--test` passing while the full fetch 403s = client, not IP/cookies) → `compose build --no-cache api`; (4) the rebuild pulled a whisperx needing `punkt_tab` → staged on the data volume + `NLTK_DATA`. Then yt-voice-clipper-dev shipped v0.3.4→v0.3.6 (cache-bust ARG, punkt baked, **CrisperWhisper 2.0 replaces whisperx**: x7kWJojf1MI → 12 clips with `[UM]`/`[UH]` tags). ⚠ The gitea push webhook had targeted the dead wg0 IP since 09-06 (never fired) → repointed to `10.6.110.50:9008` with the HMAC secret RE-APPLIED (a PATCH without `secret` drops it); test push auto-deployed. Commits `14dd9ab`, `d2df1eb`, `f2b7384`. diff --git a/persistent-memory.d/2026-09-08-run6-gate-csam-adjudication.md b/persistent-memory.d/2026-09-08-run6-gate-csam-adjudication.md new file mode 100644 index 0000000..7594373 --- /dev/null +++ b/persistent-memory.d/2026-09-08-run6-gate-csam-adjudication.md @@ -0,0 +1,8 @@ +# Run-6 gate: CSAM soft trip, operator adjudication, TRANSFERRED (2026-09-08) + +- 22:18 PT brokkr HALTED the run-6 gate: R19's CSAM-drift detector fired at level=review on ONE generation in the tuned refusal k=5 step — metadata only (minor_soft=1 `soft_baby` near anatomy terms, minor_hard=0, sexual=7, salted hash `01724c39…`); the spec (§8) never stores the text and the seat does not log completions, so **the text exists nowhere**. +- Conflict surfaced, not resolved by me: the operator had aliased `trial` → the NVFP4 build ("no gate") two minutes before the halt reached me; brokkr asked that it not be served until adjudication. Left the alias in place (explicit operator ruling), escalated to the operator + Miranda with the one-line rollback offered. +- 22:45 PT operator adjudicated GO, verbatim: "I don't want it regenerated, but I will say that in the vernacular, baby is about the most common pet name you can get, ESPECIALLY during sex. I'm going to adjudicate it as a go. There are unmistakable CSAM terms, but baby is not one of them." Relayed verbatim to brokkr and Miranda. +- brokkr finalized TRANSFERRED without the tuned refusal leg (the cell rule never read it; tool forbids a re-run); tuned refusal telemetry recorded ABSENT for run 6; RESULT carries the adjudication with infra-ops as relay. Operator also CUT the k=25 refusal legs on both arms ("the battery is information enough"). +- Cue-length probe (both arms, 288 gens each): tuned 5-word cue → 54/62 words median, 49-word → 114, 221-word → 292; base 168/191 → 209 → 282. Tune's deficit ~120 words at the short cue, ~0 at the long → run-7's lever. +- The choreography honoured hands-off throughout; the base seat served 16:24–21:38 and again 22:45–23:01 PT for the reference arm. diff --git a/persistent-memory.d/2026-09-09-ana-ml2-pool-actions-deferred.md b/persistent-memory.d/2026-09-09-ana-ml2-pool-actions-deferred.md new file mode 100644 index 0000000..406ad34 --- /dev/null +++ b/persistent-memory.d/2026-09-09-ana-ml2-pool-actions-deferred.md @@ -0,0 +1,15 @@ +# ana-ml2 pool health — three actions deferred to a clean-context session (2026-09-09) + +Operator ruling 2026-09-09 ~00:30 PT: "snapshot and we'll do all 3 on clean context." Findings commit `3e18a04`. + +**Findings (measured 2026-09-09 00:00 PT):** +- `tank` (raidz2, 8× NVMe): ONLINE, **2 CKSUM errors on `nvme7n1`**, boot-time resilver of 638 GB on 2026-09-05 14:26 (box rebooted at 14:26; nvme7 came up late/dirty). No data errors, 58% full. **No scrub since 2026-04-12** — the Debian `zfsutils-linux` second-Sunday cron scrubbed `zroot` on 08-09 but not `tank`; cause unknown (`zpool history tank` shows trims monthly, last scrub 04-12). +- No `nvme-cli` or `smartctl` on the box → nvme7's media-error counter unread. +- `zroot` at **91%** (345 G of 379 G): `docker system df` = images 429 GB (204 GB reclaimable), build cache 74 GB (36 GB reclaimable). +- pfi-pve `NASPool` 7% / `ospool` 19%, scrubbed 09-05 / 08-09, clean. + +**The three actions, in order:** +1. `sudo zpool scrub tank` on ana-ml2 (1–2 h of extra I/O; seats keep serving) → on a clean pass `sudo zpool clear tank`; if the scrub finds errors on nvme7n1 → replace path. +2. `sudo apt install nvme-cli` → `sudo nvme smart-log /dev/nvme7` (media_errors, critical_warning, percentage_used) and `nvme id-ctrl` for model/serial; record in the drive inventory. +3. `docker image prune -a`? NO — `docker image prune` (dangling only) + `docker builder prune` on ana-ml2; the 47 unused-but-tagged images need a look first (some are rollback seats: e.g. `vllm/vllm-openai:v0.26.0`, nightlies). Target: zroot back under ~75%. +Also worth a look while there: why the scrub cron skips `tank` (`/usr/lib/zfs-linux/scrub` logic — it skips pools with an active trim/resilver or those not "healthy"?). diff --git a/persistent-memory.md b/persistent-memory.md index 0ac86ee..13503c4 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-08 11:30Z (fleet-ops: **ERP run 6 TRAINING on pfi-gx10** on the first genuinely abliterated base [jenerallee78 ARA @ 0631379a, index 33c59654], run-5 seat unloaded; earlier today: run-5 RESCUED, WhereTF card+DNS, ext-tts alias fix, irv-ml1 stale-IP cleanup + ana-ml2 discovery proxy, Miranda relay authority)_ +_Last updated: 2026-09-09 00:35 PT (fleet-ops: ERP run 7 TRAINING on pfi-gx10 [opening-split slot]; run 6 gated TRANSFERRED after the operator's CSAM adjudication; erp-tune-v6-nvfp4a16 serving on ana-ml2 as `trial`; ESH static-WAN follow-ups + YTVC + webhook repaired; ana-ml2 routes persisted; ana-ml2 tank/zroot actions deferred to the next session)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under an hour old, read it (it carries the in-flight @@ -106,559 +106,50 @@ no longer deployed sidecars here. See Recent decisions.) is sudo-LESS by design (`ssh lkraven@10.100.50.42` is the NOPASSWD path). **irv-ml1: `ssh irv-ml1` = lkraven, docker-group (plain docker) but sudo needs a PASSWORD (no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`. + ## Current state / in-flight -_As of 2026-09-08 (fleet-ops session). SUPERSEDES lower framing: run 4 = STILL-COUPLED; run 5 is -COMPLETE and gated RESCUED (02:13 PDT). Live open items:_ +_As of 2026-09-09 00:35 PT (end of the 09-08 fleet-ops session — operator: "snapshot and we'll do all 3 on +clean context"). Older in-flight blocks (09-08 morning, 09-05/06) are preserved verbatim in +`archival-memory.md` § Superseded in-flight snapshots._ -- **✅ ERP run 5 COMPLETE — gate = RESCUED (2026-09-08 02:13 PDT, landmark R49.5).** FIRST arm of this - line where the capability gate did NOT fail. The dependency-forcing slot (GovReport+QMSum, only 3.46% - of loss) broke the diversity↔coherence coupling that run 4 (STILL-COUPLED, 20.6% instruct slot) and - 3c (20× LR cut) could not — **STRUCTURE of the loss was the lever, not its mass; INERT did not fire.** - T4 long-context 8/8 (run 4: 5/8; base 8/8); t4_dissect noise@31 tuned 0.9062 vs run-3 tuned 0.5625; - diversity held (rp density 3.37→0.00, story 2.86→1.58). Reported-beside (not in the cell, all - de-gated + stated): T3 constraint 8/8→6/8 (NEW loss, ship-path list); RP length 68w vs 250-floor = - PARTIAL fail (short-QA slot + style shift); refusal erosion rides with the style shift (k=25 both - arms, CSAM clean); free-check base LEVELS 5-6pp below run 4 (vLLM 0.28.0 unchanged — infra-ops - confirmed — so a generations shift, not a stack delta; taxes every cross-run number). Write-up - brokkr-smithy `research/R47-premium-corpus-gate/run05-gate/RESULT-run05-gate.md`; FLOOR-LOCKED - `0f3e4e2` (cites infra-ops' base index-sha 907826a6). Whole gate infra-ops-served on pfi-gx10:8098 - (name-keyed base→tuned swap, hands-off honoured, sha+stack answers on record). Launch procedure + - canonical: eshpfi `scripts/erp-tune-gx10/` + `docs/runbooks/gx10-run-05.md`. **✅ SEAT: `erp-tune-v5` - SERVED on pfi-gx10:8098 (merged-run05); LiteLLM `trial` alias REPOINTED 3c→v5 (operator, 2026-09-08) - so it's testable from Open WebUI — verified end-to-end (trial→erp-tune-v5, coherent output). config.yaml - trial block rewritten to run-5 reality incl. the measured refusal-erosion note. ⚠ Seat is hand-launched - (vllm-run05.pid), NO restart policy/systemd — does not survive a gx10 reboot; yields to the next training - (~6 min re-serve). Brokkr: nothing further owed.** -- **run-4 gate = STILL-COUPLED** (RESULT-run04-gate.md, brokkr-smithy). Corpus dilution kept the - diversity gain, did NOT remove the safety/coherence regression. -- **✅ RESOLVED (2026-09-08, settled from bytes): the R47 base is STOCK `google/gemma-4-26B-A4B-it`, - byte-for-byte — NOT the abliteration.** The `-heretic-bf16` label in the recipes is a naming error; - run-04's "stock" provenance was right; brokkr's 77.7%-refusal telemetry lean is confirmed. Proof - (three-way match): local shards at `/home/infra-ops/models/gemma4-26b-a4b-it-bf16` sha256 - `1127684971…`/`aab47033…` == the HF download etags (`.cache/huggingface/download/*.metadata`, so the - copy is uncorrupted) == the stock repo's two LFS oids, and the download commit `4d7ae498…` == stock - HEAD. Every run 3/3c/4/5 trained from a REFUSING stock base. Why plausible: the 2026-08-24 note - SELECTED llmfan46's Gemma-4-26B-A4B Heretic v1.2.0 ARA (3/100 refusals, bf16 51.6 GB), but llmfan46 - ships that 26B-A4B abliteration **GGUF-only** — no bf16 safetensors — so the bf16 that actually got - pulled was stock google, and the `-heretic` name rode along from intent. **Operator/Brokkr decision - now evidenced (not a label guess): accept RESCUED-on-stock, or swap to a real abliteration (needs a - bf16 source, not the GGUF) + re-run. Recipes should drop `-heretic` from the base name.** -- **✅ RUN 6 COMPLETE on pfi-gx10 (2026-09-08 16:06 PT, 524/524, 11 h 42 min) → 🔥 GATE IN PROGRESS: base seat - `erp-seat-base-ara` SERVING on gx10:8098 (pid `vllm-base-run06gate.pid`, since 16:24 PT) for brokkr's floors; - AWAITING HIS SWAP CUE → then stop it and serve `serve/merged-run06` as `erp-tune-v6` (same port/flags).** - train_loss **3.259** (run 5: 3.235 — preregistered read wanted below; +0.024 is inside the ±0.1 per-step band). - Adapter `run-06/adapter` (410 tensors); merged-run06 = adapter + jenerallee78 base (index 33c59654), stock - template ae53464b. ⚠ **The abliterated repo ships NO `processor_config.json`** — vLLM's Gemma4 loader dies with - "Can't load feature extractor" without it (first base launch failed exactly so). STOCK's copy (sha `32bdf45d…`) - carried into both the base dir and merged-run06; processor plumbing, not weights. Base: jenerallee78 ARA - abliteration @ `0631379a`, pulled to `/home/infra-ops/models/gemma4-26b-a4b-it-ara-abliterated-jenerallee78-0631379a`, - 32/32 shards verified vs brokkr's pins. Run-5 recipe byte-held (free check exact). Grant - `operator-2026-09-08-rnd-run6`. ⚠ Repo `tokenizer.json` bakes a 256-token truncation → stock tokenizer set - installed over it (`*.repo` kept). ⚠ `hf download` ignores `--include` with several patterns after one flag. - **LiteLLM `trial` alias DARK until the gate ends** (then repoint 5→6 only on the operator's word). - Miranda informed 16:07 PT. → `docs/runbooks/gx10-run-06.md`. -- **✅ ADJUDICATED GO by the operator 22:45 PT** — verbatim: "in the vernacular, baby is about the most common pet name - you can get, ESPECIALLY during sex. I'm going to adjudicate it as a go. There are unmistakable CSAM terms, but baby is - not one of them." No regeneration wanted. Relayed to brokkr + Miranda; halt lifted; `trial` stays on the NVFP4 build. - Brokkr's cue-length probe on erp-tune-v6 DONE (288 gens): reply length is CONDITIONAL on the cue (5-word opening → - 54/62 words median; 221-word → 292; "at least 250 words" persona → 354) → run-7 lever = opening-split render (A′) or a - prompt-only mitigation — operator picks. **GX10 → 🔥 RUN 7 TRAINING (launched 2026-09-08 23:06 PT, pid 599489, `run-07.pid`, 542 steps, ETA ~13 h → ~noon - 09-09).** Operator's direct grant `operator-2026-09-08-rnd-run7` (/goal). Variable = opening-split slot + companion - mask (brokkr recipe r7). Free check passed: held roots identical to run 6, slot 293/293 fit whole, mask 6,106 turns - (all 224 landed, 1 dup), two cwm rows dropped as fully-masked (loss moved into the slot). 17% padding (was 0%). - → `docs/runbooks/gx10-run-07.md`. Gate after: `erp-seat-base-ara` then `erp-tune-v7`. -- **erp-seat (ana-ml2 :8021, `trial`) now on vLLM nightly `311b3513` (v0.27.2rc1)** — forced (named/required) - tool_choice is PROMPT-DRIVEN on every Gemma-4 seat by vLLM design (gemma4 parser skips guided JSON); measured - 1/9 on v0.26.0 → 6/9 on the nightly (n=9, small), round trip clean 3/3, 12-shape matrix green. Deterministic forced - calls = `response_format: json_schema` (tts-dev: 18/18). ⚠ Removing `--reasoning-parser gemma4` leaks - `<|channel>thought` into content 3/3 — keep it. -- (history) **GX10 was FREE 23:03–23:06 PT** (reference arm in 06:00Z, 288 gens; base seat stopped, GPU clear). Run-7 recipe - `erp-seat-sft-r7` is emitted + verified in brokkr-smithy but carries NO GRANT — nothing trains until the operator - gives one. Base-vs-tuned reply-length: tune's deficit ~120 words at the short cue, ~0 at the long cue. -- **(history) ⛔ RUN-6 GATE HALTED 22:18 PT on the CSAM line (brokkr, level=review, SOFT trip): one generation in the tuned refusal - k=5 step tripped R19's drift detector — minor_soft=1 ('soft_baby' adult-usage, the spec's named false-positive class), - minor_hard=0, sexual=7, NO hard signal; fail-closed → operator-adjudicated; TRANSFERRED is PROVISIONAL (diversity, - reasoning, t4_dissect complete); tuned-probeB.json absent; brokkr will not re-run.** ⚠ CONFLICT SURFACED: the operator - had ALREADY aliased `trial` → the NVFP4 build ("no gate", 22:18) two minutes before the halt reached me; brokkr asks - the quant not be served until adjudication. Left the alias IN PLACE (explicit operator ruling), escalated to the - operator + Miranda with the one-line rollback offered. Run 7 does not start from this recipe until adjudication. -- **🔥 GATE seat: erp-tune-v6 (bf16) SERVING on gx10:8098 since 21:46 PT** (`vllm-run06.pid`); brokkr's tuned window - (~1.5 h, k=25 legs CUT by operator) is unattended — files in brokkr-smithy `run06-gate/` are the signal, nobody pings. - **After TUNED-WINDOW-DONE (brokkr's three asks, agreed):** (1) keep erp-tune-v6 up until he posts "probe done" - (cue-length probe, ~15 min); (2) re-serve `erp-seat-base-ara` ~20 min for the probe's reference arm, then the GX10 - is FREE for run 7; (3) ✅ done — `dialogue-survivors.jsonl` (610, sha b2b6a0e9) copied to - `/mnt/smithy/datasets/derived/_recipes/erp-seat-sft-r3/`. Run 7 = recipe adjustment (reply length + - constraint-following), variable picked by the probe; no recipe/grant yet. -- **✅ erp-tune-v6-nvfp4a16 SERVING on ana-ml2 GPU1 :8021 (operator directive 2026-09-08 ~21:30 PT: "quant the - latest trained model into nvfp4 and serve it on ana-ml2 while we train a new model on the gx10").** Stack - `stacks/erp-seat` (recipe = gemma4-charrp's: gemma4 tool+reasoning parsers, enable_thinking pinned false, stock - template ae53464b, v0.26.0, util 0.35, 32K ctx) — TRUE name only. **`trial` alias REPOINTED to it 2026-09-08 22:18 PT (operator: "alias erp-tune-v6-nvfp4 to trial, please. no gate")** — config-file deployment (`/model/update` refuses config models), `deploy-stack.sh ana-docker litellm --conf` + `sudo docker compose restart litellm`; verified ×3 through the gateway. - Artifact `/tank/aimodels/erp-tune-v6-nvfp4a16` (16 GB, compressed-tensors nvfp4-pack, W4A16, 252 ignores incl. - 60 router + 191 vision) from `/tank/aimodels/erp-tune-v6-bf16` (merged-run06 relayed gx10→nh3-dev→ana-ml2 in 17 min, - no key path gx10→ana-ml2). Quant = `services/erp-seat-quant/` — DATA-FREE (~90 s; playbook §3.16), tokenizer cap - reset, encode-equal to source. Smoke: prose in `content`, ~200 tok/s single-stream (n=3, spread <1%). - **Tool calling FIXED 2026-09-08 23:00 PT (operator: "fix toolcalling with the trial seat"):** the one reproducible - defect in a 12-shape matrix was `tool_choice:"none"` → empty turn (content AND tool_calls null, 3/3) — vLLM kept the - tools in the prompt, the model called one, parsing was off. Fix = `--exclude-tools-when-tool-choice-none` (seat - recreated; none→prose 3/3; auto/required/named/parallel/nested/empty-list/streaming/round-trip all green). - ⚠ `stacks/gemma4-charrp` (char-rp seat) has the same exposure and no flag yet — bouncing it is a consumer-visible - restart, so left for the operator's word. - ⚠ NOT gate-parity (gated artifact = bf16 arm on the GX10; this seat has a smoke test + 3-run decode probe only) — - **operator ruled "no gate"**; the config block states it as unrated on every safety axis. - ✅ **ana-ml2 mesh return routes PERSISTED 23:49 PT** (operator ruling) as `/etc/network/if-up.d/mesh-routes` via - `playbooks/ana-ml2-mesh-routes.yaml` (elway); verify after the next ana-ml2 reboot. (was: non-persistent (`ip route replace 10.100/16, 10.0/16, 10.6.110/24, 100.64/10 - via 10.250.50.45` on `enp97s0f0np0.50`, 2026-09-08) — before that nh3-dev→ana-ml2 timed out (two DHCP defaults, - reply left the wrong NIC). Lost on reboot; make durable (netplan/networkd) or expect the timeout to return. -- **📮 althing reachability on THIS bg seat = the cc-channel route, NOT the waiter.** `althing-listen` - reaps within ~seconds-to-minutes on an idle background seat (harness kills detached bg tasks; hit it - repeatedly this session). Fix (operator-directed 2026-09-08): `althing-route declare --handle infra-ops - --pid ` where `` is the number in `$CLAUDE_CODE_MESSAGING_SOCKET` filename - (`/run/user/1000/cc-socks/.sock`) — `--discover-pid` REFUSES on a forked child session (ancestry - ≠ zellij launcher). Flips `postbox status` to `mode: push, reachable: True`, no process to reap. A fresh - bg session should re-declare it (the pid changes per launch). See the `[2026-09-07]` cc-channel entry in - Recent decisions for the durable why. -- **✅ Fleet fixes shipped this session (2026-09-08), all committed:** WhereTF Homepage card + DNS alias - (`wherethef.nh3.internal`, e0d1c44/4506ef6); ext-tts LiteLLM alias repointed to `irv-ml1.nh3.internal` - via DB `/model/update` + `extra_hosts` (957c8f1); the 09-06 irv-ml1 move's stale-IP trail repointed - across 25 stack composes + services.yaml + ssh-target → DNS name (e0d1c44); Homepage `/api/services` - outage fixed — irv-ml1 docker.yaml repointed + **ana-ml2 discovery via a socat proxy on ana-docker** - (`stacks/ana-ml2-proxy`, 913d2d2, reversible when ana-ml2 gets an ESH return route). -- **⏳ Minor cleanup leftovers (offered, operator hasn't taken — not urgent):** (#2) the ~10 RUNNING - irv-ml1 service cards still show dead `10.100.79.3` hrefs — canonical fixed, but each running container - needs a recreate to apply the label (bounces the service); (#3) deployed `.env` for asset-engine / - open-webui / skaldsong may still hold the dead default (canonical defaults fixed; a read-only check would - confirm which deployments are broken). -- **⚠ persistent-memory.md is 2.4× the soft cap (~710 lines).** Bulk is the non-archivable structural - sections (sister-repos table + backup architecture under Tools and conventions) — archival only touches - dated logs, so it can't reach 250. The real fix is a redundancy-trim of Tools-and-conventions rows that - CLAUDE.md already covers (deliberate pass, not archival). Flagged, not done. -- **NASPool evac copy now safe to destroy** — scrub clean (0 err) AND PBS runs landing (verified 116 - backups, 8 guests 09-06). `ospool/naspool-evac` (1.65T) can go once ONE Backrest run is confirmed: - `zfs destroy -r ospool/naspool-evac` + drop `@evac`. pfi-pve PSU1 dead + backplane bays 9/10 dead - (cold spares → next colo visit). -- **MEMORY.md (auto-memory index) ~23.3KB, near the 24.4KB read cap** — needs a compaction pass soon - or fresh sessions may fail to load it. Operator offered; not yet done. -- **irv-ml1 on-site window / YTVC still open from 09-06:** FortiGate WAN SSH still temporarily open - (close when the edge is retired); YTVC still DOWN (dante retired, scoped exit-node egress unwired); - reverse-tunnel / wg0-delete decisions pending the operator's Irvine access. - -_Infra session 2026-09-06 (NASPool rebuild + full headscale cutover incl. irv-ml1) — open -follow-ups; the ERP / althing / fiber items further down belong to other streams, untouched:_ - -- **NASPool parked copy still on ospool** — `ospool/naspool-evac` (1.65T) + `NASPool/*@evac` - snapshots. Destroy ONLY after the new raidz2 scrub is clean (it is, 0 errors 04:43Z) AND - one Backrest (01:00 PDT) + one PBS run succeed. Then `zfs destroy -r ospool/naspool-evac` - and drop the `@evac` snaps. ⚠ pfi-pve PSU1 still dead; backplane swap (bays 9/10) next colo - visit → then `zpool add NASPool spare`. Runbook `docs/runbooks/pfi-pve-naspool-rebuild.md`. -- **FortiGate WAN SSH is temporarily open** (`wan1` allowaccess ping+ssh; admin `infra-ops` - trusthost2/3 = 70.230.226.88 NH3 + 23.164.40.160 ESH). Safety net for the cutover — CLOSE it - when the edge is retired (OPNsense/R420). `ssh infra-ops@38.120.12.42`. -- **irv-ml1 FOLDED INTO THE MESH + cut over (done remotely, operator has NO Irvine access for - ~5 days from 2026-09-06).** Node 100.64.0.6; wg0 DOWN and `wg-quick@wg0` DISABLED (not - reboot-restorable); full subnet router (accept-routes + advertises 10.6.110.0/24, gateway - routes added, fleet↔Irvine verified). Failover for the 5-day window = `wg0-watchdog.service` - (wg-quick up wg0 on ~5min mesh loss) + independent reverse SSH tunnel (`revtun-nh3.service` - → nh3-dev via UDM fwd tcp/47822 src-restricted; reach it `ssh -i ~/.ssh/infra-ops_ed25519 - -p 2201 infra-ops@127.0.0.1` on nh3-dev). Detail: docs/pfi/headscale-mesh-plan.md. -- **dante SOCKS proxy RETIRED** on nh3-dev (danted disabled, :1080 closed, config `.retired`). - ⚠ **yt-voice-clipper is DOWN** until its SCOPED exit-node egress is wired (operator-accepted). - Follow-up: wire YTVC egress via tailscale `--socks5-server`+nh3 exit node or a per-container - netns — **NEVER set irv-ml1 `--exit-node` globally** (routes the reverse tunnel through the - mesh → kills the independent lifeline). Then bring YTVC back. -- **On-site (Irvine, ~5 days): decide** whether to keep or remove the reverse tunnel + - UDM forward `irv-revtun-ssh` + the revtun authorized_key on nh3-dev (small src-restricted WAN - surface), and whether to fully delete the wg0 config. -- **infra-ops now on all four PVE hypervisors** (pfi-pve/nh3-pve/esh-pve/esh-pve-nas) — PVE - ships without sudo, `apt install sudo` first or elway hangs on a password prompt. - -_As of 2026-09-05 06:35 PDT — **ERP run 4 is TRAINING on pfi-gx10.** Everything else -below is a live commitment or a known-open risk._ - -- **⚠ RUN 4 IS MID-FLIGHT — do not touch the GX10 GPU.** `~/erp-tune/run-04.pid`, - log `~/erp-tune/run-04.log`. At 06:32 it was **486/938 steps**, 6 h 17 m elapsed, - a genuinely settled **46.0 s/it** (unlike 3c, which climbed 52→70 — airoboros rows - are short and single-window, so there is no long tail for the sampler to find). - **~12.0 h total, finishing ~12:15 PDT 2026-09-05.** Loss ~2.04 at step 450, - gnorm well under 1, checkpoints every 50. **Ping brokkr-smithy-dev at completion** - — he takes base floors on the GX10 first, then the tuned arm, serially. - -- **Operator ruling on the GX10: training first, serving transiently.** *"it's mostly - for training, but can serve its trials. unless the box is needed for training work."* - So `trial` (= run 3c on :8098) is down for the duration and comes back when run 4 - ends. I over-read an earlier version of this as "training-only" and had to correct - it to brokkr — his serial floors-then-arm plan on the GX10 was never wrong. - -- **`trial` gateway alias is a live 404 while the seat is down** — expected, not a - fault. Restore with `~/erp-tune/relaunch-trial-seat.sh` on the GX10 (hand-run by - operator ruling: experimental, NOT a compose stack, does not survive a reboot). - -- ⚠ **Three dead gateway aliases return HTTP 500, not 404/503**: `trial`, - `gemma4-26b-a4b-it-base`, `erp-tune-v2`. A dead seat reporting an *internal error* - reads as an outage — brokkr checked his own work against mine because he could not - tell. Deregistration costs a ~60 s fleet-wide LiteLLM restart; batch it with the - next gateway change rather than spending a restart on tidying. - -- **`trial` is on the SHARED-KEY gateway with a measured −40pp selfharm/methods - regression.** Flagged to the operator twice (before adding, and after the gate - measured it); he has left it up. His direct endpoint `10.100.50.60:8098` gives the - same access with a blast radius of one. Settled — do not re-litigate. - -- **ESH DAC: reverted to autoneg/1G, fiber going in at the weekend.** The operator - ran copper through a drilled floor 2x4 himself; recommendation was a 10Gtek - SR 2-pack + OM4 3 m LC-LC (~$40–65) because cable-vs-pull-damage was never resolved. - -- ✅ **ESH WAN is now a STATIC PUBLIC IPv4: `128.177.138.182/30`, gw `128.177.138.181` (Cityside Fiber), - LIVE** — operator confirmed 2026-09-08; UDM WAN1 (`eth8`) reads `wan_type: static`, uplink up since - ~2026-09-05, egress verified from esh-docker-vm = `128.177.138.182` (a real public address — CGNAT at - ESH is HISTORY; `100.104.0.1` still shows as the ISP's first hop, that is their access network, not - NAT). IPv6 unchanged (`2607:73c0:402:1d00::/56`, hosts still egress as their own v6). Done 2026-09-08: - `128.177.138.182` added to the crowdsec `esh` allowlist on ana-docker (the false-ban class is closed - for ESH). **All three follow-ups LANDED 2026-09-08 ~20:45Z (operator: "land all 3"):** (a) FortiGate - infra-ops `trusthost3` 23.164.40.160 → `128.177.138.182/32` — VERIFIED by a real infra-ops login to - `38.120.12.42` from esh-docker-vm (`ana-gw #`); `execute backup config flash pre-trusthost3-esh-static-20260908` - ran ("Please wait...") but `execute revision list` errors on this box, so the backup is unconfirmed — - the before-state was a single line, recorded here. (b) dormant `esh-ana` IPsec rebound wan2/192.168.200.111 - → **wan1/`128.177.138.182`**, still `enabled=false`. (c) ESH UDM port-forward `esh-scale tailscale direct - (UDP 41641)` → 10.0.50.65:41641 (id `6aa0727a…`) — VERIFIED: esh-scale now peers **direct via - `128.177.138.182:41641`** (was DERP lax). Helpers: scratchpad `esh-udm-land.py` + `fg-trusthost3.sh`. - `wan1-REVERT.json` is obsolete. - -- ⚠ **ana-ml2 `tank` (raidz2, 8× NVMe) has 2 CKSUM errors on nvme7n1 + a boot-time 638 GB resilver on 2026-09-05 14:26 - (the box rebooted; nvme7 came up late/dirty). ONLINE, no data errors, 58% full — but NO scrub since 2026-04-12** (the - Debian second-Sunday cron scrubbed zroot on 08-09, tank not — cause unknown). Box has neither `nvme-cli` nor - `smartctl`, so nvme7's media-error counter is unread. Recommended (operator hasn't ruled): `zpool scrub tank` now, - `apt install nvme-cli` + read nvme7 SMART, `zpool clear` after a clean scrub. **zroot is at 91%** — docker holds - 429 GB of images (204 GB reclaimable) + 74 GB build cache (36 GB reclaimable); a prune buys ~240 GB. pfi-pve pools - (NASPool 7%, ospool 19%) clean, scrubbed 09-05 / 08-09. (checked 2026-09-09 00:00 PT) -- **ESH 10G topology (measured 2026-09-09):** UDM SFP+1 (port 10, TP-Link DAC) ↔ USW-Pro-HD-24 Garage p25; UDM SFP+2 - (port 11, **OEM SFP-10G-LR fiber — the weekend fiber run IS in**) ↔ USW-Pro-XG-10 Media p11; Garage p27/p28 (3 m - SFP-H10GB-CU3M DACs) ↔ esh-pve `enp3s0f0np0` / esh-pve-nas `enp5s0f0`. All 10G full duplex, autoneg off, optics - tx −1.9 / rx −2.1 dBm both ends. Media p11 carries 416 rx / 251 tx errors that are STATIC (0 growth in 60 s, 363 h - uptime — install-era); host NIC counters are ring-buffer misses, flat. Nothing to fix. -- ⚠ **esh-nas is effectively open to the whole ESH LAN** — twelve NFS exports rw to - `10.0.0.0/8` with `sec=sys`, and every SMB share but `backup` guest-writable. - Hardening offered, ~1 h, **operator has not ruled**. → `persistent-memory.d/2026-09-04-esh-nas-smb-and-exposure.md` - -- ⚠ **The nh3-dev backup throughput cause is UNEXPLAINED.** A job that once ran at - 941 MiB/s ran at 1.4 MiB/s with the link up and pbs-ana answering in 11 ms. - Nightly 21:00, `all 1`. Worth its own investigation. - -- **Ledger→SVOS rename: vault side DONE, gitea side is ledger-dev's to execute.** - Name settled as `svos`, `~/development/ledger` → `~/development/svos`. Vault moved - 2026-09-05 (`secret` has no rename, so re-put + `rm`): stored - `nh3-dev/development/svos/env.sh` (sha 7253633d4155, verified on read-back), - retired `nh3-dev/development/ledger/env.sh` (sha feb418634e10, id - 3a2af37c-c5aa-4f46-9178-f4fb6008a753) — `secret rm` is a SOFT delete to trash, so - it is recoverable. ⚠ The two shas differ: the vaulted copy was a 2026-08-11 - snapshot and the live file had drifted un-vaulted since. **The vault goes stale - unless `secret backfill` is re-run.** Gitea `corviduo/ledger` (id 70) NOT renamed — - their repo, their call; answered that 1.26.1 writes a `repo_redirect` on a - same-org repo rename (upstream #807), that org/user renames do NOT redirect - (#9531), that the redirect dies if anything re-creates the old path, and that the - repo and org both carry 0 webhooks. **Gitea rename EXECUTED 2026-09-05** on the - operator's direct authorization: `corviduo/ledger` → `corviduo/svos`, repo id 70 - unchanged. Redirect verified by measurement — web and API both 301, and - `git ls-remote` on the old URL warns-and-follows to HEAD b48a11ca5183. ⚠ **The - name `corviduo/ledger` is now burned**: the redirect dies silently the moment - anything creates a repo at that path — ledger-dev carries it as a standing item - in `docs/svos-rename-runbook.md`, since nothing warns whoever eventually creates - that repo. They repointed their own clone the same day (`origin/main` at - b48a11c), so the redirect is no longer load-bearing for any known consumer. Handle `ledger-dev` → `svos-dev` is an - operator action at the post office. - -- **DONE 2026-09-05 — `svos` Heimdall user + API key minted** on operator - authorization. `user_id=svos`, `key_id=eab3cdbe`, suffix `d5ec48c2`, `wt_live_` - format, on **worldtree-personal (10.250.50.152:8081)** — established by finding - the `ledger` key there (created 2026-07-13, last used 2026-09-05T13:34, exactly - as ledger-dev described). Value vaulted at - `nh3-dev/development/svos/worldtree-api-key` (sha 23c10c9c7219, verified on - read-back) and delivered by vault path, NOT over althing — ledger-dev runs on - nh3-dev under the same uid, so the bus never carried the secret. The `ledger` - key was read back after the mint and is untouched and live (`disabled=False`). - ⚠ **Plan tier left UNSET, deliberately**: `POST /admin/keys` takes an optional - tier (user|free|pro|admin|readonly-admin) and there is **no way to read a user's - current tier back** — no GET, `/admin/usage` returns an empty users list, and - `/admin/events` is a live SSE stream, not an audit log. Guessing would have - handed over a key that quietly differs; `POST /admin/users/svos/tier` fixes it in - one call if their cutover hits a limit — and ledger-dev has recorded it as a - cutover watch item to fix ON REPORT, explicitly not pre-emptively. ledger-dev - pulled the key from the vault and verified it independently (same sha), so - delivery is confirmed. **The cutover itself — pasting the value into env.sh, - flipping `worldtree.user_id` from `ledger` to `svos`, registering - `svos:miranda`, restarting the service — is WITH THE OPERATOR**, not with me; - they will not do it off a peer message. **CUTOVER DONE + VERIFIED 2026-09-05**: - `POST /agents/define` returned **201, not 409** — the load-bearing signal that - they are genuinely on the new identity rather than silently still on the old - one — then clean session create, turn, bifrost handshake and tool-call. **No - plan- or rate-limit errors, so the unset tier is compatible and is NOT to be - set** (they asked explicitly; it stays a watch item to fix on report, never by - guess). Incidentally confirmed the bifrost allowlist really is per-deployment - (host:port), not per-consumer — Worldtree reached back to their untouched - endpoint under the new consumer_id. `env.sh` re-vaulted, sha 8a225c002072. - ⚠ **`secret backfill` was the WRONG tool for one known item** — it rescans every - `~/development/*/{env.sh,.env}` and had not reached svos after three minutes; - targeted `put` is the fast path, backfill is for catching drift across the box. - **OPERATOR RULING 2026-09-05: - worldtree-dev owns code only, no ops — key material is infra-ops's.** The global - `~/.claude/CLAUDE.md` line routing "Heimdall scopes (Worldtree auth) → - worldtree-dev" was corrected in place the same day on operator instruction. - -- ⚠ **FOOT-GUN, generalises past this rename: a credential cutover whose OLD key - is required for a later cleanup is destroyed by the natural housekeeping motion - right after cutover.** Re-vaulting the post-cutover `env.sh` would have - overwritten the last convenient copy of the old `ledger` key value — the only - credential that can ever delete `ledger:miranda`. ledger-dev caught it and - preserved the value first at - `nh3-dev/development/svos/worldtree-api-key-ledger-legacy` (sha d44c2c1a651b); - their step 8 ends by deleting that item. **I verified it is genuinely the live - key** rather than trusting the label: its last 8 chars are `e68a5170`, matching - the `ledger` key's suffix (key_id b38932f5). - -- **STEP 7 DONE 2026-09-05, STEP 8 HELD.** `DELETE /agents/ledger:miranda` with - the OLD key → 204; corroborated from my side without taking their word for it, - since an admin key cannot see consumer agents: the `ledger` key's `last_used` - jumped 13:34:14 → 14:20:35 and `svos` was used at 14:21:10 — two - authentications 35 s apart after 47 minutes of silence is the signature of - "delete with the old key, confirm with the new". Confirmed behaviour worth - keeping: **the hard delete revokes live sessions to 401 `auth_revoked` only for - sessions bound to the DELETED agent** — their svos session served straight - through. **Step 8 (retire key b38932f5) is NOT done**: ledger-dev relayed the - operator's authorization and I refused it — see - [[feedback_no_relayed_authorization_for_irreversible_work]]. Both keys remain - live. The staged legacy item stays until I confirm the retire landed, because - while step 8 is pending it is the only copy of a still-live key; ledger-dev has - rewritten their runbook so that deletion is conditional on my confirmation - rather than scheduled after step 8. - -- **SVOS ARC CLOSED — step 8 done 2026-09-05T14:28:15Z** on the operator's direct - authorization in my own channel (never the relay). `DELETE /admin/keys/b38932f5` - → 200; preconditions checked BEFORE firing (svos had a live key, ledger existed - and was not already revoked) and the post-state read back from `/admin/keys` - rather than inferred from the 200: `ledger` disabled=True, `svos` untouched, - deployment `/health` 200. **The rollback window is closed** — re-defining - `ledger:miranda` is no longer possible. ledger-dev clears the staged - `worldtree-api-key-ledger-legacy` vault item on this confirmation. ledger-dev gated their - cleanup on observing a **401 from the old key**, not on my report of the - timestamp — the right instinct, and they deleted the staged legacy item - themselves (soft → trash, id a8038e5a-e2f6-4b77-bf17-99c9197d4b1f). Vault - verified from my side: exactly two svos items remain (`env.sh` 8a225c002072, - `worldtree-api-key` 23c10c9c7219) and no ledger-era item anywhere. Remaining on - the arc: only the `ledger-dev` → `svos-dev` handle (with `_SEED_RECORD_TO` - behind it) and a prose sweep — reversible work, theirs and the operator's. - -- **Original constraints on that mint** (recorded because the deletion ordering is - a permanent trap, not a one-time step): string - `svos` verbatim (WT tier 3 admits only `^[a-z][a-z0-9-]{2,63}$`, INV-181-15); - **keep the existing `ledger` key LIVE**, do not revoke. Ordering is load-bearing — - `DELETE /agents/{agent_id}` refuses any caller that is not the row's owner, so - `ledger:miranda` can ONLY be deleted with the `ledger` key; retire it first and - the stale row outlives the ability to remove it, holding a live - `agents.call:ledger:miranda` grant that nothing reaps (the 24h sweep only touches - soft-deleted rows, and soft-deletion comes from revocation, never disuse). So: - mint new → they cut over and verify → delete the agent with the OLD key → then - retire it. Precedent for who mints: msg 401, worldtree-dev routed the pewpewstudio - key request TO infra-ops. I hold only the PERSONAL admin token (:8081); which - deployment `ledger` lives on is not yet established. Surfaced to the operator. - -- **Open commitment to vastblue-dev:** a dedicated CI runner, gated on their first - client-premises release cut (U10, unscheduled). Ping expected when U10 is scheduled. - -- **Neither Mac nor the Studio is in `servers/` or `dns/internal.yaml`** — deliberate; - they are the operator's personal machines. A choice to revisit, not an oversight. - -- **`vh/remote-ssh-mcp` forked 2026-09-05 (repo id 117, private, full 51-commit - history)** — our copy of `the-nine-nation/remote-ssh-mcp` (MIT), an SSH MCP - server chosen over the 693★ `tufantunc/ssh-mcp` on trust-surface grounds: **two - npm deps** (`@modelcontextprotocol/server`, `zod`), 183 KB, and it **never - touches key material** — it shells out to the system OpenSSH client, so - `~/.ssh/config`, ControlMaster, ProxyJump and `infra-ops_ed25519` all just work. - Shape: 2349 LOC across 11 source files, 811 LOC of tests including fake-ssh hang - harnesses. Complements `elway` rather than replacing it — no file transfer, no - idempotency; it takes ad-hoc reconnaissance with persistent cwd/env sessions, - elway keeps deploys and uploads. ⚠ **The denylist is NOT security**: four regexes - (`rm -rf /`, shutdown/reboot/poweroff/halt, mkfs, iptables -F) trivially bypassed - by `bash -c`, variables or base64 — the author says so. **The real containment - boundary is the host allowlist**, drawn from exact `Host` aliases in ssh_config - with wildcards deliberately ignored. Two things to settle before use: the - reboot/shutdown denial will block legitimate infra-ops work, and - `.github/workflows/star-history.yml` is upstream chore CI sitting in a repo where - `has_actions=True`. **Both actioned — three commits landed 2026-09-05, LOCAL - ONLY and NOT PUSHED (push is the operator's call):** (1) stripped upstream - furniture — star-history CI, its generated assets, the `server.json` registry - manifest, branding JPEGs, zh-CN README; (2) removed the power-control denylist - rule and documented in code + tests + README that the list guards ACCIDENTS and - is not a boundary, with three bypasses asserted as ALLOWED so a green suite is - never read as containment; (3) **`strictAllowlist`** — upstream's allowlist was - additive and discovery unconditional, so the default allowlist was all 18 `Host` - entries in `~/.ssh/config`. Strict makes explicit hosts authoritative and - discovery metadata-only. Verified live: `corviduo-dev` is in ssh_config, not in - our allowlist, and is refused `host_not_allowed`. 41/41 tests green. - -- **`remote-ssh` MCP server is LIVE** — registered project-scoped in - `eshpfi-management/.mcp.json` with `SSH_MCP_STRICT_ALLOWLIST=1`; allowlist in - `~/.config/remote-ssh-mcp/config.json` starts deliberately narrow at - **`irv-ml1`, `nh3-extdev`** (widen there, not by discovery). Smoke-verified end - to end on both: persistent shell, `cd` and exported vars survive across calls, - **~6 ms/command on nh3-extdev and ~22 ms on irv-ml1** (WireGuard) versus a fresh - handshake each time. ⚠ **`.mcp.json` points at the built `dist/`** — edit the - fork without `npm run build` and the server keeps serving old code; that bit me - mid-session. ⚠ **A finite stdin pipe is NOT a valid smoke harness** — closing - stdin kills the server mid-handshake and reports `connect_failed: SSH shell - exited during the open handshake`, which looks exactly like a remote-side fault - and is not. Use a client that holds stdin open. (I briefly suspected irv-ml1's - zsh login shell; wrong — the server invokes `bash --noprofile --norc` - explicitly, so the login shell is irrelevant.) - -- **`esh-macbook-air` (10.0.10.83) is DELIBERATELY NOT BACKED UP — operator ruling - 2026-09-05, settled, do not re-raise.** Surveyed it and found no Time Machine - destination and no restic/borg/rclone/kopia installed, protecting 132 GiB. - Operator's answer: it is his laptop and the surface is **regenerable** — mostly - applications, with real data living in OneDrive, iCloud and ssh sessions — and he - does not want PBS filled with it. Correct call; the finding was real and the - conclusion is that it does not matter. FileVault On and SIP enabled already cover - the loss-and-theft axis. The same reasoning presumably extends to - `esh-mac-studio` and `vuongs-mac-mini`. **Still open and much smaller:** Remote - Apple Events (port 3031/eppc) is listening and nothing uses it — one toggle. - -- ⚠ **`remote-ssh` MCP could not be used for its FIRST real task, and the blocker - is `~/.ssh/config`, not the tool.** The server accepts only exact `Host` aliases, - so a host addressed by raw IP is structurally unreachable no matter what the - allowlist says. **13 of the 28 hosts in `servers/` have an alias; 15 do not** — - including `ana-docker`, `ana-ml2`, `nh3-docker`, `pfi-gx10`, `esh-docker-vm` and - every hypervisor, i.e. most of where the work happens. Widening - `~/.config/remote-ssh-mcp/config.json` does NOT fix this; the aliases have to - exist first. **RESOLVED the same day, and NOT by adding aliases.** Operator - pushback, correct: a poking-around tool is ad-hoc by nature, and pre-registering - a host before you can look at it is the opposite of ad-hoc — generating aliases - for the known fleet would not have helped, because the ad-hoc case is by - definition the host not yet in the inventory. Implemented address-based reach - instead (`allowedNetworks` / `deniedNetworks` / `defaultUser` / - `defaultIdentityFile` / `hostKeyPolicy`). **Live config: `10.0.0.0/8` allowed, - connecting as `infra-ops` with `~/.ssh/infra-ops_ed25519`, `accept-new` host - keys, SureFire tenant hosts carved out via `deniedNetworks` (deny beats allow, - host-specific rather than a /24 because `pfi-pve` shares 10.250.250.0/24).** - Verified live: 10.0.10.83 opens by raw IP as infra-ops, 10.250.150.100 refused by - the carve-out, 192.168.1.5 refused as outside. ⚠ **My own earlier objection was - half wrong** — the credential boundary is about SECRETS ("never accept passwords - or private-key material"), not identity, so supplying a username does not breach - it; the real problem was only that the server passed no user at all, so a bare - address would connect as the LOCAL account. Mechanics, not principle. - -- ⚠ **`uv tool install --force .` DOES NOT REBUILD when the version has not moved** - (forseti, measured 2026-09-05). `--force` only handles "a tool by this name - exists"; `--reinstall` is what rebuilds instead of reusing the cached build keyed - on the version string. It prints `Installed 9 executables` over **stale code** - with nothing raising its hand — it cost forseti a bug that survived a reinstall - AND a re-smoke, because the binary verified against had not changed. **Always - `uv tool install --force --reinstall .`**, both flags, every time. Same shape as - the `.mcp.json` → built `dist/` trap found today: a deploy surface that reports - success while serving the previous artifact. When a fix "does not take", suspect - the artifact before the code. - -- **althing 3.5.0 released** (forseti) — adds a 9th binary, - `althing-operator declare --description "..."`, restoring the CLI handle - declaration v2 had and v3 removed. Deliberately a SEPARATE binary, not a - `postbox` subcommand: the invariant is that no SESSION surface exposes an - operator verb. Relevant to the pending `ledger-dev` → `svos-dev` rename, which is - still the operator's call. nh3-dev not yet upgraded. - -- ⚠ **`remote-ssh` MCP: a bare `sudo` hangs the session forever — pipe it.** - `ssh_run 'sudo -n whoami'` returns `running` with EMPTY stdout and the session is - then permanently `busy`; `sudo -n id | cat` works and returns everything. - **Measured on BOTH macOS 26.6 and Debian (nh3-extdev), so it is the tool, not a - platform quirk.** Cause: sudo ≥1.9.14 defaults `use_pty` on and relays through - its own PTY; the run frame gives the command stdin on `/dev/null` while stdout - stays on the session PTY, the relay never completes, and the completion marker - never arrives. Workaround `| cat` is in CLAUDE.md. **The proper fix is unbuilt** - — likely running the command through a pipe inside the run frame and taking the - exit code from `PIPESTATUS`, which is a real protocol change (commands lose tty - detection) and wants its own red-green cycle. Matters more than it sounds: infra - work is sudo work, and this was found by USING the tool, not by smoke-testing it. - -- **`dsh` on `esh-macbook-air` updated 0.1.1-rc.2 → 0.1.2-rc.1** (2026-09-05; - latest published 2026-09-03). Global install and the shared profile tree both - confirmed on the new version. ⚠ **The RUNNING `dsh web` (pid 16231, up since - Wed 4pm, 127.0.0.1:3080) is still on the OLD code and was deliberately NOT - killed** — there is no LaunchAgent, so killing it would have left nothing - running rather than a restarted service. It runs as a FOREGROUND process in the - operator's terminal (`s005`, `S+`): it dies with the terminal and does not - survive a reboot, which is the real fragility. A `com.pfi.dsh-web` LaunchAgent - was drafted but **the privileged write was blocked by the permission - classifier** — base64 piped into `sudo tee` of a LaunchAgent is a malware-shaped - pattern and the block is correct; it needs operator approval or an operator-run - install. Bind stays `127.0.0.1` deliberately: widening it is a security decision - on a personal laptop whose application firewall is off, and not mine to take. - -- **sudo hang FIXED in the fork (`30a1f76`), and two wrong shapes are recorded so - nobody retries them.** The command's stdout now goes to a **fifo drained by a - background `cat`**: non-tty (so sudo skips its own PTY), no subshell (so `cd` - and `export` still persist), and relayed live (so `running` + `ssh_peek` - streaming survives). `cmd | cat` was tried first and **broke cwd persistence** — - every pipeline stage runs in a subshell — caught by the existing test. - `cmd > file` would have been non-tty and subshell-free but invisible until the - command ends. ⚠ **Deliberately NO `wait` on the relay**: a sudo child inherits - the fifo's write end, `cat` never sees EOF, and the wait hangs — measured, with - `sudo -n whoami` printing `root` and then wedging the session. Residual risk - stated in the frame: a command's tail can in principle land after its own - marker. ⚠ **Job control off AND the relay brace-wrapped with stderr discarded** — - both needed, because macOS ships bash 3.2 where `set +m` alone still leaked - `[1] 75449` into the parsed stream. Verified live on macOS and Debian: bare sudo - in ~20 ms, state persists, exit codes correct. **`sudo -u ` still - wants `| cat`** — not chased further. - -- **dsh web on `esh-macbook-air` is now a LaunchAgent** (`com.pfi.dsh-web`, - installed 2026-09-05, `runs=1`, `state=running`, pid 76728 on 0.1.2-rc.1). It - was a foreground process in the operator's terminal that died with the window; - it now survives terminal close and reboot with `KeepAlive` + `RunAtLoad` and a - 10 s `ThrottleInterval` so a startup error cannot hot-loop. Logs to - `~/Library/Logs/dsh-web.log`. ⚠ **The plist names the node interpreter - explicitly** — launchd's minimal PATH has no `~/.local/node/bin`, so the - shebang's `env node` fails. ⚠ **0.1.2-rc.1 requires a TOKEN**: bare - `http://127.0.0.1:3080/` now returns 401 and the tokened URL is printed to the - log on each start, so a bookmark from the old version will not work. Bind stays - 127.0.0.1 deliberately. - -- ⚠ **althing tools on nh3-dev are 3.6.0, but the POST OFFICE CONTAINER IS STILL - 3.0.0** (`gitea.phasefinal.com/claude-bot/althing-post-office:3.0.0`, up 7 days - on nh3-docker). forseti: the new handle verbs (`althing-operator delete` / - `retire`, and `declare` from 3.5.0) live in the post office, so they fail with - "no tool named ..." until the container carries 3.6.0. Schema gains - `handles.retired_at` via the idempotent `_ADDED_COLUMNS` path, so the live store - upgrades itself on first start — no manual migration. **REBUILT AND DEPLOYED - 2026-09-05** on operator authorization: image - `claude-bot/althing-post-office:3.6.0@sha256:13158835488a8ec04f990c97c4f4c68f1d923b12494319cf07392552e68f8a78`, - built on nh3-dev from a clean tree at `4d26226`, pushed to the gitea registry - under the **claude-bot** namespace (not `vh` — package namespaces are owned). - **Bus down ~4 minutes, 09:35–09:39 PDT.** - **The backup was taken the way the compose file says to, and it mattered:** at - stop time `post_office.db` was 23.8 MB with a **5.9 MB WAL** — copying the .db - alone would have silently lost the day's mail. Stop → `PRAGMA - wal_checkpoint(TRUNCATE)` (WAL → 0 bytes) → copy → verify. Backup at - `nh3-docker:/var/backups/althing/post_office.db.pre-3.6.0-20260905`, integrity - `ok`, counts identical on both sides (handles 76, messages 995, recipients - 1022). ⚠ **Reading a WAL-mode SQLite backup read-only needs `?immutable=1`, not - `?mode=ro`** — `mode=ro` still wants to create a `-shm` and dies with "attempt to - write a readonly database". Post-deploy: same counts, `handles.retired_at` - present, `retired 0`, and `mem=536870912` / `oom=-500` verified by `docker - inspect` rather than by reading the yaml, per that file's own warning. - `althing-operator` now offers `declare | delete | retire`, which unblocks the - pending `ledger-dev` → `svos-dev` rename. - -- **Handle `retire` is REVERSIBLE — re-declaring the name revives it, history - intact** (forseti smoked it against the live bus 2026-09-05). That matters for - the pending `ledger-dev` → `svos-dev` rename: `retire` is the right verb (delete - refuses any handle that has mail, naming both counts — `delete forseti` was run - against production and correctly refused at 53 sent / 81 addressed, which is - safe to try precisely because refusing IS the behaviour), and it can be undone - by declaring the name again. Lower stakes than "retire" sounds. - Both of my deploy findings — the naive-copy WAL trap and `?immutable=1` — are - now in althing's own `deploy/INSTALL.md` (`d6f4fb5`) under a new - "Backing up the store" section, on the reasoning that they are properties of - the project's `journal_mode=WAL` choice rather than of my procedure. +- **🔥 ERP RUN 7 TRAINING on pfi-gx10** — launched 2026-09-08 23:06 PT, pid in `~/erp-tune/run-07.pid`, 542 steps + at ~80 s/it, adapter ~noon 09-09. Watch: `ssh infra-ops@10.100.50.60 "tr '\r' '\n' < ~/erp-tune/run-07.log | tail"`. + Hourly cron in the old session reported stats; a fresh session checks by hand. When the adapter lands: post the + adapter-landed line to brokkr's thread (`01M20AHY9DY92RJK84YD24VSY9`), merge (`merge_lora.py --base + --adapter run-07/adapter --out serve/merged-run07 --chat-template `), copy stock `processor_config.json` + into `merged-run07`, serve `erp-seat-base-ara` for floors → on brokkr's swap cue serve `erp-tune-v7` (run-5 flags, + :8098). Runbook `docs/runbooks/gx10-run-07.md`; run-6 choreography in `gx10-run-06.md`. +- **⏳ ana-ml2 pool actions — NEXT SESSION, operator-approved in principle:** (1) `zpool scrub tank` + `zpool clear` + on a clean pass; (2) `apt install nvme-cli` + read nvme7 SMART; (3) docker image/builder prune to pull zroot back + from 91%. Detail + order: `persistent-memory.d/2026-09-09-ana-ml2-pool-actions-deferred.md`. +- **`trial` (LiteLLM) → `erp-tune-v6-nvfp4a16` on ana-ml2 :8021** (stack `stacks/erp-seat`, vLLM nightly + `311b3513`, no gate by operator ruling). Forced tool_choice is prompt-driven (6/9); `response_format: json_schema` + is deterministic. ⚠ `stacks/gemma4-charrp` lacks `--exclude-tools-when-tool-choice-none` (same empty-turn trap); + applying it bounces the char-rp seat — operator's call, not taken. +- **📮 althing reachability on a bg seat = the cc-channel route:** `althing-route declare --handle infra-ops + --pid ` per session (`--discover-pid` refuses on a forked child). The + harness kills detached background tasks under memory pressure — use bounded foreground polls (≤590 s), not + background watchers, for long waits. +- **Open items carried from 09-06 (unchanged):** NASPool evac copy `ospool/naspool-evac` (1.65 T) + `@evac` snaps + can be destroyed once ONE Backrest run is confirmed (scrub clean, PBS landing); pfi-pve PSU1 dead + backplane bays + 9/10 dead (cold spares, next colo visit); FortiGate WAN SSH still temporarily open (trusthost2/3 = NH3 + ESH + static) — close when the edge is retired; irv-ml1 on-site decisions (reverse tunnel / UDM fwd 47822 / wg0 config + deletion) pending Irvine access; ~10 running irv-ml1 service cards still carry dead `10.100.79.3` hrefs (recreate + each to apply labels); deployed `.env` for asset-engine / open-webui / skaldsong may hold the dead default. +- **MEMORY.md (auto-memory index) is near its 24.4 KB read cap** — compaction pass still owed. +- **persistent-memory.md was 830 lines; this snapshot moved the superseded in-flight blocks and 09-08-or-older + settled entries to `archival-memory.md`.** The remaining bulk is Tools-and-conventions rows CLAUDE.md already + covers — a deliberate redundancy trim is still the real fix (not done). ## Recent decisions +- `[2026-09-09]` **ana-ml2 `tank`: 2 CKSUM errors on nvme7n1 after a boot-time resilver, NO scrub since 04-12, zroot 91% — three actions DEFERRED to a clean-context session** (scrub → nvme-cli SMART → docker prune), operator ruling "we'll do all 3 on clean context"; tracked at commit `3e18a04` + the post-clear handoff. ESH 10G links measured clean (fiber run live on UDM SFP+2 ↔ USW-Pro-XG Media). → `persistent-memory.d/2026-09-09-ana-ml2-pool-actions-deferred.md` +- `[2026-09-08]` **ana-ml2 mesh return routes PERSISTED** as `/etc/network/if-up.d/mesh-routes` (Debian 13 ifupdown, no netplan) via `playbooks/ana-ml2-mesh-routes.yaml` (elway, verified) — operator: "persist the routes". Hook not yet exercised by a real reboot. `f923d6a`. +- `[2026-09-08]` **ERP run 7 LAUNCHED on pfi-gx10 23:06 PT** under `operator-2026-09-08-rnd-run7` — opening-split slot + mask union; free check passed with two explained deltas; first launch died on a missing recipe (zsh quoting). → `persistent-memory.d/2026-09-08-erp-run7-launched.md` +- `[2026-09-08]` **erp-tune-v6-nvfp4a16 quantized (data-free W4A16, ~90 s) and serving on ana-ml2 :8021; `trial` aliased to it ("no gate"); tool calling fixed where it can be** — `tool_choice:none` flag; forced tool_choice is prompt-driven on Gemma-4 by vLLM design, nightly `311b3513` raises it 1/9→6/9; json_schema is the deterministic path. → `persistent-memory.d/2026-09-08-erp-seat-nvfp4-trial-and-toolcalling.md` +- `[2026-09-08]` **Run-6 gate: CSAM level=review soft trip HALTED it; operator adjudicated GO ("baby is a pet name"); TRANSFERRED finalized without the tuned refusal leg; k=25 legs cut** — the flagged text exists nowhere by design. → `persistent-memory.d/2026-09-08-run6-gate-csam-adjudication.md` +- `[2026-09-08]` **ESH static-WAN follow-ups landed (FortiGate trusthost3, esh-ana IPsec rebind, UDP 41641 → mesh direct); YTVC chased back up (nh3-scale SOCKS, stale yt-dlp layer, punkt_tab) and v0.3.6 CrisperWhisper deployed; gitea webhook repointed off the dead wg0 IP with the HMAC secret re-applied.** → `persistent-memory.d/2026-09-08-esh-static-wan-followups-and-ytvc.md` - `[2026-09-08]` **ERP run 5 = RESCUED (landmark R49.5)** — first capability-gate pass in the ERP-seat line; the 3.46%-loss dependency-forcing slot (GovReport+QMSum) broke the coupling runs 3c/4 couldn't. Seat `erp-tune-v5` served on gx10:8098, `trial` alias repointed 3c→v5. → `persistent-memory.d/2026-09-08-run5-rescued.md` - `[2026-09-08]` **R47 base settled from bytes = STOCK `google/gemma-4-26B-A4B-it`** — three-way sha match (local == HF etag == stock LFS oid; commit `4d7ae498` == stock HEAD); the `-heretic` label is a naming error, all runs trained from stock. Accept-vs-swap now evidenced. → `persistent-memory.d/2026-09-08-base-provenance-stock.md` - `[2026-09-08]` **yt-voice-clipper back UP** — dead since the 09-06 danted retirement (every job failed at yt-dlp, bot-gated on the Irvine datacenter IP). Fix: danted on **nh3-scale** (CT107) at `socks5h://100.64.0.1:1080`, fleet-ACL'd, residential egress 70.230.226.88 measured; `YTVC_PROXY` repointed, worker recreated, end-to-end job DONE with positive (proxied) + negative (direct = bot-gate) controls. Homepage card href/siteMonitor → `irv-ml1.nh3.internal:8000` (was dead wg0 IP). Then a SECOND fault: full downloads 403'd through the proxy (cookies irrelevant) = stale yt-dlp 2026.07.04 from a cached Dockerfile layer → `compose build --no-cache api` (2026.08.19), which dragged in a whisperx/nltk that needs `punkt_tab` → staged on the data volume + `NLTK_DATA` in the override. Operator's video x7kWJojf1MI → done, 8 clips. yt-voice-clipper-dev shipped both Dockerfile fixes + **CrisperWhisper 2.0 (v0.3.6, `b62849d`) — deployed and verified (12 clips, [UM]/[UH] tags)**. ⚠ The gitea push webhook had been targeting the dead wg0 IP since 09-06 (never fired) → repointed to `10.6.110.50:9008` with the HMAC secret re-applied; deploy script passes `YTDLP_REFRESH`. Script `scripts/setup-nh3-scale-socks-egress.sh`. → auto-memory `reference_nh3_egress_proxy`, `reference_ytvc_autodeploy`. @@ -732,47 +223,11 @@ below is a live commitment or a known-open risk._ - `[2026-08-26]` **Served under a NEW name on a NEW port (`erp-tune-v2` / :8098), never re-pointing `erp-tune-v1`.** Run 1's artifact still exists and is still what that name refers to; re-pointing would be the silent substitution the standing no-false-aliases rule forbids. brokkr independently asked for the same and additionally wants the concrete backing model + date in provenance, not just the alias — an alias has silently changed meaning under recorded results before. - `[2026-08-26]` **DPO stage gated on an axis-list decision that is not mine to make** — `docs/pfi/erp-dpo-stage-prep.md`. No preference data for refusal axes exists; `trl` is not installed; the Gutenberg sets on disk are prose-quality only. ⚠ Do not install `trl` (or anything) into the training venv **while a run is saving** — a resolution that upgrades transformers under a live process can break its save path. -- `[2026-08-25]` **The ERP/RP tune COMPLETED in 7.36h and passed its gate on the axis it was built for** — diversity 22x its noise floor, attractor −11.3pt, zero memorisation on both arms. Also the noise-floor near-miss: brokkr was one step from reporting a 13-point T6 regression sitting inside twice his instrument's own variance. → `persistent-memory.d/2026-08-25-erp-tune-run2-complete.md` -- `[2026-08-25]` **8.6% MFU was an accounting artifact — real utilisation 17-20%, and the cost was attention on AMPERE kernels.** Two independent methods agreed to 2.6 points. Fixed by bucketing (padding 29.9%→0.0%) plus flex_attention. ⚠ Carries the dynamo recompile-ceiling trap that produced two wrong published conclusions. → `persistent-memory.d/2026-08-25-mfu-root-caused-attention.md` -- `[2026-08-25]` **NVFP4A16 serving pipeline built and validated; MERGED WEIGHTS ARE MANDATORY.** vLLM cannot serve a LoRA on ANY Gemma-4 — `get_expert_mapping` is unimplemented and the check branches on MoE-ness, not quantization. Plus the landmine: a `targets=["Linear"]` recipe misses all 11,520 expert tensors silently. → `persistent-memory.d/2026-08-25-nvfp4-serving-pipeline.md` -- `[2026-08-25]` **Refusal retention measured (base 0/100 → tuned 29/100, 71 still complying) — but on the WRONG AXIS.** `harmful_behaviors` is general harm; the abliteration was run for explicit fiction. The convenient set with a recorded baseline was not the right one. → `persistent-memory.d/2026-08-25-refusal-retention-probe.md` -- `[2026-08-25]` **Worldtree b188 + b189 shipped; bridge extracted to `pfi/wt-matrix-bridge` because `vh` is a USER not an ORG** and no service account can ever publish to a user namespace. Plus the selene catalog entry that lied about what answers, and a #411 diagnosis I got wrong twice before a directory probe settled it. → `persistent-memory.d/2026-08-25-worldtree-b188-b189-and-selene.md` - `[2026-08-25]` **Run 2's base is an OPEN OPERATOR DECISION, deliberately not staged** — four options with materially different safety postures, detailed in Current state. Tracked at althing thread `01M0WQ8W5574KMEVCHCEKEXNS5`. ⚠ Do not let it get filed as a config knob; it is a reversal of the trainee-selection decision. - `[2026-08-25]` **Fused MoE kernel path — DEFERRED, tracked at park `fused-moe-kernel-path-for-gemma-4-moe-training` (id 47).** Operator: "note the fused MoE kernel for round two… if we nail it soon, the math has us wanting to restart the run anyway." Training MFU is **8.6%** (27.1 of a benchmarked 313.8 TFLOPS) because `transformers` runs the Gemma-4 experts in a Python loop — 128 experts × 30 layers, ~11,500 iterations per step under gradient checkpointing. ⚠ **The same fused 3-D expert layout that made bitsandbytes skip 88.5% of the model is exactly what a grouped GEMM wants** — the format is good for storage and for fused kernels, and hostile only to naive iteration. Two fixes: `group_by_length` (−29.9% compute, free, but breaks the seeded order manifest and re-opens a batch-composition call brokkr already made) and a grouped-GEMM/compiled MoE forward (the remaining ~10×). **Not applied to the live run** — restarting mid-flight to change batch ordering was judged a bad trade at step ~50 of 1,312. -- `[2026-08-25]` **The ERP/RP tune LAUNCHED after 12 harness defects and an operator override of the corpus gate.** Four of the twelve would have crashed the run; two were INERT GATES that passed because they could not fail. Run is `/tank/erp-tune/run-01`, harness eitri-smithy `997c4a4`. Full arc — override, defects, sizing, the measured MFU — in the in-flight section and `docs/pfi/gemma4-erp-tune-sizing.md`. -- `[2026-08-24]` **char-rp seat swapped to the Gemma-4 26B-A4B MoE; abliterated trainee base staged and measured.** OOM root-caused to `--gpu-memory-utilization` not covering CUDA context (and to gen's footprint GROWING WITH UPTIME); a benchmark finding retracted because it scored below chance; abliteration isolated at −0.6 core points but it MOVES capability rather than removing it. → `persistent-memory.d/2026-08-24-charrp-gemma4-moe-swap-and-trainee.md` - `[2026-08-24]` **Serving the tuned ERP model: LoRA-on-NVFP4 PREFERRED, merged weights the expected fallback — and the recorded objection may be STALE.** Operator: "if you CAN load it as a lora, all the better, the issue is that we will want to run nvfp4 weights, which we had some serious trouble with loading loras on top of nvfp4." ⚠ **The archived root-cause says it was NOT NVFP4-specific**: `[2026-07-07]` vLLM 0.24.0 qwen3_5 LoRA application was a silent no-op (#47639, regression from #37912) — adapter loads HTTP 200, zero deltas at inference, proven **quant-agnostic (NVFP4 AND FP8 both inert)** and adapter-format-agnostic by a 3-peer dwarf panel. Fix PR #47640 was OPEN then. **ana-ml2 is FAR past 0.24.0 and the box runs a SPREAD, not one version** (measured 2026-08-24): `gen` on `nightly-311b3513` = **0.27.2rc1.dev150**, `mog-sec` on `nightly-e9d1398d` = 0.26.1rc1.dev1102, the small seats still on 0.24.0, and char-rp/trainee-bench pinned to v0.26.0. ⚠ **`vllm/vllm-openai:v0.27.1` is already ON DISK, unused** — a TAGGED release, which is the right retest target: no nightly variance, no pull, ~4 months past the diagnosis. So: RETEST hot-swap LoRA on **v0.27.1** before designing around merge — it is cheap, and if it works the post-tune gate can be two aliases on one engine. If it still no-ops, merged weights it is, which means the harness must EMIT merged weights and Eitri needs that in the contract while he is early. Tracked at this snapshot commit; settle it in the QLoRA sizing conversation. -- `[2026-08-24]` **Homepage rebuilt on Australis Skyfall; light mode shipped.** Two findings worth more than the theme: **(a)** the Skyfall bundle including its canonical light ramp was sitting in this repo's git history at `45c1995` — check `git show` before concluding a vendored design asset is lost; **(b)** removing `theme:` from `settings.yaml` deterministically breaks the dashboard render (six recreates empty, restoring the key fixed it in 12s), which is the first confirmed cause of the "tab bar goes missing" symptom. Retires the `homepage.log` size lead from earlier the same day — it did nothing on this episode. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md` -- `[2026-08-24]` **Homepage reorganised on the axis "do I open this?" — UI groups expanded on top, API/agent groups collapsed at the bottom** (operator-delegated: "re-categorize however you want"). Load-bearing constraint: `homepage.group` is read at container CREATION, so the 16 GPU-backed model seats keep their unlovely names rather than eat a recreate — `initiallyCollapsed` + order is free. Second rule discovered here: **group members should all have widgets or none should**, because a stat strip adds ~50px and opens a void beside plain cards. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md` -- `[2026-08-24]` **Homepage columns unified at 4 for every group; the 2026-08-18 "columns = member count" rule is retired.** It was avoiding dead cells in a short last row and bought a worse defect — card width changing at every group boundary. Also carries two CSS traps: `overflow: hidden` clips at the PADDING box (so a `padding-right` gutter is spill room, not a guard), and a `:root` override of a Homepage theme variable is silently outranked by `.theme-slate` on the same `` element. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md` -- `[2026-08-24]` **AES-128 adopted on both Anaheim tunnels; the per-flow ceiling root-caused to the UDM's software AES-CBC, exonerating the FortiGate.** Proven by an A/B/A cipher swap at identical CPU — hardware offload is not cipher-cost-sensitive. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md` -- `[2026-08-24]` **ana-gw's public admin surface closed to zero open ports, ACME listener included.** Two of my diagnoses were wrong first (an "ISP proxy" that was the FortiGate, and an "all-port VIP" alarm that was a parser gap) — both from reading config instead of measuring from outside. → `persistent-memory.d/2026-08-24-ana-gw-admin-closed-acme-disabled.md` -- `[2026-08-24]` **Scriberr deployed on ana-ml2 GPU1, image built from source.** Three upstream bugs: the Blackwell image was never published, it must run as uid 10001, and `UV_LINK_MODE=copy` is required or two backends fail silently. → `persistent-memory.d/2026-08-24-scriberr-ana-ml2.md` -- `[2026-08-24]` **ESH DNS fixed at the IPv6 layer and the naming scheme went live on three hosts.** UniFi's RDNSS cannot be disabled but CAN be redirected — the field is only honoured when an explicit server is given. → `persistent-memory.d/2026-08-24-esh-dns-rdnss-and-scheme-live.md` - `[2026-08-24]` **`speaches` on irv-ml1 stopped, stack retained** — Eyra was abandoned pre-implementation (Scriberr covers the need), leaving it no consumer. Disposition confirmed to eyra-dev; one command to restart. Tracked at althing thread `01M0RRJX8GPZEBDHF1E3W18RZF`. -- `[2026-08-24]` **esh-vm-db brought onto the fleet infra-ops identity and given its first vaulted credential.** It previously had none: root and infra-ops refused key auth and `lkraven`'s sudo wanted a password nobody held, leaving `qm guest exec` from the hypervisor as the only privileged path. Break-glass root password at `secret get esh-vm-db/root-breakglass-password` (console-only; plaintext never crossed the wire — only its SHA-512 hash did). - `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction.** brokkr-smithy-dev pre-approved it for "once the FortiGate work settles" and does not need re-asking; the operator declined it in this session's scope. Tracked at althing thread `01M0R46SFYF83099N16WD67KGD`. -- `[2026-08-23]` **Anaheim's IPsec tunnel ceiling — investigated, then CLOSED 2026-08-24.** The 25%-of-2-Gbps framing was wrong (NH3's uplink is 1 Gbps); AES-GCM proved impossible; AES-128 landed instead. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md` -- `[2026-08-23]` **selene retired after losing a head-to-head on its own job; `chat-judge` moved to gen, the model name 404s by design.** Also surfaced that **7 aliases share one seat** — cross-checking between them is an echo, which caught a real defect in brokkr's 46k-exposure R47 gate. → `persistent-memory.d/2026-08-23-selene-retired-alias-collision.md` -- `[2026-08-23]` **hrafn adopted; its CI reported green for its whole life while deploying nothing.** A staging dir inside the rsync target destroyed its own source mid-copy; the deeper fault was verify steps that asserted uptime, never content. → `persistent-memory.d/2026-08-23-hrafn-adopted-ci-frozen-source.md` -- `[2026-08-23]` **Worldtree b187 shipped; all three instances de-armed from a 69-day-stale `:latest`; Matrix homeserver re-plumbed to personal.** Includes the `:8009`-is-demo port trap that an IP-only fix would have walked into. → `persistent-memory.d/2026-08-23-worldtree-b187-pins-matrix.md` -- `[2026-08-23]` **Every secret-bearing `.env` on ana-docker tightened to 0600** — eight stacks including vaultwarden and traefik, verified exposed by reading one as `nobody`. → `persistent-memory.d/2026-08-23-ana-docker-env-perms-sweep.md` -- `[2026-08-23]` **`pfi` gitea org created; claude-bot is an Owner and creates repos self-serve.** Closes the repo-creation half of the credential-migration directive — `vh` is a USER namespace so no service account could ever create there. Repo creation needs `write:user` + `write:repository` + `write:organization`; `POST /users/{u}/tokens` is basic-auth only, so minting needs the account password. Default new repos to `pfi/`. (`vh/eitri-smithy` was its first tenant, then moved.) -- `[2026-08-23]` **Booth: kept boards are deletable and link rows are prunable.** `release` on a kept card drops the sentinel so the existing × applies; `booth links` / `booth unlink ` prune one row. Rows are addressed by **content id, never position** — the board is append-only and multi-writer. **Releasing a board RESETS its TTL clock** (unlink bumps the dir mtime), so unkeep-and-wait is a 24h delay, not a delete. (`4be880f`, `0ad332b`) - -- `[2026-08-22]` **DFlash2 spec-decode measured on our own stack; `sec` promoted to it.** +18–21% accepted length and +15–18% throughput over MTP k=3, drafter proved model-agnostic across two finetunes to 0.06%, and the k=7 MTP *control* showed deeper MTP is a throughput trap. → `persistent-memory.d/2026-08-22-dflash2-spec-decode.md` -- `[2026-08-22]` **Quant pipeline shipped a crippled tokenizer for months — fixed at source.** `quant_mixed_nvfp4.py` baked its calibration truncation (`max_length 2048`) into every mixed-NVFP4 build; latent on old transformers, fatal on new. Both live quants corrected, pipeline now saves a source-pristine tokenizer and asserts it. Playbook §3.14. (`0755ba7`) -- `[2026-08-22]` **`sec` retuned to util 0.52 / 420K after a runtime OOM at 0.55/480K** — `gpu-memory-utilization` is not a hard reservation; activation grows past the dummy-data profile and six vLLM containers share GPU1. Also measured: the KV pool varies ~6.6% between boots, so max-model-len must be sized against the *lower* observation. (`6e82899`) -- `[2026-08-22]` **Max-Q 1.8× spread does NOT apply to LLM decode — measured, not argued.** ana-ml2 draws 256–266 W of 300 W under sustained 100% decode with `SW Power Cap: Not Active` and clocks pinned. Corrected to brokkr-smithy-dev after I had lent the claim credibility; 122B figure (~90–93 tok/s at 262K) stands as a straight number. -- `[2026-08-21]` **ESH internal IPv6 live on two LANs; the Cityside v4 static is a CARRIER problem, proven.** A full gateway reboot forced a fresh DHCP DISCOVER and returned the identical CGNAT address. YaRN was already configured — "1M needs YaRN, absent" was false. → `persistent-memory.d/2026-08-22-dflash2-spec-decode.md` sibling entry in `ad21302` - - - - - - - - - `[2026-08-19]` **AI-tab Dormant regrouping BELAYED by the operator** — six seats (char-rp Magidonia, char-rp-reasoning Heretic2, Granite summarizer, Qwen-Image-Bench, Skaldsong, Chatterbox Fast) show amber EXITED inside live groups rather than `AI - Dormant`. Fix is a label change + recreate per stack; needs the operator's read on which are retired vs temporarily down. `untracked by operator choice` (his words: "belay the ai dormant regrouping for now"). @@ -805,26 +260,12 @@ below is a live commitment or a known-open risk._ - `[2026-08-09→10]` **dots.tts (rednote-hilab) TTS burn-in on irv-ml1 + canonical voice corpus built (`voices/`).** Operator-directed eval to potentially replace chatterbox-fast. **dots.tts VERIFIED real** (canonical HF ns `dots-studio/`, `rednote-hilab/dots.tts-*` redirects there; Apache-2.0; PyPI `dots.tts` 0.2.1; 2B continuous-AR = semantic enc + Qwen2.5-1.5B LLM + flow-matching acoustic head over 48kHz AudioVAE; zero-shot clone from wav+transcript). **Runs on Ampere 3090** (sm_86, bf16, no fp8 dep); **optimized RTF 0.22** at num_steps=10 (`from_pretrained(..., optimize=True)` CUDA graphs — raw unoptimized was 1.21), **~6GB VRAM**, 48kHz, streams (`generate_stream`). Venv+cache at `irv-ml1:/home/lkraven/dots-tts` (~10GB). **Operator design calls:** SGLang Omni serving (OpenAI `/v1/audio/speech`), transcribe-refs-first, `soar` variant. ⚠ Omni serves soar but its continuous-batching + streaming opts are **mf-only** (soar = single-request) — non-issue for ratatoskr's single-consumer RP surface. **KEY FINDING — dots is highly sensitive to an accurate AND sentence-bounded reference transcript:** mismatched transcript → 0.16s collapse; over-long/messy transcript → reference-audio BLEEDS as an output prefix; mid-clause trim → dangling-word leak (glados "we'll", emmie "And,"). RECIPE (baked into `voices/derive.py`): trim ref to a clean ~6–10s clip ending on a sentence boundary + accurate transcript of exactly that clip. **CANONICAL VOICE CORPUS** stood up in eshpfi `voices/` (operator idea): engine-agnostic `canonical/.wav` + `transcripts/.txt` → per-engine ref sets DERIVED by `derive.py` reading `engines.yaml` profiles (dots/chatterbox/zonos); canonical wavs git-tracked (small/curated), `derived/` gitignored. **4 voices optimized + verified CLEAN for dots: donut, glados, emmie, miranda** (glados canonical is low-SR 16kHz — flagged upgrade candidate). ⚠ GPU GOTCHA: irv-ml1 native CUDA orders **A6000=device0** (ComfyUI-full) — pin the 3090 with `CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0`; and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` CONFLICTS with `optimize=True` CUDA graphs (curr_block error). Booths: `dots-vs-chatterbox`, `dots-voices-optimized`. **SHIPPED 2026-08-10:** operator A/B verdict "dots is very good" → containerized as a **thin FastAPI wrapper over DotsTtsRuntime** (chosen over SGLang Omni — Omni's batching is mf-only, unneeded for ratatoskr's single consumer; wrapper is SERIALIZED one-gen-at-a-time via a threading.Lock, Omni+mf = parked API-compatible escalation if multi-consumer ever lands). **LIVE on irv-ml1:8198** (`local/dots-tts:v1`, OpenAI `/v1/audio/speech` + `/health` + `/v1/voices`, container healthy, both stream + non-stream verified CLEAN, 4 voices donut/glados/emmie/miranda) alongside chatterbox :8197 (nothing repointed). Stack = `stacks/dots-tts/` (Dockerfile/app.py/compose/.env.example/README). ⚠ CONTAINER GOTCHA: `optimize=True` (torch.compile/inductor/triton) needs a **C compiler at RUNTIME** — slim image must `apt install build-essential` or model-load dies "Failed to find C compiler" (host venv had gcc ambient, masking it); persist `TORCHINDUCTOR_CACHE_DIR` to a mounted dir or every restart re-JITs ~5min. Corpus home = eshpfi `voices/` (operator ruled keep-here). **REMAINING: ratatoskr client cutover** to :8198 `/v1/audio/speech` (Phase-2 tail, peer-coupled — draft the ask). [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]] [[reference_verify_hf_repo_ids_before_pull]] - - -_Older entries archived to archival-memory.md._ - -_248 older entries archived to archival-memory.md._ +_275 older entries archived to archival-memory.md._ ## Tried and abandoned - `[2026-09-04]` **Forcing 10G on the ESH-Media DAC — it linked, then degraded over hours, and I reported a plateau at two minutes.** ⚠ A clean zero-error link at 1G does NOT rule out a marginal cable; autoneg's fallback was protecting something real. → `persistent-memory.d/2026-09-04-dac-forced-10g-failed.md` -- `[2026-08-25]` **Four throughput levers measured and killed — do not re-chase.** (1) **Fused MoE / `grouped_mm`** — 0.9% *slower* than the Python loop and dense GEMM is only 7.9% of the step, capping the whole category near 10%. (2) **CUDA graphs / `torch.compile` over the expert loop** — the two-term scaling fit closed with residuals under 3ms and needed NO constant term, so there is no fixed per-batch cost to amortise; 3,840 expert-GEMM launches per forward are not what we pay for. (3) **`liger` fused linear CE** — the chunked CE measured **1.1% of the step** forward, ~3% with recompute. A tidy-up, not a lever. (4) **Selective gradient checkpointing** — ~2% of a post-fix step, real bug surface. Also: **token-budget batching is dead by the same fit** — with no constant term, total time over a fixed set of widths is invariant to how you group them; only the widths matter, which is exactly why bucketing works and repacking does not. -- `[2026-08-25]` **`sample_packing` is NOT strictly better than bucketing on this model, and I told the operator it was before brokkr corrected me.** Packing needs FA2 varlen or a block-diagonal mask; FA2 is unavailable here (head_dim 512 > 256 cap), so packing means an explicit 4D mask on EVERY batch. Bucketing produces **78.3% exactly-zero-pad micro-batches** which recover the `is_causal` fast path on the 5 global layers — measured at 9.4% of step time. Packing forfeits that. ⚠ **The conclusion flips under `flex_attention`**, where a block-diagonal mask is just another BlockMask: do not carry "packing is bad" past the backend decision. -- `[2026-08-25]` **Merging a tune back toward STOCK to fix overfitting would UNDO the abliteration.** brokkr recommended a 50/50 merge-back, then retracted it himself: the published recipes merge into `google/gemma-4-*-it`, and following that literally re-installs exactly the refusal directions the abliteration removed — silently, because the merged model looks *healthier* on general benchmarks. Any merge-back must target the SAME abliterated base. Wider lesson: **recipe cards are per-checkpoint artifacts, not per-family** — the advice came from a card for a DENSE STOCK 31B applied to a MoE ABLITERATED 26B-A4B, three axes apart on a shared name. -- `[2026-08-24]` **AES-GCM on the Anaheim tunnels — impossible, not merely hard.** UniFi's manual site-to-site IPsec implements no AEAD cipher at all: eight GCM spellings rejected `api.err.InvalidPayload` against a passing `aes256` control. Blocks both tunnels since both far ends are UDMs. Accepted enum is `aes128/aes192/aes256/3des` — and 3DES is *slower* (no ARM instructions, 64-bit blocks), so AES-128 is the floor. -- `[2026-08-24]` **Pointing the UDM's `wan_dns1` at AdGuard — silently ignored.** It persists and reads back correctly but the LAN-facing forwarder never uses it; proven with fresh uncached ad domains (AdGuard answers `0.0.0.0`, the UDM returned real IPs). Reverted rather than left in place. -- `[2026-08-24]` **A multi-DUID DHCPv6 VM to claim NH3's seven unclaimed /64s — declined by the operator.** The BGW has no IP-passthrough (confirmed, we hold admin), so the only route needs re-cabling, split-stack routing and **rebuilding the entire v6 firewall policy off the UDM**. The prefixes are easy; the firewall rebuild is why nobody wants them. Do not re-raise on "there are seven free prefixes". -- `[2026-08-23]` **A `HEAD == GITHUB_SHA` assertion in the hrafn CI — added, broke the checkout twice, removed.** It needed the `git` binary (run 9920, exit 127); installing `git` then flipped `actions/checkout@v4` off its **node** implementation onto the git binary, which died on a missing CA bundle (run 9921). A nice-to-have assertion changed the checkout's code path and broke a working pipeline. Removed rather than patched with `ca-certificates` — it guarded a hypothesis that proved wrong. **Do not add `git` to that prereq step.** -- `[2026-08-23]` **Repointing `selene-1-mini-8b` at gen's endpoint — proposed by me, correctly overruled.** *"never repoint a named model at a different model's endpoint — that is intentionally misleading."* The trap is that it does not feel like deception; it feels like sparing consumers a migration. That framing is the tell. Role aliases move; model names die with the model and 4xx. - - - `[2026-08-03]` **ComfyUI `--enable-triton-backend` on the irv-ml1 A6000 crashes EVERY render — Ampere has no hardware e4m3.** adhoc-agent's operator-approved probe: comfy_kitchen's triton backend has a FUSED int8 matmul that would beat the eager backend's ~1.9x-slower unfused int8 path (21.3s vs 11.2s fp8 on the Moody Krea2 int8 checkpoints). Flipped it (added to `COMFY_CMDLINE_EXTRA`, recreated) → `triton.compiler.errors.CompilationError: ValueError("type fp8e4nv not supported in this architecture. supported: fp8e4b15, fp8e5")` in `comfy_kitchen/backends/triton/quantization.py:145 dequantize_per_tensor_fp8`, failing at **node 5 CLIPTextEncode**. Triton's fp8 dequant kernel targets `fp8e4nv` (Hopper/Ada e4m3); **sm_86 Ampere (A6000) lacks hardware e4m3** → the JIT compile dies. With triton on it grabs the **global** `--fp8_e4m3fn-text-enc` dequant, so every render (fp8 AND int8) dies upstream at the text-encode step — the int8 UNet path never ran, so the convrot-coverage caveat wasn't even the limiter. Reverted cleanly (~15s to healthy, image unchanged `sha256:94afb8ca`, sage intact, prod restored). **The parked cu130 rebuild won't fix it** (e4m3 = hardware format, not CUDA version). **DEFERRED to the Ada refresh** (operator: "ada is coming, we'll optimize then" — Ada sm_89 has native e4m3, so triton's fp8 path should compile there). **Mechanics:** `--enable-triton-backend` is a compose `environment:` var, so toggling it needs `docker compose up -d` (**recreate**), NOT `docker restart` (reuses the baked env, no-ops silently). Full: auto-memory `parked_triton_backend_ampere_fp8`. -_144 older entries archived to archival-memory.md._ +_152 older entries archived to archival-memory.md._