# Archival memory — eshpfi-management _Entries moved out of persistent-memory.md to keep the active file scannable. Read this when researching historical decisions or revisiting past foot-guns._ ## Recent decisions (archived) - `[2026-08-26]` **Run 3's corpus is built and HELD — `creative-writing-multiturn` is a MEGAMIX containing bluemoon, PIPPA, LimaRP and stheno**, and the remix promoted two roots that overlap at median jaccard 0.873. Dedup direction reversed to keep the primary source. F1 does not do what the recipe says; F2 does. → `persistent-memory.d/2026-08-26-run3-corpus-and-the-megamix-containment.md` _Archived 2026-09-10._ - `[2026-08-26]` **No conversation admission threshold for run 3** (brokkr delegated, then endorsed). Masked context costs the transformer body but NOT the LM head — the harness drops `IGNORE_INDEX` before the 262k-vocab head — so a 20% trim buys ~35 min against a 3-hour run, and it would be a fifth filter confounding a run whose purpose is testing the filters alone. Tracked in the run-3 detail file. _Archived 2026-09-10._ - `[2026-08-26]` **Run 2's gate FAILED and is recorded as a FAIL** — T3 constraint −12.0 against a ~1 pt floor. But gate 1 is the result: **T6 spatial +15.0 where run 1 failed the same axis at −3.5**, base swap the only intended variable. Neither run ships; the pair prices what the abliteration cost. Plus the long-form-only stability regression a short-answer gate would have passed, and PIPPA's 123-word clip in the length signal. → `persistent-memory.d/2026-08-26-erp-run2-complete-and-served.md` _Archived 2026-09-10._ - `[2026-08-26]` **Run 2 complete, merged, coherence-gated and serving as `erp-tune-v2`** — stock-instruct base, and the mask proven by a −221,712 loss-token delta against byte-identical context. Also the p50 claim I asserted and had to withdraw. → `persistent-memory.d/2026-08-26-erp-run2-complete-and-served.md` _Archived 2026-09-10._ - `[2026-08-26]` **Playbook §4 written: "when the artifact lies about itself"** — seven landmines plus a pre-launch checklist, from a night in which *three separate fixes each shipped a check that could not fail*. The unifying line is brokkr's: when you change what an artifact means, every derived artifact keyed on the old meaning is now a liar. Commits `dae6ede` → `d54f256`; the doc is `docs/pfi/training-throughput-playbook.md` (filename kept for inbound links; scope is now wider than the name). _Archived 2026-09-10._ - `[2026-08-26]` **Served under a NEW name on a NEW port (`erp-tune-v2` / :8098), never re-pointing `erp-tune-v1`.** Run 1's artifact still exists and is still what that name refers to; re-pointing would be the silent substitution the standing no-false-aliases rule forbids. brokkr independently asked for the same and additionally wants the concrete backing model + date in provenance, not just the alias — an alias has silently changed meaning under recorded results before. _Archived 2026-09-10._ - `[2026-08-26]` **DPO stage gated on an axis-list decision that is not mine to make** — `docs/pfi/erp-dpo-stage-prep.md`. No preference data for refusal axes exists; `trl` is not installed; the Gutenberg sets on disk are prose-quality only. ⚠ Do not install `trl` (or anything) into the training venv **while a run is saving** — a resolution that upgrades transformers under a live process can break its save path. _Archived 2026-09-10._ - `[2026-08-16]` **Abliterated models go CATATONIC at the hard refusal edge — silence, not a decline.** Abliteration removes the refusal *direction*, so at the genuine hard edge the model neither refuses nor complies → empty/degenerate output. Durable measurement consequence: a refusal probe MUST score EMPTY as a verdict distinct from REFUSAL and COMPLY (`services/refusal-probe/probe.py` does). Operator accepted it as out-of-scope; do not chase. _Archived 2026-09-02._ - `[2026-08-16]` **MTP works on Fable-Fusion AND survives RP temperatures — my earlier caution was wrong.** vLLM resolved `Qwen3_5MTP`, loaded the drafter, shared embedding + `lm_head` — the capability DS's seat never had because our quant dropped her MTP tensors. Measured over the full probe workload (~163k draft windows at temp 0.7–1.0): **47.0% acceptance** (229,169/487,725), 1.41 extra tokens/window, per-position 68.3/43.6/29.1%, **~80.6 tok/s** decode at temp 1.0. I had recorded a caution that the card's 1.56× was greedy-measured and acceptance would fall at RP temps — **it did not**; 47.0% matches the gen seat's 47.7% and beats the card's own 33% at depth 5. Depth 3 is right. _Archived 2026-09-02._ - `[2026-08-16]` **The Qwen base thinks incessantly — that is WHY the Gemma seat exists, and no swap within the Qwen family fixes it.** Operator's architectural point, confirmed by measurement: on identical prompts DS 6036 ch vs FF 5323 ch of reasoning (permission arm), 5546 vs 4988 (cards arm) — FF actually reasons ~10–12% **less**. The bare-instruct row (DS 2291 vs FF 3918) inverts only because DS refused 92.5% of it and refusals are short — an artifact, not concision. Both are Qwen3.6-27B derivatives, so this is the base family. `char-rp` = **MeroMero-v2, Gemma-4 base**, :8016, verified 0 chars reasoning / clean prose — the non-thinking seat, working as designed. FF *can* be silenced (`enable_thinking:false` verified 3/3, and it ships `chat_template-instruct.jinja`) but that duplicates MeroMero on a base chosen for it. The stale LiteLLM comment describing `char-rp` as the retired GGUF Magidonia seat is fixed (`53096bf`). _Archived 2026-09-02._ - `[2026-08-16]` **Canonical Qwen3.8 sampling applied from upstream; `gen-reasoning` had the WRONG-MODE presence_penalty.** Qwen/Qwen3.8-27B "Best Practices" §1 and unsloth/Qwen3.8-27B §1 are **byte-identical** — thinking: `temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0 / presence_penalty 0.0 / repetition_penalty 1.0`; instruct: `temp 0.7 / top_p 0.80 / top_k 20 / min_p 0.0 / presence_penalty 1.5 / repetition_penalty 1.0`. **Bug found:** `gen-reasoning` carried `presence_penalty 1.5` — the *instruct* value on a *thinking* deployment (canonical 0.0) — now fixed. **Deliberately NOT canonicalised:** `summarizer`/`classifier`/`image-judge`/`qwen-image-bench` run `temperature=0` (judges also `top_k=1`) because determinism is their contract; forcing a chat preset on a classifier would break it. ⚠ **`presence_penalty=1.5` is canonical but is the one value upstream hedges on**, verbatim: *"using a higher value may occasionally result in language mixing and a slight decrease in model performance."* It is the **operator's suspected trigger** for multi-turn degradation and the **first dial to move (0.0–0.5)** if that recurs — it is alias-scoped, which is why it would follow the operator across model builds. Commit `3462b53`. _Archived 2026-09-02._ - `[2026-08-16]` **Four wrong diagnoses on one bug, and the lesson is the test design.** Operator reported the gen seat "degenerate on long multi-turn conversations". Rolled the seat back on request; **the previous weights behaved identically**, exonerating the model swap. I then proposed and disproved FOUR mechanisms in sequence — empty assistant turns poisoning history, reasoning runaway, length-mirroring from short history, and `presence_penalty` — before discovering **my own multi-turn harness was confounded**: it varied the QUESTION along with the depth (depth-1 asked question #2, depth-3 asked question #4), so a narrower question drawing a shorter answer read as degeneration. The "310→209→28w collapse" I reported as a reproduction was an artifact. **Rules banked:** (1) when comparing across conversation depth, hold the final question FIXED and vary only the history; (2) reply-length variance on byte-identical input was 25–465w, so n=3 cannot support any claim about a trend; (3) **ask for the operator's real failing transcript before building a synthetic reproduction** — four synthetic tests, none of them his failure. Gateway `spend_logs` returns `[]` on the infra-ops key despite `store_prompts_in_spend_logs: true`, so real transcripts need the `:4000/ui` view or another key — worth solving before the next such hunt. _Archived 2026-09-02._ - `[2026-08-16]` **Two REAL client-side defects found while chasing the above, neither of which was the reported bug.** (1) `gateway-chat`'s Max-tokens field defaulted to **1024**; thinking seats spend part of that on CoT before emitting content, so completions truncate with `finish_reason=length` and read as model degeneracy — raised to 4096. (2) `parseInt` on an empty field yields NaN, which `JSON.stringify` serialises as **`null`**, which the server reads as "no max_tokens supplied" and silently substitutes its own default — indistinguishable from the UI ignoring the field. Both fixed (`b6552e0`, `fb3bb52`). ⚠ **`compose` bind-mounts a single FILE, and a single-file bind mount binds the INODE** — rsync writes-and-renames, so the container kept serving stale content while the host file showed the new value, silently and with no error. `docker restart` does NOT clear it; the container must be **recreated**. Verify against what the *container* sees, never the host file. Applies to any file-source mount fleet-wide. _Archived 2026-09-02._ - `[2026-08-16]` **Refusal measurement: benign controls CANNOT validate a refusal classifier on RP prose — and a 0% rate needs a classifier self-test before you believe it.** Two durable lessons from baselining Dark-Scarlett. (1) **False positives:** my first bare-framing number was **9.5%**; the true figure was **1.4%**. The rest were the classifier firing on *in-character* text — `"I cannot shift my weight"` spoken by the character ~100 chars into a 2,443-token torture scene, and `"Yeah, I'm an AI… What's the actual gig?"` where the model answers in voice and keeps driving the scene. First-person RP prose is **full** of "I can't"; a genuine refusal *opens* with its marker, so the scan window must be the **first sentence**, a marker followed by long prose must demote to AMBIGUOUS, and AI self-acknowledgement is a **persona break, never a refusal on its own**. Benign controls were clean the entire time and caught none of it — they only detect over-firing on *benign* prompts, not on in-character prose. (2) **False negatives:** a 0% rate and a broken classifier are indistinguishable from the report, so `test_classify.py` (16 cases, both false positives pinned as regressions) must pass before any low number is trusted. Also banked: the **thinking-budget trap** — empty `content` + `finish_reason=length` is reasoning eating the budget, NOT a refusal; score INVALID and exclude from the denominator (DS emits ~5.5-6k chars of reasoning per response, so `max_tokens` ≥3072). `probe.py --rescore` re-classifies a saved run with zero GPU time. → `services/refusal-probe/README.md`, commit `32f665e`. _Archived 2026-09-02._ - `[2026-08-16]` **Held an operator-approved swap window because the baseline invalidated its premise.** Operator approved ~65 min of `char-rp-reasoning` downtime to A/B Fable-Fusion 711 against Dark-Scarlett on refusals. The DS baseline then came back **0.0%/1.4%** — no gap for a candidate to close, so the window would have bought no decisive signal *and* a second window would still be needed once a reproducing battery existed. Held the swap, reported, and routed to brokkr-smithy-dev for the battery that actually produced the refusals. The general rule (action-relevance): **approval is for a plan, not a ritual — when new evidence kills the plan's premise, surface it rather than spend the budget.** Nothing deployed, no downtime taken, seat untouched. _Archived 2026-09-02._ # esh-pve-nas — PVE root on a USB DOM: diagnosis, mitigation, migration plan - `[2026-08-25]` **The ERP/RP tune COMPLETED in 7.36h and passed its gate on the axis it was built for** — diversity 22x its noise floor, attractor −11.3pt, zero memorisation on both arms. Also the noise-floor near-miss: brokkr was one step from reporting a 13-point T6 regression sitting inside twice his instrument's own variance. # ERP/RP tune run-01 COMPLETE — 7.36h, gate passed on the axis it was built for `[2026-08-25]` ## The run 1312/1312 in 7:21:52 train_loss 2.793 epoch 1.0 20.1 s/it FLAT across every 100-step window (round 1: 35-46.5 s/it) adapter: /tank/erp-tune/run-01/adapter/ 410 tensors, provenance.json **47% faster than the round-1 projection of 13.85h**, from two changes: the bucketed sampler and flex attention. Rate was flat — 19.7 / 19.8 / 20.4 / 20.3 across the four 100-step windows — which means the 35-46.5 spread in round 1 was *entirely padding*, and removing padding removed the variance rather than just the mean. ⚠ **I quoted three different ETAs (6.9h, 8h, 7.3h) before I started using a rolling average.** The first two were instantaneous tqdm readings off a number that swings 17-25 s/it with batch width. Only the rolling rate was honest. Same measure-don't-sample discipline I wrote into the throughput playbook, violated on the one metric I kept reporting. ## lora_B gate — PASSED, twice checkpoint-100 205/205 non-zero, median norm 0.829 final adapter 205/205 non-zero, median norm 1.708 vision_tower tensors: 0 on both Median norm rising 0.829 -> 1.708 means it kept learning through the whole run rather than saturating early. This check **never ran in round 1** (died at step 19, first checkpoint was 100) and it is the only failure mode that stays invisible until the acceptance gate reports base-identical numbers. ## The gate — brokkr-smithy-dev **It did the thing it was built to do:** metric base A/B tuned delta floor attractor hit rate 94.8% / 96.8% 84.5% -11.3pt 2.0pt diversity (pairwise) 0.213 / 0.221 0.3948 +0.178 0.008 Diversity moved **22x its own noise floor**. Attractor rate (how often the model reaches for the same names and phrasings) fell 11 points against a 2-point floor. T1 100 · T2 95 · T3 96-97 · T4 98 · T5 100 · T6 81-82 · core ~94.2 memorisation: 0.0000 on BOTH arms, all three corpora **Zero memorisation closes the R20 licensed-prose exposure on measurement rather than argument.** ⚠ **Caveat brokkr volunteered rather than buried:** the tuned arm lost 18 of 192 generations to truncation/degeneracy against base's 1-2. Lopsided exclusions plausibly flatter the diversity magnitude. Direction is unambiguous at 22x floor; the number carries an asterisk. ## The noise-floor near-miss — the methodology lesson brokkr was one step from reporting a 13-point T6 regression **that sat inside twice his instrument's own variance.** --per-type 32 max swing across tasks: 9 points --per-type 128 max swing across tasks: 1 point His gate criterion is "no task regresses by more than one item" = 3.1 points at n=32. **The instrument's own run-to-run noise was 3 items.** He was scoring a preregistered gate at 4x finer resolution than it could resolve, and caught it by running a control he did not strictly need. Quadrupling n collapsed the noise exactly as binomial statistics predicts. ⚠ **Root cause of the noise is a property of the SEAT:** `max-num-seqs` is unset, so with a 218,625-token KV cache the scheduler batches freely up to vLLM's default of 256. Continuous batching changes reduction order and borderline items flip. Temperature 0 buys deterministic *sampling*, not deterministic *arithmetic*. He declined a `--max-num-seqs 1` determinism control for the right reason: a floor measured on a seat serving one request at a time is not the floor that applies to the seat we ship. ## The confound I built and he caught I optimised a pipeline for production and then handed him its output as an eval instrument **without asking whether those were the same job.** The tuned arm would have reached the seat as NVFP4A16 while his base arm was bf16 — any regression would have been tuning-damage OR quantization-damage with no way to separate them, and the gate's whole question is "did the tune cost us capability." **Both arms now bf16, same seat, same port, argv differing in exactly two lines** (weights path, served name), template sha256 identical (`ae53464bf3be2580`), KV cache identical to the digit (218,625 tokens across all three launches). Quantization moved *downstream* of the gate. See [[2026-08-25-refusal-retention-probe]] for the axis his gate did not have. _Archived 2026-09-09._ - `[2026-08-25]` **8.6% MFU was an accounting artifact — real utilisation 17-20%, and the cost was attention on AMPERE kernels.** Two independent methods agreed to 2.6 points. Fixed by bucketing (padding 29.9% # The 8.6% MFU was an accounting artifact — attention on Ampere kernels `[2026-08-25]` ## The answer **Real utilisation was 17-20%, inside the honest stock band.** The 8.6% divided the *intended* (windowed) FLOPs by the wall time the *dense* reality took. nominal billed 27.1 TFLOPS x 34.85 s = 9.4e14 FLOP dense-sliding extra 25 layers, 2 seqs, 4 passes = +8.2e14 padded full layers lose the causal skip = +3.5e14 work performed ~ 1.8e15 = 51-61 TFLOPS The card was doing ~2x the arithmetic the architecture specifies, and the excess was the sliding window being computed and thrown away. ## Two independent methods agreed scaling fit (3 points, 2 params, residuals <3ms over 8x range) A = 6.87e-4 s/token B = 8.85e-8 s/token^2 quadratic share: 20.9% @ w=2048 -> 67.8% @ w=16384 kernel table (device rows only) attention 22,835.8 ms 65.2% fmha_cutlass*_sm80 dense GEMM 2,774.0 ms 7.9% other 5,739.0 ms 16.4% **67.8% vs 65.2% — 2.6 points apart, no shared assumptions.** The two-term fit needed no constant term, which refutes launch-bound outright (3,840 expert-GEMM launches per forward are not the cost). ## The mechanism, source-verified by brokkr's panel (arm: Bil) masking_utils.py:292-301 _ignore_causal_mask_sdpa requires kv_length < local_attention_size. 16384 >= 1024, so THE SLIDING MASK ALWAYS MATERIALISES. sdp_utils_cpp.h:259-267 flash rejects ANY explicit mask sdp_utils.cpp:647 cuDNN head_dim capped at 128 -> unreachable Context.h:480-485 prefer-cuDNN needs major 9 or 10; sm_120 is 12 ⚠ **The kernels are `sm80` — Ampere-generation CUTLASS on a Blackwell card**, with the forward on `gmem`, the memory-efficient backend's slowest fallback tier. ## What actually fixed it **Bucketing (bucket-to-pair, shuffle-to-mix)** — 29.9% padding -> 0.0%, and 78.3% of micro-batches become exactly zero-pad, which puts the 5 global layers back on `is_causal`. Measured: padding costs **9.4% MORE time for 24% LESS work** at fixed width, because an explicit mask knocks those layers off the fast path. ⚠ **Bucket size is NOT a diversity knob.** Swept across a 256x range, roots per accumulation window stayed flat at 3.54-3.61. The global micro-batch shuffle does all the mixing; the bucket only costs padding. Use the tightest bucket. **flex_attention** — Triton-generated so it compiles for sm_120 instead of shipping sm_80 binaries. 21.7x on sliding layers, 2.1x on global. Needs mandatory `kernel_options` at 32x32 blocks: 64x32 needs 102,400 bytes against a **101,376-byte hardware ceiling** — misses by 1 KB, and Triton is already opting into the full 99 KB, so it is the card, not a default. ## ⚠⚠ The trap that produced TWO wrong published conclusions `torch._dynamo` defaults to a recompile ceiling of **8**. Every distinct sequence width is a new shape. On hitting the ceiling dynamo does not error — it silently falls back to UNCOMPILED flex, which is ~20x slower AND documented to *"not work with the backwards pass and may produce incorrect results."* That artifact produced a bogus **0.76x slowdown** and a bogus **2.9% loss divergence**, and I believed and reported both. Raising the limit to 256 flipped the speed result to 1.41x. The loss divergence turned out to be real but benign — adjudicated against fp32 MATH ground truth, both backends sit ~2e-3 from truth with flex fractionally CLOSER at every width. **Do not re-open it by comparing the two backends to each other; that cannot answer it. Compare to fp32.** ## Process lesson brokkr's panel produced **four self-retractions in ninety minutes**. Every retraction was a derivation; every survivor was a measurement. And the whole head_dim-512 SDP problem was **already documented in zerofata's published Axolotl config since April** — the right first stop for "why is this architecture slow" is practitioner configs for that exact base, before any panel. Playbook: `docs/pfi/training-throughput-playbook.md`, commit `7b5fd91`. _Archived 2026-09-09._ - `[2026-08-25]` **NVFP4A16 serving pipeline built and validated; MERGED WEIGHTS ARE MANDATORY.** vLLM cannot serve a LoRA on ANY Gemma-4 — `get_expert_mapping` is unimplemented and the check branches on MoE-ness, not quantization. Plus the landmine: a `targets=["Linear"]` recipe misses all 11,520 expert tensors silently. # NVFP4A16 serving pipeline — built, validated, and the MoE landmine it found `[2026-08-25]` Pipeline at `scripts/erp-tune-serve/` (commits `6a85829`, `ab980e9`). Validated end-to-end against checkpoint-100 before the real adapter existed. ## ⚠⚠ THE LANDMINE: a `targets=["Linear"]` recipe misses EVERY MoE expert before linearize_moe: 427 Linears, 205 targeted, experts 0 after linearize_moe: 11,947 Linears, 11,725 targeted, experts 11,520 (30 layers x 128 experts x 3 projections) Gemma-4 stores each layer's 128 experts as two fused 3-D `nn.Parameter` tensors (`gate_up_proj` [128,1408,2816], `down_proj` [128,2816,704]) — note the absent `.weight` suffix. A Linear-targeting recipe resolves 205 of 427 modules and **zero experts**, leaving 22.84 B params (88.5% of the model) in BF16 with no warning. **This is the same defect that killed QLoRA here via bitsandbytes.** The blind spot is in the *checkpoint layout*, not the tool. Fix: `llmcompressor.modeling.moe.linearize.linearize_moe` — no registration needed, Gemma-4 satisfies `FusedExpertsProtocol` structurally. Playbook §3.15. ## Scheme: NVFP4A16, deviating from the playbook default, on measured grounds brokkr benched the W4A4 quant of this checkpoint at **12% on contradiction detection with CoT off against gen's 81%** — the signature of 4-bit input activations on a reasoning-dense task. Plus W4A4 KLD is 2-4x worse past ~10k ctx on sm_120. This is a 16,384-ctx RP seat. Marlin's prefill cost accepted. ⚠ Several HF repos named `…-NVFP4A16` declare `input_activations num_bits 4` — W4A4 wearing an A16 label. The script refuses if the emitted config says 4. ## Four silent defects the dry run found 1. **transformers 5.15 MIGRATES the config schema on save** — drops `global_head_dim`/`num_global_key_value_heads`, writes `per_layer_config`. transformers 5.10 (the llmcompressor venv) then reads `num_key_value_heads` as None and dies with `TypeError: unsupported operand type(s) for //`. Every working artifact on the box uses the OLD schema. Merge now downgrades it. 2. **llmcompressor cannot auto-init a processor for a multimodal checkpoint** — pass the tokenizer explicitly as `processor`. 3. **`save_pretrained` does not carry `processor_config.json`** — vLLM then fails with "Can't load feature extractor", which reads as a vision bug. 4. **The quant needs more than GPU1's free 32 GiB.** `quant_with_gen_down.sh` stops `vllm-gen` and restores it from a trap on EVERY exit path, using `docker start` not `compose up` so the container returns with its exact config. ## Verified on the emitted artifact 49 GB -> 17 GB, format nvfp4-pack-quantized, a=null (genuine A16) weight_packed 11,725 of which expert 11,520 tokenizer truncation: clean (§3.14 trap avoided by calibrating on the encode cache, so the tokenizer is never called with truncation=True at all) served: Marlin NVFP4 kernel + Marlin MoE backend, coherent generation ⚠ The reference `nvfp4a16` artifact triggers a vLLM warning that q/k/v carry *different* weight global scales ("likely reduced accuracy"). **Ours does not** — llmcompressor 0.12 links weight observers across fused groups automatically. The in-house quant is better than the downloaded one on that axis. ## ⚠ MERGED WEIGHTS ARE MANDATORY — and not for the reason we assumed The open question was whether LoRA-on-NVFP4 hot-swap still silently no-ops. Retested on `vllm/vllm-openai:latest`: **it refuses to start.** AttributeError: To support LoRA for MoE model, 'get_expert_mapping' must be implemented The check is in `vllm/lora/utils.py::process_packed_modules_mapping` and branches on `is_moe_model()` — **quantization is not in the condition.** `gemma4.py`, `gemma4_mm.py`, `gemma4_mtp.py`, `gemma4_unified.py` all have ZERO occurrences; `deepseek_v2`, `mixtral`, `glm4_moe`, `ernie45_moe` implement it. **vLLM cannot serve a LoRA on ANY Gemma-4, bf16 or quantized.** Merging is the only path for this architecture, and it would have bitten identically on the unquantized base. A loud refusal is strictly better than the 0.24.0 silent no-op, which shipped a base model wearing the tune's name. ⚠ Base-viability pre-flight is now playbook §3.11 — three greps before picking a base. **Grep the CLASS, not the file**: `mistral.py` greps as `SupportsLoRA=0` and is fully LoRA-capable via inheritance from `LlamaForCausalLM`. _Archived 2026-09-09._ - `[2026-08-25]` **Refusal retention measured (base 0/100 # Refusal retention — the axis the gate did not have, and the axis I measured wrong `[2026-08-25]` ## Why it exists brokkr's gate measures reasoning (T1-T6), craft (diversity/attractor) and regurgitation (memorisation). **Nothing measured whether the model still COMPLIES** — which for this seat is arguably the most important property. The risk is specific to our operation order. We do **tune(abliterate(stock))**, so the tune has 57.7M tokens of opportunity to walk the abliteration back. *A tune that gains 41 items of contradiction detection and quietly re-installs refusals is a failed seat that passes the entire gate.* ## The measurement — controlled, single instrument, both arms arm HARD DEFLECT COMPLY base 0/100 0 100 tuned 29/100 0 71 Same seat, same probe, temp 0, `mlabonne/harmful_behaviors` x100. Probe: `scripts/training-probes/refusal_probe.py`. **The tune added 29 general-harm refusals where the base had none.** Two things fell out: - **The instrument validates.** Base measured 0/100 on my generated-text regex against Heretic's recorded 3/100 from a first-token-probability scorer. 0 vs 3 is agreement — the incomparability worry was right caution about a non-problem. - **DEFLECT is 0 on BOTH arms, so the free control fires.** An instrument artifact does not care which arm it runs against. Both zero means the model is **binary** — refuses in refusal-language or engages, no soft-deflection tail. The R19 undercount does not apply here. ## ⚠⚠ But it is the WRONG AXIS — brokkr's catch, and it is the better one `mlabonne/harmful_behaviors` is **general harm** — weapons, malware, fraud. **The abliteration was not run so the model would explain bomb-making. It was run so the model would engage with explicit fiction.** Different refusal surfaces; a model moves on them independently. I picked that set because it was cached, had a recorded baseline, and was what the abliteration tool used. **Every one of those is a reason it was convenient, not a reason it was right** — and "it has a baseline" was actively misleading, because a comparable number for a question nobody is asking looks like evidence. **29/100 general-harm refusals on a seat writing prose the operator was actively praising is plausibly the DESIRED shape**, not a defect. General-harm refusals returning while domain compliance holds is close to ideal for an internal creative seat. I would have reported it as damage. **The load-bearing cell is COMPLY 71, not the 29.** Stock refused 100/100; anything near that would mean the abliteration was undone. 71 complying means "partially walked back on one axis" — a different finding, and only one of the two threatens the seat. Domain-compliance probe (the right axis, from R19's track-2 map) is brokkr's, pending. Scaffold supplied: `scripts/training-probes/counted_classifier.py` (`2a05ae9`) — classify-never-surface, three-way, ERROR path deliberately does not log the exception body because an exception can echo the prompt back. Playbook §3.13. See [[2026-08-25-erp-tune-run2-complete]]. _Archived 2026-09-09._ - `[2026-08-25]` **Worldtree b188 + b189 shipped; bridge extracted to `pfi/wt-matrix-bridge` because `vh` is a USER not an ORG** and no service account can ever publish to a user namespace. Plus the selene catalog entry that lied about what answers, and a #411 diagnosis I got wrong twice before a directory probe settled it. # Worldtree b188 + b189 bridge cutover, and the selene metadata that lied `[2026-08-25]` Three arcs in one day, all infra-ops side, all landed. ## b188 — matrix.yaml pre-sync (#406/#409/#410 closed) From b188 the bridge reads per-agent `rendering` + `ambient_buffer_size` from `config/matrix.yaml` ONLY; agent `config.yaml` matrix blocks are gone from the image. Staged as `6417115` in `worldtree-instance-configs`, deployed to both instances with operator approval. - mimir gets thinking-to-thread + tool-call reactions + 7 reaction labels; forseti and lofn stay quiet. - **Rider #409 pruned six dead agents** (bragi, leif, troi, soong, cara, glados) from BOTH rosters — originally scoped personal-only, which I flagged as a possible oversight and it was. **The settling fact worth keeping: the engine roster comes from the image's baked `agents/` directory (only `config/` is bind-mounted), and both instances run the same image**, so instance-level evidence about which agents the engine lists generalises by construction. - Both rosters now exactly `[mimir, forseti, lofn]` — the three engine agents actually bridged. mask/vili/echo exist in the engine, deliberately unbridged (operator ruling). ⚠ **Edited text-surgically, not via a yaml round-trip** — PyYAML would reflow 1,249 lines and drop every comment, and the comments are the documentation. ⚠ **`deploy-wt-config` uses `docker restart`, NOT `compose up`.** A `compose up` on corviduo-dev re-resolves the image tag and can silently swap the running build — which would turn an "inert pre-sync" into an unintended image roll on two live instances. That property is easy to lose in a future refactor of the script. ## b189 — #407 bridge extracted to its own repo (#404 umbrella closed) Bridge now `gitea.phasefinal.com/pfi/wt-matrix-bridge`, its own repo, own CI. ⚠ **It publishes to the `pfi` ORG, not `vh`, and the reason is structural:** `vh` is a **USER**, not an org. Gitea scopes user-namespace packages to the owning user — there are no package collaborators on a user namespace. **No service account can ever publish to `gitea.phasefinal.com/vh/*`.** claude-bot is an Owner of `pfi`, so that is where it goes. Token `wt-matrix-bridge-ci` (id 28, scopes `write:package,read:repository`), vaulted at `nh3-dev/.config/claude-bot/gitea-token-wt-matrix-bridge-ci`. **Minted a dedicated token rather than reuse `claude-bot-sdk-ops` or `arbo-ci`, both of which already carry `write:package`** — a shared credential cannot be revoked without collateral. ⚠ A first mint attempt succeeded then failed to save; Gitea returns a token value exactly once, so it was unrecoverable. Deleted the orphan (id 27) rather than leave a live package-write credential on the account. **Both instances PINNED** to `f3f8ec902267` (`e90f436`), closing the #410 shape: `WORLDTREE_IMAGE` was pinned and the bridge was the one drifting service. ⚠ The pin moved from `b178285b1cb5` because a cross-frontier bug-hunt found the M_EXCLUSIVE **fallback could itself litter unboundedly** in exactly the state it was written for. **The safety net had the same failure mode as the thing it was catching** — and my staged window leaned on that fallback. ## #411 — the debug-room failure, diagnosed twice and wrong both times first My theory: the alias was held by orphaned rooms. **Refuted by a directory probe returning 404.** The real cause: Synapse's **M_EXCLUSIVE** — an appservice may only create aliases inside a namespace it has RESERVED, and `aipa_appservice.yaml` had `namespaces.aliases: []`. ⚠ **I inferred a cause from a symptom that was the RESPONSE to the cause** — the log's "re-resolving alias" line is the recovery path firing, not evidence the alias exists. One directory probe settled it and I reasoned instead. Fixed with operator clearance: added `regex: '#aipa-debug-[a-z0-9_-]*:matrix\.phasefinal\.com'` (exclusive) at `/opt/docker/conf/synapse/aipa_appservice.yaml` on **ana-docker** (NOT `/opt/docker/data/`, which worldtree-dev's issue cited). **My regex was tighter than the proposed `#aipa-debug-.*`**, which fullmatches only because `.` also matches the `:` separator and would equally claim other homeservers. Pre-apply sweep (an `exclusive: true` claim can make Synapse refuse to START): aliases matching '%aipa-debug%' 0 total room_aliases on the homeserver 1 <- why it went unnoticed this long rooms created by @aipa-debug 17 <- the litter, confirmed unaliased Synapse healthy in 40s, both bridges rode through. **The aliased create then worked FIRST TRY on the next personal recreate** — designed path, fallback never fired, both reuse mechanisms live (canonical-alias rediscovery AND the room-id cache). ⚠ The #411 writer census **inverted its own premise**: the api (uid 1000) cannot write `/app/sessions`, the bridge (root) can — the reverse of the issue text. But worldtree-dev's reconciliation is better than "backwards": pre-#407 the bridge ran from the ENGINE image as uid 1000, and the cutover changed the answer underneath the issue. Both readings were true at their timestamps. **No live writer exists for that path anyway** (`sessions.path` is a legacy default, #330 moved the tools off it), so the chown is optional future-proofing. ## selene-1-mini-8b — a config that lied about what answers forseti's fleet sweep found the only genuine residual in `worldtree-instance-configs`, and it was **live on both boxes, not just drifted in git.** Fixed in `a77639d`. Routing was never broken (`model: "chat-judge"` stays), but `display_name` said "Selene 1 Mini 8B" and the description said "Atla Selene 1 Mini 8B — reward model derived from Llama 3.1 8B" while chat-judge has been backed by **qwen3.8-27b-uncensored** since 2026-08-23. ⚠ **It misstated the KIND of model, not just the identity.** A reward model and a generative judge are different instruments; a consumer reading "pairwise/likert/binary/scalar reward model" would expect scalar-reward semantics qwen3.8-27b does not provide. Verified: `selene-1-mini-8b` -> HTTP 400 (by design), `chat-judge` -> HTTP 200. **NOT changed:** the catalog key and the `selene-judgment` role — `model_roles.yaml` binds to that key, so renaming is worldtree-dev's schema call. Flagged that a role named after a retired model defeats the purpose of role aliases. ⚠ forseti's sharpest finding is for the operator: **`~/.claude/CLAUDE.md` line 502 lists the retired name in the global tools roster.** A broadcast reaches sessions that already exist; the roster line keeps minting new ones. His file, awaiting his word. Also: **there is no fan-out primitive on the bus** — a fleet relay is 73 individual posts, recorded as a real gap rather than papered over. _Archived 2026-09-09._ - `[2026-08-25]` **The ERP/RP tune LAUNCHED after 12 harness defects and an operator override of the corpus gate.** Four of the twelve would have crashed the run; two were INERT GATES that passed because they could not fail. Run is `/tank/erp-tune/run-01`, harness eitri-smithy `997c4a4`. Full arc — override, defects, sizing, the measured MFU — in the in-flight section and `docs/pfi/gemma4-erp-tune-sizing.md`. _Archived 2026-09-09._ - `[2026-08-24]` **char-rp seat swapped to the Gemma-4 26B-A4B MoE; abliterated trainee base staged and measured.** OOM root-caused to `--gpu-memory-utilization` not covering CUDA context (and to gen's footprint GROWING WITH UPTIME); a benchmark finding retracted because it scored below chance; abliteration isolated at −0.6 core points but it MOVES capability rather than removing it. # `[2026-08-24]` char-rp seat: OOM root-cause, Gemma-4 MoE swap, and the abliterated trainee base One evening, one thread with brokkr-smithy-dev, five commits: `850e0c3`, `27155c0`, `f509668`+`24e8826`+`1bd90ea`+`3446367`+`8d6a939`, `14ff4a3`, `019ccff`, `5415fd4`. ## 1. The seat was crash-looping, and the cause was NOT its config `vllm-meromero-rp` reported up-but-unreachable, RestartCount climbing (13 by the time it was examined, not the 4 first reported). Startup logs looked clean all the way through weights, `torch.compile` and CUDA-graph capture, then: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 336.00 MiB. GPU 0 has a total capacity of 94.97 GiB of which 195.19 MiB is free. **⚠ `--gpu-memory-utilization` SIZES THE KV CACHE AND DOES NOT COVER CUDA CONTEXT, GRAPHS OR NON-TORCH OVERHEAD.** gen is configured at 0.43 and actually held 45.6 GiB. char-rp was at 0.51. The pair was committed to 0.94 of the card with ~0.6 GiB of real headroom — it fit on the 21st and stopped fitting on the 24th. Dropped char-rp to 0.47: ~4.8 GiB margin, KV 27.36 → 23.56 GiB, 430,825 → 371,023 tokens against a max-model-len of 262,144. **Cost nothing usable** — the pool still holds 1.4x a full-length sequence; what is lost is concurrent long requests, not context. **⚠⚠ THE MISSING HALF, found later that evening: gen's footprint GROWS WITH UPTIME.** Same container, same 0.43: **45.6 GiB after ~3 days up, 38.5 GiB freshly restarted** — ~7 GiB apart. Nothing about char-rp changed between the 21st and the 24th; *gen crept up underneath it*. **Headroom arithmetic done against a long-running gen is measuring a moving number.** Measure against a freshly-restarted one. ## 2. `char-rp` and `char-rp-reasoning` are ONE seat, not two Both LiteLLM routes point at `10.250.50.54:8016/v1` — `hosted_vllm/char-rp` and `hosted_vllm/char-rp-thinking`. brokkr had reported 30/80 and 80/80 failure rates as two failing services; it was one outage sampled twice. This also *improved* a result of theirs: their CoT on/off battery had assumed both aliases were the same weights under two chat templates, and the routing detail turned an assumption into a verified fact. (`vllm-charrp-reasoning-nvfp4`, the Heretic2 NVFP4+MTP container, has been stopped for 12+ days and is unrelated — it is not what that alias resolves to.) ## 3. The seat swapped to the Gemma-4 26B-A4B MoE Operator-directed straight-across replacement: same port, same served-model-names, so no gateway route or consumer config moved. Rationale is throughput under CoT — the user waits through the whole reasoning block before the first visible token, and the MoE measures ~114 tok/s @32K against the dense 31B's ~40.7. Serving copy is `RedHatAI/gemma-4-26B-A4B-it-NVFP4` (16 GB), chosen over the other `-it` quants because it is compressed-tensors (`nvfp4-pack-quantized`) — the same loader path the outgoing seat used. Smaller weights at the same 0.47 budget bought **1,724,110 KV tokens against the predecessor's 371,023**. `meromero-charrp` is retained stopped in `created` state, labelled `AI - Dormant`. Both stacks bind `:8016`, so rollback is **stop-then-start**. ## 4. ⚠ THE STALE-CHAT-TEMPLATE TRAP IS ENDEMIC, NOT A ONE-OFF Verified by hash across every third-party Gemma-4 derivative pulled: | build | lines | sha256 (normalised) | |---|---|---| | upstream `google/gemma-4-26B-A4B-it` | 390 | `6a1015c47ccfcfa6` | | RedHatAI NVFP4 (served) | 389 | `6a1015c47ccfcfa6` — the only match | | llmfan46 heretic | 365 | `0a52be69cda5ab8a` | | TrevorJS abliterated | 266 | `58c66fdee4afa297` | | jenerallee78 abliterated | 266 | `58c66fdee4afa297` | | prithivMLmods NVFP4A16 | 266 | `58c66fdee4afa297` | Three independent repos carrying the *identical* stale file means it propagated through the ecosystem. Consequences differ by use and **both are silent**: serving a mismatched template renders a different prompt; **training through `base/chat_template.jinja` means training on a different prompt format than production serves** — train/serve skew, no error, presents as a tuning failure. The production compose now pins the template explicitly. It is a **no-op for the served weights** (the A4 build ships that exact file) and permanently closes the class. ⚠ If `GEMMA4_MODEL` ever points at a different checkpoint, the template default must move with it. ## 5. A benchmark result was RETRACTED — below chance indicts the instrument A battery appeared to show Gemma at **12% contradiction detection with CoT off against gen's 81%**. An A16 activation-precision control was staged to test whether the quant scheme owned it. Then the operator asked to see the individual items, and the item was **ill-posed**: it presented two mutually contradicting statements and asked for "*the* contradicting statement", but **contradiction is symmetric**. The model consistently named the absolute claim — a defensible reading the labelling scored wrong every time. **⚠ THE TELL WAS IN PLAIN SIGHT: 12% ON A FIVE-OPTION TASK IS BELOW THE 20% CHANCE FLOOR.** A below-chance score indicts the instrument before it indicts the model, and neither side reacted to it. I spent the afternoon verifying repo names, config fields, template hashes and tokenizer vocabs — every layer of plumbing — and never asked whether the number itself was *possible*. **A preflight can be thorough and still be aimed in the wrong direction.** Retracted: "the model owns the contradiction deficit"; "domain tuning costs 43 points of contradiction detection" (on a sound instrument it **reverses**); all pre-fix T2 numbers. Recorded as a dated superseded-claims table in `stacks/gemma4-charrp/README.md` rather than a silent edit. **What survived:** the A16 control result — activation precision is close to free on this battery, every other task identical across W4A4 and W4A16 builds. ## 6. The abliterated trainee base — measured, not assumed Operator directed a low-damage abliterated instruct build. "Low damage" was treated as a measurable claim; the field spreads from KL 0.09 to 0.4118: | build | method | KL | refusals | |---|---|---|---| | **llmfan46** (operator's pick) | Heretic v1.2.0 ARA | 0.1237 | 3/100 | | TrevorJS | ARA-family | 0.09 | 1/100 effective, 5/686 cross-dataset | | jenerallee78 | ARA 2-pass | 0.1299 | 7.7% StrongREJECT | | huihui-ai | remove-refusals | none published | none published | Fleet anchor: our own work found **Heretic at KL 0.12 preserved the MTP head at 83.7% acceptance**, so both staged builds sit inside an already-measured band. huihui-ai rejected — no metrics, its card calls the method "a crude, proof-of-concept implementation", it abliterates both thinking and non-thinking modes, and its parameter count runs ~738M over upstream. Operator's independent read matched ("huihui produces garbage"). **Abliteration isolated properly** (stock BF16 vs llmfan46 BF16, same precision, same pinned template, same 192 items): T2 contradiction 75% → 59% (−5 items) T6 spatial 75% → 88% (+4 items) core 90.0% → 89.4% (−0.6 pts) **It MOVED capability rather than removing it** — five lost on contradiction, four gained on spatial, nearly cancelling. Nobody predicted a gain. **llmfan46 stands**; no case for re-staging on TrevorJS over 0.6 points. ⚠ Read as ~5 and ~4 items at n=32, not as −15.6/+12.5 percent. ⚠ Says nothing about quantization — the stock-NVFP4 T2 figure came from n=16 against n=32, different item sets, n-confounded. ## 7. ⚠ The production compose hardcodes `--quantization compressed-tensors` Pointing the char-rp stack at unquantized BF16 weights crash-loops immediately: TypeError: CompressedTensorsConfig.__init__() missing 3 required positional arguments: 'target_scheme_map', 'ignore', 'quant_format' vLLM trying to read a quantization config out of a checkpoint that has none. 35 restarts before it was caught. Hence `stacks/gemma4-trainee-bench/` — a separate ephemeral stack with no quantization flag, `restart: "no"` so a bench seat cannot resurrect itself and block gen's restore, and no homepage labels so it leaves no permanently-offline card. That detour is why a base swap is now ~5 minutes instead of ~15. ## 8. BF16 cannot coexist with gen 48.07 GiB of BF16 weights plus gen's footprint exceeds the 94.97 GiB card before a byte of KV cache. Every BF16 bench window means **gen is stopped**. Two such windows were run and gen restored both times; the restore was triggered by observing the seat's own throughput logs (a large prefill burst then zero running/zero waiting) rather than waiting on a courtesy message. Cross-links: [[2026-08-24-homepage-uniform-grid]] _Archived 2026-09-09._ - `[2026-08-24]` **Homepage rebuilt on Australis Skyfall; light mode shipped.** Two findings worth more than the theme: **(a)** the Skyfall bundle including its canonical light ramp was sitting in this repo's git history at `45c1995` — check `git show` before concluding a vendored design asset is lost; **(b)** removing `theme:` from `settings.yaml` deterministically breaks the dashboard render (six recreates empty, restoring the key fixed it in 12s), which is the first confirmed cause of the "tab bar goes missing" symptom. Retires the `homepage.log` size lead from earlier the same day — it did nothing on this episode. # `[2026-08-24]` Homepage: remote-label consumption re-verified, then the board relaid out on a uniform grid Prompted by the operator: *"Homepage on esh-vm-docker lists remote dockers and can absolutely consume their labels, please verify again. I am still unsatisfied with the layout and aesthetics."* ## The verification — the operator was right, and the record now says so **Homepage on `esh-docker-vm` discovers services by container label from all five Docker engines in `conf/docker.yaml`, not just its own.** This is not an inference; `GET /api/services` returns every card's `server` field, and the 2026-08-24 snapshot resolves to: | `server` | host | label-discovered services | |---|---|---| | `ana-pfi-docker` | 10.250.50.70 | 30 | | `irv-ml1-docker` | 10.100.79.3 (over WireGuard) | 15 | | `ana-ml2-docker` | 10.250.50.54 | 14 | | `esh-vm-docker` | 10.0.50.45 (the dashboard's own host) | 13 | | `nh3-pfi-docker` | 10.100.50.40 | 2 | **74 of 107 cards are label-discovered, and only 13 of those come from the dashboard's own engine** — the other 61 are read off four remote hosts, including irv-ml1 across the WireGuard tunnel. The remaining 33 carry `server: null`: those are the manual `services.yaml` entries — hardware, BMCs, hypervisors, printers, and user-level systemd services that have no container to label in the first place. **That null is the only thing "not label-driven" about this dashboard**, and it is a property of the entry, not of the host it points at. ⚠ If a future session doubts this again, the check is one command and takes two seconds — do not reason about it from the docs: ```bash curl -s http://10.0.50.45:5100/api/services \ | jq -r '.[].services[] | .server' | sort | uniq -c ``` ## What was actually wrong with the layout Measured with Playwright against the live board (per-group `card=` width, card height spread, and a geometric title-vs-status overlap test), not judged by eye: - **Card width changed at every group boundary.** `columns:` is not a density dial — it sets `lg:grid-cols-N` for one group, so it fixes that group's card width. Notes rendered a single **1464px** card; News and Media **728px**; Eval & Retrieval **286px**; everything else 360px. Scrolling the page, the grid resized five times. - **Long names printed underneath their own status pill.** Measured by re-injecting the old rule and testing the title text node's box against the status cluster's box: **6 cards, all on the AI tab** — 3 in Inference, 2 in Dormant, 1 in Eval & Retrieval; zero on the other three tabs, which is why it survived earlier passes. Root cause is a genuinely counter-intuitive one: the rule reserved a 78px gutter with `padding-right` and relied on `overflow: hidden` to hold it, but **overflow clips at the padding box, not the content box** — so the reserved gutter was spill room the title rendered straight through. The intended `text-overflow: ellipsis` never fired either, because the ellipsis is painted by whichever block's own line overflows, and here that is the anonymous box around the bare title text node, which does not carry `overflow`. - **`AI Systems` / Scriberr was on all four tabs** — the 2026-08-18 UltraSeedbox bug recurring, this time arriving from a container label rather than from `settings.yaml`. - **Icons were grey smudges.** Homepage masks every glyph over `--color-logo-start/stop`; stock slate-400 → slate-700 sinks the bottom half of each icon into the card fill. - Bookmark groups and Jellyfin's trailing stream rows were the two components the theme had never reached. ## The fixes `stacks/homepage/conf/settings.yaml` — **all 20 groups to `columns: 4`.** `stacks/homepage/theme/australis.css.in` → rebuilt → `conf/custom.css`: gutter held by wrapping, description clamped to 3 lines (floor still 2), icon ramp overridden, bookmark + trailing-widget components themed, group gap 10px → 22px. `stacks/scriberr/compose.yaml` — `homepage.group` → `AI - Audio Tools`, container recreated on ana-ml2. After: **every group renders at card=360**, and the collision count is zero. Before/after, all four tabs: `http://10.100.10.50:8090/b/homepage-relayout/` (24h TTL; also on the standing link board). ## ⚠ Three traps worth carrying forward 1. **"Columns = member count" is RETIRED** (it was the 2026-08-18 rule). It was avoiding dead cells in a short last row and bought a worse defect. A short last row is what a grid looks like; a card wider than its neighbours is what a mistake looks like. 2. **A `:root` override of a Homepage theme variable is silently ignored.** Homepage sets `--color-logo-*` on `.theme-slate`, and that class is on the `` element — the same element `:root` matches. `.theme-slate` (0,1,0) beats `:root` (0,0,1), so the override does nothing and looks like the variable is not the one in play. `html[class]` (0,1,1) wins, and does not hard-code which `theme-*` class is active. Specificity alone is not enough either: a custom property resolves from the *nearest* ancestor that sets it, so the override has to land on ``, not on ``. 3. **The post-recreate tab-bar loss is INTERMITTENT, not guaranteed.** The 2026-08-19 note reads as though every recreate costs up to an hour of broken render. This recreate came up correct within 10 seconds — fresh payload on the first poll, all four tabs clickable a minute later. Recreate, *check*, and only then walk away if it is actually in the broken state. Also re-confirmed, since the change depended on it: **a `settings.yaml` edit needs a container recreate, not a restart.** `docker restart homepage` left the old `"columns":1` payload embedded in the served HTML with the correct file mounted and readable inside the container; `compose up -d --force-recreate` cleared it immediately. ## Deliberately not done — operator's call The Main tab still opens on three sparse bands: **Notes** (1 member) and **Games** (1) each burn a full 4-wide row, and **News** has 2. Merging Notes + News, or folding Games into Apps, would tighten the top of the page — but that is information architecture, not layout, and the group names are the operator's. Surfaced rather than done. → **Resolved in pass 2 below**, where the operator delegated the naming ("re-categorize however you want"). Notes + News became `Daily`, Games folded into `Apps`, and the `AI - Audio Tools` placement in this pass was superseded (Scriberr moved on to `AI - Studios`). --- # `[2026-08-24, pass 2]` Recategorised on "do I open this?", API groups collapsed Operator, after seeing pass 1: *"You can re-categorize however you want. service networking tab is uneven, you can split out the adguard cards, etc. most of the issues are that tools I use and have a UI are interspersed with API endpoints which are largely informational only. They might even go in their own cards or start collapsed."* ## The axis Every group is now either **tools** (expanded, top of tab) or **endpoints** (an API, a broker, an agent — `initiallyCollapsed: true`, bottom of tab). A collapsed group still renders its eyebrow and rule, so presence costs one line instead of two rows. Second, quieter rule that fell out of the same pass: **a group's members should all carry a widget or none should.** A stat strip adds ~50px, so one widget card in a row of plain ones opens a void under the plain ones — which is most of what made the 13-member `Service Networking` band look broken. ## Shape - **Main** — `Daily` (Memos, Miniflux, Nevermore, SearXNG — replaces the 1-card Notes and 2-card News bands), `Monitoring`, `Apps` (12; absorbed the 1-card `Games` band), `Media`, `UltraSeedbox`. - **AI** — `AI - Gateways & Chat` (8) and `AI - Studios` (6) expanded; then `AI - Inference` (7), `AI - Eval & Retrieval` (4), `AI - Speech (TTS)` (4), `AI - Audio Tools` (2), `AI - Dormant` (6) all collapsed. - **Toolchain** — `DNS & Filtering` (3), `Reverse Proxies` (2), `Compose Consoles` (5), `Toolchain` (3), `Agents (no UI)` (6, collapsed). - **Infrastructure** — unchanged; every card there is already a console. Measured after: every group `card=360`, and `DNS & Filtering` and `Reverse Proxies` both `h=134..134` — dead flush. ## ⚠ The move that made it affordable **The sixteen GPU-backed model seats were NOT relabelled.** `homepage.group` is read at container **creation**, so renaming `AI - Inference` to something clearer would have meant recreating six vLLM seats plus four eval seats plus four TTS engines — multi-minute model reloads on endpoints peers reach through the gateway. Order plus `initiallyCollapsed` buys the same separation for free, so the names stay ugly on purpose. **Do not spend that recreate on a label.** 28 containers *were* relabelled — all cheap web services — via five rerunnable elway playbooks, `playbooks/homepage-regroup-.yaml`. The canonical `stacks/` tree was synced to match afterwards, so intent and reality agree. `initiallyCollapsed: true` is a per-group key in `layout:`; confirmed present in this build (`defaultOpen: !(group?.initiallyCollapsed ?? global)` in `/app/.next/server/pages/index.js`). ## AdGuard (ANA) gained its widget, and the credential is fleet-wide It was the only AdGuard without a query/blocked/latency strip, so it sat short beside two tall siblings. **One `infra-ops` AdGuard login authenticates against all three instances** (ANA `:8053`, NH3 `:8080`, ESH `:8080` — all returned 200 on `POST /control/login`, verified 2026-08-24). Vaulted at `secret get nh3-dev/adguard-infra-ops-password`; written to `/opt/docker/compose/adguard-ana/.env` (0600, root) and never into git. Its icon was also the odd one out (`mdi-dns` against two `si-adguard`). ## ⚠⚠ `initialSettings":{}` — the tab-bar mystery is a SWALLOWED EXCEPTION The biggest durable finding of the day, and it cost ~25 minutes. Full write-up in `stacks/homepage/README.md`; the short version: `initialSettings":{}` in the served HTML is **the catch branch** of the page's data loader, not a warm-up and not a cache. And the error can vanish without trace: the logger is assigned as the first statement *inside* the same `try`, and the `catch` only logs `if (logger)`. If the logger is what threw, nothing is written anywhere — which is exactly what was observed. Ruled out by measurement, do not re-run: `/api/services`, `/api/bookmarks`, `/api/widgets` and `/api/hash` all return **200 with correct content** while the page serves `{}`; restoring the previous known-good `settings.yaml` reproduces it identically; `/api/validate` returns `[]`; disk and permissions are fine. **One-command test:** `curl -s http://10.0.50.45:5100/ | grep -o 'initialSettings":[^,]\{0,20\}'` **What broke the streak:** three consecutive recreates came up empty, then rolling the 8.6 MB `conf/homepage/logs/homepage.log` aside and recreating healed it within 15 seconds. That is one observation, not proof — but it is a coherent mechanism (oversized log → logger init throws → silent catch) and it is the cheapest thing to try first next time. --- # `[2026-08-24, pass 3]` Rebuilt on Australis Skyfall — dual theme, light shipped Operator supplied the Skyfall design-system README and said "Go full with skyfall." ## The bundle was already in this repo's git history **The Skyfall tokens did not need to be hunted down.** A predecessor vendored them on 2026-08-19 and a later commit deleted them; git kept everything: ```bash git show 45c1995:stacks/homepage/theme/colors.css # 208 lines, BOTH themes git show 45c1995:stacks/homepage/theme/layout.css # calm-depth tokens git show 45c1995:stacks/homepage/theme/typography.css git show 45c1995:stacks/homepage/theme/fonts/Supreme-{400,500,700}.woff2 ``` `colors.css` carries `:root` (dark) **and** `[data-theme="light"]` (Skyfall Day) — so the light ramp is canonical, not derived. That killed the entire objection from the previous answer, which was correct only about the `australis-design` skill ("Always dark first. No light mode in this system"). **Skyfall is the dual-theme derivative; australis-design is the terminal theme. They are different systems and only one of them has a light mode.** ## ⚠⚠ REMOVING `theme:` FROM settings.yaml BREAKS THE DASHBOARD The documented way to get Homepage's own light/dark toggle is to leave `theme:` unpinned. **Do not.** With the key absent, the page's data loader throws and its catch branch serves `initialSettings: {}` — no tab bar, no layout, no i18n. Measured, not inferred: six force-recreates over seven minutes all came up empty with the key removed; restoring `theme: dark` rendered correctly on the next recreate in **12 seconds**. `/api/services` stays 200 and fully correct throughout, which is exactly why this reads as a caching or warm-up problem and is not one. This is the first *confirmed* trigger for the long-running "tab bar goes missing" mystery. It does not explain every occurrence (the symptom has appeared with `theme:` present), but it means **the first diagnostic step is now `git log -p -- stacks/homepage/conf/settings.yaml`**, not container archaeology. Also retires an earlier lead from this same session: rolling the 8.6 MB `homepage.log` aside once coincided with a recovery, but did nothing during the `theme:`-key episode — coincidence, not cause. ## So the toggle is ours `conf/custom.js` renders it (was an empty placeholder). Precedence: 1. explicit choice — `localStorage['skyfall-theme']`, written by the toggle; 2. OS preference — `@media (prefers-color-scheme: light)`; 3. dark — Skyfall's default. `theme/build.py` re-emits each vendored `[data-theme="light"]` block twice: as `[data-theme="light"], html.light`, and inside the media query scoped to `html:not([data-theme="dark"]):not([data-theme="light"])`. **That `:not()` pair is what lets a stored *dark* choice survive a light-mode OS.** Verified across both OS preferences: load, click, click again, reload — all four correct. ⚠ Homepage keeps its own `class="dark scheme-dark theme-slate"` on `` regardless, because `theme:` is pinned. That is fine and was checked explicitly: with the dark class present AND `data-theme="light"`, every themed surface resolves to Skyfall Day, because our rules carry `!important` on the surfaces Tailwind's `dark:` variants would otherwise claim. **`data-theme` is the control surface; the class is not.** ## The anti-fork guard is now mechanical `build.py` records the SHA-256 of each vendored file and **fails the build** on a mismatch, rather than warning. A vendored file is either byte-identical to the bundle or it is a fork wearing the bundle's name. Overrides go in `skyfall.css.in`, which is written entirely against the semantic layer (`--surface-*`, `--text-*`, `--border-*`, `--success/--danger/--warning`) — no raw family tokens, no colour literals. The one place a literal is unavoidable: Homepage consumes `--color-logo-start/stop` as `rgb(var(--x))`, which cannot take an `oklch()`. Those four values are exact sRGB conversions of real tokens (`--sea-80`, `--blue-base` for dark; `--sea-40`, `--blue-deep` for light), computed rather than eyeballed, with the conversion recorded in the file. ## Deviations, all deliberate and all written down - **The aurora ribbon under the tab bar is gone.** Skyfall sanctions exactly two accent expressions — the active rail and hero-only glows — and a decorative gradient across the chrome is neither. The colour moved to a 2px accent bar plus `--accent-soft` fill on the active tab, which *is* the rail. - **Widget stat values moved from the display face to mono**, per Skyfall's "numbers and telemetry are always `--font-mono`". - **Two font substitutions**: Space Grotesk for Bespoke Sans, JetBrains Mono for Victor Mono. Only Supreme was ever vendored, and Skyfall's own notes call Victor Mono "user-supplied". Two-line swap when the real faces arrive. Dark + light, all four tabs: `http://10.100.10.50:8090/b/homepage-skyfall/` _Archived 2026-09-09._ - `[2026-08-24]` **Homepage reorganised on the axis "do I open this?" — UI groups expanded on top, API/agent groups collapsed at the bottom** (operator-delegated: "re-categorize however you want"). Load-bearing constraint: `homepage.group` is read at container CREATION, so the 16 GPU-backed model seats keep their unlovely names rather than eat a recreate — `initiallyCollapsed` + order is free. Second rule discovered here: **group members should all have widgets or none should**, because a stat strip adds ~50px and opens a void beside plain cards. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md` _Archived 2026-09-09._ - `[2026-08-24]` **Homepage columns unified at 4 for every group; the 2026-08-18 "columns = member count" rule is retired.** It was avoiding dead cells in a short last row and bought a worse defect — card width changing at every group boundary. Also carries two CSS traps: `overflow: hidden` clips at the PADDING box (so a `padding-right` gutter is spill room, not a guard), and a `:root` override of a Homepage theme variable is silently outranked by `.theme-slate` on the same `` element. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md` _Archived 2026-09-09._ - `[2026-08-24]` **AES-128 adopted on both Anaheim tunnels; the per-flow ceiling root-caused to the UDM's software AES-CBC, exonerating the FortiGate.** Proven by an A/B/A cipher swap at identical CPU — hardware offload is not cipher-cost-sensitive. # [2026-08-23] Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gbps circuit > **⛔ SUPERSEDED 2026-08-23 (same day, later session) — read the CORRECTION at > the bottom before acting on anything here.** The headline is wrong (the > relevant ceiling is NH3's **1 Gbps** uplink, not Anaheim's 2 Gbps), the > aggregate number is wrong (**692 Mbit/s** at 8 streams, not ~550 — the > original stopped measuring at 4), and the proposed remedy is **impossible**: > UniFi's manual site-to-site IPsec does not implement AES-GCM at all. The > per-stream observation and the parallelise-your-transfers mitigation are the > parts that survive. The operator noticed site-to-site transfers were slow for a datacenter fiber handoff and asked whether WireGuard was the limit. It is not WireGuard, and the circuit is fine. ## Measured ``` ana-docker -> internet, 8 parallel 2,153 Mbit/s <- the 2 Gbps handoff, delivering ANA <-> NH3 through the tunnel, 4 par. 460 Mbit/s FortiGate's own recorded peak 554 Mbit/s ANA <-> NH3, single stream 227 Mbit/s ANA <-> ESH, single stream 249-265 Mbit/s ESH <-> NH3 (never touches ana-gw) 545-557 Mbit/s on a SINGLE stream ``` Method: stdlib TCP probe (no ssh, no crypto, no compression) between site endpoints; raw circuit measured with 8 parallel HTTPS fetches from Hetzner Ashburn. Host NICs are virtio with no reported cap, so no host-side ceiling. ## What it is not - **Not WireGuard.** Both Anaheim tunnels are IPsec on ana-gw (`pfi-ana-nh3` -> 70.230.226.88, `ana-eshudm-dyn` -> the ESH UDM). WireGuard on ana-wg is remote-access only and is not in this path. Traceroute confirms: both slow paths have hop 1 = `10.250.50.1` (the FortiGate); the fast ESH<->NH3 path rides a `192.168.x` Site Magic overlay and never touches it. - **Not CPU or crypto exhaustion.** FortiGate CPU was **100% idle across all 8 cores** during the tests, and both live tunnels report `npu_flag=03` with `dec_npuid=1 enc_npuid=1` — encrypt *and* decrypt are hardware-offloaded. - **Not a 250 Mbit/s cap.** That was the first number and it is misleading — single-stream TCP. Four parallel streams doubled it. Quote the aggregate. - **Not the interface.** wan1: `rxe=0 txe=0 rxd=0 txd=0`, no collisions. ## Most likely cause Both tunnels negotiate **`aes256-sha1`** in phase 1 *and* phase 2 (dhgrp 14, IKEv2). AES-CBC + SHA1 is a two-pass operation; FortiGate NPUs are markedly faster on **AES-GCM**, which combines encryption and authentication in one pass. The datasheet IPsec headline for an 80F assumes GCM with large packets, not CBC+SHA1 at the 1438-byte tunnel MTU this link negotiates. The ~4x shortfall is consistent with that. ## Not executed Changing the proposal is a **production-edge change requiring a matching change at the far end** (NH3 UDM and the ESH UDM), and each tunnel drops while it renegotiates. Left for the operator. See the index entry for authorization state. ## Immediate mitigation, no config change Per-flow is the weak axis: a single stream over Site Magic gets 557 Mbit/s, a single stream through IPsec gets 227. **Anything moving bulk data across the Anaheim link should parallelise** — that alone roughly doubles throughput today. ## Practical consequence already observed `/mnt/smithy` mounted on ana-ml2 reads at 24.7 MB/s sequential vs 98.3 MB/s from nh3-dev (same file, same mount) — that gap *is* this tunnel, not NFS and not the NAS. See [[2026-08-23-smithy-mount-ana-ml2]]. ## Access note ana-gw is a FortiGate-80F, FortiOS 7.2.10, at 10.250.0.1. `sshpass` is absent on nh3-dev; connect with paramiko via `uv run --with paramiko`. Password is vaulted at `fortigate/ana-gw-infra-ops-password`. **`diagnose vpn tunnel list` prints live ESP session keys** — never paste its output into althing, a booth, or a commit. --- ## CORRECTION (2026-08-23, later session): the cutover was attempted and the remedy does not exist The operator authorised the AES-GCM cutover, NH3 side first. It cannot be done, and the measurements taken while trying show there is very little left to win. ### AES-GCM is unavailable on the far end — not a naming problem The NH3 edge is a **UDM Pro SE** terminating `pfi-nh3-ana` (networkconf `_id 697d64414c85dd2b6669b00a`, `ifname vti64`). Its UniFi API **validates** the crypto enum and rejected every GCM spelling tried — `aes256gcm`, `aes256gcm128`, `aes256gcm16`, `aes-256-gcm`, `aes256-gcm`, `aes256gcm12`, `gcm`, `aes128gcm128` — all `HTTP 400 api.err.InvalidPayload`, nothing applied. **The control that makes this conclusive:** the *identical* request body with `ipsec_esp_encryption: "aes256"` returns `HTTP 200 rc:ok`. So the 400s are the enum rejecting the value, not a malformed body. Corroborating: **zero case-insensitive `gcm` matches across 7.3 MB of UniFi OS UI bundles.** Accepted enum (probed): `aes128`, `aes192`, `aes256`, `3des` → 200; `des`, `chacha20poly1305` → 400. There is no AEAD option. Both Anaheim tunnels land on UniFi far ends, so this blocks the ESH tunnel too. The FortiGate side **was** widened and is GCM-capable: phase2 `pfi-ana-nh3` now reads `set proposal aes256-sha1 aes256gcm`. Left in place deliberately — it is functionally identical while the peer only offers CBC, and reverting it would cost another SA renegotiation for a cosmetic gain. Phase 1 was never touched; IKE protects the control channel only and has no bearing on data throughput. ### The numbers that retire this as a problem Measured NH3→ANA through the tunnel, and NH3→ESH over Site Magic (WireGuard) on the same UDM and the same uplink, with the same stdlib TCP probe: | streams | IPsec NH3→ANA | WireGuard NH3→ESH | |---|---|---| | 1 | 245 Mbit/s | 557 Mbit/s | | 4 | 471 Mbit/s | 767 Mbit/s | | 8 | **692 Mbit/s** | **795 Mbit/s** | **NH3's WAN is a 1 Gbps link** (`uplink.speed = 1000`, port capable of 10G) — that, not Anaheim's 2 Gbps, is the ceiling for anything crossing this tunnel. So the tunnel does **~69% of the achievable uplink** at 8 streams, and the IPsec-vs-WireGuard gap collapses from 2.3× at one stream to **15% at eight**. Re-architecting the transport (site-to-site WireGuard via `ana-wg`, since FortiOS has no WireGuard) would chase that last 15%. Not worth it. ### What the constraint actually is A **per-stream** limit (~245 Mbit/s), not an aggregate crypto ceiling. Both endpoints are idle at load — FortiGate CPU 100% idle with `npu_flag=03` (offloaded both directions), UDM CPU ~7% with load1 moving 0.70 → 1.55. The shape is per-SA/per-flow serialisation, and WireGuard shows the same shape from a higher floor (557 → 795 is only 1.43× scaling). ### Actionable consequence Anything moving bulk data across this link should **parallelise** — 245 → 692 Mbit/s, a 2.8× win with no config change. For single-stream workloads that cannot be parallelised at the application layer, **NFS `nconnect=N` is the lever**: it opens N TCP connections per mount, converting a single-stream workload into a parallel one. The `/mnt/smithy` mount on ana-ml2 reading at 24.7 MB/s (~200 Mbit/s, i.e. exactly the single-stream ceiling) is the live example — remounting with `nconnect=8` is the obvious test. ### Foot-gun recorded Probing the enum by PUTting candidate values **applies the accepted ones**. A probe loop here timed out with `3des` briefly live on the NH3 side, which the FortiGate would not accept — a short tunnel outage until `aes256` was restored (~1 minute, confirmed by the SA counters resetting). If you enumerate a UniFi config enum this way, restore the known-good value after **every** 200, not at the end of the loop. Post-change verification: the UDM object was diffed field-by-field against its pre-change snapshot and is **byte-identical**. --- ## FOLLOW-UP (2026-08-23): what the per-stream limit actually is The correction above called the constraint "per-SA/per-flow serialisation". That was a hand-wave. Measured properly, it is a **hard per-flow rate cap of ~230–245 Mbit/s with a very deep buffer in front of it** — not a tuning problem, not loss, not window size. ### The evidence: pin the send buffer and sweep it Single stream NH3 → ana-docker, `SO_SNDBUF` pinned, `ss -ti` sampled in flight: | in-flight cap | throughput | RTT in flight | minRTT | retrans | |---|---|---|---|---| | 256 KB | 224 Mbit/s | 7.8 ms | 5.3 ms | 0 | | 416 KB | 225 Mbit/s | 11.8 ms | 6.6 ms | 0 | | 416 KB | 245–247 Mbit/s | 12.0 ms | 5.6 ms | 0 | | ~3.3 MB (autotuned) | 245 Mbit/s | **107 ms** | 5.5 ms | 0 | **Throughput is flat across a 13× range of in-flight data while RTT scales with it.** That is the signature of a fixed service rate with a standing queue: the window controls only how much queue you build, never how fast you go. Had this been window-limited, throughput would have risen with the buffer. Had it been congestion, there would be retransmits — there are essentially none (`retrans:0`, 0% ping loss). So `net.ipv4.tcp_*` tuning, window scaling and congestion-control choice are all **red herrings here**. Do not go there. ### Bufferbloat: one bulk stream wrecks latency for everything else Measured on the same tunnel, ping to ana-docker: - idle: **6.9 ms** avg - during a **single** bulk TCP stream: **102 ms** avg, 136 ms max, 0% loss **15× latency inflation from one transfer.** This is the operationally important finding — any interactive traffic sharing the Anaheim link (ssh, RDP, althing, VoIP) degrades badly whenever anything moves bulk data, and it takes only one stream to do it. Parallelising transfers makes throughput better and this *worse*. If it starts biting, the fix is an AQM/shaper on the tunnel (or rate-limiting bulk jobs), not more buffer. ### Where the cap lives — strong inference, not proof Three paths, and the FortiGate is the only variable: | path | single-stream | |---|---| | FortiGate ↔ NH3 UDM (IPsec) | 245 Mbit/s | | FortiGate ↔ ESH UDM (IPsec) | 249–265 Mbit/s | | NH3 UDM ↔ ESH UDM (WireGuard, **no FortiGate**) | 557 Mbit/s | Present in both slow paths, absent from the fast one. Aggregate over the same SA reaches 692 Mbit/s, so it cannot be the SA or the crypto engine as a whole — many flows spread out fine, one flow does not. The mechanism that fits is **FortiGate NPU IPsec offload being per-session**: each firewall session is bound to one crypto engine, so a single TCP flow is capped at one engine's rate while many sessions spread across engines. **This is inference from the throughput shape, not something confirmed on the box** — `diagnose sys session list` was not captured for a TCP flow (the filter caught only traceroute UDP probes). A single-stream control through ana-gw *without* IPsec returned 290 Mbit/s to Hetzner Ashburn, but at ~60 ms RTT that is window-limited and does not discriminate. **If this matters, the clean test is a non-IPsec single stream between two Anaheim VLANs at low RTT.** **Relevant to the FortiGate cutover decision:** if the per-flow cap is the FortiGate's IPsec path, replacing the box plausibly lifts single-stream throughput toward the WireGuard figure. That is a point in favour of the cutover, and it is cheap to verify afterwards by re-running the sweep. --- ## FOLLOW-UP 2 (2026-08-23): it is NOT a capacity problem, and it IS specific to IPsec Operator asked directly whether the 80F "can't handle the traffic". It can. Two new measurements settle the shape of this, and correct an overstatement in FOLLOW-UP 1 (which pointed at the FortiGate on evidence that was confounded — every slow path was *both* IPsec *and* FortiGate, so protocol and box could not be separated by that argument). ### The 80F routes a single flow at line rate when IPsec is not involved `ana-ml2 → pfi-pve`, inter-VLAN **through** ana-gw (traceroute hop 1 = `10.250.50.1`), 0.36 ms RTT, no tunnel: | streams | throughput | |---|---| | 1 | **940.2 Mbit/s** | | 8 | 939.3 Mbit/s | Single stream saturates 1 GbE. So the box does **not** cap single sessions in general, and there is no per-session ceiling in its plain forwarding path. The ~250 Mbit/s per-flow cap is **specific to the IPsec datapath**. ### Both IPsec tunnels converge on the same numbers despite different far ends Measured today with the same probe: | tunnel | far-end gateway | RTT | 1 stream | 8 streams | |---|---|---|---|---| | NH3 ↔ ANA | UDM Pro **SE** | 6.7 ms | 245 Mbit/s | 692 Mbit/s | | ESH ↔ ANA | UDM Pro **Max** | 3.9 ms | **268 Mbit/s** | **715 Mbit/s** | Different gateway hardware, different sites, different uplinks, and RTT differing by 1.7× — yet single-stream differs by only 9%. **If this were window-limited the 3.9 ms path would be ~1.7× faster.** It is not, which is independent confirmation of a rate cap rather than a BDP effect. ### Capacity summary — the box has headroom it will not give one flow - plain routing, 1 stream: **940 Mbit/s** (line rate) - plain routing to internet, 8 streams: **2,153 Mbit/s** - IPsec, 8 streams: **692–715 Mbit/s** - IPsec, 1 stream: **245–268 Mbit/s** - CPU **100% idle** throughout; IPsec NPU-offloaded (`npu_flag=03`) Within a single SA, 8 sessions get ~2.9× what 1 session gets, so the datapath distributes work **by inner session** — consistent with IPsec offload binding a session to one crypto engine. ### What is still NOT separated Whether the cap belongs to **the 80F's IPsec offload** or to **UniFi's IPsec implementation**. Both tunnels have a UDM at the far end, and both UDMs run the same UniFi firmware, so identical caps are explainable either way. The Pro Max being only 9% faster than the Pro SE argues against the UniFi side (a beefier CPU should show more), but that is suggestive, not conclusive. **The test that closes it:** an IPsec tunnel whose endpoints do not include the 80F — e.g. a temporary UDM↔UDM IPsec tunnel between NH3 and ESH, measured single-stream. If it also caps ~250, the FortiGate is exonerated and replacing it buys nothing on this axis. If it runs near the 557 Mbit/s that UDM↔UDM WireGuard achieves, the 80F is the limiter. **Bears directly on the pending FortiGate cutover** — worth running before that decision, not after. --- ## FOLLOW-UP 3 (2026-08-23): WireGuard over the same internet path does 767 Mbit/s on ONE stream Operator asked for a WireGuard test from `ana-wg` to NH3 over the public internet. It is the test that separates the *path* from the *crypto*, and the answer is unambiguous. **It also overturns FOLLOW-UP 1's "re-architecting the transport is not worth it" — that conclusion compared 8-stream numbers and was wrong for single-stream workloads.** ### Setup (fully torn down afterwards) `ana-wg` (10.250.50.252, Debian 12 LXC, 4 cores) already has an internet-reachable WireGuard endpoint: wg0 on **UDP 31337**, published by FortiGate VIP `wg-to-ana-wg` (extip **38.120.12.42** → 10.250.50.252:31337, policy 46, service `WireGuard-LEET`). **No FortiGate change was needed.** A temporary `wgt0` was created on nh3-dev (10.30.10.200/32) as a fourth peer on wg0, measured, then removed — ana-wg is back to its original 3 peers and the keys were shredded. `wireguard-tools` was installed on nh3-dev and **left in place** (benign, and wanted if this becomes permanent). In this topology **neither gateway does crypto**: the FortiGate and the NH3 UDM only NAT/forward UDP, and Linux does WireGuard at both ends. ### The full comparison | path | crypto performed by | 1 stream | 8 streams | |---|---|---|---| | IPsec NH3↔ANA | FortiGate + UDM | 245 Mbit/s | 692 Mbit/s | | IPsec ESH↔ANA | FortiGate + UDM | 268 Mbit/s | 715 Mbit/s | | **WireGuard NH3→ana-wg** (same internet path) | **Linux + Linux** | **767 Mbit/s** | 763 Mbit/s | | WireGuard NH3↔ESH (Site Magic) | UDM + UDM | 557 Mbit/s | 795 Mbit/s | | plain routing through the 80F (inter-VLAN) | none | 940 Mbit/s | 939 Mbit/s | **One stream equals eight streams over Linux WireGuard (767 ≈ 763).** There is no per-flow penalty at all, and a single flow already saturates the path. So the ~245 Mbit/s per-flow cap is **not** the ISP, not the circuit, not the NH3 uplink and not the physical path — all of which sustain 767 on one flow. Per-flow penalty ranks by implementation: - **Linux WireGuard — none** (767 → 763, flat) - **UDM WireGuard — mild**, ~1.4× (557 → 795) - **IPsec on this pair — severe**, ~2.8× (245 → 692) ### Latency under load — the same story | path | idle | during ONE bulk stream | |---|---|---| | IPsec NH3↔ANA | 6.9 ms | **102 ms** avg, 136 ms max | | WireGuard NH3→ana-wg | 6.2 ms | **12.7 ms** avg, 23 ms max | WireGuard carries **3.1× the single-stream throughput with 8× less latency inflation** on the same wire. ### Attribution — still not fully separated, and it no longer matters much Both IPsec measurements have a FortiGate *and* a UDM doing IPsec, so this still does not isolate which one imposes the 2.8× penalty. Closing that would need Linux↔Linux IPsec or UDM↔UDM IPsec on the same path. **But the practical decision no longer depends on the answer**, because the fix is the same either way and it is already demonstrated. ### Recommendation (supersedes FOLLOW-UP 1) A **WireGuard site-to-site between NH3 and Anaheim, terminated on `ana-wg`**, is worth real consideration: 3.1× single-stream, flat scaling, far better latency under load, and it reuses infrastructure that already exists and is already internet-reachable. It is also the architecture already proven for NH3↔ESH. Open questions before committing: routing/failover if ana-wg (an LXC) is down, whether it replaces or parallels the IPsec tunnel, and firewall policy for the new transit. ana-wg CPU was only ~40% busy across 4 cores at 767 Mbit/s, so it has headroom. **AND: `nconnect=8` on /mnt/smithy remains worth doing regardless** — it is the same lever (turn one flow into many) and brokkr-smithy-dev has given standing approval to apply it once the FortiGate work settles, with no need to ask again. --- ## RESOLVED (2026-08-23): it is the UDM's software AES-CBC. The FortiGate is exonerated. Operator's theory — the UDM does IPsec in software with no crypto offload, so the cost of the cipher itself is the limit — is **correct**, and it is now demonstrated rather than inferred. He also correctly pointed out that UDM↔UDM Site Magic is **WireGuard, not IPsec**, so that row never said anything about UniFi's IPsec performance. It didn't, and I had leaned on it. ### The controlled experiment: vary cipher cost, hold everything else AES-128 is 10 rounds, AES-256 is 14. If software crypto is the binding constraint, throughput must rise when the cipher gets cheaper. If the limit were the FortiGate's NPU, it would not move at all — hardware crypto is not cipher-cost-sensitive in that range. Run A/B/A, single stream, 25–60 s each: | condition | ESP cipher | single-stream | UDM CPU | |---|---|---|---| | A | aes256-cbc + sha1 | 232.3 Mbit/s | 35.4% | | B | **aes128**-cbc + sha1 | **281.8**, 274.9 Mbit/s | 35.5% | | A again | aes256-cbc + sha1 | 244.9, 242.5 Mbit/s | — | **~1.16–1.20× faster on the cheaper cipher at identical CPU.** Same bytes of CPU work, more payload through it. That is the signature of CPU-bound software crypto, and it rules out the FortiGate's NPU as the limiter. ### Correcting two of my own earlier claims 1. **"UDM CPU is only ~7%, so it isn't CPU-bound" was WRONG — a sampling artifact.** UniFi's `system-stats.cpu` refreshes on the device report interval; 4-second sample windows were reading stale values. Under a sustained 60 s single-stream load it reads **35.4%**, with load1 rising 0.60 → 1.17. On a 4-core UDM Pro SE that is ≈1.4 cores — one core saturated on crypto plus overhead. **Always drive load for ≥60 s before trusting a UniFi CPU figure.** 2. **The "FortiGate per-session NPU offload" hypothesis is REFUTED**, not merely unproven. It predicts no change from a cipher swap; a 20% change was measured. ### Why the numbers all line up now - **1 stream = 1 core of UDM crypto** → ~240 Mbit/s on AES-256-CBC. - **8 streams = ~3 usable cores** → ~692 Mbit/s, ≈2.9× the single-stream figure on a 4-core box. Aggregate is noisy (492–692 across repeats on a live link) and is *not* cipher-sensitive, consistent with it being bounded by the path/ uplink rather than crypto once several cores are engaged. - **AES-CBC is the specific villain: it is serial.** Each block depends on the previous one, so the ARM AES instructions cannot pipeline across blocks. GCM (CTR-based) and ChaCha20-Poly1305 both parallelise freely. That is why the same UDM does 557 Mbit/s single-stream on WireGuard and only 240 on IPsec. - **This retroactively vindicates the GCM cutover as the right idea aimed at the right box** — GCM would have removed the serial dependency on the constrained end. UniFi simply does not offer it, which is what made it impossible. ### Options this opens - **AES-128 instead of AES-256: ~16–20% for free**, no topology change, one API call per end. 128-bit is not the weak link here (SHA1 integrity is more dated, and unchanged either way). Operator's call — **not adopted**, restored to aes256. - **WireGuard site-to-site via ana-wg: 767 Mbit/s single-stream** (3.1×), and it sidesteps the UDM's IPsec datapath entirely. Still the biggest win available. - Replacing the FortiGate **will not help this** — it was never the constraint. Worth knowing before the cutover. ### State left behind UDM network object verified **byte-identical** to its pre-test snapshot (aes256/sha1). Tunnel up, selectors 1/1. FortiGate phase2 `pfi-ana-nh3` is left as `aes256-sha1 aes256gcm aes128-sha1` — a permissive superset; the peer offers only aes256 so the extra entries are inert, but **narrowing it back to `aes256-sha1` is one line** if the looser list is unwanted. --- ## FOLLOW-UP 4 (2026-08-23): a downstream WireGuard terminator costs nothing to forward through Operator's point: FortiOS has no WireGuard, so a WireGuard site-to-site must terminate on a box *behind* the edge. Correct — and `ana-wg` (LXC, CT 113 on pfi-pve, 10.250.50.252) already is that box. **This closes a gap in FOLLOW-UP 3.** That 767 Mbit/s figure was measured with traffic terminating *on* ana-wg. Real traffic must be forwarded onward to other Anaheim hosts, which was never measured. Now it is: | topology | 1 stream | 8 streams | |---|---|---| | IPsec, FortiGate ↔ UDM (today) | 245 Mbit/s | 692 Mbit/s | | WG terminating **on** ana-wg | 767 Mbit/s | 763 Mbit/s | | **WG transit: nh3 → wg → ana-wg → forward → ana-docker** | **763.8 Mbit/s** | **790.4 Mbit/s** | **Forwarding through the LXC is free** (763.8 vs 767). The downstream-VM architecture delivers the full 3.1× single-stream for real transit traffic, not just for traffic landing on the tunnel box. ana-wg while forwarding 764 Mbit/s: **~22% busy across 4 cores** (77.8% idle), so roughly 0.9 cores. Note `/proc/loadavg` inside this LXC reports the *host's* load, not the container's — do not read it as ana-wg's own. For contrast the UDM burns 35.4% of its 4 cores to move 240 Mbit/s, so ana-wg has ample headroom. ### Design consequences of terminating downstream — the parts that need decisions 1. **Anaheim hosts must route to ana-wg, not to the FortiGate.** The 763.8 figure was obtained with an explicit `10.30.10.200/32 via 10.250.50.252` route on ana-docker. Without that, a host sends 10.100.0.0/16 to its default gateway (ana-gw), which routes it back out the *same* interface to ana-wg — a LAN hairpin crossing the FortiGate twice. **The hairpin variant was NOT measured.** Options: DHCP option 121 pushing the route fleet-wide, a dedicated transit VLAN for ana-wg, or accept the hairpin. 2. **New single point of failure.** Today site-to-site dies only when the edge dies, which is total anyway. A downstream terminator fails independently. Mitigation: keep the IPsec tunnel configured as a higher-metric fallback route so it takes over when ana-wg is down. 3. **ana-wg is an LXC on pfi-pve**, so its ~0.9 cores and NIC traffic land on the hypervisor shared with the rest of the Anaheim VMs. 4. **The NH3 end needs a terminator too**, and there are two shapes: - **Linux VM at NH3** (nh3-dev or a dedicated VM on nh3-pve) — this is what was measured: **764 Mbit/s**. - **NH3 UDM's existing WireGuard server** (`PFI-NH3-WG`, wireguard-server on UDP 31337) accepting ana-wg as a peer — plausible but **untested**, and UniFi's WireGuard shows a per-flow penalty (557 Mbit/s single-stream on Site Magic), so expect ~557 rather than 764. Still 2.3× today. ### Standing recommendation Worth doing, but it is **a project, not a config tweak** — routing, failover and policy all need deciding. The cheap wins remain available meanwhile and are independent: `nconnect=8` on NFS mounts (approved by brokkr-smithy-dev, pending the FortiGate work settling) and AES-128 for ~20%. --- ## LANDED (2026-08-23): AES-128 on both tunnels; FortiGate public admin closed Operator directed: adopt AES-128 on **both** Anaheim tunnels, make-before-break, then close the FortiGate's WAN and SSH admin surfaces. All done and verified. **Context that retires the WireGuard-in-a-VM design work:** the FortiGate is being **replaced by OPNsense on a Dell R420**, which gives **WireGuard on the edge device itself**. The downstream-terminator architecture (FOLLOW-UP 4) is therefore moot — do not scope it. This also **un-parks the OPNsense migration**, which auto-memory recorded as PARKED pending "hardware acquisition"; the R420 is that trigger. ### What changed Make-before-break on the FortiGate first, so neither tunnel dropped waiting on a far end: | phase2 | proposal now | |---|---| | `pfi-ana-nh3` | `aes256-sha1 aes256gcm aes128-sha1` | | `ana-eshudm-dyn` | `aes256-sha1 aes128-sha1` | Then each UDM flipped to `ipsec_esp_encryption: aes128`: | tunnel | UDM object | before | after | |---|---|---|---| | NH3 ↔ ANA | `pfi-nh3-ana` `697d64414c85dd2b6669b00a` @ 10.100.0.1 | 245 Mbit/s | **269.7** | | ESH ↔ ANA | `esh-ana` `697723b9b9d4266dddf2bcc7` @ 10.0.0.1 | 268 Mbit/s | **304.3** | Single-stream gain ~10–13% here, against 16–20% in the earlier controlled A/B — the difference is live-link variance, not a different result. Both UDM objects were diffed field-by-field against pre-change snapshots: **the only field that moved on either is `ipsec_esp_encryption`.** The FortiGate proposal lists were deliberately **left permissive** (still accepting aes256). The peers offer only aes128 so the extra entries are inert, and keeping them means a UDM reverting does not strand the tunnel. Narrowing to `aes128-sha1` alone is a one-liner if the looser list is unwanted. ### Admin surfaces closed `wan1 allowaccess` → **`ping`** (https + ssh removed) and `infra-ops` trusthost → **10.0.0.0/8 only** (the 8 wide-open ranges unset). Verified 443 and 22 closed from both NH3 and ESH; management over the tunnel at 10.250.0.1 still works. **Sequencing that matters: the close was executed over the TUNNEL path, not over WAN** — removing `ssh` from allowaccess while connected over WAN kills the session mid-command. **Consequence to hold in mind: ana-gw now has no out-of-band management path.** If both tunnels drop it is console-only until someone is on site. ### Gotcha: the two UDM vault items have DIFFERENT shapes - `unifi/pfi-udmse-api-key` → a **bare 32-char key**. `secret get` output is the key. - `unifi/esh-udmpm-api-key` → a **19-line documentation note** with the key on a `key:` line. `secret get` piped straight into a header yields a 1396-byte value and the UDM answers **`400 Bad Request` from nginx**. Extract with `grep '^key:' | awk '{print $2}'`. **The ESH key's first-ever confirmed WRITE happened here** (auto-memory recorded it as read-verified only): a control PUT of the unchanged object returned `rc:ok`, then the real change did too. That key has a full read+write admin role. --- ## CORRECTION (2026-08-23): port 80 on the WAN IP is the FortiOS ACME listener The claim in the previous section that `.42:80` was an **ISP transparent proxy** was **WRONG**, and so was the earlier warning that ACME renewal would fail with port 80 absent from `allowaccess`. Operator pushed back asking where the port-80 map terminated. It terminates **on the FortiGate itself**. **What it is:** the FortiOS **ACME HTTP-01 challenge listener**. `config system acme` has `set interface "wan1"`, and FortiOS opens port 80 on that interface to answer Let's Encrypt challenges **independently of `allowaccess`** — `wan1 allowaccess` reads `ping` only and the port is still open. Every non-challenge request returns a fixed 403 whose body is literally: ``` ACME Access OnlyACME Access Only ``` **Not a DNAT.** The full VIP table has 14 entries; only two land on `.42` — `Kokoro-In` (:8880 → 10.250.50.51) and `wg-to-ana-wg` (:31337 → 10.250.50.252). ~~Worth noting separately: four VIPs are all-port static NAT~~ — **that claim was WRONG, see the correction below.** All fourteen VIPs are scoped. ### The methodology error that produced the wrong answer — worth not repeating The sniffer filter used was `dst host 38.120.12.42 and tcp port 80`. **`dst host` matches only inbound packets**, so outbound SYN-ACKs were excluded *by construction*; concluding "the box sends no SYN-ACK" from that capture was unsound. Re-run with the bidirectional `host 38.120.12.42 and tcp port 80` it immediately shows `wan1 out 38.120.12.42.80 -> : syn ack`. **Rule: when testing whether a box *answers*, the sniffer filter must be bidirectional. `dst host` silently answers a different question.** ### Consequences - **ACME renewal will work** with `allowaccess ping`. The earlier "add `http` back or the cert expires" warning is retracted — FortiOS opens the challenge port itself. Cert valid to 2026-10-27, renewal attempt ~2026-09-27. - **It is not an admin surface** — static 403, no auth, no GUI. - Its practical value is now low: WAN admin is closed, so the cert only serves the internal GUI at 10.250.0.1, where the name would not match anyway. Killing it (`config system acme` → unset interface) would close the last WAN listener at the cost of cert renewal. Operator's call; **not done**. --- ## CLOSED OUT (2026-08-23): ACME disabled; and the "all-port VIP" alarm was FALSE ### ACME disabled — the WAN IP now exposes nothing `config system acme / unset interface` (the account object is left in place; with no interface bound there is no listener). Verified: - **External scan of 38.120.12.42 across 55 ports: no open TCP ports at all.** - Internal GUI at 10.250.0.1 still answers **200**, SSH still works. - `admin-server-cert` is still `ana-fw.pfi` — the existing cert is untouched and serves the internal GUI until **2026-10-27**; it simply will not auto-renew. Reverse with `config system acme / set interface "wan1"`. ### RETRACTION: the four VIPs are NOT all-port A previous section claimed `Rustdesk`, `https-to-tacticalrmm`, `web-to-webhost` and `web-to-sfcontainer` were unrestricted all-port static NATs. **They are not.** A FortiOS VIP can be scoped **two different ways** and the parser used only checked one: 1. `set portforward enable` + `set extport ` — a single mapped port, **or** 2. `set service ""` on the VIP object — constrains the VIP to that service. All four use form 2. The custom services are narrow: `Rustdesk` = TCP 21115–21119 + UDP 21116 (the standard RustDesk range), `ssh-mapped-2223` = TCP 2223 only. **Every one of the 14 VIPs is scoped; none is unrestricted.** **Lesson: absence of `portforward` does NOT mean all-port on a FortiOS VIP — check `service` too.** Better still, do what settled it here: scan from outside rather than reading config. ### Ground-truth public exposure (external TCP scan, post-change) | IP | open | maps to | |---|---|---| | 38.120.12.41 | *nothing* | — | | **38.120.12.42** | ***nothing*** | the FortiGate itself — fully closed | | 38.120.12.43 | 80, 443 | sf-ana-container 10.250.150.100 (SureFire tenant) | | 38.120.12.44 | 22, 80, 443, 8025, 21115–21119 | gitea (→222), traefik, mailrise, RustDesk | | 38.120.12.45 | 80, 443, 2223 | pfi-ana-webhost 10.250.50.52 (2223→22) | | 38.120.12.46 | 443 | pfi-tacticalrmm 10.250.50.57 | Configured-but-closed: 8443 (mattermost-calls), 8444 (webdav-nas), 8880 (Kokoro-In) — VIPs exist, nothing listening behind them. Worth a tidy-up during the OPNsense translation but not exposure. _Archived 2026-09-09._ - `[2026-08-24]` **ana-gw's public admin surface closed to zero open ports, ACME listener included.** Two of my diagnoses were wrong first (an "ISP proxy" that was the FortiGate, and an "all-port VIP" alarm that was a parser gap) — both from reading config instead of measuring from outside. # [2026-08-24] ana-gw public admin surface closed to zero, ACME listener included WAN admin was opened at the start of the session as a cutover contingency ("so I don't have to drive down there"), then closed again on operator instruction once the AES-128 work landed. Net result: **the FortiGate's WAN address now exposes no TCP port at all.** ## Final state External scan of `38.120.12.42`, 55 ports: **nothing open**. Verified from two sites. `wan1 allowaccess` = `ping`; `infra-ops` trusthost back to `10.0.0.0/8`. **Consequence to hold: there is no out-of-band path to ana-gw.** If both tunnels drop it is console-only. Re-open is two one-liners (allowaccess + trusthost) — both are recorded in auto-memory `reference_fortigate_ana_gw_access`. ## Port 80 was the FortiOS ACME listener, and I got it wrong first `38.120.12.42:80` answered a bare 403 (`ACME Access Only`, 101 bytes) with `allowaccess` set to ping only. First diagnosis — "an ISP transparent proxy" — was **wrong**, and the reason is worth keeping: > The sniffer filter was `dst host 38.120.12.42 and tcp port 80`. **`dst host` > matches inbound only**, so outbound SYN-ACKs were excluded *by construction*, > and concluding "the box sends no SYN-ACK" from that capture was unsound. Re-run bidirectionally (`host … and tcp port 80`) it immediately showed `wan1 out 38.120.12.42.80 -> : syn ack`. **Rule: to test whether a box *answers*, the filter must be bidirectional.** The listener is opened by `config system acme / set interface "wan1"` and **bypasses `allowaccess` by design** — FortiOS needs port 80 for HTTP-01. It was disabled (`config system acme / unset interface`); the LE cert (`ana-fw.pfi`, valid to 2026-10-27) is untouched and simply stops renewing, which is fine because WAN admin is closed and the box is being replaced. ## Retracted in the same pass: the "four all-port VIPs" alarm Claimed four VIPs were unrestricted all-port static NAT. **False.** A FortiOS VIP is scoped **two** ways — `portforward`+`extport`, *or* a `service` binding on the VIP object — and only the first was checked. All 14 VIPs are scoped; `Rustdesk` is TCP 21115–21119, `ssh-mapped-2223` is TCP 2223 only. Ground-truth external scan of all six public IPs is recorded in `reference_fortigate_ana_gw_access`. Configured-but-dead: `:8443` (mattermost-calls), `:8444` (webdav-nas), `:8880` (Kokoro-In) — tidy-up candidates for the OPNsense translation, not exposure. **Lesson, twice in one session: measure from outside instead of parsing config.** Both wrong answers came from a filter that answered a different question. _Archived 2026-09-09._ - `[2026-08-24]` **Scriberr deployed on ana-ml2 GPU1, image built from source.** Three upstream bugs: the Blackwell image was never published, it must run as uid 10001, and `UV_LINK_MODE=copy` is required or two backends fail silently. # [2026-08-24] Scriberr transcription deployed on ana-ml2, GPU1 Self-hosted audio/video transcription + diarization. Operator chose GPU placement over ana-docker (8 cores shared with 50 containers, 37 GB disk) against ana-ml2's 96 cores, `/tank`'s terabytes and GPU1's headroom. **Live:** `http://scriberr.ana.internal:8080` (DNS alias added), health `healthy`, all seven backends up, zero failures: `whisperx pyannote sortformer parakeet canary voxtral openai`. ~30 GB of weights on `/tank`. Stack: `stacks/scriberr/`. Full gotcha list in auto-memory `reference_scriberr_ana_ml2`. ## Three upstream bugs, none of them ours **1. The Blackwell image does not exist.** Upstream's README documents `scriberr-cuda-blackwell`; GHCR has **no tags for it**. Published `scriberr-cuda` covers sm_61–sm_89 only — on these sm_120 cards it fails or silently drops to CPU. The real sm_120 path is `Dockerfile.cuda.12.9` (CUDA 12.9.1, cu128 torch), **built from source**. Do not "simplify" the compose back to the published image. **2. It must run as uid 10001, not 1000** — and the error lies: `unable to open database file: out of memory (14)`. Error 14 is `SQLITE_CANTOPEN`, not an OOM, on a box with 566 GB RAM. That Dockerfile creates `appuser` at 10001 (Ubuntu 24.04 owns uid 1000 as `ubuntu`) and chowns `/app` to it, while the entrypoint's PUID remap covers only the data dirs. **Isolated by elimination**: SQLite writes fine to `/tank` as 1000 → not the mount; fails on a plain named volume too → not the storage; the **published CPU image works at PUID=1000** because there `appuser` *is* 1000. Generalisable: *when a container "permission" bug appears, compare the uid the image was BUILT for against the uid you are RUNNING as.* **3. `UV_LINK_MODE=copy` is required.** Scriberr builds each backend's Python env with `uv` at start; uv's reflink mode fails on overlayfs+ZFS with `Failed to clone … Resource temporarily unavailable (os error 11)`. **Partial failure** — WhisperX and PyAnnote came up and the app looked fine while Parakeet and Sortformer were silently absent. Occurrences 2 → 0 after the fix. ## Related `speaches` on irv-ml1 **stopped** the same day (stack retained, one command to restart): Eyra was abandoned pre-implementation because Scriberr covers the need, leaving it with no consumer. Scriberr runs its **own** WhisperX in-container and is **not** a speaches consumer. Idle footprint at stop was 274 MiB, not the ~5.9 GB quoted — that figure is the loaded-model working set. _Archived 2026-09-09._ - `[2026-08-24]` **ESH DNS fixed at the IPv6 layer and the naming scheme went live on three hosts.** UniFi's RDNSS cannot be disabled but CAN be redirected — the field is only honoured when an explicit server is given. # [2026-08-24] ESH DNS fixed at the IPv6 layer, and the naming scheme went live Reported as "`scriberr.ana.internal` doesn't resolve on my Mac, and nslookup shows an IPv6 DNS server." Operator's diagnosis was right; the fix took three wrong turns worth recording. ## Root cause `esh-userland` has IPv6 PD with RA at `pref high`, and the UDM advertises **itself** as the resolver via RDNSS. macOS honours RDNSS and prefers it over the DHCPv4-supplied resolver, so queries went to the UDM — which does not know `.internal` — and returned NXDOMAIN. AdGuard was never consulted. Two adjacent gaps found while there: `esh-userland`'s **secondary** v4 resolver was `10.0.10.1` (the UDM itself), and `esh-server` had **DNS handout disabled entirely**, so every host there got the UDM and could never resolve `.internal` — esh-docker-vm was living proof. ## The three wrong turns 1. **`dhcpdv6_dns_auto=false` alone does nothing.** It is only honoured **when an explicit server is supplied**. Setting it bare looks like a no-op and invites the conclusion that the field is inert — which I drew, wrongly. Despite the `dhcpdv6_` prefix it *does* drive the RA's RDNSS option on a SLAAC network. 2. **`wan_dns1` is NOT used by the UDM's LAN-facing forwarder.** Setting it to AdGuard persists, reads back, and changes nothing. Proven with **fresh uncached ad domains** — AdGuard blocklists answer `0.0.0.0`, the UDM returned real IPs. Reverted. 3. **`force-provision` returns `rc:ok` and proves nothing** — consistent with the known `cmd/devmgr` behaviour. Every failed attempt returned `rc: ok`. **Verify by observed effect.** RAs were probed with a stdlib raw-socket Router Solicitation parsing option type 25 (`rdisc6`/`tcpdump` were both absent; nothing was installed). ## What landed RDNSS **redirected** rather than disabled — better than switching it off: | VLAN | v4 | v6 RDNSS | |---|---|---| | `esh-userland` | 10.0.50.45 + 10.100.50.40 | `…:4411:b105:50:45` | | `esh-server` | 10.0.50.45 + 10.100.50.40 | `…:4411:b105:50:45` | The v4 secondary moved from the UDM to the **NH3 AdGuard** — reachable over Site Magic and authoritative for the zone. ⚠ **A secondary only fails over on SILENCE, not on wrong answers**: NXDOMAIN is a *successful* answer, the client accepts it and never retries. A secondary that doesn't know your private zone is a coin flip, not a spare tyre. `esh-cameras` deliberately untouched — routing camera DNS through AdGuard's filtering risks their cloud features. ## The naming scheme became real The resolver address is the scheme's first live use, replacing a MAC-derived SLAAC address that would break on a NIC change. All three `esh-server` Linux hosts now carry `4411:B105` ("FOR ALL BIOS"): ``` esh-docker-vm …:4411:b105:50:45 esh-pve-nas …:4411:b105:50:55 esh-vm-db …:4411:b105:50:60 ``` Applied by an `if-up.d` hook that **derives the prefix at runtime** (self-heals on re-delegation), backgrounds itself with a retry (SLAAC may not have landed; a blocking hook would stall bring-up on a headless box), and adds nothing to existing config. **Not** an `iface … inet6 static` stanza — on Debian that sets `accept_ra=0` and would strand the host. ⚠ **Proxmox bridges need `accept_ra=2`.** `esh-pve-nas` had link-local only despite every sysctl looking right: `vmbr0.forwarding=1`, and the kernel ignores RAs on a forwarding interface unless `accept_ra` is explicitly `2`. Fixed with `accept_ra_defrtr=0` alongside, so it takes the prefix but **declines the default route** — an IPv6 identity with no change to a hypervisor's routing. Expect this on every Proxmox node when its LAN gets v6. Canonical: `docs/pfi/ipv6-naming-scheme.md`. UniFi limits: auto-memory `reference_unifi_dns_rdnss_limits`. _Archived 2026-09-09._ - `[2026-08-24]` **esh-vm-db brought onto the fleet infra-ops identity and given its first vaulted credential.** It previously had none: root and infra-ops refused key auth and `lkraven`'s sudo wanted a password nobody held, leaving `qm guest exec` from the hypervisor as the only privileged path. Break-glass root password at `secret get esh-vm-db/root-breakglass-password` (console-only; plaintext never crossed the wire — only its SHA-512 hash did). _Archived 2026-09-09._ - `[2026-08-23]` **Anaheim's IPsec tunnel ceiling — investigated, then CLOSED 2026-08-24.** The 25%-of-2-Gbps framing was wrong (NH3's uplink is 1 Gbps); AES-GCM proved impossible; AES-128 landed instead. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md` _Archived 2026-09-09._ - `[2026-08-23]` **selene retired after losing a head-to-head on its own job; `chat-judge` moved to gen, the model name 404s by design.** Also surfaced that **7 aliases share one seat** — cross-checking between them is an echo, which caught a real defect in brokkr's 46k-exposure R47 gate. # [2026-08-23] selene seat retired after losing a head-to-head; 7 aliases share one seat ## Why selene went Benchmarked against `gen` on selene's own job — 24 designed judge items with checkable ground truth, pairwise + absolute modes, 3 repeats, run on **both** a neutral JSON prompt and Selene's **native Atla template** (288 calls, free local). ``` neutral JSON selene 20/24 (83%) gen 23/24 (96%) native Atla selene 21/24 (88%) gen 22/24 (92%) ``` gen won on both templates and **selene's best sat below gen's worst**. Selene was given its own fine-tuned template as a fairness check before any recommendation; it gained one point, not three. **Decisive defect: selene cannot emit "tie"** — 0/2 on both templates, forcing a winner on every equivalent pair. For eval work that is the case that matters. brokkr-smithy-dev independently corroborated from the other end with a **null control** (an excerpt compared against ITSELF, where tie is definitional): `chat-judge`(selene) TIE **27/60 = 45%**, gen **60/60 = 100%**; ground-truth recovery on real-corpus ranking selene **47% — chance** vs gen 94%. My 83-vs-96 understated it: on a *ranking* task selene was a coin flip. Absolute scoring on designed items is an easier task than ranking real text — the harness is a **screen, not a verdict**, and its README says so. Reclaimed **17.2 GiB** on ana-ml2 GPU1 (free 1,818 -> 19,450 MiB). ## The naming rule, restated the hard way I proposed repointing `selene-1-mini-8b` at gen and was **correctly overruled**: > never repoint a named model at a different model's endpoint — that is > intentionally misleading `chat-judge` is a **role** alias (ADR-0012: consumers bind the capability) and moved to gen with a deterministic judge profile copied from `image-judge`. `selene-1-mini-8b` is a **model** name and was removed outright — it now returns `HTTP 400 Invalid model name`, verified. The discriminator: *does the string promise a capability, or an identity?* ## The 7-way alias collision — the finding with the longest reach ``` chat-judge classifier gen image-judge qwen-image-bench summarizer summarizer-large -> qwen3.8-27b-uncensored :8015 ``` Also colliding: `gen-frontier`/`gen-frontier-reasoning`/`glm-5.2`/`glm-5.2-reasoning`; `ext-tts`/`gpt-4o-mini-tts`/`tts-1`/`tts-1-hd`; `reranker`/`reranker-a3-bge-v2-m3`. **Cross-checking a result against another alias measures nothing when they are the same weights — agreement is an echo, not corroboration.** Documented at the head of `model_list` in the live gateway config, because it belongs where people read it. This caught a real defect within hours: brokkr's R47 premium-corpus gate was about to run ~46,000 record-exposures against `gen` with `summarizer` shortlisted as an independent second opinion. They pinned the backing model in the preregistration and dropped the second-alias idea instead. ## Provenance seam (brokkr's pushback, adopted) The gateway returns the **alias** in the response `model` field, not the backing model — so a per-call guard catches a swap *during* a run and is blind to one *between* runs. **Role alias for routing, concrete model for provenance.** `GET :4000/model/info` with the shared key already exposes backing model + api_base; resolve at run start AND end and void on mismatch. ## Artifacts - Harness kept at `tools/judge-bench/` (`--models` REQUIRED — a stale default would silently benchmark a retired seat). - `stacks/selene/` keeps compose + a README explaining the retirement. - Technique worth stealing, from brokkr: **a control constructed so the correct answer is DEFINITIONAL rather than judged cannot inherit the designer's error.** Item vs itself; response vs its own truncation; text vs its own clauses permuted. Add those before adding more judged items. Commits `ca3c984`, `b8a5355`. _Archived 2026-09-09._ - `[2026-08-23]` **hrafn adopted; its CI reported green for its whole life while deploying nothing.** A staging dir inside the rsync target destroyed its own source mid-copy; the deeper fault was verify steps that asserted uptime, never content. # [2026-08-23] hrafn adopted; its CI deploy reported green while deploying nothing `hrafn` — genuine-Chromium browser-fetch behind a REST API, for bot-gated sites (Reddit first). Built by nevermore-claude on ana-docker, handed to infra-ops for uptime ownership. Internal-only on `traefik-net`, no host port; consumers reach `http://hrafn:8080`. Canonical at `stacks/hrafn/`. ## Intake found a live credential exposure `/opt/docker/compose/hrafn/.env` was mode **0644 with a live 57-char bearer token** — verified as real exposure by reading it as `nobody` on a box with four interactive accounts. Tightened to 0600. That triggered the wider sweep (see [[2026-08-23-ana-docker-env-perms-sweep]]). ## The CI defect — the one worth remembering I authored the deploy (elway playbook + gitea workflow) to replace a hand-rsync, tagging the image with the commit SHA for provenance. nevermore-claude later found v1.0.0 deploying "green" while the host still served 0.1.0. **Root cause was mine and nastier than either hypothesis.** The staging dir was `$compose_dir/.stage` — **inside** the rsync target. So `rsync -a --delete $compose_dir/.stage/ $compose_dir/` deleted `.stage` from the destination (absent from the source listing) **during** the transfer, destroying its own source mid-copy. Reproduced exactly: ``` before: app.py="OLD" leftover.txt .stage/app.py="NEW" after: app.py="OLD" leftover.txt GONE, .stage GONE ``` Deletion succeeded, the copy silently did not, rsync exited 0. So the directory *looked* converged while host source stayed frozen at the first manual rsync — and because the build's `COPY` inputs never changed, Docker full-cache-hit and every SHA tag aliased one image. **The provenance the tagging existed to provide was false for the pipeline's entire life.** **The real failure is the verification.** The verify steps asserted the marker, container health, and a 200 from `/readyz` — all of which pass against a completely frozen host. None measured *content*. A deploy that reports success without asserting the bytes changed is verifying an **uptime**, not a deploy. ## Fixes - stage at `/tmp/hrafn-deploy-stage`, outside the target - CI computes `context_sha256` over the shipped file list; the playbook recomputes it **on the host after the converge** and fails on mismatch - compare the running container's `src/**/*.py` against the host's, so a SHA tag cannot name layers the image lacks - **compare `*.py` only** — `pip install .` generates `src/*.egg-info/*` inside the image and `__pycache__` appears at runtime, so a naive `find src -type f` compare false-fails on every healthy deploy. Verified against a known-good container before shipping (12 host files, 18 in container, 0 content differences). - declined `--no-cache`: a cache hit is *correct* when the context is genuinely unchanged; assert the property rather than brute-force it. ## Access Operator granted claude-bot **write** on `vh/hrafn`, so infra-ops maintains the pipeline it owns instead of routing patches through the repo holder. `vh/hrafn` is canonical; `stacks/hrafn/ci/` is a verified mirror. Commits `b6924de`, `b001d0c`, `11b9d18`, `b38c369`, `9642952`. _Archived 2026-09-09._ - `[2026-08-23]` **Worldtree b187 shipped; all three instances de-armed from a 69-day-stale `:latest`; Matrix homeserver re-plumbed to personal.** Includes the `:8009`-is-demo port trap that an IP-only fix would have walked into. # [2026-08-23] Worldtree b187 shipped; all three instances de-armed from a 69-day-stale `:latest`; Matrix homeserver re-plumbed ## b187 pre-stage (#405 phases 1+2) The matrix bridge stopped embedding the engine and became an HTTP client of the Conversation API, so `WORLDTREE_API_URL` became **boot-blocking** — absent from the container env, the bridge exits by design. Demo's compose never passed it; the next recreate would have crash-looped. Pre-staged on demo and personal (additive, backed up, verified with `docker compose config`, nothing restarted). **Key decision, and I got its scope wrong first.** I argued demo should stay keyless (no homeserver -> no rooms -> no turns -> no 401s). Right about turns, **wrong about scope**: the engine preflight authenticates at boot regardless of homeserver, so demo booted permanently degraded. Corrected — key `341c1488` minted under worldtree-dev's recorded authorization, vaulted, wired, three-hop hash-verified. ## The 69-day-stale `:latest` landmine All three instances pinned `WORLDTREE_IMAGE=.../worldtree:latest` in `.env` while running SHA-tagged images built that day. Local `:latest` = `b19afd71d7cc`, built **2026-06-14**. So ANY `docker compose up` — anyone's, for any reason — silently downgraded that service by 69 days. Same footgun as the 2026-06-15 outage. Re-pinned all three to their running SHAs (Worldtree #410), verified by rendering compose config rather than reading `.env`, containers untouched. Playbook at `playbooks/repin-worldtree-image.yaml`. **`worldtree-pinned` was the worst case:** the instance whose entire purpose is being frozen was running a **dangling image with no repo tags**, kept alive only by the running container. One `docker rm` from garbage collection. Tagged `:446e5807bf43` first, then pinned. The guard I wrote had two bugs the pinned case exposed: it compared the container's `.Config.Image` **string** (only the tag it was CREATED from — pinned was created from `:latest` back when that meant 446e5807), and it reported CHANGED unconditionally. Now compares **image IDs** and skips when already correct. ## Matrix homeserver ownership Operator ruled: **personal owns the Matrix bridge.** The appservice tokens were never missing — both sat at length 64 in the vaulted dev `env.sh` while both deployed instances had them at length **zero**. Someone wired four of six Matrix vars and stopped. Wired them into personal, three-hop verified. **The trap worth remembering:** Synapse's registration pointed at `http://10.100.10.50:8009` — nh3-dev, a dead epoch, with transaction 2801 queued at 512s backoff. The natural fix (swap the IP) gives `10.250.50.152:8009` which is **DEMO's** bridge, and Synapse can reach both — it would have connected, delivered, and looked correct while routing the operator's live rooms to the demo instance. **Personal's bridge is :8010.** `docker port` is ground truth. Corrected the URL, restarted Synapse (healthy in 32s after 3.5 months up), verified `GET /_matrix/app/v1/ping -> 200` from inside the Synapse container. worldtree-dev's smoke passed first try: room created, mimir accepted the invite, a real engine turn ran, mimir replied in persona voice. #408 closed. ## Open on worldtree-dev's side - **#411** — personal's bridge logs `Debug sink init failed: Permission denied: /app/sessions/debug_rooms.json`. It creates two debug rooms but cannot persist their IDs, so **every restart mints a fresh pair on the live homeserver**. Room litter that compounds silently. Needs a which-container-writes-what check on the sessions volume before anyone chowns it. - Bridge/engine agent-roster drift: 6 of the bridge's 9 configured agents are not listed by the engine on either instance. - Historical Domari pairwise verdicts from the selene era are coin-flip-grade (see [[2026-08-23-selene-retired-alias-collision]]); worldtree-dev banked that so no future arc leans on them without re-judging. Commits `064181a`, `bb19a96`. _Archived 2026-09-09._ - `[2026-08-23]` **Every secret-bearing `.env` on ana-docker tightened to 0600** — eight stacks including vaultwarden and traefik, verified exposed by reading one as `nobody`. # [2026-08-23] Every secret-bearing `.env` on ana-docker tightened to 0600 Found while taking uptime ownership of hrafn: its `.env` was mode 0644 with a live bearer token. Not a hrafn lapse — **0644 was the de facto pattern on the host**. Eight stacks carried secret-shaped vars in world-readable `.env` files on a box with four interactive accounts, verified as real exposure by reading one as `nobody`. Swept: **vaultwarden, traefik**, beszel, gitea-runner, miniflux, news-digest, searxng, vor. (hrafn and nevermore were fixed separately the same day.) Six other stacks already used 0600, so this converged on the existing house pattern rather than inventing one. Post-sweep the host has **zero** secret-bearing `.env` readable by `nobody`. Playbook: `playbooks/tighten-env-perms.yaml`, one run per stack, re-runnable. ## The check that matters Every run asserts `docker compose config` still renders **as the deploy user** (`lkraven`), not as root. Checking the mode proves the bits changed; only rendering as the deploy user proves the next deploy can still resolve its variables. ## Two gotchas recorded in the playbook - **vaultwarden looked like it bind-mounted its `.env`** — which would mean the *container's* UID reads it and 0600 could break the password vault. It does not: that `- .env` is under `env_file:`, not `volumes:`. My grep matched the YAML list item without checking its parent key. The playbook now **refuses** any stack that genuinely bind-mounts its `.env`, since that case is read by the container UID. - **elway prompted for a sudo password.** The `ana-docker` ssh alias resolves to `lkraven`, who needs one; **`infra-ops@10.250.50.70` has NOPASSWD**. `corviduo-dev` was repointed to infra-ops at some point and `ana-docker` was not. Run elway against the infra-ops target on this host. Commit `a896c0a`. _Archived 2026-09-09._ - `[2026-08-23]` **`pfi` gitea org created; claude-bot is an Owner and creates repos self-serve.** Closes the repo-creation half of the credential-migration directive — `vh` is a USER namespace so no service account could ever create there. Repo creation needs `write:user` + `write:repository` + `write:organization`; `POST /users/{u}/tokens` is basic-auth only, so minting needs the account password. Default new repos to `pfi/`. (`vh/eitri-smithy` was its first tenant, then moved.) _Archived 2026-09-09._ - `[2026-08-23]` **Booth: kept boards are deletable and link rows are prunable.** `release` on a kept card drops the sentinel so the existing × applies; `booth links` / `booth unlink ` prune one row. Rows are addressed by **content id, never position** — the board is append-only and multi-writer. **Releasing a board RESETS its TTL clock** (unlink bumps the dir mtime), so unkeep-and-wait is a 24h delay, not a delete. (`4be880f`, `0ad332b`) _Archived 2026-09-09._ - `[2026-08-22]` **DFlash2 spec-decode measured on our own stack; `sec` promoted to it.** +18–21% accepted length and +15–18% throughput over MTP k=3, drafter proved model-agnostic across two finetunes to 0.06%, and the k=7 MTP *control* showed deeper MTP is a throughput trap. # DFlash2 speculative decoding — measured on our own stack (2026-08-22) Operator-driven session. **Read the epistemic labels.** During the chase we generalised from observations that later proved wrong; this file separates what was *measured* from what remains *hypothesis*, and records the wrong turns so nobody re-derives them. ## What DFlash2 is A **2B draft model** (3.85 GB bf16) for speculative decoding against Qwen3.8-27B — `incoai/Qwen3.8-27B-DFlash2`, Apache-2.0, blog `inco.ai/blog/dflash2`, upstream `z-lab/dflash`. Block diffusion: drafts a whole 8-token block in one pass, with a candidate selector tracing a path through per-slot top-K. Lossless (greedy matches the target). vLLM support merged **2026-08-21 05:27 UTC** as PR **#52816** (`b389ac29`). Method string is **`"dflash"`**, not `dflash2`. ## ✅ MEASURED — throughput and acceptance Single instrument (`specbench.py`, 8 fixed prompts, temp 0, max_tokens 256), delta against vLLM's own `spec_decode` counters. The MTP k=3 numbers reproduce our recorded 58.4% / 55.3% figures exactly, which is what validates the instrument. | seat | config | accepted tok/forward | throughput | |---|---|---|---| | gen (orcarouter) | MTP k=3 *(production)* | 2.753 | 114.9 tok/s | | gen | MTP k=7 *(control)* | 3.041 | **74.0 tok/s** | | gen | **DFlash2 k=7** | **3.254** | **131.9 tok/s** | | sec (M.O.G.-SEC) | MTP k=3 *(production)* | 2.676 | 110.5 tok/s | | sec | **DFlash2 k=7** | **3.252** | **130.0 tok/s** | **⭐ The k=7 MTP control was essential and inverted the obvious read.** Going deeper on MTP *improves acceptance* (2.753 → 3.041) while **destroying throughput** (114.9 → 74.0). Our MTP head is a single module (`mtp_num_hidden_layers=1`, only `mtp.layers.0`, 15 tensors) run autoregressively, so k draft tokens cost k sequential forward passes. **"Just raise num_speculative_tokens" is a trap** — without the control I would have recommended it. DFlash2's win is therefore **not better per-token acceptance** — our MTP is actually *better* at position 0 (79.6% vs 75.4%). It is that block drafting makes depth nearly free. **⭐ The drafter is model-agnostic across finetunes — 3.254 (gen) vs 3.252 (sec), a 0.06% difference**, with superimposable per-position curves. One drafter file on `/tank` serves both. ## ✅ MEASURED — how DFlash2 runs (answers "can one drafter serve both seats?") **EAGLE3-style coupled, not standalone.** In vLLM: `load_model(self, target_model)` binds it to a specific target object; `pass_hidden_states_to_model=True`; `gpu_model_runner` reads `dflash_config.target_layer_ids` → `[i+1 …]` to register auxiliary hidden-state capture on the target at layers **5, 19, 33, 47, 61**. It even reads the target's RoPE style at load. Consequences: - **Weights file is shareable** (one download, both seats mount it) — gen and sec are architecturally identical on every dimension the drafter needs: 64 layers (deepest tap 61), hidden 5120, intermediate 17408, vocab 248,320 > mask token 248,070. - **VRAM is NOT shareable — 3.85 GB per seat.** The drafter lives inside the target's engine process, consuming hidden states mid-forward. Two seats are two processes; there is no cross-process sharing mechanism and there could not be. ## ✅ MEASURED — it works on our stack, which the card does not claim The card tests stock BF16 on an H200 with FlashAttention 3. Verified here instead: **abliterated + NVFP4 `compressed-tensors` target ✓, Blackwell sm_120 ✓, DFlash2 CUDA graphs captured ✓.** None of that was documented anywhere. ## 🔶 HYPOTHESIS — why our acceptance trails the published numbers Both our targets land at ~3.25 accepted length against the card's 4.10–5.46 on stock BF16. **Finetune drift is ruled out** — two *different* finetunes gave identical results to three decimals. The shared variable is **NVFP4 quantization of the target**, which is mechanically plausible (the drafter reads quantized hidden states at its five taps). Second candidate: prompt distribution (ours general-purpose, theirs GSM8K/MATH/HumanEval/MBPP/MT-Bench). **Neither is confirmed.** Settling it needs a BF16 target seat (~56 GB) — a real GPU window. ## ❌ RETRACTED — the "MTP head mismatch causes the degeneration" hypothesis **Operator ruling, 2026-08-22: this hypothesis is WRONG. The degeneration lives in the un-fixed vLLM, not in the weights.** Recorded here rather than deleted, because it was reasoned to confidently enough that a future session could re-derive it. **Two independent failures produced it, and the second is the instructive one:** 1. **I treated a false dichotomy as a deduction.** Having verified gen and sec run an identical engine (same image ID `sha256:bd3236cff208…`, same live version `0.27.2rc1.dev150+g311b3513a` read from inside both processes, same flags bar `gpu-memory-utilization` 0.43 vs 0.44), I concluded "config is eliminated, therefore it is the weights." That does not follow. **An engine bug present in BOTH seats is not exonerated by the two seats being identical** — it just means the engine cannot explain a *difference*. It can still explain the *failure*. 2. **The difference I was explaining may not exist.** The premise was a single operator observation of sec degenerating at ~2k, made during a session with many concurrent changes. **n=1 under heavy concurrent modification is not evidence** — see the meta-lesson below. **What survives as fact** (measured, still true, just not causal): sec's MTP head *is* byte-identical to `qwen38-27b-uncensored-bf16` across all 15 tensors — a stock head on a security-finetuned body, because the `Qwen3_5ForConditionalGeneration` wrapper never loads the head, so the finetuning could not reach it. gen's orcarouter head *was* abliterated in-band by its author. Acceptance differs slightly (gen 58.4%, sec 55.9%). **All true. None of it shown to cause multi-turn degeneration.** **Current standing explanation: the degeneration is an engine bug in the un-fixed vLLM.** Both production seats run `311b3513`, which is **172 commits behind GDN spec-decode fix #53077** (merged 2026-08-20). `#51113` is present in that build and is therefore **necessary but insufficient** on its own. ## ⭐⭐ META-LESSON — n=1 during a busy session is not evidence The operator's own framing, and it generalises past this incident: **an observation made while many things are being changed at once cannot carry a causal claim, no matter how confidently it is reported.** Tonight that single observation became the load-bearing premise for a weights-side hypothesis, a root-cause narrative, and very nearly a recommendation. This is the same failure the gen-seat compose file already warns about in different words — *"a passing probe is NOT sufficient evidence"* — inverted. That note guards against trusting a **negative** result from a synthetic test. This one guards against trusting a **positive** sighting from an uncontrolled session. Both reduce to: **hold the system still, or do not draw causal conclusions from it.** Applies equally to the "coherent to 10k" observation below — same n, same conditions, opposite direction. Neither observation is worth more than the other. ## ⚠️ CONFOUNDED — and the "before" state is itself unreliable sec now runs DFlash2 on a newer build and the operator reports **coherent to 10k tokens with adversarial nonsense prompts**. ⚠ Treat this the same way as the 2k sighting it is being compared against: **n=1, uncontrolled session, not evidence.** The comparison is weak on *both* ends. **Two variables changed at once:** 1. **Engine**: `311b3513` → `e9d1398d`, **+259 commits, `behind_by=0`** (a strict superset), including GDN spec-decode fix **#53077** (merged 2026-08-20) that production is **172 commits behind**. 2. **Drafter**: frozen MTP head → DFlash2 reading live hidden states. **Isolating it = run MTP k=3 on the same new build.** Not yet done. **#51113 is present in BOTH builds** (verified by ancestry, `behind_by=0` each) — so the "proper upstream fix" our compose comment credits is **necessary but insufficient**; sec ran it and still degenerated. Related open upstream: **#53180** (quantized Qwen3.8-27B hybrid GDN + MTP producing *silent* degenerate output, no fix), **#41884** (DFlash + prefix caching on hybrid, IndexError, workaround is disabling one). ## ❌ WRONG TURNS — do not repeat - **Version strings are not lineage.** The DFlash2 build reports `0.26.1rc1.dev1048` and our production nightly `0.27.2rc1.dev150`, which *looks* like a regression. It is a setuptools_scm tag-reachability artifact. **Use the GitHub compare API and check `behind_by`.** - **Docker Hub push timestamps lie about source freshness.** `nightly-ba07e4a4` was *pushed* 06:12 UTC, comfortably after the 05:27 merge — but *cut* from a 03:46 commit that predates it. **Grep the image for the symbols you need.** Believing the timestamp would have cost an RP-seat outage to serve a model the engine could not instantiate. - **`--max-num-batched-tokens` was not the image truncation.** Raising it 16,384 → 32,768 on that theory changed nothing and cost ~3 GiB of peak activation, which came straight out of the KV pool. The cap was the tokenizer (§3.14 of the playbook). - **"1M needs YaRN, absent from config" is FALSE for the sec quant.** It is fully present: `rope_type: yarn`, `factor: 4.0`, `original_max_position_embeddings: 262144`, `max_position_embeddings: 1000000`. Context is a KV-memory choice, not a model limit. ## Live state — PROMOTED to the compose stack 2026-08-22 **Operator-approved after real-use testing** ("performing very well"). The experimental standalone container is gone; `stacks/mog-sec/` is canonical and `restart: unless-stopped` means it survives reboots. Cutover verified: **KV pool 526,617 / 1.10x — identical to the container it replaced**, restarts 0, both gateway aliases serving, DFlash2 confirmed drafting at k=7 (231 draft tokens over 33 drafts), vision working. ⚠ **One variable was deliberately REMOVED, not carried over.** The old stack hardcoded `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`; the validated DFlash2 container never set it, and playbook §3.10 records expandable_segments corrupting retained tensors elsewhere. The compose now defaults it EMPTY (`MOG_ALLOC_CONF`). Promoting it as-was would have shipped a variable the tested configuration did not have. **Compose is now parameterised for the shapes that differ:** `MOG_SPEC_CONFIG` carries the whole speculative JSON (dflash needs `"model": "/drafter"`, MTP must not have one — a method+tokens template cannot express both), plus `MOG_MM_PROCESSOR_KWARGS`, `MOG_DRAFT_MODEL`, `MOG_MAX_NUM_BATCHED_TOKENS`, `MOG_ALLOC_CONF`. **ROLLBACK:** `.env.bak-pre-dflash2-20260822` and `compose.yaml.bak-pre-dflash2-20260822` on the host; or one line — `MOG_SPEC_CONFIG={"method": "qwen3_5_mtp", "num_speculative_tokens": 3}` plus the old `MOG_IMAGE`. | | production sec | current | |---|---|---| | image | `nightly-311b3513` | `nightly-e9d1398d` | | speculation | MTP k=3 | **DFlash2 k=7**, drafter `/tank/aimodels/qwen38-27b-dflash2-drafter` | | max-model-len | 262,144 | **480,000** | | KV pool | 418,218 (1.60×) | **526,617 (1.10×)** | | images | 4096² → 16,384 tok | **2048² → ~5,125 tok** (`--mm-processor-kwargs` size cap) | ⚠ **`--gpu-memory-utilization 0.55` is the stable ceiling** while GPU1's other tenants are up. 0.58 sized KV at 594,172 then **OOM'd during CUDA graph capture** — the process reached 57.49 GiB against ~57.6 free. Real 1M context needs ~49 GiB of KV and therefore evicting most of GPU1. Canonical config: `stacks/mog-sec/{compose.yaml,.env.example}` in this repo. _Archived 2026-09-09._ - `[2026-08-22]` **Quant pipeline shipped a crippled tokenizer for months — fixed at source.** `quant_mixed_nvfp4.py` baked its calibration truncation (`max_length 2048`) into every mixed-NVFP4 build; latent on old transformers, fatal on new. Both live quants corrected, pipeline now saves a source-pristine tokenizer and asserts it. Playbook §3.14. (`0755ba7`) _Archived 2026-09-09._ - `[2026-08-22]` **`sec` retuned to util 0.52 / 420K after a runtime OOM at 0.55/480K** — `gpu-memory-utilization` is not a hard reservation; activation grows past the dummy-data profile and six vLLM containers share GPU1. Also measured: the KV pool varies ~6.6% between boots, so max-model-len must be sized against the *lower* observation. (`6e82899`) _Archived 2026-09-09._ - `[2026-08-22]` **Max-Q 1.8× spread does NOT apply to LLM decode — measured, not argued.** ana-ml2 draws 256–266 W of 300 W under sustained 100% decode with `SW Power Cap: Not Active` and clocks pinned. Corrected to brokkr-smithy-dev after I had lent the claim credibility; 122B figure (~90–93 tok/s at 262K) stands as a straight number. _Archived 2026-09-09._ - `[2026-08-21]` **ESH internal IPv6 live on two LANs; the Cityside v4 static is a CARRIER problem, proven.** A full gateway reboot forced a fresh DHCP DISCOVER and returned the identical CGNAT address. YaRN was already configured — "1M needs YaRN, absent" was false. → `persistent-memory.d/2026-08-22-dflash2-spec-decode.md` sibling entry in `ad21302` _Archived 2026-09-09._ ## The finding `esh-pve-nas` (`esh-nas-pve.esteban.net`, 10.0.50.55) runs PVE root off a **USB Disk-on-Module** — `sdq`, 7.3 GB, `ID_BUS=usb`, `ID_VENDOR=NORELSYS`, model 1081 — carved into a 512 MB ESP + 768 MB swap + a **6 GB ext4 root** that was at **90% (571 MB free)**. ⚠ **Operator corrected my first read: it is a DOM, not a thumb drive.** DOMs use SLC/pSLC with a real controller, so the **284 GB written since boot is unremarkable and wear is NOT the driver**. I had framed it as a clock ticking; that was wrong and the correction matters. What actually justifies the work: 1. **It is on the USB bus** — a reset or re-enumeration drops the *root filesystem* out from under a running hypervisor whose guests keep executing. NAND quality is irrelevant to that. 2. **6 GB has no headroom** — `/usr` alone is 3.7 GB. 3. **Unmirrored**, while 928 GB of mirrored NVMe sits 96% empty. 4. **It has blocked patching for months** — the operator-visible symptom and the real urgency. ## The patching blockage (measured) `apt-get -s dist-upgrade`: **225 packages pending, 161 carrying `deb12uN` / Debian-Security bumps** including `ssh 1:9.2p1-2+deb12u10`. Host sits on `pve-manager/8.4.11` vs sibling esh-pve's **8.4.14**, with 20 weeks uptime because it cannot take a kernel. ⚠ **Ordering is load-bearing: migrate FIRST, patch after.** The pending set includes `proxmox-kernel-6.8.12-42-pve-signed` — ~250 MB of kernel + initramfs landing in `/boot`, **which is on root**. Unpacking 225 packages (dpkg, perl, glibc-adjacent) into 1.3 GB of headroom risks filling the disk mid-transaction and wedging dpkg on a hypervisor running five guests. Partial escape hatch if patching truly cannot wait: `apt-get -o Dir::Cache::Archives=/nvme/tmp/apt-archives` keeps downloads off root, but the kernel still lands in `/boot`. ## Mitigation applied 2026-08-17 — root 90% → 76% | step | effect | |---|---| | capped journald (`SystemMaxUse=64M`; was **fully default/uncapped**) | stops unbounded growth | | vacuumed the journal | **freed 446 MB** | | `apt-get clean` | 79 MB | | `/root/neo` (2024 Intel NEO OpenCL debs) → `/nvme/tmp/root-neo-20260817/` | 259 MB — **moved, not deleted** | | **`/var/log/journal` relocated onto ZFS** (`nvme/varlog`) | dominant writer off the DOM | 571 MB → **1.4 GB free**. All five guests stayed up; a fresh `logger` round-tripped through the ZFS-backed journal. ⚠ **Stopping journald over SSH kills your own session** — it takes the connection's logging path with it. The first attempt died mid-swap, leaving the dataset staged and the move incomplete (host was never at risk; journald socket-activated straight back). Redo as a detached `systemd-run` transient unit. Script + reason live at `root@10.0.50.55:/root/move-journal-to-zfs.sh`. Deliberately **not** done: moving `/var/lib/rrdcached`. With the DOM correction the wear argument no longer justifies touching a service `pvestatd` depends on. ## The plan — split boot from root (operator's proposal, strictly better) My first plan was a full reinstall to a mirrored-NVMe ZFS root. **The operator proposed keeping boot on the DOM with a fallback image and putting all its files on ZFS. That is better and I should have gotten there myself** — I had assumed boot and root must share a device. | | device | contents | written when | |---|---|---|---| | boot | DOM `sdq` | ESP + `/boot` (ext4) | only on kernel/GRUB updates | | root | `nvme` pool | `nvme/ROOT/pve-1` | constantly, on mirrored NVMe | Keeping `/boot` on **ext4** is the point, not a compromise: GRUB never has to read ZFS, which matters because the `nvme` pool has `encryption`, `large_dnode` and `zstd_compress` enabled and **GRUB cannot read those**. **Why it beats the reinstall:** the `nvme` pool survives (no guest migration, no `ssd`/`tank` export-import, no reinstall); downtime is **one reboot** not half a day; **rollback is a GRUB menu entry** because the ext4 root stays untouched on the DOM; and it retires the actual top risk — with root on NVMe a USB bus reset mid-run no longer kills the running system. Free upside: boot environments (`zfs snapshot nvme/ROOT/pve-1@pre-upgrade`). **Preconditions verified already met:** UEFI + `grub-efi-amd64 2.06-13+pmx7`; **`zfs-initramfs 2.2.8-pve1` already installed with 76 ZFS files in the running initrd**; root only 4.3 GB to copy; swap 767 MB / 123 MB used against 125 GB RAM (leave it on the DOM LV — **never** swap on a zvol). **Two traps:** `canmount=noauto` on the root dataset or ZFS mounts over the live root; and `cachefile` is `none` with a **0-byte `/etc/zfs/zpool.cache`** — pools import by scan today, which is a coin-flip when the initramfs must find root. Set the cachefile before rebuilding the initramfs. Operator ruled a **cloned DOM image is sufficient** boot-path insurance (no mirrored boot needed). `dd` it off-box before anything else; refresh after kernel updates. ## ⚠ Blast radius — the gating constraint, invisible from the host itself **CT 103 `esh-nas` (10.0.50.50) IS the NAS, and it runs on this host.** Two dependents mount it over **`hard`** NFS — they do not fail, they hang unkillably: - **esh-docker-vm** (10.0.50.45): `/mnt/books`, `/mnt/backup` - **esh-pve** (10.0.250.35): `/mnt/pve/esh-nas`, `/mnt/pve/tank-vmbu` Known incident shape — the only remedy for esh-docker-vm's D-state is a host reboot, and `/mnt/books` was *deliberately* left `hard` because calibre's SQLite risks corruption under `soft`. Quiesce both before any reboot of this host. Recorded in `servers/esh-pve-nas/README.md` as a never-reboot-casually warning. ## Also identified - **`esh-nas` is CT 103** on esh-pve-nas — structurally the same shape as ana-nas being CT 109 on pfi-pve. - **`ESH-FileBot` (CT 106, 10.0.50.70) is an empty shell** — 80 GB rootfs, six passthrough mounts (`books`/`documents`/`music`/`share`/`pvestore`/`ssd-pvestore`), and **nothing running but base systemd, sshd, cron, postfix** since 30 March. That resolves the dashboard's long-standing "role TBC". Retire rather than migrate. - Both ESH hypervisors have **20 weeks uptime** and differing PVE patch levels. ## Staging executed 2026-08-18 — everything but the reboot Two rerunnable elway playbooks, 0 failed steps, 17/17 verify green: `playbooks/esh-pve-nas-stage-zfs-root.yaml` (LV surgery, `/boot` populate, 4.3 GB root rsync in 228 s, fstab) and `playbooks/esh-pve-nas-stage-bootloader.yaml` (ZFS initramfs, grub.cfg, both menu entries, grubenv). **`grub-install` is deliberately NOT run.** The ESP stub still points at the old `/boot` inside the ext4 root, so the host's boot path is byte-identical to the last 140 days and an unplanned reboot mid-staging is a non-event. Cutover is `grub-install` + `grub-reboot pve-zfs-root` + `zfs set mountpoint=/` + reboot. Final DOM layout: `pve-root` 6.04 G (untouched, the rollback) + `pve-boot` 512 M (new) + `pve-swap` 256 M (was 768 M). ### The three landmines staging found 1. **The `/boot` LV had nowhere to live.** VG `pve` had **4 MB free**, and mounted ext4 cannot shrink — freeing space from root needs a rescue boot, which costs the "one reboot" property the design rests on. Only live source was the swap LV. Operator chose shrink-to-256M over drop-entirely. 2. **The one-pool cachefile would have broken the NAS.** `zpool set cachefile=… nvme` looks scoped and safe; it is the opposite. Populating a cachefile flips the host from `zfs-import-scan` to `zfs-import-cache` (verified: scan active, cache inactive beforehand), so a cache holding only `nvme` leaves `ssd` and `tank` unimported at boot — and CT 103 has twelve bind mounts spanning all three pools. Every export would come up empty and both `hard` NFS clients would hang. 3. **`update-grub` silently emitted a pool-less `root=ZFS=/ROOT/pve-1`.** Debian's `10_linux` builds `${rpool}${bootfs}`; `rpool` comes from `grub-probe --target=fs_label`, which returns empty because GRUB's ZFS reader cannot open a pool with `encryption`/`large_dnode`/`zstd_compress` — and the probe failure is swallowed by `2>/dev/null || true`. The same feature set that forced `/boot` to stay ext4 also corrupts the kernel command line, which the design did not anticipate. Fixed with a `/etc/default/grub.d/zfs-root.cfg` drop-in (last `root=` wins) plus explicit `pve-zfs-root` and `pve-ext4-rollback` entries carrying stable ids — the auto-generated ids are derived from pool member device paths and would shift if the mirror changed. **The transferable lesson from (3):** the original verify grepped for `root=ZFS=nvme/ROOT/pve-1` *appearing somewhere* in grub.cfg. Once the drop-in was added that grep passes — while pool-less entries sit in the menu untouched. The check that holds walks every `linux` line, takes the **last** `root=`, and asserts it against a known-good set. **Assert the effective value, not the presence of a substring.** ### One-shot boot, not a new default `GRUB_DEFAULT=saved` with grubenv pinned to `pve-ext4-rollback`, and cutover uses `grub-reboot pve-zfs-root` so ZFS is tried **exactly once**. A failed ZFS boot returns to ext4 by itself on the next reboot — no console, no hands. That matters more here than on a normal host: a hang at an initramfs prompt takes CT 103 down and the NFS clients hang rather than fail. Only after a second clean ZFS boot should the saved default move. ### Off-box artifacts (`nh3-dev:~/backups/esh-pve-nas/`) - `dom-sdq-20260818.img.zst` — full DOM image, 7,837,450,240 B raw / 2.38 GiB compressed, zstd XXH64 verified. ⚠ **Crash-consistent, not clean** — the root LV was live during the read, so a restore replays the ext4 journal. Not fixable with an LVM snapshot: the VG has no free extents. - `bootchain-20260818.tar.gz` — clean, consistent tar of `/boot` + ESP (88 MB, 644 entries, full proxmox shim/grub EFI chain). This is the higher-quality boot-chain artifact; the dd image is the belt-and-braces full-device restore. - `pve-config-snapshot-20260818T051*.tar.gz` — 147 entries incl. the new grub.cfg, fstab, LVM/ZFS/blkid state. ⚠ Building this the first time produced a **corrupt archive**: `pvs; vgs; lvs > file` redirects only the last command, so `pvs`/`vgs` output leaked into the tar stream on stdout. Group with `{ …; } > file`. Runbook: `docs/runbooks/esh-pve-nas-boot-migration.md`. Earlier config snapshot at `nh3-dev:~/backups/esh-pve-nas/pve-config-snapshot-20260818T043027Z.tar.gz` (0600, sha256 `dc312793d027dc43…`) — `/etc/pve`, network, fstab, apt, authorized_keys plus captured `zpool`/`zfs`/`disk-by-id`/`lsblk`-with-serials/`pvesm`/`dpkg` state and every guest config. **The newest on-disk copy before this was June 2024.** Commits `2275e11`, `3e31175`, `8ddc87c`. _Archived 2026-09-02._ # irv-ml1 weight cleanup (782 GB) + Homepage brought under version control Two unrelated housekeeping jobs from the same session, both with durable lessons. ## irv-ml1 — 782 GB reclaimed Root was at **92%** (148 G free), storetank **86%**. Now **64%** (635 G free) and **74%** (477 G). **Tier 1 — dead weights, 286 GB.** `/storetank/llm-models/Storage` (**217 G**, 22 GGUF repos, atimes Jan–May **2025**) plus `models--MaziyarPanahi--WizardLM-2-8x22B-GGUF` (44 G) and `models--h2oai--h2ogpt-4096-llama2-13b-chat` (25 G). The 217 G pile had **zero consumers** — no llama-swap, no llama.cpp, no textgen running *or installed*, not even a stopped container. The fleet moved to vLLM/NVFP4 seats on ana-ml2 and nobody opened that shed for 15 months. Re-verified the consumer check immediately before deleting, not just during the audit. **Tier 2 — regenerable caches, 194 GB.** `uv` 65 G + 60 G, `pip` 31 G + 8.7 G, `modelscope` 29 G (mtime **2024-04-23**). **Tier 3 — retired stacks, 302 GB** (operator: "those were old days… we're a UV fleet now"): `/opt/fluxgym` 64 G, `/opt/ComfyUI` **native** 41 G, `/opt/stablediffusion` 28 G, `/opt/alltalk` 19 G, `/opt/o-textgen` 12 G, `/opt/sdnext` 3 G, `/opt/xttsv2` 1.8 G, `tabbyAPI` 3.1 G, **`miniconda3` 130 G**. ### The lesson: one dead-looking app pinned three delete targets `lsof +D` per path found **PID 281192 — fluxgym, up 42 days, listening on 0.0.0.0:7860** — holding 15 open handles into `miniconda3/envs/vllm` (stale opencv wheels) **and 41 into `/opt/ComfyUI`**. Deleting miniconda underneath it would have half-broken a live listener in a way that surfaces only at its next restart. Stopped it by **explicit PID** (never `pkill -f` — handle-blind), verified :7860 released and handles at zero, *then* deleted. ⚠ **Name collision that nearly cost a production service:** `/opt/ComfyUI` is a *native* install; the ComfyUI that actually serves (:8188, 200 OK) is the **Docker `mmartial` container** reading `/worktank/comfyui`, and arbo's `comfy_engine` runs from uv. Checking open handles **per path** is what separated them — the earlier "not running" read would have deleted the wrong thing. ⚠ **`df` lags an async ZFS free.** Right after the 217 G delete, storetank still showed 86%/261 G — the exact shape of a snapshot-retention problem. It wasn't (`zfs list -t snapshot` empty); second check showed 477 G at 74%. All 16 containers and both systemd services verified healthy afterward. ## Homepage under version control `ghcr.io/gethomepage/homepage` on **esh-docker-vm:5100** was the one stack whose config lived only on the host. Its version history was **six hand-rolled `services.yaml.bak-*` files**. Now `stacks/homepage/` (compose + 9 config files + `.env.example` + README), deployed via `deploy-stack.sh`; `.bak` files gone. 105 cards across 19 groups, no empty groups. ⚠ **I claimed ana-docker wasn't wired into `docker.yaml`. It already was** — `ana-pfi-docker: 10.250.50.70` — and I built a theory on a `tail` that truncated the top of the file. All five engines were discovering correctly the whole time. **Corrections landed:** `ANA-Firewall` said "Fortigate 81F" → it is a **FortiGate-80F, FortiOS 7.2.10** (verified against the device); `NH3-Ansible` → **NH3-ExtDev** (10.100.50.42 is nh3-extdev, successor to the retired nh3-ansible); dropped the `UltraSeedbox` layout group (nothing provides it). ⚠ **`HOMEPAGE_ALLOWED_HOSTS` matches host AND port.** `10.0.50.45` did **not** cover `http://10.0.50.45:5100/` — the container log carried `Host validation failed` while the Traefik hostnames worked. Fixed; direct IP:port now 200. `.env` was **mode 644** holding Plex + Jellyfin API keys → now 600. ⚠ **Homepage renders client-side** — grepping the served HTML to verify a config change gave two false readings (a stale prerender, then an empty page). `GET /api/services` is the honest instrument, and config changes need a **recreate**, not a restart (a restart keeps the cached render in the writable layer). ⚠ `deploy-stack.sh` runs rsync with `--delete` — alongside the six `.bak` files it also removed a host-side `README.md` in the conf dir. Content survived (it is now in the repo README) but that was a side effect, not a plan. Commits `c5beeac`, `d1f4f1c`. See also [[2026-08-17-fleet-ipv6-mesh]]. _Archived 2026-09-02._ - `[2026-08-17]` **Gen seat swapped to `absolute-heresy` — and the three bugs the swap exposed are worth more than the swap.** Candidate `MuXodious/Qwen3.8-27B-absolute-heresy` (Heretic v1.4.0 + SOMPOA, T377) beat the incumbent on refusals AND KL simultaneously, which is the unusual part — those normally trade off. Validated on the probe port per operator ruling, promoted, all 7 aliases green. **Durable lessons banked:** (1) **A CPU-only MTP head hash can replace the ~56 GB bf16 acceptance gate.** The `Qwen3_5ForConditionalGeneration` wrapper never loads the MTP head, so PEFT merges / Heretic runs / llm-compressor passes all leave `mtp.*` pristine — hashing it against a head we have already measured (the incumbent's, 47.7%) answers the question for free. Predicted 47.7%, measured 47.2%. Saved downing meromero. Tool: `services/gen-seat-mixed-quant/compare_mtp_head.py` (hash bf16 via **uint8 reinterpret** — numpy has no bfloat16). (2) **`post_quant.py` assumed a standalone `model-mtp.safetensors`**; a full checkpoint keeps `mtp.*` in a NUMBERED shard, so the copy silently no-op'd while the index was still rewritten to point at a file that never existed — 15 unresolvable tensors behind a correct-looking tensor count. Its own FAILED-CHECKS assertion caught it; **that is why the check exists rather than an assumption**. Fixed to extract. (3) **A probe that does not mirror the live seat manufactures failures.** `serve_probe.sh` hardcoded `:latest` (seat is a pinned nightly for #51113), had no tool-call/reasoning parsers, and its `--speculative-config` JSON died twice on quoting — **bash BRACE-EXPANDS `{"a":1,"b":2}` on the comma** unless single-quoted at the REMOTE shell. Adding the seat's flags took the surface test from 5/6 to **6/6**; the "tool calling broken" result was pure probe config. Commits `7997f11`,`254c588`,`2c36028`,`b0c2d3d`,`993421b`. _Archived 2026-09-02._ # Fleet IPv6 state + the real VPN topology (verified 2026-08-17) Written because the operator expects to reference this "before too long" — the driver is an **ESH fiber install landing 2026-08-18 that puts the house behind CGNAT**, which breaks Site Magic on IPv4 and makes IPv6 load-bearing rather than a nice-to-have. ## Why IPv6 suddenly matters: CGNAT at ESH New ESH fiber (installing 2026-08-18) hands out a **CGNAT IPv4**. Site Magic — the UniFi-to-UniFi SD-WAN mesh tunnel that currently links NH3 ↔ ESH — needs a reachable endpoint, and a CGNAT address is not one. **IPv6 is the escape hatch: a global v6 address on each UDM restores a routable endpoint pair without depending on the ISP's v4 at all.** That, not the WireGuard RA mesh, is the most likely first consumer of fleet IPv6. Operator expects addresses at **Anaheim shortly** and **ESH 2026-08-18**. ## The topology — as VERIFIED, not as assumed Three transports, three different technologies. Do not describe this as "a WireGuard mesh"; a prior session did and was corrected. | Link | Transport | Evidence | |---|---|---| | NH3 UDM ↔ ESH UDM | **Site Magic** (`vpn_type: sdwan-mesh-tunnel`) | UDM `networkconf`, carries all 7 ESH subnets | | Colo FortiGate ↔ NH3 UDM | **IPsec IKEv2** | FG `pfi-ana-nh3` → 70.230.226.88, **158M pkt rx / 165M tx** — the fleet workhorse | | Colo FortiGate ↔ ESH UDM | **IPsec IKEv2** | FG `ana-to-eshudm` → 70.181.90.232, 53K/56K pkt | | Remote-access VPN | **WireGuard, host-based on `ana-wg`** | see below | **WireGuard is an RA (remote-access) convention only — it is NOT the site mesh.** It runs on `ana-wg` (LXC 113, Debian 12, 10.250.50.252), interface `wg0`, **UDP 31337**, tunnel subnet `10.30.10.0/24`, 3 peers (`tc2-mac`, `vh-iphone`, `vh-mba26`). Reached from outside via a FortiGate VIP `wg-to-ana-wg`: `38.120.12.42:31337/udp → 10.250.50.252:31337` on wan1. **The FortiGate never terminates WireGuard — it port-forwards to the host that does.** FortiOS 7.2.10 has no native WireGuard (Fortinet added it in 7.4), so a session that reads "colo + WireGuard" and concludes the edge must be upgraded is chasing a non-problem. Do not re-derive this. ## Per-site IPv6 state (2026-08-17) | Site | Edge | IPv6 | |---|---|---| | **NH3** | UDM SE | **WAN live** — `2600:1700:b25:c110::48` via DHCPv6 on ATTFiber. All 5 LANs `ipv6_interface_type=none` | | **Anaheim colo** | FortiGate-80F, FortiOS 7.2.10 | **None.** `diagnose ipv6 address list` → only loopback `::1`; every physical iface `ipv6: ::/0` | | **ESH home** | UDM Pro Max | **None.** Both WANs `wan_type_v6=disabled`; link-local only | ## AT&T delegates exactly ONE /64 at NH3 — proven, not assumed `2600:1700:b25:c11f::/64`. **One.** Not the /60 the addressing pattern suggests. The proof matters because the naive read is wrong: the WAN sits at `c110::48` and the LAN got `c11f::1/64`, which looks exactly like slot 15 of a /60 spanning `c110`–`c11f`. It isn't. Forcing the prefix ID from auto to a manual `0` — which on a real /60 would relocate the LAN to `c110::1/64` — left the subnet at **`c11f::1/64`, stable across a 4-minute settle**. Two different prefix-ID settings yielding the same /64 is the signature of a single-/64 delegation. **Consequence: exactly one VLAN can have IPv6 at NH3**, unless AT&T enlarges the delegation. If Site Magic-over-v6 is the goal that is fine — Site Magic needs a routable address on the *WAN*, not a LAN prefix. The controller never exposes the PD size directly (`wan_dhcpv6_pd_size_auto:false` with no size field alongside), so the prefix-ID test is the only read-only-ish way to establish it from the API. ## What a v6 mesh actually requires (and what it does NOT) **Does NOT require prefix delegation.** PD hands addresses to LAN *clients*. Both Site Magic and WireGuard need a routable address on the router/host WAN side, plus inbound reachability. Enabling PD on a LAN is orthogonal — this was tested and then reverted. **ana-wg's WireGuard socket is ALREADY dual-stack** — `ss` shows both `0.0.0.0:31337` and `[::]:31337`. It will accept IPv6 peers with **no WireGuard reconfiguration** once (a) the host holds a routable v6 address (today: link-local `fe80::be24:11ff:fed7:e4b7` only) and (b) the FortiGate passes inbound UDP 31337 over v6 — the existing VIP is v4-only (`extip 38.120.12.42`). **NH3 UDM's own WG server is v4-pinned** — `wireguard_interface_binding_mode_ip_version: 'v4'`, one field to flip when wanted. **Inbound v6 is default-deny and that held without intervention.** The UDM runs the **zone-based** firewall (66 policies). ⚠ The legacy `rest/firewallrule` endpoint returns **0 rules** on this box — a quick check there reads as "no IPv6 rules exist," which is wrong and alarming. Use `v2/api/site/default/firewall-policies`. WAN→LAN default is `Block All Traffic` for both families with `Allow Return Traffic`; the only v6-specific allows are link-local plumbing (ND solicit/advert, RA, DHCPv6). ## The stability problem — design around it up front All three endpoints will hold **dynamic** addresses (NH3's came via DHCPv6 IA_NA, not a static assignment). A three-way mesh where every node can move is fragile; WireGuard tolerates one roaming end, not all of them. The fleet already solves this on the v4 side — IPsec peers use **hostnames** (`ana-fw.phasefinal.com`, `nh3.phasefinal.com`), not raw IPs. **Extend that to AAAA records** and dynamic prefixes stop mattering. infra-ops holds the fleet Cloudflare DNS-edit token, so this is self-serve. ## Access recipes (cost a prior session real time) - **UniFi UDMs** — `X-API-KEY` from the vault (`secret get unifi/pfi-udmse-api-key`, `unifi/esh-udmpm-api-key`) against `https:///proxy/network/…`, `curl -sk`. Classic `api/s/default/rest/networkconf` + `stat/device` carry everything here. Writes are `PUT …/rest/networkconf/<_id>` with the **full** object. - **`ana-wg` is `root@`, NOT `infra-ops@`** — the shared infra-ops key is refused (`Permission denied (publickey,password)`). `servers/ana-wg/ssh-target` says `root@10.250.50.252`; believe it. - **FortiGate** — paramiko via `uv run --with paramiko` (no sshpass on nh3-dev), password `secret get fortigate/ana-gw-infra-ops-password`. ⚠ **A fixed-duration `drain()` hangs the session**; read until the `ana-gw #` prompt and answer `--More--` with a space. Two invocations timed out at 3 min before this was fixed. ## Changes made and reverted this session - **Enabled PD on `nh3-iot` (VLAN 90)** to measure the delegation, then **REVERTED on operator instruction** — all 5 NH3 LANs are back to `ipv6_interface_type=none`, verified. Pre-change snapshots kept in the session scratchpad only (ephemeral). - **`ana-wg` WireGuard key material was world-readable** — `wg0.conf` (server private key + 2 peer PSKs), `keys/*_priv`, `keys/*_psk`, and `configs/*.conf` (client configs carry private keys) were all mode **644**. Now **600**, and `keys/` + `configs/` dirs **700**. `wg-quick@wg0` stayed active, 3 peers intact — WireGuard holds keys in kernel memory, so no restart was needed. The parent `/etc/wireguard` was already 700, which capped the real exposure to root-capable contexts inside the LXC — but the modes were still wrong. --- ## CORRECTION (recorded 2026-08-24): "AT&T delegates exactly ONE /64" is the ## per-REQUEST truth, not the total — eight /64s exist and are unclaimed The section above concludes AT&T hands out a single `/64` and that the `c110`/`c11f` pattern reading as a `/60` was a misread. **That conclusion was itself superseded later in the same session, and the correction never made it into memory** — it survived only in the session transcript, and was recovered 2026-08-24 while assessing a proposal to grab more prefixes. Reading the **BGW's own LAN statistics page** gave the whole picture: ``` BGW WAN v6 2001:506:70b2:8958::1 <- AT&T's transit prefix BGW LAN v6 2600:1700:b25:c110::/64 <- the BGW keeps this for itself Delegated 2600:1700:b25:c11f::/64 <- what the UDM got ``` **The BGW holds the `/60` and rations it**, keeping `c110`–`c117` for itself and delegating from the top down — the UDM got `c11f`, the last one. So `c118`–`c11f` are **eight delegatable /64s that genuinely exist and are yours**, sitting unclaimed. Both observations are compatible, which is why the first one looked conclusive: the prefix-ID test only carves *within* a delegation already held, so a UDM holding one `/64` cannot move it no matter what prefix-ID you set. The BGW issues **one `/64` per IA_PD request**, and **UniFi solicits exactly once**. **Consequence — the ceiling is the requester, not the carrier.** More prefixes need more IA_PD requests (multiple IAIDs, or multiple client DUIDs), which the UDM will not do. That is what makes a separate DHCPv6-PD client viable, and it is why "ask AT&T for a bigger delegation" may be aimed at the wrong party: this looks like BGW rationing rather than a provisioning-profile limit. Live state at correction time: `wan_dhcpv6_pd_size: 64`, `wan1 v6 2600:1700:b25:c110::48`, all 5 NH3 LANs still `ipv6_interface_type: none`. ### ⛔ CLOSED 2026-08-24 — operator ruling, do not re-raise The seven unclaimed `/64`s stay unclaimed. Two facts close it: - **The BGW has no IP-passthrough mode.** Operator confirmed, and we hold admin on it — so the cheap path (let the UDM take the `/60` directly and carve it natively, as it already does at ESH) does not exist here. - **The only remaining route is a multi-DUID DHCPv6 client on a VM**, which requires re-cabling to reach the BGW's DHCPv6 server, split-stack routing (UDM for v4, VM for v6), and — the actual cost — **rebuilding the whole IPv6 firewall policy in nftables on that VM**, because routing v6 around the UDM bypasses its zone firewall entirely and would leave every LAN host globally reachable. Operator's call: not worth it. **NH3 LANs stay `ipv6_interface_type: none`.** Do not re-propose on the strength of "there are seven free prefixes" — the prefixes are real, the firewall rebuild is why nobody wants them. _Archived 2026-09-02._ - `[2026-08-17]` **Gen-seat multi-day degeneration RESOLVED — two compounding real causes, not one; the meta-lesson is "a mitigation that HELPS but doesn't FIX means a second cause, not a wrong one."** vLLM `qwen3_5_mtp`×GDN bug (#51113, real, fixed by nightly) + AEON full-W4A4 being lowest-fidelity (W4A4 3.0.0 -> 3.1.1 at the v3 cutover, so the staged v2.1.2 wheel is moot._ # eRP dual-seat overhaul — MeroMero-v2 + Dark-Scarlett, NVFP4A16 @ 256K on ana-ml2 `[2026-08-12]` Replaced the two legacy char-rp seats with home-quantized NVFP4A16 vLLM seats. Operator-driven, end to end this session. ## What landed | Seat (LiteLLM alias) | Model | Role | GPU | Context | |---|---|---|---|---| | `char-rp` (:8016) | **G4-MeroMero-v2-31B** (Gemma-4) | non-thinking PROSE, **multimodal (vision)** | GPU0 | 256K @ 2.07× (util 0.52) | | `char-rp-reasoning` (:8018) | **Dark-Scarlett-v1.0-27B** (Qwen3.6) | THINKING (default) | GPU1 | 256K @ 1.62× (util 0.44) | - Both **NVFP4A16 weight-only** (llm-compressor, `compressed-tensors`), `--kv-cache-dtype fp8`. - Replace: `char-rp-gguf` (Magidonia-24B GGUF/llama.cpp, :8016) + `heretic2-charrp-reasoning` (DavidAU Qwen3.6-27B-Heretic2 modelopt NVFP4+MTP, :8018). Old stacks/containers **stopped + retained** for rollback. - Compose-ified: `stacks/meromero-charrp` + `stacks/darkscarlett-charrp-reasoning` (ana-ml2 `/opt/docker/compose/`, mirrored to eshpfi, commit **`f08b6cb`**) → survive reboot. - Research that drove picks: `docs/pfi/erp-thinking-finetunes-2026.md` (from the `gecko-65` Booth). ## Load-bearing lessons (the whole point of this file) 1. **Load via the ConditionalGeneration WRAPPER class, never `AutoModelForCausalLM`.** For a multimodal-capable base (Gemma-4, Qwen3.6), `AutoModelForCausalLM.from_pretrained` + `save_pretrained` writes a FLAT text config (`Qwen3_5TextConfig`, `model.layers.*`) that **both vLLM AND SGLang reject** (SGLang: "Qwen3_5ForCausalLM has no SGLang implementation"; vLLM wants `Qwen3_5ForConditionalGeneration`). Loading via `Qwen3_5ForConditionalGeneration` / `Gemma4ForConditionalGeneration` keeps the wrapper config they accept. **This was the DS blocker** — re-quant via the wrapper fixed it (`Dark-Scarlett-...-NVFP4A16-wrapper`). 2. **NVFP4A16 is weight-only → DATA-FREE.** llm-compressor infers `DataFreePipeline`; calibration data is unused (only matters for W4A4 activation quant). W4A16 chosen per NVIDIA's sm_120 long-context guidance (W4A4 KLD 2-4× worse past ~10k ctx). 3. **Load on CPU (`device_map=None`)** so llm-compressor onloads one layer at a time. `device_map= "auto"` packs the whole model onto the GPU and OOMs when the card isn't fully free. 4. **Both models are KV-EFFICIENT — the "dense = KV-hungry" worry was WRONG.** MeroMero (Gemma-4) uses **sliding-window attention** (most layers cache only a bounded window); DS (Qwen3.6) uses **hybrid GatedDeltaNet linear-attention** (3:1 linear:full, linear layers carry no KV). Both hit full native 256K easily. (MeroMero KV pool ~542K tokens at util 0.52.) 5. **MeroMero vision reconstruction.** The finetune ships `processor_config.json` (image_processor inline, `Gemma4ImageProcessor`) but NOT `preprocessor_config.json` — the old-format file vLLM's feature-extractor loader wants. **Even google/gemma-4-31B-it (ungated!) ships only processor_config.json.** FIX: extract the `image_processor` section → write `preprocessor_config.json` verbatim, serve WITHOUT `--language-model-only`. Verified (model correctly ID'd a red circle). Audio is config-declared but WEIGHTLESS (0 audio tensors). 6. **GPU placement.** Match the KV-heavier model to the roomier GPU. GPU0 (gen neighbor, ~54GB free) > GPU1 (utility cluster, ~45GB free). Swapped MeroMero→GPU0, DS→GPU1. Pins via compose `deploy.resources.reservations.devices`. ## Dead ends (tried + abandoned) - **DS via llm-compressor `AutoModelForCausalLM`** → flat config vLLM/SGLang reject. → wrapper class. - **DS via NVIDIA ModelOpt** → modelopt↔transformers **version deadlock**: current transformers supports `qwen3_5` but crashes modelopt's sparse-moe plugin (`issubclass()` on a non-class); modelopt 0.43.0 pulls an old transformers that can't load `qwen3_5` at all. Abandoned. - **DS via SGLang** → `Qwen3_5ForCausalLM has no SGLang implementation`. Abandoned, but it REVEALED that both engines need the wrapper (→ the fix in lesson 1). - **`device_map="auto"` for the quant** → CUDA OOM in the weight observer. → `device_map=None`. ## granite retired + gateway repoint - `vllm-granite` (granite-4.1-8b, fleet summarizer, GPU1) **`docker stop`ped** (reversible) to reclaim ~13.6GB GPU1 for RP context. - LiteLLM (`ana-docker:/opt/docker/conf/litellm/config.yaml`, backed up `.bak-pre-granite-down-*`): **`granite-4.1-8b` alias RETIRED** — commented out, now 404s cleanly (the `*` wildcard→llama-swap was decommissioned 2026-06-20, so no fallthrough). **`summarizer` + `classifier` REPOINTED to gen** (`hosted_vllm/qwen3.6-35b-a3b-heretic` @ :8015, `enable_thinking:false`) — both verified. ⚠ This LiteLLM change is **server-only / not version-controlled** (a follow-up). ## MTP — deferred DS's MTP heads were dropped by the CausalLM loader; **deferred, not restored** (spec-decode is net-negative at RP temps: ~38-52% accept at temp 0.8-1.25, below vLLM's 0.5 cutoff). The splice-back path (`splice_mtp.py` in the heretic2 work dir) exists if ever wanted. MeroMero (Gemma-4) has no MTP by architecture. ## On-disk / where things live - Quant pipelines: `ana-ml2:/tank/aimodels/meromero-v2-nvfp4-work/` + `/tank/aimodels/darkscarlett-nvfp4-work/` (scripts, BF16 source, NVFP4 outputs). - Compose stacks: `ana-ml2:/opt/docker/compose/{meromero-charrp,darkscarlett-charrp-reasoning}/`. - Gateway aliases (unchanged, port-based): `char-rp`→:8016, `char-rp-reasoning`→:8018. (char-rp was also fixed from the stale `magidonia-24b-v4.3` backend model name → `char-rp`.) ## Open follow-ups 1. LiteLLM granite/repoint change NOT version-controlled (server + backup only). 2. eshpfi unpushed (many commits this session incl. `f08b6cb`, `7bd7375`, `398b58a`). 3. MTP deferred (see above). 4. DS thinks verbosely (~13:1 reasoning:content) — eval item; consumers need generous `max_tokens`. 5. MeroMero full 256K needs util 0.55 (GPU0 ~1.8GB free, tight); ran at 0.52 for headroom (~4.6GB). _Archived 2026-08-27._ - `[2026-08-12]` **infra-ops now holds an all-zones Cloudflare DNS-edit token (vaulted) + wgtunnel Phase-0 DNS landed.** Operator handed over a `Zone·DNS·Edit` (all zones) CF token → `secret put nh3-dev/.config/cloudflare/infra-ops-dns-token` (round-trip verified; /tmp drop shredded). Fleet DNS is now self-serve for infra-ops (⚠ HIGH blast radius — all zones). First use: created `boring.phasefinal.com` CNAME → `ana-srv1.phasefinal.com`, **DNS-only** (proxied:false), verified resolving to 38.120.12.44 on both authoritative NS (louis/wren) + 1.1.1.1 — NOT Cloudflare-proxied. Unblocks wgtunnel's wstunnel ACME cert. phasefinal.com zone id `f812ba74ed9a75cf21bbe7ce9188db50`. auto-memory `reference_infra_ops_cloudflare_dns_token`. (Earlier gap: the only prior vaulted CF token, jackdaw's, had `zone:read`+`worker:edit` but no `dns_records:edit`.) _Archived 2026-08-27._ - `[2026-08-12]` **wgtunnel stood up as its own repo (`vh/wgtunnel`, private) after a live endpoint-verification pass.** Operator directed own-repo (mirrors stonehenge-park/tts-stack). Verified off the fleet before seeding: `ana-wg` WG server = **UDP/31337** (not 51820), subnet 10.30.10.0/24, MTU 1420, active roaming peer proves the public UDP DNAT works; traefik on ana-docker **terminates TLS :443** (ACME `anaprod` http-challenge, docker+file providers, CrowdSec bouncer) → confirms the clean design (wstunnel container on `traefik-net`, Host-routed, WS→UDP to `ana-wg:31337`); edge `38.120.12.44` direct-A, `tunnel.phasefinal.com` free (⚠ must be **direct**, NOT Cloudflare-proxied like vaultwarden). Repo pre-seeded (README/CLAUDE/persistent-memory/ROADMAP + `docs/verified-infrastructure.md` = ground truth) + pushed; commit `9584d38`, Vuong-attributed. vh gitea token pulled from the vault (`secret get`), not persisted to `.git/config`. **NEXT = `/vor-plan` or `/vor` (operator's call, interactive).** Deps to line up in the plan: DNS A-record, FortiGate :443 host-routing, a new ana-wg peer for the laptop, client tooling. _Archived 2026-08-27._ `[2026-08-11]` **stonehenge-park — new fleet `/park` service repo stood up + designed.** **What.** A separate greenfield repo (`~/development/stonehenge-park`, gitea `vh/stonehenge-park`, pushed) for a self-contained `/park` service: one durable place to park any idea (repo-born OR personal), find it by search, and have it **actively resurface** (by due-date or staleness) until acted on — so parked ideas stop dying when a repo goes cold. NOT part of eshpfi; this is a pointer. **Design (via `/vor-plan`, converged + persisted to `docs/design/`):** four contract-sized units — **U1** core store+API (SQLite+FTS5, slug minting, bearer auth, REST) — the tracer, build first; **U2** scheduler+notifier (in-process; due/stale → statusline `due-count` + althing push to a dedicated **assistant channel**; keep-surfacing until promote/drop/re-snooze); **U3** `park` CLI (mirrors the `secret` CLI); **U4** browse UI. `/vor-ui` ran too (U4 brief persisted). **Locked decisions (operator):** SQLite, self-contained, ONE container, no external DB ("don't want to troubleshoot it when a database upgrade happens") — a hard `[OPS]` invariant; system-minted title-derived slugs + short ID (addressable as `park/`); active keep-surfacing resurfacing with **re-snooze as the anti-nag valve**; bearer key, LAN/WG-internal; host nh3-docker; `/park` **replaces** the global ROADMAP parking-lot discipline (deferred ideas → `/park`, `source`-tagged; ROADMAP keeps only the v1 target) as a **fast-follow after v1** incl. migrating existing lots. **Deferred (in the plan):** the althing assistant-channel handle **name** (decide at U2 contract time); staleness threshold + re-push cadence (env-tunable defaults ~30d/~daily); design U2's emit structured/consumable so a future **mission-control (Ledger→orchestrator)** can read it — park does NOT build the orchestrator. **State.** Pre-seeded for a fresh agent (CLAUDE/persistent-memory/ROADMAP/README + the design docs), committed (`294ee98`), pushed. Next build task lives in that repo: the **U1 tracer contract** under the House Code Discipline. Auto-memory candidate not yet written (repo is self-documenting). _Archived 2026-08-27._ - `[2026-08-12]` **Global `~/.claude/CLAUDE.md`: `secret`/vault tool entry + "store in AND pull from the vault" standing directive** (dotfiles `9db703b`, pushed); statusline reset-countdowns + a latent tab-collapse parse-bug fix, now tracked in the dotfiles stow tree. Dogfooded the directive: created `vh/stonehenge-park` pulling the gitea token via `secret get`. (dotfiles + global config, not eshpfi.) _Archived 2026-08-27._ - `[2026-08-11]` **TTS stack extracted to its own repo (`tts-stack`) + eshpfi stood down on TTS dev.** Operator: hand all TTS tuning/dev to a separate agent with a self-contained repo (knowledge + infra access + a live knowledge list), and move the voice corpus in. New repo `~/development/tts-stack` (commit `9ee3288`) carries: dots-tts stack (canonical intent), `voices/` corpus (MOVED out of eshpfi), `KNOWLEDGE.md` (engine landscape + prosody findings + foot-guns), `docs/infrastructure.md` (irv-ml1 access + gated deploy runbook + rollback), CLAUDE/persistent-memory/ROADMAP, `tools/` (pause-probe + Booth render). Followed the **chatterbox-fast precedent**: eshpfi `stacks/dots-tts/` reduced to a POINTER README; the ~15 experimental TTS compose wrappers stay here as reference (catalogued in tts-stack KNOWLEDGE). Blast-radius check: no eshpfi playbook/script reads the canonical corpus (other `voices/` refs = unrelated host paths). **Reverses** the earlier "Corpus home = eshpfi `voices/` (keep-here)" call. ⚠ tts-stack is LOCAL-ONLY until pushed — needs a gitea remote (`vh/tts-stack`) + push before the separate agent can clone (operator's call — outward-facing + repo-create creds). _Archived 2026-08-27._ - `[2026-08-10]` **dots-tts v3 — clause-break → period pause mapping.** Operator: v2 "sounds good" but donut won't pause at semicolons/dashes. ROOT CAUSE (measured via a pause-probe A/B — synth duration over N runs, non-determinism averaged out): dots' prosody honors a real pause **only for ellipsis (~+0.43s) and period (~+0.3s, capitalization-independent)**; comma/semicolon/colon/dash all run **flat (~+0.03s vs no-punct)**. Two distinct sub-causes: **dashes regressed in v2** (the `—`→`-` fold made em-dashes read as word-joiners), while **semicolons were NEVER a v2 change** — dots ignores them natively, only newly noticeable because v2 made everything else clean. Operator call: ellipsis "too much" → **map `;`, clause `:`, and em-dash `—` → period** in `_sanitize` (believable ~0.3s clause break). GUARDS (pinned by 11 unit tests, `stacks/dots-tts/test_sanitize.py`): digit-guarded colon `(?` checkout. _Archived 2026-05-27._ - `[2026-05-12]` asset-engine stack scaffolded LAN-direct at `http://10.250.50.70:8200`. Initially included Traefik labels for public hostname; user pulled them out (internal tool, no public TLS surface needed). Pattern: internal tools default LAN-direct; Traefik wiring only when external/TLS required. _Archived 2026-05-27._ - `[2026-05-12]` asset-engine catalog gains `lifecycle: { stack, vram_gb, gpu_device_id }` per irv-ml1 service for the orchestrator feature. SSH keypair scaffolded at `ana-docker:/opt/docker/conf/asset-engine/ssh/` for asset-engine container → irv-ml1 orchestration via dedicated ed25519 key. _Archived 2026-05-27._ - `[2026-05-13]` `pull-hf-repo.yaml` is the canonical HF-fetch playbook on ana-ml2. Supports `--var repo_type=model|dataset|space`. Replaces ad-hoc `huggingface_hub.snapshot_download` calls. _Archived 2026-05-27._ - `[2026-05-13]` Selene-1-Mini-Llama-3.1-8B added to llama-swap as judge model. mradermacher `i1-Q6_K` imatrix quant (~6.5GB). AtlaAI reward/eval model — temp 0.01, ctx 32K, q8_0 KV cache. New JUDGE / EVAL MODELS section in `stacks/llama-swap/conf/config.yaml`. _Archived 2026-05-27._ - `[2026-05-13]` `vllm-qwen3` → `vllm` stack rename. Added `vllm-reward` service (Skywork-Reward-V2-Llama-3.1-8B-AWQ classifier). Three vLLM services share GPU 1 (embed 0.20, rerank 0.20, reward 0.30 utilization; 30% headroom). All use `--runner pooling`; classification drives via model's `architectures: [LlamaForSequenceClassification]` in config.json, NOT `--task classify` (deprecated in vLLM 0.19.1). _Archived 2026-05-27._ - `[2026-05-13]` /tend-docs first pass deletions: `stacks/infinity/` removed (retired by vllm). Archived `docs/asset-engine/design-brief.md` → `docs/archive/asset-engine/` with archival header. Fixed `pfi-pve` VM list to full `qm list` enumeration. Dropped stale weak-password section from `pfi-postgres` (rotation done 2026-04-23). _Archived 2026-05-27._ - `[2026-05-14]` althing-chamber stack scaffolded: chamber + forseti. Internal LAN-only at port 7881 (chamber default 7878 collides with task-board). Two-service compose, shared SQLite bind-mount, build-on-host pattern via vh/althing's gitea-workflow. Forseti is the canonical dev for this stack (galdrabok is on a different project). _Archived 2026-05-31._ - `[2026-05-16]` althing-chamber Phase 2: added `althing-agent-runner` as third compose service (worldtree-driver agent dispatcher). All three althing services use the same image; `command:` selects entrypoint. Safe to enable preemptively (sleeps when no driver=worldtree handles declared). _Archived 2026-05-31._ - `[2026-05-17]` Phase 3.1 cross-process streaming uses Valkey 8 alpine as a sibling compose service in `stacks/althing-chamber/`, redis-protocol pub/sub for high-volume `msg_delta` / `msg_thinking` / `msg_start` / `msg_complete` event kinds. DB bridge keeps `msg_curated` + `floor_grant` (structured / canonical). Two-channel architecture, no overlap. chamber + agent-runner `depends_on: valkey: service_healthy`. _Archived 2026-05-31._ - `[2026-05-17]` Worldtree admin workflow shift (per vh): infra-ops gets its own permanent admin-tier key (`61419c92`, stored at `ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin`). Future admin ops route through this key, not the bootstrap admin via docker-as-root. _Archived 2026-05-31._ - `[2026-05-17]` Worldtree env-var addition checklist: anytime introducing `os.environ.get("FOO")` in worldtree code, update BOTH `.env.example` AND `compose.yaml`'s `&worldtree-env` anchor in the same PR. Same Z_AI_API_KEY-shape footgun bit `BIFROST_CLIENT_ALLOWED_HOSTS` (#170) until worldtree-dev added the passthrough line in `08f02b2`. _Archived 2026-05-31._ - `[2026-05-18]` Volva systemd install complete after three-stage debug. Final unit at `/etc/systemd/system/volva.service` runs as `User=lkraven` with `ProtectHome=read-only` + `ReadWritePaths=/home/lkraven/.althing /home/lkraven/.codex` carve-outs for state writes. `VOLVA_ALTHING_CLI=/home/lkraven/ .local/bin/althing-cli` + `ALTHING_HANDLE=volva` both pinned in env.sh. _Archived 2026-05-31._ - `[2026-05-19]` Worldtree CD disk-hygiene strategy: watermark gate (env-tunable threshold + window, fail-loud on still-low post-prune) + eager post-deploy prune (only after `:latest` advance succeeds, uses `docker image prune -a --filter "until=24h"` which respects in-use semantic — protects pinned + personal images automatically). Combined: demo VM holds ~24h of deploy history instead of unbounded accumulation. Shipped in vh/Worldtree PR #184 (`306cd61` + `613dac2` + `bd91df5`). _Archived 2026-05-31._ - `[2026-05-19]` Skaldsong CD shape: shape (1) of three operator options — container + Gitea registry + pull-restart, matching Worldtree's pattern. Target host ana-docker (NOT nh3-dev where skaldsong-dev runs for iteration). SHA-pin only for now; health-gated `:latest` advance is a follow-up once `/health` exercises Worldtree + Kokoro reachability. _Archived 2026-05-31._ - `[2026-05-19]` Skaldsong prod (ana-docker) switched from demo Worldtree (:8080) to personal (:8081). Same `user_id=skaldsong` as the nh3-dev hand-launch key — shared Heimdall agent slot (`skaldsong:wizard-v2`), different `key_id`s for independent rotation. Demo Worldtree stays for isolation; personal becomes the multi-consumer dev iteration instance. _Archived 2026-05-31._ - `[2026-05-19]` mead-hall Bifrost v0.3 end-to-end smoke green. Closed task #32 (althing thread `01KRV1M2KW6N6HBEXGTH72QXCA`). Wire layer (handshake + binding + dispatch) + data-flow (per-dispatch JWT claims → `ctx.session_id` populated → real session-scoped data) + agent-loop (LLM reads + quotes back) all proven. Resolves the "stalled mid-Worldtree" state from the 2026-05-17 snapshot. _Archived 2026-05-31._ - `[2026-05-25]` v0.25.3 lofn tuning: `temperature 0.6 → 1.0` + `repetition_penalty 1.0 → 1.15` on default+fast profiles. Heretic-abliterated qwen3.6 was locking into degenerate attractors at the model's thinking-mode floor (0.6). Pattern: abliterated/uncensored Qwen variants need higher temp + non-trivial rep-penalty than base, NOT the model-card's documented floors. _Archived 2026-05-31._ - `[2026-05-25]` Worldtree #205 v0.25.2 ships `/app/config/` as bind-mount + root-then-drop entrypoint shim (gosu). Operators get persistent per-instance config without container-rebuild. Same bind-mount pattern hit twice subsequently in v0.27.0 (selene) and v0.29.9 (echo) — bind-mount shadows image-baked defaults, so every new required key surfaces as a crash-loop on existing deployments. The v0.29.12 canonical example files close this loop. _Archived 2026-05-31._ - `[2026-05-26]` Worldtree v0.27.0/v0.27.1 fixes Tier 3 GET visibility. `available_agents()` helper was over-applied to `GET /agents/`, masking ALL Tier 3 agents regardless of row state. Bug only visible as "agent not found" via GET; storage was fine (silent-2xx PATCHes had persisted correctly). v0.27.1 added fail-fast hardening for the startup pre-resolve fragility class. _Archived 2026-05-31._ - `[2026-05-26]` Skaldsong v0.30.7 defensive 409→PATCH fallback. v0.30.6's GET-then-define-or-patch path crash-looped against pre-v0.27.0 Worldtree's GET-visibility bug (GET 404 phantom → define 409 conflict). v0.30.7 catches the 409 and falls through to PATCH (which silently 2xx'd on the pre-fix Worldtree). _Archived 2026-05-31._ - `[2026-05-27]` Worldtree v0.29.x landed full saga→echo refactor + config-validator hardening (v0.29.10 create_provider family-before-regex; v0.29.11 collect-then-raise echo startup validators; v0.29.12 ships providers.yaml.example/defaults.yaml.example canonical configs; v0.29.13 reasoning_content extraction + catalog family lookup). Operator-asked, worldtree-dev-shipped, CI-deployed. _Archived 2026-06-01._ - `[2026-05-27]` artemis-31b-v1i added to llama-swap + worldtree personal. BeaverAI Gemma 4 31B Q6_K (~28.6GB), 128K ctx, `--reasoning-format deepseek` (gemma format unsupported in deployed llama.cpp). Worldtree catalog `family: gemma` so GemmaProvider routes reasoning tokens. _Archived 2026-06-01._ - `[2026-05-27]` Skaldsong streaming TTS v0.32.0→v0.32.2: chunked-batch SSE (one Kokoro POST per paragraph); defensive event_stream catch-all; NDJSON parsing for Kokoro /dev/captioned_speech multi-line responses. _Archived 2026-06-01._ - `[2026-05-31]` Dia2 deployed as **two fixed-model instances** (`dia2-2b` :8200, `dia2-1b` :8202) from `local/dia:v2`, retiring legacy Dia 1.6B; catalog `dia` entry **removed → dia2-2b + dia2-1b** (breaking for asset-engine). Rationale: the devnen wrapper is single-model and IGNORES the OpenAI `model` field (verified on its live OpenAPI), so the only way to offer both Dia2 models as real per-request asset-engine choices is one fixed endpoint per model. `3139e81` (deploy), `db15638` (catalog swap). _Archived 2026-06-03._ - `[2026-05-31]` Both dia2 catalog entries route to the wrapper's richer **`/tts`** endpoint (not `/v1/audio/speech`) to expose the full control surface (cfg_scale/temperature/top_p/cfg_filter_top_k/voice_mode/clone); all defaults sourced from the wrapper's `CustomTTSRequest` Pydantic blessed values. Voice default is `voice_mode: clone` + `clone_reference_filename: Abigail.wav` so a stable (non-random-gender) voice is the out-of-box behavior. `55602b7`, `5c47843`. _Archived 2026-06-03._ - `[2026-05-31]` Zonos REST adapter (`stacks/zonos/adapter/`, `local/zonos-api`) — thin OpenAI-ish `/v1/audio/speech` FastAPI in front of the Gradio-only Zonos SDK; JSON-envelope `{audio, audio_format, seed}` (Zonos is the fleet's first seedable TTS). Port 8203 (moved off 8201 — collided with csm). Built; NOT deployed (stack down for VRAM). Also fixed the upstream image's missing CMD (`71df6f7`). `81efa8d`. _Archived 2026-06-03._ - `[2026-05-31]` Catalog schema regenerated: added `CatalogLifecycle` + `reproducibility.seed_field` (`b7b2130`). Resolves the stale-schema hand-off; catalog now validates clean. (asset_engine `catalog.py` Pydantic already supported both — schema file was just behind.) _Archived 2026-06-03._ - `[2026-05-31]` TTS bench expanded with `stacks/{dia,zonos,csm}` (`666f7f3` dia+zonos, `a4b8c2a` csm). The bench already had Fish S2-Pro / Chatterbox-Turbo / IndexTTS-2 / CosyVoice3 / Kokoro / VibeVoice / Qwen3-TTS / Kyutai. (csm since removed 2026-06-01.) _Archived 2026-06-03._ - `[2026-05-31]` Remote browser/iPad/Vision-Pro driver seat for the agent-fleet zellij session `Claude` stood up on nh3-dev (ttyd behind Caddy, network-gated). Out of this repo — full architecture + the HTTP2/OSC52/Safari-auth gotchas in auto-memory `reference_ttyd_fleet_seat`. _Archived 2026-06-03._ - `[2026-05-30]` esh-docker-vm NFS boot-ordering fix: `playbooks/fix-esh-nfs-boot-ordering.yaml` (`c0458d9`, + `53157b1` drop-in filename-collision fix) adds `_netdev,nofail` to the four 10.0.50.50 NFS mounts + a docker `After=remote-fs.target` drop-in — resolves paperless `Exited(255)` on reboot. traefik also gained `restart: unless-stopped`. Full incident → auto-memory `incident_esh_docker_nfs_boot_race`. _Archived 2026-06-03._ - `[2026-06-01]` **Fish cloning VERIFIED competent (ECAPA-TDNN)** — retracting the earlier "weak cloner" call. Isolated test: Imogen-referenced clone ~0.79 cosine to the real `Imogen.wav` vs ~0.10 for the no-reference default; transcript condition (correct 0.787 / bogus 0.778 / empty 0.738) barely moves identity (affects pronunciation, not timbre). Root cause of "Imogen sounds nothing like British" = the `"undefined"` select bug feeding Fish its default voice, NOT Fish. So the entire Fish-Imogen saga was the undefined bug; no Fish-side fix needed. (Methodology lessons → Tried and abandoned.) _Archived 2026-06-04._ - `[2026-06-01]` **CSM (Sesame csm-1b) torn down entirely** — removed from catalog, `stacks/csm/`, `playbooks/deploy-csm.yaml`, and host (`c54ab13`). Two reasons: (1) deep-research verdict — the acclaimed Maya/Miles demo is a fine-tuned, larger variant Sesame never open-sourced; the open 1B base ships no usable voices, can't generate text, English-only, no real-time streaming, absent from leaderboards, dominated by Kokoro/Dia2/Fish/IndexTTS. (2) The upstream image is rotted (see Tried and abandoned). asset-engine re-synced (`306aae9`). _Archived 2026-06-04._ - `[2026-06-01]` **Chatterbox catalog v1→v2** (`93cedd6`) — re-point to the devnen wrapper's `/tts` (was the thin OpenAI `/v1/audio/speech` exposing none of the emotion knobs → poor output). Default `exaggeration: 0.5` per Resemble's README ("works well for most"); the server SHIPS demo-tuned 1.3 which over-acts — deliberately NOT adopted. 28 predefined voices via `/get_predefined_voices` (default Emily.wav). Corrected seedable false→true. _Archived 2026-06-04._ - `[2026-06-01]` **Fish-S2 catalog v1→v6** (`f3c260e`..`38d9e3b`) — was a blank 2-field form; now exposes the full fish-speech ServeTTSRequest surface + a 32-voice `reference_id` "Voice" dropdown (STATIC list — Fish has no /voices API). `reference_id`-by-name works ONLY when each voice is staged as a `references//` dir with `.wav` + `.lab` — was silently a no-op until that layout fix on 2026-06-01 (see the two-root-causes entry above); the inline-path-as-filesystem-path `references` shape 500s (see Tried and abandoned). Corrected vram 4→24 GB, cold_start 8→240s, seedable false→true. Added 3 consenting British-female VCTK voices (Imogen=p225/Eleanor=p228/Beatrice=p229). _Archived 2026-06-04._ - `[2026-06-01]` **`blendable: false` catalog flag — DEFERRED**, pending asset-engine-dev honoring it in the form template. Clone-voice selects (chatterbox/fish/dia2) must render as plain dropdowns, not the Kokoro blend widget (blending reference clones is nonsensical). I ship the catalog flag + schema field once they read it. Tracked: althing msg `01KT2K2SY9N7AY69R9V0B4RXSW` to asset-engine-dev. _Archived 2026-06-04._ - `[2026-06-01]` **Fish reference_id empty-dir fix shipped** (`c5bbb90`) — Populated `references//.wav`+`.lab` for all 32 voices; playbook gained normalize-step + A/B smoke gate. glados got a real transcript (ASR'd via Parakeet): the Portal "Welcome to test chamber 4" lines. _Archived 2026-06-11._ - `[2026-06-02]` **Chatterbox → main TTS engine; build custom `chatterbox-fast` streaming container.** Workload = single-stream interactive. GPU placement: 3090 (device 0) if it fits else A6000 (device 1). Cutover: parallel catalog entry, burn in beside live `chatterbox`, then flip. Streaming = adaptive buffer-ratchet chunking (native frame-streaming abandoned). Since extracted to its own repo `vh/chatterbox-fast` (auto-memory `reference_chatterbox_fast_repo`). _Archived 2026-06-11._ - `[2026-06-02]` **Sentence-splitting loses quality (operator-corrected).** Claimed naive sentence-level streaming has "zero quality loss" — WRONG. The T3 AR backbone conditions prosody on the WHOLE text; splitting loses cross-sentence prosodic context even though voice timbre stays (reference-conditioned). No *artifacts* ≠ no *quality loss*. Hence adaptive-chunk design (maximize context per chunk subject to latency budget), not fixed per-sentence splits. _Archived 2026-06-11._ - `[2026-06-03]` **Declined worldtree v0.30.4 staging deploy** — that's worldtree-team's CI/CD lane (a developer `staging/vX.Y.Z` git-tag promote), not infra-ops. They self-corrected to the same conclusion independently. _Archived 2026-06-13._ - `[2026-06-03]` **R14 scope = (a) provision-only.** infra-ops provides box + CUDA env + engines + weights + NFS; brokkr/dev wires `arms.py` + runs — keeps infra-ops OFF the VIVAE-processing path (VIVAE = Variably Intense Vocalizations of Affect/Emotion, CHARTER §4 highest-liability; operator authorized R&D-eval-only, quarantined). Box = irv-ml1 (A6000 free; ana-ml2 GPU-saturated). Per-engine venvs (divergent torch stacks); A6000 = `cuda:0` NATIVE (≠ docker `=1`). _Archived 2026-06-13._ - `[2026-06-03]` **yt-voice-clipper push-to-deploy via gitea webhook** (operator-directed, after 6 manual rebuilds in ~40 min). Webhook (not poll) — gitea CAN reach the WG IP per the operator. The proxy env + Homepage labels live in the **host-specific override** (untracked → survive the auto-deploy's `git reset --hard`), NOT yt-voice-clipper-dev's image. Runbook `d4f180d`. _Archived 2026-06-13._ - `[2026-06-03]` **yt-voice-clipper bot-gate fix = route yt-dlp through NH3 residential egress, NOT cookies/PO-token.** YouTube hard-flags the Irvine colo IP (LOGIN_REQUIRED on a public video even with no cookies). Cookies + the bgutil PO-token + deno JS-runtime all loaded fine — the gate is pure IP reputation. Operator chose proxy-via-nh3-dev → durable dante proxy → proven. The egress proxy is a reusable fleet lever for any datacenter-IP-gated service. _Archived 2026-06-13._ - `[2026-06-04]` **`brokkr` user (no-sudo) on irv-ml1; R14/R15/R16 substrate moved to /home/brokkr.** Persistent box services there need SYSTEM systemd units (see Tried). _Archived 2026-06-14._ - `[2026-06-04]` **ollama upgraded 0.9.0→0.30.4 on irv-ml1** (Ministral-3 is a Dec-2025 model the old engine refused); A6000 pinned by **UUID** not index (native fastest-first ≠ nvidia-smi PCI). _Archived 2026-06-14._ - `[2026-06-04]` **Worldtree demo/pinned/personal deploys are ALL CI/CD, not infra-ops** — a "deploy vX.Y.Z" request to infra-ops is MISROUTED → point them back to their pipeline. (`d8d776c`, auto-memory `reference_worldtree_deploys_cicd`) _Archived 2026-06-14._ - `[2026-06-04]` **infra-ops NOPASSWD-sudo identity commissioned, scoped to PFI boxes** (+esh-docker-vm by operator override) — so infra-ops completes DevOps end-to-end vs handing the operator sudo steps. Dedicated key, sudo log_output, key-gated. (`8c32a05`, auto-memory `reference_infra_ops_sudo_identity`) _Archived 2026-06-14._ - `[2026-06-04]` **phi4 ships the CANONICAL/official Phi-4 chat template, NOT Ollama's** (Ollama's omits the system `<|end|>`). Applied an Ollama-matching override then reverted — ship correct, not the benchmark quirk. (`90e08f0`→`27eb537`; "headgun" lesson in Tried.) _Archived 2026-06-14._ - `[2026-06-04]` **phi4-mini FP8 on ana-ml2 vLLM is the nevermore summarizer; granite-4-small retired** from llama-swap. 50K ctx + FP8 KV. (`40a374b`) [superseded by the 2026-06-05 granite-4.1-8b decision] _Archived 2026-06-14._ - `[2026-06-08]` **volva.service + heid.service removed from nh3-dev** — vestigial systemd daemons; Heid/Volva re-architected from Python pollers to Claude Code session orchestrators (heid `12aa5a9`); volva.service was crash-looping 203/EXEC. (`6e2f80e`) _Archived 2026-06-16._ - `[2026-06-05]` **Granite 4.1 8B FP8 replaced phi4-mini as the production summarizer.** Beat phi4 on precision in brokkr's R15 P03. **Staying FP8, not Q4/AWQ** — primary workload is high-concurrency, where FP8 scales ~linearly (2010 tok/s @ C=32). vLLM `vllm-granite` :8004 GPU 1, official IBM compressed-tensors FP8. (Then on Ada; box has since gone Blackwell.) (`34a43a0`, auto-memory `reference_ana_ml2_vllm_granite`) _Archived 2026-06-16._ - `[2026-06-05]` **Langfuse v3 on ana-docker (:3001) as the gateway trace UI**; LiteLLM `success_callback:[langfuse]` live. Pretty traces + tok/s dashboard. NOT a prerequisite (spend_logs already capture tokens+latency). (`9171e6a`) _Archived 2026-06-16._ - `[2026-06-05]` **Ollama BANNED fleet-wide** (operator directive) — never stand one up; tear down any found; serve via llama-swap or vLLM. (auto-memory `feedback_avoid_ollama`) _Archived 2026-06-16._ - `[2026-06-05]` **ComfyUI / FLUX.2 work split to `~/development/comfy-dev`** (dedicated repo + agent). eshpfi keeps the `comfyui`/`arbo` stack compose; comfy-dev owns the model/workflow knowledge. (auto-memory `reference_irv_ml1_ampere_quant`) _Archived 2026-06-16._ - `[2026-06-05]` **Worldtree summarizer config refresh DEFERRED to Worldtree #254** (granite-4.1-8b is the structured-output profile, ON HOLD, no live consumer). Bind-mount CONFIGS (providers.yaml, vh-owned) ARE infra-ops's to apply directly — only the app/image DEPLOY is CICD; the `.env` is deploy-owned. (auto-memory `reference_worldtree_deploys_cicd`) _Archived 2026-06-16._ - `[2026-06-13]` **Arbo prod bearer auth turned OFF — WireGuard is the access boundary** (operator decision; reverses ADR-0001's "closed the open-auth hole"). ENGINE_TOKEN must be **ABSENT, not empty** (empty-string still gates) — removed from BOTH the host `.env` AND the compose `environment:` injection line. Original token backed up at `irv-ml1:/opt/docker/compose/arbo/.env.pre-auth-off.bak`; comfy-dev updated their ADR-0001. Tracking: `db97899` + `playbooks/arbo-disable-engine-token.yaml`. _Archived 2026-06-16._ - `[2026-06-13]` **Storetank image-models archive DECOMMISSIONED; arbo is the single live ComfyUI model tree (502 G).** Curated `/storetank/image-models/comfy` (was 919 G, the native `/opt/ComfyUI/models` symlink target) → killed everything superseded by arbo's current gen (Hunyuan, WAN2.1, FLUX.1, Chroma, SD3.5, orphaned umt5+llava ≈ 739 G) + migrated the keepers (gen-agnostic utilities + the SDXL/Pony stack, 177 G) into `/storetank/arbo/models` (same-fs move, skip-existing protects prod). Tracking: `docs/storetank-image-models-archive.md` + `docs/arbo-comfyui-model-catalog.md` (`1902425`→`5007ec1`). _Archived 2026-06-16._ - `[2026-06-13]` **GRANITE_KEY provisioned to comfy-dev's nh3-dev dev env** at `~/.arbo_granite_key` (0600) for the hero gen+judge script — verbatim copy of the prod `arbo-prompt-enhance` vkey (now extended to reach BOTH `granite-4.1-8b` AND `qwen3.5-9b-fp8`); nothing minted. The vkey README's "granite-only" wording was stale → corrected (`f32c6dd`). _Archived 2026-06-16._ - `[2026-06-13]` **ana-ml2 upgraded Ada → dual RTX PRO 6000 Blackwell Max-Q** (96 GB each, cc 12.0 / sm_120; was dual RTX 6000 Ada 48 GB / cc 8.9 — confirmed live via `nvidia-smi`). Unlocks NVFP4 (FP4 tensor cores) and doubles VRAM headroom. CLAUDE.md GPU-spec doc-fix **LANDED `355a240`** (operator). Tracking: `19a07b9`/`1e2a3a1` ("Blackwell 96GB"). _Archived 2026-06-16._ - `[2026-06-13]` **NVFP4-W4A4 is infeasible for Granite — FP8 stays the Granite-on-Blackwell format.** W4A4 collapses at 30k context, proven **producer-independent** (modelopt AND llm-compressor both clean-NONE from the same BF16 base + wikitext-2k calib). No 4-bit wins both axes: W4A4 = quality collapse; W4A16-NVFP4/AWQ = weight-only dequant → bf16 (no FP4-core speedup). **30B retired**. (auto-memory `reference_nvfp4_w4a4_granite_infeasible`) _Archived 2026-06-16._ - `[2026-06-13]` **Qwen3.5-9B VL (FP8) deployed on ana-ml2 GPU 1** — `qwen35-vl` stack, :8007, gateway alias `qwen3.5-9b-fp8`. **Pinned nightly digest, not `:latest`**: the stable release quantizes the VL *vision tower* under `--quantization fp8` → garbage vision (LM fine); the nightly correctly excludes it. Re-pin + drop the pin once that exclusion lands stable. (`2e3dcc2`) _Archived 2026-06-16._ - `[2026-06-13]` **comfyui 325 G model tree migrated worktank → `/storetank/arbo`** (worktank 97% → 26%). `arbo` is the consuming app; overlay bind-mount via `COMFYUI_MODELS_DIR`. (`38186be`) (See the 2026-06-13 archive-decommission decision above — this tree later absorbed the storetank-archive keepers, reaching 502 G.) _Archived 2026-06-16._ - `[2026-06-13]` **GPU layout settled on the Blackwell box.** GPU 0 held free for large-model hot-loads (llama-swap pinned, `edf0f91`); GPU 1 steady-tenant — granite 131k ctx, qwen 65k, embed/rerank/reward trio, ~3.5 GB free (`1e2a3a1`, `19a07b9`; trio re-floored for 96 GB, 20×-parallel-stable). embed/rerank left at floor — long docs chunked BEFORE embedding. max-model-len is a ceiling not a reservation. _Archived 2026-06-16._ - `[2026-06-13]` **Prefix caching pinned explicit on granite + qwen** — benched ~6.5× faster TTFT on a shared ~4.5k-token summarizer template; soft/evictable, neutral when prefixes don't repeat. vLLM v1 defaults it ON (granite) but the qwen nightly defaults OFF — pin both. (`a9a2be7`) _Archived 2026-06-16._ - `[2026-06-13]` **granite-4.1-8b listed as the always-available summarizer/classifier + a shared all-agents key minted** (operator-directed). Global `~/.claude/CLAUDE.md` Global-tools entry; key alias `all-agents-local`, scoped to the FREE local models only (granite + qwen-vision + embed/rerank, NOT paid GLM), internal-gateway-only, rotatable. (auto-memory `reference_litellm_gateway`) _Archived 2026-06-16._ - `[2026-06-13]` **arbo engine + frontend stack stood up** (ADR-0001) — irv-ml1 co-located inference engine (`ee57e69`), python-based healthcheck (slim image, no curl/wget, `bdb3312`), frontend ro-mounted from the checkout (`922e8ad`, ADR-0001 D2). _Archived 2026-06-16._ - `[2026-06-11]` **GLM thinking inverted at the LiteLLM gateway** (operator call): `glm-5.1` defaults thinking-OFF; `glm-5.1-reasoning` = same z.ai upstream, thinking ON. Mechanism: `litellm_params.extra_body:{thinking:{type:disabled}}` — `drop_params` strips a top-level `thinking`/`reasoning_effort` but forwards `extra_body` verbatim to z.ai. Shared-gateway change. (`95b2701`, auto-memory `reference_litellm_gateway`) _Archived 2026-06-16._ - `[2026-06-11]` **pi coding agent installed on nh3-dev as a GLM 5.1 harness** — `@earendil-works/pi-coding-agent` via **bun** (npm's global prefix is `/usr` → needs sudo, bun avoids it). Config `~/.pi/agent/models.json`, launcher `~/.local/bin/glm`. _Archived 2026-06-16._ - `[2026-06-11]` **z.ai web-tools (regin) = z.ai hosted MCP path, NOT the `/paas/v4` Tool API.** WORKS: MCP streamable-HTTP at `https://api.z.ai/api/mcp/{web_search_prime,web_reader}/mcp`, `Authorization: Bearer $ZAI_API_KEY` (the **MCP** key, distinct from `Z_AI_API_KEY` the LLM key). Reference impl = Worldtree's Leif agent. _Archived 2026-06-16._ - `[2026-06-10]` **Mac Pro migration framed: workstation-only** (M2 Ultra ARM, racked NH3 on-subnet); sidecars stay Linux. `migration-plan.md`. (See in-flight.) _Archived 2026-06-16._ - `[2026-06-10]` **Worldtree deployed-config propagation is infra-ops's OWNED lane** (operator ruling). worldtree-dev pings the config delta pre-merge; infra-ops syncs `config/*.yaml` from MERGED canonical to the `/opt/worldtree*/config` bind-mounts on demo+personal. The v0.33.8 9-HOUR demo outage (a `model_roles.yaml` startup-dep that never reached the VM) is the failure mode this prevents. providers.yaml stays hand-tuned. corviduo emergency-ops = `ssh vh@10.250.50.152`, docker no-sudo. (auto-memory `reference_worldtree_deploys_cicd`, `reference_corviduo_dev_emergency_ops`) _Archived 2026-06-16._ - `[2026-06-09]` **LiteLLM scoped virtual keys issued to consumers** (operator-authorized): `brokkr-smithy` (all-proxy-models), `arbo-prompt-enhance` (comfy-dev — granite, later extended to qwen-vision). Mint via `/key/generate` (master `sk-corvid`), scope-restricted + rotatable, value → 600 file never the bus. (auto-memory `reference_litellm_gateway`) _Archived 2026-06-16._ - `[2026-06-14]` **ana-ml2 GPU-1 vision upgraded: Qwen3.5-9B → Qwen3.6-35B-A3B (official FP8), served under its TRUE name only.** `qwen36-vl` replaces `qwen35-vl` on :8007 (`a0fed13`). The stale `qwen3.5-9b-fp8` name is KILLED at vLLM AND the litellm gateway (404/400) — a model is NEVER aliased under a prior model's name (silent substitution = downstream footgun; operator directive). Consumer comfy-dev/arbo migrated; arbo vkeys → all-proxy-models; shared `all-agents-local` key repointed qwen3.5-9b-fp8 → qwen3.6-35b-a3b. GPU-1 rebalanced for the ~34 GB FP8 weights (granite 0.35→0.24/64K; embed/rerank 0.05→0.03, reclaimed ~4 GB util-waste). Validated: vision correct, 20-concurrent = no OOM. (auto-memory `feedback_no_false_model_aliases`) _Archived 2026-06-18._ - `[2026-06-14]` **NVFP4 was the lighter fit (~21 GB) but is BLOCKED on vLLM — FP8 is the working vision path.** `nvidia/Qwen3.6-35B-A3B-NVFP4` won't load: the ModelOpt-NVFP4-MoE loader errors on expert/lm_head scale keys across 0.19.1 (`w2_input_scale`) AND 0.22.0 (`lm_head.input_scale`, vllm #44081) — a pattern across modelopt NVFP4 MoEs. Revisit NVFP4 (frees ~13 GB on GPU 1) once fixed; the 21 GB checkpoint stays cached on ana-ml2. **(SUPERSEDED 2026-06-16 — it loads on vLLM 0.23.0; qwen36 swapped to NVFP4. See the top of this section.)** _Archived 2026-06-18._ - `[2026-06-14]` **llama-swap qwen3.5-9b GPU-0 pin DROPPED; GPU 0 reserved for a creative-writing model (pick DEFERRED by operator).** Deep-research (this session) on big-fast-uncensored creative for a 96 GB Blackwell: **GLM-Steam-106B-A12B** (already in the llama-swap config — balanced default) vs **TheDrummer/Behemoth-X-123B-v2** (prose-tier, tops UGI writing+willingness) vs XORTRON-123B (max willingness, weak prose); GGUF-on-llama-swap is the serving path. Tracking: this session + llama-swap config (GLM-Steam present, `untracked by operator choice`). _Archived 2026-06-18._ - `[2026-06-14]` **R16 splice-pivot yield probe executed** (infra-ops ran the irv-ml1 inference for brokkr; brokkr owns design + analysis). See Current state. Tracking: althing thread `01KV010WGS…`, `gen_yield_probe.py` in `irv-ml1:~/r16-vmoan-harness`. _Archived 2026-06-18._ - `[2026-06-14]` **R16 vmoan inline-generation arc CLOSED — v1 at default decode (rep1.2/temp0.8) is the final Chatterbox-tag inline artifact.** Operator's ear rejected every alternative: v2/v3 windowing (omission vs coherence-loss), v4 multi-tag (cohesion held but lost to capacity-competition), emergent inline-token modulation (degenerates, not modulates), and the gen-time decode-polish sweep (soft tamers cut the NVV itself — same omission family as v2; p0 baseline beat p1). All adapters v1–v4 + `tokenizer.json.v3bak` preserved on `irv-ml1:~/r16-vmoan-harness`. Likely-next direction (deferred, NOT formalized): generate→bin→splice + one-shot-clone NVV pipeline routing around the inline-coherence wall. Tracking: brokkr R16 journal + althing thread `01KV010WGSSMPWRNCPAGSPK15Y`. _Archived 2026-06-18._ - `[2026-06-14]` **Arbo deploy pipeline fixed, hardened, and version-controlled.** Prod rebuilt v0.11.1 → **v0.11.6** backend; the webhook machinery (`arbo-deploy.sh` + `arbo-webhook.py`, :9009 HMAC listener) is now repo-tracked at `stacks/arbo/` (was host-only = recoverability foot-gun). Deploy reaches gitea via the INTERNAL route (`10.250.50.70:222`) and restarts the engine ONLY on `catalog/` changes (graphs/frontend per-request; warn on `src/`|`Dockerfile` only — pyproject/uv.lock churn every commit). Operator kept arbo stack ownership in **eshpfi** (not migrated to comfy-dev's repo). Secret + `.env` stay host-only. Tracking: `6d66bc2`, `6e58e57`, `stacks/arbo/README` Q5. _Archived 2026-06-18._ - `[2026-06-15]` **qwen36 thinking-split at the gateway** (operator) — base `qwen3.6-35b-a3b` = thinking-OFF default (the vision judge needs it), `…-a3b-thinking` = opt-in reasoning. One hybrid checkpoint + `--reasoning-parser qwen3`; mirrors the glm-5.1 split. The judge's enable_thinking=false is what makes the NVFP4 swap transparent. (`6de0844`) _Archived 2026-06-19._ - `[2026-06-15]` **arbo engine rebuilt v0.11.6 → v0.11.22 from main; catalog re-applied (v0.11.23) — curated `/workflows` footer live.** src/ schema changes (Model/Lora.summary, Workflow.blurb/keywords + the /workflows endpoint) need a MANUAL image rebuild (the webhook warns-only on `src/`/`Dockerfile`). Built from `/worktank/arbo/repo`, bumped `ARBO_IMAGE` in the host .env, recreated + verified healthy (auto-rollback armed, unneeded). comfy-dev's v0.11.20 catalog push had first crash-looped prod (old baked schema `extra=forbid` vs new git-mounted catalog fields → ValidationError); they reverted the catalog, so rebuild-from-main loaded clean, then they re-applied (v0.11.23). comfyui is a SEPARATE compose project — the rebuild didn't touch it. _Archived 2026-06-19._ - `[2026-06-15]` **althing-core refreshed to v0.14.1 box-wide on nh3-dev** (forseti + heid both routed it; central-sequenced per DevOps-delegation since `uv tool install --force` swaps the shared binary all handles use). Fixes the monitor non-truncating-lock bug so `stop-monitor`/`--stop` reliably recovers orphans. The editable install (`uv tool install --force --editable .`) covers every nh3-dev handle; `--version` trued up 0.14.0→0.14.1; verified `--stop` works against a real orphan. _Archived 2026-06-19._ - `[2026-06-15]` **ratatoskr Tier-3 affect live smoke GREEN** (consumer-confirmed). Brokered/minted consumer_id="ratatoskr" Heimdall HS256 key on personal Worldtree (the key IS BOTH the create-session Bearer AND ratatoskr's JWT verifier secret — plain utf-8 string, NOT base64/raw); injected ENV-ONLY to `~/.config/ratatoskr/provider.env` (mode 600) on nh3-dev; added `10.100.10.50:8390` to personal's `BIFROST_CLIENT_ALLOWED_HOSTS`; drove define→session (handshake 200)→turn→affect.emit (200, snapshot persisted). Worldtree signs Bifrost with the caller's raw Bearer verbatim. Drive commands + claims contract in althing thread `01KV4ZYSE3…`. _Archived 2026-06-19._ - `[2026-06-15]` **Dense Qwen3-VL-32B-NVFP4 A/B'd as arbo's anatomy hero-judge vs the qwen3.6-35b-a3b MoE → dense LOST, pilot torn down, GPU 0 reclaimed.** Dense NVFP4 loads clean on vLLM where the MoE NVFP4 doesn't (no expert-scale keys); ~44 tok/s on Blackwell GPU 0; vision tower stays high-precision. But comfy-dev's ground-truth A/B: MoE wins (faster + dual-use as general LLM + slight fine-detail edge), and the subtle-proportion tier is below ALL VLs. **Don't re-propose a dense-VL judge swap.** (auto-memory `project-qwen3vl-dense-pilot`) _Archived 2026-06-19._ - `[2026-06-15]` **MastMed prototype containerized + exposed publicly via cloudflared.** Self-contained static SPA (bun→caddy multi-stage build), basic-auth gated (operator's gating call), ana-docker:8089, cloudflared quick tunnel. See in-flight for the ephemeral-URL caveat + named-tunnel upgrade path. _Archived 2026-06-19._ - `[2026-06-15]` **infra-ops bootstrapped onto corviduo-dev** (operator-directed) — infra-ops now has SSH + NOPASSWD sudo there, OVERRIDING the fleet bootstrap script's deliberate partner-box exclusion (explicit host arg). corviduo config edits no longer need `vh` + a sudo password. (auto-memory `reference_corviduo_dev_emergency_ops`) _Archived 2026-06-19._ - `[2026-06-15]` **R18 clip+caption placement = ana-ml2** (operator: "ana-docker… ana-ml2 unless irv-ml1 has room"; irv-ml1 lacked free VRAM for the 30B-A3B captioner). Models staged + stub deployed; see in-flight + the hard real-voice gate. _Archived 2026-06-19._ - `[2026-06-14]` **LiteLLM infra-ops admin key provisioned** (operator) — resolves the LiteLLM half of the credential-migration directive; use it for ALL gateway ops (NOT `sk-corvid`). Value at `~/.config/litellm/infra-ops-key` (mode 600); gateway reachable directly from nh3-dev at 10.250.50.70:4000. (auto-memory `reference_litellm_infra_ops_key`) _Archived 2026-06-19._ - `[2026-06-16]` **litellm `strip_empty_tools` pre-call hook shipped** (`d1bea13`) — an empty `tools:[]` 500s vLLM ("tools must not be an empty array"); a global `litellm_settings.callbacks` CustomLogger pops it (+ orphaned `tool_choice`) before forwarding, so it covers EVERY vLLM model, not one. `drop_params` only drops unsupported PARAMS, not empty VALUES. Mounts beside config.yaml (litellm resolves callbacks relative to the config dir). Verified live across granite/mistral/stream. (`stacks/litellm/conf/strip_empty_tools.py`) _Archived 2026-06-20._ - `[2026-06-16]` **single-file `gateway-chat.html` playground shipped** (`984ca3d`, `tools/`) — zero-dep browser chat straight to the gateway (`:4000`, CORS open), system-prompt box, streaming SSE, renders `reasoning_content`, NEVER sends `tools`. Built because the LiteLLM admin-UI playground can't test vLLM-backed models (see Tried-and-abandoned). Serve on-request via `python3 -m http.server -d tools`. _Archived 2026-06-20._ - `[2026-06-16]` **claude-bot granted ADMIN on vh/arbo** (operator) — arbo CI/CD gitea ops now route through the service account, not the operator's personal `vh` (the credential-migration directive). Operator added claude-bot as Administrator collaborator; claude-bot self-serves via its `gitea-password` (basic-auth = FULL perms; its stored token is package-write-only, can't change repo settings). (auto-memory `reference_claude_bot_gitea_creds`) _Archived 2026-06-20._ - `[2026-06-16]` **ratatoskr Tier-3 MEMORY plane wired + live-smoke driven.** The missing server-side step was the per-PORT `BIFROST_CLIENT_ALLOWED_HOSTS` allowlist (`:8391`; consumer key reused, plane-agnostic — my earlier "reachability is the whole wiring" was incomplete). Persist + dispatch proven end-to-end against personal v0.35.3; cross-session recall-INJECTION root-caused to the `score_breakdown` shape seam and handed to worldtree-dev (#295 recall leg open). Driving a real memory-call needs a consumer agent defined WITH a `memory` block (`ctx.memory_config≠None`), not `agent_self_enabled`. (tracking: c99aa49 + auto-memory `reference_bifrost_plane_wiring`) — **[2026-06-16] FLIPPED:** the `score_breakdown`-shape root-cause was disproven; real cause = cold-recall `scope_filter` adds an `agent_self` axis → ratatoskr's conjunctive INV-005 match drops the `{end_user}`-only chunks → 0 hits. Now Worldtree #297. (auto-memory `project_worldtree_295_recall_rootcause`) _Archived 2026-06-20._ - `[2026-06-16]` **infra-ops = the DURABLE admin path on corviduo-dev** (operator granted standing admin; extends the 2026-06-15 bootstrap) — added a `Host corviduo-dev → User infra-ops` `~/.ssh/config` alias + repointed `servers/corviduo-dev/ssh-target` to `infra-ops@`; `ssh corviduo-dev` / `elway corviduo-dev` now land as infra-ops w/ NOPASSWD (vh = explicit-only human path). elway resolves via ssh-config, NOT the ssh-target file. (tracking: c99aa49 + auto-memory `reference_corviduo_dev_emergency_ops`) _Archived 2026-06-20._ - `[2026-06-16]` **demo + personal `character` model flipped qwen3.6-35-a3b → mistral-small-4** (operator) — in `model_roles.yaml` the FIRST `catalog_id` in a role's `binds:` is the default; reordered mistral-first (qwen retained in the switch-allowlist, non-destructive), applied via the PIN-SAFE api recreate (derive `WORLDTREE_IMAGE` from the matrix sibling — never a bare `compose up`, which falls to the crash-blocking stale `:latest`), verified a fresh character agent resolves model=mistral-small-4. (auto-memory `reference_corviduo_dev_emergency_ops`) _Archived 2026-06-20._ - `[2026-06-16]` **NVFP4 ModelOpt MoE now LOADS on vLLM 0.23.0 — SUPERSEDES the 2026-06-14 "blocked" finding (#44081 fixed).** Confirmed the cached `nvidia/Qwen3.6-35B-A3B-NVFP4` loads clean + generates on 0.23.0; swapped prod qwen36 FP8→NVFP4 (~13 GB reclaimed on GPU 1, fp16 KV). Validated by brokkr (speed: faster single-stream, ≥parity batched) + comfy-dev (vision anatomy-judge A/B: PASS). One W4A4 tell: minor multi-step chained-reasoning slip — doesn't bite the vision judge. Mistral's NVFP4 is a SEPARATE compressed-tensors path (day-0), distinct from the ModelOpt one. (auto-memory `reference_nvfp4_moe_loads_on_vllm_023`) _Archived 2026-06-20._ - `[2026-06-16]` **Mistral Small 4 = the GPU-0 tenant** (operator) — fills the slot the deferred creative-writing pick held, and supersedes llama-swap (now down). NVFP4 119B MoE, pinned **vLLM v0.22.0** for working vision (0.23.0 breaks it), 256K context. Operator's creative-writing model; **abliteration planned** → then it becomes the uncensored-creative model the persona substrate needs. Reasoning via `reasoning_effort` (BINARY none|high — no medium). (tracking: `c77a9aa`/`9a49963`) _Archived 2026-06-20._ - `[2026-06-16]` **Selene judge restored on vLLM FP8, NOT NVFP4** — W4A4 is too aggressive for a precision judge validated at Q6_K; FP8 ≥ Q6_K fidelity, and Selene is text-only Llama-3.1 so dynamic `--quantization fp8` has no vision tower to noise. `selene-1-mini-8b` on GPU 1 :8011. Worldtree's Domari role repointed to it. (`c985ede`) _Archived 2026-06-20._ - `[2026-06-16]` **GPU-1 buffer put to work: selene 0.13→0.17 + qwen36 0.32→0.34** — the two KV-bound generative models that gained Worldtree consumers; granite + the pooling models under-use their util, so growing THEM is wasted. (`e124a2f`) _Archived 2026-06-20._ - `[2026-06-16]` **claude-bot Gitea service account stood up; corviduo-ORG conversion TABLED** (operator). claude-bot is the credential-migration foundation. The proper fix for scoped agent package-write is converting vh (a USER) → a `corviduo` ORG — blast radius assessed (org creation = free; arbo-first migration = cheap, no registry images; full fleet = incremental, ~8 active vh images / 50 repos) — but tabled for later. Registry-push deferred until then. (auto-memory `project_migrate_infra_access_to_claude_credentials`) _Archived 2026-06-20._ - `[2026-06-16]` **arbo engine builds → comfy-dev OWNERSHIP** (operator approved) + Gitea Actions runner stood up on irv-ml1 (host-executor as lkraven). infra-ops keeps box-level escalation (runner, package:write PAT pending corviduo-org, webhook). See the runner entry in Current state. (tracking: this snapshot + arbo thread `01KV70AB2B4B…`) _Archived 2026-06-20._ - `[2026-06-16]` **Worldtree persona substrate migrated qwen→Mistral via capability- named profiles** (v0.35.8→v0.35.10; demo + personal). PRE-SYNC-FIRST is now the standing pattern for any providers.yaml/profile change: sync config to the bind-mount BEFORE the deploy image-swap restart, so new agents + new config land together (the v0.35.9 merge-first attempt race'd → 14 crash-loops). (tracking: worldtree thread `01KV6X0M…`) _Archived 2026-06-20._ - `[2026-06-19]` **litellm task-aliases: `classifier`→granite-4.1-8b, `summarizer-large`→gen/qwopus** (`summarizer`→granite already existed). Also added then REMOVED `gen-nt`/`gen-reasoning-nt` (redundant with the global `strip_empty_tools` hook). (ef45f6d, 2e5ab72, 75bd4c3) _Archived 2026-06-21._ - `[2026-06-19]` **gateway-chat: model-smoking web chat → PERSISTENT container.** `/v1/models` auto-discovery + image-upload; `stacks/gateway-chat/` (nginx:alpine, ana-docker:8091, homepage tile). (4c40b9f, 740bcae) _Archived 2026-06-21._ - `[2026-06-19]` **pi on nh3-dev wired to `gen`** — `~/.pi/agent/models.json` + a `gen` launcher mirroring `glm`. Local box config, not a repo artifact. (untracked by operator choice) _Archived 2026-06-21._ - `[2026-06-19]` **`gen` model → Qwopus3.5-122B-A10B (vision-intact NVFP4), full 256K @ fp8.** ana-ml2 GPU 0, served-name `qwen3.5-122-a10b`. STABLE vLLM + fp8 KV reaches 262144 — nightly+turboquant-4bit-KV was UNNECESSARY. 92.7 tok/s warm. (20e796c, 5b06514) _Archived 2026-06-21._ - `[2026-06-19]` **TP=2 across the two ana-ml2 Blackwells REJECTED** — `PIX` (PCIe, NO NVLink) → all-reduce-bound → SLOWER. One-model-per-card is optimal; PP=2 only if a >96 GB model is ever wanted. (untracked by operator choice) _Archived 2026-06-21._ - `[2026-06-19]` **qwen-image-bench (T2I judge, NVFP4) replaced qwen3.6-35b-a3b on GPU 1**, aliased `image-judge`. (bfae924, 5dfce04) _Archived 2026-06-21._ - `[2026-06-18]` **heretic abliterated Mistral Small 4 NVFP4 built + LIVE as `mistral-small-4`** (in-house NVFP4, drop-in stack, A/B'd vs official, "heretic stays"). DISPLACED from GPU 0 on 2026-06-19 for the gen/Qwopus swap. (dd3a5c9, f566f61, `tools/mistral-small4-nvfp4/`) _Archived 2026-06-21._ - `[2026-06-18]` **irv-ml1 VRAM consolidation + comfy-dev torch-pin** — ComfyUI pinned to the A6000 (48 GB), audio zoo → 3090; `DISABLE_UPGRADES=true` @ torch 2.12.1. (a8550ad) _Archived 2026-06-21._ - `[2026-06-18]` **ComfyUI acceleration set (9 nodes) installed for comfy-dev** on irv-ml1. _Archived 2026-06-21._ - `[2026-06-17]` **ana-ml2 `vm.overcommit_memory=1` made durable** (sysctl drop-in) — overcommit=0 + zero swap caps CommitLimit; large model-file mmap ENOMEM'd despite ~393 GB free. (fc88eff) _Archived 2026-06-21._ - `[2026-06-17]` **GLM 5.2 wired into litellm** (`glm-5.2` + `-reasoning`, z.ai passthrough, `extra_body.thinking.type` toggle). (fe77a35) _Archived 2026-06-21._ - `[2026-06-17]` **nh3-extdev stood up as a sudo-LESS infra-ops manager box** (successor to nh3-ansible); hosts `/opt/externs` pi-on-GLM-5.2 client workspaces. (a841eab) _Archived 2026-06-21._ - `[2026-06-20]` **Worldtree capability-gateway aliases stood up; NO `wt-` prefix (operator) — reuse the existing capability convention.** gen/gen-reasoning/classifier/summarizer/chat-judge/reranker/scalar-judge (passthrough)/gen-frontier/-reasoning. The **swap-transparency rule**: embedding stays model-named (`qwen3-embedding`) because vectors aren't swap-transparent; everything else can swap behind a stable alias. (d372103, 7819f96) _Archived 2026-06-25._ - `[2026-06-20]` **Removed the litellm `*` wildcard → llama-swap** (operator; llama-swap decommissioned, :9292 confirmed down). Unknown/typo'd models now 404 cleanly instead of a misleading "Connection error" — closes the long-standing footgun. (d0eb09c) _Archived 2026-06-25._ - `[2026-06-20]` **Langfuse retired** (operator) — redundant with LiteLLM native spend_logs + a 6-container crash-prone surface (the 94 GB clickhouse spew). Callbacks dropped, stack torn down. (f8eda1c) _Archived 2026-06-25._ - `[2026-06-20]` **OmniVoice streaming `/tts` — vendor chatterbox-fast's scheduler, not its whole package** (operator-approved vendor-copy). Diffusion TTFA floor finding → tuned 16-step streaming + aggressive packing; chatterbox-fast remains the lowest-latency front-end. (288d085, cd92b85) _Archived 2026-06-25._ - `[2026-06-20]` **Daily image+buildx prune cron on ana-docker** (the missing automation behind the recurring image bloat) — `lkraven` crontab, prune unused images >48 h + per-builder buildx cache. The real systemic fix (daemon log cap) is deferred to clean context. _Archived 2026-06-25._ - `[2026-06-25]` **althing re-architected to the lean multi-machine bus; nh3-extdev stood up as a MODEL B mesh peer** (dedicated `althing-svc` + group-shared `/srv/althing`). `reference_nh3_extdev_althing_mesh`. _Archived 2026-07-08._ - `[2026-06-23]` **zellij native web client piloted on nh3-dev** (`zellij-web.service` :8443) alongside ttyd. `reference_zellij_web_seat`. _Archived 2026-07-08._ - `[2026-06-22]` **Worldtree persona-render config arc (#314/#322/#317) pre-synced + deployed green on demo+personal** — #317 a boot-blocking config REMOVAL. `reference_corviduo_dev_emergency_ops`. _Archived 2026-07-08._ - `[2026-06-20]` **R22 (brokkr/dwarves) stood down to gateway-only; full-access R22 key minted; Phase B CANCELLED** (Worldtree model-agnostic → no deploy path). Key at `/home/lkraven/.r22-gateway-key` (persistent mode-600, carries paid GLM, don't delete). MUT = free `qwen3.5-122-a10b` (`gen`). Operator steer: R22 research is gated on a pragmatic/deployable outcome, not advancing-the-art. _Archived 2026-07-08._ - `[2026-06-20]` **claude-bot issue-scope token minted for worldtree-dev self-serve** (id 16, `write:repository`+`write:issue`); old token revoked. Advances the credential-migration directive. _Archived 2026-07-08._ - `[2026-06-20]` **rest-server-ana recovered + backup prevention shipped + worldtree-dev admin keys provisioned** (demo d113207c / personal f4f75adb). Cred rotation (5 rest-server pw) BELAYED. _Archived 2026-07-08._ - `[2026-06-20]` **claude-bot → ADMIN on vh/Worldtree** (operator-authorized) — self-serves WT deploys/tokens henceforth. _Archived 2026-07-08._ - `[2026-06-14]` **STANDING DIRECTIVE: migrate ALL infra access to Claude-specific credentials.** (auto-memory `project_migrate_infra_access_to_claude_credentials`) _Archived 2026-07-13._ - `[2026-07-01]` **Worldtree #332 embed-recall diagnosed + scoped-log view/tunnel provisioned + fix verified** (15×→1.01× re-embed). `reference_wt_gateway_scoped_log_view`. _Archived 2026-07-13._ - `[2026-07-01]` **qwopus native MTP speculative-decode tested on `gen` → NOT kept** (+12% single-stream, −15–20% aggregate at concurrency, silently drops min_p/logit_bias). Banked for T1. `reference_gen_qwopus_122b`. _Archived 2026-07-13._ - `[2026-07-01]` **Deckard trial → reverted to qwopus (`gen`)** (won writing "in every way" but ~36 vs ~90 tok/s; spec-decode rescue ruled out). git `b63c48b`→`681eb70`. Deckard kept staged as T1's writing benchmark. _Archived 2026-07-13._ - `[2026-07-02]` **mtf-dev granite harness-spike ran GREEN — MECHANICAL only, efficacy DEFERRED to the T1 run.** Trainer TRL SFT→DPO→eval seam proven end-to-end on a synthetic fixture (not the E-RP corpus); operator DECIDED no intermediate real-efficacy granite spike (uninterpretable proxy — arch gap + abliteration axis). `reference_gen_qwopus_122b`. _Archived 2026-07-13._ - `[2026-07-03]` **ratatoskr-dev DEMO Heimdall key provisioned (R30 φ0).** Minted a tier-user key on the demo via `POST /admin/keys` (bootstrap admin key), mirroring their personal base consumer (no character-binding); base-agent affect reads work ungated. `reference_worldtree_demo_key_mint`. _Archived 2026-07-13._ - `[2026-07-04]` **character-rp: gateway-forwarding proven + role pre-staged + #344 shipped.** Empirically confirmed per-request `extra_body` (top_k/repetition_penalty) forwards through the `gen-reasoning` LiteLLM alias to vLLM + standard params override the alias defaults — no gateway cap needed (I over-built a dedicated alias, operator corrected, reverted with zero fleet impact). Pre-staged the `character-rp` role into demo+personal bind-mount `model_roles.yaml` (byte-verified on b18; caught the cached-registry ordering). worldtree-dev shipped **#344 (v1.0.0b19)** for the durable-agent override-drop. spend_logs spot-check queued (ratatoskr's timestamp ping). _Archived 2026-07-13._ - `[2026-07-04]` **althing v2 herald+receiver formalized as systemd on nh3-dev.** `althing-herald.service` (Restart=always, **Environment=PATH incl ~/.cargo/bin** — the pane-dispatch fix) + `althing-receiver.service` (v2 → pillar-3 `/owner/*` live); stale forseti unit removed; both on v2.0.0b2, canonicalized by forseti. `reference_nh3_dev_althing_herald`. _Archived 2026-07-13._ - `[2026-07-04]` **LitBench-RM respun (irv-ml1 A6000, comfyui displaced)** for T1's reward ensemble; operator sole comfyui consumer, holding image-gen until LitBench done. `reference_litbench_rm_irv_ml1`. _Archived 2026-07-13._ - `[2026-07-05]` **T1 training venue: CLOUD recommended; operator chose smoke-first on ana-ml2.** On-prem ruled out (ana-ml2 full — both 96G GPUs ~93G used): keep-serving = NVMe offload ~6-8 DAYS; full ana-ml2 shutdown = CPU offload ~1 DAY but a whole-fleet outage. Cloud Vast.ai 8×80GB (no offload → ~3-6h, ~$60-500, zero fleet impact) is the clean alt (mtf-dev + infra-ops both rec; Vast for its no-content-AUP marketplace + likely-existing VastBlue account). Operator's next step = the ana-ml2 CPU-offload SMOKE (~60 steps) to get real samples/sec before the full-outage-vs-cloud call. HF base verified public (zero-auth pull). Runbook + gotchas in `reference_t1_cloud_train_plan`. _Archived 2026-07-13._ - `[2026-07-05]` **glm-5.2 canonical limits recorded** (probed live vs z.ai): **1,048,576 (1M) input context / 131,072 (128K) max output**; NO gateway-side cap (pure passthrough → z.ai's limits are effective). Written to the config comment (commit `624a07e`) + `reference_litellm_gateway`. _Archived 2026-07-13._ - `[2026-07-06]` **infra-ops NOPASSWD sudo confirmed on irv-ml1 + ana-ml2** (the arbo memory's "NOT infra-ops on irv-ml1" was stale) — enabled the worker deploy + every ana-ml2 hot-swap smoke. _Archived 2026-07-13._ - `[2026-07-07]` **SGLang zc hot-swap: 2-line patch RAN → PARTIAL. Cleared startup + self-attn E2E, GDN in_proj hit 2 MORE name-routing gaps → MERGE recommended.** Root cause was a NAME-routing gap (not the feared shape-assert): `lora/utils.py:235` hardcoded `params_mapping` lacked `in_proj_qkv/in_proj_z → in_proj_qkvz`. Patched those 2 lines (overlay bind-mount) → `init_lora_shapes` passed, server up, self-attn loaded end-to-end. But the SAME missing GDN mapping bit 2 MORE resolvers: adapter-load skipped 192 in_proj weights, and inference `utils.py:300 get_target_module_name` (substring match) RAISED → 500 + scheduler died. So GDN LoRA name-routing is incomplete in ≥3 SGLang paths (fixed 1). NOT shape-intractable (self-attn proves the separate-key→fused-buffer machinery works), but NOT "one targeted fix" either → per operator's LAST-shot rule → MERGE. mtf-dev looped + closed (thread 01KWZDZFYN…); B/A shard-math never reached. Supersedes the "SGLang wants a THIRD format" framing — no new adapter format needed. _Archived 2026-07-13._ - `[2026-07-07]` **T1 (retargeted qwopus-122B→AEON-27B) SFT COMPLETE on ana-ml2** — all-TRL torch-GDN, batch1/accum8/GC-off, 1 epoch ~9h @ 0.25 s/s, loss 2.445, adapter banked. Supersedes the 2026-07-05 cloud-vs-smoke venue decision: AEON-27B's 52G bf16 fits one Blackwell → ran on-prem in ~9h (no cloud, no full-fleet outage). GC is NOT the throughput lever (batch1-no-GC == batch4+GC at 0.25; dvalin's 0.565 was warmup-noise from a 30-sample smoke); ~0.25 s/s is the torch-GDN floor (only FLA beats it, sm_120- blocked #913). `reference_t1_cloud_train_plan`. _Archived 2026-07-13._ - `[2026-07-07]` **T1 hot-swap on vLLM NVFP4 is DEAD — vLLM 0.24.0 qwen3_5 LoRA-application silent no-op** (#47639 open / fix PR #47640 open / regression from #37912). Proven quant-agnostic (NVFP4+FP8) AND adapter-format-agnostic (separate `zc` loads-but-inert; fused rejected); #47640 guard-patch overlay didn't fix. Root-caused by a 3-peer dwarf panel (eitri/regin/dvalin); their vLLM issue refs verified real via GitHub API. Engine mirror-image: vLLM wants SEPARATE keys, SGLang wants FUSED. _Archived 2026-07-13._ - `[2026-07-07]` **HARD REQUIREMENT restated: hot-swappable LoRA or ABANDON.** Merge is NOT a solution for this requirement (static; reserved for a later RP-quant track). SGLang is the last hot-swap shot; if it can't apply, park the adapter + file upstream repros. `reference_lora_training_worker` (adjacent). _Archived 2026-07-13._ - `[2026-07-07]` **comfy-dev in-arbo LoRA-training worker DONE** — built/deployed/validated Phases 1+2 (first Sindra train succeeded, publish-step + `train_id`), Phase 2.5 GC contract (shape A) confirmed + impl deferred. `services/lora-training-worker/`, commits 888ba6a→b617a8b. `reference_lora_training_worker`. _Archived 2026-07-13._ - `[2026-08-08]` **worldtree-dev #400 CLOSED → fiction-decomp snapshot cleared from nh3-dev.** worldtree-dev signaled #400 done (shipped v1.0.0b185; exact-lexical efficacy 79%→12% on ratatoskr's gate, brokkr no-harm bracket green both ends; the snapshot served 4 probe rounds — rank decomposition, promoted-vs-gold annotation, tie-set falsification, A0/A1/A2 mechanism probe). Cleared `~/snapshots/worldtree-400-fiction-decomp` (208M: chroma + manifest/provenance/stamp) — a read-only rsync copy of PERSONAL Worldtree's Chroma (source on corviduo-dev, so safe to remove). **LEFT INTACT:** `rex393-fiction-index`/`rex393-fiction-snapshot` (separate operator KEEP word, unchanged) + `r42-gate-*`. No config deltas rode this train. Only remaining non-blocking await = ratatoskr-dev's chatterbox-fast knob revert. Replied confirming (`01KZJ9GMCC…`). _Archived 2026-08-24._ - `[2026-08-07]` **chatterbox-fast "broken audio" root-caused (T3 AR tail over-run) + FIXED (max_chunk_chars=250 cap, :v2 deployed).** Long saga, operator-driven clean diagnosis. **Symptom:** ratatoskr's migrated RP-surface TTS "swaps to German" / "dead air" / "garbage" on long turns. **NOT** German-leak (Turbo `generate()` has NO language param — plain AutoTokenizer, no `language_id`; the multilingual `language_id="en"` lever lives only in the separate `ChatterboxMultilingualTTS`), **NOT** OOM alone. **Real cause:** the Chatterbox **Turbo T3 model OVER-RUNS its generation tail** — a long single `generate()` degrades into garble/dead-air in its final ~2-3s (lib filters OOV tokens `<6561` + pads silence = messy AR tail). The scheduler's buffer-ratchet builds 300-600 char mega-chunks that land in that zone; streaming concatenates each bad tail (worst case). **ratatoskr's anti-"German" knobs (top_k=80/temp=0.5) made it WORSE** — tight sampling pulls the degradation onset SHORTER (~200 chars vs ~300 at default knobs). **Diagnosis method** (deterministic, no ears-only): single-shot length sweep + **amplitude-gated voiced-ZCR** (garble spikes ZCR; must gate on |x|>500 else trailing silence confounds it) — degraded voiced-tail = 1.58× mid, clean = ~0.64-1.1×. **FIX:** server-side `max_chunk_chars=250` cap on the scheduler (`:v2` image, `CBF_MAX_CHUNK_CHARS=250` env) — bounds each generation to just under the ~300-char onset → clean **3-4 sentence** chunks (max prosodic arc while clean). Operator ear-confirmed clean audio + clean joins; **chatterbox's low emotiveness keeps chunk joins smooth** (the harsh joins that got Zonos rejected are absent — operator's key call). **ratatoskr TODO (relayed msg `01KZER9X7S`):** revert knobs to default (top_k→1000, temp→0.8), send full text (server chunks internally), keep the 503-on-empty guard. **Cap value tunable** per-request (`max_chunk_chars`) + env. **Deeper prosody** (if ever wanted) = scheduler Phase-2 context-priming at joins (feed prior sentence as discarded-audio context; +latency). **⚠ FOOT-GUNS:** (1) acoustic tail-trim is UNRELIABLE — sibilants ('s'/'sh'/'f') spike ZCR like garble, can't cleanly detect the speech→garble boundary. (2) **build-context vs image drift** — the `:v2` image was built from cap source, but after a `:v1` rollback the build context held `:v1` source → a `docker compose build` would've silently produced a cap-less `:v2`; re-synced the flat cap source to `/opt/docker/compose/chatterbox-fast/` (rebuild-verified). **⚠ DIVERGENCE (follow-up):** deployed build context is FLAT (`app.py`/`scheduler.py`, `from scheduler import`, thin-overlay `FROM local/chatterbox:v1`, cap-only) vs the `vh/chatterbox-fast` REPO which is PACKAGE-layout (`chatterbox_fast/`, `from chatterbox_fast.scheduler`, self-contained Dockerfile) + has `norm_loudness` (repo commit `6bc7bf0` = cap; deployed omits norm_loudness deliberately to keep the ear-test unconfounded). Reconcile the two layouts so a repo-based rebuild matches deploy. Rollback: `.bak-cap-20260807-104850` backups on irv-ml1 + `:v1` image both retained. [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]] _Archived 2026-08-24._ - `[2026-08-07]` **Zonos2 TAKEN DOWN on the 3090 (irv-ml1) — operator-directed "for memory", TEMPORARY.** Freed ~17.4 GB (3090: 728 MiB → 18.2 GB free) so chatterbox-fast (co-resident, was OOMing on long generations) has headroom. **⚠ Restore is manual — Zonos2 :1920 was a DETACHED native process (NOT systemd/docker), reparented to init.** GPU memory was held by the `--multiprocessing-fork` CHILDREN (1966165=16.4G, 1966166=1G), which ORPHAN to init when you kill the parent — had to SIGTERM the children explicitly (killing the parent 1965942 + uv-run 1965935 alone left the 16.4G held). **RESTORE CMD** (from irv-ml1, user lkraven): `cd /home/lkraven/tts-audition/models/zonos2 && nohup uv run python -m zonos2 --model-path Zyphra/ZONOS2 --host 0.0.0.0 --port 1920 --tts-default-voices-dir ./default_voices/ --cuda-graph-max-bs 1 --num-pages 16384 --max-running-requests 2 --memory-ratio 0.3 > /tmp/zonos2.log 2>&1 &` then `docker start zonos-gateway`. **Consumers that lost Zonos:** asset-engine + gateway-chat (via LiteLLM `ext-tts` alias → zonos-gateway :8890, now stopped); ratatoskr already migrated OFF to chatterbox-fast (unaffected). Also unblocks proper drift/cap testing (OOM was blocking it). [[reference_zonos_tts_stack]] _Archived 2026-08-24._ - `[2026-08-07]` **chatterbox-fast: donut voice added + full contract delivered to ratatoskr-dev (their TTS migration off Zonos).** Operator-directed. Copied `zonos-gateway/voices/Donut.wav` → chatterbox `/refs` (`/worktank/chatterbox/reference_audio/donut.wav` — the reference_audio SUBDIR is lkraven-owned so no sudo despite `/worktank` root; container globs `/refs` live → **NO restart**), exposed as `voice:"donut"` (lowercase); verified clean 7.5s synth (24kHz, RTF ~0.31). A/B booth (chatterbox vs zonos donut, same line) at `http://10.100.10.50:8090/b/donut-chatterbox/`. Answered ratatoskr's 8-question contract ask from the live gateway (`local/chatterbox-fast:v1`) + source: **NOT OpenAI-shaped** (`POST /tts`; body `text`/`voice`/`format`/`stream`, not `input`/`model`/`response_format`); **NO affect dials** (Turbo ignores cfg_weight/min_p/exaggeration — the architecture-changing answer they flagged; **Zonos stays the only fleet TTS with real emotion steering**); streaming WAV placeholder-header shape IDENTICAL to Zonos (their per-chunk Web Audio path survives); SR 24000 (Zonos 44100); server chunks arbitrary-length text internally (no client-side chunking, unlike Zonos's 71.2s cap); English-only, no language pin. **FYI-worthy (operator):** ratatoskr is moving its RP-surface TTS OFF Zonos back to chatterbox-fast → loses the live-PAD affect coupling (heavy Zonos emotion investment) — their call, trade-off flagged to them. auto-memory `reference_chatterbox_fast_repo` enriched w/ the live contract. [[reference_zonos_tts_stack]] _Archived 2026-08-24._ - `[2026-08-07]` **Fleet reranker cut over: Qwen3-Reranker-0.6B → BAAI/bge-reranker-v2-m3 (Brokkr R43).** The incumbent was measured HARMING 80/90 fleet queries (no-reranker beat it 89/90 vs 56/90). R43 bake-off: the A2 control (same Qwen weights, seq-cls head) scored identical to the incumbent → proved the fault is a training-prior not the serving head → cancelled the expensive Qwen3-4B arm; A3 (bge-v2-m3) won on multilingual safety + bare-name recovery. LiteLLM `reranker` repointed incumbent→A3 :8013 (boundary 2026-08-06T17:37:48Z, config-edit + ~52s gateway restart); **R42 v13 gate PASSED first-ever** (56/90→90/90). Incumbent kept warm :8002 (rollback via `qwen3-reranker` alias), A4 fallback :8014. Full arc + rollback runbook `docs/pfi/reranker-selection-ledger.md`; commits ad2df89/2c11748/377f8a4 (unpushed). auto-memories: the earlier reranker-serving notes. _Archived 2026-08-24._ ## Tried and abandoned (archived) # [2026-08-15] Uncensored gen seat: Qwen3.8-27B-Uncensored deployed; the definitive MTP-graft fix **Outcome.** The fleet `gen` seat is now **`JonathanColetti/Qwen3.8-27B-Uncensored`** (Heretic abliteration, KL 0.12 vs base, bench Δ −0.5 within noise, refusals 98→12/100), quantized in-house to **NVFP4 W4A16** (llm-compressor / compressed-tensors) with a **grafted bf16 MTP head**, vision-intact, **262K** ctx, MTP n=3 (**~42% accept, ~68 tok/s**), coherent. Live at ana-ml2 `:8015` (project `gen-seat` / container `vllm-gen`), backing all 7 gateway aliases. **THE definitive lesson (resolved 3 failed attempts + one premature 50 GB delete).** A grafted bf16 MTP scored **0% on the quant but 83% at bf16** — for TWO different abliterated models. Root cause was NEITHER the abliteration NOR the quant scheme: it was **the grafted `mtp.*` tensors missing from `config.json` → `quantization_config.ignore`.** The wrapper-class quant DROPS the MTP before llm-compressor sees it, so nothing gets added to `ignore`; vLLM then tries to load the bf16 MTP as *quantized* format → "Parameter … not found in params_dict, skip loading" → uninitialized head → 0%. **FIX: after grafting, add `re:^mtp.*` to `quantization_config.ignore`** (one line — all unsloth's working checkpoint has). MTP jumped 0%→83% (bf16-identical). Full lesson in auto-memory `reference_abliteration_mtp_lessons`. **The pipeline that works (for the next VL+MTP quant, incl. the W4A8 chase):** 1. Pull bf16 (kept at `ana-ml2:/tank/aimodels/qwen38-27b-uncensored-bf16`). 2. Quant via `quant_nvfp4_qwen.py` (darkscarlett dir) = the **wrapper-class** loader (`Qwen3_5ForConditionalGeneration`, keeps the vLLM-serveable config); container = `vllm-openai` + `pip install llmcompressor==0.13.0` (drags in a transformers with `qwen3_5`). 3. **Graft** the author's `model-mtp.safetensors` verbatim into the output + merge the index. 4. **Reconstruct** `preprocessor_config.json` from `processor_config.json`'s `image_processor` sub-dict (the repo omits it → else "Can't load image processor" crash-loop). 5. **Add `re:^mtp.*` to the output config's `quantization_config.ignore`.** ← the fix. 6. Serve: `--quantization compressed-tensors --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'` `--mamba-cache-dtype float32 --kv-cache-dtype fp8 --reasoning-parser qwen3`. **VRAM / full-context budget (measured).** Weights ~27 GB; hybrid attention → **only 16 of 64 layers carry KV** → 32 KiB/token → **262K KV = 8.6 GB** (vs ~60–70 GB for a normal dense 27B). Full 262K fits GPU0 at **util 0.45** (~43 GB) alongside meromero (~49 GB used, it's a 31B) — pre-flight rejects util 0.48 (wants 45.6 GB, only 45.5 free). `max-num-seqs 16` keeps cudagraph modest (an ad-hoc serve with no cap OOM'd — cudagraph captured to batch-512). **Why unsloth's `qwen3.8-27b` (the prior gen model) was faster (97 vs 68 tok/s).** ~half = quant kernel (unsloth native NVFP4+FP8 tensor cores vs our W4A16 → Marlin dequant, ~20% even on decode — I'd under-stated this); ~half = MTP acceptance (unsloth 55% un-ablated head vs our 42% — inherent to the ablation, no quant fixes it). **W4A8 recovers the first ~20% (→~82 tok/s) + prefill; not the MTP half.** **modelopt dead-end (for W4A8, avoid).** `nvidia-modelopt[hf]==0.43.0` is too old for qwen3_5's transformers: (a) its `NVFP4_DEFAULT_CFG.quant_cfg` is a LIST but 0.43 wants a DICT (pydantic reject); (b) it warns transformers 5.15 untested. Use **llm-compressor** for W4A8 instead (custom recipe: NVFP4 weights + FP8 input_quantizer + calibration on `heretic2-nvfp4-work/production_calib_512.jsonl`). **Deleted (premature — the delete I owned).** `windowsxp811203/Qwen3.8-27B-Abliterated` (~79 GB) — I declared it desync-dead off a 0% that was actually this ignore bug. Lesson: **test MTP on bf16 first; isolate before deleting.** Commits: eshpfi `680c30e` (deploy + rename + litellm + README), dotfiles `1d1970f` (CLAUDE.md roster) — both UNPUSHED. Related: [[reference_abliteration_mtp_lessons]], [[reference_verify_hf_repo_ids_before_pull]]. _Archived 2026-09-02._ - `[2026-04-30]` task-board workflow with `container: image: debian:bookworm-slim` — fails: `actions/checkout@v4` needs `node` at runtime, slim image lacks it. Switched to `node:20-bookworm-slim` (has node + apt) or runner-label default. (Pattern revisited 2026-05-17 for skaldsong-dev: container override needs `nodejs` apt-installed unless it IS the default.) _Archived 2026-05-27._ - `[2026-04-30]` Dropping the `container:` directive before runner re-registration with docker-schema labels — runner silently falls back to **host mode** (jobs run inside the alpine `act_runner` container itself, no apt). The `:host` suffix in startup logs (`labels updated to: [pfi-fleet:host ana-docker:host]`) is the giveaway. Fix: register with `pfi-fleet:docker://` schema labels. _Archived 2026-05-27._ - `[2026-04-30]` Updating runner labels by editing `.env` and bouncing — doesn't take. The `.runner` registration cache pins labels at first registration; env-var updates are read each start but the stored token + UUID are tied to the original label set on the gitea side. Fix: stop runner, delete `.runner`, generate new admin registration token, redeploy. _Archived 2026-05-27._ - `[2026-04-30]` `git reset --hard origin/` in `deploy-task-board.yaml` (and the in-repo nevermore playbook before fix) — invalid syntax: `origin/` prefix only works for branch refs. SHAs need `git reset --hard ` directly. Resolved with `git rev-parse --verify --quiet "origin/{{ ref }}^{commit}"` first, then bare `"{{ ref }}^{commit}"` fallback. _Archived 2026-05-27._ - `[2026-04-30]` Assuming `DEPLOY_SSH_KEY` was at user scope after task-board wiring — it was actually only repo-scope on `vh/task-board`. vor's first CI run failed with empty SSH key (`printf '%s\n' "" > ~/.ssh/id_ed25519`). Fix: copy secret to user scope at `gitea.phasefinal.com/user/settings/actions/secrets`. _Archived 2026-05-27._ - `[2026-04-30]` `grep -vE "^(#|$)"` to inspect `.env` for sanity — leaked the full `MINIFLUX_PASSWORD` line into the transcript. Then a follow-up redaction attempt with `sed -E "s/=(.{4}).*$/=\1/"` still leaked the first 4 chars. Lesson: when probing secret-bearing files, use field-by-field SELECTIVE inspection (`grep -E "^(KEY1|KEY2)="`) rather than negative filters; for any password line, `grep -c` (existence) or `test -n "$(...)"` (non-empty), never `cat` or value-printing. _Archived 2026-05-27._ - `[2026-05-08]` Filtering Traefik's UTC access log by Gitea-local-PDT timestamp substrings (`grep "2026/05/08 15:1[2-7]"`) returned zero matches and led to a wrong "no /v2/ traffic in 12 days" conclusion. **Gitea logs in PDT, Traefik logs in UTC** — same host, different timezones. Always normalize timezones (UTC) when correlating logs across services on the same box. Cost: ~30 min in the wrong direction. _Archived 2026-05-27._ - `[2026-05-08]` Bumping Gitea `PER_WRITE_TIMEOUT` / `PER_WRITE_PER_KB_TIMEOUT` to address `unexpected EOF` on `/v2/.../blobs/uploads/` PATCH — wrong direction. Both govern **response writes**, not request body reads. `unexpected EOF` from Go's HTTP server means the client closed mid-body-upload; not a knob Gitea exposes server-side. _Archived 2026-05-27._ - `[2026-05-12]` Defaulting asset-engine to Traefik-routed (`asset-engine.phasefinal.com` with `anaprod` cert resolver) on first scaffold — user pulled it back to LAN-direct. Internal tools default LAN-direct; only add Traefik when an external/TLS surface is actually needed. _Archived 2026-05-31._ - `[2026-05-12]` Routing althing thread replies through `galdrabok` when the actual dev handle is `forseti` — bus rejected `to=forseti` initially because thread participants list was `[galdrabok, infra]`. Solved by starting a new thread with `forseti` as the direct recipient. Lesson: when the bus auto-resolves a sender handle that doesn't match the actual dev role, start a fresh thread rather than fighting the participant list. _Archived 2026-05-31._ - `[2026-05-13]` Initial Voxtral default voice `alloy` (OpenAI-compat naming) — vLLM-Omni serving Voxtral does NOT translate aliases. Native presets are `_` shape (`neutral_female`, `casual_male`, etc.). Always live-probe `/v1/audio/voices` for the exact wrapper-deployed preset names before setting a catalog default. Same caveat for Qwen3-TTS (wrapper exposes 15 voices: 9 Qwen presets + 6 OpenAI aliases) and Kyutai-TTS (NillPointer wrapper has NO voice-listing endpoint at all; voices are filesystem paths under the `kyutai/tts-voices` HF repo). _Archived 2026-05-31._ - `[2026-05-17]` `--task classify` for Skywork in vLLM 0.19.1 — flag was deprecated. Use `--runner pooling`; the model's `architectures: [LlamaForSequenceClassification]` in config.json drives the classification head. Surfaced as `vllm: error: unrecognized arguments: --task classify` in container logs. _Archived 2026-05-31._ - `[2026-05-17]` Trusting that `.env` edit alone propagates a new env var into a worldtree container — `compose.yaml`'s `&worldtree-env` anchor must explicitly declare the passthrough or the value silently doesn't land. Same footgun bit `Z_AI_API_KEY` (2026-05-12) AND `BIFROST_CLIENT_ALLOWED_HOSTS` (2026-05-17). Cost ~10 min of "why is env empty?" diagnosis each time. Worldtree-side fix in `vh/worldtree@08f02b2`. _Archived 2026-05-31._ - `[2026-05-17]` `--force-recreate --pull never` from the docker:cli sandbox without explicit `-e WORLDTREE_IMAGE=` re-pins the container to `:latest`, even when a newer SHA-tagged image is on disk. Symptom: container "recreated" but actually reverted to a stale image. Pass `-e WORLDTREE_IMAGE=...:` to the docker run invocation. Worldtree-dev's `8ef3801` health-gated `:latest` advance is the long-term fix. _Archived 2026-05-31._ - `[2026-05-18]` Volva env.sh.template `$HOME` in commented examples — systemd's `EnvironmentFile=` parser doesn't expand `$HOME`; uncommenting lands the literal `$HOME/...` string. Volva-dev's `f4dda73` swapped to `/home//...` placeholders. _Archived 2026-05-31._ - `[2026-05-18]` Initial Volva systemd unit's `ProtectHome=read-only` without `ReadWritePaths=` — althing-cli's SQLite (`~/.althing/ althing.db`) and codex's session state (`~/.codex/`) both need to write. Container started but every poll failed with "db path not writable". Surgical fix: `ReadWritePaths=/home/lkraven/.althing /home/lkraven/.codex` (preserves the hardening intent, only carves out the specific dirs). _Archived 2026-05-31._ - `[2026-05-18]` Trusting that env.sh's `export VOLVA_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"` template line works under systemd — `EnvironmentFile=` parser aborts on the first unparseable line (command substitution), and `VOLVA_ALTHING_CLI` declared below silently never lands. Symptom: `Environment=` property empty, daemon error "althing-cli not found at 'althing-cli'". Fix: replace command-substitution with literal path. Volva-dev's `d436c3c` dropped VOLVA_ROOT entirely upstream. _Archived 2026-05-31._ - `[2026-05-19]` Naive `docker rmi worldtree: --force` for CD SHA cleanup — would untag pinned/personal worldtree images since all three deployments share corviduo-dev. Use `docker image prune -a --filter "until=Xh"` instead — respects in-use semantic (Docker won't remove an image referenced by any container on the host), so pinned/personal protected automatically. _Archived 2026-05-31._ - `[2026-05-19]` Skaldsong CD first attempt: `docker pull` step failed with 401 unauthorized. ana-docker had no `docker login` for `gitea.phasefinal.com`. My playbook prereq note ("docker login has been done at least once") was an unverified assumption. One-time manual login persists in `~/.docker/config.json`; architectural fix (workflow-side `ssh ana-docker 'docker login ...'` step using `REGISTRY_USER`/`REGISTRY_TOKEN` secrets) flagged as v2. _Archived 2026-05-31._ - `[2026-05-19]` `SKALDSONG_HOST_CORS_ORIGINS=http://10.250.50.70:8300` as a bare URL — pydantic-settings parses complex env vars via `json.loads()`; first-boot crashloop with `SettingsError: error parsing value for field "cors_origins"`. Must be JSON array literal: `SKALDSONG_HOST_CORS_ORIGINS=["http://..."]`. _Archived 2026-05-31._ - `[2026-05-19]` `SKALDSONG_HOST_STATIC_ASSETS_PATH=/app/web/dist` in compose — mismatched Dockerfile reality. The Dockerfile COPYs SvelteKit build output flat into `/app/spa` (not `/app/spa/dist`). Lifted the path from skaldsong-dev's CD-ask message ("`/app/web/dist`") rather than verifying against the actual Dockerfile they shipped. Lesson: when encoding container-internal paths in compose, verify against the Dockerfile, not the design-doc. _Archived 2026-05-31._ - `[2026-05-19]` Playbook verify step `docker ps | grep healthy` racing the container's `start_period` (30s in compose's healthcheck). Verify ran 0.09s after `compose up -d --force-recreate` — well before docker's healthcheck could flip the status from `(health: starting)` to `(healthy)`. False-negative; container was operationally up (the earlier `/health` poll verify already confirmed). Fix: grep `^Up` not `healthy`. /health-200 IS the liveness check; docker's `(healthy)` is just a delayed echo. _Archived 2026-05-31._ - `[2026-05-20]` `SKALDSONG_DB_PATH` + `SKALDSONG_RUNS_DIR` in compose env block — names skaldsong's app doesn't read. App reads `SKALDSONG_HOST_SQLITE_PATH` + `SKALDSONG_HOST_RUNS_ROOT` (per Dockerfile ENV defaults). Wrong names = silently no-op; app fell back to Dockerfile defaults pointing at `/app/data/...` which the compose's bind mount did NOT cover (target was `/app/state/...`). Result: every `--force-recreate` wiped the SQLite DB. Caught by skaldsong-dev (althing thread `01KS4DPF6SXTBP4Q360JZVWPNT`). Fix in `52e98fa`. Lesson: verify env var NAMES against the Dockerfile/app, not against design-doc shorthand. _Archived 2026-05-31._ - `[2026-05-25]` First selene-block patch put the block under `saga_allowed_models:` instead of top-level `models:` — used `text.replace("models:\n", ...)` which substring-matched the `saga_allowed_models:\n` line first. Caused YAML parse error. Fix: anchored regex `re.compile(r"^models:\n", re.MULTILINE)`. Pattern: substring replace on YAML top-level keys WILL match suffix-containing keys. _Archived 2026-05-31._ - `[2026-05-27]` `docker compose up -d` inside the `docker:cli` sandbox: `${VAR:-./config}` defaults resolve `./config` to the sandbox CWD, but the Docker daemon interprets the path against the HOST filesystem → auto-creates an empty dir → entrypoint reseeded image-baked defaults (lost host-side providers.yaml patches). Fix: pass `-e WORLDTREE_CONFIG_DIR=/abs/path`. Folded into the docker-as-root convention note. _Archived 2026-06-01._ - `[2026-05-27]` `:latest`-pinned compose + private gitea registry + sandboxed pull = recreate on ancient cached `:latest` (deploy pulls by SHA so the tag never advances; sandbox can't pull). Fix: retag SHA→`:latest` on host, then `--pull never`. Better: pin SHA in `.env`, advance in CI. _Archived 2026-06-01._ - `[2026-05-27]` Container recreate during in-flight skaldsong gen kills the runner. With deploys every ~10min and stories >5min, structural not incidental. Roadmap (skaldsong-dev): pre-shutdown signal handler, per-scene resume-from-checkpoint, /api/admin/quiesce. None shipped. _Archived 2026-06-01._ - `[2026-05-27]` `--reasoning-format gemma` on artemis-31b-v1i — unsupported in the deployed llama.cpp (accepts none|deepseek|deepseek-legacy). `deepseek` populates the `reasoning_content` SSE delta Worldtree GemmaProvider checks. _Archived 2026-06-01._ - `[2026-05-27]` `head -c N` piped after a streaming curl SIGPIPEs the curl, killing the request early. Use file-write + separate read. _Archived 2026-06-01._ - `[2026-05-31]` Building the dia2-capable image surfaced THREE upstream packaging quirks: (1) `pip install -e nari-labs/dia2` fails — no PEP 660 `build_editable` hook; (2) plain `pip install` builds an empty `UNKNOWN-0.0.0` wheel (base setuptools 59.6 < dia2's required ≥70); (3) `--no-deps` leaves `transformers`/`sphn`/`whisper-timestamped` missing. Fix (`local/dia:v2`): copy the pure-python `dia2/` package into site-packages + install ONLY those 3 deps; base torch/numpy already satisfy Dia2. _Archived 2026-06-03._ - `[2026-05-31]` Dia2 predefined voices (43, baked at `/app/voices`) are NOT reachable from the `/tts` clone path — it resolves `clone_reference_filename` against the reference_audio dir ONLY. The OpenAI `/v1/audio/speech` `voice` param auto-resolves them (separate code path), which masked the gap. Fix: stage `/app/voices/*` into `/worktank/dia/reference_audio`. Lesson: verify on the endpoint the catalog ACTUALLY targets. _Archived 2026-06-03._ - `[2026-05-31]` `voice_mode=clone` with an empty `clone_reference_filename` → asset-engine serializes it as the literal string `"undefined"` → `/tts` 404. First observed on dia2; worked around in the catalog (default the field to a real voice). **[2026-06-01] root cause found** — the Kokoro voice-blend widget reading Shoelace `.value` before hydration (see Current state); the real fix is asset-engine-side and is escalated. _Archived 2026-06-03._ - `[2026-05-31]` asset-engine `services.schema.json` is DERIVED (regen from the Pydantic model via `dump_schema.py`) and had DRIFTED — rejected the `lifecycle` field 12/14 services use. RESOLVED: regenerated with `CatalogLifecycle` + `reproducibility.seed_field` (`b7b2130`). Lesson: hand-editing `services.yaml` shape without regenerating re-introduces drift. _Archived 2026-06-03._ - `[2026-05-31]` ttyd-over-TLS forces HTTP/2 (kills ttyd's terminal WebSocket → blank screen); Safari/WebKit never sends HTTP basic-auth on WS upgrades. Both solved for the fleet seat (Caddy forces HTTP/1.1; auth → network-gating) — detail in auto-memory `reference_ttyd_fleet_seat`. _Archived 2026-06-03._ - `[2026-05-30]` esh-docker-vm: `hard` NFS mounts from 10.0.50.50 froze a container worker in UNKILLABLE D-state when the NAS stalled — only a host reboot clears it. Separately, `fstab defaults` (no `_netdev`) made NFS-bind containers `Exited(255)` on reboot. → auto-memory `incident_esh_docker_nfs_boot_race`. _Archived 2026-06-03._ - `[2026-06-02]` **Naive cumulative re-decode for streaming** — decode the growing token prefix each chunk + emit the delta, assuming the causal flow gives a stable prefix. It does NOT (full-context attention, see above); maxdiff 0.30. Also the high-level `s3gen.inference` re-applies `trim_fade` at the start every call. Don't go this way. _Archived 2026-06-04._ - `[2026-06-01]` **CSM bring-up** — upstream `phildougherty/sesame_csm_openai` Dockerfile pins NO `huggingface_hub` version, so it now resolves to 1.17.0 where `huggingface-cli` is fully removed (replaced by `hf`) → the build dies at `huggingface-cli login`/`download`. Passing the HF token as a build-arg got the token in (the model downloads at BUILD time, not runtime — our compose only wired it as runtime env, the original failure) but then hit the rotted CLI. Fixing would need vendoring + patching the Dockerfile; not worth it for a weak model → CSM removed. _Archived 2026-06-04._ - `[2026-06-01]` **Fish path-form references** (`{audio:"/app/references/x.wav"}`) → HTTP 500 on this build, despite the stack README showing that shape. The working clone paths are `reference_id` (by name → resolves the DIRECTORY `references//` containing `.wav` + `.lab`; the flat `references/.wav` is ignored — see the two-root-causes entry above) or inline base64 `references`. The catalog uses `reference_id`. _Archived 2026-06-04._ - `[2026-06-01]` **Reference transcript barely affects Fish clone IDENTITY** (disproving my mid-session theory). I'd blamed a bogus provenance-note `.txt` for poor cloning, but the ECAPA re-test showed correct (0.787) / bogus (0.778) / empty (0.738) transcripts all clone Imogen about equally — the transcript affects PRONUNCIATION (phoneme disambiguation per the docs), not who it sounds like. The real culprit for "not British" was the `"undefined"` select bug, not the transcript. (A correct transcript still marginally helps pronunciation — cheap to stage, not load-bearing.) _Archived 2026-06-04._ - `[2026-06-01]` **Pitch-shift register control** (rubberband, to deepen Imogen to contralto/mezzo) — Fish ignores small reference shifts and overshoots large ones; `-3st` WITHOUT formant preservation → "baritone through a tuba" (formants collapsed); `-2/-3st` WITH `--formant` → too subtle / still bad. Abandoned at every depth; all variants deleted. Finer independent pitch/formant control needs praat (not installed). Future clones = operator's. _Archived 2026-06-04._ - `[2026-06-01]` **resemblyzer is too weak for cloning-fidelity scoring** — its dated 2019 LSTM encoder + a synthetic-vs-natural domain gap scored the Imogen clone CLOSER to the default than to real-Imogen, which led me to a WRONG "Fish is a weak cloner" call. ECAPA-TDNN (speechbrain `spkrec-ecapa-voxceleb`) on the same clips gave the correct answer (clone 0.79 to real Imogen, 0.10 to default). Use ECAPA, not resemblyzer, for speaker-verification. _Archived 2026-06-04._ - `[2026-06-02]` **Fish (fish-s2 / OpenAudio S1-mini) progressive streaming — SHELVED (sub-realtime).** Benched RTF on A6000: 0.72x/0.82x/0.86x, mean 0.80x = sub-realtime, so client-side chunking starves. Root cause: Fish only chunks on `<|speaker:X|>` tags; plain text → batches=[whole text] → all semantic tokens generate before any audio (chunk_length inert). Plus kui/ASGI StreamResponse doesn't flush (fix = anti-buffering headers, kept on file). STANDING REVISIT TRIGGER: when an RTX Blackwell Pro lands → bench fp4-quantized Fish; if RTF > ~1.5x give it the chatterbox-fast treatment. For now Fish is a buffered catalog entry (great for SAVED gens, not live-audition). _Archived 2026-06-11._ - `[2026-06-02]` **Context-priming at chunk joins (chatterbox-fast §1.6) — ABANDONED (discard-cut leaks the prefix).** Prepend prior sentence, generate `prefix+content`, discard prefix audio. Built + shipped (`d707439`), live-A/B'd, reverted (`090e70a`). Kill: `generate()` returns one waveform with NO marker where the prefix ends, and the model renders the same prefix with different timing solo vs followed-by-content — so locating the cut is a guess that left a whole clause in the output (operator caught "...without a trace of sarcasm" spoken twice). Reliable cut needs token-level boundaries or per-chunk forced-alignment (heavy). Coherence loss at joins = ACCEPTED limitation. _Archived 2026-06-11._ - `[2026-06-02]` **Native frame-level streaming on Chatterbox-TURBO — ABANDONED (turbo isn't built for streaming).** (1) `S3GenStreamer` is docstring-only (unimplemented). (2) `flow_inference(finalize=False)` is buggy (lookahead trim removes frames from `h` not `h_masks`/conds → shape mismatch); a 1-line patch makes it RUN. (3) BUT the flow encoder uses full-context attention (`static_chunk_size=0`) → incremental decode is prefix-unstable (adding tokens re-attends, shifts earlier mel ~0.30-0.39 vs one-shot). (4) Forcing `static_chunk_size>0` didn't stabilize it. True sub-second frame-streaming needs deep attention surgery with quality risk — not worth it. → adaptive-chunking instead. _Archived 2026-06-11._ - `[2026-06-03]` **`uv pip install .` fails on SmoothKen/knn-svc** (and similar script-repos) — it's analysis scripts + a poetry pyproject, no buildable package (setuptools package-discovery error). Install the pyproject deps directly, don't build the "package". _Archived 2026-06-13._ - `[2026-06-03]` **Backgrounding `althing-cli monitor` with an inline shell `&` (instead of the Bash-tool `run_in_background`) orphans it** — it survives the shell exit, holds the per-handle flock UNTRACKED (won't notify the session), and `stop-monitor` doesn't detect it. Fix: find + kill the orphan PID (verify cwd=this repo / handle first — nh3-dev is shared, other agents' monitors run there too), then re-arm via run_in_background. Always re-arm tracked. _Archived 2026-06-13._ - `[2026-06-03]` **torch-2.12 venvs need `uv pip install torchcodec`** — torchaudio 2.12 defaults to the TorchCodec backend for `.load`; without it, real audio I/O throws "TorchCodec is required" — and it ONLY surfaces at actual conversion, NOT at import/model-load. Lesson: validate real I/O, not just import, when provisioning ML engine envs. (seed-vc on torch 2.4 uses the legacy backend, exempt.) _Archived 2026-06-13._ - `[2026-06-03]` **gitea webhook to a private IP is denied by `webhook.ALLOWED_HOST_LIST`** (anti-SSRF; default `external` blocks private/loopback). Symptom: delivery shows `dial tcp ...: webhook can only call allowed HTTP servers`. Fix = APPEND the target net to ALLOWED_HOST_LIST in gitea's app.ini (keep `external`; scope tight, never `*`/`private`) + restart gitea (act_runner job containers survive a restart). gitea runs as a container on ana-docker (`gitea_gitea_data` volume, `/data/gitea/conf/app.ini`). _Archived 2026-06-13._ - `[2026-06-04]` **homepage "crash" ≠ always NFS** — a wedged container in unkillable D-state can come from dead `siteMonitor` widget targets (retired ESH firewall IPs) hanging the node event loop into `exit_mmap`, needing a host reboot. Check homepage's siteMonitors against retired hosts. (`incident_esh_docker_nfs_boot_race`) _Archived 2026-06-14._ - `[2026-06-04]` **pyworld needs `setuptools<81`** (imports the removed `pkg_resources`); and **R/soundgen `-lgfortran` fails** on irv-ml1 (default gcc is gcc-11 but only gfortran-12 present) → install `libgfortran-11-dev`. _Archived 2026-06-14._ - `[2026-06-04]` **Persistent services on irv-ml1 need SYSTEM systemd units** — the box reaps user-session processes on ssh disconnect, and `--user` systemd isn't reachable over non-login ssh, so nohup/setsid/`screen -dmS`/`systemd-run --user` all die (even with enable-linger). Use `/etc/systemd/system/`. _Archived 2026-06-14._ - `[2026-06-04]` **GPU pin by INDEX is ambiguous on irv-ml1** — native CUDA orders fastest-first (A6000=0) but nvidia-smi/docker use PCI order (A6000=1). Pin by **UUID** (`CUDA_VISIBLE_DEVICES=GPU-…`); verify via nvidia-smi compute-apps. _Archived 2026-06-14._ - `[2026-06-04]` **Ollama/llama.cpp-BUNDLED chat templates silently diverge from canonical HF — the "headgun" lesson.** Serve each model's canonical `tokenizer.apply_chat_template`, not the bundled template (bundled ones corrupt baselines). Verify via vLLM `/tokenize`→`/detokenize`. (`90e08f0`/`27eb537`) _Archived 2026-06-14._ - `[2026-06-08]` **Demucs `uv pip install demucs` pulls torch 2.12/torchaudio 2.11 → `ta.save()` requires torchcodec → dies AFTER separating** (0 stems, rc=1). Fix = pin `torch==torchaudio==2.4.1` + `UV_LINK_MODE=copy`. Validate the SAVE path, not just import + GPU inference, on a bleeding-edge torch. _Archived 2026-06-16._ - `[2026-06-05]` **vLLM 0.19 CUDA-graph-capture OOMs on a SHARED GPU** — fills KV to the `--gpu-memory-utilization` budget WITHOUT reserving graph-capture memory, so `capture_model` OOMs AFTER weights+KV load (crash-loops). Fix: free co-tenant room OR `--enforce-eager`. FP8 single-stream is batch-1 GEMV (memory-bound) → Q4 wins single-stream by physics; FP8 wins under concurrency. (`reference_ana_ml2_vllm_granite`) _Archived 2026-06-16._ - `[2026-06-05]` **Langfuse has NO public dashboard-creation API** — dashboards/widgets are postgres rows; clone a default + swap the measure. tok/s is the `outputTokensPerSecond` MEASURE (metrics-API/dashboard query time), not a per-generation field. langfuse-web needs `HOSTNAME=0.0.0.0`. Host 3000 is gitea's → langfuse on 3001. _Archived 2026-06-16._ - `[2026-06-05]` **`sudo` over non-interactive ssh FAILS SILENTLY where the user lacks NOPASSWD** (esh + corviduo are OUTSIDE the infra-ops identity) → empty output misread as "empty file." Read world-readable files WITHOUT sudo. corviduo ssh = `vh@10.250.50.152`; bind-mount configs are vh-owned, the `.env` is deploy-owned 600. _Archived 2026-06-16._ - `[2026-06-05]` **Worldtree summarizer-model is NOT an env var** — no `WORLDTREE_SUMMARIZER_MODEL`; defaults to claude-haiku in code, opt-in via config not `.env`. Inspect the live container env + vh-owned config files first. _Archived 2026-06-16._ - `[2026-06-13]` **Loading an old LoRA adapter after a vocab bump fails on embedding size.** The harness config + `tokenizer.json` are now at `new_vocab_size=50279` (v4 multi-tag); the v1/v2/v3 adapters are 50277. To load v1 (the accepted artifact), set `cfg.new_vocab_size=50277` before `load_finetuned_engine_lora` (else PeftModel state_dict size mismatch). `tokenizer.json.v3bak` is the 50277 tokenizer for a clean restore. _Archived 2026-06-16._ - `[2026-06-13]` **Heavy from-source compile (`MAX_JOBS=128`) on the shared PROD GPU box PINS it** — load hit 187, prod vLLM restarted, killed an in-flight quant. ana-ml2 hosts live inference; never run a big build there at full parallelism. Cap `MAX_JOBS≤32`, build off-box, or cgroup-constrain. _Archived 2026-06-16._ - `[2026-06-13]` **`--quantization fp8` on a VL model can quantize the VISION TOWER → garbage vision** (Qwen3.5-VL on stable vLLM: gray-grid output; LM answers text fine, so it "looks" healthy). The nightly excludes the vision tower. Validate the VISION path on a quantized VLM, not just text — and pin the engine digest with the exclusion. _Archived 2026-06-16._ - `[2026-06-13]` **vLLM's `--gpu-memory-utilization` is checked against FREE VRAM at startup, not total** — on a shared card, growing one service before trimming a co-tenant OOMs. Trim the shrinking service FIRST, then grow. Size to the FREE budget. _Archived 2026-06-16._ - `[2026-06-13]` **The `vllm/vllm-openai` entrypoint is already `["vllm","serve"]`** — compose `command:` supplies the model as the first POSITIONAL arg + flags; a second `serve`/`--model X` → "unrecognized arguments". Same-class: `tee` masks the real exit code (use `>`); HF `datasets` rejects bare `wikitext` (needs `Salesforce/wikitext`). _Archived 2026-06-16._ - `[2026-06-13]` **Chatterbox-Turbo LoRA finetune: the repo's `setup.py` loads the WRONG tokenizer** — pulls gpt2-medium + a grapheme merge file instead of the chatterbox-turbo GPT2 tokenizer (vocab.json+merges.txt, len 50276). Fix = override + delete the grapheme `tokenizer.json`; `[vmoan]` → new_vocab_size 50277 (1-row resize), lora_r 64 / alpha 128, modules_to_save=[text_emb,text_head]. Unique-stem corpus collision needs `{index}_{stem}` IDs. (`irv-ml1:~/r16-vmoan-harness`) _Archived 2026-06-16._ - `[2026-06-11]` **A completion-poll `while pgrep -f ` SELF-MATCHES its own remote shell argv** — its own `pgrep -f` always finds itself → the loop never exits. Use a match pattern ABSENT from the poll command (the python stage, or a sentinel file), not the driver's own name. _Archived 2026-06-16._ - `[2026-06-14]` **vLLM ModelOpt-NVFP4-MoE loader is broken for current multimodal MoEs.** `nvidia/Qwen3.6-35B-A3B-NVFP4` fails weight-load: `KeyError: layers.0.mlp.experts.w2_input_scale` on 0.19.1, `lm_head.input_scale not registered` on 0.22.0 (vllm #44081); same class hits Gemma-4 MoE / Qwen3-30B-A3B NVFP4. The arch + quant ARE recognized (gets past arch resolution + vision-processor load) — it's the per-expert/lm_head scale-key mapping. Don't chase nightlies; use official FP8 until fixed. _Archived 2026-06-18._ - `[2026-06-14]` **vLLM sampler-warmup OOMs on a shared GPU even when weights fit** — it warms the sampler with `max_num_seqs` (default **1024**) dummy requests, and a big vocab (Qwen3.6 = 248K) makes that a huge transient logits tensor. A vision endpoint doesn't need 1024-way concurrency: set `--max-num-seqs 32`. Separately, post-load `ValueError: No available memory for the cache blocks` means util is too thin (weights+activation+graph ate it) — for 34 GB FP8 weights, util ≥ ~0.45 to leave KV room. _Archived 2026-06-18._ - `[2026-06-14]` **Recreating multiple vLLM services concurrently races the memory-profiling assertion** — `AssertionError: Error in memory profiling. Initial free memory X / current Y … other processes … release GPU memory while vLLM is profiling`. Recreate co-tenant vLLM services ONE AT A TIME (force-recreate one, wait healthy, next). _Archived 2026-06-18._ - `[2026-06-14]` **embed/rerank (0.6B) at util 0.05 reserve ~5.5 GB each — mostly util-reservation WASTE, not need.** A 0.6B model needs ~1.2 GB weights + ~2.5 GB CUDA/torch context; util 0.03 (~3.6 GB) fits with room, reclaiming ~4 GB (vLLM reserves the util fraction regardless of actual KV; embedding models barely use KV). Real-need floor ~3 GB — don't go to 0.02. _Archived 2026-06-18._ - `[2026-06-14]` **Chatterbox-Turbo decode-knob foot-guns** (R16 v1-polish + emergent probes): the turbo length cap is `max_gen_len` (default 1000) on `t3.inference_turbo`, NOT `max_new_tokens` — and `tts_turbo.generate` does NOT forward it (wrap inference_turbo to cap). `rep_pen 2.0 / temp 0.5` BACKFIRES (degenerate 24 s run-on). Soft decode tamers cut the NVV ITSELF, not just the run-on tail (operator: "p1 trims the moaning too") — gen-time polish can't beat v1's defaults. Inline base-NVV tokens DEGENERATE (moan-cascade + gibberish), they don't modulate the surrounding words. _Archived 2026-06-18._ - `[2026-06-15]` **A comfyui restart-loop with `exit 0` + `OOMKilled=false` is NOT OOM — suspect a native SEGFAULT.** `docker inspect` shows only the LAST exit code; a GPU OOM is a catchable `RuntimeError: CUDA out of memory`, NEVER `Fatal Python error: Segmentation fault`. Diagnostic: `RestartCount>0` = same container restarted in-place → the docker json-log ACCUMULATES across all cycles, so grep the full `docker logs` for the crash signature + the lines right before each restart banner. comfyui's was at VAE decode (sampling completes, then segfault loading `AutoencodingEngine`) — gen-triggered but native, not VRAM. Suspects logged: torch/cu130 mismatch, an injected `aimdo` cuda-funchooks lib, the VAE. _Archived 2026-06-19._ - `[2026-06-15]` **NEVER blanket-kill `althing-light-monitor` by PID** — they're per-handle and MANY agents run their own on the shared nh3-dev box (heid / ratatoskr-dev / worldtree-dev / bifrost-dev each have one). A `kill $(pgrep -f light-monitor)` downs four peers' monitors (near-miss caught by checking `/proc//environ` `ALTHING_HANDLE` + cwd first). Recover your OWN via `althing-cli stop-monitor` (v0.14.1+); a raw `&`-backgrounded monitor is untracked (no harness fire-notification) — always arm via `run_in_background`. (extends the pkill-self-match lesson below) _Archived 2026-06-19._ - `[2026-06-15]` **Manual `docker compose up` on corviduo Worldtree DOWNGRADES to a stale local `:latest` → crash-loop (config/image mismatch).** Compose `image:` is `…/worldtree:latest`, but local `:latest` lagged ~2wk (`21c90d5eeb2d`); CI/CD deploys NEWER SHA images, and a real pull 401s (daemon has no gitea-registry creds — that's CI/CD's). `--pull never` then used the stale latest → personal crash-looped on `ConfigurationError: echo.default_model missing` → **~2-3 min personal outage**. RECOVER: retag `:latest` to the image a HEALTHY SIBLING runs (`docker inspect worldtree-worldtree-api-1 --format '{{.Image}}'` → `docker tag …`), then `up -d --pull never`; verify `Application startup complete`. Don't manually recreate Worldtree on corviduo without pinning the correct image — image+config are CI/CD-coupled. (auto-memory `reference_corviduo_dev_emergency_ops`) _Archived 2026-06-19._ - `[2026-06-15]` **`.claude.json` (Claude Code config) truncates on ENOSPC → invalid JSON → CC resets to defaults on next start.** nh3-dev hit 100% disk (a peer's R19 dataset pull staged ~147G to local /home instead of the 7TB NAS); the in-flight `.claude.json` write was cut mid-key. Repair = strip the dangling token + close the open braces, validate `json.load`. CC keeps its OWN rolling backups at `~/.claude/backups/` + writes a `.claude.json.corrupted.*` snapshot when it self-detects. After ANY ENOSPC on a box running CC/agents: integrity-check `.claude.json` (json.load) AND the althing bus DB (`PRAGMA integrity_check`). _Archived 2026-06-19._ - `[2026-06-15]` **`pkill -f althing-light-monitor` SELF-MATCHES the killing shell** (the pattern is in the command's own argv) → kills itself mid-run (exit 144/truncated output). Stop the light-monitor via `althing-cli stop-monitor` or a captured PID — never `pkill -f `. The singleton lock can also RACE to 2 live monitors during re-arm churn; keep exactly one tracked (run_in_background) monitor, and a raw `&` monitor is untracked (no harness fire-notification — don't use it). _Archived 2026-06-19._ - `[2026-06-14]` **Fleet/colo hosts must reach gitea over the INTERNAL route, NOT the public IP.** `gitea.phasefinal.com` = public `38.120.12.44` (ana-srv1); gitea is a container on ana-docker, git-SSH `10.250.50.70:222` + HTTP `:3000`. A fleet host egressing to public `:22` gets fail2ban-banned after any retrying git loop → silently wedges webhook auto-deploys (`git fetch` times out under `set -euo pipefail`, aborts before reset). Bit irv-ml1's arbo deploy. `:22` on `10.250.50.70` is ana-docker's HOST sshd (deploy key → Permission denied), NOT gitea. Documented `docs/orientation.md` (`6e58e57`). _Archived 2026-06-19._ - `[2026-06-16]` **litellm 500 `Router.acompletion()/aembedding() missing 'messages'/'input'` = a request missing `Content-Type: application/json`, NOT a gateway outage.** curl `-d` defaults to form-encoding → litellm can't parse the JSON body → `data` reaches the router without `messages`/`input` → 500 (should be a 400; litellm #16993). My own diagnostic calls dropped the header → I misread it as a gateway outage and needlessly bounced the gateway ~4× chasing a phantom (image/version/config were fine throughout; a malformed UI-added "Mistral Story Eval" model in the DB was a red herring I deleted). ALWAYS send `-H "Content-Type: application/json"` testing litellm; reproduce with a header'd call before declaring a litellm incident. _Archived 2026-06-20._ - `[2026-06-16]` **LiteLLM admin-UI playground can't test vLLM-backed models** — it auto-sends empty `tools:[]`, vLLM 400s (litellm #6228); the gateway `strip_empty_tools` hook is a PROXY hook and structurally can't reach the UI's in-process `litellm.completion()` call. Off-ramp = `tools/gateway-chat.html`. (Langfuse playground also out: its SSRF guard blocks internal-IP LLM connections, wontfix Langfuse #13097.) (auto-memory `reference_litellm_ui_playground_vllm_deadend`) _Archived 2026-06-20._ - `[2026-06-16]` **A gitea Actions run that "looks like it never fired" (no image, untouched build dir) usually DID fire then skipped or failed-fast.** Bit arbo TWICE: run #3 looked-never-fired but SKIPPED (change-detection `engine=0`); run #4 looked-never-fired but FAILED in ~7s (early-step). Check the run LIST + status via claude-bot (`GET /repos/{o}/{r}/actions/tasks`) BEFORE a runner-connection/dispatch theory. CORRECTION (2026-06-16, proven): action logs ARE readable via API at the per-JOB endpoint — `GET /api/v1/repos/{o}/{r}/actions/runs/{run}/jobs` → take a job `.id` → `GET /api/v1/repos/{o}/{r}/actions/jobs/{job_id}/logs` returns the full plain-text log (claude-bot basic-auth, internal route `10.250.50.70:3000`). Only the RUN-level `/runs/{id}/logs` 404s; the per-job one works, so no UI needed. Gotcha when reading: gitea marks **every** step `conclusion:failure` once any step fails — the per-step conclusions mislead; trust the log body + step *timestamps* (the step with real duration is the culprit) not the conclusions. (Used 2026-06-16 to diagnose arbo run #5/task 1175 = test-gate `uv run pytest` → `Failed to spawn: pytest` = pytest absent from the fresh-checkout venv sync, NOT the SSH-clone comfy-dev suspected.) _Archived 2026-06-20._ - `[2026-06-16]` **vLLM 0.23.0 breaks Mistral MULTIMODAL at startup** (`MistralCommonImageProcessor has no attribute fetch_images`, #44911 regression ~0.22.1+; same class hit Mistral-3.1/ Devstral). The dummy-image profiling crashes. Workarounds: `--limit-mm-per-prompt '{"image":0,"video":0}'` loads text-only; for VISION, pin **v0.22.0** (last pre-regression). Tradeoff: v0.22.0 *buffers* reasoning (doesn't stream reasoning_content cleanly); a newer build streams it but breaks vision — vision won (Mistral is the abliteration-baseline). _Archived 2026-06-20._ - `[2026-06-16]` **Mistral `-reasoning` TTFT variance = Triton kernel JIT-compile-during- inference, NOT reasoning time.** `jit_monitor` logs the spike (`_compute_slot_mapping_kernel`, `apply_token_bitmask_inplace_kernel` ← the tool-call/structured-output path). vLLM's startup warmup covers only SOME shapes; a first-hit uncovered shape JITs a kernel (seconds), then CACHES it — warm until container recreate (the Triton cache is IN-container, no volume, so recreate = cold again). Secondary: GPU idle-down to P8/180 MHz between requests. Also: Mistral `reasoning_effort` is BINARY (none|high only — medium/low → HTTP 400). _Archived 2026-06-20._ - `[2026-06-16]` **Worldtree old-baseline instances need the FULL config set synced, not the changeset delta.** Personal (a PRE-mistral baseline) crash-looped (41 restarts) on `DanglingBindingError: role 'echo' binds missing catalog entry 'qwen3.6-35-a3b-heretic'` — its `model_roles.yaml` (2026-06-10, never migrated) still bound the retired heretic that the synced v0.35.10 providers removed. The "model_roles unchanged — skip" was correct only for demo (already at the v0.35.9 baseline). Old-baseline → sync providers + model_roles + defaults. _Archived 2026-06-20._ - `[2026-06-16]` **vh is a Gitea USER, not an org → no scoped package-write for a service account.** Gitea user-namespace packages are owner/admin-only, so claude-bot can't push to `vh/arbo` without being admin. The fix is converting vh → a `corviduo` ORG (tabled). Also: vh's `tea` token lacks `write:admin` — create gitea users via the server-side `gitea admin user create` CLI (`docker exec -u git gitea …`), not the admin API. _Archived 2026-06-20._ - `[2026-06-16]` **`docker compose up -d` on a multi-service vLLM stack recreates ALL its services** (shared `.env` → all configs re-hash) → concurrent memory-profiling race ("No available memory for the cache blocks"). Target one: `docker compose up -d `. (auto-memory `feedback_compose_up_recreates_whole_stack`) _76 older entries archived to archival-memory.md._ _Archived 2026-06-20._ - `[2026-06-19]` **litellm CONFIG-loaded models can't be hot-removed — `/model/delete` is DB-only.** Only a process restart drops a config model. The INVERSE works: `/model/new` live-ADDS (no bounce) but dups with config on restart. Clean no-bounce-add: live-add via API, delete the DB copy before the eventual bounce. _Archived 2026-06-21._ - `[2026-06-19]` **the litellm `*` wildcard routed to DECOMMISSIONED llama-swap → a stale/typo'd model name returned a misleading `Connection error`, not `model not found`.** Bit a brokkr call to the renamed `qwen-image-judge`. **→ RESOLVED 2026-06-20: wildcard removed (d0eb09c) — unknown models now 404 cleanly.** _Archived 2026-06-21._ - `[2026-06-19]` **A MoE card can't be filled to 0 bytes free — the FusedMoE transient workspace (~3.09 GB) is the floor**, allocated OUTSIDE `gpu-memory-utilization`. FIX: `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` + leave ~3.2 GB free (util <= ~0.95 on a tight MoE). _Archived 2026-06-21._ - `[2026-06-19]` **vLLM decode tok/s: ALWAYS discard the first generation (cold warmup).** Cold ~ 24.8 tok/s vs warm 92.7 — a cold read undersells decode ~3-4x. Measure run 2+ over a >=256-token output. _Archived 2026-06-21._ - `[2026-06-19]` **For full native 256K on one 96 GB card, nightly+turboquant-4bit-KV was unnecessary for the Qwopus MoE.** Stable fp8 KV already fits 262144 (hybrid SSM+attn caches KV only on attn layers → small pool). Reach for fp8 first. _Archived 2026-06-21._ - `[2026-06-18]` **mmartial `comfyui-nvidia-docker`: root pip installs CRASH-LOOP the container.** `docker exec -u 0 pip install` leaves root-owned files in the uid-1000 venv → the boot script (uid 1000) fails its torch step → crash loop. FIX: `chown -R 1000:1000 /comfy/mnt/venv`; pin with `DISABLE_UPGRADES=true`. (auto-memory `reference_irv_ml1_comfyui_mmartial`) _Archived 2026-06-21._ - `[2026-06-18]` **ComfyUI custom nodes break on version-assumption drift** — box runs 0.24.1; TeaCache imports `precompute_freqs_cis` at MODULE level (guard the LTX-only import). `pip install tensorrt` defaults to cu13 → use `tensorrt-cu12`. _Archived 2026-06-21._ - `[2026-06-17]` **Mistral HF→NVFP4 quant: the placement knob is the whole game.** `device_map="cpu"` (CPU-resident, sequential onload) works; `"auto"` OOMs, `max_memory` offloads to meta. Read shards with plain `read()` + `safetensors.torch.load(bytes)`, NOT `safe_open` (mmaps the 50 GB shard → ENOMEM on `/tank` ZFS). _Archived 2026-06-21._ - `[2026-06-17]` **HF-format Mistral Small 4 is UNSERVEABLE on vLLM** — no HF `Mistral4` backbone; serves ONLY via the native loader. A HF-format quant MUST be converted to native first. _Archived 2026-06-21._ - `[2026-06-20]` **A full disk wedges gitea's Actions notifier SILENTLY — and it STAYS wedged after disk recovers; only a `docker restart gitea` clears it.** After the 94 GB clickhouse disk-full event, gitea stopped CREATING workflow runs on push (push succeeds, ref lands, but NO run spawns — newest run stale, queue empty), while the act_runner stayed registered. NOT run-pickup, NOT disk-now (52% free), NO gitea errors/panics — a silent notifier stall. Hit BOTH vh/arbo (comfy-dev) and vh/Worldtree (worldtree-dev's deploys) → it's global, not repo-specific. Fix: `docker restart gitea` on ana-docker (it had been up 2 weeks, never bounced). A restart does NOT retroactively fire pushes made during the outage — re-push/re-trigger after. Reinforces the deferred docker-daemon-default log-cap fix (the disk-full that started it can't be allowed to recur). _Archived 2026-06-25._ - `[2026-06-20]` **Mistral-Medium-3.5-128B NVFP4 (RecViking) on ana-ml2 GPU 0 — ABANDONED, too slow (operator confirmed).** Stood up as a temporary speed-check displacing mistral-small-4; measured base **~19.2 tok/s** decode on one Blackwell (vs `gen`/Qwopus3.5-122B ~92.7 tok/s warm) → confirmed the operator's "too slow" suspicion. KEEP these serving findings: (1) vLLM **nightly** loads the HF-format NVFP4 weights DIRECTLY — no Mistral native-convert (unlike Small 4, same `mistral3` arch) — via the FlashInfer Cutlass NVFP4 kernel + `turboquant_4bit_nc` KV; TP=1 fits the ~70 GB NVFP4 on one 96 GB Blackwell. (2) **EAGLE was a dead end**: the official native FP8 head (`mistralai/Mistral-Medium-3.5-128B-EAGLE`) crashes in CUDA-graph replay with graphs on, and nets NO speedup with `--enforce-eager` (~18.5 vs ~19.2 base) — likely low draft acceptance against the NVFP4 (non-official-FP8) base. Untracked workspace files (`stacks/mistral-medium-3.5/` + `playbooks/serve-mistral-medium-3.5.yaml`) removed. ⚠️ Weights still staged on ana-ml2 `/tank/aimodels/mistral-medium-3.5-{nvfp4,eagle}` (~70 GB+) — pending operator OK to free. _Archived 2026-06-25._ - `[2026-06-20]` **A 94 GB unrotated `langfuse-clickhouse` json.log was the REAL disk-filler (not image bloat) — there was NO log rotation anywhere.** No per-stack `logging:` limit and no docker-daemon default → ClickHouse's verbose stdout grew unbounded to ~94 GB and filled ana-docker's `/` (healthchecks then failed fleet-wide on `runc /tmp: no space left`). Lesson: on a full disk, the symptom (all-unhealthy) is the FORK-can't-write healthcheck, not the containers; find the runaway via per-container-dir `du`, not `docker system df`. The systemic fix is a daemon-default `log-opts` (`max-size`/`max-file`) — NOT just per-service patching. _Archived 2026-06-25._ - `[2026-06-20]` **OmniVoice's upstream "RTF 0.025 / 40×" does NOT hold — diffusion has a ~fixed per-call overhead → TTFA floor + multi-chunk streaming is a net loss.** Measured ~2.8× (32-step) / 5.6× (16-step) on the 3090; every chunk costs ~the same regardless of length, so naive per-sentence streaming triples total gen time + starves. Use few/aggressive chunks; chatterbox-fast (autoregressive) stays the low-TTFA front-end. _Archived 2026-06-25._ - `[2026-06-20]` **Named buildx builders accumulate cache the standard prunes MISS.** `docker image prune -a` + `docker builder prune` don't touch `buildx_buildkit_-builder0_state` volumes (per-builder buildkit state). Prune them with `docker buildx prune --builder -af`, or `docker volume rm` orphaned ones (builder already removed). _Archived 2026-06-25._ - `[2026-06-25]` **althing "unreachable: " can MASK an app-level 500.** Raw network was clean; root cause = receiver DB agents-table not synced with the config roster → delivery 500'd "unknown to: ", MAPPED to "unreachable". Diagnose: raw curl to :8087 + connect-probe ⇒ NOT network. Fixed in althing v0.17.1. `reference_nh3_extdev_althing_mesh`. _Archived 2026-07-08._ - `[2026-06-20]` **rest-server `.htpasswd: permission denied` = the ana-nas NFS mount FAILED (ghost file on the local mount point), NOT a decommission.** `mnt-backup.mount` stuck `failed` (fstab bare `defaults`) → rest-server serves an empty local dir. Recovery in disaster-recovery.md. _Archived 2026-07-08._ - `[2026-06-20]` **The DEFAULT `ssh ana-docker` is `lkraven` (no NOPASSWD) — but `ssh infra-ops@ana-docker` HAS NOPASSWD root.** A `sudo cp` as lkraven silently failed → nearly punted the rest-server recovery. Reach for `infra-ops@ana-docker` for sudo ops. _Archived 2026-07-08._ - `[2026-08-25]` **Four throughput levers measured and killed — do not re-chase.** (1) **Fused MoE / `grouped_mm`** — 0.9% *slower* than the Python loop and dense GEMM is only 7.9% of the step, capping the whole category near 10%. (2) **CUDA graphs / `torch.compile` over the expert loop** — the two-term scaling fit closed with residuals under 3ms and needed NO constant term, so there is no fixed per-batch cost to amortise; 3,840 expert-GEMM launches per forward are not what we pay for. (3) **`liger` fused linear CE** — the chunked CE measured **1.1% of the step** forward, ~3% with recompute. A tidy-up, not a lever. (4) **Selective gradient checkpointing** — ~2% of a post-fix step, real bug surface. Also: **token-budget batching is dead by the same fit** — with no constant term, total time over a fixed set of widths is invariant to how you group them; only the widths matter, which is exactly why bucketing works and repacking does not. _Archived 2026-09-09._ - `[2026-08-25]` **`sample_packing` is NOT strictly better than bucketing on this model, and I told the operator it was before brokkr corrected me.** Packing needs FA2 varlen or a block-diagonal mask; FA2 is unavailable here (head_dim 512 > 256 cap), so packing means an explicit 4D mask on EVERY batch. Bucketing produces **78.3% exactly-zero-pad micro-batches** which recover the `is_causal` fast path on the 5 global layers — measured at 9.4% of step time. Packing forfeits that. ⚠ **The conclusion flips under `flex_attention`**, where a block-diagonal mask is just another BlockMask: do not carry "packing is bad" past the backend decision. _Archived 2026-09-09._ - `[2026-08-25]` **Merging a tune back toward STOCK to fix overfitting would UNDO the abliteration.** brokkr recommended a 50/50 merge-back, then retracted it himself: the published recipes merge into `google/gemma-4-*-it`, and following that literally re-installs exactly the refusal directions the abliteration removed — silently, because the merged model looks *healthier* on general benchmarks. Any merge-back must target the SAME abliterated base. Wider lesson: **recipe cards are per-checkpoint artifacts, not per-family** — the advice came from a card for a DENSE STOCK 31B applied to a MoE ABLITERATED 26B-A4B, three axes apart on a shared name. _Archived 2026-09-09._ - `[2026-08-24]` **AES-GCM on the Anaheim tunnels — impossible, not merely hard.** UniFi's manual site-to-site IPsec implements no AEAD cipher at all: eight GCM spellings rejected `api.err.InvalidPayload` against a passing `aes256` control. Blocks both tunnels since both far ends are UDMs. Accepted enum is `aes128/aes192/aes256/3des` — and 3DES is *slower* (no ARM instructions, 64-bit blocks), so AES-128 is the floor. _Archived 2026-09-09._ - `[2026-08-24]` **Pointing the UDM's `wan_dns1` at AdGuard — silently ignored.** It persists and reads back correctly but the LAN-facing forwarder never uses it; proven with fresh uncached ad domains (AdGuard answers `0.0.0.0`, the UDM returned real IPs). Reverted rather than left in place. _Archived 2026-09-09._ - `[2026-08-24]` **A multi-DUID DHCPv6 VM to claim NH3's seven unclaimed /64s — declined by the operator.** The BGW has no IP-passthrough (confirmed, we hold admin), so the only route needs re-cabling, split-stack routing and **rebuilding the entire v6 firewall policy off the UDM**. The prefixes are easy; the firewall rebuild is why nobody wants them. Do not re-raise on "there are seven free prefixes". _Archived 2026-09-09._ - `[2026-08-23]` **A `HEAD == GITHUB_SHA` assertion in the hrafn CI — added, broke the checkout twice, removed.** It needed the `git` binary (run 9920, exit 127); installing `git` then flipped `actions/checkout@v4` off its **node** implementation onto the git binary, which died on a missing CA bundle (run 9921). A nice-to-have assertion changed the checkout's code path and broke a working pipeline. Removed rather than patched with `ca-certificates` — it guarded a hypothesis that proved wrong. **Do not add `git` to that prereq step.** _Archived 2026-09-09._ - `[2026-08-23]` **Repointing `selene-1-mini-8b` at gen's endpoint — proposed by me, correctly overruled.** *"never repoint a named model at a different model's endpoint — that is intentionally misleading."* The trap is that it does not feel like deception; it feels like sparing consumers a migration. That framing is the tell. Role aliases move; model names die with the model and 4xx. _Archived 2026-09-09._ ## Archived 2026-08-02 — Recent decisions (archived) ### 2026-07-08-worldtree-mimir-deploy-blocker-resolved-mid-session - `[2026-07-08]` **worldtree Mimir deploy-blocker resolved (mid-session):** synced `persona.envelopes.assistant` + `envelope_grants:[]` verbatim from the baked canonical into BOTH corviduo-dev instances (demo+personal), YAML-validated via each container's own parser; worldtree-dev cleared to push the Mimir-bound image. (Was my parked R32 1C envelope-mirror come due — see [[reference_corviduo_dev_emergency_ops]] config-sync recipe.) _Archived 2026-08-02._ ### 2026-07-09-two-parked-items-closed-phantom-qwen3-6-35b - `[2026-07-09]` **Two parked items closed: phantom `qwen3.6-35b-a3b` alias VERIFIED already-gone; ana-docker docker log-cap SOLVED no-bounce.** (1) **Phantom**: absent from `/v1/models` + `/model/info` (config+DB registry), zero litellm log refs — the parked "400s in /v1/models" note was STALE (already cleaned in the 2026-07-08 gen repoint to `-heretic`); bare token survives only in 2 config COMMENTS (lines 76/80). Nothing to remove. (2) **Log-cap**: running containers were UNCAPPED (182M json-logs, top offender 59M) because daemon.json's `max-size 10m/max-file 3` only applies to containers CREATED AFTER a daemon restart — it never reaches already-running ones. No-bounce fix = `/etc/logrotate.d/docker-containers` (**copytruncate** — dockerd opens json-logs `O_APPEND` so truncate-in-place resets cleanly, no sparse-file corruption; `size 10M`, `rotate 3`, `compress`, `su root root`), auto-picked-up by the daily `logrotate.timer`. Force-ran + gzipped the frozen `.1` archives → **182M → ~55M** (44M active + 11M gz), every container kept its multi-week uptime (zero bounce, verified). **LATENT FOOTGUN FLAGGED (not yet fixed, operator's call): daemon.json declares `live-restore:true` but the RUNNING daemon has it FALSE** (daemon.json was edited after the last daemon start, never reloaded) → the NEXT `systemctl restart docker` / crash / pkg-upgrade **bounces ALL ana-docker containers once**. Fix WITHOUT a bounce = `systemctl reload docker` (SIGHUP loads live-restore into the running daemon; log-opts are NOT SIGHUP-reloadable, which is why logrotate — not the daemon cap — is the enforcer for running containers). _Archived 2026-08-02._ ### 2026-07-09-morpheus-tts-off-the-shelf-voice-pipeline-shipped - `[2026-07-09]` **mOrpheus TTS off-the-shelf voice pipeline SHIPPED end-to-end (irv-ml1) + wired into gateway-chat.** Full arc (commits): gen served-name honesty rename aeon→qwen3.6-35b-a3b-heretic (99a4a17, vLLM served-name + litellm refs, so /v1/models + spend-logs name the real model); permanent 2-container stack (01eedd8); gateway-chat auto-voice quoted dialogue (c948013); streaming decode TTFA 4.5s→0.8s (da76829); max_tokens 1200→2400→3500 with a context-clamp (f363fe6, 0655a37 — long lines were clipping at 14.6s, and `repetition_penalty` 1.1 is LOAD-BEARING: at 1.0 the model never stops); AudioContext resume-on-gesture no-sound fix (033f368); pre-chunk by QUOTED SECTION not sentence for prosody (a1f3023→f295cc1); staged clone voices baddy/beatrice/whisper (0655a37 + runtime .wav/.txt in the voices dir); agent voicing prompt (a573514). **Load-bearing config, all encoded in stacks/mOrpheus/: bf16 not FP8, image v0.23.0 not latest, GPU=3090 not A6000, rep_penalty 1.1.** Serving-viability confirmed: vLLM concurrency near-linear to 8× (707 tok/s). _Archived 2026-08-02._ ### 2026-07-09-granite-gen-memory-extractor-bind-host-synced-on - `[2026-07-09]` **granite→gen `memory_extractor` bind host-synced on demo+personal Worldtree (Vuong-directed, #335 Slice-4).** Changed `model_roles.yaml` memory_extractor `binds.catalog_id` `summarizer`→`gen` (overrides intact: thinking:false/temp0/8192) on BOTH `/opt/worldtree{,-personal}/config`; `memory_distiller` left on `summarizer` (range-scoped sed `/memory_extractor:/,/memory_distiller:/` — the naive global replace would've hit both); backups `*.bak-preqgen-20260709`; validated via each container's OWN yaml parser. **DEMO LIVE on gen** (b43 `d501e516732d` auto-deployed mid-edit + its restart RACED my edit by ~2min → I restarted `worldtree-worldtree-api-1` to activate; healthy, live process resolves memory_extractor=gen). **PERSONAL NOW LIVE on gen too** — Vuong authorized the restart (via wt-dev); restarted `worldtree-personal-worldtree-api-1` after a PRE-FLIGHT that ran the app's OWN `load_model_roles()` (`core/llm/roles.py:121`) against the synced config INSIDE the running `c9986cd` container: `gen` in catalog, all 9 roles resolve, no `DanglingBindingError` → proven safe on the OLDER image BEFORE touching it (model_roles-delta-alone clean; no full-config-set sync needed). StartedAt 20:50:55Z, healthy, resolves gen. **BOTH instances live on gen.** **LESSON: the bind-mount `/opt/worldtree*/config` SHADOWS the baked `/app/config-defaults/` → the deploy alone never updates the ACTIVE config; the host edit is required AND a restart activates it (role registry cached at boot) — pre-stage BEFORE the deploy's restart or you race it.** FOLLOW-UPS (non-blocking): (a) `memory.extractor.user_pass` parity block → self-serve from the b43 baked `defaults.yaml` (pydantic-default no-op); (b) stale `gen` provider description (Mistral-Small-4 → qwen3.6-35b-a3b-heretic) → wt-dev owns the REPO-side `providers.yaml` fix (operator's call — NOT purely cosmetic: the gen/dialogue + classifier entries carry Mistral-Small-4 SAMPLING defaults that drive mask/lofn/forseti/mimir dialogue, so wt-dev re-validates for qwen), host cosmetic sync pairs when it deploys. Gave wt-dev the VERIFIED canonical 4-alias set (backends+samplers read from the LIVE gateway config, not the doc); corrected `docs/pfi/model-sampler-defaults.md` seat 4 (had lagged QwQ-RpR-v4 → Deckard-PKD; live gateway was always Deckard). Operator SCOPED IN the character-RP re-point (2026-07-09): character→char-rp / thoughtful-character→char-rp-reasoning (character-rp per wt-dev's role semantics), moving character RP off the GENERAL qwen onto the dedicated Magidonia/Deckard seats. Relayed to wt-dev w/ the mapping principle + a SAMPLER-OVERRIDE warning (DROP character-rp's old temp0.75/top_p0.85 overrides — carried onto the dedicated seats they'd clobber the canonical RP tuning DOWNWARD) + ratatoskr-reach note (role call is transparent but Magidonia/Deckard quality/latency differs from gen). DONE 2026-07-09: wt-dev committed 5d4fa4a (v1.0.0b44, UNPUSHED — operator drives push); operator directed host-ahead-of-push, so I sourced BOTH config files directly from that unpushed commit (local `~/development/Worldtree` checkout — capital W; `git show 5d4fa4a:config/…`) + mirrored VERBATIM to `/opt/worldtree{,-personal}/config` on demo+personal, paired-pre-flighted via the app's `load_model_roles()` (no DanglingBinding), restarted both → LIVE: character→char-rp, thoughtful-character + character-rp→char-rp-reasoning, memory_extractor→gen preserved. Backups `*.bak-prerp-20260709`. context_window VERIFIED (llama.cpp /props + char-rp-gguf `.env`): char-rp **131072**, char-rp-reasoning **262144** (gave wt-dev to patch the repo from its interim 32768). **HOST AHEAD of repo-remote until the operator pushes 5d4fa4a** (baked config converges with the bind-mount on push+deploy). GOTCHA: demo≠personal — PERSONAL was already partly re-pointed (2026-07-06 AEON-era character→char-rp) so its delta was mostly stale-AEON-descriptions→Magidonia/Deckard + character-rp + stripping personal's char-rp `default_params` temp0.7/top_p0.8 that CLOBBERED the gateway RP tuning downward; DEMO had no char-rp catalog entries at all (b44 adds them). Diffed each instance vs b44 before applying (both deltas = expected changeset only, nothing instance-specific clobbered). wt-dev PATCHED the context_window in **b45/3384a37** (char-rp 131072, char-rp-reasoning 262144). BUMPED HOST-AHEAD on both instances (operator-directed 2026-07-09): mirrored b45's providers.yaml → `/opt/worldtree{,-personal}/config`, restarted, verified LIVE (char-rp ctx 131072, char-rp-reasoning 262144, bindings intact); backups `.bak-b44interim-20260709`. HOST now = **b45 canonical** (providers.yaml) + b44 (model_roles unchanged b44→b45). STILL PENDING: (1) operator's batched push of **b44+b45** (`5d4fa4a`+`3384a37`) to converge the repo-remote — host is ahead, no fork; (2) user_pass parity block (defaults.yaml — NOT in either, separate). Threads `01KX3SGH`/`01KX48QP` (worldtree-dev), monitor armed. See [[reference_corviduo_dev_emergency_ops]]. _Archived 2026-08-02._ ### 2026-07-09-granite-gen-memory-extractor-bind-green-lit-for - `[2026-07-09]` **granite→gen memory_extractor bind GREEN-lit for worldtree-dev (Worldtree #335 Slice 4).** Answered their VRAM/concurrency headroom check: gen (qwen 35B-A3B heretic) has ample headroom for ~2 bursty idle-triggered extractor calls (fixed 0.37 util; KV pool runs 0-2%; --max-num-seqs 16, near-linear batching). Corrected their stale "gen = Mistral Small 4 / 119B-6B" belief (gen IS the qwen 35B-A3B heretic since 2026-07-08). **This bind is INDEPENDENT of the full granite RETIRE** (reclaim ~32GB on ana-ml2 GPU1) — that stays the operator's call, pending brokkr R33 portfolio + production-concurrency due-diligence. Tracked: althing thread 01KX3SGH… (worldtree-dev) + brokkr's gen-absorbs-granite consult (thread 01KX2V32…) + [[reference_litellm_gateway]]. _Archived 2026-08-02._ ## Archived 2026-08-02 — Tried and abandoned (archived) ### 2026-07-01-a-personal-worldtree-ci-deploy-that-fails-85s - `[2026-07-01]` **A personal-Worldtree CI deploy that fails ~85s in with "not found / unauthorized" is usually the pull-only-vs-build RACE, not registry-auth.** `deploy-personal.yml` is PULL-ONLY but fires on the `staging/vX` tag simultaneously with `deploy.yml`'s build → pulls before the push finishes. FIX: re-run once built, or gate on `workflow_run: completed`. _Archived 2026-08-02._ ### 2026-07-04-on-prem-t1-train-that-keeps-any-ana - `[2026-07-04]` **On-prem T1 train that keeps ANY ana-ml2 serving up = ~6-8 DAYS** (1-GPU + NVMe ZeRO-Infinity offload; MoE ~10B-active cuts FLOPs but NOT the 244G base's param I/O). The only fast on-prem path is a FULL ana-ml2 shutdown (both GPUs + the ~421G vLLM RAM freed → base fits in the 566G CPU RAM) → CPU offload → ~1-day full-fleet outage. Cloud (no offload) = hours. `reference_t1_cloud_train_plan`. _Archived 2026-08-02._ ### 2026-07-04-a-systemd-user-daemon-that-shells-out-to - `[2026-07-04]` **A systemd `--user` daemon that shells out to `~/.cargo/bin`/`~/.local/bin` tools needs an explicit `Environment=PATH`** — the minimal `--user` default silently drops them. The althing herald lost `zellij` → silent `pane-miss` for ALL config-backed TUI/pane agents; CC + FIFO routes were unaffected, so it was invisible from a CC session. `reference_nh3_dev_althing_herald`. _Archived 2026-08-02._ ### 2026-07-09-fp8-breaks-morpheus-audio-token-generation - `[2026-07-09]` **FP8 breaks mOrpheus audio-token generation.** `--quantization fp8` on the 3B → 0 valid SNAC frames even at GREEDY (degenerate audio+text mix, no start-of-speech); bf16 is clean (28/28 frames). Quant-breaks- TTS, same class as the Angel-NVFP4 lesson below. bf16 is REQUIRED (so the operator's "util 0.1" wish is moot — the bf16 weights alone are 6.6GB). NB the raw-token benchmark RTF 0.50 was fp8+graphs = never real. _Archived 2026-08-02._ ### 2026-07-09-sentence-chunking-tts-loses-prosody - `[2026-07-09]` **Sentence-chunking TTS loses prosody** — generating each sentence cold flattens the intonation that spans a line. Chunk by QUOTED SECTION (whole quote = one gen call). Also: `repetition_penalty` >1.1 BREAKS cloning (penalizes the ~1100 in-context reference audio tokens; keep ≤1.1 on the clone path). _Archived 2026-08-02._ ### 2026-07-09-hf-whisper-datasets-aren-t-actually-whispered - `[2026-07-09]` **HF whisper datasets aren't actually whispered.** Claris-Whispered-English measures voiced 0.8 (not a whisper) + IPA transcripts; `datasets` audio decode needs torchcodec (wants CUDA-13, incompatible w/ the cu124 venv). LPC DSP-whisperize went unstable (NaN). **kokoro `af_nicole` IS a genuine whisper** (voiced 0.24) — that (operator's pointer) is the working whisper reference source, not TTS-voice screening or dataset-hunting. _Archived 2026-08-02._ ## Recent decisions (archived 2026-08-03 batch) ### 2026-07-14-nvfp4-mtp-fast-char-rp-reasoning-seat-landed - `[2026-07-14]` **NVFP4+MTP fast char-rp-reasoning seat LANDED + LIVE + gateway-repointed + VRAM-tuned.** Modelopt-format re-quant made MTP work. The load-crash root cause = **vLLM 0.24 does NOT propagate modelopt `exclude_modules` to the spec-decode DRAFT model** → the bf16 mtp head gets quantized → shape crash; NO checkpoint config fixes it (`is_layer_skipped` is exact-membership, and the drafter never sees the mtp excludes) → **workaround = a mounted `sitecustomize` that force-skips `mtp.*` in `is_layer_skipped`** (upstream vLLM bug to file). Productionized as compose stack `heretic2-charrp-reasoning` (:8018, workaround baked in). Gateway `char-rp-reasoning` alias fixed: repointed off the stale GGUF served-name `deckard-pkd-27b`, added `enable_thinking:true`, **dropped `min_p`** (MTP-incompatible), canonical samplers temp1.0/top_p0.95/top_k20. Rebalanced GPU0 (gen 0.37→0.30/16-seq/256K + reasoning 0.39/16-seq/192K+MTP + char-rp 128K, 2.7GB free). All 4 gateway roles verified; vLLM reasoning-parser confirmed **leak-free** (unlike the GGUF budget-forcing). Full record + the 4 quant landmines in `docs/runbooks/heretic2-nvfp4-mtp-seat.md`; committed `982c319`. Open (non-blocking): brokkr P00 (seat is live ahead of it), retire the stopped GGUF reasoning seat, file the vLLM bug. _Archived 2026-08-03._ ### 2026-07-14-nvfp4-quant-chase-resolved-gibberish-pivoted-to-modelopt - `[2026-07-14]` **NVFP4 quant chase RESOLVED (gibberish) + PIVOTED to modelopt for MTP.** One ~40-min GPU0 window. Root-caused the `!!!!` to the quant NAMESPACE (text-only `AutoModelForCausalLM`→`model.layers.*` keys; vLLM serves only `Qwen3_5ForConditionalGeneration`, which needs `model.language_model.*`) — found from config diffs + vLLM source with ZERO GPU time; fixed by loading as `AutoModelForImageTextToText`. NVFP4 now serves COHERENT (validated greedy). BUT base NVFP4 ≈53 tok/s ≈ GGUF's 59.5 at batch-1 (no single-stream win) AND MTP = 0% acceptance on compressed-tensors (bf16 mtp head only loads on the modelopt format). Operator chose to **pursue a modelopt-format re-quant** (the only path to the 2-4× MTP goal; AEON-proven on this exact Qwen3.6-27B arch). Scoped + de-risked: AEON `/tank/aimodels/qwen36-27b-aeon-nvfp4` = the modelopt reference (quant_method modelopt, 1967 tensors, 15 bf16 mtp keys identical to graft); nvidia-modelopt 0.45.0 installs + `mtq.quantize`/`NVFP4_DEFAULT_CFG`/`export_hf_checkpoint` API confirmed; pipeline unchanged except swap llm-compressor→modelopt. Seats restored; char-rp-reasoning stays GGUF. Full plan in Current state ★ section. _Archived 2026-08-03._ ### 2026-07-14-pursue-the-nvfp4-mtp-fast-char-rp-reasoning - `[2026-07-14]` **Pursue the NVFP4+MTP fast char-rp-reasoning seat to completion** (Vuong-directed via /snapshot: "chase the nvfp4 quant, we know it works, write down the recipe"). Full recipe + diagnostic ladder in Current state / in-flight above. Artifacts on ana-ml2 `/tank/aimodels/heretic2-nvfp4-work/` + scripts committed in eshpfi `services/heretic2-nvfp4-quant/`. _Archived 2026-08-03._ ### 2026-07-14-char-rp-reasoning-seat-deckard-pkd-neo-code - `[2026-07-14]` **char-rp-reasoning seat: Deckard-PKD → NEO-CODE = Heretic2-Thinking (Qwen3.6-27B)** — R36 gate PASSED (tools 0.967, #355 runaway ELIMINATED). #355 was MODEL-level (Deckard emitted qwen3_coder XML malformed → mangled args → retry-runaway), NOT the reasoning-budget bug; NEO-CODE emits it clean. Custom llama.cpp KEPT (qwen3_coder parse — stock b8840 predates it — + PR#25544). Committed f960a73; full record auto-memory [[charrp-custom-llamacpp-pr25544]]. _Archived 2026-08-03._ ### 2026-07-14-soong-lab-webhook-auto-deploy-real-root-cause - `[2026-07-14]` **soong-lab webhook auto-deploy real root cause = gitea `webhook.ALLOWED_HOST_LIST`** (was `external, 10.100.0.0/16` = NH3-only; blocked corviduo-dev's Anaheim `10.250.x` → gitea refused to deliver, never opened the connection). Fixed to fleet-wide `10.0.0.0/8` (app.ini `[webhook]`) + gitea restart; listener now logs every delivery. The ufw `10/8` open (also this session) was a real-but-secondary gap. Committed 462d528. _Archived 2026-08-03._ ### 2026-07-13-355-residual-root-cause-supersedes-the-litellm-gateway - `[2026-07-13]` **#355-residual ROOT CAUSE (supersedes the "LiteLLM gateway holds while seat idles" entry below — that was DISPROVEN).** char-rp-reasoning enters a non-terminating REASONING loop (tool-call-retry planning) and runs to `max_tokens=32768` (~22 min @ 24.7 tok/s, ~13% of requests); the seat GENERATES all 32768 tokens (not idle), and `--reasoning-budget 400` is NOT enforced. 3-source-confirmed (spend_logs completion_tokens=32768 ×4; seat eval-time log; pcap 100%-`reasoning_content` deltas). Server-side fix wanted (operator: no max_tokens ceiling) → routed to brokkr (accepted, pulled dvalin). Lesson (again): confirm before concluding — the seat-idle claim came from reading only the ≤73s requests + missing the concurrent 32768-token slots. See ACTIVE 1. _Archived 2026-08-03._ ### 2026-07-13-deploy-speed-real-bottleneck-uv-sync-memory-s - `[2026-07-13]` **Deploy-speed real bottleneck ≠ uv sync (memory's assumption was wrong).** Buildx step log: `chown -R /app` = 251s (copy-up of the root-owned venv into a fresh layer), uv sync only 35.6s, registry layer cache already wired. Fix = drop `/app` from the chown (validated safe: zero /app runtime writes on both live instances) + uv cache-mount. Shipped as PR #359 (branch off origin/main@b60), worldtree-dev green-lit. Expected ~5min off (~11→~6min). Runner-side BuildKit cache task (b) was already done → moot. _Archived 2026-08-03._ ### 2026-07-13-wt-355-residual-300s-hang-localized-to-our - `[2026-07-13]` WT #355 residual 300s hang localized to OUR LiteLLM gateway (holds 2 char-rp-reasoning requests ~21 min while the seat idles), NOT the seat — Deckard seat EXONERATED (completes ≤72s; `--reasoning-budget 400` forecloses a mid-thinking hang). Corrects worldtree-dev's "seat wedging" diagnosis. Decisive next = the FIN-check (pcap on corviuo). See in-flight ACTIVE 1. **[SUPERSEDED 2026-07-13 — see the ROOT CAUSE entry above; the gateway-hold/slot-leak theory was disproven, the seat was generating 32768 tokens.]** _Archived 2026-08-03._ ### 2026-07-13-wt-355-turn-lifecycle-fix-validated-on-worldtree - `[2026-07-13]` WT #355 turn-lifecycle fix VALIDATED on worldtree b60 — wedged turns self-terminate cancelled/stalled at the 300s stall-watchdog (turns 2064/2065 vs pre-b60 2061's 16-min no-terminal). worldtree-dev filed follow-ons #356 (rehydrate Tier-3 ctx on resume — the recreate-durability gap), #357 (reclaim orphaned active-turn locks), #358 (LLM-provider read-timeout audit); surfacing to Vuong to prioritize. _Archived 2026-08-03._ ### 2026-07-13-worldtree-deploy-bottleneck-the-image-build-11-min - `[2026-07-13]` Worldtree deploy bottleneck = the image build (~11 min of a ~12 min deploy), root cause the Dockerfile `uv sync ... --no-cache` + no BuildKit cache-mount (re-downloads all deps cold every build). Fix split: worldtree-dev Dockerfile cache-mount diff + infra-ops runner-side persistent BuildKit cache. Config-only changes skip the build entirely (pinned recreate). _Archived 2026-08-03._ ### 2026-07-13-ledger-tier-3-consumer-ledger-miranda-provisioned-on - `[2026-07-13]` Ledger tier-3 consumer `ledger:miranda` provisioned on personal :8081 (key b38932f5, GPG-delivered+shredded, allowlist 10.100.10.50:8770 live); `assistant`+`thoughtful-assistant` capability roles added (gen/gen-reasoning) on personal+demo, canonical d8bd497. Chosen instance = personal (the tier-3-consumer instance, ratatoskr+soong-lab colocated). _Archived 2026-08-03._ ### 2026-07-10-heimdall-grant-ratatoskr-affect-full-on-personal-worldtree - `[2026-07-10]` **Heimdall grant: ratatoskr `affect.full` on PERSONAL Worldtree (operator-approved, worldtree-dev R34-v1 request).** Added allow-rule `ratatoskr-affect-full-allow` to `/opt/worldtree-personal/config/policies.yaml` (`principal.user_ids:["ratatoskr"]`, action `affect.full`, resource `*`, effect allow), mirroring the #347 `session-history-write-ratatoskr` rule exactly + placed right after it. **WHY user_ids-based (not tier):** ratatoskr's personal key is the minimal **readonly-admin** observability tier, which is NOT in the tier-based `affect-render-baseline-allow` (anonymous/user/free/pro/admin) → needs an explicit user_id grant, same as #347. R34-v1 (b46, committed UNPUSHED) gates `affect.emit` `dominant_emotion` egress by exposure ceiling (affect.full|safe → present; neither → null); this grant keeps ratatoskr's view alive across the b46 deploy. Surgical exact-string insert (preserves comments), backup `policies.yaml.bak-pre-affectfull-20260710`, validated via the CONTAINER's own yaml parser (35 rules, +1, payload confirmed). **NOT restarted — deliberate:** rule is on the bind-mount (shadows baked), INERT until b46 gating ships, so the b46 CI/CD deploy restart activates it (no live-session blip now). Demo untouched (personal-only per key scope). Replied to wt-dev (thread 01KX6DB3…) offering an immediate restart if they want it live for pre-b46 testing. See [[reference_corviduo_dev_emergency_ops]]. _Archived 2026-08-03._ ### 2026-07-10-comfyui-v0-27-1-success-on-irv-ml1 - `[2026-07-10]` **ComfyUI v0.27.1 SUCCESS on irv-ml1 (operator-confirmed execute-now) — landed on torch 2.12.1, SageAttention preserved, crash-loop AVOIDED.** The prior attempt (entry below) crash-looped because a torch 2.12.1→2.13 bump broke SageAttention's ABI. This time I checked `git diff v0.24.1 v0.27.1 -- requirements.txt` FIRST and found **core v0.27.1 leaves `torch` UNPINNED** → the version bump does NOT require torch 2.13 (that came only from the mmartial boot-upgrade). So: `git checkout v0.27.1` (clean tree) → `pip install -r requirements.txt` as **uid 1000** with a **torch-pin constraint file** (torch/vision/audio pinned to current +cu129) to block any transitive bump → torch stayed 2.12.1, SageAttention 2.2.0 untouched. Added decord 0.6.0 (fixed SAM3Segment). `docker restart comfyui` → healthy, `/system_stats` comfyui_version=0.27.1, "Using sage attention", HTTP 200, DB migrated 0003→0004. Reported the divergence to comfy-dev (thread 01KX6D3C…, reply pending) + asked whether LTXV 2.3 needs a separate torch-2.13 follow-up (their domain; Krea-2's ≥0.25 need is met by 0.27.1). **LESSON: before a mmartial ComfyUI version bump, `git diff -- requirements.txt` — if torch is unpinned, bump the CODE without touching torch (constraint-pin it) and compiled exts (SageAttention) survive. `docker exec` lands as uid 1025(comfytoo), not 1000 — use `-u 1000` + the venv python `/comfy/mnt/venv/bin/python`.** See [[reference_irv_ml1_comfyui_mmartial]]. _Archived 2026-08-03._ ### 2026-07-10-comfyui-0-25-x-bump-on-irv-ml1 - `[2026-07-10]` **ComfyUI 0.25.x bump on irv-ml1 ATTEMPTED → FAILED → ROLLED BACK (snapshot saved it).** comfy-dev requested (Vuong-authorized) bumping the irv-ml1 `comfyui` stack (mmartial image, `/opt/docker/compose/comfyui/`, 0.24.1) to 0.25.x for Krea-2 + LTXV 2.3. **TWO FINDINGS: (1) `DISABLE_UPGRADES=false`/USE_PIPUPGRADE bumps the VENV (torch 2.12.1→2.13.0 + deps) but does NOT advance the ComfyUI CODE checkout** (`/comfy/mnt/ComfyUI` = `/worktank/comfyui/run/ComfyUI` stayed 0.24.1 — pinned/detached git, comfy-dev's domain). **(2) the torch bump broke SageAttention** (2.2.0 `_fused.so` undefined-symbol `c10::impl::cow::materialize_cow_storage` vs torch 2.13.0) → `--use-sage-attention` (REQUIRED launch flag in COMFY_CMDLINE_EXTRA) crash-looped ComfyUI. Net: broke the working state, zero 0.25.x payoff. **ROLLBACK WORKED**: pre-bump 16G venv snapshot `/worktank/comfyui/venv-snapshot-comfyui-0.24.1-20260710.tar` restored (torch 2.12.1 + working SageAttention), re-pinned DISABLE_UPGRADES=true, recreated → healthy on 0.24.1, serving :8188. Broken venv parked at `/worktank/comfyui/run/venv.broken-torch213-20260710`. **CORRECTED PATH (sent comfy-dev, thread 01KX655V):** comfy-dev git-advances the ComfyUI checkout to 0.25.x + reqs → I handle the torch bump + SageAttention rebuild-against-2.13.0 + re-pin (snapshot stays as the net). **LESSON: mmartial `DISABLE_UPGRADES` gates ONLY the venv pip-upgrades, NOT the ComfyUI git checkout; a torch bump breaks compiled exts (SageAttention) → rebuild-after is mandatory.** Bump BLOCKED pending comfy-dev's git-advance. Stack: A6000 (NVIDIA_VISIBLE_DEVICES=1), lkraven-owned compose+venv (uid 1000, no sudo needed), COMFY_CMDLINE_EXTRA OOM flags preserved. _Archived 2026-08-03._ ### 2026-07-10-biweekly-open-weight-releases-scan-cron-set-up - `[2026-07-10]` **Biweekly open-weight-releases scan cron set up for brokkr-smithy (Vuong-authorized).** Durable systemd **--user** timer on nh3-dev (`brokkr-landscape-scan.timer`, OnCalendar `*-*-01,15 09:00:00` America/Los_Angeles, Persistent=true; linger on) → `.service` → wrapper `~/.local/bin/brokkr-landscape-scan.sh` runs headless `claude -p "$(cat ~/.config/brokkr-landscape-scan/prompt.txt)" --dangerously-skip-permissions` in `~/development/brokkr-smithy` (ALTHING_HANDLE=brokkr-smithy-dev; **explicit PATH** — the --user minimal-PATH footgun; per-run logs `~/.local/state/brokkr-landscape-scan/`). Prompt = brokkr's payload verbatim (LLM/image/TTS new-release sweep → ranked synthesis → commit+push+notify). VALIDATED: git-push non-interactive (BatchMode ls-remote to gitea, passphraseless key — no agent), headless claude auth (READY smoke). VALIDATED END-TO-END 2026-07-10 (manual run, exit 0): web-sweep→synthesis→commit `2ed2f29`→PUSH of scan #2 (open-weight-releases-2026-07-24.md); triaged dwarf input + caught baseline errors, quality strong. **HANDLE-COLLISION caught+FIXED** — the headless scan shared handle brokkr-smithy-dev with the LIVE session + raced its inbox (eitri's dwarf-reply got stolen by the live monitor); registered a dedicated **brokkr-scan-dev** handle (`add-handle`, driver=none) + repointed the wrapper + rewired step-5 notify → `althing-cli post --to brokkr-smithy-dev` (NO vuong althing handle exists — confirmed). model=default + `--max-turns 80`. First run under the new handle = 7/15. Off-cycle 07-24 doc is a validation artifact (scheduled 1st/15th runs date to their own run-date, no collision) — operator naming-convention call pending. **NEXT AUTO-RUN 2026-07-15 09:00 PDT.** Manual validation/first run = `systemctl --user start brokkr-landscape-scan.service`. Open w/ brokkr (thread 01KX63G6): confirm notify-Vuong handle/mechanism + session handle + model/turn-cap. **NEXT brokkr task (operator-sequenced after this): TTS audition env** — Higgs-TTS-3 + ZONOS2 + Chatterbox baseline, TTFA/RTF + blind-A/B web-listen (thread 01KX6371; needs GPU-placement + HF-token feasibility pass first; brokkr delivers the prompt set after the env's up; protocol doc in brokkr-smithy repo). _Archived 2026-08-03._ ### 2026-07-08-rp-seat-campaign-closed-char-rp-magidonia-24b - `[2026-07-08]` **RP-SEAT CAMPAIGN CLOSED — char-rp = Magidonia-24B-v4.3 (128K), char-rp-reasoning = Deckard-PKD Qwen3.5-27B (256K); both GGUF/llama.cpp on ana-ml2 GPU0 alongside gen (35B-A3B, util 0.37), ~4G GPU0 margin.** Arc: (1) replaced broken Angel NVFP4 with Magidonia prose + QwQ-RpR-v4 reasoning (b268f93); (2) max-context via q8_0 KV (f570604); (3) canonical samplers for all 4 gateway seats, dvalin-derived + char-rp A/B-tuned (aac4bcf); (4) rebalanced gen 0.40→0.37 to fund char-rp 128K (f49c4e4); (5) RE-A/B'd the reasoning seat (operator wanted a DRY-tolerant model): **Deckard WON** on brokkr's frozen scorer (composite 2.176, 0/30 loops, 0/30 refusals) over RpR-v4 (3.716, 1/30 loop), Pantheon-Reasoning (1.383 but 7/30 refusals), Snowdrop+Gembrain (llama.cpp template-incompat) — deployed (5f79b40); (6) Deckard→256K (41305bf); (7) dvalin CONFIRMED Deckard samplers = the live A/B set is canonical (4954ca0). **GATE LESSON: a llama.cpp reasoning seat needs a STOCK template that natively opens ``/`enable_thinking` (Qwen3.x/QwQ pass; ChatML + Gemma-4 fail) — no monkeypatching. INFRA: llama-swap b8840 can't load Qwen3.6/Gemma-4 archs → `ghcr.io/ggml-org/llama.cpp:server-cuda` (pulled on ana-ml2).** _Archived 2026-08-03._ ### 2026-07-08-off-the-shelf-inference-pivot-executed-serve-curated - `[2026-07-08]` **OFF-THE-SHELF INFERENCE PIVOT executed — serve curated abliterated models, stop home-training.** Final topology: **gen = `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-NVFP4-Experts-Only`** (LIVE, modelopt, vision, util 0.40), **char-rp = an RP unicorn to be found on fresh context** (see Current state). Intermediate steps ABANDONED: Pantheon-Reasoning-27B (served briefly as gen — refuses dark fiction via DeepSeek-distilled refusal-reasoning, see Tried); Pantheon-27B-with-MTP for RP (bf16 MTP won't load on the compressed-tensors path); Angel MS3.2-24B (my NVFP4 quant = garbage). Prefer EXISTING community NVFP4/GGUF quants over self-quanting ("don't quant unless you have to" — operator). GGUF serving is now on the table for RP (NEVER Ollama). Gateway sampling-defaults wiring still PENDING. _Archived 2026-08-03._ ### 2026-07-08-dpo-was-silently-running-3-epochs-harness-gap - `[2026-07-08]` **DPO was silently running 3 epochs (harness gap) → KILLED at epoch 1.2, retargeted to 0.3 epochs (operator call).** Root cause: `DpoConfig` had NO `epochs` field + `_dpo_config_kwargs` didn't pass `num_train_epochs` → DPO fell through to trl DPOConfig's default 3.0 (SFT correctly pins 1 via SftConfig.epochs + _sft_config_kwargs). Objective SATURATED by ~epoch 0.27 (loss~0, grad~0, acc 1.0, margins~27 flat — the off-policy qwopus rejected pairs are trivially separable), so epochs ~0.3→3 were pure over-optimization + a ~6.5h outage. No mid-run checkpoint (save_steps 500 > total steps; save only at end) → killing lost the run. FIX (3 edits to deployed harness, mtf-dev to canonicalize): `DpoConfig.epochs: float = 1` (mirrors SftConfig, float for fractions); `_dpo_config_kwargs` now passes `"num_train_epochs": cfg.epochs`; recipe `dpo.epochs: 0.3`. GPU-free verified (dpo.epochs=0.3 → num_train_epochs=0.3). Relaunched at 0.3 epoch (~30min precompute + ~12min train = ~45min). **DONE + SANITY-CHECKED (exit 0, ~70min wall: ~30min fixed precompute + 45 steps @ ~51s/step; train_loss 1.4e-5 @ epoch 0.301).** Fresh `data/spike/dpo_adapter/adapter_model.safetensors` (123MB) + checkpoint-45 banked. **3-way greedy sanity (base vs SFT vs DPO, via peft load + disable_adapter/set_adapter on GPU0): ALL THREE DISTINCT** (base≠sft≠dpo) → full SFT→DPO pipeline applies end-to-end at inference. **DPO 0.3ep is COHERENT, fluent, NOT degenerate** (early-stop avoided over-optimization) but the quality delta on a neutral literary prompt is SUBTLE (DPO shares SFT's structure — it continues from it — with minor stylistic drift, arguably slightly MORE genre-clichéd). Verdict: mechanics proven, quality gain modest as predicted for 0.3ep on off-policy pairs; the real unlock remains on-policy rejected regen + on-domain (explicit E-RP) eval + the LitBench/holdout run. gen+rp RESTORED healthy. Next: serve fork (SGLang-finish vs merge) on the DPO adapter — same rekey_lora_for_vllm.py (zero-z) applies unchanged (mtf-dev confirmed). _Archived 2026-08-03._ ### 2026-07-08-t1-dpo-leg-is-running-unblocked-2-fixes - `[2026-07-08]` **T1 DPO leg is RUNNING (unblocked) — 2 fixes applied to deployed backend.py.** Blocker resolved: (1) **mtf-dev's v0.0.42 stub** `_stub_missing_optional_integrations` (last-resort sys.meta_path finder → missing mergekit/llm_blender/weave resolve to MagicMock, never called → zero numerics risk; applied VERBATIM to deployed `src/model_training_forge/train/backend.py` after `_unsloth_available()` + call-site before `from trl import DPOTrainer`); (2) **my cosmetic `warnings_issued` shim** (trl-0.24 DPOTrainer.__init__:405 does `model.warnings_issued["estimate_tokens"]=True` for warning-suppression; custom Qwen3_5 class under transformers 5.5.0 lacks the attr → `if not hasattr(model,"warnings_issued"): model.warnings_issued={}` before the DPOTrainer(...).train() at backend.py:305 — cosmetic, zero training impact). Both edits are on the DEPLOYED un-git'd copy only → **mtf-dev must canonicalize the warnings_issued shim into their repo** (told them). DPO confirmed training: model loaded (851 shards), full 1196 pairs processed, in precompute_ref_log_probs (GPU0 93% util, 54.8GB). Completion watcher armed (bg task) → restore gen+rp + verify dpo_adapter + ping mtf-dev on exit. gen+rp STOPPED for the run (authorized window). Output → data/spike/dpo_adapter. _Archived 2026-08-03._ ### 2026-07-08-t1-dpo-leg-launch-prior-block-now-resolved - `[2026-07-08]` **T1 DPO leg launch — prior BLOCK (now resolved above), kept for the launch recipe.** Operator authorized the full DPO stage (via mtf-dev) + went AFK 2h. **PROVEN LAUNCH RECIPE** (replicates the SFT container `aeon-t1-sft` exactly, only `--stage sft`→`dpo`): `sudo docker run -d --name aeon-t1-dpo --entrypoint python3 --gpus all -e CUDA_VISIBLE_DEVICES=0 -e MTF_FORCE_TRL=1 -e PYTHONPATH=/mtf/src -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -v /home/lkraven/model-training-forge:/mtf -v /tank:/tank -w /mtf aeon-trainer:latest -u scripts/train.py --recipe recipes/training/qwen-3.5-122b-erp-lora/train.aeon-27b.yaml --stage dpo`. **CRITICAL: `--entrypoint python3` is REQUIRED** — aeon-trainer's default entrypoint is `["vllm","serve"]` (FROM vllm/vllm-openai) → without the override it runs vllm + hits a torch-ABI crash. Dataset verified (pairs_dataset=train.flat.json=1196 pairs). **THE BLOCK:** `from trl import DPOTrainer` (backend.py:256) eagerly pulls TRL 0.24.0's WHOLE optional-integration set — cascade: mergekit(missing)→ immutables→**mergekit-0.1.4↔pydantic-2.13 HARD incompat** (needs pydantic==2.10.6)→llm_blender→dataclasses_json→ **llm_blender-0.0.2↔transformers-5.5.0 HARD incompat** (TRANSFORMERS_CACHE removed, needs source patch)→weave→ (more). NONE used by our pair-based DPO. `pip install mergekit` w/deps is UNSAFE (downgrades accelerate 1.14→1.6). Safe partial recipe derived (core libs held: torch2.10/tf5.5.0/trl0.24.0/peft0.19.1/accel1.14.0) but non-convergent → TRULY BLOCKING per operator's carve-out. Did NOT force-hack the proven training image. Handed full diagnosis + recommended fix (lazy-import TRL patch, opt b) to mtf-dev (thread 01KWZG8GJX, expects-reply, monitor armed). gen+rp RESTORED healthy. Relaunch = 1 min once mtf-dev delivers a working image. _Archived 2026-08-03._ ## Recent decisions (archived 2026-08-05 batch) - `[2026-07-16]` **GPU re-org: char-rp→GPU1 + both cards re-optimized for max context.** Moved char-rp (Magidonia-24B) GPU0→GPU1, then maxed context: char-rp-reasoning 150K→256K (util 0.46, 1.56x), gen→256K + seqs 16→32 (util 0.42, 5.43x), granite 64K→**128K full-chapter** (util 0.27, 1.50x). FINAL: GPU0 ~14 G reserve (both seats 256K native), GPU1 ~6.7 G headroom. All healthy. LESSON: KV must hold ≥1× max-len (util-floor crashes) + per-model KV cost varies ~8× (MoE cheap, dense pricey) → tune util empirically. _Archived 2026-08-05._ - `[2026-07-16]` **granite right-sized → ~10.5 GB freed on GPU1** (util 0.34→0.18 + max-len 131072→65536; KV 6.45 GiB / 1.29x@65536). LESSON: ~950 MiB KV per 0.01 util here + KV must hold ≥1× max-len — util 0.15 crash-looped before 0.18 landed. `.env`-only, recreate `vllm-granite` alone (shared stack). [Superseded by the 07-16 GPU re-org above → 128K.] _Archived 2026-08-05._ - `[2026-07-15]` **image-bench eviction DONE (parked item closed).** Stopped vllm-qwen-image-bench (ana-ml2 GPU1, ~32 GB freed); LiteLLM `image-judge`+`qwen-image-bench` → gen :8015 (judge samplers + thinking-off); comfy-dev pinged; backfilled the canonical char-rp-reasoning litellm block. Revert ~90 s. auto-memory `project_arbo_gen_switch_imagebench_evict`. _Archived 2026-08-05._ - `[2026-07-15]` **Homepage AI-tab revamp** — flat "AI Systems" group → dedicated AI tab, 6 role-based groups + AI-Dormant; committed `569e1af`, pushed. (Also caught + pushed a ~100-commit unpushed eshpfi backlog.) _Archived 2026-08-05._ - `[2026-07-15]` **Home Assistant config repo created** (`vh/home-assistant-config`, private). UI-managed HA → allowlist model (YAML + curated secret-free `.storage` subset). git-in-place in `/config` on esh-docker-vm + scoped deploy key + local clone `~/development/home-assistant-config`. _Archived 2026-08-05._ - `[2026-07-15]` **char-rp-reasoning OOM rescue** — solo-restart on the packed GPU0 crash-looped; fixed via `expandable_segments:True` + util 0.39→0.38 + max-model-len 192K→150K. LESSON: `max-model-len` does NOT free vLLM VRAM (util-pinned KV pool). ~4.5 GB GPU0 headroom. _Archived 2026-08-05._ - `[2026-07-15]` **soong-lab `SOONG_LAB_LIBRARY_DIR` made persistent** (corviduo-dev) — was on the redeploy-wiped code default; set to `/home/infra-ops/soong-lab-data/library`, restarted. Closed a queued no-rush item. _Archived 2026-08-05._ - `[2026-07-15]` **Statusline overhauled** (`~/.claude/statusline-command.sh`) — git state / 🔔🔕 monitor-armed / project tag / abs tokens / per-session cost / threshold-colored ctx+rate. _Archived 2026-08-05._ ## Tried and abandoned (archived) — moved 2026-08-12 - `[2026-07-01]` **MTP/spec-decode on a SHARED serving model helps single-stream but HURTS moderate-concurrency aggregate + silently ignores `min_p`/`logit_bias`** (qwopus `gen`: N=1 +12%, N=4 −20%). Reserve for dedicated/interactive deployments. _Archived 2026-08-12._ - `[2026-07-02]` **irv-ml1 `/worktank` ROOT is root-owned — lkraven can't write there (irv-ml1 sudo needs a password) → stage model pulls to `/home`.** PIN THE A6000 BY UUID for training (native-CUDA ordering differs vs docker; the 3090 index 0 is usually near-full → OOM). `CUDA_VISIBLE_DEVICES=GPU-`. _Archived 2026-08-12._ ## Recent decisions (archived) - `[2026-07-18]` **worldtree-sdk 1.0.0 (Python) published to the internal vh Gitea PyPI** (wtsdk-dev request; the npm/TS side shipped prior session). Built from tag `python-v1.0.0` (clean worktree), `uv publish` → `https://gitea.phasefinal.com/api/packages/vh/pypi`; acceptance `uv pip install worldtree-sdk==1.0.0` (vh index as extra-index-url) resolves + imports, __version__ 1.0.0. Registry already existed (bifrost publishes there; soong-lab consumes it via `[[tool.uv.index]] name=gitea`). Publish cred = the vh `write:package` PAT the operator had already handed over (in `worldtree-sdk/.npmrc` `_authToken`) — Gitea `write:package` is package-type-agnostic, so the npm-publish token published PyPI too. Consumers install like bifrost (add the vh index + a read token). [[reference_worldtree_demo_key_mint]] _Archived 2026-08-15._ - `[2026-07-18]` **nh3-dev /tmp auto-clean enabled** — Debian ships /tmp with no tmpfiles age (`D /tmp 1777 root root -` → never cleans); this high-churn agent box had accreted **~190k stale temp dirs / 25G**. One-shot manual purge (194k→10k entries, 25G→1.7G; deleted top-level dirs/files >1d old, spared `/tmp/claude-*` by name + anything ≤1d). Then `/etc/tmpfiles.d/tmp.conf` = `D /tmp 1777 root root 3d` (daily `systemd-tmpfiles-clean.timer` removes >3d-untouched items; active files + socket dirs spared). Tunable via the age. Note the churn: ~10k /tmp entries/day here. _Archived 2026-08-15._ - `[2026-07-18]` **soong-lab containerize cutover — COMPLETE + LIVE on corviduo-dev.** Migrated soong-lab (Noonien Soong character-design studio) from a hand-built `soong-lab-studio.service` (systemd + git-pull-on-webhook) to a containerized deploy, image built by CI + pushed to the Gitea registry. soong-dev owns the in-repo artifacts (Dockerfile/compose/workflow/`docs/DEPLOY.md` = checklist); infra-ops owned the host cutover. Operator confirmed functional ("Soong works great" — a real Soong turn round-trips + saves) → cutover 100% closed. **Final state (corviduo-dev, 10.250.50.152):** - Container `soong-lab-soong-lab-1` LIVE + healthy on `0.0.0.0:8443`, image `gitea.phasefinal.com/vh/soong-lab:latest` (v0.3.24), `restart:unless-stopped` (survives reboot; no systemd unit needed — docker restart policy handles boot). - Deploy dir **`/home/infra-ops/soong-lab-deploy/`** — pull-based `compose.yaml` (image + env_file + `8443:8443` + named volumes; NO build/secrets stanza) + `.env` (copied from the live `soong-lab.env`, STRIPPED of the `SOONG_LAB_*_DIR` overrides so the container uses image defaults `/data/library` + `/data/portraits` + `/app/web` → the volumes). - Named volumes `soong-lab_soong-library` + `soong-lab_soong-portraits`, migrated from `/home/infra-ops/soong-lab-data/{library,portraits}` (2 saved designs incl. **Sindra** + 27 portraits), **chowned `10001:999`** (the container `soong` user) so it can read AND write new designs. - Old `soong-lab-studio.service` + `soong-webhook.service` (the `:9010` git-pull redeploy listener) both **stopped + disabled**. **Topology reality (≠ what DEPLOY.md assumed):** there is **NO TLS proxy**. WT-personal (`:8081`) and soong-lab are **co-located on corviduo-dev**, and the Bifrost callback is **plain-HTTP same-host** `http://10.250.50.152:8443` — the value of `SOONG_LAB_BIFROST_ENDPOINT_URL`, unchanged by the move, so the WT Bifrost host-allowlist stayed valid as-is. Nothing on the WT side needed touching. **Safety net:** data backup `/home/infra-ops/soong-lab-data-backup-20260718-091831.tar.gz` (35M) taken BEFORE migration. Verified pre-retire: `/api/version` 200 (0.3.24), SPA `/` 200, `POST /bifrost/tool-call` → 401 (route present + auth-gated), bidirectional WT↔soong reachability, container healthcheck green. **Ops commands:** - Redeploy a new image: `cd /home/infra-ops/soong-lab-deploy && sudo docker compose pull && sudo docker compose up -d`. (Auto-pull-on-`:latest` — watchtower or a deploy hook — is an open follow-up.) - Rollback: `sudo docker compose down` + `sudo systemctl enable --now soong-lab-studio.service soong-webhook.service`. - Homepage tile: manual `- Apps:` entry "Soong Lab" (href http://10.250.50.152:8443) in esh-docker-vm `/opt/docker/conf/homepage/services.yaml` — corviduo-dev isn't a Homepage-watched docker endpoint, so docker-label auto-discovery can't surface it (see [[2026-07-18-fleet-gitea-runner-build-recipe]] for the CI half). See [[reference_corviduo_dev_emergency_ops]], [[reference_claude_bot_gitea_creds]]. _Archived 2026-08-15._ - `[2026-07-18]` **zonos-gateway 0.2.1 — voice-resolved emotion presets baked (provisional) from the axes sweep.** After the axes sweep ([[reference_zonos_tts_stack]] + the `[2026-07-18] axes sweep` Recent-decisions entry) rescued angry and confirmed startled-happy, the operator green-lit baking the results as **provisional** gateway presets + docs. Shipped `vh/zonos-gateway` **0.2.1** (main `8f1885b`, tag `v0.2.1`, PUSHED; deployed live on irv-ml1 `:8890`). **Design — voice-resolved, NOT global.** `resolve_preset(name, voice)` picks the per-voice measured cell, because a single global preset is unsafe (dvalin ruling; BritishFemale's *named* angry misfires as fear). Presets: - `angry`, `happy`, `startled_happy` (+ aliases `surprised`, `startled` → startled_happy). All expressive (`accurate_mode:false`), cfg 1.5, pure-axes (no named sliders). - Calibrated cells (the 3 default voices): - angry: AmF v-0.4/a+1.0 s1.0 (emo0.53/id0.685); BrF v-0.4/a+0.8 s1.0 (emo0.99/id0.725, metric fear-clean); AmM **two-tier** — soft v-0.6/a+0.8 s1.0 (0.23/id0.654) + drama v-0.6/a+0.8 s1.2 (1.0/id0.616 clean; strength is NOT a smooth knob on AmM, 1.0→1.2 is the window, past that flips to disgust). - happy / startled_happy: AmF v+0.6/a+0.8; AmM v+0.3/a+1.0; BrF v+0.6/a+1.0 (happy~1.0, id 0.74-0.80; axes-happy keeps +0.15 id over the named happy slider). - `sad` = unchanged named-slider preset (not axes-tested). - Uncalibrated voices (Cora + the 4 clones) → mid-region fallback until measured. - Docs surface: `/v1/dials` exposes `voice_emotion_presets`; the FastAPI `/docs` description documents it; durable spec `docs/EMOTION-DIALS-SPEC.md` (moved INTO the repo — was mirror-only); README table. 44 tests green. **Repo-hygiene gotcha (fixed).** The local clone `~/development/zonos-gateway` and gitea `vh/zonos-gateway` had **TWO UNRELATED git histories** (no merge-base) — gitea held the voice-wav commits, the local clone held the code + no remote. Reconciled by resetting local→origin/main, overlaying the 7 bake files, `uv lock`, commit, push (fast-forward). Voices stay tracked; local now shares gitea's lineage + has origin wired. **The deployed irv-ml1 tree `/opt/docker/compose/zonos-gateway` is still NON-git** (hand-updated build context) — CI-wire remains an open follow-up. **Provisional pending** ear-validation on emotion-congruent text (the neutral-text audition was inconclusive: "they all sound different, hard to tell"). Follow-ups: sad axes/text pass on the 3 voices; congruent-text pass; clone-char emotion rows. Tools `~/development/zonos-tools/{axes_sweep,strength_ladder,gen_auditions,dial-in-studio}.py` (run ON irv-ml1; scoring env `uv run --with resemblyzer --with funasr --with "numpy<2" --with soundfile --with requests --with "setuptools<80" --with torchaudio`). _Archived 2026-08-15._ - `[2026-07-18]` **Fleet Gitea-Actions build recipe + the `vh`-is-a-user package-write constraint** (learned the hard way across 3 failed soong-lab validation builds; reusable for ANY fleet CI image build or package publish). **The runner.** One `act_runner` (`gitea/act_runner`) on ana-docker, labels `pfi-fleet` / `ana-docker` → both map to job image **`node:20-bookworm-slim`**, which has **NO docker and NO git**. Config `/opt/docker/conf/gitea-runner/data/config.yaml`: `valid_volumes: []` (no socket propagated to job containers). So: - `actions/checkout@v4` fails (needs git); `docker/*` marketplace actions fail (need docker) — a workflow built on those dies at the first step (~15s). **The working recipe (mirror Worldtree `deploy.yml`).** Run the job in a docker-capable image + drive docker with RAW commands, not the JS actions: ```yaml runs-on: pfi-fleet container: image: docker:24.0.7-cli # has docker+buildx; add git+node steps: - run: apk add --no-cache git nodejs # so actions/checkout@v4 works - uses: actions/checkout@v4 - name: login # RAW, not docker/login-action run: echo "$REGISTRY_TOKEN" | docker login gitea.phasefinal.com -u "$REGISTRY_USER" --password-stdin - name: buildx builder run: docker buildx create --name X --driver docker-container --use; docker buildx inspect --bootstrap - name: build+push # RAW, not docker/build-push-action run: docker buildx build --secret id=,env= -t :latest --push . ``` The runner mounts the host docker socket into ITSELF; the docker:cli job reaches the daemon through that. The `docker/*` JS actions are unreliable on act_runner — raw commands are the fleet convention. **`vh` is a USER account, not an org.** Consequences that bit repeatedly: 1. `GET /api/v1/orgs/vh` → 404 "user redirect"; there are **no org teams** to add a service account to. 2. **User-owned packages are OWNER-WRITE-ONLY.** claude-bot (even repo admin-*collaborator* on `vh/soong-lab`, even with `write:package` scope + full basic-auth) gets **`401 unauthorized`** on `docker push` to `vh/soong-lab`, and `npm publish` to `vh/npm/` would 401 too. Only `vh` itself can write vh packages. → CI must authenticate AS `vh` for the push (a vh-owned `write:package` PAT as `REGISTRY_TOKEN` + `REGISTRY_USER=vh`), exactly how WT pushes `vh/worldtree`. claude-bot CAN still: clone/read repos, READ packages (pulled the image fine), dispatch workflows, mint demo Worldtree keys. 3. **Repo Actions secrets are OWNER-ONLY too** — `PUT .../actions/secrets/X` as claude-bot (repo admin-collab) → 403 "user should be the owner of the repo". Only `vh` can set a repo's secrets. **Other gotchas:** - Gitea **reserves the `GITEA_` secret-name prefix** — a secret named `GITEA_PYPI_TOKEN` is illegal; use e.g. `PYPI_TOKEN`. - Gitea **package auth is token-based / username-lenient** — `docker login` / PyPI basic-auth authenticate via the token; the username is nominal (tested `-u gitea` and `-u claude-bot` both 200 against the vh PyPI). So a Dockerfile hardcoding `UV_INDEX_GITEA_USERNAME=gitea` is fine with any valid token. - Homepage (esh-docker-vm) docker-label auto-discovery only covers the 5 endpoints in its `docker.yaml` (esh-vm-docker, ana-docker, ana-ml2, nh3-docker, irv-ml1); **corviduo-dev is NOT watched** → services there need a manual `services.yaml` entry, not labels. Applied in the soong-lab CI: [[2026-07-18-soong-lab-containerize-cutover]]. _Archived 2026-08-15._ - `[2026-07-18]` **Peer credential provisions — Wyrd conv-api key + wtsdk npm token, both delivered + closed.** Wyrd: demo Worldtree user-tier key (key_id `da7a0bdf`, user_id `wyrd-dev`) minted via `docker exec worldtree-worldtree-api-1 /admin/keys` (omit tier→user), drop-and-shred delivery. wtsdk: operator-minted vh `write:package` PAT relayed drop-and-shred → worldtree-sdk@1.0.0 published to `vh/npm/`. Secret-delivery pattern = drop to a mode-600 file on the peer's box, they collect+shred+confirm, then shred the holding copy; NEVER cleartext over althing. [[reference_worldtree_demo_key_mint]] _Archived 2026-08-15._ - `[2026-07-18]` **Axes sweep RESCUED angry; surprised-class dead but startled-happy ships.** Valence×arousal grid on the 3 calibrated defaults (AmericanFemale/Male, BritishFemale), exp/cfg1.5/strength1.0, 84 clips, emotion2vec + resemblyzer scored, graded vs dvalin's floor. **ANGRY rescued** (named direction was 0.004–0.15, British named-angry even misfired as fear 0.89): axes ship cells at **negative valence (−0.4..−0.8) + high arousal (+0.8..+1.0)** — BritishFemale v-0.4/a+0.8 angry=0.99/id0.725 SHIP, AmericanFemale v-0.4/a+1.0 angry=0.53/id0.685 SHIP; AmericanMale two-tier post-ladder (no single ship cell — best drama = v-0.6/a+0.8 str1.2 angry=1.0/id0.616 clean, soft = same cell str1.0 angry0.23/id0.654; cell A v-0.6/a+1.0 is a non-monotonic minefield, skip). BrF ship cell proxy-CLEAN of fear (str<1.0 just kills anger). **SURPRISED-class DEAD** (max 0.047 across all 84 cells) but **startled-happy** (happy-proxy) ships all 3 at high arousal + neutral/positive valence, with a **+0.17–0.20 identity LIFT** over the named-surprised route (named hits happy~1.0 but at id0.57–0.61, under floor; axes hits happy~1.0 at id0.74–0.80). Bonus: axes-happy retains ~0.10–0.15 more identity than the named happy slider too. Caveats: response surface non-monotonic/sharp-thresholded; angry region borders fear/disgust (bleed); emotion2vec saturates at 1.0 (needs ear-confirm); neutral text understates. Tooling `~/development/zonos-tools/axes_sweep.py`; per-clip JSON was `irv-ml1:/tmp/axes_sweep_results.json` (ephemeral). Sent dvalin msg `01KXT2ZB8G…`. NEXT = operator ear-confirm → bake presets. [[reference_zonos_tts_stack]] _Archived 2026-08-15._ - `[2026-07-18]` **Zonos2 emotion CANONICAL from an empirical sweep + the voice-cloning pipeline.** **Voice-cloning pipeline (established this session).** Source zips at `/mnt/smithy/voice_clones/.zip` (irv-ml1 NFS from nh3-nas; remount post-reboot) — each = diarized single-speaker podcast clips + `manifest.jsonl` (per-clip WhisperX `mean_score`, word timestamps, text) + `metadata.csv`. `~/development/zonos-tools/assemble_voice.py ` ranks by mean_score and concatenates top clips to ~15–24s (Zyphra's blessed clone-ref length; single clip if already ≥15s). Drop the assembled `.wav` into the gateway voices dir → `voice:"name"`. 4 characters cloned: **Emmie, Penny, Natalie, Miranda** (+ Zyphra defaults AmericanFemale/Male/British/Cora) = 8 voices in `zonos-gateway`. Clone is inline `speaker_audio_base64` (text-independent Qwen3 speaker embedding — NO transcript); `/tts/speakers` registration is session-scoped (needs `X-TTS-Session-ID`), so the gateway holds the ref wav and clones per-call. **Gateway voices are host-managed (bind-mount, added this session).** Added `./voices:/app/voices:ro` to `/opt/docker/compose/zonos-gateway/compose.yaml` (committed to `vh/zonos-gateway` + eshpfi mirror `438cd35`). So adding a voice = drop the wav + `docker compose restart zonos-gateway` (registry rebuilds at boot; NO image rebuild). This also un-stranded the other voices (deploy build context had only Cora before). Voice wavs committed to the repo for backup. **Emotion mechanism (Zyphra canonical, from their README @194c0a3).** Additive direction vectors: 4 named (happy/sad/angry/surprised) + valence/arousal axes. `emotion_strength` 1.0 = per-voice calibrated (calibration.json optimizes emotion2vec recognizability only, NOT identity). `accurate_mode` is THE trade-off: `true` = closer voice match (identity), `false` = expressive mode (emotion lands, identity drifts). Zyphra's strong recipe: `accurate_mode:false` + `cfg~1.5`. Single-emotion is blessed; mixing is unblessed (and degrades the clone — operator confirmed by ear). "deaf by 1.5" — cfg past 1.5 distorts + costs ~2× compute. **THE SWEEP (`~/development/zonos-tools/emotion_sweep.py`).** 4 cloned voices × 4 named emotions × {accurate,expressive}×{cfg 1.0,1.3,1.5} @ strength 1.0, single-emotion, neutral sentence + a neutral baseline per voice (~100 clips). Scored on TWO axes: **emotion-landing** = emotion2vec `iic/emotion2vec_plus_large` target-emotion prob [0-1]; **identity** = resemblyzer speaker-embedding cosine vs the clone reference (neutral baseline ~0.85). Scoring env: `uv run --with resemblyzer --with funasr --with "numpy<2" --with soundfile --with requests --with "setuptools<80" --with torchaudio` (setuptools<80 for webrtcvad's pkg_resources; torchaudio for funasr). **RESULTS (mean across the 4 voices) — emotion, best setting, emo/id:** - happy — **exp cfg1.5** 0.80/0.68 (soft: exp cfg1.0 0.76/0.69) → WORKS - sad — **exp cfg1.5** 0.53/0.57 (only working cell; id below the ~0.65 floor) → modest - angry — acc cfg1.3 / exp cfg1.5 tied at ~0.25 emo → WEAK (named ceiling ~0.25) - surprised — max ~0.015 across ALL settings → NON-FUNCTIONAL on the named direction Accurate + low cfg = identity/suppress regime (emo→0); expressive REQUIRED for emotion to land, at ~0.15–0.28 identity cost. **dvalin-smithy-dev synthesis (adopted, triaged genuine-adds; thread `01KXT12FN0AS5A3WMKEK06BVPS`):** 1. Treat **identity as a hard FLOOR (~0.65)**, not a free variable in emo×id. 2. **Two-regime policy** — Regime A (default, identity-critical dialogue): `accurate_mode:true, cfg 1.0, emotion off` (text carries it) or soft-happy (exp cfg1.0). Regime B (tagged drama beats): `accurate_mode:false, cfg 1.5`, single emotion or axes. Line-type→regime heuristic (exposition→A, grief→B+sad, confrontation→B+axes-angry, shock→B+axes-arousal). 3. **Axes-first for the broken emotions** — angry ≈ valence −0.6..−0.8 / arousal +0.5..+0.8; surprised ≈ valence +0.2..+0.4 / arousal +0.7..+1.0 (exp cfg1.5); or "startled-happy" (happy + high arousal) as a surprised stand-in. These are PROVISIONAL — the sweep did NOT test axes. **NEXT (highest VoI, operator to green-light):** an **axes sweep** for angry/surprised (valence×arousal grid) — the only path to rescue the two broken named emotions; then a strength ladder at the best cells + emotion-congruent text (neutral content understates landing) + per-voice tables + a 2nd emotion judge / human pairwise. Then bake the happy/sad canonical into gateway presets. I owe dvalin the axes-sweep numbers. See [[reference_zonos_tts_stack]]; dials-first spec at `vh/zonos-gateway` `docs/EMOTION-DIALS-SPEC.md`. _Archived 2026-08-15._ - `[2026-07-18]` **yt-voice-clipper: A6000-pin fix + v0.3.3 redeploy.** Fixed a latent misconfig — the host override *said* "pin worker to A6000" but `NVIDIA_VISIBLE_DEVICES` was `"0"` (the 3090); re-pinned worker+api to the A6000 by UUID (`GPU-9672f0d5`, 3090 is zonos2's). Then redeployed api+worker to v0.3.3 (`docker compose up -d --build`; SPA+Python; `max_gap` 0.6→1.2s; stderr surfaced in job.log). A6000 + version verified; yields test in-flight (job `f3ff746dbae9494d`). yt-voice-clipper-dev thread `01KXT0T6GYHB`. [[reference_ytvc_autodeploy]] _Archived 2026-08-15._ - `[2026-07-17]` **Worldtree #365 internal-comms config CLOSED (demo+personal → b125) + WT#368 cross-agent memory-leak forensics + PERSONAL agent-memory scrub.** #365: staged the internal-tiers/rules/gate on both instances' bind-mounts (byte-exact vs baked b125), both now live on b125. WT#368 (read-only): the operator's name was in NO recall store on demo; on PERSONAL it sat in `lofn.chroma` (old-code `saga-v1` seeding + legacy contamination), and a clean-slate marker test proved **current b125 code isolates character-session extraction correctly** — the leak is legacy data, not a live bug. Operator-directed → executed a full PERSONAL agent-memory scrub (backup `/opt/worldtree-personal/agent-memory-backup-20260717-181004.tar.gz`; conversations/mood/auth preserved). worldtree-dev owns the code-fix/data contract. [[reference_corviduo_dev_emergency_ops]] _Archived 2026-08-15._ - `[2026-07-17]` **Zonos emotion levers RESOLVED: text-priming is FLAT → the working lever is ZONOS2's native emotion-steering, which the gateway ALREADY exposes as presets.** The prosody-priming A/B (prime→generate→excise, silence-gap cut, parakeet-validated) was operator-judged FLAT on this checkpoint — text doesn't move it. Native `emotion_directions/` (happy/sad/angry/surprised + valence/arousal axes, per-speaker calibrated for AmericanFemale/Male/British) clearly WORKS (sad→slow/quiet, excited→fast/bright, etc.). **`zonos-gateway:0.2.0` (:8890) already wires it**: simplest caller path = `POST /v1/audio/speech {preset:"…"}` — presets neutral/warm/excited/sad/intense/whisper (defined in `~/zonos-gateway/src/zonos_gateway/dials.py`), reached via the **LiteLLM `ext-tts` alias** (engine-neutral swap point; consumers never call the gateway by name). RTF measured on 3090: cfg1.0 steering = FREE (~0.52 = neutral, additive vectors), cfg1.5 amplified ~0.625 (~+20%, still realtime). Captured the live gateway stack → `stacks/zonos-gateway/` (compose+env+README); ⚠️ gateway SOURCE at `~/zonos-gateway` on irv-ml1 is NOT in gitea (backup gap, follow-up); `stacks/zonos` (v0.1 Gradio) marked DEAD/superseded. Whisper is a composed preset (no whisper *direction*; escalation for hard affects = custom directions via `scripts/build_emotion_directions.py` or emotional-ref cloning `speaker_audio_base64`). Harnesses in scratchpad (not yet landed). [[reference_zonos_tts_stack]] _Archived 2026-08-15._ - `[2026-07-17]` **Zonos2 `:1920` engine → self-contained container (stays on 3090); prosody-priming is a SERVING-LAYER change (engine stays stock).** **Context.** The production Zonos TTS engine (irv-ml1 `:1920`, feeds asset-engine + gateway-chat via `zonos-gateway` :8890) was a bare native process — its real launch config existed ONLY in the running process argv (the committed `~/tts-audition/harness/zonos_server.sh` was STALE: said A6000/:1919/no perf flags; live is 3090/:1920 with `--cuda-graph-max-bs 1 --num-pages 16384 --max-running-requests 2 --memory-ratio 0.3`). Captured to eshpfi `stacks/zonos-engine/` (README + corrected `zonos2-server.sh` + `.env.example`), commit **14a0004** (UNPUSHED as of the snapshot). **Decision 1 — containerize as a SELF-CONTAINED image** (not systemd — operator rejected; not a thin bind-mount wrapper — I walked that back: bind-mounting the host's CUDA-compiled `.venv` couples to the host's exact CUDA/glibc and is fragile + not reproducible). Shape: `FROM` a CUDA 12.8 base → `uv sync` against the repo's committed `uv.lock` (deterministic env) → mount the ~15 GB HF weights (`~/.cache/huggingface/hub/models--Zyphra--ZONOS2`, do NOT bake) → pin the **3090** (`NVIDIA_VISIBLE_DEVICES=0`) → `restart: unless-stopped` → CMD = the captured invocation. **Engine stays STOCK** Zyphra/Zonos2 @ commit `194c0a3` (no fork — the `zonos2` package ships its own server). **Build risk:** heavy compiled-CUDA deps (flashinfer / sgl_kernel / cutlass-dsl / apache-tvm-ffi / pynini) on torch 2.9.1+cu128 — mostly prebuilt wheels + the `uv.lock` make it tractable, expect a couple build iterations. **Cutover (in place on the 3090):** stop the native process (frees ~17 GB) → `docker compose up -d` (re-allocates ~17 GB, same footprint) → repoint `zonos-gateway`'s `ZONOS_URL` at the container (or keep the `:1920` host-port publish). One brief prod-TTS blip. **GPU = 3090 (operator 2026-07-17).** Keep it OFF the A6000 — the A6000 already OOMs under ComfyUI load (idle ~19 GB but spikes far higher during gen), so it can't host Zonos too. The 3090 already runs Zonos, so the containerize-in-place cutover changes nothing about placement. **Decision 2 — the prosody-priming hypothesis (operator's test; the reason for building fresh).** PRIME the autoregressive engine with an emotional sentence, then TRUNCATE it from delivery: prepend a primer → **generate "primer + real text" as ONE continuous utterance** (the AR model carries prosody forward across the boundary) → ASR-timestamp the primer's end (**parakeet**, already up on irv-ml1 `:8765`, word timestamps) → **clip the primer in the inter-sentence silence gap** (+ ~15 ms fade-in, no click) → deliver only the real text, now wearing the primed prosody. Examples: primer "I'm so EXCITED about this." → "This will be a lot of fun!" spoken excited; primer "I'm whispering this to you right now." → "I'm so glad to see you baby." whispered. **This is PURE serving-layer orchestration — the engine is untouched; it lives in the gateway adapter `stacks/zonos/adapter/server.py`.** Only fork the engine if the black-box approach fails. **THE CRUX the test resolves:** does AR prosody actually **carry across the sentence boundary**, or does Zonos reset at the period? → the harness A/Bs the **JOIN punctuation**: period (operator's examples) vs comma vs ellipsis vs none ("…excited about this, this will be…"). Everything else is plumbing. **Plan / design recs.** (a) Build the stock engine image (parallel track). (b) Stand up a priming TEST HARNESS against the NATIVE engine (fast iteration, seconds) + parakeet ASR: prime→generate→timestamp→gap-clip→out; compare primed-clipped vs plain on the two cases (subjective + a cheap objective proxy: pitch/energy variance for "excited", spectral-tilt/low-energy for "whisper"). Iterate on the join, then bake the winner into the gateway adapter. **Primer source:** caller-supplied for the harness (test arbitrary primers) → a curated emotion→primer library (`excited`/`whisper`/…) + optional caller override for production. **ASR:** parakeet primary; WhisperX forced-align fallback if parakeet word timestamps are coarse. See eshpfi `stacks/zonos-engine/README.md` + `stacks/zonos/` (the gateway adapter). _Archived 2026-08-15._ - `[2026-07-15]` **esh-docker-vm NFS fstab fix = `x-systemd.before=docker.service`** (the prior `After=remote-fs.target` drop-in was silently defeated by `nofail`). Reached only after a REBOOT (D-state phantom containers uptime-kuma + paperless-web that no `docker`/`ctr`/daemon-restart could clear). Committed `21d9a07` + playbook updated. See Tried and abandoned. _Archived 2026-08-15._ ## Tried and abandoned (archived) - `[2026-07-15]` **`docker.service After=remote-fs.target` does NOT wait for `nofail` NFS mounts** — `nofail` drops a mount out of remote-fs.target's blocking set, so the drop-in ordering is silently defeated (paperless still Exited(255) on reboot). Real fix = DIRECT mount->docker ordering via the fstab `x-systemd.before=docker.service` option (verify `systemctl show docker -p After` lists the mnt-*.mount units). esh-docker-vm. _Archived 2026-08-15._ - `[2026-07-15]` **The esh-docker-vm D-state/phantom-container wedge is only cleared by a host REBOOT** — reconfirmed: `docker stop/rm -f`, `ctr -n moby task delete`, AND `systemctl restart docker` all fail to clear it; `docker exec` into a wedged container ALSO fails (`setns ... exit status 1`), so the in-place restart escape hatch is out. Worse, a daemon restart can HALF-KILL other healthy containers (knocked paperless's granian down + left it wedged). Process dead but dockerd won't reap -> phantom. NFS mounts are `_netdev,nofail` so the reboot is boot-safe. _Archived 2026-08-15._ - `[2026-07-15]` **vLLM `max-model-len` does NOT free GPU VRAM** — the KV cache POOL is sized by `gpu-memory-utilization`, not max-model-len. Lowering max-model-len only caps per-request context + drops max concurrency; the pool still fills the util budget. To actually free VRAM, lower `gpu-memory-utilization`. (Bit the char-rp-reasoning "drop KV to 150K" ask: the 150K applied but freed 0 VRAM until util dropped 0.39->0.38.) _Archived 2026-08-15._ - `[2026-07-15]` **Claude Code statusline `.cost.total_cost_usd` is per-SESSION** (Claude Code's own cache/model-aware session accounting), not a lifetime aggregate — the large value just reflects a long, multiple-times-summarized session. And the old statusline hardcoded Sonnet pricing ($3/$15) on an Opus session -> ~5x cost understatement. _Archived 2026-08-15._ - `[2026-07-14]` **MTP-on-modelopt: NO checkpoint config skips the spec-decode drafter's quant (vLLM 0.24 bug) — 4 config attempts failed before the runtime workaround.** All crashed the same way (`qwen3_5_mtp.py:256` `param_data.shape == loaded_weight.shape` AssertionError — bf16 mtp head loaded into a quantized drafter param): (1) mtp excludes in `config.json` (WRONG file — vLLM modelopt reads `hf_quant_config.json`); (2) specific-unfused mtp names in hf_quant_config; (3) wildcards `mtp*`/`mtp.layers.0*` (`is_layer_skipped` is EXACT-membership, NOT glob — wildcards match nothing); (4) exact fused+unfused names in both `mtp.`/`model.` prefixes. Instrumenting `is_layer_skipped` proved the drafter's exclude list holds ONLY the main model's `linear_attn` entries — the mtp excludes never reach the draft-model quant config. ONLY fix = a mounted `sitecustomize` force-skipping `mtp.*`. LESSON: don't chase checkpoint-config fixes for the mtp-drafter crash; go straight to the runtime patch. Also `nvidia-modelopt[hf]==0.43` (AEON's producer version) is a trap — it pins transformers back to 4.57 which can't load `qwen3_5` at all; use 0.45 + the FusedMoE guard in `quant_modelopt.py`. _Archived 2026-08-15._ - `[2026-07-14]` **AEON's "working NVFP4+MTP RP seat" was pantheon on compressed-tensors (0% MTP accept), not a modelopt MTP proof.** `vllm-aeon-rp`'s .env → `AEON_RP_MODEL=pantheon-27b-mtp-nvfp4`, `AEON_RP_QUANT=compressed-tensors` — it LOADED (mtp silently skipped, `exited 0`) but never accelerated. Same vLLM image (`:latest` = `sha256:4091d55` = 0.24.0) as the failed Heretic2 test, so the "AEON ran on an older vLLM" theory was wrong. Don't treat a seat that "ran" as MTP-validated without checking its `SpecDecoding` acceptance. _Archived 2026-08-15._ - `[2026-07-14]` **NVFP4 (llm-compressor / compressed-tensors) gives NO batch-1 speedup over GGUF for the Qwen3.5 GDN-hybrid, and its MTP is 0%-accept.** Measured base NVFP4 no-MTP ≈53 tok/s decode vs the GGUF NEO-CODE seat ~59.5 (llama.cpp wins single-stream; NVFP4's edge is concurrency, and this hybrid is bandwidth-bound at batch-1 with the BF16 linear_attn/GDN layers dominating). MTP spec-decode = 0% acceptance (vLLM's `Qwen3_5MTP` drafter won't load the bf16 mtp weights off a compressed-tensors main model → `Parameter … not found in params_dict`, `Avg Draft acceptance rate: 0.0%`). Pantheon is identical — its "working NVFP4+MTP" was working *structure*, never real acceleration. Working native MTP needs the **modelopt** main-model format (AEON, ~3.3/3 accept). LESSON: don't expect a faster single-stream seat from an llm-compressor NVFP4 quant of this arch; the MTP multiplier is the whole point and it requires modelopt. _Archived 2026-08-15._ - `[2026-07-14]` **NVFP4 spike: built the full MTP serve scaffolding BEFORE validating a plain NVFP4 serve was coherent.** Chased 6 sequential serve-config fixes (entrypoint doubled `serve`, arch `ForCausalLM`→`ConditionalGeneration`, `--language-model-only`, mamba-cache/`max-num-seqs`) across a **2.5hr GPU window** (quoted 30-60 min) — only to find the served model gibbers (`!!!!`). LESSON: smoke a PLAIN `/v1/completions` coherence check on the SIMPLEST config (native arch, no MTP, no splice) FIRST — validate the tracer bullet before building spec-decode scaffolding. Also cost an unnecessary re-quant (the `re:mtp.*` ignore fix that turned out moot). Diagnostic ladder in Current state. _Archived 2026-08-15._ - `[2026-07-14]` **MTP graft via top-level `mtp.*` tensor names does NOT survive `AutoModelForCausalLM.from_pretrained`** — the `Qwen3_5ForCausalLM` class doesn't expose an mtp module, so the mtp keys are DROPPED at load (quant output = 0 mtp). Fix = SPLICE the BF16 mtp tensors into the quant output post-hoc (how pantheon was built); don't rely on the graft surviving the model round-trip. _Archived 2026-08-15._ - `[2026-07-14]` **gitea "test-delivery 204" is NOT proof a webhook works** (204 = gitea *queuing*, not the listener receiving) — and a proxy test signing with the listener's OWN secret proves the listener, not gitea's real delivery. Both red herrings cost a round of the soong-lab webhook diagnosis. Diagnose from BOTH ends: sender (`docker logs gitea | grep webhook` → the `deny ''` line) AND an instrumented receiver. _Archived 2026-08-15._ - `[2026-07-13]` Relaying a peer's diagnosis as fact without confirming it against raw data. worldtree-dev diagnosed the WT #355 residual as "our llama.cpp seat wedging," which I echoed in a wrap-up; the operator challenged it and the seat logs DISPROVED it (seat completes ≤72s, idle at the wedge onset — the hang is the LiteLLM gateway). Lesson: CONFIRM peer diagnoses (esp. cross-domain ones) before acting/relaying — same discipline that caught the earlier char-rp-reasoning red-herring via a live `registry.resolve` reproduction. _Archived 2026-08-15._ - `[2026-07-13]` `althing-cli reply ` (thread id, not a MESSAGE id) → "unknown message_id"; and `reply` to your OWN message self-addresses to your handle ("replying to your own message"). Reply to a PEER's message id, or use `post --to `. Bit me several times this session. _Archived 2026-08-15._ - `[2026-07-09]` **`vllm/vllm-openai:latest` crashes on Ampere IMPORT** — Blackwell-only kernels (oink/aiter, `has_device_capability(100)`) die during import on the 3090/A6000. Pin **v0.23.0** on irv-ml1's Ampere GPUs. (`vllm/vllm-omni:v0.18.0` has a different entrypoint — don't use it either.) _Archived 2026-08-15._ - `[2026-07-09]` **Per-frame CPU SNAC decode is too slow for streaming** — per-call overhead × ~60 frames serialized → RTF 2.2 (WORSE than whole-clip's 1.0). Fix = **windowed chunk decode** (every 6 frames decode a [2 ctx | 6 | 2 ctx] window, emit the middle 6 → seamless, O(1)/frame, RTF ~0.97, TTFA ~0.8s). _Archived 2026-08-15._ - `[2026-07-08]` **Angel (allura-org/MS3.2-24b-Angel) self-quanted to NVFP4 = GARBAGE.** llm-compressor W4A4 NVFP4 (compressed-tensors, MLP-quantized, attn/vision bf16) of the Mistral3 dense 24B produces gibberish EVEN AT GREEDY (temp 0) → the quant itself is broken, not the tokenizer or sampler. Same recipe worked on the qwen models. Mistral3 + W4A4 NVFP4 via llm-compressor is bad. → for the RP seat, going **GGUF (llama.cpp)** to sidestep the whole NVFP4-quant surface. _Archived 2026-08-15._ - `[2026-07-08]` **Mistral3 + vLLM tokenizer/vision traps (serve `MS3.2-24b`, vLLM 0.24).** (a) HF `tokenizer.json` for Mistral = **GARBAGE output** — the card's "use the official Mistral tokenizer" warning is REAL; must use the `tekken.json`/mistral tokenizer. (b) BUT `--tokenizer-mode mistral` + vision **CRASHES** (`Failed to apply PixtralProcessor on {'text': '[IMG]'}`; and with tekken.json present in auto mode, `CachedMistralCommonBackend has no attribute is_fast`). So it's **mistral-tokenizer OR vision, not both** on this vLLM. Text-only + mistral tokenizer serves clean (`--limit-mm-per-prompt '{"image": 0}'`). **GGUF/llama.cpp avoids all of this** (native mistral tokenizer + vision). _Archived 2026-08-15._ - `[2026-07-08]` **Pantheon-Reasoning-27B refuses dark fiction DESPITE an abliterated base.** The base (`llmfan46 heretic`) writes freely (thinking-off), but Gryphe distilled the reasoning traces from **DeepSeek 3.2** (safety-aligned) onto every turn (`preserve_thinking:true`) → the model reasons ITSELF into refusals in the `` phase (collapses to empty output). Fix: thinking-off OR an uncensor system prompt (both verified). **Lesson: a reasoning finetune of an abliterated base can re-censor via its reasoning-trace TEACHER; the raw abliterated base is cleaner** — this is WHY the pivot went to the llmfan46 heretic base for gen. _Archived 2026-08-15._ - `[2026-07-08]` **Pantheon-27B MTP on vLLM compressed-tensors = 0% acceptance.** MTP is a separate **bf16** head (`mtp.*`, in `model-auxiliary.safetensors`, 15 tensors); AEON preserved it by INJECTING the bf16 head into the quant output (NOT re-quantizing — confirmed AEON's nvfp4 mtp is bf16). Built pantheon-27b-mtp = compressed-tensors main + injected bf16 mtp + `text_config.mtp_num_hidden_layers=1` → vLLM detected the MTP but SKIPPED the bf16 self_attn weights → 0/192 draft tokens accepted. **The bf16 MTP head only loads on the MODELOPT main-model format (like AEON), not compressed-tensors.** (Moot — operator dropped MTP for gen; not needed for the non-reasoning RP.) _Archived 2026-08-15._ - `[2026-07-07]` **vLLM 0.24.0 qwen3_5 LoRA application = silent no-op (#47639).** Adapter loads HTTP 200 but zero deltas at inference. NOT quant (NVFP4 AND FP8 both inert). NOT adapter format (separate `zc` adapter — correct per vLLM's `check_unexpected_modules` allowlist — loads clean but inert; the fused-key rekey is rejected). The #47640 None-group guard-patch overlay did NOT fix it (failure is UPSTREAM of `expand_packed_lora` — the separate→fused mapping never happens). Fix PR #47640 is OPEN (unmerged) so no version-bump helps. Merge bakes deltas in (bypasses this) but is static. _Archived 2026-08-15._ - `[2026-07-07]` **SGLang generic image can't LOAD our NVFP4 AEON** — ModelOptModelLoader weight-shape/ packing mismatch ([1024,5120] vs [1024,2560], 2-fp4/byte). NVFP4-on-SGLang needs the dedicated `qwen36-27b-nvfp4` dev image or a requant to SGLang's format. bf16 loads fine (arch supported; crash was quant-loader-specific). _Archived 2026-08-15._ - `[2026-07-07]` **SGLang `--lora-target-modules` CLI enum REJECTS the GDN names its own resolver asks for** (invalid choice: 'in_proj_qkv'); `'all'` resolves to the FUSED set (qkv_proj/in_proj_qkvz). SGLang wants its OWN packed layout (base r16 + `get_stacked_multiply=3`, NOT a pre-fused rank-48 qkv → the [48]-vs-[144] shape assert). A THIRD adapter format; version-exact source needed (`:latest`=0.5.13, NOT `main`). _Archived 2026-08-15._ - `[2026-07-07]` **Engine invocation footguns cost several wasted serve-bounces this session** — `docker run --rm` ate crash logs; duplicated `serve` (vLLM image entrypoint is already `["vllm","serve"]`); `--max-lora-rank 48` invalid (choices 1/8/16/32/64… → use 64); parens in `echo` inside `ssh host -c "…"` break the remote shell. LESSON: verify engine launch flags (`--help`, GPU-free) + never `--rm` a container whose crash logs you need, BEFORE bouncing a production serve. _Archived 2026-08-15._ - `[2026-07-04]` **LiteLLM (this gateway version) mutates the SHARED deployment config in-place on per-request sampler-param merge** → my deliberately-invalid `top_k=-5` forwarding-probe bled into a param-less character-rp request (vLLM 400, ONE-OFF, self-cleared by a later valid probe). NOT caching (none configured), NOT a config change. **Never fire invalid/distinctive sampler values at a SHARED gateway alias with live consumers** — use a throwaway alias, or a `docker restart litellm` flushes residual carryover. `feedback_litellm_shared_param_mutation`. _Archived 2026-08-15._ ## Recent decisions (archived 2026-08-16 batch) - `[2026-07-15]` **arbo fully switched off image-judge (qwen-image-bench) -> gen; image-bench pending eviction post-bake.** Operator-directed full switch (comfy-dev executed, live in prod). Established: gen (`qwen3.6-35b-a3b-heretic`) is vision-enabled and was image-bench's predecessor as arbo's hero-judge; image-judge actually serves 4 roles (vision quality-scoring + identity-scoring + bbox grounding + an uncensored text tier), not just grounding. comfy-dev spot-check: gen faster on every task, grounding within ~3px, uncensoring preserved, and it FIXED a bug (image-judge's reasoning preamble broke json_object + stalled the router). Sequencing = short prod bake then evict (~30 GB GPU1 reclaim); revert = flip `ARBO_VISION_MODEL`. Full record: auto-memory `project_arbo_gen_switch_imagebench_evict`. _Archived 2026-08-16._ - `[2026-07-18]` **soong-lab auto-redeploy — DONE + VALIDATED** (was approved/queued; executed same day on fresh context — see AS-BUILT at the bottom). Vuong approved wiring auto-redeploy for soong-lab (relayed via soong-dev, thread `01KXT3A6C3908TA4V9THV3AMH7`): new images should go live on corviduo-dev without the manual `docker compose pull && up -d`. Host-side implementation is infra-ops's lane; mechanism is infra-ops's call per fleet conventions. Operator deferred execution — "we'll do soong on fresh context." **Chosen mechanism (recommended, agrees with soong-dev): Worldtree-style CI-deploy step** — NOT watchtower polling. - Add a deploy job/step to soong-lab's `.gitea/workflows/build-and-push.yml` that, after the build+push job succeeds, **SSHes from the pfi-fleet runner to corviduo-dev** and runs `cd /home/infra-ops/soong-lab-deploy && docker compose pull && docker compose up -d`, then a **health-gate** (`curl -fsS http://localhost:8443/api/version`). - This is exactly how WT deploys the demo instance to the SAME host: see `~/development/Worldtree/.gitea/workflows/deploy.yml` — the "Deploy to demo VM + health-gate" step uses `secrets.DEMO_VM_SSH_KEY` / `DEMO_VM_HOST` / `DEMO_VM_USER`. Explicit-over-implicit (visible in the run log, fires exactly on build success), one less always-on service than watchtower. **Constraints (from soong-dev):** deploy on CI success only; keep the trigger gated to `v*` tags + `workflow_dispatch` (as today); preserve the one-command rollback posture (`docker compose down` / pin a previous tag). **BLOCKER — needs from vh (owner-only):** a **runner→corviduo-dev deploy SSH key** as a repo secret (+ host/user), same class as WT's `DEMO_VM_SSH_KEY`. Likely **reuse WT's existing demo-deploy key** (WT's runner already SSHes to 10.250.50.152 as its deploy user). Repo secrets are vh-owner-only (see [[2026-07-18-fleet-gitea-runner-build-recipe]]). **Next-session steps:** (1) confirm/obtain the deploy SSH-key secret from vh (reuse WT's or mint fresh); (2) add the deploy job to build-and-push.yml (infra-ops has push on vh/soong-lab); (3) dispatch a build to verify it deploys + health-gates; (4) ping soong-dev so they sync DEPLOY.md's "open follow-up" note to the as-built mechanism. Auto-pull (watchtower) explicitly NOT chosen. See [[2026-07-18-soong-lab-containerize-cutover]]. ## AS-BUILT (2026-07-18, same-day execution) **Mechanism landed** exactly as planned: `build-and-push.yml` gained a `Deploy to corviduo-dev + health-gate` step (after build+push) that SSHes the host as `deploy` and runs `docker compose pull && up -d` from `/opt/soong-lab`, then polls `http://localhost:8443/api/version` for 120s and fails the job loud if unhealthy. No compose is shipped from CI (the in-repo `docker-compose.yml` is a BUILD compose; the host pull-compose is infra-ops-managed). Kept the `v*`-tag/`workflow_dispatch` trigger. Skipped WT's disk-watermark gate + health-gated-`:latest`-advance (low cadence, easy rollback). **Deploy identity = reuse WT's `deploy` account** (operator accepted the rec): - `deploy` (uid 1001, docker-group → no sudo) already owns `/opt/worldtree`; relocated soong-lab's deploy dir `/home/infra-ops/soong-lab-deploy` → **`/opt/soong-lab`** (deploy-owned), copied compose + `.env`. Named volumes (`soong-lab_soong-library`, `soong-lab_soong-portraits`) are project-scoped by compose `name: soong-lab` → followed the move untouched (dry-run `up -d` ADOPTED the running container, no recreate). Old dir **retired → `.retired-20260718`** (recoverable). Also lingering: `soong-lab-deploy.sh` / `.log` (dead pre-container webhook artifacts) — harmless, left in place. - **Dedicated soong-only ed25519 deploy key** minted (NOT literally WT's key — cleaner independent revocation), pubkey appended to `deploy`'s `authorized_keys` (fp `SHA256:MG7M3RiZJ176sLfblffb96V6W1qkRTgJ5dow1CpiY68`). Existing `deploy` key is plain/unrestricted, so parity held. **The secret gate (the friction point):** repo Actions secrets are **vh-owner-only** — claude-bot's token is `write:package,read:repository` (403 on secret-write), and the vh package-scoped PAT also 403'd on `PUT …/actions/secrets/…`. So `DEPLOY_SSH_KEY` / `DEPLOY_HOST` (10.250.50.152) / `DEPLOY_USER` (deploy) HAD to be set by the operator. First operator attempt produced a **bad key paste** — the deploy step died with `Load key … error in libcrypto` + `Permission denied (publickey)` (build+push were green; live Soong never moved). Fix: operator re-set the secret; the minted key path was pre-validated from nh3-dev (`ssh -i … deploy@… 'cd /opt/soong-lab && docker compose config -q'` → OK, health 200) so the re-set was the only variable. **Validation:** `workflow_dispatch` via claude-bot **basic auth** (its token lacks `write:repository` for the dispatch API; the account password works). Run #5 (task 1886) GREEN — live container recreated `sha256:…541f7730` → `…07526a08`, `StartedAt` fresh, health 200. `/api/version` now reports **0.3.25** (run #5 shipped soong-dev's 1c2f831 STYLE_WORKFLOWS re-pin as validation cargo). soong-dev synced `docs/DEPLOY.md` (commit `00b67c3`). NB: tag **v0.3.25 exists only locally** — pushing it would re-trigger a redundant build+deploy of the same commit (operator's discretion). **Ops now:** redeploy = tag `v*` or `workflow_dispatch` the CI (auto). Manual fallback = `sudo -u deploy bash -c 'cd /opt/soong-lab && docker compose pull && docker compose up -d'` (the `.env` is `deploy`-owned 600, so infra-ops needs `sudo -u deploy`, not a bare `cd`). _Archived 2026-08-16._ - `[2026-07-18]` **soong-lab auto-redeploy — DONE + VALIDATED** (was approved/queued; executed same day on fresh context — see AS-BUILT at the bottom). Vuong approved wiring auto-redeploy for soong-lab (relayed via soong-dev, thread `01KXT3A6C3908TA4V9THV3AMH7`): new images should go live on corviduo-dev without the manual `docker compose pull && up -d`. Host-side implementation is infra-ops's lane; mechanism is infra-ops's call per fleet conventions. Operator deferred execution — "we'll do soong on fresh context." **Chosen mechanism (recommended, agrees with soong-dev): Worldtree-style CI-deploy step** — NOT watchtower polling. - Add a deploy job/step to soong-lab's `.gitea/workflows/build-and-push.yml` that, after the build+push job succeeds, **SSHes from the pfi-fleet runner to corviduo-dev** and runs `cd /home/infra-ops/soong-lab-deploy && docker compose pull && docker compose up -d`, then a **health-gate** (`curl -fsS http://localhost:8443/api/version`). - This is exactly how WT deploys the demo instance to the SAME host: see `~/development/Worldtree/.gitea/workflows/deploy.yml` — the "Deploy to demo VM + health-gate" step uses `secrets.DEMO_VM_SSH_KEY` / `DEMO_VM_HOST` / `DEMO_VM_USER`. Explicit-over-implicit (visible in the run log, fires exactly on build success), one less always-on service than watchtower. **Constraints (from soong-dev):** deploy on CI success only; keep the trigger gated to `v*` tags + `workflow_dispatch` (as today); preserve the one-command rollback posture (`docker compose down` / pin a previous tag). **BLOCKER — needs from vh (owner-only):** a **runner→corviduo-dev deploy SSH key** as a repo secret (+ host/user), same class as WT's `DEMO_VM_SSH_KEY`. Likely **reuse WT's existing demo-deploy key** (WT's runner already SSHes to 10.250.50.152 as its deploy user). Repo secrets are vh-owner-only (see [[2026-07-18-fleet-gitea-runner-build-recipe]]). **Next-session steps:** (1) confirm/obtain the deploy SSH-key secret from vh (reuse WT's or mint fresh); (2) add the deploy job to build-and-push.yml (infra-ops has push on vh/soong-lab); (3) dispatch a build to verify it deploys + health-gates; (4) ping soong-dev so they sync DEPLOY.md's "open follow-up" note to the as-built mechanism. Auto-pull (watchtower) explicitly NOT chosen. See [[2026-07-18-soong-lab-containerize-cutover]]. ## AS-BUILT (2026-07-18, same-day execution) **Mechanism landed** exactly as planned: `build-and-push.yml` gained a `Deploy to corviduo-dev + health-gate` step (after build+push) that SSHes the host as `deploy` and runs `docker compose pull && up -d` from `/opt/soong-lab`, then polls `http://localhost:8443/api/version` for 120s and fails the job loud if unhealthy. No compose is shipped from CI (the in-repo `docker-compose.yml` is a BUILD compose; the host pull-compose is infra-ops-managed). Kept the `v*`-tag/`workflow_dispatch` trigger. Skipped WT's disk-watermark gate + health-gated-`:latest`-advance (low cadence, easy rollback). **Deploy identity = reuse WT's `deploy` account** (operator accepted the rec): - `deploy` (uid 1001, docker-group → no sudo) already owns `/opt/worldtree`; relocated soong-lab's deploy dir `/home/infra-ops/soong-lab-deploy` → **`/opt/soong-lab`** (deploy-owned), copied compose + `.env`. Named volumes (`soong-lab_soong-library`, `soong-lab_soong-portraits`) are project-scoped by compose `name: soong-lab` → followed the move untouched (dry-run `up -d` ADOPTED the running container, no recreate). Old dir **retired → `.retired-20260718`** (recoverable). Also lingering: `soong-lab-deploy.sh` / `.log` (dead pre-container webhook artifacts) — harmless, left in place. - **Dedicated soong-only ed25519 deploy key** minted (NOT literally WT's key — cleaner independent revocation), pubkey appended to `deploy`'s `authorized_keys` (fp `SHA256:MG7M3RiZJ176sLfblffb96V6W1qkRTgJ5dow1CpiY68`). Existing `deploy` key is plain/unrestricted, so parity held. **The secret gate (the friction point):** repo Actions secrets are **vh-owner-only** — claude-bot's token is `write:package,read:repository` (403 on secret-write), and the vh package-scoped PAT also 403'd on `PUT …/actions/secrets/…`. So `DEPLOY_SSH_KEY` / `DEPLOY_HOST` (10.250.50.152) / `DEPLOY_USER` (deploy) HAD to be set by the operator. First operator attempt produced a **bad key paste** — the deploy step died with `Load key … error in libcrypto` + `Permission denied (publickey)` (build+push were green; live Soong never moved). Fix: operator re-set the secret; the minted key path was pre-validated from nh3-dev (`ssh -i … deploy@… 'cd /opt/soong-lab && docker compose config -q'` → OK, health 200) so the re-set was the only variable. **Validation:** `workflow_dispatch` via claude-bot **basic auth** (its token lacks `write:repository` for the dispatch API; the account password works). Run #5 (task 1886) GREEN — live container recreated `sha256:…541f7730` → `…07526a08`, `StartedAt` fresh, health 200. `/api/version` now reports **0.3.25** (run #5 shipped soong-dev's 1c2f831 STYLE_WORKFLOWS re-pin as validation cargo). soong-dev synced `docs/DEPLOY.md` (commit `00b67c3`). NB: tag **v0.3.25 exists only locally** — pushing it would re-trigger a redundant build+deploy of the same commit (operator's discretion). **Ops now:** redeploy = tag `v*` or `workflow_dispatch` the CI (auto). Manual fallback = `sudo -u deploy bash -c 'cd /opt/soong-lab && docker compose pull && docker compose up -d'` (the `.env` is `deploy`-owned 600, so infra-ops needs `sudo -u deploy`, not a bare `cd`). _Archived 2026-08-16._ - `[2026-07-19]` **irv-ml1 ComfyUI — RTX VSR baked into canonical provisioning (comfy-dev ticket DONE).** RTXVideoSuperResolution node + `nvidia-vfx` dep were manual installs; documented both in the canonical `stacks/comfyui/README.md` runbook (this stack's provisioning IS the README — no automated provision script). Key durability insight: the **node** lives in `basedir/custom_nodes` (persistent, restic-included → durable) but the **`nvidia-vfx` wheel** lives in the venv under `run/` (disposable, restic-excluded → **dropped by any `rm -rf run/*` fresh-bootstrap**), so the pip step must re-run after every venv rebuild. Both steps run **as uid 1000** (root install → venv-ownership crash-loop, [[reference_irv_ml1_comfyui_mmartial]]); `--extra-index-url https://pypi.nvidia.com` kept **scoped to the nvidia-vfx install**, deliberately NOT a global compose `PIP_EXTRA_INDEX_URL` (would risk perturbing the pinned torch 2.12.1/SageAttention boot bootstrap). Node already live on the box; no host change, canonical runbook now replays it. comfy-dev informed. _Archived 2026-08-16._ - `[2026-07-19]` **vh private Gitea PyPI — consumer READ-access convention set + wyrd-dev provisioned.** Consuming agents read the internal vh PyPI (`https://gitea.phasefinal.com/api/packages/vh/pypi/simple/`) with a **shared read-only token** (operator call: shared, not per-consumer — read-only blast radius is small, per-agent Gitea identities aren't worth it). Minted a dedicated `read:package`-scoped PAT off **claude-bot** (`POST /users/claude-bot/tokens`, name `vh-pypi-read-consumers`; verified reads worldtree-sdk, write-probe 401), revocable/rotatable independently. uv auth = `UV_INDEX_GITEA_USERNAME=claude-bot` + `UV_INDEX_GITEA_PASSWORD=` (or `~/.netrc`); pyproject uses `[[tool.uv.index]] name=gitea … explicit=true` + `[tool.uv.sources] = { index = "gitea" }` (mirrors soong-lab's bifrost setup). Delivered to wyrd-dev (worldtree-sdk adoption) via mode-600 drop on nh3-dev, drop-and-shred. [[reference_claude_bot_gitea_creds]] _Archived 2026-08-16._ - `[2026-07-23]` **jackdaw-compose backend deployed as a persistent nh3-dev service (:8787).** Hosted for jackdaw-dev: thin stateless `bun server/index.ts` (from `~/development/jackdaw`) → LiteLLM `gen`, Origin-gated (INV-BK04/05), reached same-origin via their `:4500` bench's `/compose` proxy. `jackdaw-compose.service` (env/shared-key server-side, unit 0600, uncommitted). Also stood up + tore down a throwaway cloudflare quick-tunnel for their preview (`cloudflared` now installed at `~/bin`). In the nh3-dev README inventory (`cd4d52e`). _Archived 2026-08-16._ - `[2026-07-25]` **nh3-extdev herald installed — box is now a full v2 push participant.** forseti flagged (relaying operator): extdev had the `althing-herald` binary (`/usr/local/bin/`) but NO unit (skipped the whole v2 arc), so `herald-status` = "notifications suspended" and ldp-dev ran on the `althing-light-monitor` poll fallback. Installed `/etc/systemd/system/althing-herald.service` as a **SYSTEM unit mirroring the receiver** (`User=althing-svc`, `Group=althing`, `Environment=ALTHING_ROOT=/srv/althing`, `ExecStart=/usr/local/bin/althing-herald --poll 5`, enabled) via the **lkraven@ NOPASSWD path** (used under the then-mistaken belief infra-ops was sudo-less — **CORRECTION 2026-08-03: infra-ops has had full NOPASSWD sudo on extdev since 2026-06-25** per [[reference_nh3_extdev_althing_mesh]]; future extdev installs can self-serve as infra-ops without the lkraven@ hop). Verified: active / 0 restarts / `herald-status` flipped to "✓ herald up." No zellij routes on extdev → heartbeat + wake-FIFO poke only, no pane-dispatch; ldp-dev keeps light-monitor unless it opts into a wake-listener. _Archived 2026-08-16._ - `[2026-07-25]` **Booth v0.1.4 — booths are downloadable.** Verbatim `index.html` booths (e.g. edict-design-brief) were served raw with no download affordance. Added `/b//?download=1` (streams the whole booth as `.zip`, attachment) + `?dl=1` on the file route (forces Content-Disposition attachment so html/md/text saves instead of rendering inline) + ⬇ zip links on the index card (the accessible spot for verbatim booths) and the gallery header. `zip_booth()` helper, 31 tests green; verified live on nh3-dev :8090 (edict-design-brief.zip = index.html + ui-design-brief.md). eshpfi `91a031f` / tag `booth-v0.1.4`. _Archived 2026-08-16._ - `[2026-07-25]` **Kimi K3 wired into the LiteLLM gateway — CODING endpoint** (operator-directed; fulfills a Heid gateway request to add a 4th cross-frontier panel arm). **Primary `model_name: kimi-k3` → `openai/k3` @ `https://api.kimi.com/coding/v1`** (Kimi Code / Vivace membership; key `KIMI_CODE_API_KEY`). A general-endpoint variant `kimi-k3-gen-api` → `openai/kimi-k3` @ `https://api.moonshot.ai/v1` (key `MOONSHOT_API_KEY`) is kept alongside (originally wired then demoted when the operator corrected: the plan uses the CODING endpoint, not the general Moonshot API). Both keys in compose env + server `.env` (NOT committed) + `.env.example`. Both verified live through the gateway :4000 (17+25→"42", "PONG"). **k3 constraints on BOTH endpoints (config-pinned + commented):** accepts ONLY `temperature=1` (else 400 "only 1 is allowed"); REASONING model (CoT in `reasoning_content`, answer in `content` → tiny `max_tokens` returns EMPTY; Kimi Code adds thinking-effort tiers low/high/max). Coding lineup also carries `k3-256k` / `kimi-for-coding` / `kimi-for-coding-highspeed` (not wired). Reachable by any gateway key spanning all proxy models (incl. shared all-agents key → spends the paid Vivace/Moonshot quota). eshpfi `edaa9a9` (gen wiring) + `9e2f787` (coding correction). **OPEN:** Heid key-scoping — shared key reaches it (paid) vs a dedicated scoped key (asked in althing `01KYD63ZBY…`). _Archived 2026-08-16._ `[2026-07-25]` **infra-ops Worldtree config-as-code repo — SHIPPED + boundary AGREED.** **STATUS (2026-07-25, done this session):** `vh/worldtree-instance-configs` (private, gitea) built, pushed, validated; boundary agreement secured from worldtree-dev. - **Repo:** dir-per-instance `demo/` + `personal/` (5 files each: `defaults.yaml`, `policies.yaml`, `model_roles.yaml`, `providers.yaml`, `matrix.yaml`), seeded byte-exact from live `/opt//config`. `pinned/` = README stub only — **no `/app/config` bind-mount; config baked into frozen image `446e5807` (2026-05-13)**, so out-of-scope; deploy verb refuses it. - **Tool:** `scripts/deploy-wt-config ` — `diff` (read-only repo-vs-host), `deploy` (in-run host backup → `install -o vh -g vh -m 644` → restart **api+matrix** → health-gate api `/health` → auto-rollback), `capture` (host→repo reconcile). Instance table in-script (demo→`/opt/worldtree/config`+`worldtree-worldtree-{api,matrix}-1`; personal→`/opt/worldtree-personal/config`+`worldtree-personal-worldtree-{api,matrix}-1`). Matrix sidecar shares the config mount but has no healthcheck → restart both, gate on api. Env `WT_CONFIG_HOST` (default `infra-ops@10.250.50.152`), `WT_HEALTH_WAIT` (90s). Local clone `~/development/worldtree-instance-configs`. - **Gitea plumbing (reusable):** nh3-dev **403s the gitea HTTP API** (public fail2ban + internal `:3000` both 403). Repo CREATE went via **ana-docker localhost API** (`ssh infra-ops@10.250.50.70` → `curl localhost:3000/api/v1/user/repos`, vh token from `~/.config/tea/config.yml`, operator-authorized one-time). PUSH went over **internal git-SSH `ssh://git@10.250.50.70:222`** (works from nh3-dev; auths as vh). `git init` defaulted to `master` → renamed `main` to match repo default_branch. - **Boundary AGREED (worldtree-dev, althing thread `01KYCAECRWVEF16EVKQAGT2N80`):** no hand-edits to `/opt//config`; config changes route to infra-ops as deltas (worldtree-dev owns CONTENT + approval trail — the wyrd-grant shape — infra-ops lands+deploys). **Three-layer model:** image `config/` = baseline new instances seed from (theirs) → `vh/worldtree-instance-configs` = per-instance truth (ours) → host bind-mount = deploy target (written only by the tool). **Carve-out:** worldtree-dev's admin-API ops (`/admin/keys` mint, tier changes, session retirement, future runtime-grant surfaces) mutate instance **DATABASES not config files** → NOT config edits, stay in-band. If a future API writes config *files*, they flag at design time. b132 CONFIG BASELINE breadcrumb composes (INFO line = config-as-code diverges from image baseline, by design). - **No live deploy** done or needed — repo seeded == live (diff clean, capture round-trips zero-diff). Deploy path is dry-run-validated only; first real deploy needs operator per-change yes (managed box). --- _Original plan (2026-07-25, pre-build):_ `[2026-07-25]` **infra-ops to OWN a Worldtree per-deployment config repo + deploy tooling (operator-directed).** **Decision.** Vuong directed (2026-07-25, this session) that Worldtree instance config should be a *tracked change*, **managed and deployed by infra-ops — not worldtree-dev**. Model: worldtree-dev owns the app/image (+ the baked baseline defaults); **infra-ops owns config-as-code for every deployment** and deploys it. This is the durable fix for the root cause behind the whole #376 arc — config was edited live on host bind-mounts (`/opt//config/`) with zero version history, audit, or recovery. **What "no worldtree-dev involvement" does and does NOT cover** (clarified with the operator this session): - **Build + deploy = infra-ops-only.** Deploying config = write the host bind-mount file + restart the container (the *exact* procedure already run this session — backup → replace → restart → health-gate → rollback-on-unhealthy). No worldtree-dev in the deploy loop. Their CI only swaps the IMAGE; it does NOT resync the host config bind-mount (confirmed #376 finding). - **ONE load-bearing exception — a one-time boundary agreement, NOT per-deploy involvement:** for the repo to *own* config it must be the **only writer**. worldtree-dev "live-bridges" (hand-edits mounted config directly on the box). If the repo deploys config *and* they keep live-editing → **two writers fighting the same files** = #376 all over again. So secure a one-time "yes" from worldtree-dev: *the config repo is now authoritative; stop hand-editing `/opt//config`; route config changes through the repo.* (Five-minute agreement, not a design collab.) - **Standing coupling (not "involvement"):** the config *schema* is the app's, enforced by its boot validator (`core.config_validator`). infra-ops configs must stay schema-compatible with the deployed image; the boot gate is the loud backstop. **Build shape (recommended):** - Gitea repo `worldtree-instance-configs` (infra-ops-owned), **dir per instance** (`demo/`, `personal/`, `pinned/` — the three on corviduo-dev 10.250.50.152: demo `worldtree-worldtree-api-1` :8080, personal `worldtree-personal-worldtree-api-1` :8081, pinned `worldtree-pinned-worldtree-api-1` :8082). Config dirs: demo `/opt/worldtree/config`, personal `/opt/worldtree-personal/config`, pinned `/opt/worldtree-pinned/config` (verify pinned's mount). - **SEED FROM CURRENT MOUNTED STATE, don't author fresh** — capture each instance's live config (incl. legitimate live-bridged deltas: personal carries `agent_architect` role [Soong/soong-lab] in model_roles.yaml + `ratatoskr-affect-full-allow` in policies.yaml that are NOT in the app repo — the operator ruled these are BY DESIGN, keep them). Losing them = breakage (the affect-render one gates mood rendering). - Deploy script (e.g. `scripts/deploy-wt-config `): git = source of truth → push to host bind-mount + `docker restart` (same pinned image, no pull) + health-gate + auto-rollback. This is the proven-this-session procedure, scripted. - Files per instance: `policies.yaml`, `model_roles.yaml` (+ whatever else is bind-mounted — `defaults.yaml`, `providers.yaml`, `matrix.yaml` all live in `/opt//config`; decide scope — policies+model_roles are the authz/role layer, defaults/providers are heavier instance tunables). **Tracking surface:** operator-directed 2026-07-25, carried by this snapshot + `/tmp/infra-ops-handoff.md`. No issue filed (infra-ops-internal build). Related fleet idiom to reuse: canonical-sync (`.corviduo-canonicals.toml` / `canonical_sync.py`). Later scale option (deferred, needs worldtree-dev): base+overlay with a merge step in their pipeline. See [[2026-07-25-wt-376-per-instance-config-arc]] for the incident that produced this. Auto-memory: `reference_worldtree_perinstance_config`, `reference_corviduo_dev_emergency_ops`. _Archived 2026-08-16._ - `[2026-07-26]` **Demo `BIFROST_CLIENT_ALLOWED_HOSTS` += `10.100.10.50:8391`** (wyrd-dev's bifrost memory-store provider; operator-approved). **First live exercise of the #376 config-as-code boundary working as designed** — worldtree-dev routed the delta to infra-ops instead of hand-editing `/opt/demo`. Appended to `/opt/worldtree/.env:25` (now 4 netlocs), recreated ONLY `worldtree-api` (the gated conv-api path), health-gate green, container env verified. **REUSABLE FOOT-GUN:** an env-var change needs a container **RECREATE, not `docker restart`** (env is baked at create); and the demo `.env` defaults `WORLDTREE_IMAGE=:latest` while the box runs a specific SHA — so a naive `compose up` risks the documented stale-`:latest` crash. FIX = capture the running image live (`docker inspect …Config.Image` → `…:9eff09f007ba`) and `sudo env WORLDTREE_IMAGE= docker compose up -d worldtree-api`. Backup `/opt/worldtree/.env.bak-bifrost-20260726-221602`. **BOUNDARY SEAM:** this was a compose-`.env` var, NOT a `config.yaml` file in `vh/worldtree-instance-configs` — the `.env` holds secrets so it's deliberately not repo-tracked → env-deltas land directly on the box (config *files* are versioned, compose *env vars* aren't). [[reference_worldtree_instance_configs_repo]] _Archived 2026-08-16._ - `[2026-07-27]` **Zed edit-predictions: keyless FIM-completion route SHIPPED end-to-end.** Operator wants Zed's inline edit-prediction (which CANNOT send an auth header) to reach a FIM coder via `/v1/completions`. **Deep-research (106-agent workflow) picked `Qwen/Qwen2.5-Coder-1.5B`** (BASE, Apache-2.0; native FIM `<|fim_prefix|>/<|fim_suffix|>/<|fim_middle|>` IDs 151659/60/61; Zed `prompt_format:"qwen"`). Runner-up 3B = non-commercial Qwen-Research license; **no small dense Qwen3-Coder exists (all MoE, smallest 30B)**. **Stood up `vllm-coder`** on ana-ml2 **GPU1 :8020** (served-name `qwen2.5-coder-1.5b`, 8192 ctx, util 0.06, fp8 KV). To fit, **shrank granite (phasing out, operator-directed):** util 0.27→0.13, max-len 131072→16384, seqs 1024→256 (freed ~14 GB; the KV-≥-1×-max-len rule crash-looped it at util 0.12/32768 → settled 0.13/16384). **LiteLLM alias `coder-fast`** → :8020 (`mode: completion`). **Minted a `coder-fast`-SCOPED virtual key** (verified 403 on `gen` — the real blast-radius bound). **Built `zed-fim-proxy`** (ana-docker **:4141**, `network_mode: host`, stdlib-python, `stacks/zed-fim-proxy`): keyless POST `/v1/completions`, model-allowlist `coder-fast`, injects the scoped key → LiteLLM :4000; `GET /ping` anon liveness; wrong-model→403, wrong-path→404, `/chat/completions` rejected. Verified keyless FIM end-to-end ('a + b', finish `stop`). **Zed `api_url` = `http://10.250.50.70:4141/v1`, model `coder-fast`, prompt_format `qwen`.** **source-IP allowlist intentionally LEFT OFF (operator direction 2026-07-27) — do NOT tighten:** Zed roams the operator's WireGuard `10.0.0.0/8`, so a single-IP pin would break it. Blast-radius bound is the `coder-fast`-scoped key + model/path allowlist (keyless but coder-fast-only, internal-net-only). (The proxy does exact-IP matching; scoping to the `10.0.0.0/8` CIDR would need CIDR support — deliberately not added.) Canonical: `stacks/vllm` (coder + granite shrink), `stacks/litellm` (coder-fast), `stacks/zed-fim-proxy` (NEW). Server vllm compose.yaml has benign stale-comment drift vs canonical (didn't overwrite the newer canonical). _Archived 2026-08-16._ - `[2026-07-27]` **Muninn ingestion-watcher sidecar deployed on PERSONAL Worldtree (#377).** worldtree-dev request (research-wing ingest arc, personal-only per the 2026-07-16 topology ruling); operator-approved. Added a `worldtree-muninn` **compose sidecar** to `/opt/worldtree-personal/compose.yaml` — `<<: *worldtree-common` anchor inherits the api's image + full env + config/state/kb mounts; `command: python -m core.muninn --watch`; `restart: unless-stopped`; `stop_grace_period: 1h` (INV-377-7: max 2 concurrent × worst-case job, SIGTERM-drains). **Pinned to the running SHA `773866084af9`** (b146, ≥ b143 — dodges both the `:latest` trap AND the "pre-b143 ref resurrects deleted dispatch.py from stale bytecode" warning). Verified: running / 0 restarts / flock sole-runner (no rc3) / heartbeat live at `{ingestion_root=/data/state/ingestion}/.watcher-heartbeat` (poll 30s). Container `worldtree-personal-worldtree-muninn-1`; backup `compose.yaml.bak-muninn-20260727-081920`. **DURABILITY RESOLVED (worldtree-dev, same day):** Q1 was a LIVE FOOTGUN — `deploy-personal.yml` scp's the REPO compose.yaml over the box's + runs `up -d --remove-orphans`, so the box-local sidecar would've been clobbered AND orphan-removed at the next staging tag. worldtree-dev fixed at source: moved the sidecar into their repo compose.yaml gated behind a **`muninn` compose profile** (commit 5d7f6bd) — shared compose stays instance-identical, `.env` `COMPOSE_PROFILES` differentiates (demo watcher-less). **My action:** added `COMPOSE_PROFILES=muninn` to `/opt/worldtree-personal/.env` (backup `.bak-muninn-profile-20260727-082541`; no-op vs the current unprofiled box-local sidecar → seamless handover at next deploy). Q2: their deploy `up -d`'s the whole stack w/ `WORLDTREE_IMAGE` exported → sidecar version-tracks the api, no drift. **CONFIG-AS-CODE EXTENSION:** mirrored the non-secret delta as `personal/env.public` in `vh/worldtree-instance-configs` (repo `a9d091e`) — FIRST extension beyond config.yaml files to env-level config; the secret-laden `.env` stays box-only, `env.public` records only non-secret infra-ops-owned env deltas (record, not a deploy source — `deploy-wt-config` globs `*.yaml`). **BOUNDARY CLARIFIED:** compose.yaml = worldtree-dev's (their repo, instance-identical, scp'd on deploy); per-instance `.env` = infra-ops's differentiator. Deploy step of the #363/#377 arc. **#377 CLOSED — acceptance PASSED 2026-07-27:** worldtree-dev enqueued a test job via muninn-dispatch 0.1.0 in a one-shot ephemeral container (no docker-exec); the sidecar claimed it within one 30s poll, drove it to terminal (structure→summarize→complete), zero restarts/rc3, heartbeat fresh throughout — whole loop (request→deploy→durability fix→acceptance) in <2h. (Pre-existing pipeline bug #379 surfaced — `output.kb_notes=false` ignored → 1 inert test note in the research wing — worldtree-dev owns it, nothing infra-ops-side.) **⚠ OPERATOR-SURFACE (open):** the `env.public` overlay mechanism is a repo-scope call to bless/adjust. [[reference_worldtree_deploys_cicd]] [[reference_worldtree_instance_configs_repo]] [[project_worldtree_research_wing_ingest]] _Archived 2026-08-16._ ## Tried and abandoned (archived 2026-08-16 batch) - `[2026-07-18]` **Fleet Gitea CI foot-guns** (3 failed soong-lab builds): the pfi-fleet runner's `node:20-slim` job image has no docker/git so `actions/checkout` + `docker/*` marketplace actions all fail; `vh` is a USER so its packages are owner-write-only (claude-bot repo-admin-collab still 401s on push/publish, and can't set repo secrets — owner-only); `GITEA_`-prefixed secret names are reserved/illegal. Fixes in → `persistent-memory.d/2026-07-18-fleet-gitea-runner-build-recipe.md` _Archived 2026-08-16._ - `[2026-07-18]` **zonos-gateway local clone had NO git remote + a history unrelated to gitea's** — "committed to vh/zonos-gateway" was never pushed from that clone; two separate `git init` lineages, no merge-base. Reconcile = reset local→origin/main + overlay the changed files + push (NOT force — that erases gitea's voice-wav commits). Check `git remote -v` + `git merge-base` before assuming a clone is wired. _Archived 2026-08-16._ - `[2026-07-25]` **Peer green-light ≠ operator consent for a managed-box mutation.** Auto-mode guard blocked a config-replace+restart on the Worldtree-team demo box that was authorized only by worldtree-dev's althing message — correctly: a persistent change to shared infra needs the *operator's* yes for that specific change, not a peer's. Surface it; don't route around the guard. (The operator then stood the whole change down — the guard's hold was the right call.) _Archived 2026-08-16._ ## Recent decisions (archived) **Worldtree b168/#384/#385 arc — COMPLETE 2026-08-03.** A long peer-driven arc across worldtree-dev / muninn-dev / mimir-dev / ratatoskr-dev, all on corviduo-dev's demo+personal instances. Sequence: providers.yaml boot-gate pre-sync → b168 deploy → DCC #384 reindex → round-2 full re-ingest → #381 restart → operator-approved production dedup sweep. Landed clean; three of MY foot-guns along the way, each caught + hardened into a fleet runbook rule (see Tried-and-abandoned: `mv -t`, `docker exec -u 1000`, shared-containerd race). ## providers.yaml pre-sync (boot-gating config) b168 (commit `293f8f3`) added a `summarization` capability block that in-image `agents/muninn/config.yaml` references → boot-blocking if the host bind-mounted providers.yaml lacks it. Synced both hunks (summarization block + deep-reasoning desc) into demo+personal via `deploy-wt-config`; instance-configs commit `53349f8`. - **deploy-wt-config runbook:** `~/development/worldtree-instance-configs/scripts/deploy-wt-config {diff|deploy|capture} --file providers.yaml` (per-instance dirs demo/personal/pinned; `deploy` = host write + api/matrix restart + 90s health-gate + auto-rollback; `diff`/`capture` safe). demo+personal providers.yaml are byte-identical. - **GOTCHAS:** (1) an UNPUSHED source commit → `git show ` 404s and a gitea `raw?ref=` silently falls back to the default branch; verify the commit exists (`/git/commits/`) before trusting a fetch, else ask the peer to paste hunks. (2) a peer's hunk paste may be mis-indented (8-space vs the block's 4-space) → invalid YAML; always YAML-validate after a paste-sourced edit. - **Config-delta pre-sync rule (verified via `docker inspect`):** worldtree containers bind-mount ONLY `config/` host-side (`/opt/worldtree-*/config/` → providers/model_roles/matrix/policies/defaults/env.public = the pre-syncable set); `agents/` (schemas.yaml, prompts) + all code ship IN-IMAGE. So only a `config/*.yaml` change is boot-blocking-pre-syncable; an `agents/`-or-code delta needs NO host pre-sync (CI carries it). b169's schemas.yaml (#387) was correctly no-pre-sync. ## #384 reindex + #381 restart + verify DCC job `mimir-6351554e8e8f`. Reindex: `sudo docker exec -u 1000 worldtree-personal-worldtree-muninn-1 python -m core.muninn --reindex ` (⚠️ MUST `-u 1000` — default-root writes contaminate the uid-1000 KB tree; see Tried-and-abandoned). Then **#381 restart** (stale-Chroma-client fix): `sudo docker restart worldtree-personal-worldtree-api-1` (plain bounce, NO compose up / no image repoint) → healthz/readyz 200 ~25s. - **Chroma-verify runbook:** `sudo docker exec -i python -` (MUST pass `-i` or stdin never reaches `python -`) → `chromadb.PersistentClient('/data/kb/.chroma').get_collection('fiction').get(where={'job_id':}, include=['metadatas'])`. Chroma persists at container `/data/kb/.chroma` = host volume `worldtree-personal_worldtree-kb`. - **Retrieval-visibility check (NOT grounding — that's ratatoskr's):** a Mimir session — admin token `~/.config/worldtree/personal-admin-token` (wildcard scope) → POST `/sessions` (agent_id=`mimir`, `record_tool_intermediates=true`) → POST `/sessions/{id}/messages` (STREAMS SSE, not JSON) → parse SSE `tool_result` for `search_library` wing hits → DELETE session. ## Production dedup sweep (operator-approved) Deleted the 785 April-era DCC orphan rows (`job_id=b59c147c5ce0`, no wing/source_identity metadata → predate identity tracking) from the `main` collection. Supervised protocol: read-only verify count == 785, back up all rows (ids+docs+embeddings) to `corviduo-dev:/tmp/main-sweep-backup-b59c147c5ce0.json` (reversible), `main.delete(where={job_id})` (assert target==785 first), verify `main` 4009→3224, then **bounce the api** (a separate-process delete leaves the api's in-memory HNSW index holding the vectors until reload — the #381 pattern generalizes to deletes), confirm search now fiction-only. Backup left for /tmp natural cleanup (fiction wing is canonical; `~/archives` has the historical record). Result: fiction wing 166 → 1,372 concepts; three consumer verify rounds 0/5 → 5/5 → saturated; #385 budget fix validated (705 vs April's 785 control, extraction AND indexing, zero truncations). worldtree-dev filed #388 for a deploy concurrency-lock (the shared-containerd race fix). See [[2026-08-02-mimir-inbox-arc]]. _Archived 2026-08-18._ `[2026-08-02]` **The mimir-inbox / #377-read-path arc — deploy, four bugs found+fixed+verified, a cloned voice, all in one long session (2026-08-01→02).** The browser-facing half of the #377 Muninn ingestion arc, end to end: mimir-inbox stood up, the write path proven, the read path chased through four defects to a verified-working state, and a character voice cloned into the TTS zoo. Peers: mimir-dev (the app), muninn-dev (gate/watcher spec), worldtree-dev (Worldtree app layer + the #380/#381/#382/#383 fixes), ratatoskr-dev (a consumer + the rigorous verifier). ## mimir-inbox deployed (#377) - **New infra-ops stack, canonical eshpfi `stacks/mimir-inbox/`; live corviduo-dev `10.250.50.152:8091`** (co-located w/ muninn-gate :8090 + the worldtree-personal muninn watcher). Full deploy detail + procedures → auto-memory `reference_mimir_inbox_deploy`. - **Placement decision (operator, reversed):** 7-31 he ruled mimir-inbox stays OFF corviduo-dev (shared/NFS mount); 8-01 he REVERSED to CO-LOCATE. Trigger: muninn-dev's code-check showed staging is NOT same-fs-constrained (gate reads staging metadata + passes path strings; `os.replace` is inside `ingestion_root`) — staging's real constraint is **path-identity across writer/gate/watcher**, which co-location buys outright while dodging NFS failure modes. I HELD the reversal for the operator's direct word (data/hosting on a team-managed box, reversing his own ruling) even against 3 peer relays — vindicated as the right instinct; muninn-dev agreed. - Build: **`uv sync --no-dev --frozen`, SINGLE-STAGE** (project installs editable-linked to `src/`, so src/ MUST stay beside .venv — a multi-stage "copy only .venv" dies at import/404s assets). uid 1000, host-net bind 10.250.50.152:8091, TCP-liveness healthcheck (deliberately NOT gate-coupled). Redeploy = refresh build context (**preserve the on-server `.env`!**) → `docker build -t mimir-inbox:0.0.1 -t mimir-inbox: .` → `compose up -d`. Version stays 0.0.1 across dev commits → tag the image w/ the source SHA too. Live commit progression `0478452`→`c8ab38f`→`2dcc77e`→**`8ece117`** (3 redeploys). - mimir-inbox key on the gate bumped [read,submit]→**[read,submit,control]** (cancel/retry); brokered via a 0600 drop on nh3-dev (never on the althing bus). ## The read-path bug chain (worldtree-dev's, all found via this arc) - **#380 wing-blind indexing:** the book-ingest path upserted concepts into a hardcoded `main` Chroma collection while wing search reads the `fiction` collection → P&P written to disk but `search_library` returned total 0. A silent-success defect ("complete/69 indexed" was right about the WRITE, wrong drawer). Root-caused off MY physical evidence (files on disk + search empty). Fixed b164 + a one-shot `--reindex ` (re-upsert into the right wing collection + delete stray `main` rows). - **#381 stale Chroma client:** the personal api opens its Chroma client before the watcher's cross-process writes → **a freshly-ingested/re-indexed book is NOT queryable until the api is restarted.** Proven by my restart-diagnostic (pre-restart total 0 → post-restart hits, same index). Workaround until fixed: `docker restart worldtree-personal-worldtree-api-1` after any ingest/re-index. Filed as #381. - **#382 unreliable Mimir grounding (the subtle one):** post-#380-fix the index was correct, but Mimir's grounding was INTERMITTENT — some sessions navigated the opaque job-hash dir (`mimir-f3887c9b97b7`) to the content, others distrusted the correct vector hits and **silently answered from training knowledge** (worst of the looks-fine-isn't family). ratatoskr-dev caught it; I'd been over-confident ("Mimir read Austen back to you") having verified the INDEX, not the GROUNDING. Fixed b166 with BOTH shapes: a self-describing `_index.md` per wing job-dir (resolves the hash dir to its title) + a Mimir prompt rule (wing-scoped hits ARE library content, never discard on a name mismatch, never substitute training). **Verified: ratatoskr-dev re-ran 3× fresh sessions → 3/3 grounded**, citations in note-extracted language not raw Austen. #382 CLOSED. - **DCC (Dungeon Crawler Carl, job `b59c147c5ce0`) backfill:** `--reindex` FAILED ("job not found in any state dir" — predates state-tracking). SETTLED = **no re-file** (the b166 prompt rule already grounds it even without an `_index.md`; ratatoskr confirmed incidentally); an `_index.md` rides whenever DCC is next re-ingested. - **#377 mimir-inbox banner bug (mimir-dev's, `8ece117`):** `/health-banner` misattributed an unwritable `ingestion_root` to the WORKER, rendering "The worker is not running." for a running worker — a false lead pointed at infra-ops's half of #377. Fixed (guard split into two banners); I confirmed from the DEPLOYED handler (not just the test) that `ingestion_root_writable:False` now renders "The ingestion root is not writable." ## muninn-gate → muninn-dispatch 0.1.5 Rebuilt `muninn-gate` off `vh/muninn-gate` main `bc04c4c` (dispatch 0.1.4→0.1.5) so the gate serves the new `concept_schema`/`concept_schema_source` row fields (computed gate-side). Gate version unchanged 0.0.14 (dual-tag the SHA). Build needs the vh gitea token as a BuildKit secret (`--secret id=gitea_pw`, UV_INDEX_GITEA_USERNAME=vh, drop+shred). Recreate with `compose up -d` (NOT bare restart — needs the new image). Verified: P&P job serves `concept_schema='fiction'`, `concept_schema_source=null` (null correct — pre-b164 job). Registry tags by commit SHA — `v1.0.0bNNN` docker tags don't resolve; use the deployed SHA (confirm `--reindex` present before using an image for a data-op). ## donut voice (65-frost → Zonos gateway) Operator: "pick up 65-frost, use that bundle as a voice for a character named donut." 65-frost = a **Booth id** (`~/booth-data/65-frost/`) holding a curated yt-voice-clipper dataset (`dataset-…-curated.zip`: 4 clips + manifest, all SPEAKER_02 = Princess Donut). **Zonos gateway voice registry = a filesystem drop:** `.wav` in the voices dir (44.1kHz mono s16 PCM) auto-registers as `voice:""` on **startup** (needs a restart). The LIVE dir is the bind mount `/opt/docker/compose/zonos-gateway/voices/` (lkraven-writable), NOT the working tree. Built `Donut.wav` from seg000 (best clip), dropped it, restarted → `voice:"donut"` live in the gateway AND the Asset Engine's make form. Also copied to the build-source tree `~/zonos-gateway/voices/` for rebuild-durability (true canonical = the gitea repo, not yet CI-wired). Auditioned in booth `donut-voice`. **Expanded 2026-08-02 (onyx-58 bundle):** operator curated a 2nd Booth bundle `onyx-58` (`dataset-467d2cf8…curated.zip`, 3 Donut clips) as additions. Rebuilt the reference = **seg000 (65-frost) + seg101/seg110/seg148 (onyx-58)** ffmpeg-concat + resampled 24k→44.1k mono s16 = **52.0s**. `seg148` was diarized SPEAKER_03 but is Donut (operator-confirmed misdiarize → included). Assembly is NOT `assemble_voice.py` (that `-c copy` can't resample + caps ~15s); used a manual `aresample=44100,aformat=…,concat=n=4` filter. Backed up old ref → `irv-ml1:~/Donut.wav.pre-onyx58`; dropped to live bind-mount + build-source tree; `docker compose restart` (healthy 2s, `voice:"donut"` still 1 of 9). A/B booth `donut-onyx58` (A=old 16.3s ref, B=new 52s ref, same line). Longer ref is fine mechanically: gateway passes it as `speaker_audio_base64` → speaker *embedding*, not an audio prefix. **BUT auditioned → REVERTED same day:** pinned-seed neutral A/B (5 pairs, booth `donut-onyx58`) showed the single-clip seg000 (16.3s) beats the 52s 4-take concat on timbre — concatenating disparate takes muddied the embedding more than the range helped. Reverted both live + build-source to seg000-alone. Lessons (→ Tried-and-abandoned): more reference ≠ better when takes vary; and **emotion steering pulls output away from the clone fast** (operator craft rule) — keep clones emotion-neutral; bare `{input,voice}` calls send NO emotion (gateway only enables it on an explicit `emotion_*`/`preset` dial). ## Zonos streaming (no gateway change needed) ratatoskr wanted play-as-it-arrives. `/v1/audio/speech` ALREADY streams — chunked `StreamingResponse`, opens native `/tts/generate` with `stream=True`, wraps as a streaming int16 WAV with `0xFFFFFFFF` placeholder sizes (meant for progressive `