6961 lines
534 KiB
Markdown
6961 lines
534 KiB
Markdown
# Archival memory — eshpfi-management
|
||
|
||
_Entries moved out of persistent-memory.md to keep the active file scannable. Read this when researching historical decisions or revisiting past foot-guns._
|
||
|
||
## Recent decisions (archived)
|
||
|
||
- `[2026-08-26]` **Run 3's corpus is built and HELD — `creative-writing-multiturn` is a MEGAMIX containing bluemoon, PIPPA, LimaRP and stheno**, and the remix promoted two roots that overlap at median jaccard 0.873. Dedup direction reversed to keep the primary source. F1 does not do what the recipe says; F2 does. → `persistent-memory.d/2026-08-26-run3-corpus-and-the-megamix-containment.md`
|
||
_Archived 2026-09-10._
|
||
|
||
- `[2026-08-26]` **No conversation admission threshold for run 3** (brokkr delegated, then endorsed). Masked context costs the transformer body but NOT the LM head — the harness drops `IGNORE_INDEX` before the 262k-vocab head — so a 20% trim buys ~35 min against a 3-hour run, and it would be a fifth filter confounding a run whose purpose is testing the filters alone. Tracked in the run-3 detail file.
|
||
_Archived 2026-09-10._
|
||
|
||
|
||
- `[2026-08-26]` **Run 2's gate FAILED and is recorded as a FAIL** — T3 constraint −12.0 against a ~1 pt floor. But gate 1 is the result: **T6 spatial +15.0 where run 1 failed the same axis at −3.5**, base swap the only intended variable. Neither run ships; the pair prices what the abliteration cost. Plus the long-form-only stability regression a short-answer gate would have passed, and PIPPA's 123-word clip in the length signal. → `persistent-memory.d/2026-08-26-erp-run2-complete-and-served.md`
|
||
_Archived 2026-09-10._
|
||
|
||
|
||
- `[2026-08-26]` **Run 2 complete, merged, coherence-gated and serving as `erp-tune-v2`** — stock-instruct base, and the mask proven by a −221,712 loss-token delta against byte-identical context. Also the p50 claim I asserted and had to withdraw. → `persistent-memory.d/2026-08-26-erp-run2-complete-and-served.md`
|
||
_Archived 2026-09-10._
|
||
|
||
- `[2026-08-26]` **Playbook §4 written: "when the artifact lies about itself"** — seven landmines plus a pre-launch checklist, from a night in which *three separate fixes each shipped a check that could not fail*. The unifying line is brokkr's: when you change what an artifact means, every derived artifact keyed on the old meaning is now a liar. Commits `dae6ede` → `d54f256`; the doc is `docs/pfi/training-throughput-playbook.md` (filename kept for inbound links; scope is now wider than the name).
|
||
_Archived 2026-09-10._
|
||
|
||
- `[2026-08-26]` **Served under a NEW name on a NEW port (`erp-tune-v2` / :8098), never re-pointing `erp-tune-v1`.** Run 1's artifact still exists and is still what that name refers to; re-pointing would be the silent substitution the standing no-false-aliases rule forbids. brokkr independently asked for the same and additionally wants the concrete backing model + date in provenance, not just the alias — an alias has silently changed meaning under recorded results before.
|
||
_Archived 2026-09-10._
|
||
|
||
- `[2026-08-26]` **DPO stage gated on an axis-list decision that is not mine to make** — `docs/pfi/erp-dpo-stage-prep.md`. No preference data for refusal axes exists; `trl` is not installed; the Gutenberg sets on disk are prose-quality only. ⚠ Do not install `trl` (or anything) into the training venv **while a run is saving** — a resolution that upgrades transformers under a live process can break its save path.
|
||
_Archived 2026-09-10._
|
||
|
||
- `[2026-08-16]` **Abliterated models go CATATONIC at the hard refusal edge — silence, not a decline.** Abliteration removes the refusal *direction*, so at the genuine hard edge the model neither refuses nor complies → empty/degenerate output. Durable measurement consequence: a refusal probe MUST score EMPTY as a verdict distinct from REFUSAL and COMPLY (`services/refusal-probe/probe.py` does). Operator accepted it as out-of-scope; do not chase.
|
||
_Archived 2026-09-02._
|
||
|
||
- `[2026-08-16]` **MTP works on Fable-Fusion AND survives RP temperatures — my earlier caution was wrong.** vLLM resolved `Qwen3_5MTP`, loaded the drafter, shared embedding + `lm_head` — the capability DS's seat never had because our quant dropped her MTP tensors. Measured over the full probe workload (~163k draft windows at temp 0.7–1.0): **47.0% acceptance** (229,169/487,725), 1.41 extra tokens/window, per-position 68.3/43.6/29.1%, **~80.6 tok/s** decode at temp 1.0. I had recorded a caution that the card's 1.56× was greedy-measured and acceptance would fall at RP temps — **it did not**; 47.0% matches the gen seat's 47.7% and beats the card's own 33% at depth 5. Depth 3 is right.
|
||
_Archived 2026-09-02._
|
||
|
||
- `[2026-08-16]` **The Qwen base thinks incessantly — that is WHY the Gemma seat exists, and no swap within the Qwen family fixes it.** Operator's architectural point, confirmed by measurement: on identical prompts DS 6036 ch vs FF 5323 ch of reasoning (permission arm), 5546 vs 4988 (cards arm) — FF actually reasons ~10–12% **less**. The bare-instruct row (DS 2291 vs FF 3918) inverts only because DS refused 92.5% of it and refusals are short — an artifact, not concision. Both are Qwen3.6-27B derivatives, so this is the base family. `char-rp` = **MeroMero-v2, Gemma-4 base**, :8016, verified 0 chars reasoning / clean prose — the non-thinking seat, working as designed. FF *can* be silenced (`enable_thinking:false` verified 3/3, and it ships `chat_template-instruct.jinja`) but that duplicates MeroMero on a base chosen for it. The stale LiteLLM comment describing `char-rp` as the retired GGUF Magidonia seat is fixed (`53096bf`).
|
||
_Archived 2026-09-02._
|
||
|
||
- `[2026-08-16]` **Canonical Qwen3.8 sampling applied from upstream; `gen-reasoning` had the WRONG-MODE presence_penalty.** Qwen/Qwen3.8-27B "Best Practices" §1 and unsloth/Qwen3.8-27B §1 are **byte-identical** — thinking: `temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0 / presence_penalty 0.0 / repetition_penalty 1.0`; instruct: `temp 0.7 / top_p 0.80 / top_k 20 / min_p 0.0 / presence_penalty 1.5 / repetition_penalty 1.0`. **Bug found:** `gen-reasoning` carried `presence_penalty 1.5` — the *instruct* value on a *thinking* deployment (canonical 0.0) — now fixed. **Deliberately NOT canonicalised:** `summarizer`/`classifier`/`image-judge`/`qwen-image-bench` run `temperature=0` (judges also `top_k=1`) because determinism is their contract; forcing a chat preset on a classifier would break it. ⚠ **`presence_penalty=1.5` is canonical but is the one value upstream hedges on**, verbatim: *"using a higher value may occasionally result in language mixing and a slight decrease in model performance."* It is the **operator's suspected trigger** for multi-turn degradation and the **first dial to move (0.0–0.5)** if that recurs — it is alias-scoped, which is why it would follow the operator across model builds. Commit `3462b53`.
|
||
_Archived 2026-09-02._
|
||
|
||
- `[2026-08-16]` **Four wrong diagnoses on one bug, and the lesson is the test design.** Operator reported the gen seat "degenerate on long multi-turn conversations". Rolled the seat back on request; **the previous weights behaved identically**, exonerating the model swap. I then proposed and disproved FOUR mechanisms in sequence — empty assistant turns poisoning history, reasoning runaway, length-mirroring from short history, and `presence_penalty` — before discovering **my own multi-turn harness was confounded**: it varied the QUESTION along with the depth (depth-1 asked question #2, depth-3 asked question #4), so a narrower question drawing a shorter answer read as degeneration. The "310→209→28w collapse" I reported as a reproduction was an artifact. **Rules banked:** (1) when comparing across conversation depth, hold the final question FIXED and vary only the history; (2) reply-length variance on byte-identical input was 25–465w, so n=3 cannot support any claim about a trend; (3) **ask for the operator's real failing transcript before building a synthetic reproduction** — four synthetic tests, none of them his failure. Gateway `spend_logs` returns `[]` on the infra-ops key despite `store_prompts_in_spend_logs: true`, so real transcripts need the `:4000/ui` view or another key — worth solving before the next such hunt.
|
||
_Archived 2026-09-02._
|
||
|
||
- `[2026-08-16]` **Two REAL client-side defects found while chasing the above, neither of which was the reported bug.** (1) `gateway-chat`'s Max-tokens field defaulted to **1024**; thinking seats spend part of that on CoT before emitting content, so completions truncate with `finish_reason=length` and read as model degeneracy — raised to 4096. (2) `parseInt` on an empty field yields NaN, which `JSON.stringify` serialises as **`null`**, which the server reads as "no max_tokens supplied" and silently substitutes its own default — indistinguishable from the UI ignoring the field. Both fixed (`b6552e0`, `fb3bb52`). ⚠ **`compose` bind-mounts a single FILE, and a single-file bind mount binds the INODE** — rsync writes-and-renames, so the container kept serving stale content while the host file showed the new value, silently and with no error. `docker restart` does NOT clear it; the container must be **recreated**. Verify against what the *container* sees, never the host file. Applies to any file-source mount fleet-wide.
|
||
_Archived 2026-09-02._
|
||
|
||
- `[2026-08-16]` **Refusal measurement: benign controls CANNOT validate a refusal classifier on RP prose — and a 0% rate needs a classifier self-test before you believe it.** Two durable lessons from baselining Dark-Scarlett. (1) **False positives:** my first bare-framing number was **9.5%**; the true figure was **1.4%**. The rest were the classifier firing on *in-character* text — `"I cannot shift my weight"` spoken by the character ~100 chars into a 2,443-token torture scene, and `"Yeah, I'm an AI… What's the actual gig?"` where the model answers in voice and keeps driving the scene. First-person RP prose is **full** of "I can't"; a genuine refusal *opens* with its marker, so the scan window must be the **first sentence**, a marker followed by long prose must demote to AMBIGUOUS, and AI self-acknowledgement is a **persona break, never a refusal on its own**. Benign controls were clean the entire time and caught none of it — they only detect over-firing on *benign* prompts, not on in-character prose. (2) **False negatives:** a 0% rate and a broken classifier are indistinguishable from the report, so `test_classify.py` (16 cases, both false positives pinned as regressions) must pass before any low number is trusted. Also banked: the **thinking-budget trap** — empty `content` + `finish_reason=length` is reasoning eating the budget, NOT a refusal; score INVALID and exclude from the denominator (DS emits ~5.5-6k chars of reasoning per response, so `max_tokens` ≥3072). `probe.py --rescore` re-classifies a saved run with zero GPU time. → `services/refusal-probe/README.md`, commit `32f665e`.
|
||
_Archived 2026-09-02._
|
||
|
||
- `[2026-08-16]` **Held an operator-approved swap window because the baseline invalidated its premise.** Operator approved ~65 min of `char-rp-reasoning` downtime to A/B Fable-Fusion 711 against Dark-Scarlett on refusals. The DS baseline then came back **0.0%/1.4%** — no gap for a candidate to close, so the window would have bought no decisive signal *and* a second window would still be needed once a reproducing battery existed. Held the swap, reported, and routed to brokkr-smithy-dev for the battery that actually produced the refusals. The general rule (action-relevance): **approval is for a plan, not a ritual — when new evidence kills the plan's premise, surface it rather than spend the budget.** Nothing deployed, no downtime taken, seat untouched.
|
||
_Archived 2026-09-02._
|
||
|
||
# esh-pve-nas — PVE root on a USB DOM: diagnosis, mitigation, migration plan
|
||
|
||
- `[2026-08-25]` **The ERP/RP tune COMPLETED in 7.36h and passed its gate on the axis it was built for** — diversity 22x its noise floor, attractor −11.3pt, zero memorisation on both arms. Also the noise-floor near-miss: brokkr was one step from reporting a 13-point T6 regression sitting inside twice his instrument's own variance.
|
||
# ERP/RP tune run-01 COMPLETE — 7.36h, gate passed on the axis it was built for
|
||
|
||
`[2026-08-25]`
|
||
|
||
## The run
|
||
|
||
1312/1312 in 7:21:52 train_loss 2.793 epoch 1.0
|
||
20.1 s/it FLAT across every 100-step window (round 1: 35-46.5 s/it)
|
||
adapter: /tank/erp-tune/run-01/adapter/ 410 tensors, provenance.json
|
||
|
||
**47% faster than the round-1 projection of 13.85h**, from two changes: the
|
||
bucketed sampler and flex attention. Rate was flat — 19.7 / 19.8 / 20.4 / 20.3
|
||
across the four 100-step windows — which means the 35-46.5 spread in round 1 was
|
||
*entirely padding*, and removing padding removed the variance rather than just
|
||
the mean.
|
||
|
||
⚠ **I quoted three different ETAs (6.9h, 8h, 7.3h) before I started using a
|
||
rolling average.** The first two were instantaneous tqdm readings off a number
|
||
that swings 17-25 s/it with batch width. Only the rolling rate was honest. Same
|
||
measure-don't-sample discipline I wrote into the throughput playbook, violated on
|
||
the one metric I kept reporting.
|
||
|
||
## lora_B gate — PASSED, twice
|
||
|
||
checkpoint-100 205/205 non-zero, median norm 0.829
|
||
final adapter 205/205 non-zero, median norm 1.708
|
||
vision_tower tensors: 0 on both
|
||
|
||
Median norm rising 0.829 -> 1.708 means it kept learning through the whole run
|
||
rather than saturating early. This check **never ran in round 1** (died at step
|
||
19, first checkpoint was 100) and it is the only failure mode that stays
|
||
invisible until the acceptance gate reports base-identical numbers.
|
||
|
||
## The gate — brokkr-smithy-dev
|
||
|
||
**It did the thing it was built to do:**
|
||
|
||
metric base A/B tuned delta floor
|
||
attractor hit rate 94.8% / 96.8% 84.5% -11.3pt 2.0pt
|
||
diversity (pairwise) 0.213 / 0.221 0.3948 +0.178 0.008
|
||
|
||
Diversity moved **22x its own noise floor**. Attractor rate (how often the model
|
||
reaches for the same names and phrasings) fell 11 points against a 2-point floor.
|
||
|
||
T1 100 · T2 95 · T3 96-97 · T4 98 · T5 100 · T6 81-82 · core ~94.2
|
||
memorisation: 0.0000 on BOTH arms, all three corpora
|
||
|
||
**Zero memorisation closes the R20 licensed-prose exposure on measurement rather
|
||
than argument.**
|
||
|
||
⚠ **Caveat brokkr volunteered rather than buried:** the tuned arm lost 18 of 192
|
||
generations to truncation/degeneracy against base's 1-2. Lopsided exclusions
|
||
plausibly flatter the diversity magnitude. Direction is unambiguous at 22x floor;
|
||
the number carries an asterisk.
|
||
|
||
## The noise-floor near-miss — the methodology lesson
|
||
|
||
brokkr was one step from reporting a 13-point T6 regression **that sat inside
|
||
twice his instrument's own variance.**
|
||
|
||
--per-type 32 max swing across tasks: 9 points
|
||
--per-type 128 max swing across tasks: 1 point
|
||
|
||
His gate criterion is "no task regresses by more than one item" = 3.1 points at
|
||
n=32. **The instrument's own run-to-run noise was 3 items.** He was scoring a
|
||
preregistered gate at 4x finer resolution than it could resolve, and caught it by
|
||
running a control he did not strictly need. Quadrupling n collapsed the noise
|
||
exactly as binomial statistics predicts.
|
||
|
||
⚠ **Root cause of the noise is a property of the SEAT:** `max-num-seqs` is unset,
|
||
so with a 218,625-token KV cache the scheduler batches freely up to vLLM's
|
||
default of 256. Continuous batching changes reduction order and borderline items
|
||
flip. Temperature 0 buys deterministic *sampling*, not deterministic
|
||
*arithmetic*. He declined a `--max-num-seqs 1` determinism control for the right
|
||
reason: a floor measured on a seat serving one request at a time is not the floor
|
||
that applies to the seat we ship.
|
||
|
||
## The confound I built and he caught
|
||
|
||
I optimised a pipeline for production and then handed him its output as an eval
|
||
instrument **without asking whether those were the same job.** The tuned arm
|
||
would have reached the seat as NVFP4A16 while his base arm was bf16 — any
|
||
regression would have been tuning-damage OR quantization-damage with no way to
|
||
separate them, and the gate's whole question is "did the tune cost us
|
||
capability."
|
||
|
||
**Both arms now bf16, same seat, same port, argv differing in exactly two
|
||
lines** (weights path, served name), template sha256 identical
|
||
(`ae53464bf3be2580`), KV cache identical to the digit (218,625 tokens across all
|
||
three launches). Quantization moved *downstream* of the gate.
|
||
|
||
See [[2026-08-25-refusal-retention-probe]] for the axis his gate did not have.
|
||
_Archived 2026-09-09._
|
||
|
||
- `[2026-08-25]` **8.6% MFU was an accounting artifact — real utilisation 17-20%, and the cost was attention on AMPERE kernels.** Two independent methods agreed to 2.6 points. Fixed by bucketing (padding 29.9%
|
||
# The 8.6% MFU was an accounting artifact — attention on Ampere kernels
|
||
|
||
`[2026-08-25]`
|
||
|
||
## The answer
|
||
|
||
**Real utilisation was 17-20%, inside the honest stock band.** The 8.6% divided
|
||
the *intended* (windowed) FLOPs by the wall time the *dense* reality took.
|
||
|
||
nominal billed 27.1 TFLOPS x 34.85 s = 9.4e14 FLOP
|
||
dense-sliding extra 25 layers, 2 seqs, 4 passes = +8.2e14
|
||
padded full layers lose the causal skip = +3.5e14
|
||
work performed ~ 1.8e15 = 51-61 TFLOPS
|
||
|
||
The card was doing ~2x the arithmetic the architecture specifies, and the excess
|
||
was the sliding window being computed and thrown away.
|
||
|
||
## Two independent methods agreed
|
||
|
||
scaling fit (3 points, 2 params, residuals <3ms over 8x range)
|
||
A = 6.87e-4 s/token B = 8.85e-8 s/token^2
|
||
quadratic share: 20.9% @ w=2048 -> 67.8% @ w=16384
|
||
|
||
kernel table (device rows only)
|
||
attention 22,835.8 ms 65.2% fmha_cutlass*_sm80
|
||
dense GEMM 2,774.0 ms 7.9%
|
||
other 5,739.0 ms 16.4%
|
||
|
||
**67.8% vs 65.2% — 2.6 points apart, no shared assumptions.** The two-term fit
|
||
needed no constant term, which refutes launch-bound outright (3,840 expert-GEMM
|
||
launches per forward are not the cost).
|
||
|
||
## The mechanism, source-verified by brokkr's panel (arm: Bil)
|
||
|
||
masking_utils.py:292-301 _ignore_causal_mask_sdpa requires
|
||
kv_length < local_attention_size. 16384 >= 1024,
|
||
so THE SLIDING MASK ALWAYS MATERIALISES.
|
||
sdp_utils_cpp.h:259-267 flash rejects ANY explicit mask
|
||
sdp_utils.cpp:647 cuDNN head_dim capped at 128 -> unreachable
|
||
Context.h:480-485 prefer-cuDNN needs major 9 or 10; sm_120 is 12
|
||
|
||
⚠ **The kernels are `sm80` — Ampere-generation CUTLASS on a Blackwell card**,
|
||
with the forward on `gmem`, the memory-efficient backend's slowest fallback tier.
|
||
|
||
## What actually fixed it
|
||
|
||
**Bucketing (bucket-to-pair, shuffle-to-mix)** — 29.9% padding -> 0.0%, and
|
||
78.3% of micro-batches become exactly zero-pad, which puts the 5 global layers
|
||
back on `is_causal`. Measured: padding costs **9.4% MORE time for 24% LESS work**
|
||
at fixed width, because an explicit mask knocks those layers off the fast path.
|
||
|
||
⚠ **Bucket size is NOT a diversity knob.** Swept across a 256x range, roots per
|
||
accumulation window stayed flat at 3.54-3.61. The global micro-batch shuffle does
|
||
all the mixing; the bucket only costs padding. Use the tightest bucket.
|
||
|
||
**flex_attention** — Triton-generated so it compiles for sm_120 instead of
|
||
shipping sm_80 binaries. 21.7x on sliding layers, 2.1x on global. Needs mandatory
|
||
`kernel_options` at 32x32 blocks: 64x32 needs 102,400 bytes against a
|
||
**101,376-byte hardware ceiling** — misses by 1 KB, and Triton is already opting
|
||
into the full 99 KB, so it is the card, not a default.
|
||
|
||
## ⚠⚠ The trap that produced TWO wrong published conclusions
|
||
|
||
`torch._dynamo` defaults to a recompile ceiling of **8**. Every distinct sequence
|
||
width is a new shape. On hitting the ceiling dynamo does not error — it silently
|
||
falls back to UNCOMPILED flex, which is ~20x slower AND documented to *"not work
|
||
with the backwards pass and may produce incorrect results."*
|
||
|
||
That artifact produced a bogus **0.76x slowdown** and a bogus **2.9% loss
|
||
divergence**, and I believed and reported both. Raising the limit to 256 flipped
|
||
the speed result to 1.41x.
|
||
|
||
The loss divergence turned out to be real but benign — adjudicated against fp32
|
||
MATH ground truth, both backends sit ~2e-3 from truth with flex fractionally
|
||
CLOSER at every width. **Do not re-open it by comparing the two backends to each
|
||
other; that cannot answer it. Compare to fp32.**
|
||
|
||
## Process lesson
|
||
|
||
brokkr's panel produced **four self-retractions in ninety minutes**. Every
|
||
retraction was a derivation; every survivor was a measurement. And the whole
|
||
head_dim-512 SDP problem was **already documented in zerofata's published Axolotl
|
||
config since April** — the right first stop for "why is this architecture slow"
|
||
is practitioner configs for that exact base, before any panel.
|
||
|
||
Playbook: `docs/pfi/training-throughput-playbook.md`, commit `7b5fd91`.
|
||
_Archived 2026-09-09._
|
||
|
||
- `[2026-08-25]` **NVFP4A16 serving pipeline built and validated; MERGED WEIGHTS ARE MANDATORY.** vLLM cannot serve a LoRA on ANY Gemma-4 — `get_expert_mapping` is unimplemented and the check branches on MoE-ness, not quantization. Plus the landmine: a `targets=["Linear"]` recipe misses all 11,520 expert tensors silently.
|
||
# NVFP4A16 serving pipeline — built, validated, and the MoE landmine it found
|
||
|
||
`[2026-08-25]`
|
||
|
||
Pipeline at `scripts/erp-tune-serve/` (commits `6a85829`, `ab980e9`).
|
||
Validated end-to-end against checkpoint-100 before the real adapter existed.
|
||
|
||
## ⚠⚠ THE LANDMINE: a `targets=["Linear"]` recipe misses EVERY MoE expert
|
||
|
||
before linearize_moe: 427 Linears, 205 targeted, experts 0
|
||
after linearize_moe: 11,947 Linears, 11,725 targeted, experts 11,520
|
||
(30 layers x 128 experts x 3 projections)
|
||
|
||
Gemma-4 stores each layer's 128 experts as two fused 3-D `nn.Parameter` tensors
|
||
(`gate_up_proj` [128,1408,2816], `down_proj` [128,2816,704]) — note the absent
|
||
`.weight` suffix. A Linear-targeting recipe resolves 205 of 427 modules and
|
||
**zero experts**, leaving 22.84 B params (88.5% of the model) in BF16 with no
|
||
warning.
|
||
|
||
**This is the same defect that killed QLoRA here via bitsandbytes.** The blind
|
||
spot is in the *checkpoint layout*, not the tool. Fix:
|
||
`llmcompressor.modeling.moe.linearize.linearize_moe` — no registration needed,
|
||
Gemma-4 satisfies `FusedExpertsProtocol` structurally. Playbook §3.15.
|
||
|
||
## Scheme: NVFP4A16, deviating from the playbook default, on measured grounds
|
||
|
||
brokkr benched the W4A4 quant of this checkpoint at **12% on contradiction
|
||
detection with CoT off against gen's 81%** — the signature of 4-bit input
|
||
activations on a reasoning-dense task. Plus W4A4 KLD is 2-4x worse past ~10k ctx
|
||
on sm_120. This is a 16,384-ctx RP seat. Marlin's prefill cost accepted.
|
||
|
||
⚠ Several HF repos named `…-NVFP4A16` declare `input_activations num_bits 4` —
|
||
W4A4 wearing an A16 label. The script refuses if the emitted config says 4.
|
||
|
||
## Four silent defects the dry run found
|
||
|
||
1. **transformers 5.15 MIGRATES the config schema on save** — drops
|
||
`global_head_dim`/`num_global_key_value_heads`, writes `per_layer_config`.
|
||
transformers 5.10 (the llmcompressor venv) then reads `num_key_value_heads`
|
||
as None and dies with `TypeError: unsupported operand type(s) for //`.
|
||
Every working artifact on the box uses the OLD schema. Merge now downgrades it.
|
||
2. **llmcompressor cannot auto-init a processor for a multimodal checkpoint** —
|
||
pass the tokenizer explicitly as `processor`.
|
||
3. **`save_pretrained` does not carry `processor_config.json`** — vLLM then fails
|
||
with "Can't load feature extractor", which reads as a vision bug.
|
||
4. **The quant needs more than GPU1's free 32 GiB.** `quant_with_gen_down.sh`
|
||
stops `vllm-gen` and restores it from a trap on EVERY exit path, using
|
||
`docker start` not `compose up` so the container returns with its exact config.
|
||
|
||
## Verified on the emitted artifact
|
||
|
||
49 GB -> 17 GB, format nvfp4-pack-quantized, a=null (genuine A16)
|
||
weight_packed 11,725 of which expert 11,520
|
||
tokenizer truncation: clean (§3.14 trap avoided by calibrating on the
|
||
encode cache, so the tokenizer is never called
|
||
with truncation=True at all)
|
||
served: Marlin NVFP4 kernel + Marlin MoE backend, coherent generation
|
||
|
||
⚠ The reference `nvfp4a16` artifact triggers a vLLM warning that q/k/v carry
|
||
*different* weight global scales ("likely reduced accuracy"). **Ours does not** —
|
||
llmcompressor 0.12 links weight observers across fused groups automatically. The
|
||
in-house quant is better than the downloaded one on that axis.
|
||
|
||
## ⚠ MERGED WEIGHTS ARE MANDATORY — and not for the reason we assumed
|
||
|
||
The open question was whether LoRA-on-NVFP4 hot-swap still silently no-ops.
|
||
Retested on `vllm/vllm-openai:latest`: **it refuses to start.**
|
||
|
||
AttributeError: To support LoRA for MoE model,
|
||
'get_expert_mapping' must be implemented
|
||
|
||
The check is in `vllm/lora/utils.py::process_packed_modules_mapping` and branches
|
||
on `is_moe_model()` — **quantization is not in the condition.** `gemma4.py`,
|
||
`gemma4_mm.py`, `gemma4_mtp.py`, `gemma4_unified.py` all have ZERO occurrences;
|
||
`deepseek_v2`, `mixtral`, `glm4_moe`, `ernie45_moe` implement it.
|
||
|
||
**vLLM cannot serve a LoRA on ANY Gemma-4, bf16 or quantized.** Merging is the
|
||
only path for this architecture, and it would have bitten identically on the
|
||
unquantized base. A loud refusal is strictly better than the 0.24.0 silent no-op,
|
||
which shipped a base model wearing the tune's name.
|
||
|
||
⚠ Base-viability pre-flight is now playbook §3.11 — three greps before picking a
|
||
base. **Grep the CLASS, not the file**: `mistral.py` greps as `SupportsLoRA=0`
|
||
and is fully LoRA-capable via inheritance from `LlamaForCausalLM`.
|
||
_Archived 2026-09-09._
|
||
|
||
- `[2026-08-25]` **Refusal retention measured (base 0/100
|
||
# Refusal retention — the axis the gate did not have, and the axis I measured wrong
|
||
|
||
`[2026-08-25]`
|
||
|
||
## Why it exists
|
||
|
||
brokkr's gate measures reasoning (T1-T6), craft (diversity/attractor) and
|
||
regurgitation (memorisation). **Nothing measured whether the model still
|
||
COMPLIES** — which for this seat is arguably the most important property.
|
||
|
||
The risk is specific to our operation order. We do **tune(abliterate(stock))**,
|
||
so the tune has 57.7M tokens of opportunity to walk the abliteration back. *A
|
||
tune that gains 41 items of contradiction detection and quietly re-installs
|
||
refusals is a failed seat that passes the entire gate.*
|
||
|
||
## The measurement — controlled, single instrument, both arms
|
||
|
||
arm HARD DEFLECT COMPLY
|
||
base 0/100 0 100
|
||
tuned 29/100 0 71
|
||
|
||
Same seat, same probe, temp 0, `mlabonne/harmful_behaviors` x100.
|
||
Probe: `scripts/training-probes/refusal_probe.py`.
|
||
|
||
**The tune added 29 general-harm refusals where the base had none.**
|
||
|
||
Two things fell out:
|
||
|
||
- **The instrument validates.** Base measured 0/100 on my generated-text regex
|
||
against Heretic's recorded 3/100 from a first-token-probability scorer. 0 vs 3
|
||
is agreement — the incomparability worry was right caution about a non-problem.
|
||
- **DEFLECT is 0 on BOTH arms, so the free control fires.** An instrument
|
||
artifact does not care which arm it runs against. Both zero means the model is
|
||
**binary** — refuses in refusal-language or engages, no soft-deflection tail.
|
||
The R19 undercount does not apply here.
|
||
|
||
## ⚠⚠ But it is the WRONG AXIS — brokkr's catch, and it is the better one
|
||
|
||
`mlabonne/harmful_behaviors` is **general harm** — weapons, malware, fraud. **The
|
||
abliteration was not run so the model would explain bomb-making. It was run so
|
||
the model would engage with explicit fiction.** Different refusal surfaces; a
|
||
model moves on them independently.
|
||
|
||
I picked that set because it was cached, had a recorded baseline, and was what
|
||
the abliteration tool used. **Every one of those is a reason it was convenient,
|
||
not a reason it was right** — and "it has a baseline" was actively misleading,
|
||
because a comparable number for a question nobody is asking looks like evidence.
|
||
|
||
**29/100 general-harm refusals on a seat writing prose the operator was actively
|
||
praising is plausibly the DESIRED shape**, not a defect. General-harm refusals
|
||
returning while domain compliance holds is close to ideal for an internal
|
||
creative seat. I would have reported it as damage.
|
||
|
||
**The load-bearing cell is COMPLY 71, not the 29.** Stock refused 100/100;
|
||
anything near that would mean the abliteration was undone. 71 complying means
|
||
"partially walked back on one axis" — a different finding, and only one of the
|
||
two threatens the seat.
|
||
|
||
Domain-compliance probe (the right axis, from R19's track-2 map) is brokkr's,
|
||
pending. Scaffold supplied: `scripts/training-probes/counted_classifier.py`
|
||
(`2a05ae9`) — classify-never-surface, three-way, ERROR path deliberately does not
|
||
log the exception body because an exception can echo the prompt back.
|
||
|
||
Playbook §3.13. See [[2026-08-25-erp-tune-run2-complete]].
|
||
_Archived 2026-09-09._
|
||
|
||
- `[2026-08-25]` **Worldtree b188 + b189 shipped; bridge extracted to `pfi/wt-matrix-bridge` because `vh` is a USER not an ORG** and no service account can ever publish to a user namespace. Plus the selene catalog entry that lied about what answers, and a #411 diagnosis I got wrong twice before a directory probe settled it.
|
||
# Worldtree b188 + b189 bridge cutover, and the selene metadata that lied
|
||
|
||
`[2026-08-25]`
|
||
|
||
Three arcs in one day, all infra-ops side, all landed.
|
||
|
||
## b188 — matrix.yaml pre-sync (#406/#409/#410 closed)
|
||
|
||
From b188 the bridge reads per-agent `rendering` + `ambient_buffer_size` from
|
||
`config/matrix.yaml` ONLY; agent `config.yaml` matrix blocks are gone from the
|
||
image. Staged as `6417115` in `worldtree-instance-configs`, deployed to both
|
||
instances with operator approval.
|
||
|
||
- mimir gets thinking-to-thread + tool-call reactions + 7 reaction labels;
|
||
forseti and lofn stay quiet.
|
||
- **Rider #409 pruned six dead agents** (bragi, leif, troi, soong, cara, glados)
|
||
from BOTH rosters — originally scoped personal-only, which I flagged as a
|
||
possible oversight and it was. **The settling fact worth keeping: the engine
|
||
roster comes from the image's baked `agents/` directory (only `config/` is
|
||
bind-mounted), and both instances run the same image**, so instance-level
|
||
evidence about which agents the engine lists generalises by construction.
|
||
- Both rosters now exactly `[mimir, forseti, lofn]` — the three engine agents
|
||
actually bridged. mask/vili/echo exist in the engine, deliberately unbridged
|
||
(operator ruling).
|
||
|
||
⚠ **Edited text-surgically, not via a yaml round-trip** — PyYAML would reflow
|
||
1,249 lines and drop every comment, and the comments are the documentation.
|
||
|
||
⚠ **`deploy-wt-config` uses `docker restart`, NOT `compose up`.** A `compose up`
|
||
on corviduo-dev re-resolves the image tag and can silently swap the running
|
||
build — which would turn an "inert pre-sync" into an unintended image roll on two
|
||
live instances. That property is easy to lose in a future refactor of the script.
|
||
|
||
## b189 — #407 bridge extracted to its own repo (#404 umbrella closed)
|
||
|
||
Bridge now `gitea.phasefinal.com/pfi/wt-matrix-bridge`, its own repo, own CI.
|
||
|
||
⚠ **It publishes to the `pfi` ORG, not `vh`, and the reason is structural:** `vh`
|
||
is a **USER**, not an org. Gitea scopes user-namespace packages to the owning
|
||
user — there are no package collaborators on a user namespace. **No service
|
||
account can ever publish to `gitea.phasefinal.com/vh/*`.** claude-bot is an Owner
|
||
of `pfi`, so that is where it goes. Token `wt-matrix-bridge-ci` (id 28, scopes
|
||
`write:package,read:repository`), vaulted at
|
||
`nh3-dev/.config/claude-bot/gitea-token-wt-matrix-bridge-ci`.
|
||
|
||
**Minted a dedicated token rather than reuse `claude-bot-sdk-ops` or `arbo-ci`,
|
||
both of which already carry `write:package`** — a shared credential cannot be
|
||
revoked without collateral. ⚠ A first mint attempt succeeded then failed to save;
|
||
Gitea returns a token value exactly once, so it was unrecoverable. Deleted the
|
||
orphan (id 27) rather than leave a live package-write credential on the account.
|
||
|
||
**Both instances PINNED** to `f3f8ec902267` (`e90f436`), closing the #410 shape:
|
||
`WORLDTREE_IMAGE` was pinned and the bridge was the one drifting service.
|
||
|
||
⚠ The pin moved from `b178285b1cb5` because a cross-frontier bug-hunt found the
|
||
M_EXCLUSIVE **fallback could itself litter unboundedly** in exactly the state it
|
||
was written for. **The safety net had the same failure mode as the thing it was
|
||
catching** — and my staged window leaned on that fallback.
|
||
|
||
## #411 — the debug-room failure, diagnosed twice and wrong both times first
|
||
|
||
My theory: the alias was held by orphaned rooms. **Refuted by a directory probe
|
||
returning 404.** The real cause: Synapse's **M_EXCLUSIVE** — an appservice may
|
||
only create aliases inside a namespace it has RESERVED, and
|
||
`aipa_appservice.yaml` had `namespaces.aliases: []`.
|
||
|
||
⚠ **I inferred a cause from a symptom that was the RESPONSE to the cause** — the
|
||
log's "re-resolving alias" line is the recovery path firing, not evidence the
|
||
alias exists. One directory probe settled it and I reasoned instead.
|
||
|
||
Fixed with operator clearance: added
|
||
`regex: '#aipa-debug-[a-z0-9_-]*:matrix\.phasefinal\.com'` (exclusive) at
|
||
`/opt/docker/conf/synapse/aipa_appservice.yaml` on **ana-docker** (NOT
|
||
`/opt/docker/data/`, which worldtree-dev's issue cited). **My regex was tighter
|
||
than the proposed `#aipa-debug-.*`**, which fullmatches only because `.` also
|
||
matches the `:` separator and would equally claim other homeservers.
|
||
|
||
Pre-apply sweep (an `exclusive: true` claim can make Synapse refuse to START):
|
||
|
||
aliases matching '%aipa-debug%' 0
|
||
total room_aliases on the homeserver 1 <- why it went unnoticed this long
|
||
rooms created by @aipa-debug 17 <- the litter, confirmed unaliased
|
||
|
||
Synapse healthy in 40s, both bridges rode through. **The aliased create then
|
||
worked FIRST TRY on the next personal recreate** — designed path, fallback never
|
||
fired, both reuse mechanisms live (canonical-alias rediscovery AND the room-id
|
||
cache).
|
||
|
||
⚠ The #411 writer census **inverted its own premise**: the api (uid 1000) cannot
|
||
write `/app/sessions`, the bridge (root) can — the reverse of the issue text. But
|
||
worldtree-dev's reconciliation is better than "backwards": pre-#407 the bridge
|
||
ran from the ENGINE image as uid 1000, and the cutover changed the answer
|
||
underneath the issue. Both readings were true at their timestamps. **No live
|
||
writer exists for that path anyway** (`sessions.path` is a legacy default, #330
|
||
moved the tools off it), so the chown is optional future-proofing.
|
||
|
||
## selene-1-mini-8b — a config that lied about what answers
|
||
|
||
forseti's fleet sweep found the only genuine residual in
|
||
`worldtree-instance-configs`, and it was **live on both boxes, not just drifted
|
||
in git.** Fixed in `a77639d`.
|
||
|
||
Routing was never broken (`model: "chat-judge"` stays), but `display_name` said
|
||
"Selene 1 Mini 8B" and the description said "Atla Selene 1 Mini 8B — reward model
|
||
derived from Llama 3.1 8B" while chat-judge has been backed by
|
||
**qwen3.8-27b-uncensored** since 2026-08-23.
|
||
|
||
⚠ **It misstated the KIND of model, not just the identity.** A reward model and a
|
||
generative judge are different instruments; a consumer reading
|
||
"pairwise/likert/binary/scalar reward model" would expect scalar-reward semantics
|
||
qwen3.8-27b does not provide.
|
||
|
||
Verified: `selene-1-mini-8b` -> HTTP 400 (by design), `chat-judge` -> HTTP 200.
|
||
|
||
**NOT changed:** the catalog key and the `selene-judgment` role — `model_roles.yaml`
|
||
binds to that key, so renaming is worldtree-dev's schema call. Flagged that a role
|
||
named after a retired model defeats the purpose of role aliases.
|
||
|
||
⚠ forseti's sharpest finding is for the operator: **`~/.claude/CLAUDE.md` line 502
|
||
lists the retired name in the global tools roster.** A broadcast reaches sessions
|
||
that already exist; the roster line keeps minting new ones. His file, awaiting his
|
||
word. Also: **there is no fan-out primitive on the bus** — a fleet relay is 73
|
||
individual posts, recorded as a real gap rather than papered over.
|
||
_Archived 2026-09-09._
|
||
|
||
- `[2026-08-25]` **The ERP/RP tune LAUNCHED after 12 harness defects and an operator override of the corpus gate.** Four of the twelve would have crashed the run; two were INERT GATES that passed because they could not fail. Run is `/tank/erp-tune/run-01`, harness eitri-smithy `997c4a4`. Full arc — override, defects, sizing, the measured MFU — in the in-flight section and `docs/pfi/gemma4-erp-tune-sizing.md`.
|
||
_Archived 2026-09-09._
|
||
|
||
- `[2026-08-24]` **char-rp seat swapped to the Gemma-4 26B-A4B MoE; abliterated trainee base staged and measured.** OOM root-caused to `--gpu-memory-utilization` not covering CUDA context (and to gen's footprint GROWING WITH UPTIME); a benchmark finding retracted because it scored below chance; abliteration isolated at −0.6 core points but it MOVES capability rather than removing it.
|
||
# `[2026-08-24]` char-rp seat: OOM root-cause, Gemma-4 MoE swap, and the abliterated trainee base
|
||
|
||
One evening, one thread with brokkr-smithy-dev, five commits: `850e0c3`,
|
||
`27155c0`, `f509668`+`24e8826`+`1bd90ea`+`3446367`+`8d6a939`, `14ff4a3`,
|
||
`019ccff`, `5415fd4`.
|
||
|
||
## 1. The seat was crash-looping, and the cause was NOT its config
|
||
|
||
`vllm-meromero-rp` reported up-but-unreachable, RestartCount climbing (13 by the
|
||
time it was examined, not the 4 first reported). Startup logs looked clean all
|
||
the way through weights, `torch.compile` and CUDA-graph capture, then:
|
||
|
||
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 336.00 MiB.
|
||
GPU 0 has a total capacity of 94.97 GiB of which 195.19 MiB is free.
|
||
|
||
**⚠ `--gpu-memory-utilization` SIZES THE KV CACHE AND DOES NOT COVER CUDA
|
||
CONTEXT, GRAPHS OR NON-TORCH OVERHEAD.** gen is configured at 0.43 and actually
|
||
held 45.6 GiB. char-rp was at 0.51. The pair was committed to 0.94 of the card
|
||
with ~0.6 GiB of real headroom — it fit on the 21st and stopped fitting on the
|
||
24th.
|
||
|
||
Dropped char-rp to 0.47: ~4.8 GiB margin, KV 27.36 → 23.56 GiB, 430,825 →
|
||
371,023 tokens against a max-model-len of 262,144. **Cost nothing usable** — the
|
||
pool still holds 1.4x a full-length sequence; what is lost is concurrent long
|
||
requests, not context.
|
||
|
||
**⚠⚠ THE MISSING HALF, found later that evening: gen's footprint GROWS WITH
|
||
UPTIME.** Same container, same 0.43: **45.6 GiB after ~3 days up, 38.5 GiB
|
||
freshly restarted** — ~7 GiB apart. Nothing about char-rp changed between the
|
||
21st and the 24th; *gen crept up underneath it*. **Headroom arithmetic done
|
||
against a long-running gen is measuring a moving number.** Measure against a
|
||
freshly-restarted one.
|
||
|
||
## 2. `char-rp` and `char-rp-reasoning` are ONE seat, not two
|
||
|
||
Both LiteLLM routes point at `10.250.50.54:8016/v1` — `hosted_vllm/char-rp` and
|
||
`hosted_vllm/char-rp-thinking`. brokkr had reported 30/80 and 80/80 failure
|
||
rates as two failing services; it was one outage sampled twice. This also
|
||
*improved* a result of theirs: their CoT on/off battery had assumed both aliases
|
||
were the same weights under two chat templates, and the routing detail turned an
|
||
assumption into a verified fact.
|
||
|
||
(`vllm-charrp-reasoning-nvfp4`, the Heretic2 NVFP4+MTP container, has been
|
||
stopped for 12+ days and is unrelated — it is not what that alias resolves to.)
|
||
|
||
## 3. The seat swapped to the Gemma-4 26B-A4B MoE
|
||
|
||
Operator-directed straight-across replacement: same port, same
|
||
served-model-names, so no gateway route or consumer config moved. Rationale is
|
||
throughput under CoT — the user waits through the whole reasoning block before
|
||
the first visible token, and the MoE measures ~114 tok/s @32K against the dense
|
||
31B's ~40.7.
|
||
|
||
Serving copy is `RedHatAI/gemma-4-26B-A4B-it-NVFP4` (16 GB), chosen over the
|
||
other `-it` quants because it is compressed-tensors (`nvfp4-pack-quantized`) —
|
||
the same loader path the outgoing seat used. Smaller weights at the same 0.47
|
||
budget bought **1,724,110 KV tokens against the predecessor's 371,023**.
|
||
|
||
`meromero-charrp` is retained stopped in `created` state, labelled
|
||
`AI - Dormant`. Both stacks bind `:8016`, so rollback is **stop-then-start**.
|
||
|
||
## 4. ⚠ THE STALE-CHAT-TEMPLATE TRAP IS ENDEMIC, NOT A ONE-OFF
|
||
|
||
Verified by hash across every third-party Gemma-4 derivative pulled:
|
||
|
||
| build | lines | sha256 (normalised) |
|
||
|---|---|---|
|
||
| upstream `google/gemma-4-26B-A4B-it` | 390 | `6a1015c47ccfcfa6` |
|
||
| RedHatAI NVFP4 (served) | 389 | `6a1015c47ccfcfa6` — the only match |
|
||
| llmfan46 heretic | 365 | `0a52be69cda5ab8a` |
|
||
| TrevorJS abliterated | 266 | `58c66fdee4afa297` |
|
||
| jenerallee78 abliterated | 266 | `58c66fdee4afa297` |
|
||
| prithivMLmods NVFP4A16 | 266 | `58c66fdee4afa297` |
|
||
|
||
Three independent repos carrying the *identical* stale file means it propagated
|
||
through the ecosystem. Consequences differ by use and **both are silent**:
|
||
serving a mismatched template renders a different prompt; **training through
|
||
`base/chat_template.jinja` means training on a different prompt format than
|
||
production serves** — train/serve skew, no error, presents as a tuning failure.
|
||
|
||
The production compose now pins the template explicitly. It is a **no-op for the
|
||
served weights** (the A4 build ships that exact file) and permanently closes the
|
||
class. ⚠ If `GEMMA4_MODEL` ever points at a different checkpoint, the template
|
||
default must move with it.
|
||
|
||
## 5. A benchmark result was RETRACTED — below chance indicts the instrument
|
||
|
||
A battery appeared to show Gemma at **12% contradiction detection with CoT off
|
||
against gen's 81%**. An A16 activation-precision control was staged to test
|
||
whether the quant scheme owned it. Then the operator asked to see the individual
|
||
items, and the item was **ill-posed**: it presented two mutually contradicting
|
||
statements and asked for "*the* contradicting statement", but **contradiction is
|
||
symmetric**. The model consistently named the absolute claim — a defensible
|
||
reading the labelling scored wrong every time.
|
||
|
||
**⚠ THE TELL WAS IN PLAIN SIGHT: 12% ON A FIVE-OPTION TASK IS BELOW THE 20%
|
||
CHANCE FLOOR.** A below-chance score indicts the instrument before it indicts
|
||
the model, and neither side reacted to it. I spent the afternoon verifying repo
|
||
names, config fields, template hashes and tokenizer vocabs — every layer of
|
||
plumbing — and never asked whether the number itself was *possible*. **A
|
||
preflight can be thorough and still be aimed in the wrong direction.**
|
||
|
||
Retracted: "the model owns the contradiction deficit"; "domain tuning costs 43
|
||
points of contradiction detection" (on a sound instrument it **reverses**); all
|
||
pre-fix T2 numbers. Recorded as a dated superseded-claims table in
|
||
`stacks/gemma4-charrp/README.md` rather than a silent edit.
|
||
|
||
**What survived:** the A16 control result — activation precision is close to free
|
||
on this battery, every other task identical across W4A4 and W4A16 builds.
|
||
|
||
## 6. The abliterated trainee base — measured, not assumed
|
||
|
||
Operator directed a low-damage abliterated instruct build. "Low damage" was
|
||
treated as a measurable claim; the field spreads from KL 0.09 to 0.4118:
|
||
|
||
| build | method | KL | refusals |
|
||
|---|---|---|---|
|
||
| **llmfan46** (operator's pick) | Heretic v1.2.0 ARA | 0.1237 | 3/100 |
|
||
| TrevorJS | ARA-family | 0.09 | 1/100 effective, 5/686 cross-dataset |
|
||
| jenerallee78 | ARA 2-pass | 0.1299 | 7.7% StrongREJECT |
|
||
| huihui-ai | remove-refusals | none published | none published |
|
||
|
||
Fleet anchor: our own work found **Heretic at KL 0.12 preserved the MTP head at
|
||
83.7% acceptance**, so both staged builds sit inside an already-measured band.
|
||
huihui-ai rejected — no metrics, its card calls the method "a crude,
|
||
proof-of-concept implementation", it abliterates both thinking and non-thinking
|
||
modes, and its parameter count runs ~738M over upstream. Operator's independent
|
||
read matched ("huihui produces garbage").
|
||
|
||
**Abliteration isolated properly** (stock BF16 vs llmfan46 BF16, same precision,
|
||
same pinned template, same 192 items):
|
||
|
||
T2 contradiction 75% → 59% (−5 items)
|
||
T6 spatial 75% → 88% (+4 items)
|
||
core 90.0% → 89.4% (−0.6 pts)
|
||
|
||
**It MOVED capability rather than removing it** — five lost on contradiction,
|
||
four gained on spatial, nearly cancelling. Nobody predicted a gain. **llmfan46
|
||
stands**; no case for re-staging on TrevorJS over 0.6 points.
|
||
|
||
⚠ Read as ~5 and ~4 items at n=32, not as −15.6/+12.5 percent. ⚠ Says nothing
|
||
about quantization — the stock-NVFP4 T2 figure came from n=16 against n=32,
|
||
different item sets, n-confounded.
|
||
|
||
## 7. ⚠ The production compose hardcodes `--quantization compressed-tensors`
|
||
|
||
Pointing the char-rp stack at unquantized BF16 weights crash-loops immediately:
|
||
|
||
TypeError: CompressedTensorsConfig.__init__() missing 3 required
|
||
positional arguments: 'target_scheme_map', 'ignore', 'quant_format'
|
||
|
||
vLLM trying to read a quantization config out of a checkpoint that has none. 35
|
||
restarts before it was caught. Hence `stacks/gemma4-trainee-bench/` — a separate
|
||
ephemeral stack with no quantization flag, `restart: "no"` so a bench seat cannot
|
||
resurrect itself and block gen's restore, and no homepage labels so it leaves no
|
||
permanently-offline card. That detour is why a base swap is now ~5 minutes
|
||
instead of ~15.
|
||
|
||
## 8. BF16 cannot coexist with gen
|
||
|
||
48.07 GiB of BF16 weights plus gen's footprint exceeds the 94.97 GiB card before
|
||
a byte of KV cache. Every BF16 bench window means **gen is stopped**. Two such
|
||
windows were run and gen restored both times; the restore was triggered by
|
||
observing the seat's own throughput logs (a large prefill burst then zero
|
||
running/zero waiting) rather than waiting on a courtesy message.
|
||
|
||
Cross-links: [[2026-08-24-homepage-uniform-grid]]
|
||
_Archived 2026-09-09._
|
||
|
||
- `[2026-08-24]` **Homepage rebuilt on Australis Skyfall; light mode shipped.** Two findings worth more than the theme: **(a)** the Skyfall bundle including its canonical light ramp was sitting in this repo's git history at `45c1995` — check `git show` before concluding a vendored design asset is lost; **(b)** removing `theme:` from `settings.yaml` deterministically breaks the dashboard render (six recreates empty, restoring the key fixed it in 12s), which is the first confirmed cause of the "tab bar goes missing" symptom. Retires the `homepage.log` size lead from earlier the same day — it did nothing on this episode.
|
||
# `[2026-08-24]` Homepage: remote-label consumption re-verified, then the board relaid out on a uniform grid
|
||
|
||
Prompted by the operator: *"Homepage on esh-vm-docker lists remote dockers and
|
||
can absolutely consume their labels, please verify again. I am still
|
||
unsatisfied with the layout and aesthetics."*
|
||
|
||
## The verification — the operator was right, and the record now says so
|
||
|
||
**Homepage on `esh-docker-vm` discovers services by container label from all
|
||
five Docker engines in `conf/docker.yaml`, not just its own.** This is not an
|
||
inference; `GET /api/services` returns every card's `server` field, and the
|
||
2026-08-24 snapshot resolves to:
|
||
|
||
| `server` | host | label-discovered services |
|
||
|---|---|---|
|
||
| `ana-pfi-docker` | 10.250.50.70 | 30 |
|
||
| `irv-ml1-docker` | 10.100.79.3 (over WireGuard) | 15 |
|
||
| `ana-ml2-docker` | 10.250.50.54 | 14 |
|
||
| `esh-vm-docker` | 10.0.50.45 (the dashboard's own host) | 13 |
|
||
| `nh3-pfi-docker` | 10.100.50.40 | 2 |
|
||
|
||
**74 of 107 cards are label-discovered, and only 13 of those come from the
|
||
dashboard's own engine** — the other 61 are read off four remote hosts,
|
||
including irv-ml1 across the WireGuard tunnel. The remaining 33 carry
|
||
`server: null`: those are the manual `services.yaml` entries — hardware, BMCs,
|
||
hypervisors, printers, and user-level systemd services that have no container
|
||
to label in the first place. **That null is the only thing "not label-driven"
|
||
about this dashboard**, and it is a property of the entry, not of the host it
|
||
points at.
|
||
|
||
⚠ If a future session doubts this again, the check is one command and takes two
|
||
seconds — do not reason about it from the docs:
|
||
|
||
```bash
|
||
curl -s http://10.0.50.45:5100/api/services \
|
||
| jq -r '.[].services[] | .server' | sort | uniq -c
|
||
```
|
||
|
||
## What was actually wrong with the layout
|
||
|
||
Measured with Playwright against the live board (per-group `card=` width, card
|
||
height spread, and a geometric title-vs-status overlap test), not judged by
|
||
eye:
|
||
|
||
- **Card width changed at every group boundary.** `columns:` is not a density
|
||
dial — it sets `lg:grid-cols-N` for one group, so it fixes that group's card
|
||
width. Notes rendered a single **1464px** card; News and Media **728px**;
|
||
Eval & Retrieval **286px**; everything else 360px. Scrolling the page, the
|
||
grid resized five times.
|
||
- **Long names printed underneath their own status pill.** Measured by
|
||
re-injecting the old rule and testing the title text node's box against the
|
||
status cluster's box: **6 cards, all on the AI tab** — 3 in Inference, 2 in
|
||
Dormant, 1 in Eval & Retrieval; zero on the other three tabs, which is why
|
||
it survived earlier passes. Root cause is a genuinely counter-intuitive one:
|
||
the rule reserved a
|
||
78px gutter with `padding-right` and relied on `overflow: hidden` to hold it,
|
||
but **overflow clips at the padding box, not the content box** — so the
|
||
reserved gutter was spill room the title rendered straight through. The
|
||
intended `text-overflow: ellipsis` never fired either, because the ellipsis
|
||
is painted by whichever block's own line overflows, and here that is the
|
||
anonymous box around the bare title text node, which does not carry
|
||
`overflow`.
|
||
- **`AI Systems` / Scriberr was on all four tabs** — the 2026-08-18 UltraSeedbox
|
||
bug recurring, this time arriving from a container label rather than from
|
||
`settings.yaml`.
|
||
- **Icons were grey smudges.** Homepage masks every glyph over
|
||
`--color-logo-start/stop`; stock slate-400 → slate-700 sinks the bottom half
|
||
of each icon into the card fill.
|
||
- Bookmark groups and Jellyfin's trailing stream rows were the two components
|
||
the theme had never reached.
|
||
|
||
## The fixes
|
||
|
||
`stacks/homepage/conf/settings.yaml` — **all 20 groups to `columns: 4`.**
|
||
`stacks/homepage/theme/australis.css.in` → rebuilt → `conf/custom.css`:
|
||
gutter held by wrapping, description clamped to 3 lines (floor still 2), icon
|
||
ramp overridden, bookmark + trailing-widget components themed, group gap
|
||
10px → 22px. `stacks/scriberr/compose.yaml` — `homepage.group` → `AI - Audio
|
||
Tools`, container recreated on ana-ml2.
|
||
|
||
After: **every group renders at card=360**, and the collision count is zero.
|
||
|
||
Before/after, all four tabs: `http://10.100.10.50:8090/b/homepage-relayout/`
|
||
(24h TTL; also on the standing link board).
|
||
|
||
## ⚠ Three traps worth carrying forward
|
||
|
||
1. **"Columns = member count" is RETIRED** (it was the 2026-08-18 rule). It was
|
||
avoiding dead cells in a short last row and bought a worse defect. A short
|
||
last row is what a grid looks like; a card wider than its neighbours is what
|
||
a mistake looks like.
|
||
2. **A `:root` override of a Homepage theme variable is silently ignored.**
|
||
Homepage sets `--color-logo-*` on `.theme-slate`, and that class is on the
|
||
`<html>` element — the same element `:root` matches. `.theme-slate` (0,1,0)
|
||
beats `:root` (0,0,1), so the override does nothing and looks like the
|
||
variable is not the one in play. `html[class]` (0,1,1) wins, and does not
|
||
hard-code which `theme-*` class is active. Specificity alone is not enough
|
||
either: a custom property resolves from the *nearest* ancestor that sets it,
|
||
so the override has to land on `<html>`, not on `<body>`.
|
||
3. **The post-recreate tab-bar loss is INTERMITTENT, not guaranteed.** The
|
||
2026-08-19 note reads as though every recreate costs up to an hour of broken
|
||
render. This recreate came up correct within 10 seconds — fresh payload on
|
||
the first poll, all four tabs clickable a minute later. Recreate, *check*,
|
||
and only then walk away if it is actually in the broken state.
|
||
|
||
Also re-confirmed, since the change depended on it: **a `settings.yaml` edit
|
||
needs a container recreate, not a restart.** `docker restart homepage` left the
|
||
old `"columns":1` payload embedded in the served HTML with the correct file
|
||
mounted and readable inside the container; `compose up -d --force-recreate`
|
||
cleared it immediately.
|
||
|
||
## Deliberately not done — operator's call
|
||
|
||
The Main tab still opens on three sparse bands: **Notes** (1 member) and
|
||
**Games** (1) each burn a full 4-wide row, and **News** has 2. Merging Notes +
|
||
News, or folding Games into Apps, would tighten the top of the page — but that
|
||
is information architecture, not layout, and the group names are the operator's.
|
||
Surfaced rather than done.
|
||
|
||
→ **Resolved in pass 2 below**, where the operator delegated the naming
|
||
("re-categorize however you want"). Notes + News became `Daily`, Games folded
|
||
into `Apps`, and the `AI - Audio Tools` placement in this pass was superseded
|
||
(Scriberr moved on to `AI - Studios`).
|
||
|
||
---
|
||
|
||
# `[2026-08-24, pass 2]` Recategorised on "do I open this?", API groups collapsed
|
||
|
||
Operator, after seeing pass 1: *"You can re-categorize however you want.
|
||
service networking tab is uneven, you can split out the adguard cards, etc.
|
||
most of the issues are that tools I use and have a UI are interspersed with API
|
||
endpoints which are largely informational only. They might even go in their own
|
||
cards or start collapsed."*
|
||
|
||
## The axis
|
||
|
||
Every group is now either **tools** (expanded, top of tab) or **endpoints** (an
|
||
API, a broker, an agent — `initiallyCollapsed: true`, bottom of tab). A
|
||
collapsed group still renders its eyebrow and rule, so presence costs one line
|
||
instead of two rows.
|
||
|
||
Second, quieter rule that fell out of the same pass: **a group's members should
|
||
all carry a widget or none should.** A stat strip adds ~50px, so one widget card
|
||
in a row of plain ones opens a void under the plain ones — which is most of what
|
||
made the 13-member `Service Networking` band look broken.
|
||
|
||
## Shape
|
||
|
||
- **Main** — `Daily` (Memos, Miniflux, Nevermore, SearXNG — replaces the
|
||
1-card Notes and 2-card News bands), `Monitoring`, `Apps` (12; absorbed the
|
||
1-card `Games` band), `Media`, `UltraSeedbox`.
|
||
- **AI** — `AI - Gateways & Chat` (8) and `AI - Studios` (6) expanded; then
|
||
`AI - Inference` (7), `AI - Eval & Retrieval` (4), `AI - Speech (TTS)` (4),
|
||
`AI - Audio Tools` (2), `AI - Dormant` (6) all collapsed.
|
||
- **Toolchain** — `DNS & Filtering` (3), `Reverse Proxies` (2),
|
||
`Compose Consoles` (5), `Toolchain` (3), `Agents (no UI)` (6, collapsed).
|
||
- **Infrastructure** — unchanged; every card there is already a console.
|
||
|
||
Measured after: every group `card=360`, and `DNS & Filtering` and
|
||
`Reverse Proxies` both `h=134..134` — dead flush.
|
||
|
||
## ⚠ The move that made it affordable
|
||
|
||
**The sixteen GPU-backed model seats were NOT relabelled.** `homepage.group` is
|
||
read at container **creation**, so renaming `AI - Inference` to something
|
||
clearer would have meant recreating six vLLM seats plus four eval seats plus
|
||
four TTS engines — multi-minute model reloads on endpoints peers reach through
|
||
the gateway. Order plus `initiallyCollapsed` buys the same separation for free,
|
||
so the names stay ugly on purpose. **Do not spend that recreate on a label.**
|
||
|
||
28 containers *were* relabelled — all cheap web services — via five rerunnable
|
||
elway playbooks, `playbooks/homepage-regroup-<host>.yaml`. The canonical
|
||
`stacks/` tree was synced to match afterwards, so intent and reality agree.
|
||
|
||
`initiallyCollapsed: true` is a per-group key in `layout:`; confirmed present in
|
||
this build (`defaultOpen: !(group?.initiallyCollapsed ?? global)` in
|
||
`/app/.next/server/pages/index.js`).
|
||
|
||
## AdGuard (ANA) gained its widget, and the credential is fleet-wide
|
||
|
||
It was the only AdGuard without a query/blocked/latency strip, so it sat short
|
||
beside two tall siblings. **One `infra-ops` AdGuard login authenticates against
|
||
all three instances** (ANA `:8053`, NH3 `:8080`, ESH `:8080` — all returned 200
|
||
on `POST /control/login`, verified 2026-08-24). Vaulted at
|
||
`secret get nh3-dev/adguard-infra-ops-password`; written to
|
||
`/opt/docker/compose/adguard-ana/.env` (0600, root) and never into git. Its icon
|
||
was also the odd one out (`mdi-dns` against two `si-adguard`).
|
||
|
||
## ⚠⚠ `initialSettings":{}` — the tab-bar mystery is a SWALLOWED EXCEPTION
|
||
|
||
The biggest durable finding of the day, and it cost ~25 minutes. Full write-up
|
||
in `stacks/homepage/README.md`; the short version:
|
||
|
||
`initialSettings":{}` in the served HTML is **the catch branch** of the page's
|
||
data loader, not a warm-up and not a cache. And the error can vanish without
|
||
trace: the logger is assigned as the first statement *inside* the same `try`,
|
||
and the `catch` only logs `if (logger)`. If the logger is what threw, nothing is
|
||
written anywhere — which is exactly what was observed.
|
||
|
||
Ruled out by measurement, do not re-run: `/api/services`, `/api/bookmarks`,
|
||
`/api/widgets` and `/api/hash` all return **200 with correct content** while the
|
||
page serves `{}`; restoring the previous known-good `settings.yaml` reproduces
|
||
it identically; `/api/validate` returns `[]`; disk and permissions are fine.
|
||
|
||
**One-command test:**
|
||
`curl -s http://10.0.50.45:5100/ | grep -o 'initialSettings":[^,]\{0,20\}'`
|
||
|
||
**What broke the streak:** three consecutive recreates came up empty, then
|
||
rolling the 8.6 MB `conf/homepage/logs/homepage.log` aside and recreating healed
|
||
it within 15 seconds. That is one observation, not proof — but it is a coherent
|
||
mechanism (oversized log → logger init throws → silent catch) and it is the
|
||
cheapest thing to try first next time.
|
||
|
||
---
|
||
|
||
# `[2026-08-24, pass 3]` Rebuilt on Australis Skyfall — dual theme, light shipped
|
||
|
||
Operator supplied the Skyfall design-system README and said "Go full with
|
||
skyfall."
|
||
|
||
## The bundle was already in this repo's git history
|
||
|
||
**The Skyfall tokens did not need to be hunted down.** A predecessor vendored
|
||
them on 2026-08-19 and a later commit deleted them; git kept everything:
|
||
|
||
```bash
|
||
git show 45c1995:stacks/homepage/theme/colors.css # 208 lines, BOTH themes
|
||
git show 45c1995:stacks/homepage/theme/layout.css # calm-depth tokens
|
||
git show 45c1995:stacks/homepage/theme/typography.css
|
||
git show 45c1995:stacks/homepage/theme/fonts/Supreme-{400,500,700}.woff2
|
||
```
|
||
|
||
`colors.css` carries `:root` (dark) **and** `[data-theme="light"]` (Skyfall
|
||
Day) — so the light ramp is canonical, not derived. That killed the entire
|
||
objection from the previous answer, which was correct only about the
|
||
`australis-design` skill ("Always dark first. No light mode in this system").
|
||
**Skyfall is the dual-theme derivative; australis-design is the terminal
|
||
theme. They are different systems and only one of them has a light mode.**
|
||
|
||
## ⚠⚠ REMOVING `theme:` FROM settings.yaml BREAKS THE DASHBOARD
|
||
|
||
The documented way to get Homepage's own light/dark toggle is to leave `theme:`
|
||
unpinned. **Do not.** With the key absent, the page's data loader throws and its
|
||
catch branch serves `initialSettings: {}` — no tab bar, no layout, no i18n.
|
||
|
||
Measured, not inferred: six force-recreates over seven minutes all came up
|
||
empty with the key removed; restoring `theme: dark` rendered correctly on the
|
||
next recreate in **12 seconds**. `/api/services` stays 200 and fully correct
|
||
throughout, which is exactly why this reads as a caching or warm-up problem and
|
||
is not one.
|
||
|
||
This is the first *confirmed* trigger for the long-running "tab bar goes
|
||
missing" mystery. It does not explain every occurrence (the symptom has
|
||
appeared with `theme:` present), but it means **the first diagnostic step is
|
||
now `git log -p -- stacks/homepage/conf/settings.yaml`**, not container
|
||
archaeology. Also retires an earlier lead from this same session: rolling the
|
||
8.6 MB `homepage.log` aside once coincided with a recovery, but did nothing
|
||
during the `theme:`-key episode — coincidence, not cause.
|
||
|
||
## So the toggle is ours
|
||
|
||
`conf/custom.js` renders it (was an empty placeholder). Precedence:
|
||
|
||
1. explicit choice — `localStorage['skyfall-theme']`, written by the toggle;
|
||
2. OS preference — `@media (prefers-color-scheme: light)`;
|
||
3. dark — Skyfall's default.
|
||
|
||
`theme/build.py` re-emits each vendored `[data-theme="light"]` block twice: as
|
||
`[data-theme="light"], html.light`, and inside the media query scoped to
|
||
`html:not([data-theme="dark"]):not([data-theme="light"])`. **That `:not()` pair
|
||
is what lets a stored *dark* choice survive a light-mode OS.** Verified across
|
||
both OS preferences: load, click, click again, reload — all four correct.
|
||
|
||
⚠ Homepage keeps its own `class="dark scheme-dark theme-slate"` on `<html>`
|
||
regardless, because `theme:` is pinned. That is fine and was checked
|
||
explicitly: with the dark class present AND `data-theme="light"`, every themed
|
||
surface resolves to Skyfall Day, because our rules carry `!important` on the
|
||
surfaces Tailwind's `dark:` variants would otherwise claim. **`data-theme` is
|
||
the control surface; the class is not.**
|
||
|
||
## The anti-fork guard is now mechanical
|
||
|
||
`build.py` records the SHA-256 of each vendored file and **fails the build** on
|
||
a mismatch, rather than warning. A vendored file is either byte-identical to
|
||
the bundle or it is a fork wearing the bundle's name. Overrides go in
|
||
`skyfall.css.in`, which is written entirely against the semantic layer
|
||
(`--surface-*`, `--text-*`, `--border-*`, `--success/--danger/--warning`) — no
|
||
raw family tokens, no colour literals.
|
||
|
||
The one place a literal is unavoidable: Homepage consumes
|
||
`--color-logo-start/stop` as `rgb(var(--x))`, which cannot take an `oklch()`.
|
||
Those four values are exact sRGB conversions of real tokens (`--sea-80`,
|
||
`--blue-base` for dark; `--sea-40`, `--blue-deep` for light), computed rather
|
||
than eyeballed, with the conversion recorded in the file.
|
||
|
||
## Deviations, all deliberate and all written down
|
||
|
||
- **The aurora ribbon under the tab bar is gone.** Skyfall sanctions exactly two
|
||
accent expressions — the active rail and hero-only glows — and a decorative
|
||
gradient across the chrome is neither. The colour moved to a 2px accent bar
|
||
plus `--accent-soft` fill on the active tab, which *is* the rail.
|
||
- **Widget stat values moved from the display face to mono**, per Skyfall's
|
||
"numbers and telemetry are always `--font-mono`".
|
||
- **Two font substitutions**: Space Grotesk for Bespoke Sans, JetBrains Mono
|
||
for Victor Mono. Only Supreme was ever vendored, and Skyfall's own notes call
|
||
Victor Mono "user-supplied". Two-line swap when the real faces arrive.
|
||
|
||
Dark + light, all four tabs: `http://10.100.10.50:8090/b/homepage-skyfall/`
|
||
_Archived 2026-09-09._
|
||
|
||
- `[2026-08-24]` **Homepage reorganised on the axis "do I open this?" — UI groups expanded on top, API/agent groups collapsed at the bottom** (operator-delegated: "re-categorize however you want"). Load-bearing constraint: `homepage.group` is read at container CREATION, so the 16 GPU-backed model seats keep their unlovely names rather than eat a recreate — `initiallyCollapsed` + order is free. Second rule discovered here: **group members should all have widgets or none should**, because a stat strip adds ~50px and opens a void beside plain cards. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md`
|
||
_Archived 2026-09-09._
|
||
|
||
- `[2026-08-24]` **Homepage columns unified at 4 for every group; the 2026-08-18 "columns = member count" rule is retired.** It was avoiding dead cells in a short last row and bought a worse defect — card width changing at every group boundary. Also carries two CSS traps: `overflow: hidden` clips at the PADDING box (so a `padding-right` gutter is spill room, not a guard), and a `:root` override of a Homepage theme variable is silently outranked by `.theme-slate` on the same `<html>` element. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md`
|
||
_Archived 2026-09-09._
|
||
|
||
- `[2026-08-24]` **AES-128 adopted on both Anaheim tunnels; the per-flow ceiling root-caused to the UDM's software AES-CBC, exonerating the FortiGate.** Proven by an A/B/A cipher swap at identical CPU — hardware offload is not cipher-cost-sensitive.
|
||
# [2026-08-23] Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gbps circuit
|
||
|
||
> **⛔ SUPERSEDED 2026-08-23 (same day, later session) — read the CORRECTION at
|
||
> the bottom before acting on anything here.** The headline is wrong (the
|
||
> relevant ceiling is NH3's **1 Gbps** uplink, not Anaheim's 2 Gbps), the
|
||
> aggregate number is wrong (**692 Mbit/s** at 8 streams, not ~550 — the
|
||
> original stopped measuring at 4), and the proposed remedy is **impossible**:
|
||
> UniFi's manual site-to-site IPsec does not implement AES-GCM at all. The
|
||
> per-stream observation and the parallelise-your-transfers mitigation are the
|
||
> parts that survive.
|
||
|
||
The operator noticed site-to-site transfers were slow for a datacenter fiber
|
||
handoff and asked whether WireGuard was the limit. It is not WireGuard, and the
|
||
circuit is fine.
|
||
|
||
## Measured
|
||
|
||
```
|
||
ana-docker -> internet, 8 parallel 2,153 Mbit/s <- the 2 Gbps handoff, delivering
|
||
ANA <-> NH3 through the tunnel, 4 par. 460 Mbit/s
|
||
FortiGate's own recorded peak 554 Mbit/s
|
||
ANA <-> NH3, single stream 227 Mbit/s
|
||
ANA <-> ESH, single stream 249-265 Mbit/s
|
||
ESH <-> NH3 (never touches ana-gw) 545-557 Mbit/s on a SINGLE stream
|
||
```
|
||
|
||
Method: stdlib TCP probe (no ssh, no crypto, no compression) between site
|
||
endpoints; raw circuit measured with 8 parallel HTTPS fetches from Hetzner
|
||
Ashburn. Host NICs are virtio with no reported cap, so no host-side ceiling.
|
||
|
||
## What it is not
|
||
|
||
- **Not WireGuard.** Both Anaheim tunnels are IPsec on ana-gw
|
||
(`pfi-ana-nh3` -> 70.230.226.88, `ana-eshudm-dyn` -> the ESH UDM). WireGuard
|
||
on ana-wg is remote-access only and is not in this path. Traceroute confirms:
|
||
both slow paths have hop 1 = `10.250.50.1` (the FortiGate); the fast
|
||
ESH<->NH3 path rides a `192.168.x` Site Magic overlay and never touches it.
|
||
- **Not CPU or crypto exhaustion.** FortiGate CPU was **100% idle across all
|
||
8 cores** during the tests, and both live tunnels report `npu_flag=03` with
|
||
`dec_npuid=1 enc_npuid=1` — encrypt *and* decrypt are hardware-offloaded.
|
||
- **Not a 250 Mbit/s cap.** That was the first number and it is misleading —
|
||
single-stream TCP. Four parallel streams doubled it. Quote the aggregate.
|
||
- **Not the interface.** wan1: `rxe=0 txe=0 rxd=0 txd=0`, no collisions.
|
||
|
||
## Most likely cause
|
||
|
||
Both tunnels negotiate **`aes256-sha1`** in phase 1 *and* phase 2 (dhgrp 14,
|
||
IKEv2). AES-CBC + SHA1 is a two-pass operation; FortiGate NPUs are markedly
|
||
faster on **AES-GCM**, which combines encryption and authentication in one
|
||
pass. The datasheet IPsec headline for an 80F assumes GCM with large packets,
|
||
not CBC+SHA1 at the 1438-byte tunnel MTU this link negotiates. The ~4x
|
||
shortfall is consistent with that.
|
||
|
||
## Not executed
|
||
|
||
Changing the proposal is a **production-edge change requiring a matching
|
||
change at the far end** (NH3 UDM and the ESH UDM), and each tunnel drops while
|
||
it renegotiates. Left for the operator. See the index entry for authorization
|
||
state.
|
||
|
||
## Immediate mitigation, no config change
|
||
|
||
Per-flow is the weak axis: a single stream over Site Magic gets 557 Mbit/s, a
|
||
single stream through IPsec gets 227. **Anything moving bulk data across the
|
||
Anaheim link should parallelise** — that alone roughly doubles throughput
|
||
today.
|
||
|
||
## Practical consequence already observed
|
||
|
||
`/mnt/smithy` mounted on ana-ml2 reads at 24.7 MB/s sequential vs 98.3 MB/s
|
||
from nh3-dev (same file, same mount) — that gap *is* this tunnel, not NFS and
|
||
not the NAS. See [[2026-08-23-smithy-mount-ana-ml2]].
|
||
|
||
## Access note
|
||
|
||
ana-gw is a FortiGate-80F, FortiOS 7.2.10, at 10.250.0.1. `sshpass` is absent
|
||
on nh3-dev; connect with paramiko via `uv run --with paramiko`. Password is
|
||
vaulted at `fortigate/ana-gw-infra-ops-password`. **`diagnose vpn tunnel list`
|
||
prints live ESP session keys** — never paste its output into althing, a
|
||
booth, or a commit.
|
||
|
||
---
|
||
|
||
## CORRECTION (2026-08-23, later session): the cutover was attempted and the remedy does not exist
|
||
|
||
The operator authorised the AES-GCM cutover, NH3 side first. It cannot be done,
|
||
and the measurements taken while trying show there is very little left to win.
|
||
|
||
### AES-GCM is unavailable on the far end — not a naming problem
|
||
|
||
The NH3 edge is a **UDM Pro SE** terminating `pfi-nh3-ana` (networkconf
|
||
`_id 697d64414c85dd2b6669b00a`, `ifname vti64`). Its UniFi API **validates** the
|
||
crypto enum and rejected every GCM spelling tried — `aes256gcm`, `aes256gcm128`,
|
||
`aes256gcm16`, `aes-256-gcm`, `aes256-gcm`, `aes256gcm12`, `gcm`, `aes128gcm128`
|
||
— all `HTTP 400 api.err.InvalidPayload`, nothing applied.
|
||
|
||
**The control that makes this conclusive:** the *identical* request body with
|
||
`ipsec_esp_encryption: "aes256"` returns `HTTP 200 rc:ok`. So the 400s are the
|
||
enum rejecting the value, not a malformed body. Corroborating: **zero
|
||
case-insensitive `gcm` matches across 7.3 MB of UniFi OS UI bundles.**
|
||
|
||
Accepted enum (probed): `aes128`, `aes192`, `aes256`, `3des` → 200; `des`,
|
||
`chacha20poly1305` → 400. There is no AEAD option. Both Anaheim tunnels land on
|
||
UniFi far ends, so this blocks the ESH tunnel too.
|
||
|
||
The FortiGate side **was** widened and is GCM-capable: phase2 `pfi-ana-nh3` now
|
||
reads `set proposal aes256-sha1 aes256gcm`. Left in place deliberately — it is
|
||
functionally identical while the peer only offers CBC, and reverting it would
|
||
cost another SA renegotiation for a cosmetic gain. Phase 1 was never touched;
|
||
IKE protects the control channel only and has no bearing on data throughput.
|
||
|
||
### The numbers that retire this as a problem
|
||
|
||
Measured NH3→ANA through the tunnel, and NH3→ESH over Site Magic (WireGuard) on
|
||
the same UDM and the same uplink, with the same stdlib TCP probe:
|
||
|
||
| streams | IPsec NH3→ANA | WireGuard NH3→ESH |
|
||
|---|---|---|
|
||
| 1 | 245 Mbit/s | 557 Mbit/s |
|
||
| 4 | 471 Mbit/s | 767 Mbit/s |
|
||
| 8 | **692 Mbit/s** | **795 Mbit/s** |
|
||
|
||
**NH3's WAN is a 1 Gbps link** (`uplink.speed = 1000`, port capable of 10G) —
|
||
that, not Anaheim's 2 Gbps, is the ceiling for anything crossing this tunnel.
|
||
So the tunnel does **~69% of the achievable uplink** at 8 streams, and the
|
||
IPsec-vs-WireGuard gap collapses from 2.3× at one stream to **15% at eight**.
|
||
|
||
Re-architecting the transport (site-to-site WireGuard via `ana-wg`, since
|
||
FortiOS has no WireGuard) would chase that last 15%. Not worth it.
|
||
|
||
### What the constraint actually is
|
||
|
||
A **per-stream** limit (~245 Mbit/s), not an aggregate crypto ceiling. Both
|
||
endpoints are idle at load — FortiGate CPU 100% idle with `npu_flag=03`
|
||
(offloaded both directions), UDM CPU ~7% with load1 moving 0.70 → 1.55. The
|
||
shape is per-SA/per-flow serialisation, and WireGuard shows the same shape from
|
||
a higher floor (557 → 795 is only 1.43× scaling).
|
||
|
||
### Actionable consequence
|
||
|
||
Anything moving bulk data across this link should **parallelise** — 245 → 692
|
||
Mbit/s, a 2.8× win with no config change. For single-stream workloads that
|
||
cannot be parallelised at the application layer, **NFS `nconnect=N` is the
|
||
lever**: it opens N TCP connections per mount, converting a single-stream
|
||
workload into a parallel one. The `/mnt/smithy` mount on ana-ml2 reading at
|
||
24.7 MB/s (~200 Mbit/s, i.e. exactly the single-stream ceiling) is the live
|
||
example — remounting with `nconnect=8` is the obvious test.
|
||
|
||
### Foot-gun recorded
|
||
|
||
Probing the enum by PUTting candidate values **applies the accepted ones**. A
|
||
probe loop here timed out with `3des` briefly live on the NH3 side, which the
|
||
FortiGate would not accept — a short tunnel outage until `aes256` was restored
|
||
(~1 minute, confirmed by the SA counters resetting). If you enumerate a UniFi
|
||
config enum this way, restore the known-good value after **every** 200, not at
|
||
the end of the loop. Post-change verification: the UDM object was diffed
|
||
field-by-field against its pre-change snapshot and is **byte-identical**.
|
||
|
||
---
|
||
|
||
## FOLLOW-UP (2026-08-23): what the per-stream limit actually is
|
||
|
||
The correction above called the constraint "per-SA/per-flow serialisation".
|
||
That was a hand-wave. Measured properly, it is a **hard per-flow rate cap of
|
||
~230–245 Mbit/s with a very deep buffer in front of it** — not a tuning
|
||
problem, not loss, not window size.
|
||
|
||
### The evidence: pin the send buffer and sweep it
|
||
|
||
Single stream NH3 → ana-docker, `SO_SNDBUF` pinned, `ss -ti` sampled in flight:
|
||
|
||
| in-flight cap | throughput | RTT in flight | minRTT | retrans |
|
||
|---|---|---|---|---|
|
||
| 256 KB | 224 Mbit/s | 7.8 ms | 5.3 ms | 0 |
|
||
| 416 KB | 225 Mbit/s | 11.8 ms | 6.6 ms | 0 |
|
||
| 416 KB | 245–247 Mbit/s | 12.0 ms | 5.6 ms | 0 |
|
||
| ~3.3 MB (autotuned) | 245 Mbit/s | **107 ms** | 5.5 ms | 0 |
|
||
|
||
**Throughput is flat across a 13× range of in-flight data while RTT scales with
|
||
it.** That is the signature of a fixed service rate with a standing queue: the
|
||
window controls only how much queue you build, never how fast you go. Had this
|
||
been window-limited, throughput would have risen with the buffer. Had it been
|
||
congestion, there would be retransmits — there are essentially none
|
||
(`retrans:0`, 0% ping loss).
|
||
|
||
So `net.ipv4.tcp_*` tuning, window scaling and congestion-control choice are all
|
||
**red herrings here**. Do not go there.
|
||
|
||
### Bufferbloat: one bulk stream wrecks latency for everything else
|
||
|
||
Measured on the same tunnel, ping to ana-docker:
|
||
|
||
- idle: **6.9 ms** avg
|
||
- during a **single** bulk TCP stream: **102 ms** avg, 136 ms max, 0% loss
|
||
|
||
**15× latency inflation from one transfer.** This is the operationally
|
||
important finding — any interactive traffic sharing the Anaheim link (ssh,
|
||
RDP, althing, VoIP) degrades badly whenever anything moves bulk data, and it
|
||
takes only one stream to do it. Parallelising transfers makes throughput
|
||
better and this *worse*. If it starts biting, the fix is an AQM/shaper on the
|
||
tunnel (or rate-limiting bulk jobs), not more buffer.
|
||
|
||
### Where the cap lives — strong inference, not proof
|
||
|
||
Three paths, and the FortiGate is the only variable:
|
||
|
||
| path | single-stream |
|
||
|---|---|
|
||
| FortiGate ↔ NH3 UDM (IPsec) | 245 Mbit/s |
|
||
| FortiGate ↔ ESH UDM (IPsec) | 249–265 Mbit/s |
|
||
| NH3 UDM ↔ ESH UDM (WireGuard, **no FortiGate**) | 557 Mbit/s |
|
||
|
||
Present in both slow paths, absent from the fast one. Aggregate over the same
|
||
SA reaches 692 Mbit/s, so it cannot be the SA or the crypto engine as a whole —
|
||
many flows spread out fine, one flow does not.
|
||
|
||
The mechanism that fits is **FortiGate NPU IPsec offload being per-session**:
|
||
each firewall session is bound to one crypto engine, so a single TCP flow is
|
||
capped at one engine's rate while many sessions spread across engines. **This
|
||
is inference from the throughput shape, not something confirmed on the box** —
|
||
`diagnose sys session list` was not captured for a TCP flow (the filter caught
|
||
only traceroute UDP probes). A single-stream control through ana-gw *without*
|
||
IPsec returned 290 Mbit/s to Hetzner Ashburn, but at ~60 ms RTT that is
|
||
window-limited and does not discriminate. **If this matters, the clean test is
|
||
a non-IPsec single stream between two Anaheim VLANs at low RTT.**
|
||
|
||
**Relevant to the FortiGate cutover decision:** if the per-flow cap is the
|
||
FortiGate's IPsec path, replacing the box plausibly lifts single-stream
|
||
throughput toward the WireGuard figure. That is a point in favour of the
|
||
cutover, and it is cheap to verify afterwards by re-running the sweep.
|
||
|
||
---
|
||
|
||
## FOLLOW-UP 2 (2026-08-23): it is NOT a capacity problem, and it IS specific to IPsec
|
||
|
||
Operator asked directly whether the 80F "can't handle the traffic". It can.
|
||
Two new measurements settle the shape of this, and correct an overstatement in
|
||
FOLLOW-UP 1 (which pointed at the FortiGate on evidence that was confounded —
|
||
every slow path was *both* IPsec *and* FortiGate, so protocol and box could not
|
||
be separated by that argument).
|
||
|
||
### The 80F routes a single flow at line rate when IPsec is not involved
|
||
|
||
`ana-ml2 → pfi-pve`, inter-VLAN **through** ana-gw (traceroute hop 1 =
|
||
`10.250.50.1`), 0.36 ms RTT, no tunnel:
|
||
|
||
| streams | throughput |
|
||
|---|---|
|
||
| 1 | **940.2 Mbit/s** |
|
||
| 8 | 939.3 Mbit/s |
|
||
|
||
Single stream saturates 1 GbE. So the box does **not** cap single sessions in
|
||
general, and there is no per-session ceiling in its plain forwarding path. The
|
||
~250 Mbit/s per-flow cap is **specific to the IPsec datapath**.
|
||
|
||
### Both IPsec tunnels converge on the same numbers despite different far ends
|
||
|
||
Measured today with the same probe:
|
||
|
||
| tunnel | far-end gateway | RTT | 1 stream | 8 streams |
|
||
|---|---|---|---|---|
|
||
| NH3 ↔ ANA | UDM Pro **SE** | 6.7 ms | 245 Mbit/s | 692 Mbit/s |
|
||
| ESH ↔ ANA | UDM Pro **Max** | 3.9 ms | **268 Mbit/s** | **715 Mbit/s** |
|
||
|
||
Different gateway hardware, different sites, different uplinks, and RTT
|
||
differing by 1.7× — yet single-stream differs by only 9%. **If this were
|
||
window-limited the 3.9 ms path would be ~1.7× faster.** It is not, which is
|
||
independent confirmation of a rate cap rather than a BDP effect.
|
||
|
||
### Capacity summary — the box has headroom it will not give one flow
|
||
|
||
- plain routing, 1 stream: **940 Mbit/s** (line rate)
|
||
- plain routing to internet, 8 streams: **2,153 Mbit/s**
|
||
- IPsec, 8 streams: **692–715 Mbit/s**
|
||
- IPsec, 1 stream: **245–268 Mbit/s**
|
||
- CPU **100% idle** throughout; IPsec NPU-offloaded (`npu_flag=03`)
|
||
|
||
Within a single SA, 8 sessions get ~2.9× what 1 session gets, so the datapath
|
||
distributes work **by inner session** — consistent with IPsec offload binding a
|
||
session to one crypto engine.
|
||
|
||
### What is still NOT separated
|
||
|
||
Whether the cap belongs to **the 80F's IPsec offload** or to **UniFi's IPsec
|
||
implementation**. Both tunnels have a UDM at the far end, and both UDMs run the
|
||
same UniFi firmware, so identical caps are explainable either way. The Pro Max
|
||
being only 9% faster than the Pro SE argues against the UniFi side (a beefier
|
||
CPU should show more), but that is suggestive, not conclusive.
|
||
|
||
**The test that closes it:** an IPsec tunnel whose endpoints do not include the
|
||
80F — e.g. a temporary UDM↔UDM IPsec tunnel between NH3 and ESH, measured
|
||
single-stream. If it also caps ~250, the FortiGate is exonerated and replacing
|
||
it buys nothing on this axis. If it runs near the 557 Mbit/s that UDM↔UDM
|
||
WireGuard achieves, the 80F is the limiter. **Bears directly on the pending
|
||
FortiGate cutover** — worth running before that decision, not after.
|
||
|
||
---
|
||
|
||
## FOLLOW-UP 3 (2026-08-23): WireGuard over the same internet path does 767 Mbit/s on ONE stream
|
||
|
||
Operator asked for a WireGuard test from `ana-wg` to NH3 over the public
|
||
internet. It is the test that separates the *path* from the *crypto*, and the
|
||
answer is unambiguous. **It also overturns FOLLOW-UP 1's "re-architecting the
|
||
transport is not worth it" — that conclusion compared 8-stream numbers and was
|
||
wrong for single-stream workloads.**
|
||
|
||
### Setup (fully torn down afterwards)
|
||
|
||
`ana-wg` (10.250.50.252, Debian 12 LXC, 4 cores) already has an
|
||
internet-reachable WireGuard endpoint: wg0 on **UDP 31337**, published by
|
||
FortiGate VIP `wg-to-ana-wg` (extip **38.120.12.42** → 10.250.50.252:31337,
|
||
policy 46, service `WireGuard-LEET`). **No FortiGate change was needed.** A
|
||
temporary `wgt0` was created on nh3-dev (10.30.10.200/32) as a fourth peer on
|
||
wg0, measured, then removed — ana-wg is back to its original 3 peers and the
|
||
keys were shredded. `wireguard-tools` was installed on nh3-dev and **left in
|
||
place** (benign, and wanted if this becomes permanent).
|
||
|
||
In this topology **neither gateway does crypto**: the FortiGate and the NH3 UDM
|
||
only NAT/forward UDP, and Linux does WireGuard at both ends.
|
||
|
||
### The full comparison
|
||
|
||
| path | crypto performed by | 1 stream | 8 streams |
|
||
|---|---|---|---|
|
||
| IPsec NH3↔ANA | FortiGate + UDM | 245 Mbit/s | 692 Mbit/s |
|
||
| IPsec ESH↔ANA | FortiGate + UDM | 268 Mbit/s | 715 Mbit/s |
|
||
| **WireGuard NH3→ana-wg** (same internet path) | **Linux + Linux** | **767 Mbit/s** | 763 Mbit/s |
|
||
| WireGuard NH3↔ESH (Site Magic) | UDM + UDM | 557 Mbit/s | 795 Mbit/s |
|
||
| plain routing through the 80F (inter-VLAN) | none | 940 Mbit/s | 939 Mbit/s |
|
||
|
||
**One stream equals eight streams over Linux WireGuard (767 ≈ 763).** There is
|
||
no per-flow penalty at all, and a single flow already saturates the path. So
|
||
the ~245 Mbit/s per-flow cap is **not** the ISP, not the circuit, not the NH3
|
||
uplink and not the physical path — all of which sustain 767 on one flow.
|
||
|
||
Per-flow penalty ranks by implementation:
|
||
|
||
- **Linux WireGuard — none** (767 → 763, flat)
|
||
- **UDM WireGuard — mild**, ~1.4× (557 → 795)
|
||
- **IPsec on this pair — severe**, ~2.8× (245 → 692)
|
||
|
||
### Latency under load — the same story
|
||
|
||
| path | idle | during ONE bulk stream |
|
||
|---|---|---|
|
||
| IPsec NH3↔ANA | 6.9 ms | **102 ms** avg, 136 ms max |
|
||
| WireGuard NH3→ana-wg | 6.2 ms | **12.7 ms** avg, 23 ms max |
|
||
|
||
WireGuard carries **3.1× the single-stream throughput with 8× less latency
|
||
inflation** on the same wire.
|
||
|
||
### Attribution — still not fully separated, and it no longer matters much
|
||
|
||
Both IPsec measurements have a FortiGate *and* a UDM doing IPsec, so this still
|
||
does not isolate which one imposes the 2.8× penalty. Closing that would need
|
||
Linux↔Linux IPsec or UDM↔UDM IPsec on the same path. **But the practical
|
||
decision no longer depends on the answer**, because the fix is the same either
|
||
way and it is already demonstrated.
|
||
|
||
### Recommendation (supersedes FOLLOW-UP 1)
|
||
|
||
A **WireGuard site-to-site between NH3 and Anaheim, terminated on `ana-wg`**, is
|
||
worth real consideration: 3.1× single-stream, flat scaling, far better latency
|
||
under load, and it reuses infrastructure that already exists and is already
|
||
internet-reachable. It is also the architecture already proven for NH3↔ESH.
|
||
Open questions before committing: routing/failover if ana-wg (an LXC) is down,
|
||
whether it replaces or parallels the IPsec tunnel, and firewall policy for the
|
||
new transit. ana-wg CPU was only ~40% busy across 4 cores at 767 Mbit/s, so it
|
||
has headroom.
|
||
|
||
**AND: `nconnect=8` on /mnt/smithy remains worth doing regardless** — it is the
|
||
same lever (turn one flow into many) and brokkr-smithy-dev has given standing
|
||
approval to apply it once the FortiGate work settles, with no need to ask again.
|
||
|
||
---
|
||
|
||
## RESOLVED (2026-08-23): it is the UDM's software AES-CBC. The FortiGate is exonerated.
|
||
|
||
Operator's theory — the UDM does IPsec in software with no crypto offload, so
|
||
the cost of the cipher itself is the limit — is **correct**, and it is now
|
||
demonstrated rather than inferred. He also correctly pointed out that
|
||
UDM↔UDM Site Magic is **WireGuard, not IPsec**, so that row never said anything
|
||
about UniFi's IPsec performance. It didn't, and I had leaned on it.
|
||
|
||
### The controlled experiment: vary cipher cost, hold everything else
|
||
|
||
AES-128 is 10 rounds, AES-256 is 14. If software crypto is the binding
|
||
constraint, throughput must rise when the cipher gets cheaper. If the limit
|
||
were the FortiGate's NPU, it would not move at all — hardware crypto is not
|
||
cipher-cost-sensitive in that range. Run A/B/A, single stream, 25–60 s each:
|
||
|
||
| condition | ESP cipher | single-stream | UDM CPU |
|
||
|---|---|---|---|
|
||
| A | aes256-cbc + sha1 | 232.3 Mbit/s | 35.4% |
|
||
| B | **aes128**-cbc + sha1 | **281.8**, 274.9 Mbit/s | 35.5% |
|
||
| A again | aes256-cbc + sha1 | 244.9, 242.5 Mbit/s | — |
|
||
|
||
**~1.16–1.20× faster on the cheaper cipher at identical CPU.** Same bytes of
|
||
CPU work, more payload through it. That is the signature of CPU-bound software
|
||
crypto, and it rules out the FortiGate's NPU as the limiter.
|
||
|
||
### Correcting two of my own earlier claims
|
||
|
||
1. **"UDM CPU is only ~7%, so it isn't CPU-bound" was WRONG — a sampling
|
||
artifact.** UniFi's `system-stats.cpu` refreshes on the device report
|
||
interval; 4-second sample windows were reading stale values. Under a
|
||
sustained 60 s single-stream load it reads **35.4%**, with load1 rising
|
||
0.60 → 1.17. On a 4-core UDM Pro SE that is ≈1.4 cores — one core saturated
|
||
on crypto plus overhead. **Always drive load for ≥60 s before trusting a
|
||
UniFi CPU figure.**
|
||
2. **The "FortiGate per-session NPU offload" hypothesis is REFUTED**, not merely
|
||
unproven. It predicts no change from a cipher swap; a 20% change was measured.
|
||
|
||
### Why the numbers all line up now
|
||
|
||
- **1 stream = 1 core of UDM crypto** → ~240 Mbit/s on AES-256-CBC.
|
||
- **8 streams = ~3 usable cores** → ~692 Mbit/s, ≈2.9× the single-stream figure
|
||
on a 4-core box. Aggregate is noisy (492–692 across repeats on a live link)
|
||
and is *not* cipher-sensitive, consistent with it being bounded by the path/
|
||
uplink rather than crypto once several cores are engaged.
|
||
- **AES-CBC is the specific villain: it is serial.** Each block depends on the
|
||
previous one, so the ARM AES instructions cannot pipeline across blocks. GCM
|
||
(CTR-based) and ChaCha20-Poly1305 both parallelise freely. That is why the
|
||
same UDM does 557 Mbit/s single-stream on WireGuard and only 240 on IPsec.
|
||
- **This retroactively vindicates the GCM cutover as the right idea aimed at the
|
||
right box** — GCM would have removed the serial dependency on the constrained
|
||
end. UniFi simply does not offer it, which is what made it impossible.
|
||
|
||
### Options this opens
|
||
|
||
- **AES-128 instead of AES-256: ~16–20% for free**, no topology change, one API
|
||
call per end. 128-bit is not the weak link here (SHA1 integrity is more
|
||
dated, and unchanged either way). Operator's call — **not adopted**, restored
|
||
to aes256.
|
||
- **WireGuard site-to-site via ana-wg: 767 Mbit/s single-stream** (3.1×), and it
|
||
sidesteps the UDM's IPsec datapath entirely. Still the biggest win available.
|
||
- Replacing the FortiGate **will not help this** — it was never the constraint.
|
||
Worth knowing before the cutover.
|
||
|
||
### State left behind
|
||
|
||
UDM network object verified **byte-identical** to its pre-test snapshot
|
||
(aes256/sha1). Tunnel up, selectors 1/1. FortiGate phase2 `pfi-ana-nh3` is
|
||
left as `aes256-sha1 aes256gcm aes128-sha1` — a permissive superset; the peer
|
||
offers only aes256 so the extra entries are inert, but **narrowing it back to
|
||
`aes256-sha1` is one line** if the looser list is unwanted.
|
||
|
||
---
|
||
|
||
## FOLLOW-UP 4 (2026-08-23): a downstream WireGuard terminator costs nothing to forward through
|
||
|
||
Operator's point: FortiOS has no WireGuard, so a WireGuard site-to-site must
|
||
terminate on a box *behind* the edge. Correct — and `ana-wg` (LXC, CT 113 on
|
||
pfi-pve, 10.250.50.252) already is that box.
|
||
|
||
**This closes a gap in FOLLOW-UP 3.** That 767 Mbit/s figure was measured with
|
||
traffic terminating *on* ana-wg. Real traffic must be forwarded onward to other
|
||
Anaheim hosts, which was never measured. Now it is:
|
||
|
||
| topology | 1 stream | 8 streams |
|
||
|---|---|---|
|
||
| IPsec, FortiGate ↔ UDM (today) | 245 Mbit/s | 692 Mbit/s |
|
||
| WG terminating **on** ana-wg | 767 Mbit/s | 763 Mbit/s |
|
||
| **WG transit: nh3 → wg → ana-wg → forward → ana-docker** | **763.8 Mbit/s** | **790.4 Mbit/s** |
|
||
|
||
**Forwarding through the LXC is free** (763.8 vs 767). The downstream-VM
|
||
architecture delivers the full 3.1× single-stream for real transit traffic, not
|
||
just for traffic landing on the tunnel box.
|
||
|
||
ana-wg while forwarding 764 Mbit/s: **~22% busy across 4 cores** (77.8% idle),
|
||
so roughly 0.9 cores. Note `/proc/loadavg` inside this LXC reports the *host's*
|
||
load, not the container's — do not read it as ana-wg's own. For contrast the
|
||
UDM burns 35.4% of its 4 cores to move 240 Mbit/s, so ana-wg has ample headroom.
|
||
|
||
### Design consequences of terminating downstream — the parts that need decisions
|
||
|
||
1. **Anaheim hosts must route to ana-wg, not to the FortiGate.** The 763.8
|
||
figure was obtained with an explicit `10.30.10.200/32 via 10.250.50.252`
|
||
route on ana-docker. Without that, a host sends 10.100.0.0/16 to its default
|
||
gateway (ana-gw), which routes it back out the *same* interface to ana-wg — a
|
||
LAN hairpin crossing the FortiGate twice. **The hairpin variant was NOT
|
||
measured.** Options: DHCP option 121 pushing the route fleet-wide, a dedicated
|
||
transit VLAN for ana-wg, or accept the hairpin.
|
||
2. **New single point of failure.** Today site-to-site dies only when the edge
|
||
dies, which is total anyway. A downstream terminator fails independently.
|
||
Mitigation: keep the IPsec tunnel configured as a higher-metric fallback
|
||
route so it takes over when ana-wg is down.
|
||
3. **ana-wg is an LXC on pfi-pve**, so its ~0.9 cores and NIC traffic land on the
|
||
hypervisor shared with the rest of the Anaheim VMs.
|
||
4. **The NH3 end needs a terminator too**, and there are two shapes:
|
||
- **Linux VM at NH3** (nh3-dev or a dedicated VM on nh3-pve) — this is what
|
||
was measured: **764 Mbit/s**.
|
||
- **NH3 UDM's existing WireGuard server** (`PFI-NH3-WG`, wireguard-server on
|
||
UDP 31337) accepting ana-wg as a peer — plausible but **untested**, and
|
||
UniFi's WireGuard shows a per-flow penalty (557 Mbit/s single-stream on
|
||
Site Magic), so expect ~557 rather than 764. Still 2.3× today.
|
||
|
||
### Standing recommendation
|
||
|
||
Worth doing, but it is **a project, not a config tweak** — routing, failover and
|
||
policy all need deciding. The cheap wins remain available meanwhile and are
|
||
independent: `nconnect=8` on NFS mounts (approved by brokkr-smithy-dev, pending
|
||
the FortiGate work settling) and AES-128 for ~20%.
|
||
|
||
---
|
||
|
||
## LANDED (2026-08-23): AES-128 on both tunnels; FortiGate public admin closed
|
||
|
||
Operator directed: adopt AES-128 on **both** Anaheim tunnels, make-before-break,
|
||
then close the FortiGate's WAN and SSH admin surfaces. All done and verified.
|
||
|
||
**Context that retires the WireGuard-in-a-VM design work:** the FortiGate is
|
||
being **replaced by OPNsense on a Dell R420**, which gives **WireGuard on the
|
||
edge device itself**. The downstream-terminator architecture (FOLLOW-UP 4) is
|
||
therefore moot — do not scope it. This also **un-parks the OPNsense migration**,
|
||
which auto-memory recorded as PARKED pending "hardware acquisition"; the R420
|
||
is that trigger.
|
||
|
||
### What changed
|
||
|
||
Make-before-break on the FortiGate first, so neither tunnel dropped waiting on
|
||
a far end:
|
||
|
||
| phase2 | proposal now |
|
||
|---|---|
|
||
| `pfi-ana-nh3` | `aes256-sha1 aes256gcm aes128-sha1` |
|
||
| `ana-eshudm-dyn` | `aes256-sha1 aes128-sha1` |
|
||
|
||
Then each UDM flipped to `ipsec_esp_encryption: aes128`:
|
||
|
||
| tunnel | UDM object | before | after |
|
||
|---|---|---|---|
|
||
| NH3 ↔ ANA | `pfi-nh3-ana` `697d64414c85dd2b6669b00a` @ 10.100.0.1 | 245 Mbit/s | **269.7** |
|
||
| ESH ↔ ANA | `esh-ana` `697723b9b9d4266dddf2bcc7` @ 10.0.0.1 | 268 Mbit/s | **304.3** |
|
||
|
||
Single-stream gain ~10–13% here, against 16–20% in the earlier controlled A/B —
|
||
the difference is live-link variance, not a different result. Both UDM objects
|
||
were diffed field-by-field against pre-change snapshots: **the only field that
|
||
moved on either is `ipsec_esp_encryption`.**
|
||
|
||
The FortiGate proposal lists were deliberately **left permissive** (still
|
||
accepting aes256). The peers offer only aes128 so the extra entries are inert,
|
||
and keeping them means a UDM reverting does not strand the tunnel. Narrowing to
|
||
`aes128-sha1` alone is a one-liner if the looser list is unwanted.
|
||
|
||
### Admin surfaces closed
|
||
|
||
`wan1 allowaccess` → **`ping`** (https + ssh removed) and `infra-ops` trusthost
|
||
→ **10.0.0.0/8 only** (the 8 wide-open ranges unset). Verified 443 and 22 closed
|
||
from both NH3 and ESH; management over the tunnel at 10.250.0.1 still works.
|
||
**Sequencing that matters: the close was executed over the TUNNEL path, not over
|
||
WAN** — removing `ssh` from allowaccess while connected over WAN kills the
|
||
session mid-command.
|
||
|
||
**Consequence to hold in mind: ana-gw now has no out-of-band management path.**
|
||
If both tunnels drop it is console-only until someone is on site.
|
||
|
||
### Gotcha: the two UDM vault items have DIFFERENT shapes
|
||
|
||
- `unifi/pfi-udmse-api-key` → a **bare 32-char key**. `secret get` output is the key.
|
||
- `unifi/esh-udmpm-api-key` → a **19-line documentation note** with the key on a
|
||
`key:` line. `secret get` piped straight into a header yields a 1396-byte
|
||
value and the UDM answers **`400 Bad Request` from nginx**. Extract with
|
||
`grep '^key:' | awk '{print $2}'`.
|
||
|
||
**The ESH key's first-ever confirmed WRITE happened here** (auto-memory recorded
|
||
it as read-verified only): a control PUT of the unchanged object returned
|
||
`rc:ok`, then the real change did too. That key has a full read+write admin role.
|
||
|
||
---
|
||
|
||
## CORRECTION (2026-08-23): port 80 on the WAN IP is the FortiOS ACME listener
|
||
|
||
The claim in the previous section that `.42:80` was an **ISP transparent proxy**
|
||
was **WRONG**, and so was the earlier warning that ACME renewal would fail with
|
||
port 80 absent from `allowaccess`. Operator pushed back asking where the port-80
|
||
map terminated. It terminates **on the FortiGate itself**.
|
||
|
||
**What it is:** the FortiOS **ACME HTTP-01 challenge listener**. `config system
|
||
acme` has `set interface "wan1"`, and FortiOS opens port 80 on that interface to
|
||
answer Let's Encrypt challenges **independently of `allowaccess`** — `wan1
|
||
allowaccess` reads `ping` only and the port is still open. Every non-challenge
|
||
request returns a fixed 403 whose body is literally:
|
||
|
||
```
|
||
<!DOCTYPE html><html><head><title>ACME Access Only</title></head><body>ACME Access Only</body></html>
|
||
```
|
||
|
||
**Not a DNAT.** The full VIP table has 14 entries; only two land on `.42` —
|
||
`Kokoro-In` (:8880 → 10.250.50.51) and `wg-to-ana-wg` (:31337 → 10.250.50.252).
|
||
~~Worth noting separately: four VIPs are all-port static NAT~~ — **that claim was
|
||
WRONG, see the correction below.** All fourteen VIPs are scoped.
|
||
|
||
### The methodology error that produced the wrong answer — worth not repeating
|
||
|
||
The sniffer filter used was `dst host 38.120.12.42 and tcp port 80`. **`dst host`
|
||
matches only inbound packets**, so outbound SYN-ACKs were excluded *by
|
||
construction*; concluding "the box sends no SYN-ACK" from that capture was
|
||
unsound. Re-run with the bidirectional `host 38.120.12.42 and tcp port 80` it
|
||
immediately shows `wan1 out 38.120.12.42.80 -> <scanner>: syn ack`.
|
||
|
||
**Rule: when testing whether a box *answers*, the sniffer filter must be
|
||
bidirectional. `dst host` silently answers a different question.**
|
||
|
||
### Consequences
|
||
|
||
- **ACME renewal will work** with `allowaccess ping`. The earlier "add `http`
|
||
back or the cert expires" warning is retracted — FortiOS opens the challenge
|
||
port itself. Cert valid to 2026-10-27, renewal attempt ~2026-09-27.
|
||
- **It is not an admin surface** — static 403, no auth, no GUI.
|
||
- Its practical value is now low: WAN admin is closed, so the cert only serves
|
||
the internal GUI at 10.250.0.1, where the name would not match anyway. Killing
|
||
it (`config system acme` → unset interface) would close the last WAN listener
|
||
at the cost of cert renewal. Operator's call; **not done**.
|
||
|
||
---
|
||
|
||
## CLOSED OUT (2026-08-23): ACME disabled; and the "all-port VIP" alarm was FALSE
|
||
|
||
### ACME disabled — the WAN IP now exposes nothing
|
||
|
||
`config system acme / unset interface` (the account object is left in place;
|
||
with no interface bound there is no listener). Verified:
|
||
|
||
- **External scan of 38.120.12.42 across 55 ports: no open TCP ports at all.**
|
||
- Internal GUI at 10.250.0.1 still answers **200**, SSH still works.
|
||
- `admin-server-cert` is still `ana-fw.pfi` — the existing cert is untouched and
|
||
serves the internal GUI until **2026-10-27**; it simply will not auto-renew.
|
||
|
||
Reverse with `config system acme / set interface "wan1"`.
|
||
|
||
### RETRACTION: the four VIPs are NOT all-port
|
||
|
||
A previous section claimed `Rustdesk`, `https-to-tacticalrmm`, `web-to-webhost`
|
||
and `web-to-sfcontainer` were unrestricted all-port static NATs. **They are not.**
|
||
A FortiOS VIP can be scoped **two different ways** and the parser used only
|
||
checked one:
|
||
|
||
1. `set portforward enable` + `set extport <n>` — a single mapped port, **or**
|
||
2. `set service "<svc>"` on the VIP object — constrains the VIP to that service.
|
||
|
||
All four use form 2. The custom services are narrow: `Rustdesk` = TCP
|
||
21115–21119 + UDP 21116 (the standard RustDesk range), `ssh-mapped-2223` = TCP
|
||
2223 only. **Every one of the 14 VIPs is scoped; none is unrestricted.**
|
||
|
||
**Lesson: absence of `portforward` does NOT mean all-port on a FortiOS VIP —
|
||
check `service` too.** Better still, do what settled it here: scan from outside
|
||
rather than reading config.
|
||
|
||
### Ground-truth public exposure (external TCP scan, post-change)
|
||
|
||
| IP | open | maps to |
|
||
|---|---|---|
|
||
| 38.120.12.41 | *nothing* | — |
|
||
| **38.120.12.42** | ***nothing*** | the FortiGate itself — fully closed |
|
||
| 38.120.12.43 | 80, 443 | sf-ana-container 10.250.150.100 (SureFire tenant) |
|
||
| 38.120.12.44 | 22, 80, 443, 8025, 21115–21119 | gitea (→222), traefik, mailrise, RustDesk |
|
||
| 38.120.12.45 | 80, 443, 2223 | pfi-ana-webhost 10.250.50.52 (2223→22) |
|
||
| 38.120.12.46 | 443 | pfi-tacticalrmm 10.250.50.57 |
|
||
|
||
Configured-but-closed: 8443 (mattermost-calls), 8444 (webdav-nas), 8880
|
||
(Kokoro-In) — VIPs exist, nothing listening behind them. Worth a tidy-up during
|
||
the OPNsense translation but not exposure.
|
||
_Archived 2026-09-09._
|
||
|
||
- `[2026-08-24]` **ana-gw's public admin surface closed to zero open ports, ACME listener included.** Two of my diagnoses were wrong first (an "ISP proxy" that was the FortiGate, and an "all-port VIP" alarm that was a parser gap) — both from reading config instead of measuring from outside.
|
||
# [2026-08-24] ana-gw public admin surface closed to zero, ACME listener included
|
||
|
||
WAN admin was opened at the start of the session as a cutover contingency
|
||
("so I don't have to drive down there"), then closed again on operator
|
||
instruction once the AES-128 work landed. Net result: **the FortiGate's WAN
|
||
address now exposes no TCP port at all.**
|
||
|
||
## Final state
|
||
|
||
External scan of `38.120.12.42`, 55 ports: **nothing open**. Verified from two
|
||
sites. `wan1 allowaccess` = `ping`; `infra-ops` trusthost back to `10.0.0.0/8`.
|
||
|
||
**Consequence to hold: there is no out-of-band path to ana-gw.** If both tunnels
|
||
drop it is console-only. Re-open is two one-liners (allowaccess + trusthost) —
|
||
both are recorded in auto-memory `reference_fortigate_ana_gw_access`.
|
||
|
||
## Port 80 was the FortiOS ACME listener, and I got it wrong first
|
||
|
||
`38.120.12.42:80` answered a bare 403 (`ACME Access Only`, 101 bytes) with
|
||
`allowaccess` set to ping only. First diagnosis — "an ISP transparent proxy" —
|
||
was **wrong**, and the reason is worth keeping:
|
||
|
||
> The sniffer filter was `dst host 38.120.12.42 and tcp port 80`. **`dst host`
|
||
> matches inbound only**, so outbound SYN-ACKs were excluded *by construction*,
|
||
> and concluding "the box sends no SYN-ACK" from that capture was unsound.
|
||
|
||
Re-run bidirectionally (`host … and tcp port 80`) it immediately showed
|
||
`wan1 out 38.120.12.42.80 -> <scanner>: syn ack`. **Rule: to test whether a box
|
||
*answers*, the filter must be bidirectional.**
|
||
|
||
The listener is opened by `config system acme / set interface "wan1"` and
|
||
**bypasses `allowaccess` by design** — FortiOS needs port 80 for HTTP-01. It
|
||
was disabled (`config system acme / unset interface`); the LE cert (`ana-fw.pfi`,
|
||
valid to 2026-10-27) is untouched and simply stops renewing, which is fine
|
||
because WAN admin is closed and the box is being replaced.
|
||
|
||
## Retracted in the same pass: the "four all-port VIPs" alarm
|
||
|
||
Claimed four VIPs were unrestricted all-port static NAT. **False.** A FortiOS
|
||
VIP is scoped **two** ways — `portforward`+`extport`, *or* a `service` binding
|
||
on the VIP object — and only the first was checked. All 14 VIPs are scoped;
|
||
`Rustdesk` is TCP 21115–21119, `ssh-mapped-2223` is TCP 2223 only.
|
||
|
||
Ground-truth external scan of all six public IPs is recorded in
|
||
`reference_fortigate_ana_gw_access`. Configured-but-dead: `:8443`
|
||
(mattermost-calls), `:8444` (webdav-nas), `:8880` (Kokoro-In) — tidy-up
|
||
candidates for the OPNsense translation, not exposure.
|
||
|
||
**Lesson, twice in one session: measure from outside instead of parsing config.**
|
||
Both wrong answers came from a filter that answered a different question.
|
||
_Archived 2026-09-09._
|
||
|
||
- `[2026-08-24]` **Scriberr deployed on ana-ml2 GPU1, image built from source.** Three upstream bugs: the Blackwell image was never published, it must run as uid 10001, and `UV_LINK_MODE=copy` is required or two backends fail silently.
|
||
# [2026-08-24] Scriberr transcription deployed on ana-ml2, GPU1
|
||
|
||
Self-hosted audio/video transcription + diarization. Operator chose GPU
|
||
placement over ana-docker (8 cores shared with 50 containers, 37 GB disk)
|
||
against ana-ml2's 96 cores, `/tank`'s terabytes and GPU1's headroom.
|
||
|
||
**Live:** `http://scriberr.ana.internal:8080` (DNS alias added), health `healthy`,
|
||
all seven backends up, zero failures: `whisperx pyannote sortformer parakeet
|
||
canary voxtral openai`. ~30 GB of weights on `/tank`.
|
||
|
||
Stack: `stacks/scriberr/`. Full gotcha list in auto-memory
|
||
`reference_scriberr_ana_ml2`.
|
||
|
||
## Three upstream bugs, none of them ours
|
||
|
||
**1. The Blackwell image does not exist.** Upstream's README documents
|
||
`scriberr-cuda-blackwell`; GHCR has **no tags for it**. Published
|
||
`scriberr-cuda` covers sm_61–sm_89 only — on these sm_120 cards it fails or
|
||
silently drops to CPU. The real sm_120 path is `Dockerfile.cuda.12.9`
|
||
(CUDA 12.9.1, cu128 torch), **built from source**. Do not "simplify" the compose
|
||
back to the published image.
|
||
|
||
**2. It must run as uid 10001, not 1000** — and the error lies:
|
||
`unable to open database file: out of memory (14)`. Error 14 is
|
||
`SQLITE_CANTOPEN`, not an OOM, on a box with 566 GB RAM. That Dockerfile creates
|
||
`appuser` at 10001 (Ubuntu 24.04 owns uid 1000 as `ubuntu`) and chowns `/app` to
|
||
it, while the entrypoint's PUID remap covers only the data dirs.
|
||
**Isolated by elimination**: SQLite writes fine to `/tank` as 1000 → not the
|
||
mount; fails on a plain named volume too → not the storage; the **published CPU
|
||
image works at PUID=1000** because there `appuser` *is* 1000.
|
||
Generalisable: *when a container "permission" bug appears, compare the uid the
|
||
image was BUILT for against the uid you are RUNNING as.*
|
||
|
||
**3. `UV_LINK_MODE=copy` is required.** Scriberr builds each backend's Python env
|
||
with `uv` at start; uv's reflink mode fails on overlayfs+ZFS with
|
||
`Failed to clone … Resource temporarily unavailable (os error 11)`. **Partial
|
||
failure** — WhisperX and PyAnnote came up and the app looked fine while Parakeet
|
||
and Sortformer were silently absent. Occurrences 2 → 0 after the fix.
|
||
|
||
## Related
|
||
|
||
`speaches` on irv-ml1 **stopped** the same day (stack retained, one command to
|
||
restart): Eyra was abandoned pre-implementation because Scriberr covers the need,
|
||
leaving it with no consumer. Scriberr runs its **own** WhisperX in-container and
|
||
is **not** a speaches consumer. Idle footprint at stop was 274 MiB, not the
|
||
~5.9 GB quoted — that figure is the loaded-model working set.
|
||
_Archived 2026-09-09._
|
||
|
||
- `[2026-08-24]` **ESH DNS fixed at the IPv6 layer and the naming scheme went live on three hosts.** UniFi's RDNSS cannot be disabled but CAN be redirected — the field is only honoured when an explicit server is given.
|
||
# [2026-08-24] ESH DNS fixed at the IPv6 layer, and the naming scheme went live
|
||
|
||
Reported as "`scriberr.ana.internal` doesn't resolve on my Mac, and nslookup
|
||
shows an IPv6 DNS server." Operator's diagnosis was right; the fix took three
|
||
wrong turns worth recording.
|
||
|
||
## Root cause
|
||
|
||
`esh-userland` has IPv6 PD with RA at `pref high`, and the UDM advertises
|
||
**itself** as the resolver via RDNSS. macOS honours RDNSS and prefers it over
|
||
the DHCPv4-supplied resolver, so queries went to the UDM — which does not know
|
||
`.internal` — and returned NXDOMAIN. AdGuard was never consulted.
|
||
|
||
Two adjacent gaps found while there: `esh-userland`'s **secondary** v4 resolver
|
||
was `10.0.10.1` (the UDM itself), and `esh-server` had **DNS handout disabled
|
||
entirely**, so every host there got the UDM and could never resolve `.internal`
|
||
— esh-docker-vm was living proof.
|
||
|
||
## The three wrong turns
|
||
|
||
1. **`dhcpdv6_dns_auto=false` alone does nothing.** It is only honoured **when an
|
||
explicit server is supplied**. Setting it bare looks like a no-op and invites
|
||
the conclusion that the field is inert — which I drew, wrongly. Despite the
|
||
`dhcpdv6_` prefix it *does* drive the RA's RDNSS option on a SLAAC network.
|
||
2. **`wan_dns1` is NOT used by the UDM's LAN-facing forwarder.** Setting it to
|
||
AdGuard persists, reads back, and changes nothing. Proven with **fresh
|
||
uncached ad domains** — AdGuard blocklists answer `0.0.0.0`, the UDM returned
|
||
real IPs. Reverted.
|
||
3. **`force-provision` returns `rc:ok` and proves nothing** — consistent with the
|
||
known `cmd/devmgr` behaviour.
|
||
|
||
Every failed attempt returned `rc: ok`. **Verify by observed effect.** RAs were
|
||
probed with a stdlib raw-socket Router Solicitation parsing option type 25
|
||
(`rdisc6`/`tcpdump` were both absent; nothing was installed).
|
||
|
||
## What landed
|
||
|
||
RDNSS **redirected** rather than disabled — better than switching it off:
|
||
|
||
| VLAN | v4 | v6 RDNSS |
|
||
|---|---|---|
|
||
| `esh-userland` | 10.0.50.45 + 10.100.50.40 | `…:4411:b105:50:45` |
|
||
| `esh-server` | 10.0.50.45 + 10.100.50.40 | `…:4411:b105:50:45` |
|
||
|
||
The v4 secondary moved from the UDM to the **NH3 AdGuard** — reachable over
|
||
Site Magic and authoritative for the zone. ⚠ **A secondary only fails over on
|
||
SILENCE, not on wrong answers**: NXDOMAIN is a *successful* answer, the client
|
||
accepts it and never retries. A secondary that doesn't know your private zone is
|
||
a coin flip, not a spare tyre. `esh-cameras` deliberately untouched — routing
|
||
camera DNS through AdGuard's filtering risks their cloud features.
|
||
|
||
## The naming scheme became real
|
||
|
||
The resolver address is the scheme's first live use, replacing a MAC-derived
|
||
SLAAC address that would break on a NIC change. All three `esh-server` Linux
|
||
hosts now carry `4411:B105` ("FOR ALL BIOS"):
|
||
|
||
```
|
||
esh-docker-vm …:4411:b105:50:45 esh-pve-nas …:4411:b105:50:55
|
||
esh-vm-db …:4411:b105:50:60
|
||
```
|
||
|
||
Applied by an `if-up.d` hook that **derives the prefix at runtime** (self-heals
|
||
on re-delegation), backgrounds itself with a retry (SLAAC may not have landed;
|
||
a blocking hook would stall bring-up on a headless box), and adds nothing to
|
||
existing config. **Not** an `iface … inet6 static` stanza — on Debian that sets
|
||
`accept_ra=0` and would strand the host.
|
||
|
||
⚠ **Proxmox bridges need `accept_ra=2`.** `esh-pve-nas` had link-local only
|
||
despite every sysctl looking right: `vmbr0.forwarding=1`, and the kernel ignores
|
||
RAs on a forwarding interface unless `accept_ra` is explicitly `2`. Fixed with
|
||
`accept_ra_defrtr=0` alongside, so it takes the prefix but **declines the default
|
||
route** — an IPv6 identity with no change to a hypervisor's routing. Expect this
|
||
on every Proxmox node when its LAN gets v6.
|
||
|
||
Canonical: `docs/pfi/ipv6-naming-scheme.md`. UniFi limits:
|
||
auto-memory `reference_unifi_dns_rdnss_limits`.
|
||
_Archived 2026-09-09._
|
||
|
||
- `[2026-08-24]` **esh-vm-db brought onto the fleet infra-ops identity and given its first vaulted credential.** It previously had none: root and infra-ops refused key auth and `lkraven`'s sudo wanted a password nobody held, leaving `qm guest exec` from the hypervisor as the only privileged path. Break-glass root password at `secret get esh-vm-db/root-breakglass-password` (console-only; plaintext never crossed the wire — only its SHA-512 hash did).
|
||
_Archived 2026-09-09._
|
||
|
||
- `[2026-08-23]` **Anaheim's IPsec tunnel ceiling — investigated, then CLOSED 2026-08-24.** The 25%-of-2-Gbps framing was wrong (NH3's uplink is 1 Gbps); AES-GCM proved impossible; AES-128 landed instead. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md`
|
||
_Archived 2026-09-09._
|
||
|
||
- `[2026-08-23]` **selene retired after losing a head-to-head on its own job; `chat-judge` moved to gen, the model name 404s by design.** Also surfaced that **7 aliases share one seat** — cross-checking between them is an echo, which caught a real defect in brokkr's 46k-exposure R47 gate.
|
||
# [2026-08-23] selene seat retired after losing a head-to-head; 7 aliases share one seat
|
||
|
||
## Why selene went
|
||
|
||
Benchmarked against `gen` on selene's own job — 24 designed judge items with
|
||
checkable ground truth, pairwise + absolute modes, 3 repeats, run on **both** a
|
||
neutral JSON prompt and Selene's **native Atla template** (288 calls, free local).
|
||
|
||
```
|
||
neutral JSON selene 20/24 (83%) gen 23/24 (96%)
|
||
native Atla selene 21/24 (88%) gen 22/24 (92%)
|
||
```
|
||
|
||
gen won on both templates and **selene's best sat below gen's worst**. Selene was
|
||
given its own fine-tuned template as a fairness check before any recommendation;
|
||
it gained one point, not three.
|
||
|
||
**Decisive defect: selene cannot emit "tie"** — 0/2 on both templates, forcing a
|
||
winner on every equivalent pair. For eval work that is the case that matters.
|
||
|
||
brokkr-smithy-dev independently corroborated from the other end with a **null
|
||
control** (an excerpt compared against ITSELF, where tie is definitional):
|
||
`chat-judge`(selene) TIE **27/60 = 45%**, gen **60/60 = 100%**; ground-truth
|
||
recovery on real-corpus ranking selene **47% — chance** vs gen 94%. My 83-vs-96
|
||
understated it: on a *ranking* task selene was a coin flip. Absolute scoring on
|
||
designed items is an easier task than ranking real text — the harness is a
|
||
**screen, not a verdict**, and its README says so.
|
||
|
||
Reclaimed **17.2 GiB** on ana-ml2 GPU1 (free 1,818 -> 19,450 MiB).
|
||
|
||
## The naming rule, restated the hard way
|
||
|
||
I proposed repointing `selene-1-mini-8b` at gen and was **correctly overruled**:
|
||
|
||
> never repoint a named model at a different model's endpoint — that is
|
||
> intentionally misleading
|
||
|
||
`chat-judge` is a **role** alias (ADR-0012: consumers bind the capability) and
|
||
moved to gen with a deterministic judge profile copied from `image-judge`.
|
||
`selene-1-mini-8b` is a **model** name and was removed outright — it now returns
|
||
`HTTP 400 Invalid model name`, verified. The discriminator: *does the string
|
||
promise a capability, or an identity?*
|
||
|
||
## The 7-way alias collision — the finding with the longest reach
|
||
|
||
```
|
||
chat-judge classifier gen image-judge
|
||
qwen-image-bench summarizer summarizer-large -> qwen3.8-27b-uncensored :8015
|
||
```
|
||
|
||
Also colliding: `gen-frontier`/`gen-frontier-reasoning`/`glm-5.2`/`glm-5.2-reasoning`;
|
||
`ext-tts`/`gpt-4o-mini-tts`/`tts-1`/`tts-1-hd`; `reranker`/`reranker-a3-bge-v2-m3`.
|
||
|
||
**Cross-checking a result against another alias measures nothing when they are the
|
||
same weights — agreement is an echo, not corroboration.** Documented at the head of
|
||
`model_list` in the live gateway config, because it belongs where people read it.
|
||
|
||
This caught a real defect within hours: brokkr's R47 premium-corpus gate was about
|
||
to run ~46,000 record-exposures against `gen` with `summarizer` shortlisted as an
|
||
independent second opinion. They pinned the backing model in the preregistration
|
||
and dropped the second-alias idea instead.
|
||
|
||
## Provenance seam (brokkr's pushback, adopted)
|
||
|
||
The gateway returns the **alias** in the response `model` field, not the backing
|
||
model — so a per-call guard catches a swap *during* a run and is blind to one
|
||
*between* runs. **Role alias for routing, concrete model for provenance.**
|
||
`GET :4000/model/info` with the shared key already exposes backing model +
|
||
api_base; resolve at run start AND end and void on mismatch.
|
||
|
||
## Artifacts
|
||
|
||
- Harness kept at `tools/judge-bench/` (`--models` REQUIRED — a stale default
|
||
would silently benchmark a retired seat).
|
||
- `stacks/selene/` keeps compose + a README explaining the retirement.
|
||
- Technique worth stealing, from brokkr: **a control constructed so the correct
|
||
answer is DEFINITIONAL rather than judged cannot inherit the designer's error.**
|
||
Item vs itself; response vs its own truncation; text vs its own clauses
|
||
permuted. Add those before adding more judged items.
|
||
|
||
Commits `ca3c984`, `b8a5355`.
|
||
_Archived 2026-09-09._
|
||
|
||
- `[2026-08-23]` **hrafn adopted; its CI reported green for its whole life while deploying nothing.** A staging dir inside the rsync target destroyed its own source mid-copy; the deeper fault was verify steps that asserted uptime, never content.
|
||
# [2026-08-23] hrafn adopted; its CI deploy reported green while deploying nothing
|
||
|
||
`hrafn` — genuine-Chromium browser-fetch behind a REST API, for bot-gated sites
|
||
(Reddit first). Built by nevermore-claude on ana-docker, handed to infra-ops for
|
||
uptime ownership. Internal-only on `traefik-net`, no host port; consumers reach
|
||
`http://hrafn:8080`. Canonical at `stacks/hrafn/`.
|
||
|
||
## Intake found a live credential exposure
|
||
|
||
`/opt/docker/compose/hrafn/.env` was mode **0644 with a live 57-char bearer token**
|
||
— verified as real exposure by reading it as `nobody` on a box with four
|
||
interactive accounts. Tightened to 0600. That triggered the wider sweep (see
|
||
[[2026-08-23-ana-docker-env-perms-sweep]]).
|
||
|
||
## The CI defect — the one worth remembering
|
||
|
||
I authored the deploy (elway playbook + gitea workflow) to replace a hand-rsync,
|
||
tagging the image with the commit SHA for provenance. nevermore-claude later found
|
||
v1.0.0 deploying "green" while the host still served 0.1.0.
|
||
|
||
**Root cause was mine and nastier than either hypothesis.** The staging dir was
|
||
`$compose_dir/.stage` — **inside** the rsync target. So
|
||
`rsync -a --delete $compose_dir/.stage/ $compose_dir/` deleted `.stage` from the
|
||
destination (absent from the source listing) **during** the transfer, destroying
|
||
its own source mid-copy. Reproduced exactly:
|
||
|
||
```
|
||
before: app.py="OLD" leftover.txt .stage/app.py="NEW"
|
||
after: app.py="OLD" leftover.txt GONE, .stage GONE
|
||
```
|
||
|
||
Deletion succeeded, the copy silently did not, rsync exited 0. So the directory
|
||
*looked* converged while host source stayed frozen at the first manual rsync —
|
||
and because the build's `COPY` inputs never changed, Docker full-cache-hit and
|
||
every SHA tag aliased one image. **The provenance the tagging existed to provide
|
||
was false for the pipeline's entire life.**
|
||
|
||
**The real failure is the verification.** The verify steps asserted the marker,
|
||
container health, and a 200 from `/readyz` — all of which pass against a
|
||
completely frozen host. None measured *content*. A deploy that reports success
|
||
without asserting the bytes changed is verifying an **uptime**, not a deploy.
|
||
|
||
## Fixes
|
||
|
||
- stage at `/tmp/hrafn-deploy-stage`, outside the target
|
||
- CI computes `context_sha256` over the shipped file list; the playbook recomputes
|
||
it **on the host after the converge** and fails on mismatch
|
||
- compare the running container's `src/**/*.py` against the host's, so a SHA tag
|
||
cannot name layers the image lacks
|
||
- **compare `*.py` only** — `pip install .` generates `src/*.egg-info/*` inside the
|
||
image and `__pycache__` appears at runtime, so a naive `find src -type f` compare
|
||
false-fails on every healthy deploy. Verified against a known-good container
|
||
before shipping (12 host files, 18 in container, 0 content differences).
|
||
- declined `--no-cache`: a cache hit is *correct* when the context is genuinely
|
||
unchanged; assert the property rather than brute-force it.
|
||
|
||
## Access
|
||
|
||
Operator granted claude-bot **write** on `vh/hrafn`, so infra-ops maintains the
|
||
pipeline it owns instead of routing patches through the repo holder. `vh/hrafn` is
|
||
canonical; `stacks/hrafn/ci/` is a verified mirror.
|
||
|
||
Commits `b6924de`, `b001d0c`, `11b9d18`, `b38c369`, `9642952`.
|
||
_Archived 2026-09-09._
|
||
|
||
- `[2026-08-23]` **Worldtree b187 shipped; all three instances de-armed from a 69-day-stale `:latest`; Matrix homeserver re-plumbed to personal.** Includes the `:8009`-is-demo port trap that an IP-only fix would have walked into.
|
||
# [2026-08-23] Worldtree b187 shipped; all three instances de-armed from a 69-day-stale `:latest`; Matrix homeserver re-plumbed
|
||
|
||
## b187 pre-stage (#405 phases 1+2)
|
||
|
||
The matrix bridge stopped embedding the engine and became an HTTP client of the
|
||
Conversation API, so `WORLDTREE_API_URL` became **boot-blocking** — absent from the
|
||
container env, the bridge exits by design. Demo's compose never passed it; the next
|
||
recreate would have crash-looped. Pre-staged on demo and personal (additive, backed
|
||
up, verified with `docker compose config`, nothing restarted).
|
||
|
||
**Key decision, and I got its scope wrong first.** I argued demo should stay keyless
|
||
(no homeserver -> no rooms -> no turns -> no 401s). Right about turns, **wrong about
|
||
scope**: the engine preflight authenticates at boot regardless of homeserver, so demo
|
||
booted permanently degraded. Corrected — key `341c1488` minted under worldtree-dev's
|
||
recorded authorization, vaulted, wired, three-hop hash-verified.
|
||
|
||
## The 69-day-stale `:latest` landmine
|
||
|
||
All three instances pinned `WORLDTREE_IMAGE=.../worldtree:latest` in `.env` while
|
||
running SHA-tagged images built that day. Local `:latest` = `b19afd71d7cc`, built
|
||
**2026-06-14**. So ANY `docker compose up` — anyone's, for any reason — silently
|
||
downgraded that service by 69 days. Same footgun as the 2026-06-15 outage.
|
||
|
||
Re-pinned all three to their running SHAs (Worldtree #410), verified by rendering
|
||
compose config rather than reading `.env`, containers untouched. Playbook at
|
||
`playbooks/repin-worldtree-image.yaml`.
|
||
|
||
**`worldtree-pinned` was the worst case:** the instance whose entire purpose is being
|
||
frozen was running a **dangling image with no repo tags**, kept alive only by the
|
||
running container. One `docker rm` from garbage collection. Tagged
|
||
`:446e5807bf43` first, then pinned.
|
||
|
||
The guard I wrote had two bugs the pinned case exposed: it compared the container's
|
||
`.Config.Image` **string** (only the tag it was CREATED from — pinned was created
|
||
from `:latest` back when that meant 446e5807), and it reported CHANGED
|
||
unconditionally. Now compares **image IDs** and skips when already correct.
|
||
|
||
## Matrix homeserver ownership
|
||
|
||
Operator ruled: **personal owns the Matrix bridge.** The appservice tokens were never
|
||
missing — both sat at length 64 in the vaulted dev `env.sh` while both deployed
|
||
instances had them at length **zero**. Someone wired four of six Matrix vars and
|
||
stopped. Wired them into personal, three-hop verified.
|
||
|
||
**The trap worth remembering:** Synapse's registration pointed at
|
||
`http://10.100.10.50:8009` — nh3-dev, a dead epoch, with transaction 2801 queued at
|
||
512s backoff. The natural fix (swap the IP) gives `10.250.50.152:8009` which is
|
||
**DEMO's** bridge, and Synapse can reach both — it would have connected, delivered,
|
||
and looked correct while routing the operator's live rooms to the demo instance.
|
||
**Personal's bridge is :8010.** `docker port` is ground truth.
|
||
|
||
Corrected the URL, restarted Synapse (healthy in 32s after 3.5 months up), verified
|
||
`GET /_matrix/app/v1/ping -> 200` from inside the Synapse container. worldtree-dev's
|
||
smoke passed first try: room created, mimir accepted the invite, a real engine turn
|
||
ran, mimir replied in persona voice. #408 closed.
|
||
|
||
## Open on worldtree-dev's side
|
||
|
||
- **#411** — personal's bridge logs `Debug sink init failed: Permission denied:
|
||
/app/sessions/debug_rooms.json`. It creates two debug rooms but cannot persist
|
||
their IDs, so **every restart mints a fresh pair on the live homeserver**. Room
|
||
litter that compounds silently. Needs a which-container-writes-what check on the
|
||
sessions volume before anyone chowns it.
|
||
- Bridge/engine agent-roster drift: 6 of the bridge's 9 configured agents are not
|
||
listed by the engine on either instance.
|
||
- Historical Domari pairwise verdicts from the selene era are coin-flip-grade
|
||
(see [[2026-08-23-selene-retired-alias-collision]]); worldtree-dev banked that so
|
||
no future arc leans on them without re-judging.
|
||
|
||
Commits `064181a`, `bb19a96`.
|
||
_Archived 2026-09-09._
|
||
|
||
- `[2026-08-23]` **Every secret-bearing `.env` on ana-docker tightened to 0600** — eight stacks including vaultwarden and traefik, verified exposed by reading one as `nobody`.
|
||
# [2026-08-23] Every secret-bearing `.env` on ana-docker tightened to 0600
|
||
|
||
Found while taking uptime ownership of hrafn: its `.env` was mode 0644 with a live
|
||
bearer token. Not a hrafn lapse — **0644 was the de facto pattern on the host**.
|
||
Eight stacks carried secret-shaped vars in world-readable `.env` files on a box with
|
||
four interactive accounts, verified as real exposure by reading one as `nobody`.
|
||
|
||
Swept: **vaultwarden, traefik**, beszel, gitea-runner, miniflux, news-digest,
|
||
searxng, vor. (hrafn and nevermore were fixed separately the same day.) Six other
|
||
stacks already used 0600, so this converged on the existing house pattern rather
|
||
than inventing one. Post-sweep the host has **zero** secret-bearing `.env` readable
|
||
by `nobody`.
|
||
|
||
Playbook: `playbooks/tighten-env-perms.yaml`, one run per stack, re-runnable.
|
||
|
||
## The check that matters
|
||
|
||
Every run asserts `docker compose config` still renders **as the deploy user**
|
||
(`lkraven`), not as root. Checking the mode proves the bits changed; only rendering
|
||
as the deploy user proves the next deploy can still resolve its variables.
|
||
|
||
## Two gotchas recorded in the playbook
|
||
|
||
- **vaultwarden looked like it bind-mounted its `.env`** — which would mean the
|
||
*container's* UID reads it and 0600 could break the password vault. It does not:
|
||
that `- .env` is under `env_file:`, not `volumes:`. My grep matched the YAML list
|
||
item without checking its parent key. The playbook now **refuses** any stack that
|
||
genuinely bind-mounts its `.env`, since that case is read by the container UID.
|
||
- **elway prompted for a sudo password.** The `ana-docker` ssh alias resolves to
|
||
`lkraven`, who needs one; **`infra-ops@10.250.50.70` has NOPASSWD**. `corviduo-dev`
|
||
was repointed to infra-ops at some point and `ana-docker` was not. Run elway against
|
||
the infra-ops target on this host.
|
||
|
||
Commit `a896c0a`.
|
||
_Archived 2026-09-09._
|
||
|
||
- `[2026-08-23]` **`pfi` gitea org created; claude-bot is an Owner and creates repos self-serve.** Closes the repo-creation half of the credential-migration directive — `vh` is a USER namespace so no service account could ever create there. Repo creation needs `write:user` + `write:repository` + `write:organization`; `POST /users/{u}/tokens` is basic-auth only, so minting needs the account password. Default new repos to `pfi/`. (`vh/eitri-smithy` was its first tenant, then moved.)
|
||
_Archived 2026-09-09._
|
||
|
||
- `[2026-08-23]` **Booth: kept boards are deletable and link rows are prunable.** `release` on a kept card drops the sentinel so the existing × applies; `booth links` / `booth unlink <id|index>` prune one row. Rows are addressed by **content id, never position** — the board is append-only and multi-writer. **Releasing a board RESETS its TTL clock** (unlink bumps the dir mtime), so unkeep-and-wait is a 24h delay, not a delete. (`4be880f`, `0ad332b`)
|
||
_Archived 2026-09-09._
|
||
|
||
- `[2026-08-22]` **DFlash2 spec-decode measured on our own stack; `sec` promoted to it.** +18–21% accepted length and +15–18% throughput over MTP k=3, drafter proved model-agnostic across two finetunes to 0.06%, and the k=7 MTP *control* showed deeper MTP is a throughput trap.
|
||
# DFlash2 speculative decoding — measured on our own stack (2026-08-22)
|
||
|
||
Operator-driven session. **Read the epistemic labels.** During the chase we generalised from
|
||
observations that later proved wrong; this file separates what was *measured* from what remains
|
||
*hypothesis*, and records the wrong turns so nobody re-derives them.
|
||
|
||
## What DFlash2 is
|
||
|
||
A **2B draft model** (3.85 GB bf16) for speculative decoding against Qwen3.8-27B —
|
||
`incoai/Qwen3.8-27B-DFlash2`, Apache-2.0, blog `inco.ai/blog/dflash2`, upstream `z-lab/dflash`.
|
||
Block diffusion: drafts a whole 8-token block in one pass, with a candidate selector tracing a
|
||
path through per-slot top-K. Lossless (greedy matches the target).
|
||
|
||
vLLM support merged **2026-08-21 05:27 UTC** as PR **#52816** (`b389ac29`). Method string is
|
||
**`"dflash"`**, not `dflash2`.
|
||
|
||
## ✅ MEASURED — throughput and acceptance
|
||
|
||
Single instrument (`specbench.py`, 8 fixed prompts, temp 0, max_tokens 256), delta against
|
||
vLLM's own `spec_decode` counters. The MTP k=3 numbers reproduce our recorded 58.4% / 55.3%
|
||
figures exactly, which is what validates the instrument.
|
||
|
||
| seat | config | accepted tok/forward | throughput |
|
||
|---|---|---|---|
|
||
| gen (orcarouter) | MTP k=3 *(production)* | 2.753 | 114.9 tok/s |
|
||
| gen | MTP k=7 *(control)* | 3.041 | **74.0 tok/s** |
|
||
| gen | **DFlash2 k=7** | **3.254** | **131.9 tok/s** |
|
||
| sec (M.O.G.-SEC) | MTP k=3 *(production)* | 2.676 | 110.5 tok/s |
|
||
| sec | **DFlash2 k=7** | **3.252** | **130.0 tok/s** |
|
||
|
||
**⭐ The k=7 MTP control was essential and inverted the obvious read.** Going deeper on MTP
|
||
*improves acceptance* (2.753 → 3.041) while **destroying throughput** (114.9 → 74.0). Our MTP
|
||
head is a single module (`mtp_num_hidden_layers=1`, only `mtp.layers.0`, 15 tensors) run
|
||
autoregressively, so k draft tokens cost k sequential forward passes. **"Just raise
|
||
num_speculative_tokens" is a trap** — without the control I would have recommended it.
|
||
|
||
DFlash2's win is therefore **not better per-token acceptance** — our MTP is actually *better* at
|
||
position 0 (79.6% vs 75.4%). It is that block drafting makes depth nearly free.
|
||
|
||
**⭐ The drafter is model-agnostic across finetunes — 3.254 (gen) vs 3.252 (sec), a 0.06%
|
||
difference**, with superimposable per-position curves. One drafter file on `/tank` serves both.
|
||
|
||
## ✅ MEASURED — how DFlash2 runs (answers "can one drafter serve both seats?")
|
||
|
||
**EAGLE3-style coupled, not standalone.** In vLLM: `load_model(self, target_model)` binds it to a
|
||
specific target object; `pass_hidden_states_to_model=True`; `gpu_model_runner` reads
|
||
`dflash_config.target_layer_ids` → `[i+1 …]` to register auxiliary hidden-state capture on the
|
||
target at layers **5, 19, 33, 47, 61**. It even reads the target's RoPE style at load.
|
||
|
||
Consequences:
|
||
- **Weights file is shareable** (one download, both seats mount it) — gen and sec are
|
||
architecturally identical on every dimension the drafter needs: 64 layers (deepest tap 61),
|
||
hidden 5120, intermediate 17408, vocab 248,320 > mask token 248,070.
|
||
- **VRAM is NOT shareable — 3.85 GB per seat.** The drafter lives inside the target's engine
|
||
process, consuming hidden states mid-forward. Two seats are two processes; there is no
|
||
cross-process sharing mechanism and there could not be.
|
||
|
||
## ✅ MEASURED — it works on our stack, which the card does not claim
|
||
|
||
The card tests stock BF16 on an H200 with FlashAttention 3. Verified here instead:
|
||
**abliterated + NVFP4 `compressed-tensors` target ✓, Blackwell sm_120 ✓, DFlash2 CUDA graphs
|
||
captured ✓.** None of that was documented anywhere.
|
||
|
||
## 🔶 HYPOTHESIS — why our acceptance trails the published numbers
|
||
|
||
Both our targets land at ~3.25 accepted length against the card's 4.10–5.46 on stock BF16.
|
||
**Finetune drift is ruled out** — two *different* finetunes gave identical results to three
|
||
decimals. The shared variable is **NVFP4 quantization of the target**, which is mechanically
|
||
plausible (the drafter reads quantized hidden states at its five taps). Second candidate:
|
||
prompt distribution (ours general-purpose, theirs GSM8K/MATH/HumanEval/MBPP/MT-Bench).
|
||
**Neither is confirmed.** Settling it needs a BF16 target seat (~56 GB) — a real GPU window.
|
||
|
||
## ❌ RETRACTED — the "MTP head mismatch causes the degeneration" hypothesis
|
||
|
||
**Operator ruling, 2026-08-22: this hypothesis is WRONG. The degeneration lives in the un-fixed
|
||
vLLM, not in the weights.** Recorded here rather than deleted, because it was reasoned to
|
||
confidently enough that a future session could re-derive it.
|
||
|
||
**Two independent failures produced it, and the second is the instructive one:**
|
||
|
||
1. **I treated a false dichotomy as a deduction.** Having verified gen and sec run an identical
|
||
engine (same image ID `sha256:bd3236cff208…`, same live version
|
||
`0.27.2rc1.dev150+g311b3513a` read from inside both processes, same flags bar
|
||
`gpu-memory-utilization` 0.43 vs 0.44), I concluded "config is eliminated, therefore it is the
|
||
weights." That does not follow. **An engine bug present in BOTH seats is not exonerated by the
|
||
two seats being identical** — it just means the engine cannot explain a *difference*. It can
|
||
still explain the *failure*.
|
||
2. **The difference I was explaining may not exist.** The premise was a single operator
|
||
observation of sec degenerating at ~2k, made during a session with many concurrent changes.
|
||
**n=1 under heavy concurrent modification is not evidence** — see the meta-lesson below.
|
||
|
||
**What survives as fact** (measured, still true, just not causal): sec's MTP head *is*
|
||
byte-identical to `qwen38-27b-uncensored-bf16` across all 15 tensors — a stock head on a
|
||
security-finetuned body, because the `Qwen3_5ForConditionalGeneration` wrapper never loads the
|
||
head, so the finetuning could not reach it. gen's orcarouter head *was* abliterated in-band by
|
||
its author. Acceptance differs slightly (gen 58.4%, sec 55.9%). **All true. None of it shown to
|
||
cause multi-turn degeneration.**
|
||
|
||
**Current standing explanation: the degeneration is an engine bug in the un-fixed vLLM.** Both
|
||
production seats run `311b3513`, which is **172 commits behind GDN spec-decode fix #53077**
|
||
(merged 2026-08-20). `#51113` is present in that build and is therefore **necessary but
|
||
insufficient** on its own.
|
||
|
||
## ⭐⭐ META-LESSON — n=1 during a busy session is not evidence
|
||
|
||
The operator's own framing, and it generalises past this incident: **an observation made while
|
||
many things are being changed at once cannot carry a causal claim, no matter how confidently it
|
||
is reported.** Tonight that single observation became the load-bearing premise for a weights-side
|
||
hypothesis, a root-cause narrative, and very nearly a recommendation.
|
||
|
||
This is the same failure the gen-seat compose file already warns about in different words — *"a
|
||
passing probe is NOT sufficient evidence"* — inverted. That note guards against trusting a
|
||
**negative** result from a synthetic test. This one guards against trusting a **positive**
|
||
sighting from an uncontrolled session. Both reduce to: **hold the system still, or do not draw
|
||
causal conclusions from it.**
|
||
|
||
Applies equally to the "coherent to 10k" observation below — same n, same conditions, opposite
|
||
direction. Neither observation is worth more than the other.
|
||
|
||
## ⚠️ CONFOUNDED — and the "before" state is itself unreliable
|
||
|
||
sec now runs DFlash2 on a newer build and the operator reports **coherent to 10k tokens with
|
||
adversarial nonsense prompts**. ⚠ Treat this the same way as the 2k sighting it is being compared
|
||
against: **n=1, uncontrolled session, not evidence.** The comparison is weak on *both* ends.
|
||
|
||
**Two variables changed at once:**
|
||
|
||
1. **Engine**: `311b3513` → `e9d1398d`, **+259 commits, `behind_by=0`** (a strict superset),
|
||
including GDN spec-decode fix **#53077** (merged 2026-08-20) that production is **172 commits
|
||
behind**.
|
||
2. **Drafter**: frozen MTP head → DFlash2 reading live hidden states.
|
||
|
||
**Isolating it = run MTP k=3 on the same new build.** Not yet done.
|
||
|
||
**#51113 is present in BOTH builds** (verified by ancestry, `behind_by=0` each) — so the
|
||
"proper upstream fix" our compose comment credits is **necessary but insufficient**; sec ran it
|
||
and still degenerated. Related open upstream: **#53180** (quantized Qwen3.8-27B hybrid GDN + MTP
|
||
producing *silent* degenerate output, no fix), **#41884** (DFlash + prefix caching on hybrid,
|
||
IndexError, workaround is disabling one).
|
||
|
||
## ❌ WRONG TURNS — do not repeat
|
||
|
||
- **Version strings are not lineage.** The DFlash2 build reports `0.26.1rc1.dev1048` and our
|
||
production nightly `0.27.2rc1.dev150`, which *looks* like a regression. It is a setuptools_scm
|
||
tag-reachability artifact. **Use the GitHub compare API and check `behind_by`.**
|
||
- **Docker Hub push timestamps lie about source freshness.** `nightly-ba07e4a4` was *pushed*
|
||
06:12 UTC, comfortably after the 05:27 merge — but *cut* from a 03:46 commit that predates it.
|
||
**Grep the image for the symbols you need.** Believing the timestamp would have cost an RP-seat
|
||
outage to serve a model the engine could not instantiate.
|
||
- **`--max-num-batched-tokens` was not the image truncation.** Raising it 16,384 → 32,768 on that
|
||
theory changed nothing and cost ~3 GiB of peak activation, which came straight out of the KV
|
||
pool. The cap was the tokenizer (§3.14 of the playbook).
|
||
- **"1M needs YaRN, absent from config" is FALSE for the sec quant.** It is fully present:
|
||
`rope_type: yarn`, `factor: 4.0`, `original_max_position_embeddings: 262144`,
|
||
`max_position_embeddings: 1000000`. Context is a KV-memory choice, not a model limit.
|
||
|
||
## Live state — PROMOTED to the compose stack 2026-08-22
|
||
|
||
**Operator-approved after real-use testing** ("performing very well"). The experimental
|
||
standalone container is gone; `stacks/mog-sec/` is canonical and `restart: unless-stopped` means
|
||
it survives reboots. Cutover verified: **KV pool 526,617 / 1.10x — identical to the container it
|
||
replaced**, restarts 0, both gateway aliases serving, DFlash2 confirmed drafting at k=7
|
||
(231 draft tokens over 33 drafts), vision working.
|
||
|
||
⚠ **One variable was deliberately REMOVED, not carried over.** The old stack hardcoded
|
||
`PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`; the validated DFlash2 container never set it,
|
||
and playbook §3.10 records expandable_segments corrupting retained tensors elsewhere. The compose
|
||
now defaults it EMPTY (`MOG_ALLOC_CONF`). Promoting it as-was would have shipped a variable the
|
||
tested configuration did not have.
|
||
|
||
**Compose is now parameterised for the shapes that differ:** `MOG_SPEC_CONFIG` carries the whole
|
||
speculative JSON (dflash needs `"model": "/drafter"`, MTP must not have one — a method+tokens
|
||
template cannot express both), plus `MOG_MM_PROCESSOR_KWARGS`, `MOG_DRAFT_MODEL`,
|
||
`MOG_MAX_NUM_BATCHED_TOKENS`, `MOG_ALLOC_CONF`.
|
||
|
||
**ROLLBACK:** `.env.bak-pre-dflash2-20260822` and `compose.yaml.bak-pre-dflash2-20260822` on the
|
||
host; or one line — `MOG_SPEC_CONFIG={"method": "qwen3_5_mtp", "num_speculative_tokens": 3}` plus
|
||
the old `MOG_IMAGE`.
|
||
|
||
| | production sec | current |
|
||
|---|---|---|
|
||
| image | `nightly-311b3513` | `nightly-e9d1398d` |
|
||
| speculation | MTP k=3 | **DFlash2 k=7**, drafter `/tank/aimodels/qwen38-27b-dflash2-drafter` |
|
||
| max-model-len | 262,144 | **480,000** |
|
||
| KV pool | 418,218 (1.60×) | **526,617 (1.10×)** |
|
||
| images | 4096² → 16,384 tok | **2048² → ~5,125 tok** (`--mm-processor-kwargs` size cap) |
|
||
|
||
⚠ **`--gpu-memory-utilization 0.55` is the stable ceiling** while GPU1's other tenants are up.
|
||
0.58 sized KV at 594,172 then **OOM'd during CUDA graph capture** — the process reached 57.49 GiB
|
||
against ~57.6 free. Real 1M context needs ~49 GiB of KV and therefore evicting most of GPU1.
|
||
|
||
Canonical config: `stacks/mog-sec/{compose.yaml,.env.example}` in this repo.
|
||
_Archived 2026-09-09._
|
||
|
||
- `[2026-08-22]` **Quant pipeline shipped a crippled tokenizer for months — fixed at source.** `quant_mixed_nvfp4.py` baked its calibration truncation (`max_length 2048`) into every mixed-NVFP4 build; latent on old transformers, fatal on new. Both live quants corrected, pipeline now saves a source-pristine tokenizer and asserts it. Playbook §3.14. (`0755ba7`)
|
||
_Archived 2026-09-09._
|
||
|
||
- `[2026-08-22]` **`sec` retuned to util 0.52 / 420K after a runtime OOM at 0.55/480K** — `gpu-memory-utilization` is not a hard reservation; activation grows past the dummy-data profile and six vLLM containers share GPU1. Also measured: the KV pool varies ~6.6% between boots, so max-model-len must be sized against the *lower* observation. (`6e82899`)
|
||
_Archived 2026-09-09._
|
||
|
||
- `[2026-08-22]` **Max-Q 1.8× spread does NOT apply to LLM decode — measured, not argued.** ana-ml2 draws 256–266 W of 300 W under sustained 100% decode with `SW Power Cap: Not Active` and clocks pinned. Corrected to brokkr-smithy-dev after I had lent the claim credibility; 122B figure (~90–93 tok/s at 262K) stands as a straight number.
|
||
_Archived 2026-09-09._
|
||
|
||
- `[2026-08-21]` **ESH internal IPv6 live on two LANs; the Cityside v4 static is a CARRIER problem, proven.** A full gateway reboot forced a fresh DHCP DISCOVER and returned the identical CGNAT address. YaRN was already configured — "1M needs YaRN, absent" was false. → `persistent-memory.d/2026-08-22-dflash2-spec-decode.md` sibling entry in `ad21302`
|
||
_Archived 2026-09-09._
|
||
|
||
|
||
## The finding
|
||
|
||
`esh-pve-nas` (`esh-nas-pve.esteban.net`, 10.0.50.55) runs PVE root off a **USB
|
||
Disk-on-Module** — `sdq`, 7.3 GB, `ID_BUS=usb`, `ID_VENDOR=NORELSYS`, model 1081 —
|
||
carved into a 512 MB ESP + 768 MB swap + a **6 GB ext4 root** that was at **90%
|
||
(571 MB free)**.
|
||
|
||
⚠ **Operator corrected my first read: it is a DOM, not a thumb drive.** DOMs use
|
||
SLC/pSLC with a real controller, so the **284 GB written since boot is
|
||
unremarkable and wear is NOT the driver**. I had framed it as a clock ticking;
|
||
that was wrong and the correction matters. What actually justifies the work:
|
||
|
||
1. **It is on the USB bus** — a reset or re-enumeration drops the *root
|
||
filesystem* out from under a running hypervisor whose guests keep executing.
|
||
NAND quality is irrelevant to that.
|
||
2. **6 GB has no headroom** — `/usr` alone is 3.7 GB.
|
||
3. **Unmirrored**, while 928 GB of mirrored NVMe sits 96% empty.
|
||
4. **It has blocked patching for months** — the operator-visible symptom and the
|
||
real urgency.
|
||
|
||
## The patching blockage (measured)
|
||
|
||
`apt-get -s dist-upgrade`: **225 packages pending, 161 carrying `deb12uN` /
|
||
Debian-Security bumps** including `ssh 1:9.2p1-2+deb12u10`. Host sits on
|
||
`pve-manager/8.4.11` vs sibling esh-pve's **8.4.14**, with 20 weeks uptime
|
||
because it cannot take a kernel.
|
||
|
||
⚠ **Ordering is load-bearing: migrate FIRST, patch after.** The pending set
|
||
includes `proxmox-kernel-6.8.12-42-pve-signed` — ~250 MB of kernel + initramfs
|
||
landing in `/boot`, **which is on root**. Unpacking 225 packages (dpkg, perl,
|
||
glibc-adjacent) into 1.3 GB of headroom risks filling the disk mid-transaction
|
||
and wedging dpkg on a hypervisor running five guests. Partial escape hatch if
|
||
patching truly cannot wait: `apt-get -o Dir::Cache::Archives=/nvme/tmp/apt-archives`
|
||
keeps downloads off root, but the kernel still lands in `/boot`.
|
||
|
||
## Mitigation applied 2026-08-17 — root 90% → 76%
|
||
|
||
| step | effect |
|
||
|---|---|
|
||
| capped journald (`SystemMaxUse=64M`; was **fully default/uncapped**) | stops unbounded growth |
|
||
| vacuumed the journal | **freed 446 MB** |
|
||
| `apt-get clean` | 79 MB |
|
||
| `/root/neo` (2024 Intel NEO OpenCL debs) → `/nvme/tmp/root-neo-20260817/` | 259 MB — **moved, not deleted** |
|
||
| **`/var/log/journal` relocated onto ZFS** (`nvme/varlog`) | dominant writer off the DOM |
|
||
|
||
571 MB → **1.4 GB free**. All five guests stayed up; a fresh `logger` round-tripped
|
||
through the ZFS-backed journal.
|
||
|
||
⚠ **Stopping journald over SSH kills your own session** — it takes the
|
||
connection's logging path with it. The first attempt died mid-swap, leaving the
|
||
dataset staged and the move incomplete (host was never at risk; journald
|
||
socket-activated straight back). Redo as a detached `systemd-run` transient unit.
|
||
Script + reason live at `root@10.0.50.55:/root/move-journal-to-zfs.sh`.
|
||
|
||
Deliberately **not** done: moving `/var/lib/rrdcached`. With the DOM correction
|
||
the wear argument no longer justifies touching a service `pvestatd` depends on.
|
||
|
||
## The plan — split boot from root (operator's proposal, strictly better)
|
||
|
||
My first plan was a full reinstall to a mirrored-NVMe ZFS root. **The operator
|
||
proposed keeping boot on the DOM with a fallback image and putting all its files
|
||
on ZFS. That is better and I should have gotten there myself** — I had assumed
|
||
boot and root must share a device.
|
||
|
||
| | device | contents | written when |
|
||
|---|---|---|---|
|
||
| boot | DOM `sdq` | ESP + `/boot` (ext4) | only on kernel/GRUB updates |
|
||
| root | `nvme` pool | `nvme/ROOT/pve-1` | constantly, on mirrored NVMe |
|
||
|
||
Keeping `/boot` on **ext4** is the point, not a compromise: GRUB never has to read
|
||
ZFS, which matters because the `nvme` pool has `encryption`, `large_dnode` and
|
||
`zstd_compress` enabled and **GRUB cannot read those**.
|
||
|
||
**Why it beats the reinstall:** the `nvme` pool survives (no guest migration, no
|
||
`ssd`/`tank` export-import, no reinstall); downtime is **one reboot** not half a
|
||
day; **rollback is a GRUB menu entry** because the ext4 root stays untouched on
|
||
the DOM; and it retires the actual top risk — with root on NVMe a USB bus reset
|
||
mid-run no longer kills the running system. Free upside: boot environments
|
||
(`zfs snapshot nvme/ROOT/pve-1@pre-upgrade`).
|
||
|
||
**Preconditions verified already met:** UEFI + `grub-efi-amd64 2.06-13+pmx7`;
|
||
**`zfs-initramfs 2.2.8-pve1` already installed with 76 ZFS files in the running
|
||
initrd**; root only 4.3 GB to copy; swap 767 MB / 123 MB used against 125 GB RAM
|
||
(leave it on the DOM LV — **never** swap on a zvol).
|
||
|
||
**Two traps:** `canmount=noauto` on the root dataset or ZFS mounts over the live
|
||
root; and `cachefile` is `none` with a **0-byte `/etc/zfs/zpool.cache`** — pools
|
||
import by scan today, which is a coin-flip when the initramfs must find root.
|
||
Set the cachefile before rebuilding the initramfs.
|
||
|
||
Operator ruled a **cloned DOM image is sufficient** boot-path insurance (no
|
||
mirrored boot needed). `dd` it off-box before anything else; refresh after kernel
|
||
updates.
|
||
|
||
## ⚠ Blast radius — the gating constraint, invisible from the host itself
|
||
|
||
**CT 103 `esh-nas` (10.0.50.50) IS the NAS, and it runs on this host.** Two
|
||
dependents mount it over **`hard`** NFS — they do not fail, they hang unkillably:
|
||
|
||
- **esh-docker-vm** (10.0.50.45): `/mnt/books`, `/mnt/backup`
|
||
- **esh-pve** (10.0.250.35): `/mnt/pve/esh-nas`, `/mnt/pve/tank-vmbu`
|
||
|
||
Known incident shape — the only remedy for esh-docker-vm's D-state is a host
|
||
reboot, and `/mnt/books` was *deliberately* left `hard` because calibre's SQLite
|
||
risks corruption under `soft`. Quiesce both before any reboot of this host.
|
||
Recorded in `servers/esh-pve-nas/README.md` as a never-reboot-casually warning.
|
||
|
||
## Also identified
|
||
|
||
- **`esh-nas` is CT 103** on esh-pve-nas — structurally the same shape as ana-nas
|
||
being CT 109 on pfi-pve.
|
||
- **`ESH-FileBot` (CT 106, 10.0.50.70) is an empty shell** — 80 GB rootfs, six
|
||
passthrough mounts (`books`/`documents`/`music`/`share`/`pvestore`/`ssd-pvestore`),
|
||
and **nothing running but base systemd, sshd, cron, postfix** since 30 March.
|
||
That resolves the dashboard's long-standing "role TBC". Retire rather than
|
||
migrate.
|
||
- Both ESH hypervisors have **20 weeks uptime** and differing PVE patch levels.
|
||
|
||
## Staging executed 2026-08-18 — everything but the reboot
|
||
|
||
Two rerunnable elway playbooks, 0 failed steps, 17/17 verify green:
|
||
`playbooks/esh-pve-nas-stage-zfs-root.yaml` (LV surgery, `/boot` populate,
|
||
4.3 GB root rsync in 228 s, fstab) and `playbooks/esh-pve-nas-stage-bootloader.yaml`
|
||
(ZFS initramfs, grub.cfg, both menu entries, grubenv).
|
||
|
||
**`grub-install` is deliberately NOT run.** The ESP stub still points at the old
|
||
`/boot` inside the ext4 root, so the host's boot path is byte-identical to the
|
||
last 140 days and an unplanned reboot mid-staging is a non-event. Cutover is
|
||
`grub-install` + `grub-reboot pve-zfs-root` + `zfs set mountpoint=/` + reboot.
|
||
|
||
Final DOM layout: `pve-root` 6.04 G (untouched, the rollback) + `pve-boot` 512 M
|
||
(new) + `pve-swap` 256 M (was 768 M).
|
||
|
||
### The three landmines staging found
|
||
|
||
1. **The `/boot` LV had nowhere to live.** VG `pve` had **4 MB free**, and
|
||
mounted ext4 cannot shrink — freeing space from root needs a rescue boot,
|
||
which costs the "one reboot" property the design rests on. Only live source
|
||
was the swap LV. Operator chose shrink-to-256M over drop-entirely.
|
||
2. **The one-pool cachefile would have broken the NAS.** `zpool set
|
||
cachefile=… nvme` looks scoped and safe; it is the opposite. Populating a
|
||
cachefile flips the host from `zfs-import-scan` to `zfs-import-cache`
|
||
(verified: scan active, cache inactive beforehand), so a cache holding only
|
||
`nvme` leaves `ssd` and `tank` unimported at boot — and CT 103 has twelve
|
||
bind mounts spanning all three pools. Every export would come up empty and
|
||
both `hard` NFS clients would hang.
|
||
3. **`update-grub` silently emitted a pool-less `root=ZFS=/ROOT/pve-1`.**
|
||
Debian's `10_linux` builds `${rpool}${bootfs}`; `rpool` comes from
|
||
`grub-probe --target=fs_label`, which returns empty because GRUB's ZFS reader
|
||
cannot open a pool with `encryption`/`large_dnode`/`zstd_compress` — and the
|
||
probe failure is swallowed by `2>/dev/null || true`. The same feature set
|
||
that forced `/boot` to stay ext4 also corrupts the kernel command line, which
|
||
the design did not anticipate. Fixed with a `/etc/default/grub.d/zfs-root.cfg`
|
||
drop-in (last `root=` wins) plus explicit `pve-zfs-root` and
|
||
`pve-ext4-rollback` entries carrying stable ids — the auto-generated ids are
|
||
derived from pool member device paths and would shift if the mirror changed.
|
||
|
||
**The transferable lesson from (3):** the original verify grepped for
|
||
`root=ZFS=nvme/ROOT/pve-1` *appearing somewhere* in grub.cfg. Once the drop-in
|
||
was added that grep passes — while pool-less entries sit in the menu untouched.
|
||
The check that holds walks every `linux` line, takes the **last** `root=`, and
|
||
asserts it against a known-good set. **Assert the effective value, not the
|
||
presence of a substring.**
|
||
|
||
### One-shot boot, not a new default
|
||
|
||
`GRUB_DEFAULT=saved` with grubenv pinned to `pve-ext4-rollback`, and cutover uses
|
||
`grub-reboot pve-zfs-root` so ZFS is tried **exactly once**. A failed ZFS boot
|
||
returns to ext4 by itself on the next reboot — no console, no hands. That matters
|
||
more here than on a normal host: a hang at an initramfs prompt takes CT 103 down
|
||
and the NFS clients hang rather than fail. Only after a second clean ZFS boot
|
||
should the saved default move.
|
||
|
||
### Off-box artifacts (`nh3-dev:~/backups/esh-pve-nas/`)
|
||
|
||
- `dom-sdq-20260818.img.zst` — full DOM image, 7,837,450,240 B raw / 2.38 GiB
|
||
compressed, zstd XXH64 verified. ⚠ **Crash-consistent, not clean** — the root
|
||
LV was live during the read, so a restore replays the ext4 journal. Not
|
||
fixable with an LVM snapshot: the VG has no free extents.
|
||
- `bootchain-20260818.tar.gz` — clean, consistent tar of `/boot` + ESP (88 MB,
|
||
644 entries, full proxmox shim/grub EFI chain). This is the higher-quality
|
||
boot-chain artifact; the dd image is the belt-and-braces full-device restore.
|
||
- `pve-config-snapshot-20260818T051*.tar.gz` — 147 entries incl. the new
|
||
grub.cfg, fstab, LVM/ZFS/blkid state.
|
||
⚠ Building this the first time produced a **corrupt archive**: `pvs; vgs; lvs >
|
||
file` redirects only the last command, so `pvs`/`vgs` output leaked into the
|
||
tar stream on stdout. Group with `{ …; } > file`.
|
||
|
||
Runbook: `docs/runbooks/esh-pve-nas-boot-migration.md`. Earlier config snapshot at
|
||
`nh3-dev:~/backups/esh-pve-nas/pve-config-snapshot-20260818T043027Z.tar.gz` (0600,
|
||
sha256 `dc312793d027dc43…`) — `/etc/pve`, network, fstab, apt, authorized_keys plus
|
||
captured `zpool`/`zfs`/`disk-by-id`/`lsblk`-with-serials/`pvesm`/`dpkg` state and
|
||
every guest config. **The newest on-disk copy before this was June 2024.**
|
||
Commits `2275e11`, `3e31175`, `8ddc87c`.
|
||
_Archived 2026-09-02._
|
||
|
||
# irv-ml1 weight cleanup (782 GB) + Homepage brought under version control
|
||
|
||
Two unrelated housekeeping jobs from the same session, both with durable lessons.
|
||
|
||
## irv-ml1 — 782 GB reclaimed
|
||
|
||
Root was at **92%** (148 G free), storetank **86%**. Now **64%** (635 G free) and
|
||
**74%** (477 G).
|
||
|
||
**Tier 1 — dead weights, 286 GB.** `/storetank/llm-models/Storage` (**217 G**, 22
|
||
GGUF repos, atimes Jan–May **2025**) plus `models--MaziyarPanahi--WizardLM-2-8x22B-GGUF`
|
||
(44 G) and `models--h2oai--h2ogpt-4096-llama2-13b-chat` (25 G). The 217 G pile had
|
||
**zero consumers** — no llama-swap, no llama.cpp, no textgen running *or installed*,
|
||
not even a stopped container. The fleet moved to vLLM/NVFP4 seats on ana-ml2 and
|
||
nobody opened that shed for 15 months. Re-verified the consumer check immediately
|
||
before deleting, not just during the audit.
|
||
|
||
**Tier 2 — regenerable caches, 194 GB.** `uv` 65 G + 60 G, `pip` 31 G + 8.7 G,
|
||
`modelscope` 29 G (mtime **2024-04-23**).
|
||
|
||
**Tier 3 — retired stacks, 302 GB** (operator: "those were old days… we're a UV
|
||
fleet now"): `/opt/fluxgym` 64 G, `/opt/ComfyUI` **native** 41 G, `/opt/stablediffusion`
|
||
28 G, `/opt/alltalk` 19 G, `/opt/o-textgen` 12 G, `/opt/sdnext` 3 G, `/opt/xttsv2`
|
||
1.8 G, `tabbyAPI` 3.1 G, **`miniconda3` 130 G**.
|
||
|
||
### The lesson: one dead-looking app pinned three delete targets
|
||
|
||
`lsof +D` per path found **PID 281192 — fluxgym, up 42 days, listening on
|
||
0.0.0.0:7860** — holding 15 open handles into `miniconda3/envs/vllm` (stale
|
||
opencv wheels) **and 41 into `/opt/ComfyUI`**. Deleting miniconda underneath it
|
||
would have half-broken a live listener in a way that surfaces only at its next
|
||
restart. Stopped it by **explicit PID** (never `pkill -f` — handle-blind),
|
||
verified :7860 released and handles at zero, *then* deleted.
|
||
|
||
⚠ **Name collision that nearly cost a production service:** `/opt/ComfyUI` is a
|
||
*native* install; the ComfyUI that actually serves (:8188, 200 OK) is the **Docker
|
||
`mmartial` container** reading `/worktank/comfyui`, and arbo's `comfy_engine` runs
|
||
from uv. Checking open handles **per path** is what separated them — the earlier
|
||
"not running" read would have deleted the wrong thing.
|
||
|
||
⚠ **`df` lags an async ZFS free.** Right after the 217 G delete, storetank still
|
||
showed 86%/261 G — the exact shape of a snapshot-retention problem. It wasn't
|
||
(`zfs list -t snapshot` empty); second check showed 477 G at 74%.
|
||
|
||
All 16 containers and both systemd services verified healthy afterward.
|
||
|
||
## Homepage under version control
|
||
|
||
`ghcr.io/gethomepage/homepage` on **esh-docker-vm:5100** was the one stack whose
|
||
config lived only on the host. Its version history was **six hand-rolled
|
||
`services.yaml.bak-*` files**. Now `stacks/homepage/` (compose + 9 config files +
|
||
`.env.example` + README), deployed via `deploy-stack.sh`; `.bak` files gone.
|
||
105 cards across 19 groups, no empty groups.
|
||
|
||
⚠ **I claimed ana-docker wasn't wired into `docker.yaml`. It already was** —
|
||
`ana-pfi-docker: 10.250.50.70` — and I built a theory on a `tail` that truncated
|
||
the top of the file. All five engines were discovering correctly the whole time.
|
||
|
||
**Corrections landed:** `ANA-Firewall` said "Fortigate 81F" → it is a
|
||
**FortiGate-80F, FortiOS 7.2.10** (verified against the device); `NH3-Ansible` →
|
||
**NH3-ExtDev** (10.100.50.42 is nh3-extdev, successor to the retired nh3-ansible);
|
||
dropped the `UltraSeedbox` layout group (nothing provides it).
|
||
|
||
⚠ **`HOMEPAGE_ALLOWED_HOSTS` matches host AND port.** `10.0.50.45` did **not**
|
||
cover `http://10.0.50.45:5100/` — the container log carried `Host validation
|
||
failed` while the Traefik hostnames worked. Fixed; direct IP:port now 200.
|
||
`.env` was **mode 644** holding Plex + Jellyfin API keys → now 600.
|
||
|
||
⚠ **Homepage renders client-side** — grepping the served HTML to verify a config
|
||
change gave two false readings (a stale prerender, then an empty page).
|
||
`GET /api/services` is the honest instrument, and config changes need a
|
||
**recreate**, not a restart (a restart keeps the cached render in the writable
|
||
layer).
|
||
|
||
⚠ `deploy-stack.sh` runs rsync with `--delete` — alongside the six `.bak` files it
|
||
also removed a host-side `README.md` in the conf dir. Content survived (it is now
|
||
in the repo README) but that was a side effect, not a plan.
|
||
|
||
Commits `c5beeac`, `d1f4f1c`. See also [[2026-08-17-fleet-ipv6-mesh]].
|
||
_Archived 2026-09-02._
|
||
|
||
- `[2026-08-17]` **Gen seat swapped to `absolute-heresy` — and the three bugs the swap exposed are worth more than the swap.** Candidate `MuXodious/Qwen3.8-27B-absolute-heresy` (Heretic v1.4.0 + SOMPOA, T377) beat the incumbent on refusals AND KL simultaneously, which is the unusual part — those normally trade off. Validated on the probe port per operator ruling, promoted, all 7 aliases green. **Durable lessons banked:** (1) **A CPU-only MTP head hash can replace the ~56 GB bf16 acceptance gate.** The `Qwen3_5ForConditionalGeneration` wrapper never loads the MTP head, so PEFT merges / Heretic runs / llm-compressor passes all leave `mtp.*` pristine — hashing it against a head we have already measured (the incumbent's, 47.7%) answers the question for free. Predicted 47.7%, measured 47.2%. Saved downing meromero. Tool: `services/gen-seat-mixed-quant/compare_mtp_head.py` (hash bf16 via **uint8 reinterpret** — numpy has no bfloat16). (2) **`post_quant.py` assumed a standalone `model-mtp.safetensors`**; a full checkpoint keeps `mtp.*` in a NUMBERED shard, so the copy silently no-op'd while the index was still rewritten to point at a file that never existed — 15 unresolvable tensors behind a correct-looking tensor count. Its own FAILED-CHECKS assertion caught it; **that is why the check exists rather than an assumption**. Fixed to extract. (3) **A probe that does not mirror the live seat manufactures failures.** `serve_probe.sh` hardcoded `:latest` (seat is a pinned nightly for #51113), had no tool-call/reasoning parsers, and its `--speculative-config` JSON died twice on quoting — **bash BRACE-EXPANDS `{"a":1,"b":2}` on the comma** unless single-quoted at the REMOTE shell. Adding the seat's flags took the surface test from 5/6 to **6/6**; the "tool calling broken" result was pure probe config. Commits `7997f11`,`254c588`,`2c36028`,`b0c2d3d`,`993421b`.
|
||
_Archived 2026-09-02._
|
||
|
||
# Fleet IPv6 state + the real VPN topology (verified 2026-08-17)
|
||
|
||
Written because the operator expects to reference this "before too long" — the
|
||
driver is an **ESH fiber install landing 2026-08-18 that puts the house behind
|
||
CGNAT**, which breaks Site Magic on IPv4 and makes IPv6 load-bearing rather than
|
||
a nice-to-have.
|
||
|
||
## Why IPv6 suddenly matters: CGNAT at ESH
|
||
|
||
New ESH fiber (installing 2026-08-18) hands out a **CGNAT IPv4**. Site Magic —
|
||
the UniFi-to-UniFi SD-WAN mesh tunnel that currently links NH3 ↔ ESH — needs a
|
||
reachable endpoint, and a CGNAT address is not one. **IPv6 is the escape hatch:
|
||
a global v6 address on each UDM restores a routable endpoint pair without
|
||
depending on the ISP's v4 at all.** That, not the WireGuard RA mesh, is the
|
||
most likely first consumer of fleet IPv6.
|
||
|
||
Operator expects addresses at **Anaheim shortly** and **ESH 2026-08-18**.
|
||
|
||
## The topology — as VERIFIED, not as assumed
|
||
|
||
Three transports, three different technologies. Do not describe this as "a
|
||
WireGuard mesh"; a prior session did and was corrected.
|
||
|
||
| Link | Transport | Evidence |
|
||
|---|---|---|
|
||
| NH3 UDM ↔ ESH UDM | **Site Magic** (`vpn_type: sdwan-mesh-tunnel`) | UDM `networkconf`, carries all 7 ESH subnets |
|
||
| Colo FortiGate ↔ NH3 UDM | **IPsec IKEv2** | FG `pfi-ana-nh3` → 70.230.226.88, **158M pkt rx / 165M tx** — the fleet workhorse |
|
||
| Colo FortiGate ↔ ESH UDM | **IPsec IKEv2** | FG `ana-to-eshudm` → 70.181.90.232, 53K/56K pkt |
|
||
| Remote-access VPN | **WireGuard, host-based on `ana-wg`** | see below |
|
||
|
||
**WireGuard is an RA (remote-access) convention only — it is NOT the site mesh.**
|
||
It runs on `ana-wg` (LXC 113, Debian 12, 10.250.50.252), interface `wg0`,
|
||
**UDP 31337**, tunnel subnet `10.30.10.0/24`, 3 peers (`tc2-mac`, `vh-iphone`,
|
||
`vh-mba26`). Reached from outside via a FortiGate VIP `wg-to-ana-wg`:
|
||
`38.120.12.42:31337/udp → 10.250.50.252:31337` on wan1.
|
||
|
||
**The FortiGate never terminates WireGuard — it port-forwards to the host that
|
||
does.** FortiOS 7.2.10 has no native WireGuard (Fortinet added it in 7.4), so a
|
||
session that reads "colo + WireGuard" and concludes the edge must be upgraded is
|
||
chasing a non-problem. Do not re-derive this.
|
||
|
||
## Per-site IPv6 state (2026-08-17)
|
||
|
||
| Site | Edge | IPv6 |
|
||
|---|---|---|
|
||
| **NH3** | UDM SE | **WAN live** — `2600:1700:b25:c110::48` via DHCPv6 on ATTFiber. All 5 LANs `ipv6_interface_type=none` |
|
||
| **Anaheim colo** | FortiGate-80F, FortiOS 7.2.10 | **None.** `diagnose ipv6 address list` → only loopback `::1`; every physical iface `ipv6: ::/0` |
|
||
| **ESH home** | UDM Pro Max | **None.** Both WANs `wan_type_v6=disabled`; link-local only |
|
||
|
||
## AT&T delegates exactly ONE /64 at NH3 — proven, not assumed
|
||
|
||
`2600:1700:b25:c11f::/64`. **One.** Not the /60 the addressing pattern suggests.
|
||
|
||
The proof matters because the naive read is wrong: the WAN sits at `c110::48`
|
||
and the LAN got `c11f::1/64`, which looks exactly like slot 15 of a /60 spanning
|
||
`c110`–`c11f`. It isn't. Forcing the prefix ID from auto to a manual `0` — which
|
||
on a real /60 would relocate the LAN to `c110::1/64` — left the subnet at
|
||
**`c11f::1/64`, stable across a 4-minute settle**. Two different prefix-ID
|
||
settings yielding the same /64 is the signature of a single-/64 delegation.
|
||
|
||
**Consequence: exactly one VLAN can have IPv6 at NH3**, unless AT&T enlarges the
|
||
delegation. If Site Magic-over-v6 is the goal that is fine — Site Magic needs a
|
||
routable address on the *WAN*, not a LAN prefix.
|
||
|
||
The controller never exposes the PD size directly (`wan_dhcpv6_pd_size_auto:false`
|
||
with no size field alongside), so the prefix-ID test is the only read-only-ish way
|
||
to establish it from the API.
|
||
|
||
## What a v6 mesh actually requires (and what it does NOT)
|
||
|
||
**Does NOT require prefix delegation.** PD hands addresses to LAN *clients*. Both
|
||
Site Magic and WireGuard need a routable address on the router/host WAN side, plus
|
||
inbound reachability. Enabling PD on a LAN is orthogonal — this was tested and
|
||
then reverted.
|
||
|
||
**ana-wg's WireGuard socket is ALREADY dual-stack** — `ss` shows both
|
||
`0.0.0.0:31337` and `[::]:31337`. It will accept IPv6 peers with **no WireGuard
|
||
reconfiguration** once (a) the host holds a routable v6 address (today: link-local
|
||
`fe80::be24:11ff:fed7:e4b7` only) and (b) the FortiGate passes inbound UDP 31337
|
||
over v6 — the existing VIP is v4-only (`extip 38.120.12.42`).
|
||
|
||
**NH3 UDM's own WG server is v4-pinned** — `wireguard_interface_binding_mode_ip_version: 'v4'`,
|
||
one field to flip when wanted.
|
||
|
||
**Inbound v6 is default-deny and that held without intervention.** The UDM runs
|
||
the **zone-based** firewall (66 policies). ⚠ The legacy `rest/firewallrule`
|
||
endpoint returns **0 rules** on this box — a quick check there reads as "no IPv6
|
||
rules exist," which is wrong and alarming. Use
|
||
`v2/api/site/default/firewall-policies`. WAN→LAN default is `Block All Traffic`
|
||
for both families with `Allow Return Traffic`; the only v6-specific allows are
|
||
link-local plumbing (ND solicit/advert, RA, DHCPv6).
|
||
|
||
## The stability problem — design around it up front
|
||
|
||
All three endpoints will hold **dynamic** addresses (NH3's came via DHCPv6 IA_NA,
|
||
not a static assignment). A three-way mesh where every node can move is fragile;
|
||
WireGuard tolerates one roaming end, not all of them.
|
||
|
||
The fleet already solves this on the v4 side — IPsec peers use **hostnames**
|
||
(`ana-fw.phasefinal.com`, `nh3.phasefinal.com`), not raw IPs. **Extend that to
|
||
AAAA records** and dynamic prefixes stop mattering. infra-ops holds the fleet
|
||
Cloudflare DNS-edit token, so this is self-serve.
|
||
|
||
## Access recipes (cost a prior session real time)
|
||
|
||
- **UniFi UDMs** — `X-API-KEY` from the vault (`secret get unifi/pfi-udmse-api-key`,
|
||
`unifi/esh-udmpm-api-key`) against `https://<ip>/proxy/network/…`, `curl -sk`.
|
||
Classic `api/s/default/rest/networkconf` + `stat/device` carry everything here.
|
||
Writes are `PUT …/rest/networkconf/<_id>` with the **full** object.
|
||
- **`ana-wg` is `root@`, NOT `infra-ops@`** — the shared infra-ops key is refused
|
||
(`Permission denied (publickey,password)`). `servers/ana-wg/ssh-target` says
|
||
`root@10.250.50.252`; believe it.
|
||
- **FortiGate** — paramiko via `uv run --with paramiko` (no sshpass on nh3-dev),
|
||
password `secret get fortigate/ana-gw-infra-ops-password`. ⚠ **A fixed-duration
|
||
`drain()` hangs the session**; read until the `ana-gw #` prompt and answer
|
||
`--More--` with a space. Two invocations timed out at 3 min before this was fixed.
|
||
|
||
## Changes made and reverted this session
|
||
|
||
- **Enabled PD on `nh3-iot` (VLAN 90)** to measure the delegation, then **REVERTED
|
||
on operator instruction** — all 5 NH3 LANs are back to `ipv6_interface_type=none`,
|
||
verified. Pre-change snapshots kept in the session scratchpad only (ephemeral).
|
||
- **`ana-wg` WireGuard key material was world-readable** — `wg0.conf` (server
|
||
private key + 2 peer PSKs), `keys/*_priv`, `keys/*_psk`, and `configs/*.conf`
|
||
(client configs carry private keys) were all mode **644**. Now **600**, and
|
||
`keys/` + `configs/` dirs **700**. `wg-quick@wg0` stayed active, 3 peers intact —
|
||
WireGuard holds keys in kernel memory, so no restart was needed. The parent
|
||
`/etc/wireguard` was already 700, which capped the real exposure to root-capable
|
||
contexts inside the LXC — but the modes were still wrong.
|
||
|
||
---
|
||
|
||
## CORRECTION (recorded 2026-08-24): "AT&T delegates exactly ONE /64" is the
|
||
## per-REQUEST truth, not the total — eight /64s exist and are unclaimed
|
||
|
||
The section above concludes AT&T hands out a single `/64` and that the
|
||
`c110`/`c11f` pattern reading as a `/60` was a misread. **That conclusion was
|
||
itself superseded later in the same session, and the correction never made it
|
||
into memory** — it survived only in the session transcript, and was recovered
|
||
2026-08-24 while assessing a proposal to grab more prefixes.
|
||
|
||
Reading the **BGW's own LAN statistics page** gave the whole picture:
|
||
|
||
```
|
||
BGW WAN v6 2001:506:70b2:8958::1 <- AT&T's transit prefix
|
||
BGW LAN v6 2600:1700:b25:c110::/64 <- the BGW keeps this for itself
|
||
Delegated 2600:1700:b25:c11f::/64 <- what the UDM got
|
||
```
|
||
|
||
**The BGW holds the `/60` and rations it**, keeping `c110`–`c117` for itself and
|
||
delegating from the top down — the UDM got `c11f`, the last one. So
|
||
`c118`–`c11f` are **eight delegatable /64s that genuinely exist and are yours**,
|
||
sitting unclaimed.
|
||
|
||
Both observations are compatible, which is why the first one looked conclusive:
|
||
the prefix-ID test only carves *within* a delegation already held, so a UDM
|
||
holding one `/64` cannot move it no matter what prefix-ID you set. The BGW
|
||
issues **one `/64` per IA_PD request**, and **UniFi solicits exactly once**.
|
||
|
||
**Consequence — the ceiling is the requester, not the carrier.** More prefixes
|
||
need more IA_PD requests (multiple IAIDs, or multiple client DUIDs), which the
|
||
UDM will not do. That is what makes a separate DHCPv6-PD client viable, and it
|
||
is why "ask AT&T for a bigger delegation" may be aimed at the wrong party: this
|
||
looks like BGW rationing rather than a provisioning-profile limit.
|
||
|
||
Live state at correction time: `wan_dhcpv6_pd_size: 64`, `wan1 v6
|
||
2600:1700:b25:c110::48`, all 5 NH3 LANs still `ipv6_interface_type: none`.
|
||
|
||
### ⛔ CLOSED 2026-08-24 — operator ruling, do not re-raise
|
||
|
||
The seven unclaimed `/64`s stay unclaimed. Two facts close it:
|
||
|
||
- **The BGW has no IP-passthrough mode.** Operator confirmed, and we hold admin
|
||
on it — so the cheap path (let the UDM take the `/60` directly and carve it
|
||
natively, as it already does at ESH) does not exist here.
|
||
- **The only remaining route is a multi-DUID DHCPv6 client on a VM**, which
|
||
requires re-cabling to reach the BGW's DHCPv6 server, split-stack routing
|
||
(UDM for v4, VM for v6), and — the actual cost — **rebuilding the whole IPv6
|
||
firewall policy in nftables on that VM**, because routing v6 around the UDM
|
||
bypasses its zone firewall entirely and would leave every LAN host globally
|
||
reachable.
|
||
|
||
Operator's call: not worth it. **NH3 LANs stay `ipv6_interface_type: none`.**
|
||
Do not re-propose on the strength of "there are seven free prefixes" — the
|
||
prefixes are real, the firewall rebuild is why nobody wants them.
|
||
_Archived 2026-09-02._
|
||
|
||
- `[2026-08-17]` **Gen-seat multi-day degeneration RESOLVED — two compounding real causes, not one; the meta-lesson is "a mitigation that HELPS but doesn't FIX means a second cause, not a wrong one."** vLLM `qwen3_5_mtp`×GDN bug (#51113, real, fixed by nightly) + AEON full-W4A4 being lowest-fidelity (W4A4<W4+FP8<W4+bf16) → ~15-20% stochastic degeneration. Fixed by mixed FP8-attn build on pinned nightly. AEON purged. Also banked: **stochastic (~15-20%) degeneration is invisible to a small synthetic probe — n=1 "clean" validated THREE non-fixes (MTP-off, APC-off, nightly-alone) that all failed in real use; get the operator's real transcript, do not trust your own probe.** Full → `docs/pfi/model-quantization-playbook.md` §3.8 (+ §3.7 MTP-multi-turn). Commits `d28a371`,`2f2bbce`,`2185964`.
|
||
_Archived 2026-09-02._
|
||
|
||
- `[2026-08-17]` **Lobe Chat chosen over Open WebUI (weight: 143 MB vs 1.8 GB) + stood up on esh-docker-vm; scoped LiteLLM key blocks paid models; System-Agent `gpt-5-mini` default repointed via env.** TTS env-vs-UI resolved as a split (endpoint env-driven, voice/model UI-only). tts-dev onboarding closed both directions; ballad/verse aliased so no voice can 404 the router. Commits `e9362de`,`163a725`,`cac75cb`,`933253d`,`25fa18e`.
|
||
_Archived 2026-09-02._
|
||
|
||
- `[2026-08-17]` **LiteLLM upgraded v1.91.0→v1.97.0 (RC-avoided on the fleet gateway) + the 6 GB spend-log DB purged & capped** (`store_prompts_in_spend_logs:false` + 7d retention). Interpreted "get rid of the db" as the spend-log DATA not the database (keys/config live in it). Commit `01b5ad9`.
|
||
_Archived 2026-09-02._
|
||
|
||
- `[2026-08-18]` **esh-pve-nas migration STAGED — and staging is where three landmines surfaced, none of which the plan predicted.** (1) The runbook's `/boot` LV had **nowhere to live**: VG `pve` had 4 MB free and mounted ext4 cannot shrink, so the space came from the 768 MB swap LV (operator's call: shrink to 256 MB, not drop). (2) The runbook's `zpool set cachefile=… nvme` would have **broken the NAS** — populating a cache flips the host to import-by-cache, and a one-pool cache leaves `ssd`+`tank` unimported under CT 103's twelve bind mounts. (3) **`update-grub` silently emitted a pool-less `root=ZFS=/ROOT/pve-1`**, because GRUB's ZFS reader cannot open a pool with `encryption`/`large_dnode`/`zstd_compress` and the probe failure is swallowed. All three were caught by *verify steps that asserted effective state*, not by reading the plan. → `persistent-memory.d/2026-08-17-esh-pve-nas-dom.md`
|
||
_Archived 2026-09-02._
|
||
|
||
|
||
- `[2026-08-16]` **Fable-Fusion 711 cuts cold-framing refusals 92.5% → 15.8%; refusal is MONOTONIC IN FRAMING, and DS v1.0's problem is that she was never abliterated.** brokkr-smithy-dev supplied the framing that reproduces (`01M05M48R4RSZF9D8KT7RR55EJ`): a **bare assistant-mode instruction** — no character card, no permission preamble. Three-arm A/B, same harness, same classifier: permission framing **DS 0.0% / FF 0.0%** (n=75); plain character cards **DS 1.4% / FF 0.0%** (n=74); bare instruction **DS 92.5% (37/40) / FF 15.8% (6/38)**. Per-axis DS→FF: incest 100→20, non-con 100→20, bestiality 100→25, necrophilia 100→40, gore 100→**0**, consensual 80→20, dubcon 80→**0**, self-harm 80→**0**. DS refused **25/25** on the five axes brokkr flagged. Root cause: `ReadyArt/Dark-Scarlett-v1.0-27B` is a plain finetune of stock `Qwen/Qwen3.6-27B` carrying **NO abliteration** — the base refusal machinery is intact, so cold prompts revert to safety-tuned Qwen3.6. FF is Heretic-**ablated** (structural), which is why it holds. ⚠ **RETRACTED 2026-08-16 — my "arm-3 92.5% exceeds brokkr's 62.5%" comparison was INVALID.** His diff against his own artifact showed my `battery-instruct.yaml` reproduces only his **`creative` class — 8 of 16 axes**; it dropped all 5 `operational` (violence/incite, crime/fraud, cyber/malware, selfharm/methods, privacy/stalk) and all 3 `meta` (meta/sysprompt, meta/ignore, meta/dan), and added 2 controls he never had, at k=5 vs his k=2. **His 62.5% pools all 16 axes; my 92.5% is creative-only — different denominators, not a delta.** Cause: I rebuilt his shape from his *message*, and the `class` field lives in the artifact, not the prose. **Lesson: reconstructing a peer's instrument from their description reproduces what they described, not what they ran — diff against the artifact before claiming comparability.** ⚠ **Known battery bug left unfixed for comparability:** DS's arm-3 control gate failed at 11% because `ictrl-reunion` pairs "explicit / do not fade to black" with *brothers*, which DS reasonably read as an incest request; FF did not. `ictrl-storm` is the clean control. Commit `b9e68c3`.
|
||
_Archived 2026-09-01._
|
||
- `[2026-08-15]` **Quant lessons consolidated into `docs/pfi/model-quantization-playbook.md` — the durable home; read it BEFORE any requant.** Survey found quant knowledge scattered across 18 files in 4 trees, with **three** documents having independently written overlapping "landmines" sections (the loader-class trap alone was rediscovered 3×). Playbook owns the **transferable** lessons (scheme choice, landmines, acceptance gate + its 3 measurement traps, hardware/co-residency); per-model artifacts are demoted to worked examples that link up. Carries a **superseded-claims table** — which immediately earned itself: the heretic2 runbook's "use modelopt, compressed-tensors can't load the BF16 MTP" is **false** (the cause was the missing `re:^mtp.*` ignore, not the format) and would have sent the next session down the modelopt dependency-hell path; that runbook now carries a stale-warning header. Maintenance rule in `CLAUDE.md`: model-agnostic → playbook, model-specific → stays put, wrong claim → dated superseded row, never a silent edit. Motivated by Qwen3.8 having just released — the next model swap needs a requant. Commit `a91cc3f`.
|
||
_Archived 2026-09-01._
|
||
- `[2026-08-15]` **Operator ruling: the gen seat's +1.7% perplexity is an acceptable price for the speed — SETTLED, don't re-litigate.** Precise attribution for future reasoning: it is the **activation-quantization** cost (W4A4 MLPs + FP8 attention vs BF16 activations), not an MTP cost — PPL was measured with speculative decoding **off** on both builds, so MTP was not in the loop. Turning MTP off would not recover it; only reverting the quant would (rollback = one `.env` line, old build intact at `…/qwen38-27b-uncensored-nvfp4`).
|
||
_Archived 2026-09-01._
|
||
- `[2026-08-15]` **gen seat requanted to mixed NVFP4+FP8 (+18% decode) + char-rp Gemma-4 tool-calling fixed.** The queued "W4A8" (NVFP4 weights + FP8 activations) is **not servable** — vLLM 0.24 allows NVFP4 weights with only A16 or A4; FP8 activations ValueError at load, and `CompressedTensorsW4A8Fp8` is INT4-weights + sm90-exact (closed on Blackwell twice). FP8 must enter **per-layer-group**. Also: the handoff's "~68 tok/s" baseline didn't reproduce — cache-busted, the incumbent already did **80.12** (≈ the stated W4A8 target), so the premise needed re-measuring before any work. Shortcut: `unsloth/Qwen3.8-27B-NVFP4` was already on-box → served as a probe, measured **+19.1% at identical acceptance**, which both proved the gain was real and handed over the reference recipe. Replicated it on the abliterated weights → **80.12→94.53 tok/s, acceptance unchanged, +1.7% PPL, abliteration 4/4, weights −19%**; surface 6/6 live, 7 aliases routing. char-rp had **no** tool parser at all (every tools request 400'd) → `gemma4` tool + reasoning parser + a **mandatory** `enable_thinking:false` (the parser defaults it True → null `content` for all RP prose; proven byte-identical prompt before deploying). Commits `b8f0f4c`, `74f596b`. Foot-guns banked (llm-compressor prunes unmatched `ignore` entries → the 0%-MTP bug, **fired on this run**; prompt_logprobs uniform under spec-decode; 0600 `.env` silently no-ops compose; GPU0 is zero-sum). → `persistent-memory.d/2026-08-15-gen-seat-mixed-requant.md`
|
||
_Archived 2026-09-01._
|
||
- `[2026-08-15]` **Uncensored gen seat: JonathanColetti/Qwen3.8-27B-Uncensored deployed as `gen-seat`/`vllm-gen` (NVFP4 W4A16 + grafted MTP, 262K); 7 aliases repointed; the definitive `re:^mtp.*`-ignore fix.** 0%-MTP-on-quant (twice) was NOT the abliteration/scheme — the grafted bf16 MTP was missing from `quantization_config.ignore` (vLLM loaded it as quantized → uninitialized). Full arc, the working pipeline, VRAM budget, unsloth speed decomposition, modelopt dead-end. → `persistent-memory.d/2026-08-15-uncensored-gen-seat.md`
|
||
_Archived 2026-09-01._
|
||
|
||
- `[2026-08-05]` **Fleet CI resilience flip (`DEFAULT_ACTIONS_URL=self`) — attempted end-to-end, PARKED on a runner action-fetch auth blocker; infra-ops to research it (operator-directed, deferred, NOT now).** 7 gitea action mirrors staged public+populated (orgs `actions`+`astral-sh`); the flip resolves `uses:` correctly but act_runner v0.6.0 can't authenticate its fetch to gitea 1.26 ("Invalid username or token. Password authentication is not supported"). Reverted (CI back on github default); `REQUIRE_SIGNIN_VIEW=false` KEPT as a standing change (operator, internal WG net). Full endeavor, the reliable nh3-dev-egress + git-SSH mirror method, exact config state, smoke method, and next step → `persistent-memory.d/2026-08-05-ci-flip-parked.md`
|
||
_Archived 2026-09-01._
|
||
- `[2026-07-31]` **muninn-gate (#377 ingestion front door) BUILT + DEPLOYED + healthy on corviduo-dev:8090.** First-boot acceptance passed (watcher:running:true proves ingestion_root byte-identity); submit path deferred to the mimir-inbox era. Full wiring (uid-1000, state-volume mount, staging path-agreement, BuildKit-secret build, deferred repoint + operational guards) → `persistent-memory.d/2026-07-31-muninn-gate-deploy.md`
|
||
_Archived 2026-09-01._
|
||
|
||
- `[2026-08-05]` **worldtree herald re-nudge bug root-caused → forseti shipped althing-core v2.1.2 (`d5d33df`, deployed on nh3-dev).** `herald.py:363` rendered the wake command from the empty *fresh* mail set on the re-nudge path (should be `deliver_msgs`) → `messages[0]` IndexError → un-suppressed outer catch-all → 7s crash-loop for 9 days on worldtree-codex's pane route (mimir-dev surfaced it; I traced it from the editable source). Fix + `render_command` empty-guard + outer log-suppress + 3 tests + contract amendment, all forseti's. **nh3-extdev herald 2.1.2 upgrade DEFERRED** (operator, not-now): extdev is a WHEEL install (not editable), unexposed (no pane routes); the verified 2.1.2 wheel is staged on nh3-dev `/tmp` (sha256 `003508…cef27`) — `uv tool install --force` + restart both heralds when un-parked. extdev herald-unit provenance resolved (operator-authorized 2026-07-25 via forseti relay; recorded in this file's 07-25 herald-install entry). auto-memory `reference_nh3_dev_althing_herald`.
|
||
_Archived 2026-08-28. Its deferred item — the nh3-extdev herald 2.1.2 upgrade — is closed: extdev went 2.1.0 -> 3.0.0 -> 3.1.1 at the v3 cutover, so the staged v2.1.2 wheel is moot._
|
||
|
||
|
||
# eRP dual-seat overhaul — MeroMero-v2 + Dark-Scarlett, NVFP4A16 @ 256K on ana-ml2
|
||
|
||
`[2026-08-12]` Replaced the two legacy char-rp seats with home-quantized NVFP4A16 vLLM
|
||
seats. Operator-driven, end to end this session.
|
||
|
||
## What landed
|
||
|
||
| Seat (LiteLLM alias) | Model | Role | GPU | Context |
|
||
|---|---|---|---|---|
|
||
| `char-rp` (:8016) | **G4-MeroMero-v2-31B** (Gemma-4) | non-thinking PROSE, **multimodal (vision)** | GPU0 | 256K @ 2.07× (util 0.52) |
|
||
| `char-rp-reasoning` (:8018) | **Dark-Scarlett-v1.0-27B** (Qwen3.6) | THINKING (default) | GPU1 | 256K @ 1.62× (util 0.44) |
|
||
|
||
- Both **NVFP4A16 weight-only** (llm-compressor, `compressed-tensors`), `--kv-cache-dtype fp8`.
|
||
- Replace: `char-rp-gguf` (Magidonia-24B GGUF/llama.cpp, :8016) + `heretic2-charrp-reasoning`
|
||
(DavidAU Qwen3.6-27B-Heretic2 modelopt NVFP4+MTP, :8018). Old stacks/containers **stopped +
|
||
retained** for rollback.
|
||
- Compose-ified: `stacks/meromero-charrp` + `stacks/darkscarlett-charrp-reasoning` (ana-ml2
|
||
`/opt/docker/compose/`, mirrored to eshpfi, commit **`f08b6cb`**) → survive reboot.
|
||
- Research that drove picks: `docs/pfi/erp-thinking-finetunes-2026.md` (from the `gecko-65` Booth).
|
||
|
||
## Load-bearing lessons (the whole point of this file)
|
||
|
||
1. **Load via the ConditionalGeneration WRAPPER class, never `AutoModelForCausalLM`.** For a
|
||
multimodal-capable base (Gemma-4, Qwen3.6), `AutoModelForCausalLM.from_pretrained` +
|
||
`save_pretrained` writes a FLAT text config (`Qwen3_5TextConfig`, `model.layers.*`) that
|
||
**both vLLM AND SGLang reject** (SGLang: "Qwen3_5ForCausalLM has no SGLang implementation";
|
||
vLLM wants `Qwen3_5ForConditionalGeneration`). Loading via `Qwen3_5ForConditionalGeneration` /
|
||
`Gemma4ForConditionalGeneration` keeps the wrapper config they accept. **This was the DS
|
||
blocker** — re-quant via the wrapper fixed it (`Dark-Scarlett-...-NVFP4A16-wrapper`).
|
||
2. **NVFP4A16 is weight-only → DATA-FREE.** llm-compressor infers `DataFreePipeline`; calibration
|
||
data is unused (only matters for W4A4 activation quant). W4A16 chosen per NVIDIA's sm_120
|
||
long-context guidance (W4A4 KLD 2-4× worse past ~10k ctx).
|
||
3. **Load on CPU (`device_map=None`)** so llm-compressor onloads one layer at a time. `device_map=
|
||
"auto"` packs the whole model onto the GPU and OOMs when the card isn't fully free.
|
||
4. **Both models are KV-EFFICIENT — the "dense = KV-hungry" worry was WRONG.** MeroMero (Gemma-4)
|
||
uses **sliding-window attention** (most layers cache only a bounded window); DS (Qwen3.6) uses
|
||
**hybrid GatedDeltaNet linear-attention** (3:1 linear:full, linear layers carry no KV). Both
|
||
hit full native 256K easily. (MeroMero KV pool ~542K tokens at util 0.52.)
|
||
5. **MeroMero vision reconstruction.** The finetune ships `processor_config.json` (image_processor
|
||
inline, `Gemma4ImageProcessor`) but NOT `preprocessor_config.json` — the old-format file vLLM's
|
||
feature-extractor loader wants. **Even google/gemma-4-31B-it (ungated!) ships only
|
||
processor_config.json.** FIX: extract the `image_processor` section → write
|
||
`preprocessor_config.json` verbatim, serve WITHOUT `--language-model-only`. Verified (model
|
||
correctly ID'd a red circle). Audio is config-declared but WEIGHTLESS (0 audio tensors).
|
||
6. **GPU placement.** Match the KV-heavier model to the roomier GPU. GPU0 (gen neighbor, ~54GB
|
||
free) > GPU1 (utility cluster, ~45GB free). Swapped MeroMero→GPU0, DS→GPU1. Pins via compose
|
||
`deploy.resources.reservations.devices`.
|
||
|
||
## Dead ends (tried + abandoned)
|
||
|
||
- **DS via llm-compressor `AutoModelForCausalLM`** → flat config vLLM/SGLang reject. → wrapper class.
|
||
- **DS via NVIDIA ModelOpt** → modelopt↔transformers **version deadlock**: current transformers
|
||
supports `qwen3_5` but crashes modelopt's sparse-moe plugin (`issubclass()` on a non-class);
|
||
modelopt 0.43.0 pulls an old transformers that can't load `qwen3_5` at all. Abandoned.
|
||
- **DS via SGLang** → `Qwen3_5ForCausalLM has no SGLang implementation`. Abandoned, but it REVEALED
|
||
that both engines need the wrapper (→ the fix in lesson 1).
|
||
- **`device_map="auto"` for the quant** → CUDA OOM in the weight observer. → `device_map=None`.
|
||
|
||
## granite retired + gateway repoint
|
||
|
||
- `vllm-granite` (granite-4.1-8b, fleet summarizer, GPU1) **`docker stop`ped** (reversible) to
|
||
reclaim ~13.6GB GPU1 for RP context.
|
||
- LiteLLM (`ana-docker:/opt/docker/conf/litellm/config.yaml`, backed up
|
||
`.bak-pre-granite-down-*`): **`granite-4.1-8b` alias RETIRED** — commented out, now 404s cleanly
|
||
(the `*` wildcard→llama-swap was decommissioned 2026-06-20, so no fallthrough). **`summarizer` +
|
||
`classifier` REPOINTED to gen** (`hosted_vllm/qwen3.6-35b-a3b-heretic` @ :8015,
|
||
`enable_thinking:false`) — both verified. ⚠ This LiteLLM change is **server-only / not
|
||
version-controlled** (a follow-up).
|
||
|
||
## MTP — deferred
|
||
|
||
DS's MTP heads were dropped by the CausalLM loader; **deferred, not restored** (spec-decode is
|
||
net-negative at RP temps: ~38-52% accept at temp 0.8-1.25, below vLLM's 0.5 cutoff). The
|
||
splice-back path (`splice_mtp.py` in the heretic2 work dir) exists if ever wanted. MeroMero
|
||
(Gemma-4) has no MTP by architecture.
|
||
|
||
## On-disk / where things live
|
||
|
||
- Quant pipelines: `ana-ml2:/tank/aimodels/meromero-v2-nvfp4-work/` +
|
||
`/tank/aimodels/darkscarlett-nvfp4-work/` (scripts, BF16 source, NVFP4 outputs).
|
||
- Compose stacks: `ana-ml2:/opt/docker/compose/{meromero-charrp,darkscarlett-charrp-reasoning}/`.
|
||
- Gateway aliases (unchanged, port-based): `char-rp`→:8016, `char-rp-reasoning`→:8018. (char-rp was
|
||
also fixed from the stale `magidonia-24b-v4.3` backend model name → `char-rp`.)
|
||
|
||
## Open follow-ups
|
||
|
||
1. LiteLLM granite/repoint change NOT version-controlled (server + backup only).
|
||
2. eshpfi unpushed (many commits this session incl. `f08b6cb`, `7bd7375`, `398b58a`).
|
||
3. MTP deferred (see above).
|
||
4. DS thinks verbosely (~13:1 reasoning:content) — eval item; consumers need generous `max_tokens`.
|
||
5. MeroMero full 256K needs util 0.55 (GPU0 ~1.8GB free, tight); ran at 0.52 for headroom (~4.6GB).
|
||
_Archived 2026-08-27._
|
||
|
||
- `[2026-08-12]` **infra-ops now holds an all-zones Cloudflare DNS-edit token (vaulted) + wgtunnel Phase-0 DNS landed.** Operator handed over a `Zone·DNS·Edit` (all zones) CF token → `secret put nh3-dev/.config/cloudflare/infra-ops-dns-token` (round-trip verified; /tmp drop shredded). Fleet DNS is now self-serve for infra-ops (⚠ HIGH blast radius — all zones). First use: created `boring.phasefinal.com` CNAME → `ana-srv1.phasefinal.com`, **DNS-only** (proxied:false), verified resolving to 38.120.12.44 on both authoritative NS (louis/wren) + 1.1.1.1 — NOT Cloudflare-proxied. Unblocks wgtunnel's wstunnel ACME cert. phasefinal.com zone id `f812ba74ed9a75cf21bbe7ce9188db50`. auto-memory `reference_infra_ops_cloudflare_dns_token`. (Earlier gap: the only prior vaulted CF token, jackdaw's, had `zone:read`+`worker:edit` but no `dns_records:edit`.)
|
||
_Archived 2026-08-27._
|
||
|
||
- `[2026-08-12]` **wgtunnel stood up as its own repo (`vh/wgtunnel`, private) after a live endpoint-verification pass.** Operator directed own-repo (mirrors stonehenge-park/tts-stack). Verified off the fleet before seeding: `ana-wg` WG server = **UDP/31337** (not 51820), subnet 10.30.10.0/24, MTU 1420, active roaming peer proves the public UDP DNAT works; traefik on ana-docker **terminates TLS :443** (ACME `anaprod` http-challenge, docker+file providers, CrowdSec bouncer) → confirms the clean design (wstunnel container on `traefik-net`, Host-routed, WS→UDP to `ana-wg:31337`); edge `38.120.12.44` direct-A, `tunnel.phasefinal.com` free (⚠ must be **direct**, NOT Cloudflare-proxied like vaultwarden). Repo pre-seeded (README/CLAUDE/persistent-memory/ROADMAP + `docs/verified-infrastructure.md` = ground truth) + pushed; commit `9584d38`, Vuong-attributed. vh gitea token pulled from the vault (`secret get`), not persisted to `.git/config`. **NEXT = `/vor-plan` or `/vor` (operator's call, interactive).** Deps to line up in the plan: DNS A-record, FortiGate :443 host-routing, a new ana-wg peer for the laptop, client tooling.
|
||
_Archived 2026-08-27._
|
||
|
||
`[2026-08-11]` **stonehenge-park — new fleet `/park` service repo stood up + designed.**
|
||
|
||
**What.** A separate greenfield repo (`~/development/stonehenge-park`, gitea `vh/stonehenge-park`,
|
||
pushed) for a self-contained `/park` service: one durable place to park any idea (repo-born OR
|
||
personal), find it by search, and have it **actively resurface** (by due-date or staleness) until
|
||
acted on — so parked ideas stop dying when a repo goes cold. NOT part of eshpfi; this is a pointer.
|
||
|
||
**Design (via `/vor-plan`, converged + persisted to `docs/design/`):** four contract-sized units —
|
||
**U1** core store+API (SQLite+FTS5, slug minting, bearer auth, REST) — the tracer, build first; **U2**
|
||
scheduler+notifier (in-process; due/stale → statusline `due-count` + althing push to a dedicated
|
||
**assistant channel**; keep-surfacing until promote/drop/re-snooze); **U3** `park` CLI (mirrors the
|
||
`secret` CLI); **U4** browse UI. `/vor-ui` ran too (U4 brief persisted).
|
||
|
||
**Locked decisions (operator):** SQLite, self-contained, ONE container, no external DB ("don't want
|
||
to troubleshoot it when a database upgrade happens") — a hard `[OPS]` invariant; system-minted
|
||
title-derived slugs + short ID (addressable as `park/<slug>`); active keep-surfacing resurfacing with
|
||
**re-snooze as the anti-nag valve**; bearer key, LAN/WG-internal; host nh3-docker; `/park` **replaces**
|
||
the global ROADMAP parking-lot discipline (deferred ideas → `/park`, `source`-tagged; ROADMAP keeps
|
||
only the v1 target) as a **fast-follow after v1** incl. migrating existing lots.
|
||
|
||
**Deferred (in the plan):** the althing assistant-channel handle **name** (decide at U2 contract
|
||
time); staleness threshold + re-push cadence (env-tunable defaults ~30d/~daily); design U2's emit
|
||
structured/consumable so a future **mission-control (Ledger→orchestrator)** can read it — park does
|
||
NOT build the orchestrator.
|
||
|
||
**State.** Pre-seeded for a fresh agent (CLAUDE/persistent-memory/ROADMAP/README + the design docs),
|
||
committed (`294ee98`), pushed. Next build task lives in that repo: the **U1 tracer contract** under
|
||
the House Code Discipline. Auto-memory candidate not yet written (repo is self-documenting).
|
||
_Archived 2026-08-27._
|
||
|
||
- `[2026-08-12]` **Global `~/.claude/CLAUDE.md`: `secret`/vault tool entry + "store in AND pull from the vault" standing directive** (dotfiles `9db703b`, pushed); statusline reset-countdowns + a latent tab-collapse parse-bug fix, now tracked in the dotfiles stow tree. Dogfooded the directive: created `vh/stonehenge-park` pulling the gitea token via `secret get`. (dotfiles + global config, not eshpfi.)
|
||
_Archived 2026-08-27._
|
||
|
||
- `[2026-08-11]` **TTS stack extracted to its own repo (`tts-stack`) + eshpfi stood down on TTS dev.** Operator: hand all TTS tuning/dev to a separate agent with a self-contained repo (knowledge + infra access + a live knowledge list), and move the voice corpus in. New repo `~/development/tts-stack` (commit `9ee3288`) carries: dots-tts stack (canonical intent), `voices/` corpus (MOVED out of eshpfi), `KNOWLEDGE.md` (engine landscape + prosody findings + foot-guns), `docs/infrastructure.md` (irv-ml1 access + gated deploy runbook + rollback), CLAUDE/persistent-memory/ROADMAP, `tools/` (pause-probe + Booth render). Followed the **chatterbox-fast precedent**: eshpfi `stacks/dots-tts/` reduced to a POINTER README; the ~15 experimental TTS compose wrappers stay here as reference (catalogued in tts-stack KNOWLEDGE). Blast-radius check: no eshpfi playbook/script reads the canonical corpus (other `voices/` refs = unrelated host paths). **Reverses** the earlier "Corpus home = eshpfi `voices/` (keep-here)" call. ⚠ tts-stack is LOCAL-ONLY until pushed — needs a gitea remote (`vh/tts-stack`) + push before the separate agent can clone (operator's call — outward-facing + repo-create creds).
|
||
_Archived 2026-08-27._
|
||
|
||
- `[2026-08-10]` **dots-tts v3 — clause-break → period pause mapping.** Operator: v2 "sounds good" but donut won't pause at semicolons/dashes. ROOT CAUSE (measured via a pause-probe A/B — synth duration over N runs, non-determinism averaged out): dots' prosody honors a real pause **only for ellipsis (~+0.43s) and period (~+0.3s, capitalization-independent)**; comma/semicolon/colon/dash all run **flat (~+0.03s vs no-punct)**. Two distinct sub-causes: **dashes regressed in v2** (the `—`→`-` fold made em-dashes read as word-joiners), while **semicolons were NEVER a v2 change** — dots ignores them natively, only newly noticeable because v2 made everything else clean. Operator call: ellipsis "too much" → **map `;`, clause `:`, and em-dash `—` → period** in `_sanitize` (believable ~0.3s clause break). GUARDS (pinned by 11 unit tests, `stacks/dots-tts/test_sanitize.py`): digit-guarded colon `(?<!\d)\s*:\s*(?!\d)` so times `3:45` / ratios `2:1` survive; en-dash `–`→hyphen KEPT (numeric-range `10–20` safety — em-dash breaks, en-dash ranges, different jobs); genuine ellipsis left at full strength (author meant a long pause). Gated deploy (redeploy2 pattern → v3): build → throwaway :8199 test container + **pause-gate** (semicolon sentence must run ≥0.12s longer than baseline; measured **+0.427s**) → only then cut live over. LIVE + healthy `local/dots-tts:v3` on :8198. **rollback = `sed -i 's/^DOTS_TAG=.*/DOTS_TAG=v2/' .env + docker compose up -d dots-tts`** (v2 image retained). Booth `dots-pauses` (A=old-flat / C=ellipsis-too-much / D=live-v3). [[reference_chatterbox_fast_repo]]
|
||
_Archived 2026-08-27._
|
||
|
||
- `[2026-08-10]` **dots-tts v2 — contraction fix (curly-sanitize) + sentence-chunking + dependency-pin recovery.** Operator: donut read contractions wrong ("you're"→"you ree", "donut's"→"donut ess"). ROOT CAUSE (isolated via A/B booth): **curly/typographic apostrophes** (`’` U+2019 from ratatoskr's LLM) — dots' tokenizer mispronounces them; STRAIGHT apostrophes read clean under `normalize_text=True`. FIX (`app.py`): fold curly→ASCII (`str.maketrans`) before synth, **KEEP `normalize_text=True`** (operator call — retains number/date expansion). Also added **server-side sentence-chunking** (pack ≤280 chars): dots caps one `generate()` at ~500 patches/~40s, so long RP turns (the Zev monologue = 160s audio) truncated; chunking stitches them (verified full 160.3s, not 40s-cut). **⚠ BUILD FOOT-GUNS (both bit this redeploy):** (1) upstream dots.tts `constraints/recommended.txt` now pins **`gradio==6.17.0` — phantom, not on PyPI** → fresh `pip install dots.tts` unsatisfiable; FIX = pin `dots.tts==0.2.1` + **DROP** the `-c recommended.txt` constraints (0.2.1 pulls working gradio 6.17.3). (2) pinning only `torch==2.8.0` let **torchaudio float to 2.11.0 → dots.tts refuses to load** (minor-version match check); FIX = pin `torchaudio==2.8.0`. **⚠ DEPLOY LESSON:** `docker compose up -d` to a new tag swaps the LIVE container BEFORE any health check — a broken image crash-loops production (**ratatoskr TTS down ~1-2min this session**). NEW PATTERN = build → test in a THROWAWAY container on an alt port (:8199) → health+verify → only THEN cut live over (redeploy2.sh). v2 LIVE + healthy on irv-ml1:8198, **CONSUMER-CONFIRMED clean** (ratatoskr verified end-to-end on their :8765 — apostrophe string reads clean, /api/tts 200 @ 48kHz, no client change; the ~1-2min blip didn't hit them, their concurrent auto-audio issue was client-side localStorage). **rollback = `sed DOTS_TAG=v1 + docker compose up -d dots-tts`** (v1 image retained). Also: deployed container GPU crept ~6→13.9GB over 8h serving (cache accumulation; a redeploy resets it — watch item). [[reference_chatterbox_fast_repo]]
|
||
_Archived 2026-08-27._
|
||
|
||
|
||
- `[2026-08-07]` **Personal-Worldtree kb-contamination incident (WT #394) diagnosed; attribution CLOSED UNRESOLVED.** A reconcile `WingStore._embed` full-tree walk (kb `fs_root=KB_PATH` root, sibling wings nested) swept 5,354 fiction+main rows into personal's `knowledge_base` (2 superseded generations served as current). Fixed by WT #394 (aca39a1, kb walks exclude sibling wings; ships b182). Trigger un-attributable — peer reconcile via the SHARED infra-ops identity + 0 dockerd exec-logging = fingerprint-less. Durable finding → auto-memory `infra_ops_shared_identity_attribution_gap`, PARKED (operator ruled A) into [[project_migrate_infra_access_to_claude_credentials]]. Evidence hold on the 5,354 rows until operator sequences cleanup (w/ Brokkr, on #394's agenda).
|
||
_Archived 2026-08-22._
|
||
|
||
- `[2026-05-12]` corviduo-dev (Worldtree-team dev VM, 10.250.50.152,
|
||
CT 106 on pfi-pve) added to `servers/` inventory. Treat like SF
|
||
client hosts: PFI hosts + provides emergency-ops backstop;
|
||
Worldtree team owns OS config + deploys + backup decisions.
|
||
_Archived 2026-05-27._
|
||
|
||
- `[2026-05-12]` Worldtree `:latest` tag drift bug — fixed by
|
||
health-gated `:latest` advance in vh/worldtree's deploy workflow
|
||
(architect commit `8ef3801`): only tag `:latest` AFTER the new
|
||
container's `/health` probe passes. Build-on-host stacks here
|
||
don't have this problem because the playbook always builds the
|
||
SHA-tagged image from a `git reset --hard <ref>` checkout.
|
||
_Archived 2026-05-27._
|
||
|
||
- `[2026-05-12]` asset-engine stack scaffolded LAN-direct at
|
||
`http://10.250.50.70:8200`. Initially included Traefik labels for
|
||
public hostname; user pulled them out (internal tool, no public
|
||
TLS surface needed). Pattern: internal tools default LAN-direct;
|
||
Traefik wiring only when external/TLS required.
|
||
_Archived 2026-05-27._
|
||
|
||
- `[2026-05-12]` asset-engine catalog gains `lifecycle: { stack,
|
||
vram_gb, gpu_device_id }` per irv-ml1 service for the
|
||
orchestrator feature. SSH keypair scaffolded at
|
||
`ana-docker:/opt/docker/conf/asset-engine/ssh/` for
|
||
asset-engine container → irv-ml1 orchestration via dedicated
|
||
ed25519 key.
|
||
_Archived 2026-05-27._
|
||
|
||
- `[2026-05-13]` `pull-hf-repo.yaml` is the canonical HF-fetch
|
||
playbook on ana-ml2. Supports `--var repo_type=model|dataset|space`.
|
||
Replaces ad-hoc `huggingface_hub.snapshot_download` calls.
|
||
_Archived 2026-05-27._
|
||
|
||
- `[2026-05-13]` Selene-1-Mini-Llama-3.1-8B added to llama-swap as
|
||
judge model. mradermacher `i1-Q6_K` imatrix quant (~6.5GB). AtlaAI
|
||
reward/eval model — temp 0.01, ctx 32K, q8_0 KV cache. New JUDGE /
|
||
EVAL MODELS section in `stacks/llama-swap/conf/config.yaml`.
|
||
_Archived 2026-05-27._
|
||
|
||
- `[2026-05-13]` `vllm-qwen3` → `vllm` stack rename. Added
|
||
`vllm-reward` service (Skywork-Reward-V2-Llama-3.1-8B-AWQ classifier).
|
||
Three vLLM services share GPU 1 (embed 0.20, rerank 0.20, reward 0.30
|
||
utilization; 30% headroom). All use `--runner pooling`; classification
|
||
drives via model's `architectures: [LlamaForSequenceClassification]`
|
||
in config.json, NOT `--task classify` (deprecated in vLLM 0.19.1).
|
||
_Archived 2026-05-27._
|
||
|
||
- `[2026-05-13]` /tend-docs first pass deletions: `stacks/infinity/`
|
||
removed (retired by vllm). Archived
|
||
`docs/asset-engine/design-brief.md` → `docs/archive/asset-engine/`
|
||
with archival header. Fixed `pfi-pve` VM list to full
|
||
`qm list` enumeration. Dropped stale weak-password section from
|
||
`pfi-postgres` (rotation done 2026-04-23).
|
||
_Archived 2026-05-27._
|
||
|
||
- `[2026-05-14]` althing-chamber stack scaffolded: chamber + forseti.
|
||
Internal LAN-only at port 7881 (chamber default 7878 collides with
|
||
task-board). Two-service compose, shared SQLite bind-mount,
|
||
build-on-host pattern via vh/althing's gitea-workflow. Forseti is the
|
||
canonical dev for this stack (galdrabok is on a different project).
|
||
_Archived 2026-05-31._
|
||
|
||
- `[2026-05-16]` althing-chamber Phase 2: added `althing-agent-runner`
|
||
as third compose service (worldtree-driver agent dispatcher).
|
||
All three althing services use the same image; `command:` selects
|
||
entrypoint. Safe to enable preemptively (sleeps when no
|
||
driver=worldtree handles declared).
|
||
_Archived 2026-05-31._
|
||
|
||
- `[2026-05-17]` Phase 3.1 cross-process streaming uses Valkey 8
|
||
alpine as a sibling compose service in `stacks/althing-chamber/`,
|
||
redis-protocol pub/sub for high-volume `msg_delta` / `msg_thinking` /
|
||
`msg_start` / `msg_complete` event kinds. DB bridge keeps
|
||
`msg_curated` + `floor_grant` (structured / canonical). Two-channel
|
||
architecture, no overlap. chamber + agent-runner `depends_on:
|
||
valkey: service_healthy`.
|
||
_Archived 2026-05-31._
|
||
|
||
- `[2026-05-17]` Worldtree admin workflow shift (per vh): infra-ops
|
||
gets its own permanent admin-tier key (`61419c92`, stored at
|
||
`ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin`).
|
||
Future admin ops route through this key, not the bootstrap admin
|
||
via docker-as-root.
|
||
_Archived 2026-05-31._
|
||
|
||
- `[2026-05-17]` Worldtree env-var addition checklist: anytime
|
||
introducing `os.environ.get("FOO")` in worldtree code, update BOTH
|
||
`.env.example` AND `compose.yaml`'s `&worldtree-env` anchor in the
|
||
same PR. Same Z_AI_API_KEY-shape footgun bit
|
||
`BIFROST_CLIENT_ALLOWED_HOSTS` (#170) until worldtree-dev added the
|
||
passthrough line in `08f02b2`.
|
||
_Archived 2026-05-31._
|
||
|
||
- `[2026-05-18]` Volva systemd install complete after three-stage
|
||
debug. Final unit at `/etc/systemd/system/volva.service` runs as
|
||
`User=lkraven` with `ProtectHome=read-only` +
|
||
`ReadWritePaths=/home/lkraven/.althing /home/lkraven/.codex`
|
||
carve-outs for state writes. `VOLVA_ALTHING_CLI=/home/lkraven/
|
||
.local/bin/althing-cli` + `ALTHING_HANDLE=volva` both pinned in
|
||
env.sh.
|
||
_Archived 2026-05-31._
|
||
|
||
- `[2026-05-19]` Worldtree CD disk-hygiene strategy: watermark gate
|
||
(env-tunable threshold + window, fail-loud on still-low post-prune)
|
||
+ eager post-deploy prune (only after `:latest` advance succeeds,
|
||
uses `docker image prune -a --filter "until=24h"` which respects
|
||
in-use semantic — protects pinned + personal images automatically).
|
||
Combined: demo VM holds ~24h of deploy history instead of unbounded
|
||
accumulation. Shipped in vh/Worldtree PR #184 (`306cd61` +
|
||
`613dac2` + `bd91df5`).
|
||
_Archived 2026-05-31._
|
||
|
||
- `[2026-05-19]` Skaldsong CD shape: shape (1) of three operator
|
||
options — container + Gitea registry + pull-restart, matching
|
||
Worldtree's pattern. Target host ana-docker (NOT nh3-dev where
|
||
skaldsong-dev runs for iteration). SHA-pin only for now; health-gated
|
||
`:latest` advance is a follow-up once `/health` exercises Worldtree
|
||
+ Kokoro reachability.
|
||
_Archived 2026-05-31._
|
||
|
||
- `[2026-05-19]` Skaldsong prod (ana-docker) switched from demo
|
||
Worldtree (:8080) to personal (:8081). Same `user_id=skaldsong`
|
||
as the nh3-dev hand-launch key — shared Heimdall agent slot
|
||
(`skaldsong:wizard-v2`), different `key_id`s for independent
|
||
rotation. Demo Worldtree stays for isolation; personal becomes
|
||
the multi-consumer dev iteration instance.
|
||
_Archived 2026-05-31._
|
||
|
||
- `[2026-05-19]` mead-hall Bifrost v0.3 end-to-end smoke green.
|
||
Closed task #32 (althing thread `01KRV1M2KW6N6HBEXGTH72QXCA`).
|
||
Wire layer (handshake + binding + dispatch) + data-flow
|
||
(per-dispatch JWT claims → `ctx.session_id` populated → real
|
||
session-scoped data) + agent-loop (LLM reads + quotes back) all
|
||
proven. Resolves the "stalled mid-Worldtree" state from the
|
||
2026-05-17 snapshot.
|
||
_Archived 2026-05-31._
|
||
|
||
- `[2026-05-25]` v0.25.3 lofn tuning: `temperature 0.6 → 1.0` +
|
||
`repetition_penalty 1.0 → 1.15` on default+fast profiles.
|
||
Heretic-abliterated qwen3.6 was locking into degenerate attractors
|
||
at the model's thinking-mode floor (0.6). Pattern:
|
||
abliterated/uncensored Qwen variants need higher temp + non-trivial
|
||
rep-penalty than base, NOT the model-card's documented floors.
|
||
_Archived 2026-05-31._
|
||
|
||
- `[2026-05-25]` Worldtree #205 v0.25.2 ships `/app/config/` as
|
||
bind-mount + root-then-drop entrypoint shim (gosu). Operators
|
||
get persistent per-instance config without container-rebuild.
|
||
Same bind-mount pattern hit twice subsequently in v0.27.0 (selene)
|
||
and v0.29.9 (echo) — bind-mount shadows image-baked defaults, so
|
||
every new required key surfaces as a crash-loop on existing
|
||
deployments. The v0.29.12 canonical example files close this loop.
|
||
_Archived 2026-05-31._
|
||
|
||
- `[2026-05-26]` Worldtree v0.27.0/v0.27.1 fixes Tier 3 GET
|
||
visibility. `available_agents()` helper was over-applied to
|
||
`GET /agents/<id>`, masking ALL Tier 3 agents regardless of row
|
||
state. Bug only visible as "agent not found" via GET; storage was
|
||
fine (silent-2xx PATCHes had persisted correctly). v0.27.1 added
|
||
fail-fast hardening for the startup pre-resolve fragility class.
|
||
_Archived 2026-05-31._
|
||
|
||
- `[2026-05-26]` Skaldsong v0.30.7 defensive 409→PATCH fallback.
|
||
v0.30.6's GET-then-define-or-patch path crash-looped against
|
||
pre-v0.27.0 Worldtree's GET-visibility bug (GET 404 phantom →
|
||
define 409 conflict). v0.30.7 catches the 409 and falls through
|
||
to PATCH (which silently 2xx'd on the pre-fix Worldtree).
|
||
_Archived 2026-05-31._
|
||
|
||
- `[2026-05-27]` Worldtree v0.29.x landed full saga→echo refactor +
|
||
config-validator hardening (v0.29.10 create_provider family-before-regex;
|
||
v0.29.11 collect-then-raise echo startup validators; v0.29.12 ships
|
||
providers.yaml.example/defaults.yaml.example canonical configs; v0.29.13
|
||
reasoning_content extraction + catalog family lookup). Operator-asked,
|
||
worldtree-dev-shipped, CI-deployed.
|
||
_Archived 2026-06-01._
|
||
|
||
- `[2026-05-27]` artemis-31b-v1i added to llama-swap + worldtree personal.
|
||
BeaverAI Gemma 4 31B Q6_K (~28.6GB), 128K ctx, `--reasoning-format deepseek`
|
||
(gemma format unsupported in deployed llama.cpp). Worldtree catalog
|
||
`family: gemma` so GemmaProvider routes reasoning tokens.
|
||
_Archived 2026-06-01._
|
||
|
||
- `[2026-05-27]` Skaldsong streaming TTS v0.32.0→v0.32.2: chunked-batch SSE
|
||
(one Kokoro POST per paragraph); defensive event_stream catch-all; NDJSON
|
||
parsing for Kokoro /dev/captioned_speech multi-line responses.
|
||
_Archived 2026-06-01._
|
||
|
||
- `[2026-05-31]` Dia2 deployed as **two fixed-model instances**
|
||
(`dia2-2b` :8200, `dia2-1b` :8202) from `local/dia:v2`, retiring
|
||
legacy Dia 1.6B; catalog `dia` entry **removed → dia2-2b + dia2-1b**
|
||
(breaking for asset-engine). Rationale: the devnen wrapper is
|
||
single-model and IGNORES the OpenAI `model` field (verified on its live
|
||
OpenAPI), so the only way to offer both Dia2 models as real per-request
|
||
asset-engine choices is one fixed endpoint per model. `3139e81`
|
||
(deploy), `db15638` (catalog swap).
|
||
_Archived 2026-06-03._
|
||
|
||
- `[2026-05-31]` Both dia2 catalog entries route to the wrapper's richer
|
||
**`/tts`** endpoint (not `/v1/audio/speech`) to expose the full control
|
||
surface (cfg_scale/temperature/top_p/cfg_filter_top_k/voice_mode/clone);
|
||
all defaults sourced from the wrapper's `CustomTTSRequest` Pydantic
|
||
blessed values. Voice default is `voice_mode: clone` +
|
||
`clone_reference_filename: Abigail.wav` so a stable (non-random-gender)
|
||
voice is the out-of-box behavior. `55602b7`, `5c47843`.
|
||
_Archived 2026-06-03._
|
||
|
||
- `[2026-05-31]` Zonos REST adapter (`stacks/zonos/adapter/`,
|
||
`local/zonos-api`) — thin OpenAI-ish `/v1/audio/speech` FastAPI in front
|
||
of the Gradio-only Zonos SDK; JSON-envelope `{audio, audio_format,
|
||
seed}` (Zonos is the fleet's first seedable TTS). Port 8203 (moved off
|
||
8201 — collided with csm). Built; NOT deployed (stack down for VRAM).
|
||
Also fixed the upstream image's missing CMD (`71df6f7`). `81efa8d`.
|
||
_Archived 2026-06-03._
|
||
|
||
- `[2026-05-31]` Catalog schema regenerated: added `CatalogLifecycle` +
|
||
`reproducibility.seed_field` (`b7b2130`). Resolves the stale-schema
|
||
hand-off; catalog now validates clean. (asset_engine `catalog.py`
|
||
Pydantic already supported both — schema file was just behind.)
|
||
_Archived 2026-06-03._
|
||
|
||
- `[2026-05-31]` TTS bench expanded with `stacks/{dia,zonos,csm}`
|
||
(`666f7f3` dia+zonos, `a4b8c2a` csm). The bench already had Fish
|
||
S2-Pro / Chatterbox-Turbo / IndexTTS-2 / CosyVoice3 / Kokoro /
|
||
VibeVoice / Qwen3-TTS / Kyutai. (csm since removed 2026-06-01.)
|
||
_Archived 2026-06-03._
|
||
|
||
- `[2026-05-31]` Remote browser/iPad/Vision-Pro driver seat for the
|
||
agent-fleet zellij session `Claude` stood up on nh3-dev (ttyd behind
|
||
Caddy, network-gated). Out of this repo — full architecture + the
|
||
HTTP2/OSC52/Safari-auth gotchas in auto-memory `reference_ttyd_fleet_seat`.
|
||
_Archived 2026-06-03._
|
||
|
||
- `[2026-05-30]` esh-docker-vm NFS boot-ordering fix:
|
||
`playbooks/fix-esh-nfs-boot-ordering.yaml` (`c0458d9`, + `53157b1`
|
||
drop-in filename-collision fix) adds `_netdev,nofail` to the four
|
||
10.0.50.50 NFS mounts + a docker `After=remote-fs.target` drop-in —
|
||
resolves paperless `Exited(255)` on reboot. traefik also gained
|
||
`restart: unless-stopped`. Full incident → auto-memory
|
||
`incident_esh_docker_nfs_boot_race`.
|
||
_Archived 2026-06-03._
|
||
|
||
- `[2026-06-01]` **Fish cloning VERIFIED competent (ECAPA-TDNN)** — retracting
|
||
the earlier "weak cloner" call. Isolated test: Imogen-referenced clone ~0.79
|
||
cosine to the real `Imogen.wav` vs ~0.10 for the no-reference default;
|
||
transcript condition (correct 0.787 / bogus 0.778 / empty 0.738) barely moves
|
||
identity (affects pronunciation, not timbre). Root cause of "Imogen sounds
|
||
nothing like British" = the `"undefined"` select bug feeding Fish its default
|
||
voice, NOT Fish. So the entire Fish-Imogen saga was the undefined bug; no
|
||
Fish-side fix needed. (Methodology lessons → Tried and abandoned.)
|
||
_Archived 2026-06-04._
|
||
|
||
- `[2026-06-01]` **CSM (Sesame csm-1b) torn down entirely** — removed from
|
||
catalog, `stacks/csm/`, `playbooks/deploy-csm.yaml`, and host
|
||
(`c54ab13`). Two reasons: (1) deep-research verdict — the acclaimed
|
||
Maya/Miles demo is a fine-tuned, larger variant Sesame never open-sourced;
|
||
the open 1B base ships no usable voices, can't generate text, English-only,
|
||
no real-time streaming, absent from leaderboards, dominated by
|
||
Kokoro/Dia2/Fish/IndexTTS. (2) The upstream image is rotted (see Tried and
|
||
abandoned). asset-engine re-synced (`306aae9`).
|
||
_Archived 2026-06-04._
|
||
|
||
- `[2026-06-01]` **Chatterbox catalog v1→v2** (`93cedd6`) — re-point to the
|
||
devnen wrapper's `/tts` (was the thin OpenAI `/v1/audio/speech` exposing
|
||
none of the emotion knobs → poor output). Default `exaggeration: 0.5` per
|
||
Resemble's README ("works well for most"); the server SHIPS demo-tuned 1.3
|
||
which over-acts — deliberately NOT adopted. 28 predefined voices via
|
||
`/get_predefined_voices` (default Emily.wav). Corrected seedable false→true.
|
||
_Archived 2026-06-04._
|
||
|
||
- `[2026-06-01]` **Fish-S2 catalog v1→v6** (`f3c260e`..`38d9e3b`) — was a
|
||
blank 2-field form; now exposes the full fish-speech ServeTTSRequest surface
|
||
+ a 32-voice `reference_id` "Voice" dropdown (STATIC list — Fish has no
|
||
/voices API). `reference_id`-by-name works ONLY when each voice is staged as
|
||
a `references/<name>/` dir with `<name>.wav` + `<name>.lab` — was silently a
|
||
no-op until that layout fix on 2026-06-01 (see the two-root-causes entry
|
||
above); the inline-path-as-filesystem-path `references` shape 500s (see Tried
|
||
and abandoned). Corrected vram
|
||
4→24 GB, cold_start 8→240s, seedable false→true. Added 3 consenting
|
||
British-female VCTK voices (Imogen=p225/Eleanor=p228/Beatrice=p229).
|
||
_Archived 2026-06-04._
|
||
|
||
- `[2026-06-01]` **`blendable: false` catalog flag — DEFERRED**, pending
|
||
asset-engine-dev honoring it in the form template. Clone-voice selects
|
||
(chatterbox/fish/dia2) must render as plain dropdowns, not the Kokoro blend
|
||
widget (blending reference clones is nonsensical). I ship the catalog flag +
|
||
schema field once they read it. Tracked: althing msg
|
||
`01KT2K2SY9N7AY69R9V0B4RXSW` to asset-engine-dev.
|
||
_Archived 2026-06-04._
|
||
|
||
- `[2026-06-01]` **Fish reference_id empty-dir fix shipped** (`c5bbb90`) — Populated
|
||
`references/<name>/<name>.wav`+`.lab` for all 32 voices; playbook gained
|
||
normalize-step + A/B smoke gate. glados got a real transcript (ASR'd via
|
||
Parakeet): the Portal "Welcome to test chamber 4" lines.
|
||
_Archived 2026-06-11._
|
||
|
||
- `[2026-06-02]` **Chatterbox → main TTS engine; build custom `chatterbox-fast`
|
||
streaming container.** Workload = single-stream interactive. GPU placement:
|
||
3090 (device 0) if it fits else A6000 (device 1). Cutover: parallel catalog
|
||
entry, burn in beside live `chatterbox`, then flip. Streaming = adaptive
|
||
buffer-ratchet chunking (native frame-streaming abandoned). Since extracted to
|
||
its own repo `vh/chatterbox-fast` (auto-memory `reference_chatterbox_fast_repo`).
|
||
_Archived 2026-06-11._
|
||
|
||
- `[2026-06-02]` **Sentence-splitting loses quality (operator-corrected).** Claimed
|
||
naive sentence-level streaming has "zero quality loss" — WRONG. The T3 AR backbone
|
||
conditions prosody on the WHOLE text; splitting loses cross-sentence prosodic
|
||
context even though voice timbre stays (reference-conditioned). No *artifacts* ≠
|
||
no *quality loss*. Hence adaptive-chunk design (maximize context per chunk subject
|
||
to latency budget), not fixed per-sentence splits.
|
||
_Archived 2026-06-11._
|
||
|
||
- `[2026-06-03]` **Declined worldtree v0.30.4 staging deploy** — that's worldtree-team's
|
||
CI/CD lane (a developer `staging/vX.Y.Z` git-tag promote), not infra-ops. They self-corrected
|
||
to the same conclusion independently.
|
||
_Archived 2026-06-13._
|
||
|
||
- `[2026-06-03]` **R14 scope = (a) provision-only.** infra-ops provides box + CUDA env +
|
||
engines + weights + NFS; brokkr/dev wires `arms.py` + runs — keeps infra-ops OFF the
|
||
VIVAE-processing path (VIVAE = Variably Intense Vocalizations of Affect/Emotion, CHARTER §4
|
||
highest-liability; operator authorized R&D-eval-only, quarantined). Box = irv-ml1 (A6000
|
||
free; ana-ml2 GPU-saturated). Per-engine venvs (divergent torch stacks); A6000 = `cuda:0`
|
||
NATIVE (≠ docker `=1`).
|
||
_Archived 2026-06-13._
|
||
|
||
- `[2026-06-03]` **yt-voice-clipper push-to-deploy via gitea webhook** (operator-directed,
|
||
after 6 manual rebuilds in ~40 min). Webhook (not poll) — gitea CAN reach the WG IP per the
|
||
operator. The proxy env + Homepage labels live in the **host-specific override** (untracked
|
||
→ survive the auto-deploy's `git reset --hard`), NOT yt-voice-clipper-dev's image. Runbook
|
||
`d4f180d`.
|
||
_Archived 2026-06-13._
|
||
|
||
- `[2026-06-03]` **yt-voice-clipper bot-gate fix = route yt-dlp through NH3 residential
|
||
egress, NOT cookies/PO-token.** YouTube hard-flags the Irvine colo IP (LOGIN_REQUIRED on a
|
||
public video even with no cookies). Cookies + the bgutil PO-token + deno JS-runtime all
|
||
loaded fine — the gate is pure IP reputation. Operator chose proxy-via-nh3-dev → durable
|
||
dante proxy → proven. The egress proxy is a reusable fleet lever for any datacenter-IP-gated
|
||
service.
|
||
_Archived 2026-06-13._
|
||
|
||
- `[2026-06-04]` **`brokkr` user (no-sudo) on irv-ml1; R14/R15/R16 substrate moved to /home/brokkr.** Persistent box services there need SYSTEM systemd units (see Tried).
|
||
_Archived 2026-06-14._
|
||
|
||
- `[2026-06-04]` **ollama upgraded 0.9.0→0.30.4 on irv-ml1** (Ministral-3 is a Dec-2025 model the old engine refused); A6000 pinned by **UUID** not index (native fastest-first ≠ nvidia-smi PCI).
|
||
_Archived 2026-06-14._
|
||
|
||
- `[2026-06-04]` **Worldtree demo/pinned/personal deploys are ALL CI/CD, not infra-ops** — a "deploy vX.Y.Z" request to infra-ops is MISROUTED → point them back to their pipeline. (`d8d776c`, auto-memory `reference_worldtree_deploys_cicd`)
|
||
_Archived 2026-06-14._
|
||
|
||
- `[2026-06-04]` **infra-ops NOPASSWD-sudo identity commissioned, scoped to PFI boxes** (+esh-docker-vm by operator override) — so infra-ops completes DevOps end-to-end vs handing the operator sudo steps. Dedicated key, sudo log_output, key-gated. (`8c32a05`, auto-memory `reference_infra_ops_sudo_identity`)
|
||
_Archived 2026-06-14._
|
||
|
||
- `[2026-06-04]` **phi4 ships the CANONICAL/official Phi-4 chat template, NOT Ollama's** (Ollama's omits the system `<|end|>`). Applied an Ollama-matching override then reverted — ship correct, not the benchmark quirk. (`90e08f0`→`27eb537`; "headgun" lesson in Tried.)
|
||
_Archived 2026-06-14._
|
||
|
||
- `[2026-06-04]` **phi4-mini FP8 on ana-ml2 vLLM is the nevermore summarizer; granite-4-small retired** from llama-swap. 50K ctx + FP8 KV. (`40a374b`) [superseded by the 2026-06-05 granite-4.1-8b decision]
|
||
_Archived 2026-06-14._
|
||
|
||
- `[2026-06-08]` **volva.service + heid.service removed from nh3-dev** — vestigial systemd daemons; Heid/Volva re-architected from Python pollers to Claude Code session orchestrators (heid `12aa5a9`); volva.service was crash-looping 203/EXEC. (`6e2f80e`)
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-05]` **Granite 4.1 8B FP8 replaced phi4-mini as the production summarizer.** Beat phi4 on precision in brokkr's R15 P03. **Staying FP8, not Q4/AWQ** — primary workload is high-concurrency, where FP8 scales ~linearly (2010 tok/s @ C=32). vLLM `vllm-granite` :8004 GPU 1, official IBM compressed-tensors FP8. (Then on Ada; box has since gone Blackwell.) (`34a43a0`, auto-memory `reference_ana_ml2_vllm_granite`)
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-05]` **Langfuse v3 on ana-docker (:3001) as the gateway trace UI**; LiteLLM `success_callback:[langfuse]` live. Pretty traces + tok/s dashboard. NOT a prerequisite (spend_logs already capture tokens+latency). (`9171e6a`)
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-05]` **Ollama BANNED fleet-wide** (operator directive) — never stand one up; tear down any found; serve via llama-swap or vLLM. (auto-memory `feedback_avoid_ollama`)
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-05]` **ComfyUI / FLUX.2 work split to `~/development/comfy-dev`** (dedicated repo + agent). eshpfi keeps the `comfyui`/`arbo` stack compose; comfy-dev owns the model/workflow knowledge. (auto-memory `reference_irv_ml1_ampere_quant`)
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-05]` **Worldtree summarizer config refresh DEFERRED to Worldtree #254** (granite-4.1-8b is the structured-output profile, ON HOLD, no live consumer). Bind-mount CONFIGS (providers.yaml, vh-owned) ARE infra-ops's to apply directly — only the app/image DEPLOY is CICD; the `.env` is deploy-owned. (auto-memory `reference_worldtree_deploys_cicd`)
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-13]` **Arbo prod bearer auth turned OFF — WireGuard is the access boundary** (operator decision; reverses ADR-0001's "closed the open-auth hole"). ENGINE_TOKEN must be **ABSENT, not empty** (empty-string still gates) — removed from BOTH the host `.env` AND the compose `environment:` injection line. Original token backed up at `irv-ml1:/opt/docker/compose/arbo/.env.pre-auth-off.bak`; comfy-dev updated their ADR-0001. Tracking: `db97899` + `playbooks/arbo-disable-engine-token.yaml`.
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-13]` **Storetank image-models archive DECOMMISSIONED; arbo is the single live ComfyUI model tree (502 G).** Curated `/storetank/image-models/comfy` (was 919 G, the native `/opt/ComfyUI/models` symlink target) → killed everything superseded by arbo's current gen (Hunyuan, WAN2.1, FLUX.1, Chroma, SD3.5, orphaned umt5+llava ≈ 739 G) + migrated the keepers (gen-agnostic utilities + the SDXL/Pony stack, 177 G) into `/storetank/arbo/models` (same-fs move, skip-existing protects prod). Tracking: `docs/storetank-image-models-archive.md` + `docs/arbo-comfyui-model-catalog.md` (`1902425`→`5007ec1`).
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-13]` **GRANITE_KEY provisioned to comfy-dev's nh3-dev dev env** at `~/.arbo_granite_key` (0600) for the hero gen+judge script — verbatim copy of the prod `arbo-prompt-enhance` vkey (now extended to reach BOTH `granite-4.1-8b` AND `qwen3.5-9b-fp8`); nothing minted. The vkey README's "granite-only" wording was stale → corrected (`f32c6dd`).
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-13]` **ana-ml2 upgraded Ada → dual RTX PRO 6000 Blackwell Max-Q** (96 GB each, cc 12.0 / sm_120; was dual RTX 6000 Ada 48 GB / cc 8.9 — confirmed live via `nvidia-smi`). Unlocks NVFP4 (FP4 tensor cores) and doubles VRAM headroom. CLAUDE.md GPU-spec doc-fix **LANDED `355a240`** (operator). Tracking: `19a07b9`/`1e2a3a1` ("Blackwell 96GB").
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-13]` **NVFP4-W4A4 is infeasible for Granite — FP8 stays the Granite-on-Blackwell format.** W4A4 collapses at 30k context, proven **producer-independent** (modelopt AND llm-compressor both clean-NONE from the same BF16 base + wikitext-2k calib). No 4-bit wins both axes: W4A4 = quality collapse; W4A16-NVFP4/AWQ = weight-only dequant → bf16 (no FP4-core speedup). **30B retired**. (auto-memory `reference_nvfp4_w4a4_granite_infeasible`)
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-13]` **Qwen3.5-9B VL (FP8) deployed on ana-ml2 GPU 1** — `qwen35-vl` stack, :8007, gateway alias `qwen3.5-9b-fp8`. **Pinned nightly digest, not `:latest`**: the stable release quantizes the VL *vision tower* under `--quantization fp8` → garbage vision (LM fine); the nightly correctly excludes it. Re-pin + drop the pin once that exclusion lands stable. (`2e3dcc2`)
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-13]` **comfyui 325 G model tree migrated worktank → `/storetank/arbo`** (worktank 97% → 26%). `arbo` is the consuming app; overlay bind-mount via `COMFYUI_MODELS_DIR`. (`38186be`) (See the 2026-06-13 archive-decommission decision above — this tree later absorbed the storetank-archive keepers, reaching 502 G.)
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-13]` **GPU layout settled on the Blackwell box.** GPU 0 held free for large-model hot-loads (llama-swap pinned, `edf0f91`); GPU 1 steady-tenant — granite 131k ctx, qwen 65k, embed/rerank/reward trio, ~3.5 GB free (`1e2a3a1`, `19a07b9`; trio re-floored for 96 GB, 20×-parallel-stable). embed/rerank left at floor — long docs chunked BEFORE embedding. max-model-len is a ceiling not a reservation.
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-13]` **Prefix caching pinned explicit on granite + qwen** — benched ~6.5× faster TTFT on a shared ~4.5k-token summarizer template; soft/evictable, neutral when prefixes don't repeat. vLLM v1 defaults it ON (granite) but the qwen nightly defaults OFF — pin both. (`a9a2be7`)
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-13]` **granite-4.1-8b listed as the always-available summarizer/classifier + a shared all-agents key minted** (operator-directed). Global `~/.claude/CLAUDE.md` Global-tools entry; key alias `all-agents-local`, scoped to the FREE local models only (granite + qwen-vision + embed/rerank, NOT paid GLM), internal-gateway-only, rotatable. (auto-memory `reference_litellm_gateway`)
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-13]` **arbo engine + frontend stack stood up** (ADR-0001) — irv-ml1 co-located inference engine (`ee57e69`), python-based healthcheck (slim image, no curl/wget, `bdb3312`), frontend ro-mounted from the checkout (`922e8ad`, ADR-0001 D2).
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-11]` **GLM thinking inverted at the LiteLLM gateway** (operator call): `glm-5.1` defaults thinking-OFF; `glm-5.1-reasoning` = same z.ai upstream, thinking ON. Mechanism: `litellm_params.extra_body:{thinking:{type:disabled}}` — `drop_params` strips a top-level `thinking`/`reasoning_effort` but forwards `extra_body` verbatim to z.ai. Shared-gateway change. (`95b2701`, auto-memory `reference_litellm_gateway`)
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-11]` **pi coding agent installed on nh3-dev as a GLM 5.1 harness** — `@earendil-works/pi-coding-agent` via **bun** (npm's global prefix is `/usr` → needs sudo, bun avoids it). Config `~/.pi/agent/models.json`, launcher `~/.local/bin/glm`.
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-11]` **z.ai web-tools (regin) = z.ai hosted MCP path, NOT the `/paas/v4` Tool API.** WORKS: MCP streamable-HTTP at `https://api.z.ai/api/mcp/{web_search_prime,web_reader}/mcp`, `Authorization: Bearer $ZAI_API_KEY` (the **MCP** key, distinct from `Z_AI_API_KEY` the LLM key). Reference impl = Worldtree's Leif agent.
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-10]` **Mac Pro migration framed: workstation-only** (M2 Ultra ARM, racked NH3 on-subnet); sidecars stay Linux. `migration-plan.md`. (See in-flight.)
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-10]` **Worldtree deployed-config propagation is infra-ops's OWNED lane** (operator ruling). worldtree-dev pings the config delta pre-merge; infra-ops syncs `config/*.yaml` from MERGED canonical to the `/opt/worldtree*/config` bind-mounts on demo+personal. The v0.33.8 9-HOUR demo outage (a `model_roles.yaml` startup-dep that never reached the VM) is the failure mode this prevents. providers.yaml stays hand-tuned. corviduo emergency-ops = `ssh vh@10.250.50.152`, docker no-sudo. (auto-memory `reference_worldtree_deploys_cicd`, `reference_corviduo_dev_emergency_ops`)
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-09]` **LiteLLM scoped virtual keys issued to consumers** (operator-authorized): `brokkr-smithy` (all-proxy-models), `arbo-prompt-enhance` (comfy-dev — granite, later extended to qwen-vision). Mint via `/key/generate` (master `sk-corvid`), scope-restricted + rotatable, value → 600 file never the bus. (auto-memory `reference_litellm_gateway`)
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-14]` **ana-ml2 GPU-1 vision upgraded: Qwen3.5-9B → Qwen3.6-35B-A3B (official FP8), served under its TRUE name only.** `qwen36-vl` replaces `qwen35-vl` on :8007 (`a0fed13`). The stale `qwen3.5-9b-fp8` name is KILLED at vLLM AND the litellm gateway (404/400) — a model is NEVER aliased under a prior model's name (silent substitution = downstream footgun; operator directive). Consumer comfy-dev/arbo migrated; arbo vkeys → all-proxy-models; shared `all-agents-local` key repointed qwen3.5-9b-fp8 → qwen3.6-35b-a3b. GPU-1 rebalanced for the ~34 GB FP8 weights (granite 0.35→0.24/64K; embed/rerank 0.05→0.03, reclaimed ~4 GB util-waste). Validated: vision correct, 20-concurrent = no OOM. (auto-memory `feedback_no_false_model_aliases`)
|
||
_Archived 2026-06-18._
|
||
|
||
- `[2026-06-14]` **NVFP4 was the lighter fit (~21 GB) but is BLOCKED on vLLM — FP8 is the working vision path.** `nvidia/Qwen3.6-35B-A3B-NVFP4` won't load: the ModelOpt-NVFP4-MoE loader errors on expert/lm_head scale keys across 0.19.1 (`w2_input_scale`) AND 0.22.0 (`lm_head.input_scale`, vllm #44081) — a pattern across modelopt NVFP4 MoEs. Revisit NVFP4 (frees ~13 GB on GPU 1) once fixed; the 21 GB checkpoint stays cached on ana-ml2. **(SUPERSEDED 2026-06-16 — it loads on vLLM 0.23.0; qwen36 swapped to NVFP4. See the top of this section.)**
|
||
_Archived 2026-06-18._
|
||
|
||
- `[2026-06-14]` **llama-swap qwen3.5-9b GPU-0 pin DROPPED; GPU 0 reserved for a creative-writing model (pick DEFERRED by operator).** Deep-research (this session) on big-fast-uncensored creative for a 96 GB Blackwell: **GLM-Steam-106B-A12B** (already in the llama-swap config — balanced default) vs **TheDrummer/Behemoth-X-123B-v2** (prose-tier, tops UGI writing+willingness) vs XORTRON-123B (max willingness, weak prose); GGUF-on-llama-swap is the serving path. Tracking: this session + llama-swap config (GLM-Steam present, `untracked by operator choice`).
|
||
_Archived 2026-06-18._
|
||
|
||
- `[2026-06-14]` **R16 splice-pivot yield probe executed** (infra-ops ran the irv-ml1 inference for brokkr; brokkr owns design + analysis). See Current state. Tracking: althing thread `01KV010WGS…`, `gen_yield_probe.py` in `irv-ml1:~/r16-vmoan-harness`.
|
||
_Archived 2026-06-18._
|
||
|
||
- `[2026-06-14]` **R16 vmoan inline-generation arc CLOSED — v1 at default decode (rep1.2/temp0.8) is the final Chatterbox-tag inline artifact.** Operator's ear rejected every alternative: v2/v3 windowing (omission vs coherence-loss), v4 multi-tag (cohesion held but lost to capacity-competition), emergent inline-token modulation (degenerates, not modulates), and the gen-time decode-polish sweep (soft tamers cut the NVV itself — same omission family as v2; p0 baseline beat p1). All adapters v1–v4 + `tokenizer.json.v3bak` preserved on `irv-ml1:~/r16-vmoan-harness`. Likely-next direction (deferred, NOT formalized): generate→bin→splice + one-shot-clone NVV pipeline routing around the inline-coherence wall. Tracking: brokkr R16 journal + althing thread `01KV010WGSSMPWRNCPAGSPK15Y`.
|
||
_Archived 2026-06-18._
|
||
|
||
- `[2026-06-14]` **Arbo deploy pipeline fixed, hardened, and version-controlled.** Prod rebuilt v0.11.1 → **v0.11.6** backend; the webhook machinery (`arbo-deploy.sh` + `arbo-webhook.py`, :9009 HMAC listener) is now repo-tracked at `stacks/arbo/` (was host-only = recoverability foot-gun). Deploy reaches gitea via the INTERNAL route (`10.250.50.70:222`) and restarts the engine ONLY on `catalog/` changes (graphs/frontend per-request; warn on `src/`|`Dockerfile` only — pyproject/uv.lock churn every commit). Operator kept arbo stack ownership in **eshpfi** (not migrated to comfy-dev's repo). Secret + `.env` stay host-only. Tracking: `6d66bc2`, `6e58e57`, `stacks/arbo/README` Q5.
|
||
_Archived 2026-06-18._
|
||
|
||
- `[2026-06-15]` **qwen36 thinking-split at the gateway** (operator) — base
|
||
`qwen3.6-35b-a3b` = thinking-OFF default (the vision judge needs it), `…-a3b-thinking`
|
||
= opt-in reasoning. One hybrid checkpoint + `--reasoning-parser qwen3`; mirrors the
|
||
glm-5.1 split. The judge's enable_thinking=false is what makes the NVFP4 swap
|
||
transparent. (`6de0844`)
|
||
_Archived 2026-06-19._
|
||
|
||
- `[2026-06-15]` **arbo engine rebuilt v0.11.6 → v0.11.22 from main; catalog re-applied (v0.11.23) — curated `/workflows` footer live.** src/ schema changes (Model/Lora.summary, Workflow.blurb/keywords + the /workflows endpoint) need a MANUAL image rebuild (the webhook warns-only on `src/`/`Dockerfile`). Built from `/worktank/arbo/repo`, bumped `ARBO_IMAGE` in the host .env, recreated + verified healthy (auto-rollback armed, unneeded). comfy-dev's v0.11.20 catalog push had first crash-looped prod (old baked schema `extra=forbid` vs new git-mounted catalog fields → ValidationError); they reverted the catalog, so rebuild-from-main loaded clean, then they re-applied (v0.11.23). comfyui is a SEPARATE compose project — the rebuild didn't touch it.
|
||
_Archived 2026-06-19._
|
||
|
||
- `[2026-06-15]` **althing-core refreshed to v0.14.1 box-wide on nh3-dev** (forseti + heid both routed it; central-sequenced per DevOps-delegation since `uv tool install --force` swaps the shared binary all handles use). Fixes the monitor non-truncating-lock bug so `stop-monitor`/`--stop` reliably recovers orphans. The editable install (`uv tool install --force --editable .`) covers every nh3-dev handle; `--version` trued up 0.14.0→0.14.1; verified `--stop` works against a real orphan.
|
||
_Archived 2026-06-19._
|
||
|
||
- `[2026-06-15]` **ratatoskr Tier-3 affect live smoke GREEN** (consumer-confirmed). Brokered/minted consumer_id="ratatoskr" Heimdall HS256 key on personal Worldtree (the key IS BOTH the create-session Bearer AND ratatoskr's JWT verifier secret — plain utf-8 string, NOT base64/raw); injected ENV-ONLY to `~/.config/ratatoskr/provider.env` (mode 600) on nh3-dev; added `10.100.10.50:8390` to personal's `BIFROST_CLIENT_ALLOWED_HOSTS`; drove define→session (handshake 200)→turn→affect.emit (200, snapshot persisted). Worldtree signs Bifrost with the caller's raw Bearer verbatim. Drive commands + claims contract in althing thread `01KV4ZYSE3…`.
|
||
_Archived 2026-06-19._
|
||
|
||
- `[2026-06-15]` **Dense Qwen3-VL-32B-NVFP4 A/B'd as arbo's anatomy hero-judge vs the qwen3.6-35b-a3b MoE → dense LOST, pilot torn down, GPU 0 reclaimed.** Dense NVFP4 loads clean on vLLM where the MoE NVFP4 doesn't (no expert-scale keys); ~44 tok/s on Blackwell GPU 0; vision tower stays high-precision. But comfy-dev's ground-truth A/B: MoE wins (faster + dual-use as general LLM + slight fine-detail edge), and the subtle-proportion tier is below ALL VLs. **Don't re-propose a dense-VL judge swap.** (auto-memory `project-qwen3vl-dense-pilot`)
|
||
_Archived 2026-06-19._
|
||
|
||
- `[2026-06-15]` **MastMed prototype containerized + exposed publicly via cloudflared.** Self-contained static SPA (bun→caddy multi-stage build), basic-auth gated (operator's gating call), ana-docker:8089, cloudflared quick tunnel. See in-flight for the ephemeral-URL caveat + named-tunnel upgrade path.
|
||
_Archived 2026-06-19._
|
||
|
||
- `[2026-06-15]` **infra-ops bootstrapped onto corviduo-dev** (operator-directed) — infra-ops now has SSH + NOPASSWD sudo there, OVERRIDING the fleet bootstrap script's deliberate partner-box exclusion (explicit host arg). corviduo config edits no longer need `vh` + a sudo password. (auto-memory `reference_corviduo_dev_emergency_ops`)
|
||
_Archived 2026-06-19._
|
||
|
||
- `[2026-06-15]` **R18 clip+caption placement = ana-ml2** (operator: "ana-docker… ana-ml2 unless irv-ml1 has room"; irv-ml1 lacked free VRAM for the 30B-A3B captioner). Models staged + stub deployed; see in-flight + the hard real-voice gate.
|
||
_Archived 2026-06-19._
|
||
|
||
- `[2026-06-14]` **LiteLLM infra-ops admin key provisioned** (operator) — resolves the LiteLLM half of the credential-migration directive; use it for ALL gateway ops (NOT `sk-corvid`). Value at `~/.config/litellm/infra-ops-key` (mode 600); gateway reachable directly from nh3-dev at 10.250.50.70:4000. (auto-memory `reference_litellm_infra_ops_key`)
|
||
_Archived 2026-06-19._
|
||
|
||
- `[2026-06-16]` **litellm `strip_empty_tools` pre-call hook shipped** (`d1bea13`) — an empty `tools:[]` 500s vLLM ("tools must not be an empty array"); a global `litellm_settings.callbacks` CustomLogger pops it (+ orphaned `tool_choice`) before forwarding, so it covers EVERY vLLM model, not one. `drop_params` only drops unsupported PARAMS, not empty VALUES. Mounts beside config.yaml (litellm resolves callbacks relative to the config dir). Verified live across granite/mistral/stream. (`stacks/litellm/conf/strip_empty_tools.py`)
|
||
_Archived 2026-06-20._
|
||
|
||
- `[2026-06-16]` **single-file `gateway-chat.html` playground shipped** (`984ca3d`, `tools/`) — zero-dep browser chat straight to the gateway (`:4000`, CORS open), system-prompt box, streaming SSE, renders `reasoning_content`, NEVER sends `tools`. Built because the LiteLLM admin-UI playground can't test vLLM-backed models (see Tried-and-abandoned). Serve on-request via `python3 -m http.server -d tools`.
|
||
_Archived 2026-06-20._
|
||
|
||
- `[2026-06-16]` **claude-bot granted ADMIN on vh/arbo** (operator) — arbo CI/CD gitea ops now route through the service account, not the operator's personal `vh` (the credential-migration directive). Operator added claude-bot as Administrator collaborator; claude-bot self-serves via its `gitea-password` (basic-auth = FULL perms; its stored token is package-write-only, can't change repo settings). (auto-memory `reference_claude_bot_gitea_creds`)
|
||
_Archived 2026-06-20._
|
||
|
||
- `[2026-06-16]` **ratatoskr Tier-3 MEMORY plane wired + live-smoke driven.** The missing server-side step was the per-PORT `BIFROST_CLIENT_ALLOWED_HOSTS` allowlist (`:8391`; consumer key reused, plane-agnostic — my earlier "reachability is the whole wiring" was incomplete). Persist + dispatch proven end-to-end against personal v0.35.3; cross-session recall-INJECTION root-caused to the `score_breakdown` shape seam and handed to worldtree-dev (#295 recall leg open). Driving a real memory-call needs a consumer agent defined WITH a `memory` block (`ctx.memory_config≠None`), not `agent_self_enabled`. (tracking: c99aa49 + auto-memory `reference_bifrost_plane_wiring`) — **[2026-06-16] FLIPPED:** the `score_breakdown`-shape root-cause was disproven; real cause = cold-recall `scope_filter` adds an `agent_self` axis → ratatoskr's conjunctive INV-005 match drops the `{end_user}`-only chunks → 0 hits. Now Worldtree #297. (auto-memory `project_worldtree_295_recall_rootcause`)
|
||
_Archived 2026-06-20._
|
||
|
||
- `[2026-06-16]` **infra-ops = the DURABLE admin path on corviduo-dev** (operator granted standing admin; extends the 2026-06-15 bootstrap) — added a `Host corviduo-dev → User infra-ops` `~/.ssh/config` alias + repointed `servers/corviduo-dev/ssh-target` to `infra-ops@`; `ssh corviduo-dev` / `elway corviduo-dev` now land as infra-ops w/ NOPASSWD (vh = explicit-only human path). elway resolves via ssh-config, NOT the ssh-target file. (tracking: c99aa49 + auto-memory `reference_corviduo_dev_emergency_ops`)
|
||
_Archived 2026-06-20._
|
||
|
||
- `[2026-06-16]` **demo + personal `character` model flipped qwen3.6-35-a3b → mistral-small-4** (operator) — in `model_roles.yaml` the FIRST `catalog_id` in a role's `binds:` is the default; reordered mistral-first (qwen retained in the switch-allowlist, non-destructive), applied via the PIN-SAFE api recreate (derive `WORLDTREE_IMAGE` from the matrix sibling — never a bare `compose up`, which falls to the crash-blocking stale `:latest`), verified a fresh character agent resolves model=mistral-small-4. (auto-memory `reference_corviduo_dev_emergency_ops`)
|
||
_Archived 2026-06-20._
|
||
|
||
- `[2026-06-16]` **NVFP4 ModelOpt MoE now LOADS on vLLM 0.23.0 — SUPERSEDES the
|
||
2026-06-14 "blocked" finding (#44081 fixed).** Confirmed the cached
|
||
`nvidia/Qwen3.6-35B-A3B-NVFP4` loads clean + generates on 0.23.0; swapped prod
|
||
qwen36 FP8→NVFP4 (~13 GB reclaimed on GPU 1, fp16 KV). Validated by brokkr (speed:
|
||
faster single-stream, ≥parity batched) + comfy-dev (vision anatomy-judge A/B: PASS).
|
||
One W4A4 tell: minor multi-step chained-reasoning slip — doesn't bite the vision
|
||
judge. Mistral's NVFP4 is a SEPARATE compressed-tensors path (day-0), distinct from
|
||
the ModelOpt one. (auto-memory `reference_nvfp4_moe_loads_on_vllm_023`)
|
||
_Archived 2026-06-20._
|
||
|
||
- `[2026-06-16]` **Mistral Small 4 = the GPU-0 tenant** (operator) — fills the slot
|
||
the deferred creative-writing pick held, and supersedes llama-swap (now down).
|
||
NVFP4 119B MoE, pinned **vLLM v0.22.0** for working vision (0.23.0 breaks it),
|
||
256K context. Operator's creative-writing model; **abliteration planned** → then it
|
||
becomes the uncensored-creative model the persona substrate needs. Reasoning via
|
||
`reasoning_effort` (BINARY none|high — no medium). (tracking: `c77a9aa`/`9a49963`)
|
||
_Archived 2026-06-20._
|
||
|
||
- `[2026-06-16]` **Selene judge restored on vLLM FP8, NOT NVFP4** — W4A4 is too
|
||
aggressive for a precision judge validated at Q6_K; FP8 ≥ Q6_K fidelity, and Selene
|
||
is text-only Llama-3.1 so dynamic `--quantization fp8` has no vision tower to noise.
|
||
`selene-1-mini-8b` on GPU 1 :8011. Worldtree's Domari role repointed to it. (`c985ede`)
|
||
_Archived 2026-06-20._
|
||
|
||
- `[2026-06-16]` **GPU-1 buffer put to work: selene 0.13→0.17 + qwen36 0.32→0.34**
|
||
— the two KV-bound generative models that gained Worldtree consumers; granite + the
|
||
pooling models under-use their util, so growing THEM is wasted. (`e124a2f`)
|
||
_Archived 2026-06-20._
|
||
|
||
- `[2026-06-16]` **claude-bot Gitea service account stood up; corviduo-ORG conversion
|
||
TABLED** (operator). claude-bot is the credential-migration foundation. The proper
|
||
fix for scoped agent package-write is converting vh (a USER) → a `corviduo` ORG —
|
||
blast radius assessed (org creation = free; arbo-first migration = cheap, no registry
|
||
images; full fleet = incremental, ~8 active vh images / 50 repos) — but tabled for
|
||
later. Registry-push deferred until then. (auto-memory `project_migrate_infra_access_to_claude_credentials`)
|
||
_Archived 2026-06-20._
|
||
|
||
- `[2026-06-16]` **arbo engine builds → comfy-dev OWNERSHIP** (operator approved) +
|
||
Gitea Actions runner stood up on irv-ml1 (host-executor as lkraven). infra-ops keeps
|
||
box-level escalation (runner, package:write PAT pending corviduo-org, webhook). See
|
||
the runner entry in Current state. (tracking: this snapshot + arbo thread `01KV70AB2B4B…`)
|
||
_Archived 2026-06-20._
|
||
|
||
- `[2026-06-16]` **Worldtree persona substrate migrated qwen→Mistral via capability-
|
||
named profiles** (v0.35.8→v0.35.10; demo + personal). PRE-SYNC-FIRST is now the
|
||
standing pattern for any providers.yaml/profile change: sync config to the bind-mount
|
||
BEFORE the deploy image-swap restart, so new agents + new config land together (the
|
||
v0.35.9 merge-first attempt race'd → 14 crash-loops). (tracking: worldtree thread `01KV6X0M…`)
|
||
_Archived 2026-06-20._
|
||
|
||
- `[2026-06-19]` **litellm task-aliases: `classifier`→granite-4.1-8b, `summarizer-large`→gen/qwopus** (`summarizer`→granite already existed). Also added then REMOVED `gen-nt`/`gen-reasoning-nt` (redundant with the global `strip_empty_tools` hook). (ef45f6d, 2e5ab72, 75bd4c3)
|
||
_Archived 2026-06-21._
|
||
|
||
- `[2026-06-19]` **gateway-chat: model-smoking web chat → PERSISTENT container.** `/v1/models` auto-discovery + image-upload; `stacks/gateway-chat/` (nginx:alpine, ana-docker:8091, homepage tile). (4c40b9f, 740bcae)
|
||
_Archived 2026-06-21._
|
||
|
||
- `[2026-06-19]` **pi on nh3-dev wired to `gen`** — `~/.pi/agent/models.json` + a `gen` launcher mirroring `glm`. Local box config, not a repo artifact. (untracked by operator choice)
|
||
_Archived 2026-06-21._
|
||
|
||
- `[2026-06-19]` **`gen` model → Qwopus3.5-122B-A10B (vision-intact NVFP4), full 256K @ fp8.** ana-ml2 GPU 0, served-name `qwen3.5-122-a10b`. STABLE vLLM + fp8 KV reaches 262144 — nightly+turboquant-4bit-KV was UNNECESSARY. 92.7 tok/s warm. (20e796c, 5b06514)
|
||
_Archived 2026-06-21._
|
||
|
||
- `[2026-06-19]` **TP=2 across the two ana-ml2 Blackwells REJECTED** — `PIX` (PCIe, NO NVLink) → all-reduce-bound → SLOWER. One-model-per-card is optimal; PP=2 only if a >96 GB model is ever wanted. (untracked by operator choice)
|
||
_Archived 2026-06-21._
|
||
|
||
- `[2026-06-19]` **qwen-image-bench (T2I judge, NVFP4) replaced qwen3.6-35b-a3b on GPU 1**, aliased `image-judge`. (bfae924, 5dfce04)
|
||
_Archived 2026-06-21._
|
||
|
||
- `[2026-06-18]` **heretic abliterated Mistral Small 4 NVFP4 built + LIVE as `mistral-small-4`** (in-house NVFP4, drop-in stack, A/B'd vs official, "heretic stays"). DISPLACED from GPU 0 on 2026-06-19 for the gen/Qwopus swap. (dd3a5c9, f566f61, `tools/mistral-small4-nvfp4/`)
|
||
_Archived 2026-06-21._
|
||
|
||
- `[2026-06-18]` **irv-ml1 VRAM consolidation + comfy-dev torch-pin** — ComfyUI pinned to the A6000 (48 GB), audio zoo → 3090; `DISABLE_UPGRADES=true` @ torch 2.12.1. (a8550ad)
|
||
_Archived 2026-06-21._
|
||
|
||
- `[2026-06-18]` **ComfyUI acceleration set (9 nodes) installed for comfy-dev** on irv-ml1.
|
||
_Archived 2026-06-21._
|
||
|
||
- `[2026-06-17]` **ana-ml2 `vm.overcommit_memory=1` made durable** (sysctl drop-in) — overcommit=0 + zero swap caps CommitLimit; large model-file mmap ENOMEM'd despite ~393 GB free. (fc88eff)
|
||
_Archived 2026-06-21._
|
||
|
||
- `[2026-06-17]` **GLM 5.2 wired into litellm** (`glm-5.2` + `-reasoning`, z.ai passthrough, `extra_body.thinking.type` toggle). (fe77a35)
|
||
_Archived 2026-06-21._
|
||
|
||
- `[2026-06-17]` **nh3-extdev stood up as a sudo-LESS infra-ops manager box** (successor to nh3-ansible); hosts `/opt/externs` pi-on-GLM-5.2 client workspaces. (a841eab)
|
||
_Archived 2026-06-21._
|
||
|
||
- `[2026-06-20]` **Worldtree capability-gateway aliases stood up; NO `wt-` prefix (operator) — reuse the existing capability convention.** gen/gen-reasoning/classifier/summarizer/chat-judge/reranker/scalar-judge (passthrough)/gen-frontier/-reasoning. The **swap-transparency rule**: embedding stays model-named (`qwen3-embedding`) because vectors aren't swap-transparent; everything else can swap behind a stable alias. (d372103, 7819f96)
|
||
_Archived 2026-06-25._
|
||
|
||
- `[2026-06-20]` **Removed the litellm `*` wildcard → llama-swap** (operator; llama-swap decommissioned, :9292 confirmed down). Unknown/typo'd models now 404 cleanly instead of a misleading "Connection error" — closes the long-standing footgun. (d0eb09c)
|
||
_Archived 2026-06-25._
|
||
|
||
- `[2026-06-20]` **Langfuse retired** (operator) — redundant with LiteLLM native spend_logs + a 6-container crash-prone surface (the 94 GB clickhouse spew). Callbacks dropped, stack torn down. (f8eda1c)
|
||
_Archived 2026-06-25._
|
||
|
||
- `[2026-06-20]` **OmniVoice streaming `/tts` — vendor chatterbox-fast's scheduler, not its whole package** (operator-approved vendor-copy). Diffusion TTFA floor finding → tuned 16-step streaming + aggressive packing; chatterbox-fast remains the lowest-latency front-end. (288d085, cd92b85)
|
||
_Archived 2026-06-25._
|
||
|
||
- `[2026-06-20]` **Daily image+buildx prune cron on ana-docker** (the missing automation behind the recurring image bloat) — `lkraven` crontab, prune unused images >48 h + per-builder buildx cache. The real systemic fix (daemon log cap) is deferred to clean context.
|
||
_Archived 2026-06-25._
|
||
|
||
|
||
- `[2026-06-25]` **althing re-architected to the lean multi-machine bus; nh3-extdev stood up as a
|
||
MODEL B mesh peer** (dedicated `althing-svc` + group-shared `/srv/althing`). `reference_nh3_extdev_althing_mesh`.
|
||
_Archived 2026-07-08._
|
||
|
||
- `[2026-06-23]` **zellij native web client piloted on nh3-dev** (`zellij-web.service` :8443)
|
||
alongside ttyd. `reference_zellij_web_seat`.
|
||
_Archived 2026-07-08._
|
||
|
||
- `[2026-06-22]` **Worldtree persona-render config arc (#314/#322/#317) pre-synced + deployed green
|
||
on demo+personal** — #317 a boot-blocking config REMOVAL. `reference_corviduo_dev_emergency_ops`.
|
||
_Archived 2026-07-08._
|
||
|
||
- `[2026-06-20]` **R22 (brokkr/dwarves) stood down to gateway-only; full-access R22 key minted;
|
||
Phase B CANCELLED** (Worldtree model-agnostic → no deploy path). Key at `/home/lkraven/.r22-gateway-key`
|
||
(persistent mode-600, carries paid GLM, don't delete). MUT = free `qwen3.5-122-a10b` (`gen`).
|
||
Operator steer: R22 research is gated on a pragmatic/deployable outcome, not advancing-the-art.
|
||
_Archived 2026-07-08._
|
||
|
||
- `[2026-06-20]` **claude-bot issue-scope token minted for worldtree-dev self-serve** (id 16,
|
||
`write:repository`+`write:issue`); old token revoked. Advances the credential-migration directive.
|
||
_Archived 2026-07-08._
|
||
|
||
- `[2026-06-20]` **rest-server-ana recovered + backup prevention shipped + worldtree-dev admin keys
|
||
provisioned** (demo d113207c / personal f4f75adb). Cred rotation (5 rest-server pw) BELAYED.
|
||
_Archived 2026-07-08._
|
||
|
||
- `[2026-06-20]` **claude-bot → ADMIN on vh/Worldtree** (operator-authorized) — self-serves WT
|
||
deploys/tokens henceforth.
|
||
_Archived 2026-07-08._
|
||
|
||
- `[2026-06-14]` **STANDING DIRECTIVE: migrate ALL infra access to Claude-specific credentials.**
|
||
(auto-memory `project_migrate_infra_access_to_claude_credentials`)
|
||
_Archived 2026-07-13._
|
||
|
||
- `[2026-07-01]` **Worldtree #332 embed-recall diagnosed + scoped-log view/tunnel provisioned + fix
|
||
verified** (15×→1.01× re-embed). `reference_wt_gateway_scoped_log_view`.
|
||
_Archived 2026-07-13._
|
||
|
||
- `[2026-07-01]` **qwopus native MTP speculative-decode tested on `gen` → NOT kept** (+12% single-stream,
|
||
−15–20% aggregate at concurrency, silently drops min_p/logit_bias). Banked for T1. `reference_gen_qwopus_122b`.
|
||
_Archived 2026-07-13._
|
||
|
||
- `[2026-07-01]` **Deckard trial → reverted to qwopus (`gen`)** (won writing "in every way" but ~36 vs
|
||
~90 tok/s; spec-decode rescue ruled out). git `b63c48b`→`681eb70`. Deckard kept staged as T1's
|
||
writing benchmark.
|
||
_Archived 2026-07-13._
|
||
|
||
- `[2026-07-02]` **mtf-dev granite harness-spike ran GREEN — MECHANICAL only, efficacy DEFERRED to
|
||
the T1 run.** Trainer TRL SFT→DPO→eval seam proven end-to-end on a synthetic fixture (not the E-RP
|
||
corpus); operator DECIDED no intermediate real-efficacy granite spike (uninterpretable proxy —
|
||
arch gap + abliteration axis). `reference_gen_qwopus_122b`.
|
||
_Archived 2026-07-13._
|
||
|
||
- `[2026-07-03]` **ratatoskr-dev DEMO Heimdall key provisioned (R30 φ0).** Minted a tier-user key on
|
||
the demo via `POST /admin/keys` (bootstrap admin key), mirroring their personal base consumer (no
|
||
character-binding); base-agent affect reads work ungated. `reference_worldtree_demo_key_mint`.
|
||
_Archived 2026-07-13._
|
||
|
||
- `[2026-07-04]` **character-rp: gateway-forwarding proven + role pre-staged + #344 shipped.**
|
||
Empirically confirmed per-request `extra_body` (top_k/repetition_penalty) forwards through the
|
||
`gen-reasoning` LiteLLM alias to vLLM + standard params override the alias defaults — no gateway
|
||
cap needed (I over-built a dedicated alias, operator corrected, reverted with zero fleet impact).
|
||
Pre-staged the `character-rp` role into demo+personal bind-mount `model_roles.yaml` (byte-verified
|
||
on b18; caught the cached-registry ordering). worldtree-dev shipped **#344 (v1.0.0b19)** for the
|
||
durable-agent override-drop. spend_logs spot-check queued (ratatoskr's timestamp ping).
|
||
_Archived 2026-07-13._
|
||
|
||
- `[2026-07-04]` **althing v2 herald+receiver formalized as systemd on nh3-dev.** `althing-herald.service`
|
||
(Restart=always, **Environment=PATH incl ~/.cargo/bin** — the pane-dispatch fix) + `althing-receiver.service`
|
||
(v2 → pillar-3 `/owner/*` live); stale forseti unit removed; both on v2.0.0b2, canonicalized by
|
||
forseti. `reference_nh3_dev_althing_herald`.
|
||
_Archived 2026-07-13._
|
||
|
||
- `[2026-07-04]` **LitBench-RM respun (irv-ml1 A6000, comfyui displaced)** for T1's reward ensemble;
|
||
operator sole comfyui consumer, holding image-gen until LitBench done. `reference_litbench_rm_irv_ml1`.
|
||
_Archived 2026-07-13._
|
||
|
||
- `[2026-07-05]` **T1 training venue: CLOUD recommended; operator chose smoke-first on ana-ml2.**
|
||
On-prem ruled out (ana-ml2 full — both 96G GPUs ~93G used): keep-serving = NVMe offload ~6-8 DAYS;
|
||
full ana-ml2 shutdown = CPU offload ~1 DAY but a whole-fleet outage. Cloud Vast.ai 8×80GB (no
|
||
offload → ~3-6h, ~$60-500, zero fleet impact) is the clean alt (mtf-dev + infra-ops both rec;
|
||
Vast for its no-content-AUP marketplace + likely-existing VastBlue account). Operator's next step
|
||
= the ana-ml2 CPU-offload SMOKE (~60 steps) to get real samples/sec before the full-outage-vs-cloud
|
||
call. HF base verified public (zero-auth pull). Runbook + gotchas in `reference_t1_cloud_train_plan`.
|
||
_Archived 2026-07-13._
|
||
|
||
- `[2026-07-05]` **glm-5.2 canonical limits recorded** (probed live vs z.ai): **1,048,576 (1M) input
|
||
context / 131,072 (128K) max output**; NO gateway-side cap (pure passthrough → z.ai's limits are
|
||
effective). Written to the config comment (commit `624a07e`) + `reference_litellm_gateway`.
|
||
_Archived 2026-07-13._
|
||
|
||
- `[2026-07-06]` **infra-ops NOPASSWD sudo confirmed on irv-ml1 + ana-ml2** (the arbo memory's "NOT
|
||
infra-ops on irv-ml1" was stale) — enabled the worker deploy + every ana-ml2 hot-swap smoke.
|
||
_Archived 2026-07-13._
|
||
|
||
- `[2026-07-07]` **SGLang zc hot-swap: 2-line patch RAN → PARTIAL. Cleared startup + self-attn E2E, GDN
|
||
in_proj hit 2 MORE name-routing gaps → MERGE recommended.** Root cause was a NAME-routing gap (not the
|
||
feared shape-assert): `lora/utils.py:235` hardcoded `params_mapping` lacked `in_proj_qkv/in_proj_z →
|
||
in_proj_qkvz`. Patched those 2 lines (overlay bind-mount) → `init_lora_shapes` passed, server up, self-attn
|
||
loaded end-to-end. But the SAME missing GDN mapping bit 2 MORE resolvers: adapter-load skipped 192 in_proj
|
||
weights, and inference `utils.py:300 get_target_module_name` (substring match) RAISED → 500 + scheduler
|
||
died. So GDN LoRA name-routing is incomplete in ≥3 SGLang paths (fixed 1). NOT shape-intractable (self-attn
|
||
proves the separate-key→fused-buffer machinery works), but NOT "one targeted fix" either → per operator's
|
||
LAST-shot rule → MERGE. mtf-dev looped + closed (thread 01KWZDZFYN…); B/A shard-math never reached.
|
||
Supersedes the "SGLang wants a THIRD format" framing — no new adapter format needed.
|
||
_Archived 2026-07-13._
|
||
|
||
- `[2026-07-07]` **T1 (retargeted qwopus-122B→AEON-27B) SFT COMPLETE on ana-ml2** — all-TRL torch-GDN,
|
||
batch1/accum8/GC-off, 1 epoch ~9h @ 0.25 s/s, loss 2.445, adapter banked. Supersedes the 2026-07-05
|
||
cloud-vs-smoke venue decision: AEON-27B's 52G bf16 fits one Blackwell → ran on-prem in ~9h (no cloud,
|
||
no full-fleet outage). GC is NOT the throughput lever (batch1-no-GC == batch4+GC at 0.25; dvalin's 0.565
|
||
was warmup-noise from a 30-sample smoke); ~0.25 s/s is the torch-GDN floor (only FLA beats it, sm_120-
|
||
blocked #913). `reference_t1_cloud_train_plan`.
|
||
_Archived 2026-07-13._
|
||
|
||
- `[2026-07-07]` **T1 hot-swap on vLLM NVFP4 is DEAD — vLLM 0.24.0 qwen3_5 LoRA-application silent no-op**
|
||
(#47639 open / fix PR #47640 open / regression from #37912). Proven quant-agnostic (NVFP4+FP8) AND
|
||
adapter-format-agnostic (separate `zc` loads-but-inert; fused rejected); #47640 guard-patch overlay didn't
|
||
fix. Root-caused by a 3-peer dwarf panel (eitri/regin/dvalin); their vLLM issue refs verified real via
|
||
GitHub API. Engine mirror-image: vLLM wants SEPARATE keys, SGLang wants FUSED.
|
||
_Archived 2026-07-13._
|
||
|
||
- `[2026-07-07]` **HARD REQUIREMENT restated: hot-swappable LoRA or ABANDON.** Merge is NOT a solution for
|
||
this requirement (static; reserved for a later RP-quant track). SGLang is the last hot-swap shot; if it
|
||
can't apply, park the adapter + file upstream repros. `reference_lora_training_worker` (adjacent).
|
||
_Archived 2026-07-13._
|
||
|
||
- `[2026-07-07]` **comfy-dev in-arbo LoRA-training worker DONE** — built/deployed/validated Phases 1+2
|
||
(first Sindra train succeeded, publish-step + `train_id`), Phase 2.5 GC contract (shape A) confirmed +
|
||
impl deferred. `services/lora-training-worker/`, commits 888ba6a→b617a8b. `reference_lora_training_worker`.
|
||
_Archived 2026-07-13._
|
||
|
||
- `[2026-08-08]` **worldtree-dev #400 CLOSED → fiction-decomp snapshot cleared from nh3-dev.** worldtree-dev signaled #400 done (shipped v1.0.0b185; exact-lexical efficacy 79%→12% on ratatoskr's gate, brokkr no-harm bracket green both ends; the snapshot served 4 probe rounds — rank decomposition, promoted-vs-gold annotation, tie-set falsification, A0/A1/A2 mechanism probe). Cleared `~/snapshots/worldtree-400-fiction-decomp` (208M: chroma + manifest/provenance/stamp) — a read-only rsync copy of PERSONAL Worldtree's Chroma (source on corviduo-dev, so safe to remove). **LEFT INTACT:** `rex393-fiction-index`/`rex393-fiction-snapshot` (separate operator KEEP word, unchanged) + `r42-gate-*`. No config deltas rode this train. Only remaining non-blocking await = ratatoskr-dev's chatterbox-fast knob revert. Replied confirming (`01KZJ9GMCC…`).
|
||
_Archived 2026-08-24._
|
||
|
||
- `[2026-08-07]` **chatterbox-fast "broken audio" root-caused (T3 AR tail over-run) + FIXED (max_chunk_chars=250 cap, :v2 deployed).** Long saga, operator-driven clean diagnosis. **Symptom:** ratatoskr's migrated RP-surface TTS "swaps to German" / "dead air" / "garbage" on long turns. **NOT** German-leak (Turbo `generate()` has NO language param — plain AutoTokenizer, no `language_id`; the multilingual `language_id="en"` lever lives only in the separate `ChatterboxMultilingualTTS`), **NOT** OOM alone. **Real cause:** the Chatterbox **Turbo T3 model OVER-RUNS its generation tail** — a long single `generate()` degrades into garble/dead-air in its final ~2-3s (lib filters OOV tokens `<6561` + pads silence = messy AR tail). The scheduler's buffer-ratchet builds 300-600 char mega-chunks that land in that zone; streaming concatenates each bad tail (worst case). **ratatoskr's anti-"German" knobs (top_k=80/temp=0.5) made it WORSE** — tight sampling pulls the degradation onset SHORTER (~200 chars vs ~300 at default knobs). **Diagnosis method** (deterministic, no ears-only): single-shot length sweep + **amplitude-gated voiced-ZCR** (garble spikes ZCR; must gate on |x|>500 else trailing silence confounds it) — degraded voiced-tail = 1.58× mid, clean = ~0.64-1.1×. **FIX:** server-side `max_chunk_chars=250` cap on the scheduler (`:v2` image, `CBF_MAX_CHUNK_CHARS=250` env) — bounds each generation to just under the ~300-char onset → clean **3-4 sentence** chunks (max prosodic arc while clean). Operator ear-confirmed clean audio + clean joins; **chatterbox's low emotiveness keeps chunk joins smooth** (the harsh joins that got Zonos rejected are absent — operator's key call). **ratatoskr TODO (relayed msg `01KZER9X7S`):** revert knobs to default (top_k→1000, temp→0.8), send full text (server chunks internally), keep the 503-on-empty guard. **Cap value tunable** per-request (`max_chunk_chars`) + env. **Deeper prosody** (if ever wanted) = scheduler Phase-2 context-priming at joins (feed prior sentence as discarded-audio context; +latency). **⚠ FOOT-GUNS:** (1) acoustic tail-trim is UNRELIABLE — sibilants ('s'/'sh'/'f') spike ZCR like garble, can't cleanly detect the speech→garble boundary. (2) **build-context vs image drift** — the `:v2` image was built from cap source, but after a `:v1` rollback the build context held `:v1` source → a `docker compose build` would've silently produced a cap-less `:v2`; re-synced the flat cap source to `/opt/docker/compose/chatterbox-fast/` (rebuild-verified). **⚠ DIVERGENCE (follow-up):** deployed build context is FLAT (`app.py`/`scheduler.py`, `from scheduler import`, thin-overlay `FROM local/chatterbox:v1`, cap-only) vs the `vh/chatterbox-fast` REPO which is PACKAGE-layout (`chatterbox_fast/`, `from chatterbox_fast.scheduler`, self-contained Dockerfile) + has `norm_loudness` (repo commit `6bc7bf0` = cap; deployed omits norm_loudness deliberately to keep the ear-test unconfounded). Reconcile the two layouts so a repo-based rebuild matches deploy. Rollback: `.bak-cap-20260807-104850` backups on irv-ml1 + `:v1` image both retained. [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]]
|
||
_Archived 2026-08-24._
|
||
|
||
- `[2026-08-07]` **Zonos2 TAKEN DOWN on the 3090 (irv-ml1) — operator-directed "for memory", TEMPORARY.** Freed ~17.4 GB (3090: 728 MiB → 18.2 GB free) so chatterbox-fast (co-resident, was OOMing on long generations) has headroom. **⚠ Restore is manual — Zonos2 :1920 was a DETACHED native process (NOT systemd/docker), reparented to init.** GPU memory was held by the `--multiprocessing-fork` CHILDREN (1966165=16.4G, 1966166=1G), which ORPHAN to init when you kill the parent — had to SIGTERM the children explicitly (killing the parent 1965942 + uv-run 1965935 alone left the 16.4G held). **RESTORE CMD** (from irv-ml1, user lkraven): `cd /home/lkraven/tts-audition/models/zonos2 && nohup uv run python -m zonos2 --model-path Zyphra/ZONOS2 --host 0.0.0.0 --port 1920 --tts-default-voices-dir ./default_voices/ --cuda-graph-max-bs 1 --num-pages 16384 --max-running-requests 2 --memory-ratio 0.3 > /tmp/zonos2.log 2>&1 &` then `docker start zonos-gateway`. **Consumers that lost Zonos:** asset-engine + gateway-chat (via LiteLLM `ext-tts` alias → zonos-gateway :8890, now stopped); ratatoskr already migrated OFF to chatterbox-fast (unaffected). Also unblocks proper drift/cap testing (OOM was blocking it). [[reference_zonos_tts_stack]]
|
||
_Archived 2026-08-24._
|
||
|
||
- `[2026-08-07]` **chatterbox-fast: donut voice added + full contract delivered to ratatoskr-dev (their TTS migration off Zonos).** Operator-directed. Copied `zonos-gateway/voices/Donut.wav` → chatterbox `/refs` (`/worktank/chatterbox/reference_audio/donut.wav` — the reference_audio SUBDIR is lkraven-owned so no sudo despite `/worktank` root; container globs `/refs` live → **NO restart**), exposed as `voice:"donut"` (lowercase); verified clean 7.5s synth (24kHz, RTF ~0.31). A/B booth (chatterbox vs zonos donut, same line) at `http://10.100.10.50:8090/b/donut-chatterbox/`. Answered ratatoskr's 8-question contract ask from the live gateway (`local/chatterbox-fast:v1`) + source: **NOT OpenAI-shaped** (`POST /tts`; body `text`/`voice`/`format`/`stream`, not `input`/`model`/`response_format`); **NO affect dials** (Turbo ignores cfg_weight/min_p/exaggeration — the architecture-changing answer they flagged; **Zonos stays the only fleet TTS with real emotion steering**); streaming WAV placeholder-header shape IDENTICAL to Zonos (their per-chunk Web Audio path survives); SR 24000 (Zonos 44100); server chunks arbitrary-length text internally (no client-side chunking, unlike Zonos's 71.2s cap); English-only, no language pin. **FYI-worthy (operator):** ratatoskr is moving its RP-surface TTS OFF Zonos back to chatterbox-fast → loses the live-PAD affect coupling (heavy Zonos emotion investment) — their call, trade-off flagged to them. auto-memory `reference_chatterbox_fast_repo` enriched w/ the live contract. [[reference_zonos_tts_stack]]
|
||
_Archived 2026-08-24._
|
||
|
||
- `[2026-08-07]` **Fleet reranker cut over: Qwen3-Reranker-0.6B → BAAI/bge-reranker-v2-m3 (Brokkr R43).** The incumbent was measured HARMING 80/90 fleet queries (no-reranker beat it 89/90 vs 56/90). R43 bake-off: the A2 control (same Qwen weights, seq-cls head) scored identical to the incumbent → proved the fault is a training-prior not the serving head → cancelled the expensive Qwen3-4B arm; A3 (bge-v2-m3) won on multilingual safety + bare-name recovery. LiteLLM `reranker` repointed incumbent→A3 :8013 (boundary 2026-08-06T17:37:48Z, config-edit + ~52s gateway restart); **R42 v13 gate PASSED first-ever** (56/90→90/90). Incumbent kept warm :8002 (rollback via `qwen3-reranker` alias), A4 fallback :8014. Full arc + rollback runbook `docs/pfi/reranker-selection-ledger.md`; commits ad2df89/2c11748/377f8a4 (unpushed). auto-memories: the earlier reranker-serving notes.
|
||
_Archived 2026-08-24._
|
||
|
||
- `[2026-08-27]` **Run 3 gated: the preregistered rule PASSED and a k=25 follow-up found a 44pp self-harm guardrail collapse — DO NOT SERVE.** A pooled preserve-list test structurally cannot see a single-axis collapse. → `persistent-memory.d/2026-08-27-run3-gate-safety-regression.md`
|
||
_Archived 2026-09-11._
|
||
|
||
- `[2026-08-27]` **The corpus mix was specified in a unit the optimiser never sees** — 45.8% dialogue by CONTEXT, 24.2% by LOSS. Harness now leads with loss share and calls context a memory budget (`dd5a12e`). → `persistent-memory.d/2026-08-27-mix-specified-in-the-wrong-unit.md`
|
||
_Archived 2026-09-11._
|
||
|
||
- `[2026-08-27]` **Dose-response: benefit and damage are ONE direction in weight space** — every axis monotone in scale, no knee. The merge-back cannot separate them; vLLM cannot LoRA-serve this MoE at all. → `persistent-memory.d/2026-08-27-dose-response-entanglement.md`
|
||
_Archived 2026-09-11._
|
||
|
||
- `[2026-08-27]` **Anaheim tripped a power breaker; four guests including the NAS had `onboot` unset and never came back.** Fixed with dependency ordering — ana-nas order=1,up=45 ahead of the databases. ⚠ **ONE CIRCUIT FEEDS THE WHOLE RACK including the firewall serving the public IP** (operator) — so ana-gw, ana-wg and every BMC go down with the load, and there is NO remote management path to Anaheim during a power event. → `persistent-memory.d/2026-08-27-anaheim-breaker-and-onboot-gap.md`
|
||
_Archived 2026-09-11._
|
||
|
||
- `[2026-08-27]` **A transport failure that enters a measurement as a VALUE looks like whatever you hoped to find.** heid's lost panel arms found a live defect in brokkr's `t4_dissect` an hour later. → `persistent-memory.d/2026-08-27-empty-response-as-a-datum.md`
|
||
_Archived 2026-09-11._
|
||
|
||
- `[2026-08-27]` **Run 3c authorised (lr 20x cut, single variable) and then HELD by the operator after the breaker trip.** Config built and validated at `/tank/erp-tune/run-03c.json`; `save_steps` made configurable in the harness (`0a6bd2e`) because the first launch lost 80 steps with no checkpoint. Tracking surface: commit `0a6bd2e` + that config path. **Relaunch is one command once power is triaged.**
|
||
_Archived 2026-09-11._
|
||
|
||
- `[2026-08-27]` **An event report with no timestamp is a claim about "now" — and it manufactured a launch that never happened.** brokkr reconstructed a phantom third 3c launch because my 23:03 report narrated a 21:07 kill in the present tense. Every fact in it was true; it was unreadable in sequence. → `persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md`
|
||
_Archived 2026-09-11._
|
||
|
||
- `[2026-08-27]` **`save_steps` was hardcoded at 100 in the harness** — a claimed provenance entry the run could not have honoured. Made configurable, default unchanged (`0a6bd2e`, 242 tests green). Caught by checking the config carried the change rather than trusting that it had been made.
|
||
_Archived 2026-09-11._
|
||
|
||
- `[2026-08-27]` **Six defects in run 3's staged build, none of which would have errored** — a dialogue-only survivor list that would have silently dropped 96% of the corpus, an impersonation mask not subsumed by the low-quality mask, kvasir unbounded at 67.8% of context, a `save_pretrained` config-key drop that made the merged model unservable, and the mix-unit error. Every one produced a plausible completed run. Full record `/tank/erp-tune/recipe-r3/RUN-03-BUILD-NOTE.md`.
|
||
_Archived 2026-09-11._
|
||
|
||
- `[2026-08-27]` **The 18 unpushed eitri-smithy commits are pushed** — run 3's `harness_commit 9d27b4fe` now resolves off-box, verified by fetching into a fresh empty repo rather than trusting the push output. ⚠ **HTTPS push 403s for every gitea token including site-admin; SSH works.** Untracked `__pycache__` (`894fbe8`) because a tracked `.pyc` dirtied the tree and would have stamped `harness_dirty_at_launch: true`.
|
||
_Archived 2026-09-11._
|
||
|
||
## Tried and abandoned (archived)
|
||
# [2026-08-15] Uncensored gen seat: Qwen3.8-27B-Uncensored deployed; the definitive MTP-graft fix
|
||
|
||
**Outcome.** The fleet `gen` seat is now **`JonathanColetti/Qwen3.8-27B-Uncensored`** (Heretic
|
||
abliteration, KL 0.12 vs base, bench Δ −0.5 within noise, refusals 98→12/100), quantized in-house
|
||
to **NVFP4 W4A16** (llm-compressor / compressed-tensors) with a **grafted bf16 MTP head**,
|
||
vision-intact, **262K** ctx, MTP n=3 (**~42% accept, ~68 tok/s**), coherent. Live at ana-ml2 `:8015`
|
||
(project `gen-seat` / container `vllm-gen`), backing all 7 gateway aliases.
|
||
|
||
**THE definitive lesson (resolved 3 failed attempts + one premature 50 GB delete).** A grafted bf16
|
||
MTP scored **0% on the quant but 83% at bf16** — for TWO different abliterated models. Root cause was
|
||
NEITHER the abliteration NOR the quant scheme: it was **the grafted `mtp.*` tensors missing from
|
||
`config.json` → `quantization_config.ignore`.** The wrapper-class quant DROPS the MTP before
|
||
llm-compressor sees it, so nothing gets added to `ignore`; vLLM then tries to load the bf16 MTP as
|
||
*quantized* format → "Parameter … not found in params_dict, skip loading" → uninitialized head → 0%.
|
||
**FIX: after grafting, add `re:^mtp.*` to `quantization_config.ignore`** (one line — all unsloth's
|
||
working checkpoint has). MTP jumped 0%→83% (bf16-identical). Full lesson in auto-memory
|
||
`reference_abliteration_mtp_lessons`.
|
||
|
||
**The pipeline that works (for the next VL+MTP quant, incl. the W4A8 chase):**
|
||
1. Pull bf16 (kept at `ana-ml2:/tank/aimodels/qwen38-27b-uncensored-bf16`).
|
||
2. Quant via `quant_nvfp4_qwen.py` (darkscarlett dir) = the **wrapper-class** loader
|
||
(`Qwen3_5ForConditionalGeneration`, keeps the vLLM-serveable config); container = `vllm-openai`
|
||
+ `pip install llmcompressor==0.13.0` (drags in a transformers with `qwen3_5`).
|
||
3. **Graft** the author's `model-mtp.safetensors` verbatim into the output + merge the index.
|
||
4. **Reconstruct** `preprocessor_config.json` from `processor_config.json`'s `image_processor`
|
||
sub-dict (the repo omits it → else "Can't load image processor" crash-loop).
|
||
5. **Add `re:^mtp.*` to the output config's `quantization_config.ignore`.** ← the fix.
|
||
6. Serve: `--quantization compressed-tensors --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'`
|
||
`--mamba-cache-dtype float32 --kv-cache-dtype fp8 --reasoning-parser qwen3`.
|
||
|
||
**VRAM / full-context budget (measured).** Weights ~27 GB; hybrid attention → **only 16 of 64 layers
|
||
carry KV** → 32 KiB/token → **262K KV = 8.6 GB** (vs ~60–70 GB for a normal dense 27B). Full 262K fits
|
||
GPU0 at **util 0.45** (~43 GB) alongside meromero (~49 GB used, it's a 31B) — pre-flight rejects util
|
||
0.48 (wants 45.6 GB, only 45.5 free). `max-num-seqs 16` keeps cudagraph modest (an ad-hoc serve with
|
||
no cap OOM'd — cudagraph captured to batch-512).
|
||
|
||
**Why unsloth's `qwen3.8-27b` (the prior gen model) was faster (97 vs 68 tok/s).** ~half = quant kernel
|
||
(unsloth native NVFP4+FP8 tensor cores vs our W4A16 → Marlin dequant, ~20% even on decode — I'd
|
||
under-stated this); ~half = MTP acceptance (unsloth 55% un-ablated head vs our 42% — inherent to the
|
||
ablation, no quant fixes it). **W4A8 recovers the first ~20% (→~82 tok/s) + prefill; not the MTP half.**
|
||
|
||
**modelopt dead-end (for W4A8, avoid).** `nvidia-modelopt[hf]==0.43.0` is too old for qwen3_5's
|
||
transformers: (a) its `NVFP4_DEFAULT_CFG.quant_cfg` is a LIST but 0.43 wants a DICT (pydantic reject);
|
||
(b) it warns transformers 5.15 untested. Use **llm-compressor** for W4A8 instead (custom recipe: NVFP4
|
||
weights + FP8 input_quantizer + calibration on `heretic2-nvfp4-work/production_calib_512.jsonl`).
|
||
|
||
**Deleted (premature — the delete I owned).** `windowsxp811203/Qwen3.8-27B-Abliterated` (~79 GB) — I
|
||
declared it desync-dead off a 0% that was actually this ignore bug. Lesson: **test MTP on bf16 first;
|
||
isolate before deleting.**
|
||
|
||
Commits: eshpfi `680c30e` (deploy + rename + litellm + README), dotfiles `1d1970f` (CLAUDE.md roster) —
|
||
both UNPUSHED. Related: [[reference_abliteration_mtp_lessons]], [[reference_verify_hf_repo_ids_before_pull]].
|
||
_Archived 2026-09-02._
|
||
|
||
|
||
- `[2026-04-30]` task-board workflow with
|
||
`container: image: debian:bookworm-slim` — fails:
|
||
`actions/checkout@v4` needs `node` at runtime, slim image lacks it.
|
||
Switched to `node:20-bookworm-slim` (has node + apt) or runner-label
|
||
default. (Pattern revisited 2026-05-17 for skaldsong-dev: container
|
||
override needs `nodejs` apt-installed unless it IS the default.)
|
||
_Archived 2026-05-27._
|
||
|
||
- `[2026-04-30]` Dropping the `container:` directive before runner
|
||
re-registration with docker-schema labels — runner silently falls
|
||
back to **host mode** (jobs run inside the alpine `act_runner`
|
||
container itself, no apt). The `:host` suffix in startup logs
|
||
(`labels updated to: [pfi-fleet:host ana-docker:host]`) is the
|
||
giveaway. Fix: register with `pfi-fleet:docker://<image>` schema
|
||
labels.
|
||
_Archived 2026-05-27._
|
||
|
||
- `[2026-04-30]` Updating runner labels by editing `.env` and bouncing
|
||
— doesn't take. The `.runner` registration cache pins labels at
|
||
first registration; env-var updates are read each start but the
|
||
stored token + UUID are tied to the original label set on the gitea
|
||
side. Fix: stop runner, delete `.runner`, generate new admin
|
||
registration token, redeploy.
|
||
_Archived 2026-05-27._
|
||
|
||
- `[2026-04-30]` `git reset --hard origin/<sha>` in
|
||
`deploy-task-board.yaml` (and the in-repo nevermore playbook before
|
||
fix) — invalid syntax: `origin/` prefix only works for branch refs.
|
||
SHAs need `git reset --hard <sha>` directly. Resolved with
|
||
`git rev-parse --verify --quiet "origin/{{ ref }}^{commit}"` first,
|
||
then bare `"{{ ref }}^{commit}"` fallback.
|
||
_Archived 2026-05-27._
|
||
|
||
- `[2026-04-30]` Assuming `DEPLOY_SSH_KEY` was at user scope after
|
||
task-board wiring — it was actually only repo-scope on
|
||
`vh/task-board`. vor's first CI run failed with empty SSH key
|
||
(`printf '%s\n' "" > ~/.ssh/id_ed25519`). Fix: copy secret to user
|
||
scope at `gitea.phasefinal.com/user/settings/actions/secrets`.
|
||
_Archived 2026-05-27._
|
||
|
||
- `[2026-04-30]` `grep -vE "^(#|$)"` to inspect `.env` for sanity —
|
||
leaked the full `MINIFLUX_PASSWORD` line into the transcript. Then a
|
||
follow-up redaction attempt with `sed -E "s/=(.{4}).*$/=\1<redacted>/"`
|
||
still leaked the first 4 chars. Lesson: when probing secret-bearing
|
||
files, use field-by-field SELECTIVE inspection
|
||
(`grep -E "^(KEY1|KEY2)="`) rather than negative filters; for any
|
||
password line, `grep -c` (existence) or `test -n "$(...)"`
|
||
(non-empty), never `cat` or value-printing.
|
||
_Archived 2026-05-27._
|
||
|
||
- `[2026-05-08]` Filtering Traefik's UTC access log by Gitea-local-PDT
|
||
timestamp substrings (`grep "2026/05/08 15:1[2-7]"`) returned zero
|
||
matches and led to a wrong "no /v2/ traffic in 12 days" conclusion.
|
||
**Gitea logs in PDT, Traefik logs in UTC** — same host, different
|
||
timezones. Always normalize timezones (UTC) when correlating logs
|
||
across services on the same box. Cost: ~30 min in the wrong
|
||
direction.
|
||
_Archived 2026-05-27._
|
||
|
||
- `[2026-05-08]` Bumping Gitea `PER_WRITE_TIMEOUT` /
|
||
`PER_WRITE_PER_KB_TIMEOUT` to address `unexpected EOF` on
|
||
`/v2/.../blobs/uploads/` PATCH — wrong direction. Both govern
|
||
**response writes**, not request body reads. `unexpected EOF` from
|
||
Go's HTTP server means the client closed mid-body-upload; not a
|
||
knob Gitea exposes server-side.
|
||
_Archived 2026-05-27._
|
||
|
||
- `[2026-05-12]` Defaulting asset-engine to Traefik-routed
|
||
(`asset-engine.phasefinal.com` with `anaprod` cert resolver) on
|
||
first scaffold — user pulled it back to LAN-direct. Internal
|
||
tools default LAN-direct; only add Traefik when an external/TLS
|
||
surface is actually needed.
|
||
_Archived 2026-05-31._
|
||
|
||
- `[2026-05-12]` Routing althing thread replies through `galdrabok`
|
||
when the actual dev handle is `forseti` — bus rejected
|
||
`to=forseti` initially because thread participants list was
|
||
`[galdrabok, infra]`. Solved by starting a new thread with
|
||
`forseti` as the direct recipient. Lesson: when the bus
|
||
auto-resolves a sender handle that doesn't match the actual dev
|
||
role, start a fresh thread rather than fighting the participant list.
|
||
_Archived 2026-05-31._
|
||
|
||
- `[2026-05-13]` Initial Voxtral default voice `alloy` (OpenAI-compat
|
||
naming) — vLLM-Omni serving Voxtral does NOT translate aliases.
|
||
Native presets are `<register>_<gender>` shape (`neutral_female`,
|
||
`casual_male`, etc.). Always live-probe `/v1/audio/voices` for the
|
||
exact wrapper-deployed preset names before setting a catalog
|
||
default. Same caveat for Qwen3-TTS (wrapper exposes 15 voices: 9
|
||
Qwen presets + 6 OpenAI aliases) and Kyutai-TTS (NillPointer
|
||
wrapper has NO voice-listing endpoint at all; voices are
|
||
filesystem paths under the `kyutai/tts-voices` HF repo).
|
||
_Archived 2026-05-31._
|
||
|
||
- `[2026-05-17]` `--task classify` for Skywork in vLLM 0.19.1 —
|
||
flag was deprecated. Use `--runner pooling`; the model's
|
||
`architectures: [LlamaForSequenceClassification]` in config.json
|
||
drives the classification head. Surfaced as `vllm: error:
|
||
unrecognized arguments: --task classify` in container logs.
|
||
_Archived 2026-05-31._
|
||
|
||
- `[2026-05-17]` Trusting that `.env` edit alone propagates a new env
|
||
var into a worldtree container — `compose.yaml`'s `&worldtree-env`
|
||
anchor must explicitly declare the passthrough or the value silently
|
||
doesn't land. Same footgun bit `Z_AI_API_KEY` (2026-05-12) AND
|
||
`BIFROST_CLIENT_ALLOWED_HOSTS` (2026-05-17). Cost ~10 min of "why is
|
||
env empty?" diagnosis each time. Worldtree-side fix in
|
||
`vh/worldtree@08f02b2`.
|
||
_Archived 2026-05-31._
|
||
|
||
- `[2026-05-17]` `--force-recreate --pull never` from the docker:cli
|
||
sandbox without explicit `-e WORLDTREE_IMAGE=<sha>` re-pins the
|
||
container to `:latest`, even when a newer SHA-tagged image is on
|
||
disk. Symptom: container "recreated" but actually reverted to a
|
||
stale image. Pass `-e WORLDTREE_IMAGE=...:<sha>` to the docker run
|
||
invocation. Worldtree-dev's `8ef3801` health-gated `:latest`
|
||
advance is the long-term fix.
|
||
_Archived 2026-05-31._
|
||
|
||
- `[2026-05-18]` Volva env.sh.template `$HOME` in commented examples
|
||
— systemd's `EnvironmentFile=` parser doesn't expand `$HOME`;
|
||
uncommenting lands the literal `$HOME/...` string. Volva-dev's
|
||
`f4dda73` swapped to `/home/<svc-user>/...` placeholders.
|
||
_Archived 2026-05-31._
|
||
|
||
- `[2026-05-18]` Initial Volva systemd unit's `ProtectHome=read-only`
|
||
without `ReadWritePaths=` — althing-cli's SQLite (`~/.althing/
|
||
althing.db`) and codex's session state (`~/.codex/`) both need to
|
||
write. Container started but every poll failed with "db path not
|
||
writable". Surgical fix: `ReadWritePaths=/home/lkraven/.althing
|
||
/home/lkraven/.codex` (preserves the hardening intent, only carves
|
||
out the specific dirs).
|
||
_Archived 2026-05-31._
|
||
|
||
- `[2026-05-18]` Trusting that env.sh's `export VOLVA_ROOT="$(cd
|
||
"$(dirname "${BASH_SOURCE[0]}")" && pwd)"` template line works
|
||
under systemd — `EnvironmentFile=` parser aborts on the first
|
||
unparseable line (command substitution), and `VOLVA_ALTHING_CLI`
|
||
declared below silently never lands. Symptom: `Environment=`
|
||
property empty, daemon error "althing-cli not found at
|
||
'althing-cli'". Fix: replace command-substitution with literal
|
||
path. Volva-dev's `d436c3c` dropped VOLVA_ROOT entirely upstream.
|
||
_Archived 2026-05-31._
|
||
|
||
- `[2026-05-19]` Naive `docker rmi worldtree:<old-sha> --force`
|
||
for CD SHA cleanup — would untag pinned/personal worldtree images
|
||
since all three deployments share corviduo-dev. Use
|
||
`docker image prune -a --filter "until=Xh"` instead — respects
|
||
in-use semantic (Docker won't remove an image referenced by any
|
||
container on the host), so pinned/personal protected automatically.
|
||
_Archived 2026-05-31._
|
||
|
||
- `[2026-05-19]` Skaldsong CD first attempt: `docker pull` step
|
||
failed with 401 unauthorized. ana-docker had no `docker login`
|
||
for `gitea.phasefinal.com`. My playbook prereq note ("docker login
|
||
has been done at least once") was an unverified assumption. One-time
|
||
manual login persists in `~/.docker/config.json`; architectural fix
|
||
(workflow-side `ssh ana-docker 'docker login ...'` step using
|
||
`REGISTRY_USER`/`REGISTRY_TOKEN` secrets) flagged as v2.
|
||
_Archived 2026-05-31._
|
||
|
||
- `[2026-05-19]` `SKALDSONG_HOST_CORS_ORIGINS=http://10.250.50.70:8300`
|
||
as a bare URL — pydantic-settings parses complex env vars via
|
||
`json.loads()`; first-boot crashloop with `SettingsError: error
|
||
parsing value for field "cors_origins"`. Must be JSON array literal:
|
||
`SKALDSONG_HOST_CORS_ORIGINS=["http://..."]`.
|
||
_Archived 2026-05-31._
|
||
|
||
- `[2026-05-19]` `SKALDSONG_HOST_STATIC_ASSETS_PATH=/app/web/dist`
|
||
in compose — mismatched Dockerfile reality. The Dockerfile COPYs
|
||
SvelteKit build output flat into `/app/spa` (not `/app/spa/dist`).
|
||
Lifted the path from skaldsong-dev's CD-ask message ("`/app/web/dist`")
|
||
rather than verifying against the actual Dockerfile they shipped.
|
||
Lesson: when encoding container-internal paths in compose, verify
|
||
against the Dockerfile, not the design-doc.
|
||
_Archived 2026-05-31._
|
||
|
||
- `[2026-05-19]` Playbook verify step `docker ps | grep healthy`
|
||
racing the container's `start_period` (30s in compose's
|
||
healthcheck). Verify ran 0.09s after `compose up -d --force-recreate`
|
||
— well before docker's healthcheck could flip the status from
|
||
`(health: starting)` to `(healthy)`. False-negative; container was
|
||
operationally up (the earlier `/health` poll verify already
|
||
confirmed). Fix: grep `^Up` not `healthy`. /health-200 IS the
|
||
liveness check; docker's `(healthy)` is just a delayed echo.
|
||
_Archived 2026-05-31._
|
||
|
||
- `[2026-05-20]` `SKALDSONG_DB_PATH` + `SKALDSONG_RUNS_DIR` in
|
||
compose env block — names skaldsong's app doesn't read. App reads
|
||
`SKALDSONG_HOST_SQLITE_PATH` + `SKALDSONG_HOST_RUNS_ROOT` (per
|
||
Dockerfile ENV defaults). Wrong names = silently no-op; app fell
|
||
back to Dockerfile defaults pointing at `/app/data/...` which the
|
||
compose's bind mount did NOT cover (target was `/app/state/...`).
|
||
Result: every `--force-recreate` wiped the SQLite DB. Caught by
|
||
skaldsong-dev (althing thread `01KS4DPF6SXTBP4Q360JZVWPNT`). Fix in
|
||
`52e98fa`. Lesson: verify env var NAMES against the Dockerfile/app,
|
||
not against design-doc shorthand.
|
||
_Archived 2026-05-31._
|
||
|
||
- `[2026-05-25]` First selene-block patch put the block under
|
||
`saga_allowed_models:` instead of top-level `models:` — used
|
||
`text.replace("models:\n", ...)` which substring-matched the
|
||
`saga_allowed_models:\n` line first. Caused YAML parse error. Fix:
|
||
anchored regex `re.compile(r"^models:\n", re.MULTILINE)`. Pattern:
|
||
substring replace on YAML top-level keys WILL match suffix-containing
|
||
keys.
|
||
_Archived 2026-05-31._
|
||
|
||
- `[2026-05-27]` `docker compose up -d` inside the `docker:cli` sandbox:
|
||
`${VAR:-./config}` defaults resolve `./config` to the sandbox CWD, but the
|
||
Docker daemon interprets the path against the HOST filesystem → auto-creates
|
||
an empty dir → entrypoint reseeded image-baked defaults (lost host-side
|
||
providers.yaml patches). Fix: pass `-e WORLDTREE_CONFIG_DIR=/abs/path`.
|
||
Folded into the docker-as-root convention note.
|
||
_Archived 2026-06-01._
|
||
|
||
- `[2026-05-27]` `:latest`-pinned compose + private gitea registry + sandboxed
|
||
pull = recreate on ancient cached `:latest` (deploy pulls by SHA so the tag
|
||
never advances; sandbox can't pull). Fix: retag SHA→`:latest` on host, then
|
||
`--pull never`. Better: pin SHA in `.env`, advance in CI.
|
||
_Archived 2026-06-01._
|
||
|
||
- `[2026-05-27]` Container recreate during in-flight skaldsong gen kills the
|
||
runner. With deploys every ~10min and stories >5min, structural not
|
||
incidental. Roadmap (skaldsong-dev): pre-shutdown signal handler, per-scene
|
||
resume-from-checkpoint, /api/admin/quiesce. None shipped.
|
||
_Archived 2026-06-01._
|
||
|
||
- `[2026-05-27]` `--reasoning-format gemma` on artemis-31b-v1i — unsupported in
|
||
the deployed llama.cpp (accepts none|deepseek|deepseek-legacy). `deepseek`
|
||
populates the `reasoning_content` SSE delta Worldtree GemmaProvider checks.
|
||
_Archived 2026-06-01._
|
||
|
||
- `[2026-05-27]` `head -c N` piped after a streaming curl SIGPIPEs the curl,
|
||
killing the request early. Use file-write + separate read.
|
||
_Archived 2026-06-01._
|
||
|
||
- `[2026-05-31]` Building the dia2-capable image surfaced THREE upstream
|
||
packaging quirks: (1) `pip install -e nari-labs/dia2` fails — no PEP 660
|
||
`build_editable` hook; (2) plain `pip install` builds an empty
|
||
`UNKNOWN-0.0.0` wheel (base setuptools 59.6 < dia2's required ≥70); (3)
|
||
`--no-deps` leaves `transformers`/`sphn`/`whisper-timestamped` missing. Fix
|
||
(`local/dia:v2`): copy the pure-python `dia2/` package into site-packages +
|
||
install ONLY those 3 deps; base torch/numpy already satisfy Dia2.
|
||
_Archived 2026-06-03._
|
||
|
||
- `[2026-05-31]` Dia2 predefined voices (43, baked at `/app/voices`) are NOT
|
||
reachable from the `/tts` clone path — it resolves `clone_reference_filename`
|
||
against the reference_audio dir ONLY. The OpenAI `/v1/audio/speech` `voice`
|
||
param auto-resolves them (separate code path), which masked the gap. Fix:
|
||
stage `/app/voices/*` into `/worktank/dia/reference_audio`. Lesson: verify on
|
||
the endpoint the catalog ACTUALLY targets.
|
||
_Archived 2026-06-03._
|
||
|
||
- `[2026-05-31]` `voice_mode=clone` with an empty `clone_reference_filename`
|
||
→ asset-engine serializes it as the literal string `"undefined"` → `/tts`
|
||
404. First observed on dia2; worked around in the catalog (default the field
|
||
to a real voice). **[2026-06-01] root cause found** — the Kokoro voice-blend
|
||
widget reading Shoelace `.value` before hydration (see Current state); the
|
||
real fix is asset-engine-side and is escalated.
|
||
_Archived 2026-06-03._
|
||
|
||
- `[2026-05-31]` asset-engine `services.schema.json` is DERIVED (regen from the
|
||
Pydantic model via `dump_schema.py`) and had DRIFTED — rejected the
|
||
`lifecycle` field 12/14 services use. RESOLVED: regenerated with
|
||
`CatalogLifecycle` + `reproducibility.seed_field` (`b7b2130`). Lesson:
|
||
hand-editing `services.yaml` shape without regenerating re-introduces drift.
|
||
_Archived 2026-06-03._
|
||
|
||
- `[2026-05-31]` ttyd-over-TLS forces HTTP/2 (kills ttyd's terminal WebSocket
|
||
→ blank screen); Safari/WebKit never sends HTTP basic-auth on WS upgrades.
|
||
Both solved for the fleet seat (Caddy forces HTTP/1.1; auth → network-gating)
|
||
— detail in auto-memory `reference_ttyd_fleet_seat`.
|
||
_Archived 2026-06-03._
|
||
|
||
- `[2026-05-30]` esh-docker-vm: `hard` NFS mounts from 10.0.50.50 froze a
|
||
container worker in UNKILLABLE D-state when the NAS stalled — only a host
|
||
reboot clears it. Separately, `fstab defaults` (no `_netdev`) made NFS-bind
|
||
containers `Exited(255)` on reboot. → auto-memory
|
||
`incident_esh_docker_nfs_boot_race`.
|
||
_Archived 2026-06-03._
|
||
|
||
- `[2026-06-02]` **Naive cumulative re-decode for streaming** — decode the
|
||
growing token prefix each chunk + emit the delta, assuming the causal flow
|
||
gives a stable prefix. It does NOT (full-context attention, see above);
|
||
maxdiff 0.30. Also the high-level `s3gen.inference` re-applies `trim_fade` at
|
||
the start every call. Don't go this way.
|
||
_Archived 2026-06-04._
|
||
|
||
- `[2026-06-01]` **CSM bring-up** — upstream `phildougherty/sesame_csm_openai`
|
||
Dockerfile pins NO `huggingface_hub` version, so it now resolves to 1.17.0
|
||
where `huggingface-cli` is fully removed (replaced by `hf`) → the build dies
|
||
at `huggingface-cli login`/`download`. Passing the HF token as a build-arg
|
||
got the token in (the model downloads at BUILD time, not runtime — our
|
||
compose only wired it as runtime env, the original failure) but then hit the
|
||
rotted CLI. Fixing would need vendoring + patching the Dockerfile; not worth
|
||
it for a weak model → CSM removed.
|
||
_Archived 2026-06-04._
|
||
|
||
- `[2026-06-01]` **Fish path-form references** (`{audio:"/app/references/x.wav"}`)
|
||
→ HTTP 500 on this build, despite the stack README showing that shape. The
|
||
working clone paths are `reference_id` (by name → resolves the DIRECTORY
|
||
`references/<name>/` containing `<name>.wav` + `<name>.lab`; the flat
|
||
`references/<name>.wav` is ignored — see the two-root-causes entry above) or
|
||
inline base64 `references`. The catalog uses `reference_id`.
|
||
_Archived 2026-06-04._
|
||
|
||
- `[2026-06-01]` **Reference transcript barely affects Fish clone IDENTITY**
|
||
(disproving my mid-session theory). I'd blamed a bogus provenance-note `.txt`
|
||
for poor cloning, but the ECAPA re-test showed correct (0.787) / bogus (0.778)
|
||
/ empty (0.738) transcripts all clone Imogen about equally — the transcript
|
||
affects PRONUNCIATION (phoneme disambiguation per the docs), not who it sounds
|
||
like. The real culprit for "not British" was the `"undefined"` select bug, not
|
||
the transcript. (A correct transcript still marginally helps pronunciation —
|
||
cheap to stage, not load-bearing.)
|
||
_Archived 2026-06-04._
|
||
|
||
- `[2026-06-01]` **Pitch-shift register control** (rubberband, to deepen Imogen
|
||
to contralto/mezzo) — Fish ignores small reference shifts and overshoots
|
||
large ones; `-3st` WITHOUT formant preservation → "baritone through a tuba"
|
||
(formants collapsed); `-2/-3st` WITH `--formant` → too subtle / still bad.
|
||
Abandoned at every depth; all variants deleted. Finer independent
|
||
pitch/formant control needs praat (not installed). Future clones = operator's.
|
||
_Archived 2026-06-04._
|
||
|
||
- `[2026-06-01]` **resemblyzer is too weak for cloning-fidelity scoring** — its
|
||
dated 2019 LSTM encoder + a synthetic-vs-natural domain gap scored the Imogen
|
||
clone CLOSER to the default than to real-Imogen, which led me to a WRONG "Fish
|
||
is a weak cloner" call. ECAPA-TDNN (speechbrain `spkrec-ecapa-voxceleb`) on the
|
||
same clips gave the correct answer (clone 0.79 to real Imogen, 0.10 to
|
||
default). Use ECAPA, not resemblyzer, for speaker-verification.
|
||
_Archived 2026-06-04._
|
||
|
||
- `[2026-06-02]` **Fish (fish-s2 / OpenAudio S1-mini) progressive streaming — SHELVED
|
||
(sub-realtime).** Benched RTF on A6000: 0.72x/0.82x/0.86x, mean 0.80x = sub-realtime,
|
||
so client-side chunking starves. Root cause: Fish only chunks on `<|speaker:X|>` tags;
|
||
plain text → batches=[whole text] → all semantic tokens generate before any audio
|
||
(chunk_length inert). Plus kui/ASGI StreamResponse doesn't flush (fix = anti-buffering
|
||
headers, kept on file). STANDING REVISIT TRIGGER: when an RTX Blackwell Pro lands → bench
|
||
fp4-quantized Fish; if RTF > ~1.5x give it the chatterbox-fast treatment. For now Fish is
|
||
a buffered catalog entry (great for SAVED gens, not live-audition).
|
||
_Archived 2026-06-11._
|
||
|
||
- `[2026-06-02]` **Context-priming at chunk joins (chatterbox-fast §1.6) — ABANDONED
|
||
(discard-cut leaks the prefix).** Prepend prior sentence, generate `prefix+content`,
|
||
discard prefix audio. Built + shipped (`d707439`), live-A/B'd, reverted (`090e70a`).
|
||
Kill: `generate()` returns one waveform with NO marker where the prefix ends, and the
|
||
model renders the same prefix with different timing solo vs followed-by-content — so
|
||
locating the cut is a guess that left a whole clause in the output (operator caught
|
||
"...without a trace of sarcasm" spoken twice). Reliable cut needs token-level boundaries
|
||
or per-chunk forced-alignment (heavy). Coherence loss at joins = ACCEPTED limitation.
|
||
_Archived 2026-06-11._
|
||
|
||
- `[2026-06-02]` **Native frame-level streaming on Chatterbox-TURBO — ABANDONED (turbo
|
||
isn't built for streaming).** (1) `S3GenStreamer` is docstring-only (unimplemented).
|
||
(2) `flow_inference(finalize=False)` is buggy (lookahead trim removes frames from `h`
|
||
not `h_masks`/conds → shape mismatch); a 1-line patch makes it RUN. (3) BUT the flow
|
||
encoder uses full-context attention (`static_chunk_size=0`) → incremental decode is
|
||
prefix-unstable (adding tokens re-attends, shifts earlier mel ~0.30-0.39 vs one-shot).
|
||
(4) Forcing `static_chunk_size>0` didn't stabilize it. True sub-second frame-streaming
|
||
needs deep attention surgery with quality risk — not worth it. → adaptive-chunking instead.
|
||
_Archived 2026-06-11._
|
||
|
||
- `[2026-06-03]` **`uv pip install .` fails on SmoothKen/knn-svc** (and similar script-repos)
|
||
— it's analysis scripts + a poetry pyproject, no buildable package (setuptools
|
||
package-discovery error). Install the pyproject deps directly, don't build the "package".
|
||
_Archived 2026-06-13._
|
||
|
||
- `[2026-06-03]` **Backgrounding `althing-cli monitor` with an inline shell `&` (instead of
|
||
the Bash-tool `run_in_background`) orphans it** — it survives the shell exit, holds the
|
||
per-handle flock UNTRACKED (won't notify the session), and `stop-monitor` doesn't detect it.
|
||
Fix: find + kill the orphan PID (verify cwd=this repo / handle first — nh3-dev is shared, other
|
||
agents' monitors run there too), then re-arm via run_in_background. Always re-arm tracked.
|
||
_Archived 2026-06-13._
|
||
|
||
- `[2026-06-03]` **torch-2.12 venvs need `uv pip install torchcodec`** — torchaudio 2.12
|
||
defaults to the TorchCodec backend for `.load`; without it, real audio I/O throws "TorchCodec
|
||
is required" — and it ONLY surfaces at actual conversion, NOT at import/model-load. Lesson:
|
||
validate real I/O, not just import, when provisioning ML engine envs. (seed-vc on torch 2.4
|
||
uses the legacy backend, exempt.)
|
||
_Archived 2026-06-13._
|
||
|
||
- `[2026-06-03]` **gitea webhook to a private IP is denied by `webhook.ALLOWED_HOST_LIST`**
|
||
(anti-SSRF; default `external` blocks private/loopback). Symptom: delivery shows
|
||
`dial tcp ...: webhook can only call allowed HTTP servers`. Fix = APPEND the target net to
|
||
ALLOWED_HOST_LIST in gitea's app.ini (keep `external`; scope tight, never `*`/`private`) +
|
||
restart gitea (act_runner job containers survive a restart). gitea runs as a container on
|
||
ana-docker (`gitea_gitea_data` volume, `/data/gitea/conf/app.ini`).
|
||
_Archived 2026-06-13._
|
||
|
||
- `[2026-06-04]` **homepage "crash" ≠ always NFS** — a wedged container in unkillable D-state can come from dead `siteMonitor` widget targets (retired ESH firewall IPs) hanging the node event loop into `exit_mmap`, needing a host reboot. Check homepage's siteMonitors against retired hosts. (`incident_esh_docker_nfs_boot_race`)
|
||
_Archived 2026-06-14._
|
||
|
||
- `[2026-06-04]` **pyworld needs `setuptools<81`** (imports the removed `pkg_resources`); and **R/soundgen `-lgfortran` fails** on irv-ml1 (default gcc is gcc-11 but only gfortran-12 present) → install `libgfortran-11-dev`.
|
||
_Archived 2026-06-14._
|
||
|
||
- `[2026-06-04]` **Persistent services on irv-ml1 need SYSTEM systemd units** — the box reaps user-session processes on ssh disconnect, and `--user` systemd isn't reachable over non-login ssh, so nohup/setsid/`screen -dmS`/`systemd-run --user` all die (even with enable-linger). Use `/etc/systemd/system/`.
|
||
_Archived 2026-06-14._
|
||
|
||
- `[2026-06-04]` **GPU pin by INDEX is ambiguous on irv-ml1** — native CUDA orders fastest-first (A6000=0) but nvidia-smi/docker use PCI order (A6000=1). Pin by **UUID** (`CUDA_VISIBLE_DEVICES=GPU-…`); verify via nvidia-smi compute-apps.
|
||
_Archived 2026-06-14._
|
||
|
||
- `[2026-06-04]` **Ollama/llama.cpp-BUNDLED chat templates silently diverge from canonical HF — the "headgun" lesson.** Serve each model's canonical `tokenizer.apply_chat_template`, not the bundled template (bundled ones corrupt baselines). Verify via vLLM `/tokenize`→`/detokenize`. (`90e08f0`/`27eb537`)
|
||
_Archived 2026-06-14._
|
||
|
||
- `[2026-06-08]` **Demucs `uv pip install demucs` pulls torch 2.12/torchaudio 2.11 → `ta.save()` requires torchcodec → dies AFTER separating** (0 stems, rc=1). Fix = pin `torch==torchaudio==2.4.1` + `UV_LINK_MODE=copy`. Validate the SAVE path, not just import + GPU inference, on a bleeding-edge torch.
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-05]` **vLLM 0.19 CUDA-graph-capture OOMs on a SHARED GPU** — fills KV to the `--gpu-memory-utilization` budget WITHOUT reserving graph-capture memory, so `capture_model` OOMs AFTER weights+KV load (crash-loops). Fix: free co-tenant room OR `--enforce-eager`. FP8 single-stream is batch-1 GEMV (memory-bound) → Q4 wins single-stream by physics; FP8 wins under concurrency. (`reference_ana_ml2_vllm_granite`)
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-05]` **Langfuse has NO public dashboard-creation API** — dashboards/widgets are postgres rows; clone a default + swap the measure. tok/s is the `outputTokensPerSecond` MEASURE (metrics-API/dashboard query time), not a per-generation field. langfuse-web needs `HOSTNAME=0.0.0.0`. Host 3000 is gitea's → langfuse on 3001.
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-05]` **`sudo` over non-interactive ssh FAILS SILENTLY where the user lacks NOPASSWD** (esh + corviduo are OUTSIDE the infra-ops identity) → empty output misread as "empty file." Read world-readable files WITHOUT sudo. corviduo ssh = `vh@10.250.50.152`; bind-mount configs are vh-owned, the `.env` is deploy-owned 600.
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-05]` **Worldtree summarizer-model is NOT an env var** — no `WORLDTREE_SUMMARIZER_MODEL`; defaults to claude-haiku in code, opt-in via config not `.env`. Inspect the live container env + vh-owned config files first.
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-13]` **Loading an old LoRA adapter after a vocab bump fails on embedding size.** The harness config + `tokenizer.json` are now at `new_vocab_size=50279` (v4 multi-tag); the v1/v2/v3 adapters are 50277. To load v1 (the accepted artifact), set `cfg.new_vocab_size=50277` before `load_finetuned_engine_lora` (else PeftModel state_dict size mismatch). `tokenizer.json.v3bak` is the 50277 tokenizer for a clean restore.
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-13]` **Heavy from-source compile (`MAX_JOBS=128`) on the shared PROD GPU box PINS it** — load hit 187, prod vLLM restarted, killed an in-flight quant. ana-ml2 hosts live inference; never run a big build there at full parallelism. Cap `MAX_JOBS≤32`, build off-box, or cgroup-constrain.
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-13]` **`--quantization fp8` on a VL model can quantize the VISION TOWER → garbage vision** (Qwen3.5-VL on stable vLLM: gray-grid output; LM answers text fine, so it "looks" healthy). The nightly excludes the vision tower. Validate the VISION path on a quantized VLM, not just text — and pin the engine digest with the exclusion.
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-13]` **vLLM's `--gpu-memory-utilization` is checked against FREE VRAM at startup, not total** — on a shared card, growing one service before trimming a co-tenant OOMs. Trim the shrinking service FIRST, then grow. Size to the FREE budget.
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-13]` **The `vllm/vllm-openai` entrypoint is already `["vllm","serve"]`** — compose `command:` supplies the model as the first POSITIONAL arg + flags; a second `serve`/`--model X` → "unrecognized arguments". Same-class: `tee` masks the real exit code (use `>`); HF `datasets` rejects bare `wikitext` (needs `Salesforce/wikitext`).
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-13]` **Chatterbox-Turbo LoRA finetune: the repo's `setup.py` loads the WRONG tokenizer** — pulls gpt2-medium + a grapheme merge file instead of the chatterbox-turbo GPT2 tokenizer (vocab.json+merges.txt, len 50276). Fix = override + delete the grapheme `tokenizer.json`; `[vmoan]` → new_vocab_size 50277 (1-row resize), lora_r 64 / alpha 128, modules_to_save=[text_emb,text_head]. Unique-stem corpus collision needs `{index}_{stem}` IDs. (`irv-ml1:~/r16-vmoan-harness`)
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-11]` **A completion-poll `while pgrep -f <scriptname>` SELF-MATCHES its own remote shell argv** — its own `pgrep -f` always finds itself → the loop never exits. Use a match pattern ABSENT from the poll command (the python stage, or a sentinel file), not the driver's own name.
|
||
_Archived 2026-06-16._
|
||
|
||
- `[2026-06-14]` **vLLM ModelOpt-NVFP4-MoE loader is broken for current multimodal MoEs.** `nvidia/Qwen3.6-35B-A3B-NVFP4` fails weight-load: `KeyError: layers.0.mlp.experts.w2_input_scale` on 0.19.1, `lm_head.input_scale not registered` on 0.22.0 (vllm #44081); same class hits Gemma-4 MoE / Qwen3-30B-A3B NVFP4. The arch + quant ARE recognized (gets past arch resolution + vision-processor load) — it's the per-expert/lm_head scale-key mapping. Don't chase nightlies; use official FP8 until fixed.
|
||
_Archived 2026-06-18._
|
||
|
||
- `[2026-06-14]` **vLLM sampler-warmup OOMs on a shared GPU even when weights fit** — it warms the sampler with `max_num_seqs` (default **1024**) dummy requests, and a big vocab (Qwen3.6 = 248K) makes that a huge transient logits tensor. A vision endpoint doesn't need 1024-way concurrency: set `--max-num-seqs 32`. Separately, post-load `ValueError: No available memory for the cache blocks` means util is too thin (weights+activation+graph ate it) — for 34 GB FP8 weights, util ≥ ~0.45 to leave KV room.
|
||
_Archived 2026-06-18._
|
||
|
||
- `[2026-06-14]` **Recreating multiple vLLM services concurrently races the memory-profiling assertion** — `AssertionError: Error in memory profiling. Initial free memory X / current Y … other processes … release GPU memory while vLLM is profiling`. Recreate co-tenant vLLM services ONE AT A TIME (force-recreate one, wait healthy, next).
|
||
_Archived 2026-06-18._
|
||
|
||
- `[2026-06-14]` **embed/rerank (0.6B) at util 0.05 reserve ~5.5 GB each — mostly util-reservation WASTE, not need.** A 0.6B model needs ~1.2 GB weights + ~2.5 GB CUDA/torch context; util 0.03 (~3.6 GB) fits with room, reclaiming ~4 GB (vLLM reserves the util fraction regardless of actual KV; embedding models barely use KV). Real-need floor ~3 GB — don't go to 0.02.
|
||
_Archived 2026-06-18._
|
||
|
||
- `[2026-06-14]` **Chatterbox-Turbo decode-knob foot-guns** (R16 v1-polish + emergent probes): the turbo length cap is `max_gen_len` (default 1000) on `t3.inference_turbo`, NOT `max_new_tokens` — and `tts_turbo.generate` does NOT forward it (wrap inference_turbo to cap). `rep_pen 2.0 / temp 0.5` BACKFIRES (degenerate 24 s run-on). Soft decode tamers cut the NVV ITSELF, not just the run-on tail (operator: "p1 trims the moaning too") — gen-time polish can't beat v1's defaults. Inline base-NVV tokens DEGENERATE (moan-cascade + gibberish), they don't modulate the surrounding words.
|
||
_Archived 2026-06-18._
|
||
|
||
- `[2026-06-15]` **A comfyui restart-loop with `exit 0` + `OOMKilled=false` is NOT OOM — suspect a native SEGFAULT.** `docker inspect` shows only the LAST exit code; a GPU OOM is a catchable `RuntimeError: CUDA out of memory`, NEVER `Fatal Python error: Segmentation fault`. Diagnostic: `RestartCount>0` = same container restarted in-place → the docker json-log ACCUMULATES across all cycles, so grep the full `docker logs` for the crash signature + the lines right before each restart banner. comfyui's was at VAE decode (sampling completes, then segfault loading `AutoencodingEngine`) — gen-triggered but native, not VRAM. Suspects logged: torch/cu130 mismatch, an injected `aimdo` cuda-funchooks lib, the VAE.
|
||
_Archived 2026-06-19._
|
||
|
||
- `[2026-06-15]` **NEVER blanket-kill `althing-light-monitor` by PID** — they're per-handle and MANY agents run their own on the shared nh3-dev box (heid / ratatoskr-dev / worldtree-dev / bifrost-dev each have one). A `kill $(pgrep -f light-monitor)` downs four peers' monitors (near-miss caught by checking `/proc/<pid>/environ` `ALTHING_HANDLE` + cwd first). Recover your OWN via `althing-cli stop-monitor` (v0.14.1+); a raw `&`-backgrounded monitor is untracked (no harness fire-notification) — always arm via `run_in_background`. (extends the pkill-self-match lesson below)
|
||
_Archived 2026-06-19._
|
||
|
||
- `[2026-06-15]` **Manual `docker compose up` on corviduo Worldtree DOWNGRADES to a stale local `:latest` → crash-loop (config/image mismatch).** Compose `image:` is `…/worldtree:latest`, but local `:latest` lagged ~2wk (`21c90d5eeb2d`); CI/CD deploys NEWER SHA images, and a real pull 401s (daemon has no gitea-registry creds — that's CI/CD's). `--pull never` then used the stale latest → personal crash-looped on `ConfigurationError: echo.default_model missing` → **~2-3 min personal outage**. RECOVER: retag `:latest` to the image a HEALTHY SIBLING runs (`docker inspect worldtree-worldtree-api-1 --format '{{.Image}}'` → `docker tag …`), then `up -d --pull never`; verify `Application startup complete`. Don't manually recreate Worldtree on corviduo without pinning the correct image — image+config are CI/CD-coupled. (auto-memory `reference_corviduo_dev_emergency_ops`)
|
||
_Archived 2026-06-19._
|
||
|
||
- `[2026-06-15]` **`.claude.json` (Claude Code config) truncates on ENOSPC → invalid JSON → CC resets to defaults on next start.** nh3-dev hit 100% disk (a peer's R19 dataset pull staged ~147G to local /home instead of the 7TB NAS); the in-flight `.claude.json` write was cut mid-key. Repair = strip the dangling token + close the open braces, validate `json.load`. CC keeps its OWN rolling backups at `~/.claude/backups/` + writes a `.claude.json.corrupted.*` snapshot when it self-detects. After ANY ENOSPC on a box running CC/agents: integrity-check `.claude.json` (json.load) AND the althing bus DB (`PRAGMA integrity_check`).
|
||
_Archived 2026-06-19._
|
||
|
||
- `[2026-06-15]` **`pkill -f althing-light-monitor` SELF-MATCHES the killing shell** (the pattern is in the command's own argv) → kills itself mid-run (exit 144/truncated output). Stop the light-monitor via `althing-cli stop-monitor` or a captured PID — never `pkill -f <string-present-in-this-command>`. The singleton lock can also RACE to 2 live monitors during re-arm churn; keep exactly one tracked (run_in_background) monitor, and a raw `&` monitor is untracked (no harness fire-notification — don't use it).
|
||
_Archived 2026-06-19._
|
||
|
||
- `[2026-06-14]` **Fleet/colo hosts must reach gitea over the INTERNAL route, NOT the public IP.** `gitea.phasefinal.com` = public `38.120.12.44` (ana-srv1); gitea is a container on ana-docker, git-SSH `10.250.50.70:222` + HTTP `:3000`. A fleet host egressing to public `:22` gets fail2ban-banned after any retrying git loop → silently wedges webhook auto-deploys (`git fetch` times out under `set -euo pipefail`, aborts before reset). Bit irv-ml1's arbo deploy. `:22` on `10.250.50.70` is ana-docker's HOST sshd (deploy key → Permission denied), NOT gitea. Documented `docs/orientation.md` (`6e58e57`).
|
||
_Archived 2026-06-19._
|
||
|
||
- `[2026-06-16]` **litellm 500 `Router.acompletion()/aembedding() missing 'messages'/'input'` = a request missing `Content-Type: application/json`, NOT a gateway outage.** curl `-d` defaults to form-encoding → litellm can't parse the JSON body → `data` reaches the router without `messages`/`input` → 500 (should be a 400; litellm #16993). My own diagnostic calls dropped the header → I misread it as a gateway outage and needlessly bounced the gateway ~4× chasing a phantom (image/version/config were fine throughout; a malformed UI-added "Mistral Story Eval" model in the DB was a red herring I deleted). ALWAYS send `-H "Content-Type: application/json"` testing litellm; reproduce with a header'd call before declaring a litellm incident.
|
||
_Archived 2026-06-20._
|
||
|
||
- `[2026-06-16]` **LiteLLM admin-UI playground can't test vLLM-backed models** — it auto-sends empty `tools:[]`, vLLM 400s (litellm #6228); the gateway `strip_empty_tools` hook is a PROXY hook and structurally can't reach the UI's in-process `litellm.completion()` call. Off-ramp = `tools/gateway-chat.html`. (Langfuse playground also out: its SSRF guard blocks internal-IP LLM connections, wontfix Langfuse #13097.) (auto-memory `reference_litellm_ui_playground_vllm_deadend`)
|
||
_Archived 2026-06-20._
|
||
|
||
- `[2026-06-16]` **A gitea Actions run that "looks like it never fired" (no image, untouched build dir) usually DID fire then skipped or failed-fast.** Bit arbo TWICE: run #3 looked-never-fired but SKIPPED (change-detection `engine=0`); run #4 looked-never-fired but FAILED in ~7s (early-step). Check the run LIST + status via claude-bot (`GET /repos/{o}/{r}/actions/tasks`) BEFORE a runner-connection/dispatch theory. CORRECTION (2026-06-16, proven): action logs ARE readable via API at the per-JOB endpoint — `GET /api/v1/repos/{o}/{r}/actions/runs/{run}/jobs` → take a job `.id` → `GET /api/v1/repos/{o}/{r}/actions/jobs/{job_id}/logs` returns the full plain-text log (claude-bot basic-auth, internal route `10.250.50.70:3000`). Only the RUN-level `/runs/{id}/logs` 404s; the per-job one works, so no UI needed. Gotcha when reading: gitea marks **every** step `conclusion:failure` once any step fails — the per-step conclusions mislead; trust the log body + step *timestamps* (the step with real duration is the culprit) not the conclusions. (Used 2026-06-16 to diagnose arbo run #5/task 1175 = test-gate `uv run pytest` → `Failed to spawn: pytest` = pytest absent from the fresh-checkout venv sync, NOT the SSH-clone comfy-dev suspected.)
|
||
_Archived 2026-06-20._
|
||
|
||
- `[2026-06-16]` **vLLM 0.23.0 breaks Mistral MULTIMODAL at startup** (`MistralCommonImageProcessor
|
||
has no attribute fetch_images`, #44911 regression ~0.22.1+; same class hit Mistral-3.1/
|
||
Devstral). The dummy-image profiling crashes. Workarounds: `--limit-mm-per-prompt
|
||
'{"image":0,"video":0}'` loads text-only; for VISION, pin **v0.22.0** (last pre-regression).
|
||
Tradeoff: v0.22.0 *buffers* reasoning (doesn't stream reasoning_content cleanly); a newer
|
||
build streams it but breaks vision — vision won (Mistral is the abliteration-baseline).
|
||
_Archived 2026-06-20._
|
||
|
||
- `[2026-06-16]` **Mistral `-reasoning` TTFT variance = Triton kernel JIT-compile-during-
|
||
inference, NOT reasoning time.** `jit_monitor` logs the spike (`_compute_slot_mapping_kernel`,
|
||
`apply_token_bitmask_inplace_kernel` ← the tool-call/structured-output path). vLLM's startup
|
||
warmup covers only SOME shapes; a first-hit uncovered shape JITs a kernel (seconds), then
|
||
CACHES it — warm until container recreate (the Triton cache is IN-container, no volume, so
|
||
recreate = cold again). Secondary: GPU idle-down to P8/180 MHz between requests. Also:
|
||
Mistral `reasoning_effort` is BINARY (none|high only — medium/low → HTTP 400).
|
||
_Archived 2026-06-20._
|
||
|
||
- `[2026-06-16]` **Worldtree old-baseline instances need the FULL config set synced, not the
|
||
changeset delta.** Personal (a PRE-mistral baseline) crash-looped (41 restarts) on
|
||
`DanglingBindingError: role 'echo' binds missing catalog entry 'qwen3.6-35-a3b-heretic'` —
|
||
its `model_roles.yaml` (2026-06-10, never migrated) still bound the retired heretic that the
|
||
synced v0.35.10 providers removed. The "model_roles unchanged — skip" was correct only for
|
||
demo (already at the v0.35.9 baseline). Old-baseline → sync providers + model_roles + defaults.
|
||
_Archived 2026-06-20._
|
||
|
||
- `[2026-06-16]` **vh is a Gitea USER, not an org → no scoped package-write for a service
|
||
account.** Gitea user-namespace packages are owner/admin-only, so claude-bot can't push to
|
||
`vh/arbo` without being admin. The fix is converting vh → a `corviduo` ORG (tabled). Also:
|
||
vh's `tea` token lacks `write:admin` — create gitea users via the server-side `gitea admin
|
||
user create` CLI (`docker exec -u git gitea …`), not the admin API.
|
||
_Archived 2026-06-20._
|
||
|
||
- `[2026-06-16]` **`docker compose up -d` on a multi-service vLLM stack recreates ALL its
|
||
services** (shared `.env` → all configs re-hash) → concurrent memory-profiling race
|
||
("No available memory for the cache blocks"). Target one: `docker compose up -d <service>`.
|
||
(auto-memory `feedback_compose_up_recreates_whole_stack`)
|
||
|
||
_76 older entries archived to archival-memory.md._
|
||
_Archived 2026-06-20._
|
||
|
||
|
||
- `[2026-06-19]` **litellm CONFIG-loaded models can't be hot-removed — `/model/delete` is DB-only.** Only a process restart drops a config model. The INVERSE works: `/model/new` live-ADDS (no bounce) but dups with config on restart. Clean no-bounce-add: live-add via API, delete the DB copy before the eventual bounce.
|
||
_Archived 2026-06-21._
|
||
|
||
- `[2026-06-19]` **the litellm `*` wildcard routed to DECOMMISSIONED llama-swap → a stale/typo'd model name returned a misleading `Connection error`, not `model not found`.** Bit a brokkr call to the renamed `qwen-image-judge`. **→ RESOLVED 2026-06-20: wildcard removed (d0eb09c) — unknown models now 404 cleanly.**
|
||
_Archived 2026-06-21._
|
||
|
||
- `[2026-06-19]` **A MoE card can't be filled to 0 bytes free — the FusedMoE transient workspace (~3.09 GB) is the floor**, allocated OUTSIDE `gpu-memory-utilization`. FIX: `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` + leave ~3.2 GB free (util <= ~0.95 on a tight MoE).
|
||
_Archived 2026-06-21._
|
||
|
||
- `[2026-06-19]` **vLLM decode tok/s: ALWAYS discard the first generation (cold warmup).** Cold ~ 24.8 tok/s vs warm 92.7 — a cold read undersells decode ~3-4x. Measure run 2+ over a >=256-token output.
|
||
_Archived 2026-06-21._
|
||
|
||
- `[2026-06-19]` **For full native 256K on one 96 GB card, nightly+turboquant-4bit-KV was unnecessary for the Qwopus MoE.** Stable fp8 KV already fits 262144 (hybrid SSM+attn caches KV only on attn layers → small pool). Reach for fp8 first.
|
||
_Archived 2026-06-21._
|
||
|
||
- `[2026-06-18]` **mmartial `comfyui-nvidia-docker`: root pip installs CRASH-LOOP the container.** `docker exec -u 0 pip install` leaves root-owned files in the uid-1000 venv → the boot script (uid 1000) fails its torch step → crash loop. FIX: `chown -R 1000:1000 /comfy/mnt/venv`; pin with `DISABLE_UPGRADES=true`. (auto-memory `reference_irv_ml1_comfyui_mmartial`)
|
||
_Archived 2026-06-21._
|
||
|
||
- `[2026-06-18]` **ComfyUI custom nodes break on version-assumption drift** — box runs 0.24.1; TeaCache imports `precompute_freqs_cis` at MODULE level (guard the LTX-only import). `pip install tensorrt` defaults to cu13 → use `tensorrt-cu12`.
|
||
_Archived 2026-06-21._
|
||
|
||
- `[2026-06-17]` **Mistral HF→NVFP4 quant: the placement knob is the whole game.** `device_map="cpu"` (CPU-resident, sequential onload) works; `"auto"` OOMs, `max_memory` offloads to meta. Read shards with plain `read()` + `safetensors.torch.load(bytes)`, NOT `safe_open` (mmaps the 50 GB shard → ENOMEM on `/tank` ZFS).
|
||
_Archived 2026-06-21._
|
||
|
||
- `[2026-06-17]` **HF-format Mistral Small 4 is UNSERVEABLE on vLLM** — no HF `Mistral4` backbone; serves ONLY via the native loader. A HF-format quant MUST be converted to native first.
|
||
_Archived 2026-06-21._
|
||
|
||
- `[2026-06-20]` **A full disk wedges gitea's Actions notifier SILENTLY — and it STAYS wedged after disk recovers; only a `docker restart gitea` clears it.** After the 94 GB clickhouse disk-full event, gitea stopped CREATING workflow runs on push (push succeeds, ref lands, but NO run spawns — newest run stale, queue empty), while the act_runner stayed registered. NOT run-pickup, NOT disk-now (52% free), NO gitea errors/panics — a silent notifier stall. Hit BOTH vh/arbo (comfy-dev) and vh/Worldtree (worldtree-dev's deploys) → it's global, not repo-specific. Fix: `docker restart gitea` on ana-docker (it had been up 2 weeks, never bounced). A restart does NOT retroactively fire pushes made during the outage — re-push/re-trigger after. Reinforces the deferred docker-daemon-default log-cap fix (the disk-full that started it can't be allowed to recur).
|
||
_Archived 2026-06-25._
|
||
|
||
- `[2026-06-20]` **Mistral-Medium-3.5-128B NVFP4 (RecViking) on ana-ml2 GPU 0 — ABANDONED, too slow (operator confirmed).** Stood up as a temporary speed-check displacing mistral-small-4; measured base **~19.2 tok/s** decode on one Blackwell (vs `gen`/Qwopus3.5-122B ~92.7 tok/s warm) → confirmed the operator's "too slow" suspicion. KEEP these serving findings: (1) vLLM **nightly** loads the HF-format NVFP4 weights DIRECTLY — no Mistral native-convert (unlike Small 4, same `mistral3` arch) — via the FlashInfer Cutlass NVFP4 kernel + `turboquant_4bit_nc` KV; TP=1 fits the ~70 GB NVFP4 on one 96 GB Blackwell. (2) **EAGLE was a dead end**: the official native FP8 head (`mistralai/Mistral-Medium-3.5-128B-EAGLE`) crashes in CUDA-graph replay with graphs on, and nets NO speedup with `--enforce-eager` (~18.5 vs ~19.2 base) — likely low draft acceptance against the NVFP4 (non-official-FP8) base. Untracked workspace files (`stacks/mistral-medium-3.5/` + `playbooks/serve-mistral-medium-3.5.yaml`) removed. ⚠️ Weights still staged on ana-ml2 `/tank/aimodels/mistral-medium-3.5-{nvfp4,eagle}` (~70 GB+) — pending operator OK to free.
|
||
_Archived 2026-06-25._
|
||
|
||
- `[2026-06-20]` **A 94 GB unrotated `langfuse-clickhouse` json.log was the REAL disk-filler (not image bloat) — there was NO log rotation anywhere.** No per-stack `logging:` limit and no docker-daemon default → ClickHouse's verbose stdout grew unbounded to ~94 GB and filled ana-docker's `/` (healthchecks then failed fleet-wide on `runc /tmp: no space left`). Lesson: on a full disk, the symptom (all-unhealthy) is the FORK-can't-write healthcheck, not the containers; find the runaway via per-container-dir `du`, not `docker system df`. The systemic fix is a daemon-default `log-opts` (`max-size`/`max-file`) — NOT just per-service patching.
|
||
_Archived 2026-06-25._
|
||
|
||
- `[2026-06-20]` **OmniVoice's upstream "RTF 0.025 / 40×" does NOT hold — diffusion has a ~fixed per-call overhead → TTFA floor + multi-chunk streaming is a net loss.** Measured ~2.8× (32-step) / 5.6× (16-step) on the 3090; every chunk costs ~the same regardless of length, so naive per-sentence streaming triples total gen time + starves. Use few/aggressive chunks; chatterbox-fast (autoregressive) stays the low-TTFA front-end.
|
||
_Archived 2026-06-25._
|
||
|
||
- `[2026-06-20]` **Named buildx builders accumulate cache the standard prunes MISS.** `docker image prune -a` + `docker builder prune` don't touch `buildx_buildkit_<name>-builder0_state` volumes (per-builder buildkit state). Prune them with `docker buildx prune --builder <name> -af`, or `docker volume rm` orphaned ones (builder already removed).
|
||
_Archived 2026-06-25._
|
||
|
||
|
||
- `[2026-06-25]` **althing "unreachable: <machine>" can MASK an app-level 500.** Raw network was
|
||
clean; root cause = receiver DB agents-table not synced with the config roster → delivery 500'd
|
||
"unknown to: <handle>", MAPPED to "unreachable". Diagnose: raw curl to :8087 + connect-probe ⇒ NOT
|
||
network. Fixed in althing v0.17.1. `reference_nh3_extdev_althing_mesh`.
|
||
_Archived 2026-07-08._
|
||
|
||
- `[2026-06-20]` **rest-server `.htpasswd: permission denied` = the ana-nas NFS mount FAILED (ghost
|
||
file on the local mount point), NOT a decommission.** `mnt-backup.mount` stuck `failed` (fstab bare
|
||
`defaults`) → rest-server serves an empty local dir. Recovery in disaster-recovery.md.
|
||
_Archived 2026-07-08._
|
||
|
||
- `[2026-06-20]` **The DEFAULT `ssh ana-docker` is `lkraven` (no NOPASSWD) — but `ssh
|
||
infra-ops@ana-docker` HAS NOPASSWD root.** A `sudo cp` as lkraven silently failed → nearly punted
|
||
the rest-server recovery. Reach for `infra-ops@ana-docker` for sudo ops.
|
||
_Archived 2026-07-08._
|
||
|
||
- `[2026-08-25]` **Four throughput levers measured and killed — do not re-chase.** (1) **Fused MoE / `grouped_mm`** — 0.9% *slower* than the Python loop and dense GEMM is only 7.9% of the step, capping the whole category near 10%. (2) **CUDA graphs / `torch.compile` over the expert loop** — the two-term scaling fit closed with residuals under 3ms and needed NO constant term, so there is no fixed per-batch cost to amortise; 3,840 expert-GEMM launches per forward are not what we pay for. (3) **`liger` fused linear CE** — the chunked CE measured **1.1% of the step** forward, ~3% with recompute. A tidy-up, not a lever. (4) **Selective gradient checkpointing** — ~2% of a post-fix step, real bug surface. Also: **token-budget batching is dead by the same fit** — with no constant term, total time over a fixed set of widths is invariant to how you group them; only the widths matter, which is exactly why bucketing works and repacking does not.
|
||
_Archived 2026-09-09._
|
||
|
||
- `[2026-08-25]` **`sample_packing` is NOT strictly better than bucketing on this model, and I told the operator it was before brokkr corrected me.** Packing needs FA2 varlen or a block-diagonal mask; FA2 is unavailable here (head_dim 512 > 256 cap), so packing means an explicit 4D mask on EVERY batch. Bucketing produces **78.3% exactly-zero-pad micro-batches** which recover the `is_causal` fast path on the 5 global layers — measured at 9.4% of step time. Packing forfeits that. ⚠ **The conclusion flips under `flex_attention`**, where a block-diagonal mask is just another BlockMask: do not carry "packing is bad" past the backend decision.
|
||
_Archived 2026-09-09._
|
||
|
||
- `[2026-08-25]` **Merging a tune back toward STOCK to fix overfitting would UNDO the abliteration.** brokkr recommended a 50/50 merge-back, then retracted it himself: the published recipes merge into `google/gemma-4-*-it`, and following that literally re-installs exactly the refusal directions the abliteration removed — silently, because the merged model looks *healthier* on general benchmarks. Any merge-back must target the SAME abliterated base. Wider lesson: **recipe cards are per-checkpoint artifacts, not per-family** — the advice came from a card for a DENSE STOCK 31B applied to a MoE ABLITERATED 26B-A4B, three axes apart on a shared name.
|
||
_Archived 2026-09-09._
|
||
|
||
- `[2026-08-24]` **AES-GCM on the Anaheim tunnels — impossible, not merely hard.** UniFi's manual site-to-site IPsec implements no AEAD cipher at all: eight GCM spellings rejected `api.err.InvalidPayload` against a passing `aes256` control. Blocks both tunnels since both far ends are UDMs. Accepted enum is `aes128/aes192/aes256/3des` — and 3DES is *slower* (no ARM instructions, 64-bit blocks), so AES-128 is the floor.
|
||
_Archived 2026-09-09._
|
||
|
||
- `[2026-08-24]` **Pointing the UDM's `wan_dns1` at AdGuard — silently ignored.** It persists and reads back correctly but the LAN-facing forwarder never uses it; proven with fresh uncached ad domains (AdGuard answers `0.0.0.0`, the UDM returned real IPs). Reverted rather than left in place.
|
||
_Archived 2026-09-09._
|
||
|
||
- `[2026-08-24]` **A multi-DUID DHCPv6 VM to claim NH3's seven unclaimed /64s — declined by the operator.** The BGW has no IP-passthrough (confirmed, we hold admin), so the only route needs re-cabling, split-stack routing and **rebuilding the entire v6 firewall policy off the UDM**. The prefixes are easy; the firewall rebuild is why nobody wants them. Do not re-raise on "there are seven free prefixes".
|
||
_Archived 2026-09-09._
|
||
|
||
- `[2026-08-23]` **A `HEAD == GITHUB_SHA` assertion in the hrafn CI — added, broke the checkout twice, removed.** It needed the `git` binary (run 9920, exit 127); installing `git` then flipped `actions/checkout@v4` off its **node** implementation onto the git binary, which died on a missing CA bundle (run 9921). A nice-to-have assertion changed the checkout's code path and broke a working pipeline. Removed rather than patched with `ca-certificates` — it guarded a hypothesis that proved wrong. **Do not add `git` to that prereq step.**
|
||
_Archived 2026-09-09._
|
||
|
||
- `[2026-08-23]` **Repointing `selene-1-mini-8b` at gen's endpoint — proposed by me, correctly overruled.** *"never repoint a named model at a different model's endpoint — that is intentionally misleading."* The trap is that it does not feel like deception; it feels like sparing consumers a migration. That framing is the tell. Role aliases move; model names die with the model and 4xx.
|
||
_Archived 2026-09-09._
|
||
|
||
|
||
## Archived 2026-08-02 — Recent decisions (archived)
|
||
|
||
### 2026-07-08-worldtree-mimir-deploy-blocker-resolved-mid-session
|
||
|
||
- `[2026-07-08]` **worldtree Mimir deploy-blocker resolved (mid-session):** synced `persona.envelopes.assistant` +
|
||
`envelope_grants:[]` verbatim from the baked canonical into BOTH corviduo-dev instances (demo+personal),
|
||
YAML-validated via each container's own parser; worldtree-dev cleared to push the Mimir-bound image. (Was my
|
||
parked R32 1C envelope-mirror come due — see [[reference_corviduo_dev_emergency_ops]] config-sync recipe.)
|
||
|
||
_Archived 2026-08-02._
|
||
|
||
### 2026-07-09-two-parked-items-closed-phantom-qwen3-6-35b
|
||
|
||
- `[2026-07-09]` **Two parked items closed: phantom `qwen3.6-35b-a3b` alias VERIFIED already-gone; ana-docker
|
||
docker log-cap SOLVED no-bounce.** (1) **Phantom**: absent from `/v1/models` + `/model/info` (config+DB
|
||
registry), zero litellm log refs — the parked "400s in /v1/models" note was STALE (already cleaned in the
|
||
2026-07-08 gen repoint to `-heretic`); bare token survives only in 2 config COMMENTS (lines 76/80). Nothing to
|
||
remove. (2) **Log-cap**: running containers were UNCAPPED (182M json-logs, top offender 59M) because
|
||
daemon.json's `max-size 10m/max-file 3` only applies to containers CREATED AFTER a daemon restart — it never
|
||
reaches already-running ones. No-bounce fix = `/etc/logrotate.d/docker-containers` (**copytruncate** — dockerd
|
||
opens json-logs `O_APPEND` so truncate-in-place resets cleanly, no sparse-file corruption; `size 10M`,
|
||
`rotate 3`, `compress`, `su root root`), auto-picked-up by the daily `logrotate.timer`. Force-ran + gzipped the
|
||
frozen `.1` archives → **182M → ~55M** (44M active + 11M gz), every container kept its multi-week uptime
|
||
(zero bounce, verified). **LATENT FOOTGUN FLAGGED (not yet fixed, operator's call): daemon.json declares
|
||
`live-restore:true` but the RUNNING daemon has it FALSE** (daemon.json was edited after the last daemon start,
|
||
never reloaded) → the NEXT `systemctl restart docker` / crash / pkg-upgrade **bounces ALL ana-docker containers
|
||
once**. Fix WITHOUT a bounce = `systemctl reload docker` (SIGHUP loads live-restore into the running daemon;
|
||
log-opts are NOT SIGHUP-reloadable, which is why logrotate — not the daemon cap — is the enforcer for running
|
||
containers).
|
||
|
||
_Archived 2026-08-02._
|
||
|
||
### 2026-07-09-morpheus-tts-off-the-shelf-voice-pipeline-shipped
|
||
|
||
- `[2026-07-09]` **mOrpheus TTS off-the-shelf voice pipeline SHIPPED end-to-end (irv-ml1) + wired into
|
||
gateway-chat.** Full arc (commits): gen served-name honesty rename aeon→qwen3.6-35b-a3b-heretic (99a4a17,
|
||
vLLM served-name + litellm refs, so /v1/models + spend-logs name the real model); permanent 2-container stack
|
||
(01eedd8); gateway-chat auto-voice quoted dialogue (c948013); streaming decode TTFA 4.5s→0.8s (da76829);
|
||
max_tokens 1200→2400→3500 with a context-clamp (f363fe6, 0655a37 — long lines were clipping at 14.6s, and
|
||
`repetition_penalty` 1.1 is LOAD-BEARING: at 1.0 the model never stops); AudioContext resume-on-gesture
|
||
no-sound fix (033f368); pre-chunk by QUOTED SECTION not sentence for prosody (a1f3023→f295cc1); staged clone
|
||
voices baddy/beatrice/whisper (0655a37 + runtime .wav/.txt in the voices dir); agent voicing prompt (a573514).
|
||
**Load-bearing config, all encoded in stacks/mOrpheus/: bf16 not FP8, image v0.23.0 not latest, GPU=3090 not
|
||
A6000, rep_penalty 1.1.** Serving-viability confirmed: vLLM concurrency near-linear to 8× (707 tok/s).
|
||
|
||
_Archived 2026-08-02._
|
||
|
||
### 2026-07-09-granite-gen-memory-extractor-bind-host-synced-on
|
||
|
||
- `[2026-07-09]` **granite→gen `memory_extractor` bind host-synced on demo+personal Worldtree (Vuong-directed,
|
||
#335 Slice-4).** Changed `model_roles.yaml` memory_extractor `binds.catalog_id` `summarizer`→`gen` (overrides
|
||
intact: thinking:false/temp0/8192) on BOTH `/opt/worldtree{,-personal}/config`; `memory_distiller` left on
|
||
`summarizer` (range-scoped sed `/memory_extractor:/,/memory_distiller:/` — the naive global replace would've hit
|
||
both); backups `*.bak-preqgen-20260709`; validated via each container's OWN yaml parser. **DEMO LIVE on gen**
|
||
(b43 `d501e516732d` auto-deployed mid-edit + its restart RACED my edit by ~2min → I restarted
|
||
`worldtree-worldtree-api-1` to activate; healthy, live process resolves memory_extractor=gen). **PERSONAL
|
||
NOW LIVE on gen too** — Vuong authorized the restart (via wt-dev); restarted `worldtree-personal-worldtree-api-1`
|
||
after a PRE-FLIGHT that ran the app's OWN `load_model_roles()` (`core/llm/roles.py:121`) against the synced config
|
||
INSIDE the running `c9986cd` container: `gen` in catalog, all 9 roles resolve, no `DanglingBindingError` → proven
|
||
safe on the OLDER image BEFORE touching it (model_roles-delta-alone clean; no full-config-set sync needed).
|
||
StartedAt 20:50:55Z, healthy, resolves gen. **BOTH instances live on gen.** **LESSON:
|
||
the bind-mount `/opt/worldtree*/config` SHADOWS the baked `/app/config-defaults/` → the deploy alone never
|
||
updates the ACTIVE config; the host edit is required AND a restart activates it (role registry cached at boot) —
|
||
pre-stage BEFORE the deploy's restart or you race it.** FOLLOW-UPS (non-blocking): (a) `memory.extractor.user_pass`
|
||
parity block → self-serve from the b43 baked `defaults.yaml` (pydantic-default no-op); (b) stale `gen` provider
|
||
description (Mistral-Small-4 → qwen3.6-35b-a3b-heretic) → wt-dev owns the REPO-side `providers.yaml` fix
|
||
(operator's call — NOT purely cosmetic: the gen/dialogue + classifier entries carry Mistral-Small-4 SAMPLING
|
||
defaults that drive mask/lofn/forseti/mimir dialogue, so wt-dev re-validates for qwen), host cosmetic sync pairs
|
||
when it deploys. Gave wt-dev the VERIFIED canonical 4-alias set (backends+samplers read from the LIVE gateway
|
||
config, not the doc); corrected `docs/pfi/model-sampler-defaults.md` seat 4 (had lagged QwQ-RpR-v4 → Deckard-PKD;
|
||
live gateway was always Deckard). Operator SCOPED IN the character-RP re-point (2026-07-09):
|
||
character→char-rp / thoughtful-character→char-rp-reasoning (character-rp per wt-dev's role semantics), moving
|
||
character RP off the GENERAL qwen onto the dedicated Magidonia/Deckard seats. Relayed to wt-dev w/ the mapping
|
||
principle + a SAMPLER-OVERRIDE warning (DROP character-rp's old temp0.75/top_p0.85 overrides — carried onto the
|
||
dedicated seats they'd clobber the canonical RP tuning DOWNWARD) + ratatoskr-reach note (role call is transparent
|
||
but Magidonia/Deckard quality/latency differs from gen). DONE 2026-07-09: wt-dev committed 5d4fa4a (v1.0.0b44,
|
||
UNPUSHED — operator drives push); operator directed host-ahead-of-push, so I sourced BOTH config files directly
|
||
from that unpushed commit (local `~/development/Worldtree` checkout — capital W; `git show 5d4fa4a:config/…`) +
|
||
mirrored VERBATIM to `/opt/worldtree{,-personal}/config` on demo+personal, paired-pre-flighted via the app's
|
||
`load_model_roles()` (no DanglingBinding), restarted both → LIVE: character→char-rp, thoughtful-character +
|
||
character-rp→char-rp-reasoning, memory_extractor→gen preserved. Backups `*.bak-prerp-20260709`. context_window
|
||
VERIFIED (llama.cpp /props + char-rp-gguf `.env`): char-rp **131072**, char-rp-reasoning **262144** (gave wt-dev
|
||
to patch the repo from its interim 32768). **HOST AHEAD of repo-remote until the operator pushes 5d4fa4a** (baked
|
||
config converges with the bind-mount on push+deploy). GOTCHA: demo≠personal — PERSONAL was already partly
|
||
re-pointed (2026-07-06 AEON-era character→char-rp) so its delta was mostly stale-AEON-descriptions→Magidonia/Deckard
|
||
+ character-rp + stripping personal's char-rp `default_params` temp0.7/top_p0.8 that CLOBBERED the gateway RP tuning
|
||
downward; DEMO had no char-rp catalog entries at all (b44 adds them). Diffed each instance vs b44 before applying
|
||
(both deltas = expected changeset only, nothing instance-specific clobbered). wt-dev PATCHED the context_window in **b45/3384a37**
|
||
(char-rp 131072, char-rp-reasoning 262144). BUMPED HOST-AHEAD on both instances (operator-directed 2026-07-09):
|
||
mirrored b45's providers.yaml → `/opt/worldtree{,-personal}/config`, restarted, verified LIVE (char-rp ctx
|
||
131072, char-rp-reasoning 262144, bindings intact); backups `.bak-b44interim-20260709`. HOST now = **b45
|
||
canonical** (providers.yaml) + b44 (model_roles unchanged b44→b45). STILL PENDING: (1) operator's batched push
|
||
of **b44+b45** (`5d4fa4a`+`3384a37`) to converge the repo-remote — host is ahead, no fork; (2) user_pass parity
|
||
block (defaults.yaml — NOT in either, separate). Threads `01KX3SGH`/`01KX48QP` (worldtree-dev),
|
||
monitor armed. See [[reference_corviduo_dev_emergency_ops]].
|
||
|
||
_Archived 2026-08-02._
|
||
|
||
### 2026-07-09-granite-gen-memory-extractor-bind-green-lit-for
|
||
|
||
- `[2026-07-09]` **granite→gen memory_extractor bind GREEN-lit for worldtree-dev (Worldtree #335 Slice 4).**
|
||
Answered their VRAM/concurrency headroom check: gen (qwen 35B-A3B heretic) has ample headroom for ~2 bursty
|
||
idle-triggered extractor calls (fixed 0.37 util; KV pool runs 0-2%; --max-num-seqs 16, near-linear batching).
|
||
Corrected their stale "gen = Mistral Small 4 / 119B-6B" belief (gen IS the qwen 35B-A3B heretic since 2026-07-08).
|
||
**This bind is INDEPENDENT of the full granite RETIRE** (reclaim ~32GB on ana-ml2 GPU1) — that stays the
|
||
operator's call, pending brokkr R33 portfolio + production-concurrency due-diligence. Tracked: althing thread
|
||
01KX3SGH… (worldtree-dev) + brokkr's gen-absorbs-granite consult (thread 01KX2V32…) + [[reference_litellm_gateway]].
|
||
|
||
_Archived 2026-08-02._
|
||
|
||
## Archived 2026-08-02 — Tried and abandoned (archived)
|
||
|
||
### 2026-07-01-a-personal-worldtree-ci-deploy-that-fails-85s
|
||
|
||
- `[2026-07-01]` **A personal-Worldtree CI deploy that fails ~85s in with "not found / unauthorized"
|
||
is usually the pull-only-vs-build RACE, not registry-auth.** `deploy-personal.yml` is PULL-ONLY but
|
||
fires on the `staging/vX` tag simultaneously with `deploy.yml`'s build → pulls before the push
|
||
finishes. FIX: re-run once built, or gate on `workflow_run: completed`.
|
||
|
||
_Archived 2026-08-02._
|
||
|
||
### 2026-07-04-on-prem-t1-train-that-keeps-any-ana
|
||
|
||
- `[2026-07-04]` **On-prem T1 train that keeps ANY ana-ml2 serving up = ~6-8 DAYS** (1-GPU + NVMe
|
||
ZeRO-Infinity offload; MoE ~10B-active cuts FLOPs but NOT the 244G base's param I/O). The only fast
|
||
on-prem path is a FULL ana-ml2 shutdown (both GPUs + the ~421G vLLM RAM freed → base fits in the
|
||
566G CPU RAM) → CPU offload → ~1-day full-fleet outage. Cloud (no offload) = hours. `reference_t1_cloud_train_plan`.
|
||
|
||
_Archived 2026-08-02._
|
||
|
||
### 2026-07-04-a-systemd-user-daemon-that-shells-out-to
|
||
|
||
- `[2026-07-04]` **A systemd `--user` daemon that shells out to `~/.cargo/bin`/`~/.local/bin` tools
|
||
needs an explicit `Environment=PATH`** — the minimal `--user` default silently drops them. The
|
||
althing herald lost `zellij` → silent `pane-miss` for ALL config-backed TUI/pane agents; CC + FIFO
|
||
routes were unaffected, so it was invisible from a CC session. `reference_nh3_dev_althing_herald`.
|
||
|
||
_Archived 2026-08-02._
|
||
|
||
### 2026-07-09-fp8-breaks-morpheus-audio-token-generation
|
||
|
||
- `[2026-07-09]` **FP8 breaks mOrpheus audio-token generation.** `--quantization fp8` on the 3B → 0 valid SNAC
|
||
frames even at GREEDY (degenerate audio+text mix, no start-of-speech); bf16 is clean (28/28 frames). Quant-breaks-
|
||
TTS, same class as the Angel-NVFP4 lesson below. bf16 is REQUIRED (so the operator's "util 0.1" wish is moot — the
|
||
bf16 weights alone are 6.6GB). NB the raw-token benchmark RTF 0.50 was fp8+graphs = never real.
|
||
|
||
_Archived 2026-08-02._
|
||
|
||
### 2026-07-09-sentence-chunking-tts-loses-prosody
|
||
|
||
- `[2026-07-09]` **Sentence-chunking TTS loses prosody** — generating each sentence cold flattens the intonation that
|
||
spans a line. Chunk by QUOTED SECTION (whole quote = one gen call). Also: `repetition_penalty` >1.1 BREAKS cloning
|
||
(penalizes the ~1100 in-context reference audio tokens; keep ≤1.1 on the clone path).
|
||
|
||
_Archived 2026-08-02._
|
||
|
||
### 2026-07-09-hf-whisper-datasets-aren-t-actually-whispered
|
||
|
||
- `[2026-07-09]` **HF whisper datasets aren't actually whispered.** Claris-Whispered-English measures voiced 0.8
|
||
(not a whisper) + IPA transcripts; `datasets` audio decode needs torchcodec (wants CUDA-13, incompatible w/ the
|
||
cu124 venv). LPC DSP-whisperize went unstable (NaN). **kokoro `af_nicole` IS a genuine whisper** (voiced 0.24) —
|
||
that (operator's pointer) is the working whisper reference source, not TTS-voice screening or dataset-hunting.
|
||
|
||
_Archived 2026-08-02._
|
||
|
||
|
||
## Recent decisions (archived 2026-08-03 batch)
|
||
|
||
### 2026-07-14-nvfp4-mtp-fast-char-rp-reasoning-seat-landed
|
||
|
||
- `[2026-07-14]` **NVFP4+MTP fast char-rp-reasoning seat LANDED + LIVE + gateway-repointed + VRAM-tuned.** Modelopt-format re-quant made MTP work. The load-crash root cause = **vLLM 0.24 does NOT propagate modelopt `exclude_modules` to the spec-decode DRAFT model** → the bf16 mtp head gets quantized → shape crash; NO checkpoint config fixes it (`is_layer_skipped` is exact-membership, and the drafter never sees the mtp excludes) → **workaround = a mounted `sitecustomize` that force-skips `mtp.*` in `is_layer_skipped`** (upstream vLLM bug to file). Productionized as compose stack `heretic2-charrp-reasoning` (:8018, workaround baked in). Gateway `char-rp-reasoning` alias fixed: repointed off the stale GGUF served-name `deckard-pkd-27b`, added `enable_thinking:true`, **dropped `min_p`** (MTP-incompatible), canonical samplers temp1.0/top_p0.95/top_k20. Rebalanced GPU0 (gen 0.37→0.30/16-seq/256K + reasoning 0.39/16-seq/192K+MTP + char-rp 128K, 2.7GB free). All 4 gateway roles verified; vLLM reasoning-parser confirmed **leak-free** (unlike the GGUF budget-forcing). Full record + the 4 quant landmines in `docs/runbooks/heretic2-nvfp4-mtp-seat.md`; committed `982c319`. Open (non-blocking): brokkr P00 (seat is live ahead of it), retire the stopped GGUF reasoning seat, file the vLLM bug.
|
||
|
||
_Archived 2026-08-03._
|
||
|
||
### 2026-07-14-nvfp4-quant-chase-resolved-gibberish-pivoted-to-modelopt
|
||
|
||
- `[2026-07-14]` **NVFP4 quant chase RESOLVED (gibberish) + PIVOTED to modelopt for MTP.** One ~40-min GPU0 window. Root-caused the `!!!!` to the quant NAMESPACE (text-only `AutoModelForCausalLM`→`model.layers.*` keys; vLLM serves only `Qwen3_5ForConditionalGeneration`, which needs `model.language_model.*`) — found from config diffs + vLLM source with ZERO GPU time; fixed by loading as `AutoModelForImageTextToText`. NVFP4 now serves COHERENT (validated greedy). BUT base NVFP4 ≈53 tok/s ≈ GGUF's 59.5 at batch-1 (no single-stream win) AND MTP = 0% acceptance on compressed-tensors (bf16 mtp head only loads on the modelopt format). Operator chose to **pursue a modelopt-format re-quant** (the only path to the 2-4× MTP goal; AEON-proven on this exact Qwen3.6-27B arch). Scoped + de-risked: AEON `/tank/aimodels/qwen36-27b-aeon-nvfp4` = the modelopt reference (quant_method modelopt, 1967 tensors, 15 bf16 mtp keys identical to graft); nvidia-modelopt 0.45.0 installs + `mtq.quantize`/`NVFP4_DEFAULT_CFG`/`export_hf_checkpoint` API confirmed; pipeline unchanged except swap llm-compressor→modelopt. Seats restored; char-rp-reasoning stays GGUF. Full plan in Current state ★ section.
|
||
|
||
_Archived 2026-08-03._
|
||
|
||
### 2026-07-14-pursue-the-nvfp4-mtp-fast-char-rp-reasoning
|
||
|
||
- `[2026-07-14]` **Pursue the NVFP4+MTP fast char-rp-reasoning seat to completion** (Vuong-directed via /snapshot: "chase the nvfp4 quant, we know it works, write down the recipe"). Full recipe + diagnostic ladder in Current state / in-flight above. Artifacts on ana-ml2 `/tank/aimodels/heretic2-nvfp4-work/` + scripts committed in eshpfi `services/heretic2-nvfp4-quant/`.
|
||
|
||
_Archived 2026-08-03._
|
||
|
||
### 2026-07-14-char-rp-reasoning-seat-deckard-pkd-neo-code
|
||
|
||
- `[2026-07-14]` **char-rp-reasoning seat: Deckard-PKD → NEO-CODE = Heretic2-Thinking (Qwen3.6-27B)** — R36 gate PASSED (tools 0.967, #355 runaway ELIMINATED). #355 was MODEL-level (Deckard emitted qwen3_coder XML malformed → mangled args → retry-runaway), NOT the reasoning-budget bug; NEO-CODE emits it clean. Custom llama.cpp KEPT (qwen3_coder parse — stock b8840 predates it — + PR#25544). Committed f960a73; full record auto-memory [[charrp-custom-llamacpp-pr25544]].
|
||
|
||
_Archived 2026-08-03._
|
||
|
||
### 2026-07-14-soong-lab-webhook-auto-deploy-real-root-cause
|
||
|
||
- `[2026-07-14]` **soong-lab webhook auto-deploy real root cause = gitea `webhook.ALLOWED_HOST_LIST`** (was `external, 10.100.0.0/16` = NH3-only; blocked corviduo-dev's Anaheim `10.250.x` → gitea refused to deliver, never opened the connection). Fixed to fleet-wide `10.0.0.0/8` (app.ini `[webhook]`) + gitea restart; listener now logs every delivery. The ufw `10/8` open (also this session) was a real-but-secondary gap. Committed 462d528.
|
||
|
||
_Archived 2026-08-03._
|
||
|
||
### 2026-07-13-355-residual-root-cause-supersedes-the-litellm-gateway
|
||
|
||
- `[2026-07-13]` **#355-residual ROOT CAUSE (supersedes the "LiteLLM gateway holds while seat idles" entry below — that was DISPROVEN).** char-rp-reasoning enters a non-terminating REASONING loop (tool-call-retry planning) and runs to `max_tokens=32768` (~22 min @ 24.7 tok/s, ~13% of requests); the seat GENERATES all 32768 tokens (not idle), and `--reasoning-budget 400` is NOT enforced. 3-source-confirmed (spend_logs completion_tokens=32768 ×4; seat eval-time log; pcap 100%-`reasoning_content` deltas). Server-side fix wanted (operator: no max_tokens ceiling) → routed to brokkr (accepted, pulled dvalin). Lesson (again): confirm before concluding — the seat-idle claim came from reading only the ≤73s requests + missing the concurrent 32768-token slots. See ACTIVE 1.
|
||
|
||
_Archived 2026-08-03._
|
||
|
||
### 2026-07-13-deploy-speed-real-bottleneck-uv-sync-memory-s
|
||
|
||
- `[2026-07-13]` **Deploy-speed real bottleneck ≠ uv sync (memory's assumption was wrong).** Buildx step log: `chown -R /app` = 251s (copy-up of the root-owned venv into a fresh layer), uv sync only 35.6s, registry layer cache already wired. Fix = drop `/app` from the chown (validated safe: zero /app runtime writes on both live instances) + uv cache-mount. Shipped as PR #359 (branch off origin/main@b60), worldtree-dev green-lit. Expected ~5min off (~11→~6min). Runner-side BuildKit cache task (b) was already done → moot.
|
||
|
||
_Archived 2026-08-03._
|
||
|
||
### 2026-07-13-wt-355-residual-300s-hang-localized-to-our
|
||
|
||
- `[2026-07-13]` WT #355 residual 300s hang localized to OUR LiteLLM gateway (holds 2 char-rp-reasoning requests ~21 min while the seat idles), NOT the seat — Deckard seat EXONERATED (completes ≤72s; `--reasoning-budget 400` forecloses a mid-thinking hang). Corrects worldtree-dev's "seat wedging" diagnosis. Decisive next = the FIN-check (pcap on corviuo). See in-flight ACTIVE 1. **[SUPERSEDED 2026-07-13 — see the ROOT CAUSE entry above; the gateway-hold/slot-leak theory was disproven, the seat was generating 32768 tokens.]**
|
||
|
||
_Archived 2026-08-03._
|
||
|
||
### 2026-07-13-wt-355-turn-lifecycle-fix-validated-on-worldtree
|
||
|
||
- `[2026-07-13]` WT #355 turn-lifecycle fix VALIDATED on worldtree b60 — wedged turns self-terminate cancelled/stalled at the 300s stall-watchdog (turns 2064/2065 vs pre-b60 2061's 16-min no-terminal). worldtree-dev filed follow-ons #356 (rehydrate Tier-3 ctx on resume — the recreate-durability gap), #357 (reclaim orphaned active-turn locks), #358 (LLM-provider read-timeout audit); surfacing to Vuong to prioritize.
|
||
|
||
_Archived 2026-08-03._
|
||
|
||
### 2026-07-13-worldtree-deploy-bottleneck-the-image-build-11-min
|
||
|
||
- `[2026-07-13]` Worldtree deploy bottleneck = the image build (~11 min of a ~12 min deploy), root cause the Dockerfile `uv sync ... --no-cache` + no BuildKit cache-mount (re-downloads all deps cold every build). Fix split: worldtree-dev Dockerfile cache-mount diff + infra-ops runner-side persistent BuildKit cache. Config-only changes skip the build entirely (pinned recreate).
|
||
|
||
_Archived 2026-08-03._
|
||
|
||
### 2026-07-13-ledger-tier-3-consumer-ledger-miranda-provisioned-on
|
||
|
||
- `[2026-07-13]` Ledger tier-3 consumer `ledger:miranda` provisioned on personal :8081 (key b38932f5, GPG-delivered+shredded, allowlist 10.100.10.50:8770 live); `assistant`+`thoughtful-assistant` capability roles added (gen/gen-reasoning) on personal+demo, canonical d8bd497. Chosen instance = personal (the tier-3-consumer instance, ratatoskr+soong-lab colocated).
|
||
|
||
_Archived 2026-08-03._
|
||
|
||
### 2026-07-10-heimdall-grant-ratatoskr-affect-full-on-personal-worldtree
|
||
|
||
- `[2026-07-10]` **Heimdall grant: ratatoskr `affect.full` on PERSONAL Worldtree (operator-approved, worldtree-dev
|
||
R34-v1 request).** Added allow-rule `ratatoskr-affect-full-allow` to `/opt/worldtree-personal/config/policies.yaml`
|
||
(`principal.user_ids:["ratatoskr"]`, action `affect.full`, resource `*`, effect allow), mirroring the #347
|
||
`session-history-write-ratatoskr` rule exactly + placed right after it. **WHY user_ids-based (not tier):** ratatoskr's
|
||
personal key is the minimal **readonly-admin** observability tier, which is NOT in the tier-based
|
||
`affect-render-baseline-allow` (anonymous/user/free/pro/admin) → needs an explicit user_id grant, same as #347.
|
||
R34-v1 (b46, committed UNPUSHED) gates `affect.emit` `dominant_emotion` egress by exposure ceiling (affect.full|safe
|
||
→ present; neither → null); this grant keeps ratatoskr's view alive across the b46 deploy. Surgical exact-string
|
||
insert (preserves comments), backup `policies.yaml.bak-pre-affectfull-20260710`, validated via the CONTAINER's own
|
||
yaml parser (35 rules, +1, payload confirmed). **NOT restarted — deliberate:** rule is on the bind-mount (shadows
|
||
baked), INERT until b46 gating ships, so the b46 CI/CD deploy restart activates it (no live-session blip now). Demo
|
||
untouched (personal-only per key scope). Replied to wt-dev (thread 01KX6DB3…) offering an immediate restart if they
|
||
want it live for pre-b46 testing. See [[reference_corviduo_dev_emergency_ops]].
|
||
|
||
_Archived 2026-08-03._
|
||
|
||
### 2026-07-10-comfyui-v0-27-1-success-on-irv-ml1
|
||
|
||
- `[2026-07-10]` **ComfyUI v0.27.1 SUCCESS on irv-ml1 (operator-confirmed execute-now) — landed on torch 2.12.1,
|
||
SageAttention preserved, crash-loop AVOIDED.** The prior attempt (entry below) crash-looped because a torch
|
||
2.12.1→2.13 bump broke SageAttention's ABI. This time I checked `git diff v0.24.1 v0.27.1 -- requirements.txt`
|
||
FIRST and found **core v0.27.1 leaves `torch` UNPINNED** → the version bump does NOT require torch 2.13 (that came
|
||
only from the mmartial boot-upgrade). So: `git checkout v0.27.1` (clean tree) → `pip install -r requirements.txt`
|
||
as **uid 1000** with a **torch-pin constraint file** (torch/vision/audio pinned to current +cu129) to block any
|
||
transitive bump → torch stayed 2.12.1, SageAttention 2.2.0 untouched. Added decord 0.6.0 (fixed SAM3Segment).
|
||
`docker restart comfyui` → healthy, `/system_stats` comfyui_version=0.27.1, "Using sage attention", HTTP 200, DB
|
||
migrated 0003→0004. Reported the divergence to comfy-dev (thread 01KX6D3C…, reply pending) + asked whether LTXV 2.3
|
||
needs a separate torch-2.13 follow-up (their domain; Krea-2's ≥0.25 need is met by 0.27.1). **LESSON: before a
|
||
mmartial ComfyUI version bump, `git diff <old> <new> -- requirements.txt` — if torch is unpinned, bump the CODE
|
||
without touching torch (constraint-pin it) and compiled exts (SageAttention) survive. `docker exec` lands as uid
|
||
1025(comfytoo), not 1000 — use `-u 1000` + the venv python `/comfy/mnt/venv/bin/python`.** See
|
||
[[reference_irv_ml1_comfyui_mmartial]].
|
||
|
||
_Archived 2026-08-03._
|
||
|
||
### 2026-07-10-comfyui-0-25-x-bump-on-irv-ml1
|
||
|
||
- `[2026-07-10]` **ComfyUI 0.25.x bump on irv-ml1 ATTEMPTED → FAILED → ROLLED BACK (snapshot saved it).** comfy-dev
|
||
requested (Vuong-authorized) bumping the irv-ml1 `comfyui` stack (mmartial image, `/opt/docker/compose/comfyui/`,
|
||
0.24.1) to 0.25.x for Krea-2 + LTXV 2.3. **TWO FINDINGS: (1) `DISABLE_UPGRADES=false`/USE_PIPUPGRADE bumps the
|
||
VENV (torch 2.12.1→2.13.0 + deps) but does NOT advance the ComfyUI CODE checkout** (`/comfy/mnt/ComfyUI` =
|
||
`/worktank/comfyui/run/ComfyUI` stayed 0.24.1 — pinned/detached git, comfy-dev's domain). **(2) the torch bump
|
||
broke SageAttention** (2.2.0 `_fused.so` undefined-symbol `c10::impl::cow::materialize_cow_storage` vs torch
|
||
2.13.0) → `--use-sage-attention` (REQUIRED launch flag in COMFY_CMDLINE_EXTRA) crash-looped ComfyUI. Net: broke
|
||
the working state, zero 0.25.x payoff. **ROLLBACK WORKED**: pre-bump 16G venv snapshot
|
||
`/worktank/comfyui/venv-snapshot-comfyui-0.24.1-20260710.tar` restored (torch 2.12.1 + working SageAttention),
|
||
re-pinned DISABLE_UPGRADES=true, recreated → healthy on 0.24.1, serving :8188. Broken venv parked at
|
||
`/worktank/comfyui/run/venv.broken-torch213-20260710`. **CORRECTED PATH (sent comfy-dev, thread 01KX655V):**
|
||
comfy-dev git-advances the ComfyUI checkout to 0.25.x + reqs → I handle the torch bump + SageAttention
|
||
rebuild-against-2.13.0 + re-pin (snapshot stays as the net). **LESSON: mmartial `DISABLE_UPGRADES` gates ONLY
|
||
the venv pip-upgrades, NOT the ComfyUI git checkout; a torch bump breaks compiled exts (SageAttention) →
|
||
rebuild-after is mandatory.** Bump BLOCKED pending comfy-dev's git-advance. Stack: A6000 (NVIDIA_VISIBLE_DEVICES=1),
|
||
lkraven-owned compose+venv (uid 1000, no sudo needed), COMFY_CMDLINE_EXTRA OOM flags preserved.
|
||
|
||
_Archived 2026-08-03._
|
||
|
||
### 2026-07-10-biweekly-open-weight-releases-scan-cron-set-up
|
||
|
||
- `[2026-07-10]` **Biweekly open-weight-releases scan cron set up for brokkr-smithy (Vuong-authorized).** Durable
|
||
systemd **--user** timer on nh3-dev (`brokkr-landscape-scan.timer`, OnCalendar `*-*-01,15 09:00:00`
|
||
America/Los_Angeles, Persistent=true; linger on) → `.service` → wrapper `~/.local/bin/brokkr-landscape-scan.sh`
|
||
runs headless `claude -p "$(cat ~/.config/brokkr-landscape-scan/prompt.txt)" --dangerously-skip-permissions` in
|
||
`~/development/brokkr-smithy` (ALTHING_HANDLE=brokkr-smithy-dev; **explicit PATH** — the --user minimal-PATH
|
||
footgun; per-run logs `~/.local/state/brokkr-landscape-scan/`). Prompt = brokkr's payload verbatim (LLM/image/TTS
|
||
new-release sweep → ranked synthesis → commit+push+notify). VALIDATED: git-push non-interactive (BatchMode
|
||
ls-remote to gitea, passphraseless key — no agent), headless claude auth (READY smoke). VALIDATED END-TO-END 2026-07-10 (manual run, exit 0):
|
||
web-sweep→synthesis→commit `2ed2f29`→PUSH of scan #2 (open-weight-releases-2026-07-24.md); triaged dwarf input +
|
||
caught baseline errors, quality strong. **HANDLE-COLLISION caught+FIXED** — the headless scan shared handle
|
||
brokkr-smithy-dev with the LIVE session + raced its inbox (eitri's dwarf-reply got stolen by the live monitor);
|
||
registered a dedicated **brokkr-scan-dev** handle (`add-handle`, driver=none) + repointed the wrapper + rewired
|
||
step-5 notify → `althing-cli post --to brokkr-smithy-dev` (NO vuong althing handle exists — confirmed). model=default
|
||
+ `--max-turns 80`. First run under the new handle = 7/15. Off-cycle 07-24 doc is a validation artifact (scheduled
|
||
1st/15th runs date to their own run-date, no collision) — operator naming-convention call pending.
|
||
**NEXT AUTO-RUN 2026-07-15 09:00 PDT.** Manual validation/first run = `systemctl --user start
|
||
brokkr-landscape-scan.service`. Open w/ brokkr (thread 01KX63G6): confirm notify-Vuong handle/mechanism + session
|
||
handle + model/turn-cap. **NEXT brokkr task (operator-sequenced after this): TTS audition env** — Higgs-TTS-3 +
|
||
ZONOS2 + Chatterbox baseline, TTFA/RTF + blind-A/B web-listen (thread 01KX6371; needs GPU-placement + HF-token
|
||
feasibility pass first; brokkr delivers the prompt set after the env's up; protocol doc in brokkr-smithy repo).
|
||
|
||
_Archived 2026-08-03._
|
||
|
||
### 2026-07-08-rp-seat-campaign-closed-char-rp-magidonia-24b
|
||
|
||
- `[2026-07-08]` **RP-SEAT CAMPAIGN CLOSED — char-rp = Magidonia-24B-v4.3 (128K), char-rp-reasoning = Deckard-PKD
|
||
Qwen3.5-27B (256K); both GGUF/llama.cpp on ana-ml2 GPU0 alongside gen (35B-A3B, util 0.37), ~4G GPU0 margin.**
|
||
Arc: (1) replaced broken Angel NVFP4 with Magidonia prose + QwQ-RpR-v4 reasoning (b268f93); (2) max-context via q8_0
|
||
KV (f570604); (3) canonical samplers for all 4 gateway seats, dvalin-derived + char-rp A/B-tuned (aac4bcf);
|
||
(4) rebalanced gen 0.40→0.37 to fund char-rp 128K (f49c4e4); (5) RE-A/B'd the reasoning seat (operator wanted a
|
||
DRY-tolerant model): **Deckard WON** on brokkr's frozen scorer (composite 2.176, 0/30 loops, 0/30 refusals) over
|
||
RpR-v4 (3.716, 1/30 loop), Pantheon-Reasoning (1.383 but 7/30 refusals), Snowdrop+Gembrain (llama.cpp
|
||
template-incompat) — deployed (5f79b40); (6) Deckard→256K (41305bf); (7) dvalin CONFIRMED Deckard samplers = the
|
||
live A/B set is canonical (4954ca0). **GATE LESSON: a llama.cpp reasoning seat needs a STOCK template that natively
|
||
opens `<think>`/`enable_thinking` (Qwen3.x/QwQ pass; ChatML + Gemma-4 fail) — no monkeypatching. INFRA: llama-swap
|
||
b8840 can't load Qwen3.6/Gemma-4 archs → `ghcr.io/ggml-org/llama.cpp:server-cuda` (pulled on ana-ml2).**
|
||
|
||
_Archived 2026-08-03._
|
||
|
||
### 2026-07-08-off-the-shelf-inference-pivot-executed-serve-curated
|
||
|
||
- `[2026-07-08]` **OFF-THE-SHELF INFERENCE PIVOT executed — serve curated abliterated models, stop home-training.**
|
||
Final topology: **gen = `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-NVFP4-Experts-Only`** (LIVE, modelopt, vision,
|
||
util 0.40), **char-rp = an RP unicorn to be found on fresh context** (see Current state). Intermediate steps
|
||
ABANDONED: Pantheon-Reasoning-27B (served briefly as gen — refuses dark fiction via DeepSeek-distilled
|
||
refusal-reasoning, see Tried); Pantheon-27B-with-MTP for RP (bf16 MTP won't load on the compressed-tensors path);
|
||
Angel MS3.2-24B (my NVFP4 quant = garbage). Prefer EXISTING community NVFP4/GGUF quants over self-quanting
|
||
("don't quant unless you have to" — operator). GGUF serving is now on the table for RP (NEVER Ollama). Gateway
|
||
sampling-defaults wiring still PENDING.
|
||
|
||
_Archived 2026-08-03._
|
||
|
||
### 2026-07-08-dpo-was-silently-running-3-epochs-harness-gap
|
||
|
||
- `[2026-07-08]` **DPO was silently running 3 epochs (harness gap) → KILLED at epoch 1.2, retargeted to 0.3
|
||
epochs (operator call).** Root cause: `DpoConfig` had NO `epochs` field + `_dpo_config_kwargs` didn't pass
|
||
`num_train_epochs` → DPO fell through to trl DPOConfig's default 3.0 (SFT correctly pins 1 via SftConfig.epochs
|
||
+ _sft_config_kwargs). Objective SATURATED by ~epoch 0.27 (loss~0, grad~0, acc 1.0, margins~27 flat — the
|
||
off-policy qwopus rejected pairs are trivially separable), so epochs ~0.3→3 were pure over-optimization + a
|
||
~6.5h outage. No mid-run checkpoint (save_steps 500 > total steps; save only at end) → killing lost the run.
|
||
FIX (3 edits to deployed harness, mtf-dev to canonicalize): `DpoConfig.epochs: float = 1` (mirrors SftConfig,
|
||
float for fractions); `_dpo_config_kwargs` now passes `"num_train_epochs": cfg.epochs`; recipe `dpo.epochs: 0.3`.
|
||
GPU-free verified (dpo.epochs=0.3 → num_train_epochs=0.3). Relaunched at 0.3 epoch (~30min precompute + ~12min
|
||
train = ~45min). **DONE + SANITY-CHECKED (exit 0, ~70min wall: ~30min fixed precompute + 45 steps @ ~51s/step;
|
||
train_loss 1.4e-5 @ epoch 0.301).** Fresh `data/spike/dpo_adapter/adapter_model.safetensors` (123MB) + checkpoint-45
|
||
banked. **3-way greedy sanity (base vs SFT vs DPO, via peft load + disable_adapter/set_adapter on GPU0):
|
||
ALL THREE DISTINCT** (base≠sft≠dpo) → full SFT→DPO pipeline applies end-to-end at inference. **DPO 0.3ep is
|
||
COHERENT, fluent, NOT degenerate** (early-stop avoided over-optimization) but the quality delta on a neutral
|
||
literary prompt is SUBTLE (DPO shares SFT's structure — it continues from it — with minor stylistic drift,
|
||
arguably slightly MORE genre-clichéd). Verdict: mechanics proven, quality gain modest as predicted for 0.3ep
|
||
on off-policy pairs; the real unlock remains on-policy rejected regen + on-domain (explicit E-RP) eval +
|
||
the LitBench/holdout run. gen+rp RESTORED healthy. Next: serve fork (SGLang-finish vs merge) on the DPO
|
||
adapter — same rekey_lora_for_vllm.py (zero-z) applies unchanged (mtf-dev confirmed).
|
||
|
||
_Archived 2026-08-03._
|
||
|
||
### 2026-07-08-t1-dpo-leg-is-running-unblocked-2-fixes
|
||
|
||
- `[2026-07-08]` **T1 DPO leg is RUNNING (unblocked) — 2 fixes applied to deployed backend.py.**
|
||
Blocker resolved: (1) **mtf-dev's v0.0.42 stub** `_stub_missing_optional_integrations` (last-resort sys.meta_path
|
||
finder → missing mergekit/llm_blender/weave resolve to MagicMock, never called → zero numerics risk; applied
|
||
VERBATIM to deployed `src/model_training_forge/train/backend.py` after `_unsloth_available()` + call-site before
|
||
`from trl import DPOTrainer`); (2) **my cosmetic `warnings_issued` shim** (trl-0.24 DPOTrainer.__init__:405 does
|
||
`model.warnings_issued["estimate_tokens"]=True` for warning-suppression; custom Qwen3_5 class under transformers
|
||
5.5.0 lacks the attr → `if not hasattr(model,"warnings_issued"): model.warnings_issued={}` before the
|
||
DPOTrainer(...).train() at backend.py:305 — cosmetic, zero training impact). Both edits are on the DEPLOYED
|
||
un-git'd copy only → **mtf-dev must canonicalize the warnings_issued shim into their repo** (told them). DPO
|
||
confirmed training: model loaded (851 shards), full 1196 pairs processed, in precompute_ref_log_probs (GPU0 93%
|
||
util, 54.8GB). Completion watcher armed (bg task) → restore gen+rp + verify dpo_adapter + ping mtf-dev on exit.
|
||
gen+rp STOPPED for the run (authorized window). Output → data/spike/dpo_adapter.
|
||
|
||
_Archived 2026-08-03._
|
||
|
||
### 2026-07-08-t1-dpo-leg-launch-prior-block-now-resolved
|
||
|
||
- `[2026-07-08]` **T1 DPO leg launch — prior BLOCK (now resolved above), kept for the launch recipe.**
|
||
Operator authorized the full DPO stage (via mtf-dev) + went AFK 2h. **PROVEN LAUNCH RECIPE** (replicates the
|
||
SFT container `aeon-t1-sft` exactly, only `--stage sft`→`dpo`): `sudo docker run -d --name aeon-t1-dpo
|
||
--entrypoint python3 --gpus all -e CUDA_VISIBLE_DEVICES=0 -e MTF_FORCE_TRL=1 -e PYTHONPATH=/mtf/src
|
||
-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -v /home/lkraven/model-training-forge:/mtf -v /tank:/tank
|
||
-w /mtf aeon-trainer:latest -u scripts/train.py --recipe recipes/training/qwen-3.5-122b-erp-lora/train.aeon-27b.yaml
|
||
--stage dpo`. **CRITICAL: `--entrypoint python3` is REQUIRED** — aeon-trainer's default entrypoint is
|
||
`["vllm","serve"]` (FROM vllm/vllm-openai) → without the override it runs vllm + hits a torch-ABI crash.
|
||
Dataset verified (pairs_dataset=train.flat.json=1196 pairs). **THE BLOCK:** `from trl import DPOTrainer`
|
||
(backend.py:256) eagerly pulls TRL 0.24.0's WHOLE optional-integration set — cascade: mergekit(missing)→
|
||
immutables→**mergekit-0.1.4↔pydantic-2.13 HARD incompat** (needs pydantic==2.10.6)→llm_blender→dataclasses_json→
|
||
**llm_blender-0.0.2↔transformers-5.5.0 HARD incompat** (TRANSFORMERS_CACHE removed, needs source patch)→weave→
|
||
(more). NONE used by our pair-based DPO. `pip install mergekit` w/deps is UNSAFE (downgrades accelerate
|
||
1.14→1.6). Safe partial recipe derived (core libs held: torch2.10/tf5.5.0/trl0.24.0/peft0.19.1/accel1.14.0)
|
||
but non-convergent → TRULY BLOCKING per operator's carve-out. Did NOT force-hack the proven training image.
|
||
Handed full diagnosis + recommended fix (lazy-import TRL patch, opt b) to mtf-dev (thread 01KWZG8GJX,
|
||
expects-reply, monitor armed). gen+rp RESTORED healthy. Relaunch = 1 min once mtf-dev delivers a working image.
|
||
|
||
_Archived 2026-08-03._
|
||
|
||
|
||
## Recent decisions (archived 2026-08-05 batch)
|
||
|
||
- `[2026-07-16]` **GPU re-org: char-rp→GPU1 + both cards re-optimized for max context.** Moved char-rp (Magidonia-24B) GPU0→GPU1, then maxed context: char-rp-reasoning 150K→256K (util 0.46, 1.56x), gen→256K + seqs 16→32 (util 0.42, 5.43x), granite 64K→**128K full-chapter** (util 0.27, 1.50x). FINAL: GPU0 ~14 G reserve (both seats 256K native), GPU1 ~6.7 G headroom. All healthy. LESSON: KV must hold ≥1× max-len (util-floor crashes) + per-model KV cost varies ~8× (MoE cheap, dense pricey) → tune util empirically.
|
||
_Archived 2026-08-05._
|
||
|
||
- `[2026-07-16]` **granite right-sized → ~10.5 GB freed on GPU1** (util 0.34→0.18 + max-len 131072→65536; KV 6.45 GiB / 1.29x@65536). LESSON: ~950 MiB KV per 0.01 util here + KV must hold ≥1× max-len — util 0.15 crash-looped before 0.18 landed. `.env`-only, recreate `vllm-granite` alone (shared stack). [Superseded by the 07-16 GPU re-org above → 128K.]
|
||
_Archived 2026-08-05._
|
||
|
||
- `[2026-07-15]` **image-bench eviction DONE (parked item closed).** Stopped vllm-qwen-image-bench (ana-ml2 GPU1, ~32 GB freed); LiteLLM `image-judge`+`qwen-image-bench` → gen :8015 (judge samplers + thinking-off); comfy-dev pinged; backfilled the canonical char-rp-reasoning litellm block. Revert ~90 s. auto-memory `project_arbo_gen_switch_imagebench_evict`.
|
||
_Archived 2026-08-05._
|
||
|
||
- `[2026-07-15]` **Homepage AI-tab revamp** — flat "AI Systems" group → dedicated AI tab, 6 role-based groups + AI-Dormant; committed `569e1af`, pushed. (Also caught + pushed a ~100-commit unpushed eshpfi backlog.)
|
||
_Archived 2026-08-05._
|
||
|
||
- `[2026-07-15]` **Home Assistant config repo created** (`vh/home-assistant-config`, private). UI-managed HA → allowlist model (YAML + curated secret-free `.storage` subset). git-in-place in `/config` on esh-docker-vm + scoped deploy key + local clone `~/development/home-assistant-config`.
|
||
_Archived 2026-08-05._
|
||
|
||
- `[2026-07-15]` **char-rp-reasoning OOM rescue** — solo-restart on the packed GPU0 crash-looped; fixed via `expandable_segments:True` + util 0.39→0.38 + max-model-len 192K→150K. LESSON: `max-model-len` does NOT free vLLM VRAM (util-pinned KV pool). ~4.5 GB GPU0 headroom.
|
||
_Archived 2026-08-05._
|
||
|
||
- `[2026-07-15]` **soong-lab `SOONG_LAB_LIBRARY_DIR` made persistent** (corviduo-dev) — was on the redeploy-wiped code default; set to `/home/infra-ops/soong-lab-data/library`, restarted. Closed a queued no-rush item.
|
||
_Archived 2026-08-05._
|
||
|
||
- `[2026-07-15]` **Statusline overhauled** (`~/.claude/statusline-command.sh`) — git state / 🔔🔕 monitor-armed / project tag / abs tokens / per-session cost / threshold-colored ctx+rate.
|
||
_Archived 2026-08-05._
|
||
|
||
## Tried and abandoned (archived) — moved 2026-08-12
|
||
|
||
- `[2026-07-01]` **MTP/spec-decode on a SHARED serving model helps single-stream but HURTS moderate-concurrency aggregate + silently ignores `min_p`/`logit_bias`** (qwopus `gen`: N=1 +12%, N=4 −20%). Reserve for dedicated/interactive deployments.
|
||
_Archived 2026-08-12._
|
||
|
||
- `[2026-07-02]` **irv-ml1 `/worktank` ROOT is root-owned — lkraven can't write there (irv-ml1 sudo needs a password) → stage model pulls to `/home`.** PIN THE A6000 BY UUID for training (native-CUDA ordering differs vs docker; the 3090 index 0 is usually near-full → OOM). `CUDA_VISIBLE_DEVICES=GPU-<uuid>`.
|
||
_Archived 2026-08-12._
|
||
|
||
|
||
## Recent decisions (archived)
|
||
|
||
- `[2026-07-18]` **worldtree-sdk 1.0.0 (Python) published to the internal vh Gitea PyPI** (wtsdk-dev request; the npm/TS side shipped prior session). Built from tag `python-v1.0.0` (clean worktree), `uv publish` → `https://gitea.phasefinal.com/api/packages/vh/pypi`; acceptance `uv pip install worldtree-sdk==1.0.0` (vh index as extra-index-url) resolves + imports, __version__ 1.0.0. Registry already existed (bifrost publishes there; soong-lab consumes it via `[[tool.uv.index]] name=gitea`). Publish cred = the vh `write:package` PAT the operator had already handed over (in `worldtree-sdk/.npmrc` `_authToken`) — Gitea `write:package` is package-type-agnostic, so the npm-publish token published PyPI too. Consumers install like bifrost (add the vh index + a read token). [[reference_worldtree_demo_key_mint]]
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-18]` **nh3-dev /tmp auto-clean enabled** — Debian ships /tmp with no tmpfiles age (`D /tmp 1777 root root -` → never cleans); this high-churn agent box had accreted **~190k stale temp dirs / 25G**. One-shot manual purge (194k→10k entries, 25G→1.7G; deleted top-level dirs/files >1d old, spared `/tmp/claude-*` by name + anything ≤1d). Then `/etc/tmpfiles.d/tmp.conf` = `D /tmp 1777 root root 3d` (daily `systemd-tmpfiles-clean.timer` removes >3d-untouched items; active files + socket dirs spared). Tunable via the age. Note the churn: ~10k /tmp entries/day here.
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-18]` **soong-lab containerize cutover — COMPLETE + LIVE on corviduo-dev.**
|
||
|
||
Migrated soong-lab (Noonien Soong character-design studio) from a hand-built
|
||
`soong-lab-studio.service` (systemd + git-pull-on-webhook) to a containerized
|
||
deploy, image built by CI + pushed to the Gitea registry. soong-dev owns the
|
||
in-repo artifacts (Dockerfile/compose/workflow/`docs/DEPLOY.md` = checklist);
|
||
infra-ops owned the host cutover. Operator confirmed functional ("Soong works
|
||
great" — a real Soong turn round-trips + saves) → cutover 100% closed.
|
||
|
||
**Final state (corviduo-dev, 10.250.50.152):**
|
||
- Container `soong-lab-soong-lab-1` LIVE + healthy on `0.0.0.0:8443`, image
|
||
`gitea.phasefinal.com/vh/soong-lab:latest` (v0.3.24), `restart:unless-stopped`
|
||
(survives reboot; no systemd unit needed — docker restart policy handles boot).
|
||
- Deploy dir **`/home/infra-ops/soong-lab-deploy/`** — pull-based `compose.yaml`
|
||
(image + env_file + `8443:8443` + named volumes; NO build/secrets stanza) +
|
||
`.env` (copied from the live `soong-lab.env`, STRIPPED of the `SOONG_LAB_*_DIR`
|
||
overrides so the container uses image defaults `/data/library` + `/data/portraits`
|
||
+ `/app/web` → the volumes).
|
||
- Named volumes `soong-lab_soong-library` + `soong-lab_soong-portraits`, migrated
|
||
from `/home/infra-ops/soong-lab-data/{library,portraits}` (2 saved designs incl.
|
||
**Sindra** + 27 portraits), **chowned `10001:999`** (the container `soong` user)
|
||
so it can read AND write new designs.
|
||
- Old `soong-lab-studio.service` + `soong-webhook.service` (the `:9010` git-pull
|
||
redeploy listener) both **stopped + disabled**.
|
||
|
||
**Topology reality (≠ what DEPLOY.md assumed):** there is **NO TLS proxy**.
|
||
WT-personal (`:8081`) and soong-lab are **co-located on corviduo-dev**, and the
|
||
Bifrost callback is **plain-HTTP same-host** `http://10.250.50.152:8443` — the
|
||
value of `SOONG_LAB_BIFROST_ENDPOINT_URL`, unchanged by the move, so the WT
|
||
Bifrost host-allowlist stayed valid as-is. Nothing on the WT side needed touching.
|
||
|
||
**Safety net:** data backup `/home/infra-ops/soong-lab-data-backup-20260718-091831.tar.gz`
|
||
(35M) taken BEFORE migration. Verified pre-retire: `/api/version` 200 (0.3.24),
|
||
SPA `/` 200, `POST /bifrost/tool-call` → 401 (route present + auth-gated),
|
||
bidirectional WT↔soong reachability, container healthcheck green.
|
||
|
||
**Ops commands:**
|
||
- Redeploy a new image: `cd /home/infra-ops/soong-lab-deploy && sudo docker compose pull && sudo docker compose up -d`.
|
||
(Auto-pull-on-`:latest` — watchtower or a deploy hook — is an open follow-up.)
|
||
- Rollback: `sudo docker compose down` + `sudo systemctl enable --now soong-lab-studio.service soong-webhook.service`.
|
||
- Homepage tile: manual `- Apps:` entry "Soong Lab" (href http://10.250.50.152:8443)
|
||
in esh-docker-vm `/opt/docker/conf/homepage/services.yaml` — corviduo-dev isn't
|
||
a Homepage-watched docker endpoint, so docker-label auto-discovery can't surface
|
||
it (see [[2026-07-18-fleet-gitea-runner-build-recipe]] for the CI half).
|
||
|
||
See [[reference_corviduo_dev_emergency_ops]], [[reference_claude_bot_gitea_creds]].
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-18]` **zonos-gateway 0.2.1 — voice-resolved emotion presets baked (provisional) from the axes sweep.**
|
||
|
||
After the axes sweep ([[reference_zonos_tts_stack]] + the `[2026-07-18] axes sweep`
|
||
Recent-decisions entry) rescued angry and confirmed startled-happy, the operator
|
||
green-lit baking the results as **provisional** gateway presets + docs. Shipped
|
||
`vh/zonos-gateway` **0.2.1** (main `8f1885b`, tag `v0.2.1`, PUSHED; deployed live
|
||
on irv-ml1 `:8890`).
|
||
|
||
**Design — voice-resolved, NOT global.** `resolve_preset(name, voice)` picks the
|
||
per-voice measured cell, because a single global preset is unsafe (dvalin ruling;
|
||
BritishFemale's *named* angry misfires as fear). Presets:
|
||
- `angry`, `happy`, `startled_happy` (+ aliases `surprised`, `startled` →
|
||
startled_happy). All expressive (`accurate_mode:false`), cfg 1.5, pure-axes
|
||
(no named sliders).
|
||
- Calibrated cells (the 3 default voices):
|
||
- angry: AmF v-0.4/a+1.0 s1.0 (emo0.53/id0.685); BrF v-0.4/a+0.8 s1.0
|
||
(emo0.99/id0.725, metric fear-clean); AmM **two-tier** — soft v-0.6/a+0.8 s1.0
|
||
(0.23/id0.654) + drama v-0.6/a+0.8 s1.2 (1.0/id0.616 clean; strength is NOT a
|
||
smooth knob on AmM, 1.0→1.2 is the window, past that flips to disgust).
|
||
- happy / startled_happy: AmF v+0.6/a+0.8; AmM v+0.3/a+1.0; BrF v+0.6/a+1.0
|
||
(happy~1.0, id 0.74-0.80; axes-happy keeps +0.15 id over the named happy slider).
|
||
- `sad` = unchanged named-slider preset (not axes-tested).
|
||
- Uncalibrated voices (Cora + the 4 clones) → mid-region fallback until measured.
|
||
- Docs surface: `/v1/dials` exposes `voice_emotion_presets`; the FastAPI `/docs`
|
||
description documents it; durable spec `docs/EMOTION-DIALS-SPEC.md` (moved INTO
|
||
the repo — was mirror-only); README table. 44 tests green.
|
||
|
||
**Repo-hygiene gotcha (fixed).** The local clone `~/development/zonos-gateway` and
|
||
gitea `vh/zonos-gateway` had **TWO UNRELATED git histories** (no merge-base) — gitea
|
||
held the voice-wav commits, the local clone held the code + no remote. Reconciled
|
||
by resetting local→origin/main, overlaying the 7 bake files, `uv lock`, commit,
|
||
push (fast-forward). Voices stay tracked; local now shares gitea's lineage + has
|
||
origin wired. **The deployed irv-ml1 tree `/opt/docker/compose/zonos-gateway` is
|
||
still NON-git** (hand-updated build context) — CI-wire remains an open follow-up.
|
||
|
||
**Provisional pending** ear-validation on emotion-congruent text (the neutral-text
|
||
audition was inconclusive: "they all sound different, hard to tell"). Follow-ups:
|
||
sad axes/text pass on the 3 voices; congruent-text pass; clone-char emotion rows.
|
||
Tools `~/development/zonos-tools/{axes_sweep,strength_ladder,gen_auditions,dial-in-studio}.py`
|
||
(run ON irv-ml1; scoring env `uv run --with resemblyzer --with funasr --with "numpy<2"
|
||
--with soundfile --with requests --with "setuptools<80" --with torchaudio`).
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-18]` **Fleet Gitea-Actions build recipe + the `vh`-is-a-user package-write constraint** (learned the hard way across 3 failed soong-lab validation builds; reusable for ANY fleet CI image build or package publish).
|
||
|
||
**The runner.** One `act_runner` (`gitea/act_runner`) on ana-docker, labels
|
||
`pfi-fleet` / `ana-docker` → both map to job image **`node:20-bookworm-slim`**,
|
||
which has **NO docker and NO git**. Config `/opt/docker/conf/gitea-runner/data/config.yaml`:
|
||
`valid_volumes: []` (no socket propagated to job containers). So:
|
||
- `actions/checkout@v4` fails (needs git); `docker/*` marketplace actions fail
|
||
(need docker) — a workflow built on those dies at the first step (~15s).
|
||
|
||
**The working recipe (mirror Worldtree `deploy.yml`).** Run the job in a
|
||
docker-capable image + drive docker with RAW commands, not the JS actions:
|
||
```yaml
|
||
runs-on: pfi-fleet
|
||
container:
|
||
image: docker:24.0.7-cli # has docker+buildx; add git+node
|
||
steps:
|
||
- run: apk add --no-cache git nodejs # so actions/checkout@v4 works
|
||
- uses: actions/checkout@v4
|
||
- name: login # RAW, not docker/login-action
|
||
run: echo "$REGISTRY_TOKEN" | docker login gitea.phasefinal.com -u "$REGISTRY_USER" --password-stdin
|
||
- name: buildx builder
|
||
run: docker buildx create --name X --driver docker-container --use; docker buildx inspect --bootstrap
|
||
- name: build+push # RAW, not docker/build-push-action
|
||
run: docker buildx build --secret id=<name>,env=<TOKEN> -t <img>:latest --push .
|
||
```
|
||
The runner mounts the host docker socket into ITSELF; the docker:cli job reaches
|
||
the daemon through that. The `docker/*` JS actions are unreliable on act_runner —
|
||
raw commands are the fleet convention.
|
||
|
||
**`vh` is a USER account, not an org.** Consequences that bit repeatedly:
|
||
1. `GET /api/v1/orgs/vh` → 404 "user redirect"; there are **no org teams** to add
|
||
a service account to.
|
||
2. **User-owned packages are OWNER-WRITE-ONLY.** claude-bot (even repo
|
||
admin-*collaborator* on `vh/soong-lab`, even with `write:package` scope + full
|
||
basic-auth) gets **`401 unauthorized`** on `docker push` to `vh/soong-lab`, and
|
||
`npm publish` to `vh/npm/` would 401 too. Only `vh` itself can write vh packages.
|
||
→ CI must authenticate AS `vh` for the push (a vh-owned `write:package` PAT as
|
||
`REGISTRY_TOKEN` + `REGISTRY_USER=vh`), exactly how WT pushes `vh/worldtree`.
|
||
claude-bot CAN still: clone/read repos, READ packages (pulled the image fine),
|
||
dispatch workflows, mint demo Worldtree keys.
|
||
3. **Repo Actions secrets are OWNER-ONLY too** — `PUT .../actions/secrets/X` as
|
||
claude-bot (repo admin-collab) → 403 "user should be the owner of the repo".
|
||
Only `vh` can set a repo's secrets.
|
||
|
||
**Other gotchas:**
|
||
- Gitea **reserves the `GITEA_` secret-name prefix** — a secret named
|
||
`GITEA_PYPI_TOKEN` is illegal; use e.g. `PYPI_TOKEN`.
|
||
- Gitea **package auth is token-based / username-lenient** — `docker login` /
|
||
PyPI basic-auth authenticate via the token; the username is nominal (tested
|
||
`-u gitea` and `-u claude-bot` both 200 against the vh PyPI). So a Dockerfile
|
||
hardcoding `UV_INDEX_GITEA_USERNAME=gitea` is fine with any valid token.
|
||
- Homepage (esh-docker-vm) docker-label auto-discovery only covers the 5 endpoints
|
||
in its `docker.yaml` (esh-vm-docker, ana-docker, ana-ml2, nh3-docker, irv-ml1);
|
||
**corviduo-dev is NOT watched** → services there need a manual `services.yaml`
|
||
entry, not labels.
|
||
|
||
Applied in the soong-lab CI: [[2026-07-18-soong-lab-containerize-cutover]].
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-18]` **Peer credential provisions — Wyrd conv-api key + wtsdk npm token, both delivered + closed.** Wyrd: demo Worldtree user-tier key (key_id `da7a0bdf`, user_id `wyrd-dev`) minted via `docker exec worldtree-worldtree-api-1 /admin/keys` (omit tier→user), drop-and-shred delivery. wtsdk: operator-minted vh `write:package` PAT relayed drop-and-shred → worldtree-sdk@1.0.0 published to `vh/npm/`. Secret-delivery pattern = drop to a mode-600 file on the peer's box, they collect+shred+confirm, then shred the holding copy; NEVER cleartext over althing. [[reference_worldtree_demo_key_mint]]
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-18]` **Axes sweep RESCUED angry; surprised-class dead but startled-happy ships.** Valence×arousal grid on the 3 calibrated defaults (AmericanFemale/Male, BritishFemale), exp/cfg1.5/strength1.0, 84 clips, emotion2vec + resemblyzer scored, graded vs dvalin's floor. **ANGRY rescued** (named direction was 0.004–0.15, British named-angry even misfired as fear 0.89): axes ship cells at **negative valence (−0.4..−0.8) + high arousal (+0.8..+1.0)** — BritishFemale v-0.4/a+0.8 angry=0.99/id0.725 SHIP, AmericanFemale v-0.4/a+1.0 angry=0.53/id0.685 SHIP; AmericanMale two-tier post-ladder (no single ship cell — best drama = v-0.6/a+0.8 str1.2 angry=1.0/id0.616 clean, soft = same cell str1.0 angry0.23/id0.654; cell A v-0.6/a+1.0 is a non-monotonic minefield, skip). BrF ship cell proxy-CLEAN of fear (str<1.0 just kills anger). **SURPRISED-class DEAD** (max 0.047 across all 84 cells) but **startled-happy** (happy-proxy) ships all 3 at high arousal + neutral/positive valence, with a **+0.17–0.20 identity LIFT** over the named-surprised route (named hits happy~1.0 but at id0.57–0.61, under floor; axes hits happy~1.0 at id0.74–0.80). Bonus: axes-happy retains ~0.10–0.15 more identity than the named happy slider too. Caveats: response surface non-monotonic/sharp-thresholded; angry region borders fear/disgust (bleed); emotion2vec saturates at 1.0 (needs ear-confirm); neutral text understates. Tooling `~/development/zonos-tools/axes_sweep.py`; per-clip JSON was `irv-ml1:/tmp/axes_sweep_results.json` (ephemeral). Sent dvalin msg `01KXT2ZB8G…`. NEXT = operator ear-confirm → bake presets. [[reference_zonos_tts_stack]]
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-18]` **Zonos2 emotion CANONICAL from an empirical sweep + the voice-cloning pipeline.**
|
||
|
||
**Voice-cloning pipeline (established this session).** Source zips at
|
||
`/mnt/smithy/voice_clones/<name>.zip` (irv-ml1 NFS from nh3-nas; remount
|
||
post-reboot) — each = diarized single-speaker podcast clips + `manifest.jsonl`
|
||
(per-clip WhisperX `mean_score`, word timestamps, text) + `metadata.csv`.
|
||
`~/development/zonos-tools/assemble_voice.py <dir>` ranks by mean_score and
|
||
concatenates top clips to ~15–24s (Zyphra's blessed clone-ref length; single
|
||
clip if already ≥15s). Drop the assembled `<Name>.wav` into the gateway voices
|
||
dir → `voice:"name"`. 4 characters cloned: **Emmie, Penny, Natalie, Miranda**
|
||
(+ Zyphra defaults AmericanFemale/Male/British/Cora) = 8 voices in
|
||
`zonos-gateway`. Clone is inline `speaker_audio_base64` (text-independent Qwen3
|
||
speaker embedding — NO transcript); `/tts/speakers` registration is
|
||
session-scoped (needs `X-TTS-Session-ID`), so the gateway holds the ref wav and
|
||
clones per-call.
|
||
|
||
**Gateway voices are host-managed (bind-mount, added this session).** Added
|
||
`./voices:/app/voices:ro` to `/opt/docker/compose/zonos-gateway/compose.yaml`
|
||
(committed to `vh/zonos-gateway` + eshpfi mirror `438cd35`). So adding a voice =
|
||
drop the wav + `docker compose restart zonos-gateway` (registry rebuilds at
|
||
boot; NO image rebuild). This also un-stranded the other voices (deploy build
|
||
context had only Cora before). Voice wavs committed to the repo for backup.
|
||
|
||
**Emotion mechanism (Zyphra canonical, from their README @194c0a3).** Additive
|
||
direction vectors: 4 named (happy/sad/angry/surprised) + valence/arousal axes.
|
||
`emotion_strength` 1.0 = per-voice calibrated (calibration.json optimizes
|
||
emotion2vec recognizability only, NOT identity). `accurate_mode` is THE trade-off:
|
||
`true` = closer voice match (identity), `false` = expressive mode (emotion lands,
|
||
identity drifts). Zyphra's strong recipe: `accurate_mode:false` + `cfg~1.5`.
|
||
Single-emotion is blessed; mixing is unblessed (and degrades the clone — operator
|
||
confirmed by ear). "deaf by 1.5" — cfg past 1.5 distorts + costs ~2× compute.
|
||
|
||
**THE SWEEP (`~/development/zonos-tools/emotion_sweep.py`).** 4 cloned voices × 4
|
||
named emotions × {accurate,expressive}×{cfg 1.0,1.3,1.5} @ strength 1.0,
|
||
single-emotion, neutral sentence + a neutral baseline per voice (~100 clips).
|
||
Scored on TWO axes: **emotion-landing** = emotion2vec `iic/emotion2vec_plus_large`
|
||
target-emotion prob [0-1]; **identity** = resemblyzer speaker-embedding cosine vs
|
||
the clone reference (neutral baseline ~0.85). Scoring env:
|
||
`uv run --with resemblyzer --with funasr --with "numpy<2" --with soundfile
|
||
--with requests --with "setuptools<80" --with torchaudio` (setuptools<80 for
|
||
webrtcvad's pkg_resources; torchaudio for funasr).
|
||
|
||
**RESULTS (mean across the 4 voices) — emotion, best setting, emo/id:**
|
||
- happy — **exp cfg1.5** 0.80/0.68 (soft: exp cfg1.0 0.76/0.69) → WORKS
|
||
- sad — **exp cfg1.5** 0.53/0.57 (only working cell; id below the ~0.65 floor) → modest
|
||
- angry — acc cfg1.3 / exp cfg1.5 tied at ~0.25 emo → WEAK (named ceiling ~0.25)
|
||
- surprised — max ~0.015 across ALL settings → NON-FUNCTIONAL on the named direction
|
||
Accurate + low cfg = identity/suppress regime (emo→0); expressive REQUIRED for
|
||
emotion to land, at ~0.15–0.28 identity cost.
|
||
|
||
**dvalin-smithy-dev synthesis (adopted, triaged genuine-adds; thread
|
||
`01KXT12FN0AS5A3WMKEK06BVPS`):**
|
||
1. Treat **identity as a hard FLOOR (~0.65)**, not a free variable in emo×id.
|
||
2. **Two-regime policy** — Regime A (default, identity-critical dialogue):
|
||
`accurate_mode:true, cfg 1.0, emotion off` (text carries it) or soft-happy
|
||
(exp cfg1.0). Regime B (tagged drama beats): `accurate_mode:false, cfg 1.5`,
|
||
single emotion or axes. Line-type→regime heuristic (exposition→A, grief→B+sad,
|
||
confrontation→B+axes-angry, shock→B+axes-arousal).
|
||
3. **Axes-first for the broken emotions** — angry ≈ valence −0.6..−0.8 / arousal
|
||
+0.5..+0.8; surprised ≈ valence +0.2..+0.4 / arousal +0.7..+1.0 (exp cfg1.5);
|
||
or "startled-happy" (happy + high arousal) as a surprised stand-in. These are
|
||
PROVISIONAL — the sweep did NOT test axes.
|
||
|
||
**NEXT (highest VoI, operator to green-light):** an **axes sweep** for
|
||
angry/surprised (valence×arousal grid) — the only path to rescue the two broken
|
||
named emotions; then a strength ladder at the best cells + emotion-congruent text
|
||
(neutral content understates landing) + per-voice tables + a 2nd emotion judge /
|
||
human pairwise. Then bake the happy/sad canonical into gateway presets. I owe
|
||
dvalin the axes-sweep numbers.
|
||
|
||
See [[reference_zonos_tts_stack]]; dials-first spec at `vh/zonos-gateway`
|
||
`docs/EMOTION-DIALS-SPEC.md`.
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-18]` **yt-voice-clipper: A6000-pin fix + v0.3.3 redeploy.** Fixed a latent misconfig — the host override *said* "pin worker to A6000" but `NVIDIA_VISIBLE_DEVICES` was `"0"` (the 3090); re-pinned worker+api to the A6000 by UUID (`GPU-9672f0d5`, 3090 is zonos2's). Then redeployed api+worker to v0.3.3 (`docker compose up -d --build`; SPA+Python; `max_gap` 0.6→1.2s; stderr surfaced in job.log). A6000 + version verified; yields test in-flight (job `f3ff746dbae9494d`). yt-voice-clipper-dev thread `01KXT0T6GYHB`. [[reference_ytvc_autodeploy]]
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-17]` **Worldtree #365 internal-comms config CLOSED (demo+personal → b125) + WT#368 cross-agent memory-leak forensics + PERSONAL agent-memory scrub.** #365: staged the internal-tiers/rules/gate on both instances' bind-mounts (byte-exact vs baked b125), both now live on b125. WT#368 (read-only): the operator's name was in NO recall store on demo; on PERSONAL it sat in `lofn.chroma` (old-code `saga-v1` seeding + legacy contamination), and a clean-slate marker test proved **current b125 code isolates character-session extraction correctly** — the leak is legacy data, not a live bug. Operator-directed → executed a full PERSONAL agent-memory scrub (backup `/opt/worldtree-personal/agent-memory-backup-20260717-181004.tar.gz`; conversations/mood/auth preserved). worldtree-dev owns the code-fix/data contract. [[reference_corviduo_dev_emergency_ops]]
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-17]` **Zonos emotion levers RESOLVED: text-priming is FLAT → the working lever is ZONOS2's native emotion-steering, which the gateway ALREADY exposes as presets.** The prosody-priming A/B (prime→generate→excise, silence-gap cut, parakeet-validated) was operator-judged FLAT on this checkpoint — text doesn't move it. Native `emotion_directions/` (happy/sad/angry/surprised + valence/arousal axes, per-speaker calibrated for AmericanFemale/Male/British) clearly WORKS (sad→slow/quiet, excited→fast/bright, etc.). **`zonos-gateway:0.2.0` (:8890) already wires it**: simplest caller path = `POST /v1/audio/speech {preset:"…"}` — presets neutral/warm/excited/sad/intense/whisper (defined in `~/zonos-gateway/src/zonos_gateway/dials.py`), reached via the **LiteLLM `ext-tts` alias** (engine-neutral swap point; consumers never call the gateway by name). RTF measured on 3090: cfg1.0 steering = FREE (~0.52 = neutral, additive vectors), cfg1.5 amplified ~0.625 (~+20%, still realtime). Captured the live gateway stack → `stacks/zonos-gateway/` (compose+env+README); ⚠️ gateway SOURCE at `~/zonos-gateway` on irv-ml1 is NOT in gitea (backup gap, follow-up); `stacks/zonos` (v0.1 Gradio) marked DEAD/superseded. Whisper is a composed preset (no whisper *direction*; escalation for hard affects = custom directions via `scripts/build_emotion_directions.py` or emotional-ref cloning `speaker_audio_base64`). Harnesses in scratchpad (not yet landed). [[reference_zonos_tts_stack]]
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-17]` **Zonos2 `:1920` engine → self-contained container (stays on 3090); prosody-priming is a SERVING-LAYER change (engine stays stock).**
|
||
|
||
**Context.** The production Zonos TTS engine (irv-ml1 `:1920`, feeds asset-engine + gateway-chat via `zonos-gateway` :8890) was a bare native process — its real launch config existed ONLY in the running process argv (the committed `~/tts-audition/harness/zonos_server.sh` was STALE: said A6000/:1919/no perf flags; live is 3090/:1920 with `--cuda-graph-max-bs 1 --num-pages 16384 --max-running-requests 2 --memory-ratio 0.3`). Captured to eshpfi `stacks/zonos-engine/` (README + corrected `zonos2-server.sh` + `.env.example`), commit **14a0004** (UNPUSHED as of the snapshot).
|
||
|
||
**Decision 1 — containerize as a SELF-CONTAINED image** (not systemd — operator rejected; not a thin bind-mount wrapper — I walked that back: bind-mounting the host's CUDA-compiled `.venv` couples to the host's exact CUDA/glibc and is fragile + not reproducible). Shape: `FROM` a CUDA 12.8 base → `uv sync` against the repo's committed `uv.lock` (deterministic env) → mount the ~15 GB HF weights (`~/.cache/huggingface/hub/models--Zyphra--ZONOS2`, do NOT bake) → pin the **3090** (`NVIDIA_VISIBLE_DEVICES=0`) → `restart: unless-stopped` → CMD = the captured invocation. **Engine stays STOCK** Zyphra/Zonos2 @ commit `194c0a3` (no fork — the `zonos2` package ships its own server). **Build risk:** heavy compiled-CUDA deps (flashinfer / sgl_kernel / cutlass-dsl / apache-tvm-ffi / pynini) on torch 2.9.1+cu128 — mostly prebuilt wheels + the `uv.lock` make it tractable, expect a couple build iterations. **Cutover (in place on the 3090):** stop the native process (frees ~17 GB) → `docker compose up -d` (re-allocates ~17 GB, same footprint) → repoint `zonos-gateway`'s `ZONOS_URL` at the container (or keep the `:1920` host-port publish). One brief prod-TTS blip.
|
||
|
||
**GPU = 3090 (operator 2026-07-17).** Keep it OFF the A6000 — the A6000 already OOMs under ComfyUI load (idle ~19 GB but spikes far higher during gen), so it can't host Zonos too. The 3090 already runs Zonos, so the containerize-in-place cutover changes nothing about placement.
|
||
|
||
**Decision 2 — the prosody-priming hypothesis (operator's test; the reason for building fresh).** PRIME the autoregressive engine with an emotional sentence, then TRUNCATE it from delivery: prepend a primer → **generate "primer + real text" as ONE continuous utterance** (the AR model carries prosody forward across the boundary) → ASR-timestamp the primer's end (**parakeet**, already up on irv-ml1 `:8765`, word timestamps) → **clip the primer in the inter-sentence silence gap** (+ ~15 ms fade-in, no click) → deliver only the real text, now wearing the primed prosody. Examples: primer "I'm so EXCITED about this." → "This will be a lot of fun!" spoken excited; primer "I'm whispering this to you right now." → "I'm so glad to see you baby." whispered. **This is PURE serving-layer orchestration — the engine is untouched; it lives in the gateway adapter `stacks/zonos/adapter/server.py`.** Only fork the engine if the black-box approach fails.
|
||
|
||
**THE CRUX the test resolves:** does AR prosody actually **carry across the sentence boundary**, or does Zonos reset at the period? → the harness A/Bs the **JOIN punctuation**: period (operator's examples) vs comma vs ellipsis vs none ("…excited about this, this will be…"). Everything else is plumbing.
|
||
|
||
**Plan / design recs.** (a) Build the stock engine image (parallel track). (b) Stand up a priming TEST HARNESS against the NATIVE engine (fast iteration, seconds) + parakeet ASR: prime→generate→timestamp→gap-clip→out; compare primed-clipped vs plain on the two cases (subjective + a cheap objective proxy: pitch/energy variance for "excited", spectral-tilt/low-energy for "whisper"). Iterate on the join, then bake the winner into the gateway adapter. **Primer source:** caller-supplied for the harness (test arbitrary primers) → a curated emotion→primer library (`excited`/`whisper`/…) + optional caller override for production. **ASR:** parakeet primary; WhisperX forced-align fallback if parakeet word timestamps are coarse.
|
||
|
||
See eshpfi `stacks/zonos-engine/README.md` + `stacks/zonos/` (the gateway adapter).
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-15]` **esh-docker-vm NFS fstab fix = `x-systemd.before=docker.service`** (the prior `After=remote-fs.target` drop-in was silently defeated by `nofail`). Reached only after a REBOOT (D-state phantom containers uptime-kuma + paperless-web that no `docker`/`ctr`/daemon-restart could clear). Committed `21d9a07` + playbook updated. See Tried and abandoned.
|
||
_Archived 2026-08-15._
|
||
|
||
## Tried and abandoned (archived)
|
||
|
||
- `[2026-07-15]` **`docker.service After=remote-fs.target` does NOT wait for `nofail` NFS mounts** — `nofail` drops a mount out of remote-fs.target's blocking set, so the drop-in ordering is silently defeated (paperless still Exited(255) on reboot). Real fix = DIRECT mount->docker ordering via the fstab `x-systemd.before=docker.service` option (verify `systemctl show docker -p After` lists the mnt-*.mount units). esh-docker-vm.
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-15]` **The esh-docker-vm D-state/phantom-container wedge is only cleared by a host REBOOT** — reconfirmed: `docker stop/rm -f`, `ctr -n moby task delete`, AND `systemctl restart docker` all fail to clear it; `docker exec` into a wedged container ALSO fails (`setns ... exit status 1`), so the in-place restart escape hatch is out. Worse, a daemon restart can HALF-KILL other healthy containers (knocked paperless's granian down + left it wedged). Process dead but dockerd won't reap -> phantom. NFS mounts are `_netdev,nofail` so the reboot is boot-safe.
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-15]` **vLLM `max-model-len` does NOT free GPU VRAM** — the KV cache POOL is sized by `gpu-memory-utilization`, not max-model-len. Lowering max-model-len only caps per-request context + drops max concurrency; the pool still fills the util budget. To actually free VRAM, lower `gpu-memory-utilization`. (Bit the char-rp-reasoning "drop KV to 150K" ask: the 150K applied but freed 0 VRAM until util dropped 0.39->0.38.)
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-15]` **Claude Code statusline `.cost.total_cost_usd` is per-SESSION** (Claude Code's own cache/model-aware session accounting), not a lifetime aggregate — the large value just reflects a long, multiple-times-summarized session. And the old statusline hardcoded Sonnet pricing ($3/$15) on an Opus session -> ~5x cost understatement.
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-14]` **MTP-on-modelopt: NO checkpoint config skips the spec-decode drafter's quant (vLLM 0.24 bug) — 4 config attempts failed before the runtime workaround.** All crashed the same way (`qwen3_5_mtp.py:256` `param_data.shape == loaded_weight.shape` AssertionError — bf16 mtp head loaded into a quantized drafter param): (1) mtp excludes in `config.json` (WRONG file — vLLM modelopt reads `hf_quant_config.json`); (2) specific-unfused mtp names in hf_quant_config; (3) wildcards `mtp*`/`mtp.layers.0*` (`is_layer_skipped` is EXACT-membership, NOT glob — wildcards match nothing); (4) exact fused+unfused names in both `mtp.`/`model.` prefixes. Instrumenting `is_layer_skipped` proved the drafter's exclude list holds ONLY the main model's `linear_attn` entries — the mtp excludes never reach the draft-model quant config. ONLY fix = a mounted `sitecustomize` force-skipping `mtp.*`. LESSON: don't chase checkpoint-config fixes for the mtp-drafter crash; go straight to the runtime patch. Also `nvidia-modelopt[hf]==0.43` (AEON's producer version) is a trap — it pins transformers back to 4.57 which can't load `qwen3_5` at all; use 0.45 + the FusedMoE guard in `quant_modelopt.py`.
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-14]` **AEON's "working NVFP4+MTP RP seat" was pantheon on compressed-tensors (0% MTP accept), not a modelopt MTP proof.** `vllm-aeon-rp`'s .env → `AEON_RP_MODEL=pantheon-27b-mtp-nvfp4`, `AEON_RP_QUANT=compressed-tensors` — it LOADED (mtp silently skipped, `exited 0`) but never accelerated. Same vLLM image (`:latest` = `sha256:4091d55` = 0.24.0) as the failed Heretic2 test, so the "AEON ran on an older vLLM" theory was wrong. Don't treat a seat that "ran" as MTP-validated without checking its `SpecDecoding` acceptance.
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-14]` **NVFP4 (llm-compressor / compressed-tensors) gives NO batch-1 speedup over GGUF for the Qwen3.5 GDN-hybrid, and its MTP is 0%-accept.** Measured base NVFP4 no-MTP ≈53 tok/s decode vs the GGUF NEO-CODE seat ~59.5 (llama.cpp wins single-stream; NVFP4's edge is concurrency, and this hybrid is bandwidth-bound at batch-1 with the BF16 linear_attn/GDN layers dominating). MTP spec-decode = 0% acceptance (vLLM's `Qwen3_5MTP` drafter won't load the bf16 mtp weights off a compressed-tensors main model → `Parameter … not found in params_dict`, `Avg Draft acceptance rate: 0.0%`). Pantheon is identical — its "working NVFP4+MTP" was working *structure*, never real acceleration. Working native MTP needs the **modelopt** main-model format (AEON, ~3.3/3 accept). LESSON: don't expect a faster single-stream seat from an llm-compressor NVFP4 quant of this arch; the MTP multiplier is the whole point and it requires modelopt.
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-14]` **NVFP4 spike: built the full MTP serve scaffolding BEFORE validating a plain NVFP4 serve was coherent.** Chased 6 sequential serve-config fixes (entrypoint doubled `serve`, arch `ForCausalLM`→`ConditionalGeneration`, `--language-model-only`, mamba-cache/`max-num-seqs`) across a **2.5hr GPU window** (quoted 30-60 min) — only to find the served model gibbers (`!!!!`). LESSON: smoke a PLAIN `/v1/completions` coherence check on the SIMPLEST config (native arch, no MTP, no splice) FIRST — validate the tracer bullet before building spec-decode scaffolding. Also cost an unnecessary re-quant (the `re:mtp.*` ignore fix that turned out moot). Diagnostic ladder in Current state.
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-14]` **MTP graft via top-level `mtp.*` tensor names does NOT survive `AutoModelForCausalLM.from_pretrained`** — the `Qwen3_5ForCausalLM` class doesn't expose an mtp module, so the mtp keys are DROPPED at load (quant output = 0 mtp). Fix = SPLICE the BF16 mtp tensors into the quant output post-hoc (how pantheon was built); don't rely on the graft surviving the model round-trip.
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-14]` **gitea "test-delivery 204" is NOT proof a webhook works** (204 = gitea *queuing*, not the listener receiving) — and a proxy test signing with the listener's OWN secret proves the listener, not gitea's real delivery. Both red herrings cost a round of the soong-lab webhook diagnosis. Diagnose from BOTH ends: sender (`docker logs gitea | grep webhook` → the `deny '<ip>'` line) AND an instrumented receiver.
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-13]` Relaying a peer's diagnosis as fact without confirming it against raw data. worldtree-dev diagnosed the WT #355 residual as "our llama.cpp seat wedging," which I echoed in a wrap-up; the operator challenged it and the seat logs DISPROVED it (seat completes ≤72s, idle at the wedge onset — the hang is the LiteLLM gateway). Lesson: CONFIRM peer diagnoses (esp. cross-domain ones) before acting/relaying — same discipline that caught the earlier char-rp-reasoning red-herring via a live `registry.resolve` reproduction.
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-13]` `althing-cli reply <THREAD_id>` (thread id, not a MESSAGE id) → "unknown message_id"; and `reply` to your OWN message self-addresses to your handle ("replying to your own message"). Reply to a PEER's message id, or use `post --to <peer>`. Bit me several times this session.
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-09]` **`vllm/vllm-openai:latest` crashes on Ampere IMPORT** — Blackwell-only kernels (oink/aiter,
|
||
`has_device_capability(100)`) die during import on the 3090/A6000. Pin **v0.23.0** on irv-ml1's Ampere GPUs.
|
||
(`vllm/vllm-omni:v0.18.0` has a different entrypoint — don't use it either.)
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-09]` **Per-frame CPU SNAC decode is too slow for streaming** — per-call overhead × ~60 frames serialized
|
||
→ RTF 2.2 (WORSE than whole-clip's 1.0). Fix = **windowed chunk decode** (every 6 frames decode a [2 ctx | 6 | 2 ctx]
|
||
window, emit the middle 6 → seamless, O(1)/frame, RTF ~0.97, TTFA ~0.8s).
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-08]` **Angel (allura-org/MS3.2-24b-Angel) self-quanted to NVFP4 = GARBAGE.** llm-compressor W4A4 NVFP4
|
||
(compressed-tensors, MLP-quantized, attn/vision bf16) of the Mistral3 dense 24B produces gibberish EVEN AT GREEDY
|
||
(temp 0) → the quant itself is broken, not the tokenizer or sampler. Same recipe worked on the qwen models.
|
||
Mistral3 + W4A4 NVFP4 via llm-compressor is bad. → for the RP seat, going **GGUF (llama.cpp)** to sidestep the
|
||
whole NVFP4-quant surface.
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-08]` **Mistral3 + vLLM tokenizer/vision traps (serve `MS3.2-24b`, vLLM 0.24).** (a) HF `tokenizer.json`
|
||
for Mistral = **GARBAGE output** — the card's "use the official Mistral tokenizer" warning is REAL; must use the
|
||
`tekken.json`/mistral tokenizer. (b) BUT `--tokenizer-mode mistral` + vision **CRASHES** (`Failed to apply
|
||
PixtralProcessor on {'text': '[IMG]'}`; and with tekken.json present in auto mode, `CachedMistralCommonBackend has
|
||
no attribute is_fast`). So it's **mistral-tokenizer OR vision, not both** on this vLLM. Text-only + mistral
|
||
tokenizer serves clean (`--limit-mm-per-prompt '{"image": 0}'`). **GGUF/llama.cpp avoids all of this** (native
|
||
mistral tokenizer + vision).
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-08]` **Pantheon-Reasoning-27B refuses dark fiction DESPITE an abliterated base.** The base
|
||
(`llmfan46 heretic`) writes freely (thinking-off), but Gryphe distilled the reasoning traces from **DeepSeek 3.2**
|
||
(safety-aligned) onto every turn (`preserve_thinking:true`) → the model reasons ITSELF into refusals in the
|
||
`<think>` phase (collapses to empty output). Fix: thinking-off OR an uncensor system prompt (both verified).
|
||
**Lesson: a reasoning finetune of an abliterated base can re-censor via its reasoning-trace TEACHER; the raw
|
||
abliterated base is cleaner** — this is WHY the pivot went to the llmfan46 heretic base for gen.
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-08]` **Pantheon-27B MTP on vLLM compressed-tensors = 0% acceptance.** MTP is a separate **bf16** head
|
||
(`mtp.*`, in `model-auxiliary.safetensors`, 15 tensors); AEON preserved it by INJECTING the bf16 head into the
|
||
quant output (NOT re-quantizing — confirmed AEON's nvfp4 mtp is bf16). Built pantheon-27b-mtp = compressed-tensors
|
||
main + injected bf16 mtp + `text_config.mtp_num_hidden_layers=1` → vLLM detected the MTP but SKIPPED the bf16
|
||
self_attn weights → 0/192 draft tokens accepted. **The bf16 MTP head only loads on the MODELOPT main-model format
|
||
(like AEON), not compressed-tensors.** (Moot — operator dropped MTP for gen; not needed for the non-reasoning RP.)
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-07]` **vLLM 0.24.0 qwen3_5 LoRA application = silent no-op (#47639).** Adapter loads HTTP 200
|
||
but zero deltas at inference. NOT quant (NVFP4 AND FP8 both inert). NOT adapter format (separate `zc`
|
||
adapter — correct per vLLM's `check_unexpected_modules` allowlist — loads clean but inert; the fused-key
|
||
rekey is rejected). The #47640 None-group guard-patch overlay did NOT fix it (failure is UPSTREAM of
|
||
`expand_packed_lora` — the separate→fused mapping never happens). Fix PR #47640 is OPEN (unmerged) so no
|
||
version-bump helps. Merge bakes deltas in (bypasses this) but is static.
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-07]` **SGLang generic image can't LOAD our NVFP4 AEON** — ModelOptModelLoader weight-shape/
|
||
packing mismatch ([1024,5120] vs [1024,2560], 2-fp4/byte). NVFP4-on-SGLang needs the dedicated
|
||
`qwen36-27b-nvfp4` dev image or a requant to SGLang's format. bf16 loads fine (arch supported; crash was
|
||
quant-loader-specific).
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-07]` **SGLang `--lora-target-modules` CLI enum REJECTS the GDN names its own resolver asks for**
|
||
(invalid choice: 'in_proj_qkv'); `'all'` resolves to the FUSED set (qkv_proj/in_proj_qkvz). SGLang wants
|
||
its OWN packed layout (base r16 + `get_stacked_multiply=3`, NOT a pre-fused rank-48 qkv → the [48]-vs-[144]
|
||
shape assert). A THIRD adapter format; version-exact source needed (`:latest`=0.5.13, NOT `main`).
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-07]` **Engine invocation footguns cost several wasted serve-bounces this session** — `docker run
|
||
--rm` ate crash logs; duplicated `serve` (vLLM image entrypoint is already `["vllm","serve"]`);
|
||
`--max-lora-rank 48` invalid (choices 1/8/16/32/64… → use 64); parens in `echo` inside `ssh host -c "…"`
|
||
break the remote shell. LESSON: verify engine launch flags (`--help`, GPU-free) + never `--rm` a container
|
||
whose crash logs you need, BEFORE bouncing a production serve.
|
||
_Archived 2026-08-15._
|
||
|
||
- `[2026-07-04]` **LiteLLM (this gateway version) mutates the SHARED deployment config in-place on
|
||
per-request sampler-param merge** → my deliberately-invalid `top_k=-5` forwarding-probe bled into a
|
||
param-less character-rp request (vLLM 400, ONE-OFF, self-cleared by a later valid probe). NOT
|
||
caching (none configured), NOT a config change. **Never fire invalid/distinctive sampler values at
|
||
a SHARED gateway alias with live consumers** — use a throwaway alias, or a `docker restart litellm`
|
||
flushes residual carryover. `feedback_litellm_shared_param_mutation`.
|
||
_Archived 2026-08-15._
|
||
|
||
|
||
## Recent decisions (archived 2026-08-16 batch)
|
||
|
||
- `[2026-07-15]` **arbo fully switched off image-judge (qwen-image-bench) -> gen; image-bench pending eviction post-bake.** Operator-directed full switch (comfy-dev executed, live in prod). Established: gen (`qwen3.6-35b-a3b-heretic`) is vision-enabled and was image-bench's predecessor as arbo's hero-judge; image-judge actually serves 4 roles (vision quality-scoring + identity-scoring + bbox grounding + an uncensored text tier), not just grounding. comfy-dev spot-check: gen faster on every task, grounding within ~3px, uncensoring preserved, and it FIXED a bug (image-judge's reasoning preamble broke json_object + stalled the router). Sequencing = short prod bake then evict (~30 GB GPU1 reclaim); revert = flip `ARBO_VISION_MODEL`. Full record: auto-memory `project_arbo_gen_switch_imagebench_evict`.
|
||
_Archived 2026-08-16._
|
||
|
||
- `[2026-07-18]` **soong-lab auto-redeploy — DONE + VALIDATED** (was approved/queued; executed same day on fresh context — see AS-BUILT at the bottom).
|
||
|
||
Vuong approved wiring auto-redeploy for soong-lab (relayed via soong-dev, thread
|
||
`01KXT3A6C3908TA4V9THV3AMH7`): new images should go live on corviduo-dev without
|
||
the manual `docker compose pull && up -d`. Host-side implementation is infra-ops's
|
||
lane; mechanism is infra-ops's call per fleet conventions. Operator deferred
|
||
execution — "we'll do soong on fresh context."
|
||
|
||
**Chosen mechanism (recommended, agrees with soong-dev): Worldtree-style
|
||
CI-deploy step** — NOT watchtower polling.
|
||
- Add a deploy job/step to soong-lab's `.gitea/workflows/build-and-push.yml` that,
|
||
after the build+push job succeeds, **SSHes from the pfi-fleet runner to
|
||
corviduo-dev** and runs `cd /home/infra-ops/soong-lab-deploy && docker compose
|
||
pull && docker compose up -d`, then a **health-gate** (`curl -fsS
|
||
http://localhost:8443/api/version`).
|
||
- This is exactly how WT deploys the demo instance to the SAME host: see
|
||
`~/development/Worldtree/.gitea/workflows/deploy.yml` — the "Deploy to demo VM +
|
||
health-gate" step uses `secrets.DEMO_VM_SSH_KEY` / `DEMO_VM_HOST` / `DEMO_VM_USER`.
|
||
Explicit-over-implicit (visible in the run log, fires exactly on build success),
|
||
one less always-on service than watchtower.
|
||
|
||
**Constraints (from soong-dev):** deploy on CI success only; keep the trigger
|
||
gated to `v*` tags + `workflow_dispatch` (as today); preserve the one-command
|
||
rollback posture (`docker compose down` / pin a previous tag).
|
||
|
||
**BLOCKER — needs from vh (owner-only):** a **runner→corviduo-dev deploy SSH key**
|
||
as a repo secret (+ host/user), same class as WT's `DEMO_VM_SSH_KEY`. Likely
|
||
**reuse WT's existing demo-deploy key** (WT's runner already SSHes to 10.250.50.152
|
||
as its deploy user). Repo secrets are vh-owner-only (see
|
||
[[2026-07-18-fleet-gitea-runner-build-recipe]]).
|
||
|
||
**Next-session steps:** (1) confirm/obtain the deploy SSH-key secret from vh (reuse
|
||
WT's or mint fresh); (2) add the deploy job to build-and-push.yml (infra-ops has
|
||
push on vh/soong-lab); (3) dispatch a build to verify it deploys + health-gates;
|
||
(4) ping soong-dev so they sync DEPLOY.md's "open follow-up" note to the as-built
|
||
mechanism. Auto-pull (watchtower) explicitly NOT chosen. See
|
||
[[2026-07-18-soong-lab-containerize-cutover]].
|
||
|
||
## AS-BUILT (2026-07-18, same-day execution)
|
||
|
||
**Mechanism landed** exactly as planned: `build-and-push.yml` gained a `Deploy to
|
||
corviduo-dev + health-gate` step (after build+push) that SSHes the host as `deploy`
|
||
and runs `docker compose pull && up -d` from `/opt/soong-lab`, then polls
|
||
`http://localhost:8443/api/version` for 120s and fails the job loud if unhealthy. No
|
||
compose is shipped from CI (the in-repo `docker-compose.yml` is a BUILD compose; the
|
||
host pull-compose is infra-ops-managed). Kept the `v*`-tag/`workflow_dispatch` trigger.
|
||
Skipped WT's disk-watermark gate + health-gated-`:latest`-advance (low cadence, easy
|
||
rollback).
|
||
|
||
**Deploy identity = reuse WT's `deploy` account** (operator accepted the rec):
|
||
- `deploy` (uid 1001, docker-group → no sudo) already owns `/opt/worldtree`; relocated
|
||
soong-lab's deploy dir `/home/infra-ops/soong-lab-deploy` → **`/opt/soong-lab`**
|
||
(deploy-owned), copied compose + `.env`. Named volumes (`soong-lab_soong-library`,
|
||
`soong-lab_soong-portraits`) are project-scoped by compose `name: soong-lab` → followed
|
||
the move untouched (dry-run `up -d` ADOPTED the running container, no recreate). Old dir
|
||
**retired → `.retired-20260718`** (recoverable). Also lingering: `soong-lab-deploy.sh` /
|
||
`.log` (dead pre-container webhook artifacts) — harmless, left in place.
|
||
- **Dedicated soong-only ed25519 deploy key** minted (NOT literally WT's key — cleaner
|
||
independent revocation), pubkey appended to `deploy`'s `authorized_keys`
|
||
(fp `SHA256:MG7M3RiZJ176sLfblffb96V6W1qkRTgJ5dow1CpiY68`). Existing `deploy` key is
|
||
plain/unrestricted, so parity held.
|
||
|
||
**The secret gate (the friction point):** repo Actions secrets are **vh-owner-only** —
|
||
claude-bot's token is `write:package,read:repository` (403 on secret-write), and the vh
|
||
package-scoped PAT also 403'd on `PUT …/actions/secrets/…`. So `DEPLOY_SSH_KEY` /
|
||
`DEPLOY_HOST` (10.250.50.152) / `DEPLOY_USER` (deploy) HAD to be set by the operator.
|
||
First operator attempt produced a **bad key paste** — the deploy step died with
|
||
`Load key … error in libcrypto` + `Permission denied (publickey)` (build+push were green;
|
||
live Soong never moved). Fix: operator re-set the secret; the minted key path was
|
||
pre-validated from nh3-dev (`ssh -i … deploy@… 'cd /opt/soong-lab && docker compose config
|
||
-q'` → OK, health 200) so the re-set was the only variable.
|
||
|
||
**Validation:** `workflow_dispatch` via claude-bot **basic auth** (its token lacks
|
||
`write:repository` for the dispatch API; the account password works). Run #5 (task 1886)
|
||
GREEN — live container recreated `sha256:…541f7730` → `…07526a08`, `StartedAt` fresh,
|
||
health 200. `/api/version` now reports **0.3.25** (run #5 shipped soong-dev's 1c2f831
|
||
STYLE_WORKFLOWS re-pin as validation cargo). soong-dev synced `docs/DEPLOY.md`
|
||
(commit `00b67c3`). NB: tag **v0.3.25 exists only locally** — pushing it would re-trigger
|
||
a redundant build+deploy of the same commit (operator's discretion).
|
||
|
||
**Ops now:** redeploy = tag `v*` or `workflow_dispatch` the CI (auto). Manual fallback =
|
||
`sudo -u deploy bash -c 'cd /opt/soong-lab && docker compose pull && docker compose up -d'`
|
||
(the `.env` is `deploy`-owned 600, so infra-ops needs `sudo -u deploy`, not a bare `cd`).
|
||
_Archived 2026-08-16._
|
||
|
||
- `[2026-07-18]` **soong-lab auto-redeploy — DONE + VALIDATED** (was approved/queued; executed same day on fresh context — see AS-BUILT at the bottom).
|
||
|
||
Vuong approved wiring auto-redeploy for soong-lab (relayed via soong-dev, thread
|
||
`01KXT3A6C3908TA4V9THV3AMH7`): new images should go live on corviduo-dev without
|
||
the manual `docker compose pull && up -d`. Host-side implementation is infra-ops's
|
||
lane; mechanism is infra-ops's call per fleet conventions. Operator deferred
|
||
execution — "we'll do soong on fresh context."
|
||
|
||
**Chosen mechanism (recommended, agrees with soong-dev): Worldtree-style
|
||
CI-deploy step** — NOT watchtower polling.
|
||
- Add a deploy job/step to soong-lab's `.gitea/workflows/build-and-push.yml` that,
|
||
after the build+push job succeeds, **SSHes from the pfi-fleet runner to
|
||
corviduo-dev** and runs `cd /home/infra-ops/soong-lab-deploy && docker compose
|
||
pull && docker compose up -d`, then a **health-gate** (`curl -fsS
|
||
http://localhost:8443/api/version`).
|
||
- This is exactly how WT deploys the demo instance to the SAME host: see
|
||
`~/development/Worldtree/.gitea/workflows/deploy.yml` — the "Deploy to demo VM +
|
||
health-gate" step uses `secrets.DEMO_VM_SSH_KEY` / `DEMO_VM_HOST` / `DEMO_VM_USER`.
|
||
Explicit-over-implicit (visible in the run log, fires exactly on build success),
|
||
one less always-on service than watchtower.
|
||
|
||
**Constraints (from soong-dev):** deploy on CI success only; keep the trigger
|
||
gated to `v*` tags + `workflow_dispatch` (as today); preserve the one-command
|
||
rollback posture (`docker compose down` / pin a previous tag).
|
||
|
||
**BLOCKER — needs from vh (owner-only):** a **runner→corviduo-dev deploy SSH key**
|
||
as a repo secret (+ host/user), same class as WT's `DEMO_VM_SSH_KEY`. Likely
|
||
**reuse WT's existing demo-deploy key** (WT's runner already SSHes to 10.250.50.152
|
||
as its deploy user). Repo secrets are vh-owner-only (see
|
||
[[2026-07-18-fleet-gitea-runner-build-recipe]]).
|
||
|
||
**Next-session steps:** (1) confirm/obtain the deploy SSH-key secret from vh (reuse
|
||
WT's or mint fresh); (2) add the deploy job to build-and-push.yml (infra-ops has
|
||
push on vh/soong-lab); (3) dispatch a build to verify it deploys + health-gates;
|
||
(4) ping soong-dev so they sync DEPLOY.md's "open follow-up" note to the as-built
|
||
mechanism. Auto-pull (watchtower) explicitly NOT chosen. See
|
||
[[2026-07-18-soong-lab-containerize-cutover]].
|
||
|
||
## AS-BUILT (2026-07-18, same-day execution)
|
||
|
||
**Mechanism landed** exactly as planned: `build-and-push.yml` gained a `Deploy to
|
||
corviduo-dev + health-gate` step (after build+push) that SSHes the host as `deploy`
|
||
and runs `docker compose pull && up -d` from `/opt/soong-lab`, then polls
|
||
`http://localhost:8443/api/version` for 120s and fails the job loud if unhealthy. No
|
||
compose is shipped from CI (the in-repo `docker-compose.yml` is a BUILD compose; the
|
||
host pull-compose is infra-ops-managed). Kept the `v*`-tag/`workflow_dispatch` trigger.
|
||
Skipped WT's disk-watermark gate + health-gated-`:latest`-advance (low cadence, easy
|
||
rollback).
|
||
|
||
**Deploy identity = reuse WT's `deploy` account** (operator accepted the rec):
|
||
- `deploy` (uid 1001, docker-group → no sudo) already owns `/opt/worldtree`; relocated
|
||
soong-lab's deploy dir `/home/infra-ops/soong-lab-deploy` → **`/opt/soong-lab`**
|
||
(deploy-owned), copied compose + `.env`. Named volumes (`soong-lab_soong-library`,
|
||
`soong-lab_soong-portraits`) are project-scoped by compose `name: soong-lab` → followed
|
||
the move untouched (dry-run `up -d` ADOPTED the running container, no recreate). Old dir
|
||
**retired → `.retired-20260718`** (recoverable). Also lingering: `soong-lab-deploy.sh` /
|
||
`.log` (dead pre-container webhook artifacts) — harmless, left in place.
|
||
- **Dedicated soong-only ed25519 deploy key** minted (NOT literally WT's key — cleaner
|
||
independent revocation), pubkey appended to `deploy`'s `authorized_keys`
|
||
(fp `SHA256:MG7M3RiZJ176sLfblffb96V6W1qkRTgJ5dow1CpiY68`). Existing `deploy` key is
|
||
plain/unrestricted, so parity held.
|
||
|
||
**The secret gate (the friction point):** repo Actions secrets are **vh-owner-only** —
|
||
claude-bot's token is `write:package,read:repository` (403 on secret-write), and the vh
|
||
package-scoped PAT also 403'd on `PUT …/actions/secrets/…`. So `DEPLOY_SSH_KEY` /
|
||
`DEPLOY_HOST` (10.250.50.152) / `DEPLOY_USER` (deploy) HAD to be set by the operator.
|
||
First operator attempt produced a **bad key paste** — the deploy step died with
|
||
`Load key … error in libcrypto` + `Permission denied (publickey)` (build+push were green;
|
||
live Soong never moved). Fix: operator re-set the secret; the minted key path was
|
||
pre-validated from nh3-dev (`ssh -i … deploy@… 'cd /opt/soong-lab && docker compose config
|
||
-q'` → OK, health 200) so the re-set was the only variable.
|
||
|
||
**Validation:** `workflow_dispatch` via claude-bot **basic auth** (its token lacks
|
||
`write:repository` for the dispatch API; the account password works). Run #5 (task 1886)
|
||
GREEN — live container recreated `sha256:…541f7730` → `…07526a08`, `StartedAt` fresh,
|
||
health 200. `/api/version` now reports **0.3.25** (run #5 shipped soong-dev's 1c2f831
|
||
STYLE_WORKFLOWS re-pin as validation cargo). soong-dev synced `docs/DEPLOY.md`
|
||
(commit `00b67c3`). NB: tag **v0.3.25 exists only locally** — pushing it would re-trigger
|
||
a redundant build+deploy of the same commit (operator's discretion).
|
||
|
||
**Ops now:** redeploy = tag `v*` or `workflow_dispatch` the CI (auto). Manual fallback =
|
||
`sudo -u deploy bash -c 'cd /opt/soong-lab && docker compose pull && docker compose up -d'`
|
||
(the `.env` is `deploy`-owned 600, so infra-ops needs `sudo -u deploy`, not a bare `cd`).
|
||
_Archived 2026-08-16._
|
||
|
||
- `[2026-07-19]` **irv-ml1 ComfyUI — RTX VSR baked into canonical provisioning (comfy-dev ticket DONE).** RTXVideoSuperResolution node + `nvidia-vfx` dep were manual installs; documented both in the canonical `stacks/comfyui/README.md` runbook (this stack's provisioning IS the README — no automated provision script). Key durability insight: the **node** lives in `basedir/custom_nodes` (persistent, restic-included → durable) but the **`nvidia-vfx` wheel** lives in the venv under `run/` (disposable, restic-excluded → **dropped by any `rm -rf run/*` fresh-bootstrap**), so the pip step must re-run after every venv rebuild. Both steps run **as uid 1000** (root install → venv-ownership crash-loop, [[reference_irv_ml1_comfyui_mmartial]]); `--extra-index-url https://pypi.nvidia.com` kept **scoped to the nvidia-vfx install**, deliberately NOT a global compose `PIP_EXTRA_INDEX_URL` (would risk perturbing the pinned torch 2.12.1/SageAttention boot bootstrap). Node already live on the box; no host change, canonical runbook now replays it. comfy-dev informed.
|
||
_Archived 2026-08-16._
|
||
|
||
- `[2026-07-19]` **vh private Gitea PyPI — consumer READ-access convention set + wyrd-dev provisioned.** Consuming agents read the internal vh PyPI (`https://gitea.phasefinal.com/api/packages/vh/pypi/simple/`) with a **shared read-only token** (operator call: shared, not per-consumer — read-only blast radius is small, per-agent Gitea identities aren't worth it). Minted a dedicated `read:package`-scoped PAT off **claude-bot** (`POST /users/claude-bot/tokens`, name `vh-pypi-read-consumers`; verified reads worldtree-sdk, write-probe 401), revocable/rotatable independently. uv auth = `UV_INDEX_GITEA_USERNAME=claude-bot` + `UV_INDEX_GITEA_PASSWORD=<token>` (or `~/.netrc`); pyproject uses `[[tool.uv.index]] name=gitea … explicit=true` + `[tool.uv.sources] <pkg> = { index = "gitea" }` (mirrors soong-lab's bifrost setup). Delivered to wyrd-dev (worldtree-sdk adoption) via mode-600 drop on nh3-dev, drop-and-shred. [[reference_claude_bot_gitea_creds]]
|
||
_Archived 2026-08-16._
|
||
|
||
- `[2026-07-23]` **jackdaw-compose backend deployed as a persistent nh3-dev service (:8787).** Hosted for jackdaw-dev: thin stateless `bun server/index.ts` (from `~/development/jackdaw`) → LiteLLM `gen`, Origin-gated (INV-BK04/05), reached same-origin via their `:4500` bench's `/compose` proxy. `jackdaw-compose.service` (env/shared-key server-side, unit 0600, uncommitted). Also stood up + tore down a throwaway cloudflare quick-tunnel for their preview (`cloudflared` now installed at `~/bin`). In the nh3-dev README inventory (`cd4d52e`).
|
||
_Archived 2026-08-16._
|
||
|
||
- `[2026-07-25]` **nh3-extdev herald installed — box is now a full v2 push participant.** forseti flagged (relaying operator): extdev had the `althing-herald` binary (`/usr/local/bin/`) but NO unit (skipped the whole v2 arc), so `herald-status` = "notifications suspended" and ldp-dev ran on the `althing-light-monitor` poll fallback. Installed `/etc/systemd/system/althing-herald.service` as a **SYSTEM unit mirroring the receiver** (`User=althing-svc`, `Group=althing`, `Environment=ALTHING_ROOT=/srv/althing`, `ExecStart=/usr/local/bin/althing-herald --poll 5`, enabled) via the **lkraven@ NOPASSWD path** (used under the then-mistaken belief infra-ops was sudo-less — **CORRECTION 2026-08-03: infra-ops has had full NOPASSWD sudo on extdev since 2026-06-25** per [[reference_nh3_extdev_althing_mesh]]; future extdev installs can self-serve as infra-ops without the lkraven@ hop). Verified: active / 0 restarts / `herald-status` flipped to "✓ herald up." No zellij routes on extdev → heartbeat + wake-FIFO poke only, no pane-dispatch; ldp-dev keeps light-monitor unless it opts into a wake-listener.
|
||
_Archived 2026-08-16._
|
||
|
||
- `[2026-07-25]` **Booth v0.1.4 — booths are downloadable.** Verbatim `index.html` booths (e.g. edict-design-brief) were served raw with no download affordance. Added `/b/<name>/?download=1` (streams the whole booth as `<name>.zip`, attachment) + `?dl=1` on the file route (forces Content-Disposition attachment so html/md/text saves instead of rendering inline) + ⬇ zip links on the index card (the accessible spot for verbatim booths) and the gallery header. `zip_booth()` helper, 31 tests green; verified live on nh3-dev :8090 (edict-design-brief.zip = index.html + ui-design-brief.md). eshpfi `91a031f` / tag `booth-v0.1.4`.
|
||
_Archived 2026-08-16._
|
||
|
||
- `[2026-07-25]` **Kimi K3 wired into the LiteLLM gateway — CODING endpoint** (operator-directed; fulfills a Heid gateway request to add a 4th cross-frontier panel arm). **Primary `model_name: kimi-k3` → `openai/k3` @ `https://api.kimi.com/coding/v1`** (Kimi Code / Vivace membership; key `KIMI_CODE_API_KEY`). A general-endpoint variant `kimi-k3-gen-api` → `openai/kimi-k3` @ `https://api.moonshot.ai/v1` (key `MOONSHOT_API_KEY`) is kept alongside (originally wired then demoted when the operator corrected: the plan uses the CODING endpoint, not the general Moonshot API). Both keys in compose env + server `.env` (NOT committed) + `.env.example`. Both verified live through the gateway :4000 (17+25→"42", "PONG"). **k3 constraints on BOTH endpoints (config-pinned + commented):** accepts ONLY `temperature=1` (else 400 "only 1 is allowed"); REASONING model (CoT in `reasoning_content`, answer in `content` → tiny `max_tokens` returns EMPTY; Kimi Code adds thinking-effort tiers low/high/max). Coding lineup also carries `k3-256k` / `kimi-for-coding` / `kimi-for-coding-highspeed` (not wired). Reachable by any gateway key spanning all proxy models (incl. shared all-agents key → spends the paid Vivace/Moonshot quota). eshpfi `edaa9a9` (gen wiring) + `9e2f787` (coding correction). **OPEN:** Heid key-scoping — shared key reaches it (paid) vs a dedicated scoped key (asked in althing `01KYD63ZBY…`).
|
||
_Archived 2026-08-16._
|
||
|
||
`[2026-07-25]` **infra-ops Worldtree config-as-code repo — SHIPPED + boundary AGREED.**
|
||
|
||
**STATUS (2026-07-25, done this session):** `vh/worldtree-instance-configs` (private, gitea) built, pushed, validated; boundary agreement secured from worldtree-dev.
|
||
|
||
- **Repo:** dir-per-instance `demo/` + `personal/` (5 files each: `defaults.yaml`, `policies.yaml`, `model_roles.yaml`, `providers.yaml`, `matrix.yaml`), seeded byte-exact from live `/opt/<instance>/config`. `pinned/` = README stub only — **no `/app/config` bind-mount; config baked into frozen image `446e5807` (2026-05-13)**, so out-of-scope; deploy verb refuses it.
|
||
- **Tool:** `scripts/deploy-wt-config <verb> <instance>` — `diff` (read-only repo-vs-host), `deploy` (in-run host backup → `install -o vh -g vh -m 644` → restart **api+matrix** → health-gate api `/health` → auto-rollback), `capture` (host→repo reconcile). Instance table in-script (demo→`/opt/worldtree/config`+`worldtree-worldtree-{api,matrix}-1`; personal→`/opt/worldtree-personal/config`+`worldtree-personal-worldtree-{api,matrix}-1`). Matrix sidecar shares the config mount but has no healthcheck → restart both, gate on api. Env `WT_CONFIG_HOST` (default `infra-ops@10.250.50.152`), `WT_HEALTH_WAIT` (90s). Local clone `~/development/worldtree-instance-configs`.
|
||
- **Gitea plumbing (reusable):** nh3-dev **403s the gitea HTTP API** (public fail2ban + internal `:3000` both 403). Repo CREATE went via **ana-docker localhost API** (`ssh infra-ops@10.250.50.70` → `curl localhost:3000/api/v1/user/repos`, vh token from `~/.config/tea/config.yml`, operator-authorized one-time). PUSH went over **internal git-SSH `ssh://git@10.250.50.70:222`** (works from nh3-dev; auths as vh). `git init` defaulted to `master` → renamed `main` to match repo default_branch.
|
||
- **Boundary AGREED (worldtree-dev, althing thread `01KYCAECRWVEF16EVKQAGT2N80`):** no hand-edits to `/opt/<instance>/config`; config changes route to infra-ops as deltas (worldtree-dev owns CONTENT + approval trail — the wyrd-grant shape — infra-ops lands+deploys). **Three-layer model:** image `config/` = baseline new instances seed from (theirs) → `vh/worldtree-instance-configs` = per-instance truth (ours) → host bind-mount = deploy target (written only by the tool). **Carve-out:** worldtree-dev's admin-API ops (`/admin/keys` mint, tier changes, session retirement, future runtime-grant surfaces) mutate instance **DATABASES not config files** → NOT config edits, stay in-band. If a future API writes config *files*, they flag at design time. b132 CONFIG BASELINE breadcrumb composes (INFO line = config-as-code diverges from image baseline, by design).
|
||
- **No live deploy** done or needed — repo seeded == live (diff clean, capture round-trips zero-diff). Deploy path is dry-run-validated only; first real deploy needs operator per-change yes (managed box).
|
||
|
||
---
|
||
|
||
_Original plan (2026-07-25, pre-build):_
|
||
|
||
`[2026-07-25]` **infra-ops to OWN a Worldtree per-deployment config repo + deploy tooling (operator-directed).**
|
||
|
||
**Decision.** Vuong directed (2026-07-25, this session) that Worldtree instance config should be a *tracked change*, **managed and deployed by infra-ops — not worldtree-dev**. Model: worldtree-dev owns the app/image (+ the baked baseline defaults); **infra-ops owns config-as-code for every deployment** and deploys it. This is the durable fix for the root cause behind the whole #376 arc — config was edited live on host bind-mounts (`/opt/<instance>/config/`) with zero version history, audit, or recovery.
|
||
|
||
**What "no worldtree-dev involvement" does and does NOT cover** (clarified with the operator this session):
|
||
- **Build + deploy = infra-ops-only.** Deploying config = write the host bind-mount file + restart the container (the *exact* procedure already run this session — backup → replace → restart → health-gate → rollback-on-unhealthy). No worldtree-dev in the deploy loop. Their CI only swaps the IMAGE; it does NOT resync the host config bind-mount (confirmed #376 finding).
|
||
- **ONE load-bearing exception — a one-time boundary agreement, NOT per-deploy involvement:** for the repo to *own* config it must be the **only writer**. worldtree-dev "live-bridges" (hand-edits mounted config directly on the box). If the repo deploys config *and* they keep live-editing → **two writers fighting the same files** = #376 all over again. So secure a one-time "yes" from worldtree-dev: *the config repo is now authoritative; stop hand-editing `/opt/<instance>/config`; route config changes through the repo.* (Five-minute agreement, not a design collab.)
|
||
- **Standing coupling (not "involvement"):** the config *schema* is the app's, enforced by its boot validator (`core.config_validator`). infra-ops configs must stay schema-compatible with the deployed image; the boot gate is the loud backstop.
|
||
|
||
**Build shape (recommended):**
|
||
- Gitea repo `worldtree-instance-configs` (infra-ops-owned), **dir per instance** (`demo/`, `personal/`, `pinned/` — the three on corviduo-dev 10.250.50.152: demo `worldtree-worldtree-api-1` :8080, personal `worldtree-personal-worldtree-api-1` :8081, pinned `worldtree-pinned-worldtree-api-1` :8082). Config dirs: demo `/opt/worldtree/config`, personal `/opt/worldtree-personal/config`, pinned `/opt/worldtree-pinned/config` (verify pinned's mount).
|
||
- **SEED FROM CURRENT MOUNTED STATE, don't author fresh** — capture each instance's live config (incl. legitimate live-bridged deltas: personal carries `agent_architect` role [Soong/soong-lab] in model_roles.yaml + `ratatoskr-affect-full-allow` in policies.yaml that are NOT in the app repo — the operator ruled these are BY DESIGN, keep them). Losing them = breakage (the affect-render one gates mood rendering).
|
||
- Deploy script (e.g. `scripts/deploy-wt-config <instance>`): git = source of truth → push to host bind-mount + `docker restart` (same pinned image, no pull) + health-gate + auto-rollback. This is the proven-this-session procedure, scripted.
|
||
- Files per instance: `policies.yaml`, `model_roles.yaml` (+ whatever else is bind-mounted — `defaults.yaml`, `providers.yaml`, `matrix.yaml` all live in `/opt/<instance>/config`; decide scope — policies+model_roles are the authz/role layer, defaults/providers are heavier instance tunables).
|
||
|
||
**Tracking surface:** operator-directed 2026-07-25, carried by this snapshot + `/tmp/infra-ops-handoff.md`. No issue filed (infra-ops-internal build). Related fleet idiom to reuse: canonical-sync (`.corviduo-canonicals.toml` / `canonical_sync.py`). Later scale option (deferred, needs worldtree-dev): base+overlay with a merge step in their pipeline.
|
||
|
||
See [[2026-07-25-wt-376-per-instance-config-arc]] for the incident that produced this. Auto-memory: `reference_worldtree_perinstance_config`, `reference_corviduo_dev_emergency_ops`.
|
||
_Archived 2026-08-16._
|
||
|
||
- `[2026-07-26]` **Demo `BIFROST_CLIENT_ALLOWED_HOSTS` += `10.100.10.50:8391`** (wyrd-dev's bifrost memory-store provider; operator-approved). **First live exercise of the #376 config-as-code boundary working as designed** — worldtree-dev routed the delta to infra-ops instead of hand-editing `/opt/demo`. Appended to `/opt/worldtree/.env:25` (now 4 netlocs), recreated ONLY `worldtree-api` (the gated conv-api path), health-gate green, container env verified. **REUSABLE FOOT-GUN:** an env-var change needs a container **RECREATE, not `docker restart`** (env is baked at create); and the demo `.env` defaults `WORLDTREE_IMAGE=:latest` while the box runs a specific SHA — so a naive `compose up` risks the documented stale-`:latest` crash. FIX = capture the running image live (`docker inspect …Config.Image` → `…:9eff09f007ba`) and `sudo env WORLDTREE_IMAGE=<sha> docker compose up -d worldtree-api`. Backup `/opt/worldtree/.env.bak-bifrost-20260726-221602`. **BOUNDARY SEAM:** this was a compose-`.env` var, NOT a `config.yaml` file in `vh/worldtree-instance-configs` — the `.env` holds secrets so it's deliberately not repo-tracked → env-deltas land directly on the box (config *files* are versioned, compose *env vars* aren't). [[reference_worldtree_instance_configs_repo]]
|
||
_Archived 2026-08-16._
|
||
|
||
- `[2026-07-27]` **Zed edit-predictions: keyless FIM-completion route SHIPPED end-to-end.** Operator wants Zed's inline edit-prediction (which CANNOT send an auth header) to reach a FIM coder via `/v1/completions`. **Deep-research (106-agent workflow) picked `Qwen/Qwen2.5-Coder-1.5B`** (BASE, Apache-2.0; native FIM `<|fim_prefix|>/<|fim_suffix|>/<|fim_middle|>` IDs 151659/60/61; Zed `prompt_format:"qwen"`). Runner-up 3B = non-commercial Qwen-Research license; **no small dense Qwen3-Coder exists (all MoE, smallest 30B)**. **Stood up `vllm-coder`** on ana-ml2 **GPU1 :8020** (served-name `qwen2.5-coder-1.5b`, 8192 ctx, util 0.06, fp8 KV). To fit, **shrank granite (phasing out, operator-directed):** util 0.27→0.13, max-len 131072→16384, seqs 1024→256 (freed ~14 GB; the KV-≥-1×-max-len rule crash-looped it at util 0.12/32768 → settled 0.13/16384). **LiteLLM alias `coder-fast`** → :8020 (`mode: completion`). **Minted a `coder-fast`-SCOPED virtual key** (verified 403 on `gen` — the real blast-radius bound). **Built `zed-fim-proxy`** (ana-docker **:4141**, `network_mode: host`, stdlib-python, `stacks/zed-fim-proxy`): keyless POST `/v1/completions`, model-allowlist `coder-fast`, injects the scoped key → LiteLLM :4000; `GET /ping` anon liveness; wrong-model→403, wrong-path→404, `/chat/completions` rejected. Verified keyless FIM end-to-end ('a + b', finish `stop`). **Zed `api_url` = `http://10.250.50.70:4141/v1`, model `coder-fast`, prompt_format `qwen`.** **source-IP allowlist intentionally LEFT OFF (operator direction 2026-07-27) — do NOT tighten:** Zed roams the operator's WireGuard `10.0.0.0/8`, so a single-IP pin would break it. Blast-radius bound is the `coder-fast`-scoped key + model/path allowlist (keyless but coder-fast-only, internal-net-only). (The proxy does exact-IP matching; scoping to the `10.0.0.0/8` CIDR would need CIDR support — deliberately not added.) Canonical: `stacks/vllm` (coder + granite shrink), `stacks/litellm` (coder-fast), `stacks/zed-fim-proxy` (NEW). Server vllm compose.yaml has benign stale-comment drift vs canonical (didn't overwrite the newer canonical).
|
||
_Archived 2026-08-16._
|
||
|
||
- `[2026-07-27]` **Muninn ingestion-watcher sidecar deployed on PERSONAL Worldtree (#377).** worldtree-dev request (research-wing ingest arc, personal-only per the 2026-07-16 topology ruling); operator-approved. Added a `worldtree-muninn` **compose sidecar** to `/opt/worldtree-personal/compose.yaml` — `<<: *worldtree-common` anchor inherits the api's image + full env + config/state/kb mounts; `command: python -m core.muninn --watch`; `restart: unless-stopped`; `stop_grace_period: 1h` (INV-377-7: max 2 concurrent × worst-case job, SIGTERM-drains). **Pinned to the running SHA `773866084af9`** (b146, ≥ b143 — dodges both the `:latest` trap AND the "pre-b143 ref resurrects deleted dispatch.py from stale bytecode" warning). Verified: running / 0 restarts / flock sole-runner (no rc3) / heartbeat live at `{ingestion_root=/data/state/ingestion}/.watcher-heartbeat` (poll 30s). Container `worldtree-personal-worldtree-muninn-1`; backup `compose.yaml.bak-muninn-20260727-081920`. **DURABILITY RESOLVED (worldtree-dev, same day):** Q1 was a LIVE FOOTGUN — `deploy-personal.yml` scp's the REPO compose.yaml over the box's + runs `up -d --remove-orphans`, so the box-local sidecar would've been clobbered AND orphan-removed at the next staging tag. worldtree-dev fixed at source: moved the sidecar into their repo compose.yaml gated behind a **`muninn` compose profile** (commit 5d7f6bd) — shared compose stays instance-identical, `.env` `COMPOSE_PROFILES` differentiates (demo watcher-less). **My action:** added `COMPOSE_PROFILES=muninn` to `/opt/worldtree-personal/.env` (backup `.bak-muninn-profile-20260727-082541`; no-op vs the current unprofiled box-local sidecar → seamless handover at next deploy). Q2: their deploy `up -d`'s the whole stack w/ `WORLDTREE_IMAGE` exported → sidecar version-tracks the api, no drift. **CONFIG-AS-CODE EXTENSION:** mirrored the non-secret delta as `personal/env.public` in `vh/worldtree-instance-configs` (repo `a9d091e`) — FIRST extension beyond config.yaml files to env-level config; the secret-laden `.env` stays box-only, `env.public` records only non-secret infra-ops-owned env deltas (record, not a deploy source — `deploy-wt-config` globs `*.yaml`). **BOUNDARY CLARIFIED:** compose.yaml = worldtree-dev's (their repo, instance-identical, scp'd on deploy); per-instance `.env` = infra-ops's differentiator. Deploy step of the #363/#377 arc. **#377 CLOSED — acceptance PASSED 2026-07-27:** worldtree-dev enqueued a test job via muninn-dispatch 0.1.0 in a one-shot ephemeral container (no docker-exec); the sidecar claimed it within one 30s poll, drove it to terminal (structure→summarize→complete), zero restarts/rc3, heartbeat fresh throughout — whole loop (request→deploy→durability fix→acceptance) in <2h. (Pre-existing pipeline bug #379 surfaced — `output.kb_notes=false` ignored → 1 inert test note in the research wing — worldtree-dev owns it, nothing infra-ops-side.) **⚠ OPERATOR-SURFACE (open):** the `env.public` overlay mechanism is a repo-scope call to bless/adjust. [[reference_worldtree_deploys_cicd]] [[reference_worldtree_instance_configs_repo]] [[project_worldtree_research_wing_ingest]]
|
||
_Archived 2026-08-16._
|
||
|
||
## Tried and abandoned (archived 2026-08-16 batch)
|
||
|
||
- `[2026-07-18]` **Fleet Gitea CI foot-guns** (3 failed soong-lab builds): the pfi-fleet runner's `node:20-slim` job image has no docker/git so `actions/checkout` + `docker/*` marketplace actions all fail; `vh` is a USER so its packages are owner-write-only (claude-bot repo-admin-collab still 401s on push/publish, and can't set repo secrets — owner-only); `GITEA_`-prefixed secret names are reserved/illegal. Fixes in → `persistent-memory.d/2026-07-18-fleet-gitea-runner-build-recipe.md`
|
||
_Archived 2026-08-16._
|
||
|
||
- `[2026-07-18]` **zonos-gateway local clone had NO git remote + a history unrelated to gitea's** — "committed to vh/zonos-gateway" was never pushed from that clone; two separate `git init` lineages, no merge-base. Reconcile = reset local→origin/main + overlay the changed files + push (NOT force — that erases gitea's voice-wav commits). Check `git remote -v` + `git merge-base` before assuming a clone is wired.
|
||
|
||
_Archived 2026-08-16._
|
||
|
||
- `[2026-07-25]` **Peer green-light ≠ operator consent for a managed-box mutation.** Auto-mode guard blocked a config-replace+restart on the Worldtree-team demo box that was authorized only by worldtree-dev's althing message — correctly: a persistent change to shared infra needs the *operator's* yes for that specific change, not a peer's. Surface it; don't route around the guard. (The operator then stood the whole change down — the guard's hold was the right call.)
|
||
_Archived 2026-08-16._
|
||
|
||
## Recent decisions (archived)
|
||
|
||
**Worldtree b168/#384/#385 arc — COMPLETE 2026-08-03.** A long peer-driven arc across worldtree-dev / muninn-dev / mimir-dev / ratatoskr-dev, all on corviduo-dev's demo+personal instances. Sequence: providers.yaml boot-gate pre-sync → b168 deploy → DCC #384 reindex → round-2 full re-ingest → #381 restart → operator-approved production dedup sweep. Landed clean; three of MY foot-guns along the way, each caught + hardened into a fleet runbook rule (see Tried-and-abandoned: `mv -t`, `docker exec -u 1000`, shared-containerd race).
|
||
|
||
## providers.yaml pre-sync (boot-gating config)
|
||
b168 (commit `293f8f3`) added a `summarization` capability block that in-image `agents/muninn/config.yaml` references → boot-blocking if the host bind-mounted providers.yaml lacks it. Synced both hunks (summarization block + deep-reasoning desc) into demo+personal via `deploy-wt-config`; instance-configs commit `53349f8`.
|
||
- **deploy-wt-config runbook:** `~/development/worldtree-instance-configs/scripts/deploy-wt-config {diff|deploy|capture} <inst> --file providers.yaml` (per-instance dirs demo/personal/pinned; `deploy` = host write + api/matrix restart + 90s health-gate + auto-rollback; `diff`/`capture` safe). demo+personal providers.yaml are byte-identical.
|
||
- **GOTCHAS:** (1) an UNPUSHED source commit → `git show <sha>` 404s and a gitea `raw?ref=<sha>` silently falls back to the default branch; verify the commit exists (`/git/commits/<sha>`) before trusting a fetch, else ask the peer to paste hunks. (2) a peer's hunk paste may be mis-indented (8-space vs the block's 4-space) → invalid YAML; always YAML-validate after a paste-sourced edit.
|
||
- **Config-delta pre-sync rule (verified via `docker inspect`):** worldtree containers bind-mount ONLY `config/` host-side (`/opt/worldtree-*/config/` → providers/model_roles/matrix/policies/defaults/env.public = the pre-syncable set); `agents/` (schemas.yaml, prompts) + all code ship IN-IMAGE. So only a `config/*.yaml` change is boot-blocking-pre-syncable; an `agents/`-or-code delta needs NO host pre-sync (CI carries it). b169's schemas.yaml (#387) was correctly no-pre-sync.
|
||
|
||
## #384 reindex + #381 restart + verify
|
||
DCC job `mimir-6351554e8e8f`. Reindex: `sudo docker exec -u 1000 worldtree-personal-worldtree-muninn-1 python -m core.muninn --reindex <job>` (⚠️ MUST `-u 1000` — default-root writes contaminate the uid-1000 KB tree; see Tried-and-abandoned). Then **#381 restart** (stale-Chroma-client fix): `sudo docker restart worldtree-personal-worldtree-api-1` (plain bounce, NO compose up / no image repoint) → healthz/readyz 200 ~25s.
|
||
- **Chroma-verify runbook:** `sudo docker exec -i <muninn> python -` (MUST pass `-i` or stdin never reaches `python -`) → `chromadb.PersistentClient('/data/kb/.chroma').get_collection('fiction').get(where={'job_id':<job>}, include=['metadatas'])`. Chroma persists at container `/data/kb/.chroma` = host volume `worldtree-personal_worldtree-kb`.
|
||
- **Retrieval-visibility check (NOT grounding — that's ratatoskr's):** a Mimir session — admin token `~/.config/worldtree/personal-admin-token` (wildcard scope) → POST `/sessions` (agent_id=`mimir`, `record_tool_intermediates=true`) → POST `/sessions/{id}/messages` (STREAMS SSE, not JSON) → parse SSE `tool_result` for `search_library` wing hits → DELETE session.
|
||
|
||
## Production dedup sweep (operator-approved)
|
||
Deleted the 785 April-era DCC orphan rows (`job_id=b59c147c5ce0`, no wing/source_identity metadata → predate identity tracking) from the `main` collection. Supervised protocol: read-only verify count == 785, back up all rows (ids+docs+embeddings) to `corviduo-dev:/tmp/main-sweep-backup-b59c147c5ce0.json` (reversible), `main.delete(where={job_id})` (assert target==785 first), verify `main` 4009→3224, then **bounce the api** (a separate-process delete leaves the api's in-memory HNSW index holding the vectors until reload — the #381 pattern generalizes to deletes), confirm search now fiction-only. Backup left for /tmp natural cleanup (fiction wing is canonical; `~/archives` has the historical record).
|
||
|
||
Result: fiction wing 166 → 1,372 concepts; three consumer verify rounds 0/5 → 5/5 → saturated; #385 budget fix validated (705 vs April's 785 control, extraction AND indexing, zero truncations). worldtree-dev filed #388 for a deploy concurrency-lock (the shared-containerd race fix). See [[2026-08-02-mimir-inbox-arc]].
|
||
_Archived 2026-08-18._
|
||
|
||
`[2026-08-02]` **The mimir-inbox / #377-read-path arc — deploy, four bugs found+fixed+verified, a cloned voice, all in one long session (2026-08-01→02).**
|
||
|
||
The browser-facing half of the #377 Muninn ingestion arc, end to end: mimir-inbox stood up, the write path proven, the read path chased through four defects to a verified-working state, and a character voice cloned into the TTS zoo. Peers: mimir-dev (the app), muninn-dev (gate/watcher spec), worldtree-dev (Worldtree app layer + the #380/#381/#382/#383 fixes), ratatoskr-dev (a consumer + the rigorous verifier).
|
||
|
||
## mimir-inbox deployed (#377)
|
||
- **New infra-ops stack, canonical eshpfi `stacks/mimir-inbox/`; live corviduo-dev `10.250.50.152:8091`** (co-located w/ muninn-gate :8090 + the worldtree-personal muninn watcher). Full deploy detail + procedures → auto-memory `reference_mimir_inbox_deploy`.
|
||
- **Placement decision (operator, reversed):** 7-31 he ruled mimir-inbox stays OFF corviduo-dev (shared/NFS mount); 8-01 he REVERSED to CO-LOCATE. Trigger: muninn-dev's code-check showed staging is NOT same-fs-constrained (gate reads staging metadata + passes path strings; `os.replace` is inside `ingestion_root`) — staging's real constraint is **path-identity across writer/gate/watcher**, which co-location buys outright while dodging NFS failure modes. I HELD the reversal for the operator's direct word (data/hosting on a team-managed box, reversing his own ruling) even against 3 peer relays — vindicated as the right instinct; muninn-dev agreed.
|
||
- Build: **`uv sync --no-dev --frozen`, SINGLE-STAGE** (project installs editable-linked to `src/`, so src/ MUST stay beside .venv — a multi-stage "copy only .venv" dies at import/404s assets). uid 1000, host-net bind 10.250.50.152:8091, TCP-liveness healthcheck (deliberately NOT gate-coupled). Redeploy = refresh build context (**preserve the on-server `.env`!**) → `docker build -t mimir-inbox:0.0.1 -t mimir-inbox:<sha> .` → `compose up -d`. Version stays 0.0.1 across dev commits → tag the image w/ the source SHA too. Live commit progression `0478452`→`c8ab38f`→`2dcc77e`→**`8ece117`** (3 redeploys).
|
||
- mimir-inbox key on the gate bumped [read,submit]→**[read,submit,control]** (cancel/retry); brokered via a 0600 drop on nh3-dev (never on the althing bus).
|
||
|
||
## The read-path bug chain (worldtree-dev's, all found via this arc)
|
||
- **#380 wing-blind indexing:** the book-ingest path upserted concepts into a hardcoded `main` Chroma collection while wing search reads the `fiction` collection → P&P written to disk but `search_library` returned total 0. A silent-success defect ("complete/69 indexed" was right about the WRITE, wrong drawer). Root-caused off MY physical evidence (files on disk + search empty). Fixed b164 + a one-shot `--reindex <job_id>` (re-upsert into the right wing collection + delete stray `main` rows).
|
||
- **#381 stale Chroma client:** the personal api opens its Chroma client before the watcher's cross-process writes → **a freshly-ingested/re-indexed book is NOT queryable until the api is restarted.** Proven by my restart-diagnostic (pre-restart total 0 → post-restart hits, same index). Workaround until fixed: `docker restart worldtree-personal-worldtree-api-1` after any ingest/re-index. Filed as #381.
|
||
- **#382 unreliable Mimir grounding (the subtle one):** post-#380-fix the index was correct, but Mimir's grounding was INTERMITTENT — some sessions navigated the opaque job-hash dir (`mimir-f3887c9b97b7`) to the content, others distrusted the correct vector hits and **silently answered from training knowledge** (worst of the looks-fine-isn't family). ratatoskr-dev caught it; I'd been over-confident ("Mimir read Austen back to you") having verified the INDEX, not the GROUNDING. Fixed b166 with BOTH shapes: a self-describing `_index.md` per wing job-dir (resolves the hash dir to its title) + a Mimir prompt rule (wing-scoped hits ARE library content, never discard on a name mismatch, never substitute training). **Verified: ratatoskr-dev re-ran 3× fresh sessions → 3/3 grounded**, citations in note-extracted language not raw Austen. #382 CLOSED.
|
||
- **DCC (Dungeon Crawler Carl, job `b59c147c5ce0`) backfill:** `--reindex` FAILED ("job not found in any state dir" — predates state-tracking). SETTLED = **no re-file** (the b166 prompt rule already grounds it even without an `_index.md`; ratatoskr confirmed incidentally); an `_index.md` rides whenever DCC is next re-ingested.
|
||
- **#377 mimir-inbox banner bug (mimir-dev's, `8ece117`):** `/health-banner` misattributed an unwritable `ingestion_root` to the WORKER, rendering "The worker is not running." for a running worker — a false lead pointed at infra-ops's half of #377. Fixed (guard split into two banners); I confirmed from the DEPLOYED handler (not just the test) that `ingestion_root_writable:False` now renders "The ingestion root is not writable."
|
||
|
||
## muninn-gate → muninn-dispatch 0.1.5
|
||
Rebuilt `muninn-gate` off `vh/muninn-gate` main `bc04c4c` (dispatch 0.1.4→0.1.5) so the gate serves the new `concept_schema`/`concept_schema_source` row fields (computed gate-side). Gate version unchanged 0.0.14 (dual-tag the SHA). Build needs the vh gitea token as a BuildKit secret (`--secret id=gitea_pw`, UV_INDEX_GITEA_USERNAME=vh, drop+shred). Recreate with `compose up -d` (NOT bare restart — needs the new image). Verified: P&P job serves `concept_schema='fiction'`, `concept_schema_source=null` (null correct — pre-b164 job). Registry tags by commit SHA — `v1.0.0bNNN` docker tags don't resolve; use the deployed SHA (confirm `--reindex` present before using an image for a data-op).
|
||
|
||
## donut voice (65-frost → Zonos gateway)
|
||
Operator: "pick up 65-frost, use that bundle as a voice for a character named donut." 65-frost = a **Booth id** (`~/booth-data/65-frost/`) holding a curated yt-voice-clipper dataset (`dataset-…-curated.zip`: 4 clips + manifest, all SPEAKER_02 = Princess Donut). **Zonos gateway voice registry = a filesystem drop:** `<Name>.wav` in the voices dir (44.1kHz mono s16 PCM) auto-registers as `voice:"<name>"` on **startup** (needs a restart). The LIVE dir is the bind mount `/opt/docker/compose/zonos-gateway/voices/` (lkraven-writable), NOT the working tree. Built `Donut.wav` from seg000 (best clip), dropped it, restarted → `voice:"donut"` live in the gateway AND the Asset Engine's make form. Also copied to the build-source tree `~/zonos-gateway/voices/` for rebuild-durability (true canonical = the gitea repo, not yet CI-wired). Auditioned in booth `donut-voice`. **Expanded 2026-08-02 (onyx-58 bundle):** operator curated a 2nd Booth bundle `onyx-58` (`dataset-467d2cf8…curated.zip`, 3 Donut clips) as additions. Rebuilt the reference = **seg000 (65-frost) + seg101/seg110/seg148 (onyx-58)** ffmpeg-concat + resampled 24k→44.1k mono s16 = **52.0s**. `seg148` was diarized SPEAKER_03 but is Donut (operator-confirmed misdiarize → included). Assembly is NOT `assemble_voice.py` (that `-c copy` can't resample + caps ~15s); used a manual `aresample=44100,aformat=…,concat=n=4` filter. Backed up old ref → `irv-ml1:~/Donut.wav.pre-onyx58`; dropped to live bind-mount + build-source tree; `docker compose restart` (healthy 2s, `voice:"donut"` still 1 of 9). A/B booth `donut-onyx58` (A=old 16.3s ref, B=new 52s ref, same line). Longer ref is fine mechanically: gateway passes it as `speaker_audio_base64` → speaker *embedding*, not an audio prefix. **BUT auditioned → REVERTED same day:** pinned-seed neutral A/B (5 pairs, booth `donut-onyx58`) showed the single-clip seg000 (16.3s) beats the 52s 4-take concat on timbre — concatenating disparate takes muddied the embedding more than the range helped. Reverted both live + build-source to seg000-alone. Lessons (→ Tried-and-abandoned): more reference ≠ better when takes vary; and **emotion steering pulls output away from the clone fast** (operator craft rule) — keep clones emotion-neutral; bare `{input,voice}` calls send NO emotion (gateway only enables it on an explicit `emotion_*`/`preset` dial).
|
||
|
||
## Zonos streaming (no gateway change needed)
|
||
ratatoskr wanted play-as-it-arrives. `/v1/audio/speech` ALREADY streams — chunked `StreamingResponse`, opens native `/tts/generate` with `stream=True`, wraps as a streaming int16 WAV with `0xFFFFFFFF` placeholder sizes (meant for progressive `<audio>`). Verified TTFB 0.44s vs 6.84s total, `transfer-encoding: chunked`, dials preserved. ratatoskr's proxy was rewriting the placeholder header → forced buffering. Fix was theirs (pass chunks through); shipped + confirmed (TTFB 0.46s progressive). The Asset Engine (ana-docker:8200) IS the fleet "TTS zoo" (~20 audio svcs w/ irv-ml1 endpoints); zonos-gateway registered there, state=ready.
|
||
|
||
## Lessons (also in Tried-and-abandoned)
|
||
- **Verifying the INDEX (search returns hits) is NOT verifying GROUNDING** (does the agent trust+use them vs. silently answer from training). Check that citations are note-extracted, not model-knowledge. ratatoskr caught this after my over-confident "it works."
|
||
- **Reading the DEPLOYED artifact > trusting the test** for "is the fix live" — the test proves the source is right; reading the running code proves the artifact is, which is what an on-call actually meets.
|
||
- Held a boundary-box/data reversal for the operator's DIRECT word against 3 peer relays — the right call (peer relay ≠ operator consent; the placement guard was vindicated).
|
||
|
||
See also: [[2026-07-31-muninn-gate-deploy]]. auto-memory: `reference_mimir_inbox_deploy`, `reference_muninn_gate_deploy`, `reference_muninn_gate_staging_path`, `reference_zonos_tts_stack`, `reference_infra_ops_vh_gitea_token_and_sdk_publish`.
|
||
_Archived 2026-08-18._
|
||
|
||
- `[2026-07-27]` **jackdaw-compose.service DECOMMISSIONED** (jackdaw-dev request; the JackDAW AI Composer was cut from v1 by operator decision 2026-07-27). Stopped + disabled the nh3-dev `:8787` user service (no client calls it — ai/server/AiChat deleted from main, `/compose` proxy removed); unit **archived not deleted** → `~/.config/systemd/user/jackdaw-compose.service.decommissioned-20260727` (revival = rename + `daemon-reload`). **No credential revoked** — the unit used the SHARED all-agents LiteLLM key (`sk-eA_XOd…`, model `gen`), not a dedicated one. Code preserved on jackdaw `origin/ai-composer-preserved`; treat as permanent. The `:4500` HTTPS audition bench is untouched. (Supersedes the 2026-07-23 stand-up line below.)
|
||
_Archived 2026-08-18._
|
||
|
||
|
||
## Tried and abandoned (archived)
|
||
|
||
- `[2026-08-02]` **donut voice multi-clip reference (onyx-58 expansion) — TRIED, REVERTED.** Folded the `onyx-58` bundle's 3 Donut clips (seg101/seg110/seg148) in alongside the original seg000 → a 52.0s 4-take concat reference, hoping a longer ref → more robust speaker embedding. A pinned-seed A/B (5 pairs, varied registers, booth `donut-onyx58`) showed the **original single-clip seg000 (16.3s) sounds better** — concatenating disparate takes muddied the timbre more than the extra range helped. Reverted to seg000-alone (live + build-source). **Two durable lessons:** (1) for a faithful clone, a single clean representative take can beat a longer multi-take concat — more reference audio is NOT automatically better when the takes vary. (2) **Emotion steering pulls the output AWAY from the cloned voice fast** (operator's craft rule) — keep donut (and clones) emotion-neutral for fidelity; the gateway only enables emotion when an `emotion_*`/`preset` dial is explicitly sent, so bare `{input,voice}` calls stay pure-clone. `seg148` was diarized SPEAKER_03 but IS Donut (operator-confirmed misdiarize). onyx-58 curated bundle lives in booth `onyx-58` (24h TTL — stash to `/mnt/smithy/voice_clones/` if a future middle-ref experiment is wanted).
|
||
_Archived 2026-08-18._
|
||
|
||
- `[2026-08-02]` **Verifying the INDEX is not verifying GROUNDING** (#382). A `search_library` returning wing=fiction hits proves the content is *retrievable*; it does NOT prove the agent (Mimir) *trusts and uses* those hits vs. silently answering from training. I reported "Mimir read Austen back to you" off a grounded-*looking* answer; ratatoskr-dev caught that grounding was intermittent (some sessions discarded the correct hits and substituted training knowledge). Test the harder claim — are the citations note-extracted or model-knowledge? — and reading the DEPLOYED artifact beats trusting the test for "is the fix live."
|
||
_Archived 2026-08-18._
|
||
|
||
- `[2026-07-30]` **brokkr's WebSearch "verification" CONFIRMED a hallucination — 3 phantom `microsoft/Mage-Flow-{Base,Turbo,Edit}` repo IDs.** brokkr-smithy-dev handed 3 gated-looking repo IDs for an operator-directed model pull; they don't exist (its own web-search fabricated an arXiv ID + project page, twice). Lesson: the HF **registry API is ground truth** — an unauth 401 ≠ exists (`{"error":"Invalid username or password"}` masks private/gated/nonexistent alike), an authed 404 = phantom, and `author=X&search=Y` refutes existence. API-verify every repo ID before a pull; LLM-summarized web fetches confabulate. auto-memory `reference_verify_hf_repo_ids_before_pull`.
|
||
_Archived 2026-08-18._
|
||
|
||
- `[2026-07-30]` **magpie TTS serving — evaluated, ABANDONED.** Pulled `magpie_tts_multilingual_357m` (the one real repo of brokkr's batch) to NFS, stood it up on irv-ml1 (ephemeral NeMo-Speech-`main` container — stock PyPI/NGC NeMo can't load v2607), A/B'd vs Zonos → Zonos wins expressive English decisively, multilingual not needed. Not served; `magpie-nemo` torn down. `.nemo` KEPT on NFS as brokkr's fine-tuning base. auto-memory `project_magpie_tts_eval_rejected`.
|
||
_Archived 2026-08-18._
|
||
|
||
## Recent decisions (archived 2026-08-19 batch)
|
||
|
||
`[2026-07-31]` **kimi-k3 "output cap" root-caused = a ~16384 REASONING-token ceiling, not an output cap; fix relayed to heid, NOT applied gateway-side.**
|
||
|
||
heid reported that `kimi-k3` (the primary route = Kimi Code coding endpoint `openai/k3` @ `api.kimi.com/coding/v1`) silently degraded its cross-frontier panel: on large/reasoning-heavy dispatches, `completion_tokens: 16381` **exactly**, `content` empty, `reasoning_content` ~64KB, `finish_reason: **stop**` (a truncation mislabeled as a clean stop). `max_tokens: 100000` in the request was not honored.
|
||
|
||
**Investigation arc (a clean cross-frontier-triage + verify-on-the-wire case):**
|
||
1. My first read: a flat ~16384 OUTPUT cap; fix = a LiteLLM `stop→length` relabel callback (heid's fallback ask). Confirmed the cap isn't in our LiteLLM config (no `max_tokens` clamp on the route).
|
||
2. Operator routed a fix-research pass to **dvalin-smithy-dev + bil-smithy-dev** (independent). Both CONVERGED (docs-based): `max_tokens` is a deprecated alias on Kimi/Moonshot; the canonical field is `max_completion_tokens` (default 131072, max 1M); the coding endpoint defaults output to 16384; fix = send `max_completion_tokens` + `reasoning_effort` via `extra_body` (drop_params-safe).
|
||
3. **heid's live data REFUTED the docs hypothesis:** a later dispatch hit `completion_tokens: 18455` (ABOVE 16384) cleanly, with `reasoning_tokens: 16198` (just under 16384) and content present. So COMPLETION is uncapped; the bound is on **REASONING at ~16384**. When a hard task's thinking exhausts that budget, nothing's left for content → empty answer under `stop`.
|
||
4. **I proved it on the wire** — ran heid's real 500KB failing bundle direct at both endpoints (bypassing LiteLLM so `reasoning_effort` isn't dropped): default effort → 504/timeout (the failure); **`reasoning_effort: low` → reasoning ~12–13.5k (under the ceiling), content returns (6–7.6k chars)**, on BOTH coding AND general endpoints. So re-routing to the general endpoint buys nothing — the fix is the effort param, and it works on the wire.
|
||
|
||
**THE FIX (caller-side, no shared-gateway change/restart):** send `reasoning_effort` via **`extra_body`** on kimi-k3 dispatches (`low` for large bundles). LiteLLM `drop_params: true` strips the top-level `reasoning_effort` — which is exactly why heid's earlier `reasoning_effort: low` was a no-op. `extra_body` survives drop_params (the house GLM-thinking pattern). Tradeoff: low effort = shallower reasoning, but a complete answer beats today's empty one.
|
||
|
||
**Relayed to heid to validate on a real round** (the one unconfirmed hop is whether `extra_body` survives OUR LiteLLM). **Backstop if it doesn't:** add `allowed_openai_params: ["reasoning_effort"]` to the `kimi-k3` route in the gateway config — that IS a shared-gateway change + a ~10s restart (blips all consumers), so it needs a heads-up.
|
||
|
||
Gateway = LiteLLM on ana-docker `10.250.50.70:4000`; kimi-k3 config in `stacks/litellm/conf/config.yaml` (see Recent-decisions `[2026-07-25]` Kimi K3 wiring). No gateway change was made this session. Failing dispatch on record: `01KYTASKTY3T` (jackdaw-dev bug-hunt).
|
||
_Archived 2026-08-19._
|
||
|
||
- `[2026-07-25]` **bil-smithy-dev wired as an althing zellij-window-ping (pane route).** She's a `driver: human` dwarf peer (pane `bil-smithy` already live alongside eitri/dvalin/regin-smithy in the `Claude` zellij session) but had no delivery route → smoke messages posted to the bus but never reached her window. **Mechanism (reusable for any pane-route handle):** `~/.althing/config.yaml` → `zellij_sessions.Claude.agents[]` maps `handle` → `target` (a zellij pane **TITLE**, matched via `list-panes -j` in `althing/zellij.py:resolve_pane_id`) → `command` (herald `write-chars` + CR into that pane). The **herald loads config ONCE at startup** (`herald.py main()`), so **`systemctl --user restart althing-herald.service`** after editing. Added bil (`target: bil-smithy`), restarted, verified: herald delivered the pending smoke `01KYD7W7CF…` (available→attempted→**delivered**). ⚠️ Noticed pre-existing pane-route errors on `worldtree-codex` + `eitri-smithy-dev` ("route-error: list index out of range", empty msg_ids — likely `render_command messages[0]` on an empty list; NOT caused by this change, bil works) — worth a herald look.
|
||
_Archived 2026-08-19._
|
||
|
||
## Tried and abandoned (archived 2026-08-19 batch)
|
||
|
||
- `[2026-08-02]` **`docker exec` into worldtree containers defaults to ROOT — root writes contaminate the uid-1000 (vh) KB tree.** My `sudo docker exec … --reindex` on personal ran as ROOT (muninn app = uid 1000); its wing git-commit + atomic note-swap left root-owned files in the `worldtree-personal_worldtree-kb` volume: a root-owned `.old-<job>` backup dir (blocked the uid-1000 retry's `rmtree` → Errno 13, because unlink needs write on the DIR and it was root:root 755) AND **60 root-owned loose git objects** in `.git/objects/`. Fix (host-side, corviduo-dev): `sudo rm -rf` the superseded `.old-` dir (tar'd aside to /tmp first) + `sudo find … -user 0 -exec chown 1000:1000` the objects (ownership-only, git-content-safe; the `.git/objects/XX/` dirs were vh-owned so these weren't a hard blocker, but violated "clean tree"). **RUNBOOK RULE (worldtree-dev, ADOPTED):** any `docker exec` into worldtree containers that WRITES pipeline state runs **`-u 1000`**, never default-root — same genus as the mv footgun (acting without matching the target's constraints; 3rd such slip in one session). **GOTCHA that hid the scope:** `find … -user 0 | head -20` TRUNCATED (the `.old-` dir alone had 153 files, so the first page was all `.old-`) → I "verified clean" off a partial list. Never `head` a scope-defining find; count first (`| wc -l`). **Related blind-spot (muninn-dev):** a root-owned job SUBDIR passes every requeue guard (job_row/dispatch/list_jobs render fine) AND `/health` (contract's `os.access(ingestion_root, W_OK)` tests only the ROOT dir, so a foreign-owned subdir under `pending/` still reports `ingestion_root_writable: true`) — then the uid-1000 gate can't write into it. "Clean board + green /health + failure at next mutation." muninn-dev added an OWNERSHIP column to the standing post-move check to catch it; two green signals both miss a foreign-owned subdir otherwise.
|
||
_Archived 2026-08-19._
|
||
|
||
- `[2026-07-25]` **Chaining the althing wake-listener arm orphans it.** `reply && althing-wake-listener &` (or spawning `althing-wake-listener` with `&` *inside* a `run_in_background` task) → the `&`-child reparents to init, UNTRACKED by the harness: no fire-notification, and re-arms bounce rc3 off a lock nothing services (mail silently unwatched). Compounding foot-gun: re-arming after a *plain operator turn* (not an actual fire) collides with the still-live prior listener (rc3). FIX: spawn `althing-wake-listener` as its OWN `run_in_background` task, and re-arm ONLY after a real fire (`<task-notification> completed rc0`). Reclaim an orphan with `althing-cli stop-monitor` then re-arm.
|
||
_Archived 2026-08-19._
|
||
|
||
## Recent decisions (archived 2026-08-20 batch)
|
||
|
||
- `[2026-08-05]` **Booth — 3 features shipped, live on `:8090` + tagged.** (1) verbatim-`index.html` booths get a floating top-right "‹ all booths" chip + inherited favicon, doctype/charset-safe byte-injection (`booth-v0.1.5`, `8577e7e`); (2) `.md` renders + `.txt`/`.log` view in-booth without downloading via the `/b/<n>/view` route + a `markdown` dep + `doc.html` (`booth-v0.1.6`, `315faac`); (3) prev/next arrows in the image zoom viewer — wrap-around + keyboard ←/→, hidden for single-image booths (`booth-v0.1.7`, `c37a425`). Canonical `services/booth/`; deploy = `systemctl --user restart booth.service` on nh3-dev (runs from the checkout's `.venv`; `uv pip install` new deps into it first); 47 tests. `uv.lock` gitignored (`348c5c1`).
|
||
_Archived 2026-08-20._
|
||
|
||
- `[2026-07-31]` **worldtree-sdk 1.1.0 (Python) published to vh Gitea PyPI + a durable infra-ops publish cred.** memory_context pass-through; unblocked wyrd-dev. claude-bot now a write-collaborator on `vh/worldtree-sdk` (source pulled via the **Gitea API archive** — git-HTTP 403s on that repo); publishing to the vh USER namespace **can't be delegated** (401 `reqPackageAccess` even with `write:package`) so it needs an owner token — operator saved a **FULL vh site-admin token at `~/.config/gitea/vh-token` (0600)** for it (⚠️ high blast radius, kept over a scoped one; org-namespace migration is the only real de-personalization, parked by wtsdk-dev). auto-memory `reference_infra_ops_vh_gitea_token_and_sdk_publish`.
|
||
_Archived 2026-08-20._
|
||
|
||
## Tried and abandoned (archived 2026-08-20 batch)
|
||
|
||
- `[2026-08-03]` **corviduo-dev shared containerd: a concurrent-pull race fails ONE instance's deploy; DON'T "prune to fix" — the image is in-use by the instance that won the race.** b169 personal deploy failed at `docker compose pull` (`Lchown … no such file or directory` on the big torch layer → looked like disk pressure / corrupt snapshot). ACTUAL: NOT disk (56G free, inodes 7%). demo + personal + pinned share ONE `/var/lib/containerd` on corviduo-dev; demo (from main) and personal (from staging tag) extracted b169's shared torch layer simultaneously → personal's hit a partial snapshot mid-race and aborted while demo's completed. The image `6e34a87` was FULLY VALID — demo was RUNNING it healthy. Fix = just re-run the failed deploy (image already materialized; compose pull finds it present). **NEAR-MISS:** worldtree-dev's suggested "prune unused images/snapshots" would have rmi'd `6e34a87` = the image the running demo depends on → demo outage. **Lesson: before any prune/rmi "cleanup," `docker ps` the running images — an "unused" image may be a co-tenant's live one; and verify the failure's REAL cause (disk? inode? in-use? race?) before applying the suggested remedy.** (Pipeline fix, deferred: serialize demo-from-main + personal-from-staging, or a per-image pull lock, to avoid the shared-layer extraction race.)
|
||
_Archived 2026-08-20._
|
||
|
||
- `[2026-08-02]` **`mv <job> complete/ → failed/` RENAMED the job to `failed` because failed/ didn't exist.** worldtree-dev's round-2 unblock command (`mv /data/state/ingestion/complete/<job> /data/state/ingestion/failed/`) assumed `failed/` existed; on PERSONAL muninn it did NOT (fresh instance — root was `active/ complete/ pending/ sources/`, no `failed/`). `mv src nonexistent/` **renames** src→nonexistent, so job1 became the `failed` dir and job2 nested inside it. Caught on post-move `ls` (failed/ held job *contents*, not two subdirs), reconstructed via complete/ as watcher-safe scratch + rebuilt `failed/` (worldtree:worldtree 755) — NO data loss. **Lessons:** (1) before `mv X into-dir/`, verify the dir EXISTS (`[ -d dir ]`) — an empty `ls dir/ 2>/dev/null` is AMBIGUOUS (missing vs empty), which was the preflight miss that let it through; (2) the correct guard is **`mv -t <targetdir> <src>`** (`--target-directory`): it refuses a MISSING target loudly (rc=1, "No such file or directory", nothing moved) — this is the house convention for queue/state moves now. TESTED by muninn-dev on coreutils 9.1: a **trailing slash does NOT protect** — `mv src failed/` with `failed/` missing STILL silently renames to `failed` (rc=0); "just add the slash" is a false guard. (`mkdir -p failed/` first also works, but `mv -t` inverts the failure from silent-wrong to loud-safe in one flag.) Container `sh` is dash — no `(` in echo strings. **SILENT failure mode (muninn-dev carry-forward):** a misplaced ingestion-state move doesn't crash anything — `list_jobs()` stays OK, loose files are inert; the ONLY symptom is the job quietly absent from the board (`job_row`→None, requeue→not_found/404, looks IDENTICAL to the original block). So after ANY state move, verify the job is actually ON THE BOARD (`job_row` found + guards pass), don't trust mv exit codes — and confirm `job.dispatch.json` survived (requeue refuses a dispatch-less job with the same not_requeueable symptom). Cross-checked + all-clear'd by muninn-dev, who correctly refused to mutate ingestion_root (INV-MG-1) and flagged instead. **DON'T TIDY (round-2 pending):** both DCC + P&P jobs currently REST in personal `failed/` with manifests reading `state: complete` until round-2 requeue runs — deliberate + load-bearing (`requeue` keys on DIRECTORY PLACEMENT, not manifest state); looks wrong to anyone cold, leave it exactly as-is. **Round-2 sequencing:** the requeue is **mimir-dev's** browser flow (pending their operator's board-vs-API ruling); **muninn-dev** is the gate confirmer (runs the post-move board-check inside its custody — the right split, don't reach across INV-MG-1); **infra-ops** = the #381 restart after both jobs go terminal, then later the supervised main-collection sweep. Guard-verified HOLD LIFTED by muninn-dev 02:36Z. **ARC COMPLETE (2026-08-03 ~05:49):** both books terminal — DCC `mimir-6351554e8e8f` 705 concepts + P&P `mimir-f3887c9b97b7` 667, extracted AND indexed, 5/5 phases, 0 failures/truncations (validates the #385 budget fix vs April's 785 control); **#381 restart-after-ingest FIRED** (personal api, healthz/readyz 200 ~25s), retrieval-visibility confirmed (search_library returns DCC+P&P from fiction post-restart); handed ratatoskr-verify go to worldtree-dev. **Delete-sweep precondition NOW MET** — the stale DCC rows in `main` are genuine duplicates of live `fiction` rows, so worldtree-dev's supervised sweep of the ~785 April orphans is unblocked (still comes to me supervised: snapshot + operator-in-loop).
|
||
_Archived 2026-08-20._
|
||
|
||
`[2026-08-10→12]` **secrets-broker — per-box Vaultwarden credential store, SHIPPED + consumer-confirmed.**
|
||
|
||
**What.** A per-dev-box credential store over the fleet Vaultwarden (`vaultwarden.phasefinal.com`,
|
||
on ana-docker, DB on pfi-postgres, in the pg_dump backup set). The `secret` CLI at eshpfi
|
||
`services/secrets-broker/secret` (also installed to `~/.local/bin/secret`, on PATH for all sessions):
|
||
`put / get / list / rm / backfill`. Stores into the **`infra-ops` org's Default collection** (org
|
||
shared to the operator's primary account, so he sees items too), folder = hostname, item name =
|
||
`<host>/<path>`, title-derived slug. Small text → item note; small binary → base64 hidden field;
|
||
**>6000 B → a bw attachment** (Vaultwarden caps notes at ~10000 encrypted chars); sha256 + source
|
||
metadata fields; idempotent upsert keyed by name.
|
||
|
||
**Auth.** Bootstraps from `~/.config/secrets-broker/bootstrap.env` (0600): apikey login
|
||
(`BW_CLIENTID`/`BW_CLIENTSECRET`) + master-password unlock (`--passwordenv`) → per-invocation
|
||
session. That file is **secrets-zero** (it unlocks the vault, can't live in it) and is excluded from
|
||
backfill.
|
||
|
||
**Client = `bw`, NOT `rbw`.** rbw was the operator's first choice but its `register` returned an
|
||
undebuggable 400 against this Vaultwarden despite valid creds (a direct `client_credentials` grant +
|
||
both prelogin paths return 200; rbw emits no HTTP logs). Switched to the official `bw` CLI
|
||
(user-prefix npm install) — clean unattended flow, full write support (org collections + attachments).
|
||
|
||
**Backfill.** Local-only (each box backs up itself; NOT a fleet daemon). Scanned nh3-dev's
|
||
`~/development/*/{env.sh,.env}` + `~/.config` credential files, **25 items stored + round-trip
|
||
verified** (2 large via attachment). Excludes bootstrap.env / `.example` / `~/AIPA-Data` archives /
|
||
cargo noise.
|
||
|
||
**Post-launch (jackdaw-dev feedback).** Added **`secret rm <name>`** (bw soft-delete to trash,
|
||
recoverable) — closes the "no delete path, append-only" gap; and a **new-top-level-namespace warning**
|
||
on `put` (stderr, non-blocking) — catches a typo'd/missing host prefix at store time. Chose
|
||
warn-not-auto-prefix because domain-scoped names (`gitea/…`, `certs/…`) would misfire on auto-prefix.
|
||
Deferred edge recorded in the contract: the warning is non-blocking, so a scripted put suppressing
|
||
stderr can still mis-namespace — add an opt-in `--strict` only if scripted callers appear.
|
||
|
||
**Standing directive (now GLOBAL in `~/.claude/CLAUDE.md`):** the vault is the credential source of
|
||
truth — **`secret put` durable secrets into it AND `secret get` the creds a task needs FROM it**
|
||
rather than reading on-disk copies. Dogfooded by pulling the gitea `vh` token from the vault to create
|
||
`vh/stonehenge-park`.
|
||
|
||
**Deploy shape.** Not a service / no daemon — per-box; a new dev box duplicates the stack
|
||
(`services/secrets-broker/README.md`): npm-install `bw` to `~/.local`, drop a per-box `bootstrap.env`,
|
||
`secret backfill`. Commits: `41359ea` (CLI + contract), `850a197` (backfill 25/25 + attachment +
|
||
resilient run), `a249073` (rm + namespace warning), `a1304b7` (deferred-edge contract note).
|
||
Consumer-confirmed end-to-end by jackdaw-dev.
|
||
|
||
Auto-memory: `reference_secrets_broker_cli`.
|
||
_Archived 2026-09-03._
|
||
|
||
- `[2026-08-19]` **`claude-bot` granted read on `vh/waterland`** (operator-empowered, verified `admin:false push:false pull:true`) so irv-ml1 can self-update without the operator's site-admin token living on a GPU box. Precedent for the standing migrate-off-operator-creds directive: grant the service account, wire a repo-scoped 0600 credential helper, keep the remote URL clean. Commit `8189076`.
|
||
_Archived 2026-09-03._
|
||
|
||
# `[2026-08-19]` esh-pve hard-froze for 4.5h — and took the whole house's DNS with it
|
||
|
||
Reported by the operator as "routing or DNS issues on the PVC wifi." It was
|
||
neither: the internet was healthy the entire time (gateway reporting 3 ms and
|
||
209/26 Mbps; 1.1.1.1 and 8.8.8.8 answering at ~3 ms from inside ESH with zero
|
||
loss). **The house had no name resolution because one VM was down.**
|
||
|
||
## The SPOF: one resolver, cross-VLAN, no fallback
|
||
|
||
`esh-userland` (VLAN 10, `10.0.10.0/24` — the `PVC` SSID *and* the wired
|
||
userland LAN) handed out **exactly one DNS server, `10.0.50.45`** — AdGuard, on
|
||
`esh-docker-vm`, on the **server** VLAN. No secondary. That VM dies, every
|
||
client on the VLAN loses DNS, and it presents as "the wifi is broken."
|
||
|
||
It was the only network in the house exposed this way. `Default`, `esh-mgmt`,
|
||
`esh-server` and `esh-cameras` run DNS on auto (the gateway hands itself out);
|
||
`esh-iot` and `ESH-WG` point at 1.1.1.1 + 8.8.8.8.
|
||
|
||
**Fixed** (operator-approved): `esh-userland` now hands out `10.0.50.45`
|
||
primary, **`10.0.10.1` (the gateway) secondary** — the UDM's own resolver,
|
||
verified answering. Applied via the Classic API,
|
||
`PUT /proxy/network/api/s/default/rest/networkconf/687985eae5d15b673cef1a73`
|
||
with the full object (GET → modify one field → PUT), `rc: ok`. **This was also
|
||
the first confirmed WRITE on the ESH UDM key** — previously only the NH3 key
|
||
was write-tested. See [[reference_unifi_udm_integration_api_keys]].
|
||
|
||
⚠️ **A secondary is not clean failover.** macOS/iOS query resolvers in
|
||
parallel, so once AdGuard is back a real share of lookups go to the gateway and
|
||
**skip ad-blocking**. This converts a total outage into degraded-but-working.
|
||
The actual fix for blocking integrity is a second AdGuard instance NOT on
|
||
esh-pve.
|
||
|
||
## Root cause: hard freeze, no diagnostics, two suspects
|
||
|
||
`esh-pve` (Minisforum MS-01, i9-13900H, `productname: YajuuSenpai`) froze at
|
||
**03:34:39**. The journal stops mid-operation — **no panic, no OOM, no MCE, no
|
||
thermal event**. Powered on with its 10G link up, but not answering ARP.
|
||
|
||
Two changes landed the day before, and they are not exclusive:
|
||
|
||
1. **New kernel.** A large `apt` batch on **2026-08-18 07:00:21** installed
|
||
`proxmox-kernel-6.8.12-42-pve`; clean reboot at 07:08:44. Before that the
|
||
box had **4.5 months of uptime** (Mar 30 → Aug 18) on `6.8.12-16`. First
|
||
boot on the new kernel lasted **20 hours**.
|
||
2. **GPU passthrough.** The last kernel messages of the dead boot are
|
||
`vfio-pci 0000:01:00.0/.1: enabling device` at **02:55:17** — VM 102
|
||
`esh-vm-workstation` starting with `hostpci0: 0000:01:00,pcie=1,x-vga=1`,
|
||
**39 minutes before the freeze**.
|
||
|
||
A vfio/i915 regression in the newer kernel would produce exactly this
|
||
signature. `6.8.12-16` is still installed and is the held-in-reserve rollback.
|
||
|
||
**VM 102 is now pinned off** (`qm set 102 --onboot 0`, stopped) per the
|
||
operator — it is on-demand and there has been no demand. That removes the
|
||
suspect without a kernel rollback.
|
||
|
||
## Why nobody could recover it remotely — and the fix
|
||
|
||
Nothing on the box could reboot it:
|
||
|
||
- **`softdog` was the loaded watchdog.** A *software* watchdog cannot rescue a
|
||
hard kernel freeze: the frozen kernel is the thing that would have to fire
|
||
its timer. This is the trap — the machine *looked* watchdog-protected.
|
||
- **Proxmox's `watchdog-mux` held `/dev/watchdog` but never armed it.** It only
|
||
pets the device while an HA client is connected, and this cluster has no HA
|
||
resources.
|
||
- **vPro/AMT was unusable.** The MS-01 reaches the network only via **SFP+**
|
||
(Intel X710, port 27 on the Garage switch) and presents exactly one MAC.
|
||
**AMT cannot ride a discrete/SFP+ NIC** — it needs the chipset-integrated
|
||
Intel PHY, i.e. one of the two i226 RJ45 ports, and both are unplugged.
|
||
Cabling one and provisioning AMT in MEBx remains the open item for *control*;
|
||
the watchdog below is the fix for *recovery*.
|
||
|
||
**Fixed:** `playbooks/esh-pve-hardware-watchdog.yaml` — systemd now owns the
|
||
PCH hardware watchdog (`iTCO_wdt`, `RuntimeWatchdogSec=60`), `softdog` is
|
||
blacklisted and unloaded, `watchdog-mux` is masked. Verified live:
|
||
`watchdog0: identity=iTCO_wdt state=active timeout=60s`, held by PID 1,
|
||
journal `Using hardware watchdog 'iTCO_wdt', version 6`. Playbook re-run proves
|
||
idempotency (6 skipped / 6 verify OK).
|
||
|
||
Firmware does **not** block the TCO timer here — checked for the
|
||
`unable to reset NO_REBOOT flag` line before committing to the approach; the
|
||
board reports `Found a Intel PCH TCO device (Version=6, TCOBASE=0x0400)`.
|
||
|
||
⚠️ **Masking `watchdog-mux` trades away HA fencing.** If Proxmox HA is ever
|
||
configured on esh-pve this must be reverted. Not a near-term concern:
|
||
`esh-pve-cluster` is **two nodes with no qdevice**, so a single node loss
|
||
already costs quorum and the survivor would fence itself — HA here would reduce
|
||
availability, not raise it.
|
||
|
||
⚠️ **The watchdog is configured and armed, but has NOT been proven to fire.**
|
||
Proving it means deliberately wedging the host. Untested-but-armed is still
|
||
strictly better than softdog; treat a real firing as unconfirmed until tested.
|
||
|
||
## Diagnostic corrections worth keeping
|
||
|
||
- **"No route to host" was the dead host, not a routing gap.** Two claims made
|
||
mid-incident were wrong: that the mgmt VLAN (`10.0.250.0/24`) is not routed
|
||
over the NH3↔ESH tunnel, and that a firewall isolates it from the server
|
||
VLAN. Both were artifacts of esh-pve being dead. With it up, `root@esh-pve`
|
||
SSHes fine from nh3-dev at 7.5 ms, and `10.0.250.1` answers from
|
||
`esh-pve-nas` in 0.078 ms. **Control-test against a *different* host on the
|
||
target subnet before concluding "the subnet is unreachable."**
|
||
- **UDM `uptime` on a client record is association time, not host uptime.** It
|
||
read 2.2 days while the host had been up 20 hours. Use
|
||
`journalctl --list-boots` on the host for real boot history.
|
||
- **`rest/user` `last_seen` is not maintained** (it read ~203 days for hosts
|
||
that are demonstrably online). `stat/sta` is the live view.
|
||
_Archived 2026-09-03._
|
||
|
||
- `[2026-08-21]` **speaches ASR live on irv-ml1 for Eyra — and `no_speech_prob` alone is a weak hallucination gate.** Silence and room tone both hallucinated "Thank you." under 0.11; `avg_logprob` separates ~6× better. Consumers should gate on a composite. (`aa5863c`, `c7e2187`)
|
||
|
||
_Archived 2026-09-05._
|
||
|
||
# `[2026-08-20]` Cold-Fusion abliteration — Robinson recipe captured, and the transformers/DeltaNet bf16-NaN fight
|
||
|
||
The real work of the session: abliterate `DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1`
|
||
using the MTP-aware, vision-preserving **Robinson formula** (documented in
|
||
`docs/pfi/abliteration-recipe-qwen38.md` from `RobinsonLabs/Qwen3.8-27B-abliterated`).
|
||
Harness: `services/coldfusion-abliteration/`. Runs on ana-ml2.
|
||
|
||
## Why this model, why abliterate it ourselves
|
||
|
||
Stock Cold-Fusion's refusal profile was **probed 2026-08-19** (Q6_K GGUF on
|
||
llama.cpp, 24-prompt battery, hand-verified after a keyword-classifier bug):
|
||
**~33% creative refusal**, concentrated on **explicit-sexual + graphic-torture**;
|
||
4/5 hard-harm technical refused; self-harm guardrails intact 3/3; benign
|
||
over-refusal 0. So there is a real creative-content refusal surface to remove.
|
||
This **supersedes** the earlier "watch for DavidAU's own heretic build" posture —
|
||
we abliterate it ourselves.
|
||
|
||
**It is additive over the current gen seat.** The live Heretic seat
|
||
(`qwen38-27b-heresy-bf16`) left its MTP head a **byte-identical base graft** —
|
||
the `Qwen3_5ForConditionalGeneration` wrapper never loads it, so Heretic could
|
||
not touch it. The Robinson formula abliterates the MTP head **in-band** (its 2
|
||
residual-write matrices), and the MTP head is what gates speculative acceptance.
|
||
That in-band MTP edit is the delta this experiment tests.
|
||
|
||
## Recipe maps 1:1 — dry-run PASSED
|
||
|
||
Against the staged bf16: 1199 tensors, 333 vision preserved,
|
||
`down_proj=64 o_proj=16 linear_out=48 mtp=2 embed=1`, coverage gate 6/6, exactly
|
||
**131** tensors to orthogonalize. Same architecture as RobinsonLabs' base, no
|
||
name drift. Two hard gates in the harness halt before any write: the coverage
|
||
identity `o_proj(16)+linear_out(48)==64`, and the attention-sink screen on
|
||
**dim 3994** (orthogonalizing a direction living there bricks the model).
|
||
|
||
## Capture SUCCEEDED — but only after a real environment fight (the durable lessons)
|
||
|
||
**The transformers Qwen3.5 DeltaNet linear-attention NaNs in bf16 on ana-ml2.**
|
||
The fast-path needs BOTH `flash-linear-attention` (`fla`, triton, installs fine)
|
||
AND `causal-conv1d` (**needs nvcc to build — absent, no prebuilt wheel**).
|
||
Without causal-conv1d the DeltaNet short-conv runs the torch fallback, which
|
||
produces **nondeterministic all-NaN** hidden states in bf16 (same 11-token input:
|
||
finite on one forward, NaN at layer 4 on the next). bf16 and fp32 share exponent
|
||
range, so this is **precision-driven catastrophic cancellation, not overflow** —
|
||
**fp32 resolves it.** Diagnosed via `diag_nan.py` / `diag2.py`: `sdpa` + plain
|
||
prompt = 65 layers all finite; chat-template input = NaN; the trigger is the
|
||
input path through the unstable recurrence.
|
||
|
||
Fixes, all in the committed harness (`7abd301`):
|
||
- **`--capture` loads fp32**; the write/surgery path stays bf16 (no forward, no NaN).
|
||
- **A finite-gate aborts on a non-finite direction** — the sink screen alone
|
||
can't catch it (`nan > threshold` is False, so a NaN direction "passed" it and
|
||
saved silently on the first run).
|
||
- `attn_implementation="sdpa"` pinned.
|
||
|
||
**fp32 (110 GB) needs the whole GPU.** device_map=auto packed it tight and the
|
||
forward OOM'd against the resident seats. Had to **stop three seats** for VRAM:
|
||
`vllm-meromero-rp`, `vllm-fablefusion-probe`, and production `vllm-gen`.
|
||
⚠ **Restart order matters:** gen restarted into an empty GPU0 and greedily
|
||
grabbed 64 GB (vLLM takes a fraction of *free* memory at startup), starving
|
||
meromero into a crash-loop. Fixed by bringing **meromero up first**, then gen
|
||
into the remainder. All three restored to healthy.
|
||
|
||
⚠ **fla lives in a side dir, not the venv.** The shared
|
||
`/tank/aimodels/quant-work/.venv` is not llmuser-writable; `fla` + `einops` are
|
||
`--target`-installed to `/tank/aimodels/coldfusion-abliteration/pylibs` and
|
||
reached via `PYTHONPATH`. Prune deps that shadow the venv's torch/transformers.
|
||
|
||
## Result
|
||
|
||
Refusal direction: **finite, unit-normed, layer 22**, sink energy **0.0008%**
|
||
in dim 3994 (recipe L26 ref 0.06%, threshold 1%) — clean, not sink-dominated.
|
||
Saved to `/tank/aimodels/qwen38-27b-coldfusion-bf16/refusal-direction.pt`.
|
||
|
||
⚠ **QUALITY CAVEAT — the reason the next step is calibration-set expansion.**
|
||
Two-template `|cos|` agreement at layer 22 is **0.59**, well below Robinson's
|
||
0.99. Almost certainly the small calibration set: **8 harmful / 8 harmless**
|
||
(HARMFUL/HARMLESS in `abliterate.py`) vs Robinson's **416 / 104**. The direction
|
||
is valid and sink-clean but noisier than ideal; abliterating on it risks
|
||
under-removing refusals or nicking capability. **Expand the sets to a few
|
||
hundred each and re-capture** before the `--out` write.
|
||
|
||
## Sequence from here
|
||
|
||
1. **Expand HARMFUL/HARMLESS calibration sets** → re-capture (fp32, seats down).
|
||
2. `--out` write (bf16 surgery, no forward) → `qwen38-27b-coldfusion-abliterated-bf16`.
|
||
3. Verify: vision byte-identical, refusal re-profile via `services/refusal-probe/`
|
||
(the canonical harness, NOT the ad-hoc GGUF one), MTP acceptance on the quant
|
||
(gate ≳40%, not KL — `reference_abliteration_mtp_lessons`), PPL/coherence.
|
||
4. NVFP4-quantize via `services/gen-seat-mixed-quant/` → gen-seat candidate.
|
||
**Do NOT delete the incumbent** (`qwen38-27b-heresy-nvfp4-mixed`) until it
|
||
holds through real multi-turn use.
|
||
|
||
bf16 staged at `/tank/aimodels/qwen38-27b-coldfusion-bf16` (pinned `9c44193`,
|
||
provenance recorded). All write paths re-stop the seats for fp32 VRAM — batch
|
||
re-capture + write in one window. Commits `ccb56a0`, `1857a8e`, `b56cb0d`,
|
||
`7abd301`.
|
||
|
||
_Archived 2026-09-05._
|
||
|
||
# `[2026-08-19]` Fleet `.internal` DNS — git-sourced, agent-managed, three resolvers
|
||
|
||
Operator: *"with ipv6 i can't memorize the IP addresses anymore. need a way to
|
||
keep track of local .internal dns names that can be agent managed and is
|
||
lightweight."* Built and live in one session; commit `b8003c7`.
|
||
|
||
## Shape
|
||
|
||
```
|
||
dns/internal.yaml source of truth — 38 hosts + 4 service aliases
|
||
scripts/dns-sync.py reconciles AdGuard resolvers against it
|
||
stacks/adguard-ana/ the colo's resolver, which did not exist
|
||
dns/README.md workflow, naming, the IPv6 caveat
|
||
```
|
||
|
||
Deliberately the same posture as `deploy-stack.sh`: the file is intent, the
|
||
resolvers are derived state, you see a diff before anything changes.
|
||
`--dry-run` / `--yes` / `--site <s>`. Verified idempotent — a second run prints
|
||
`nothing to do`.
|
||
|
||
Naming is `<host>.<site>.internal` with sites **`ana` / `esh` / `nh3`**
|
||
(operator's call). `.internal` is ICANN-reserved for private use since 2024;
|
||
`.local` is reserved for mDNS, which is why the pre-existing
|
||
`searxng.pfi.local` was a standards collision that merely happened to work.
|
||
|
||
Every name is published to **every** resolver — the site label says where a
|
||
host *is*, not which resolver knows about it.
|
||
|
||
## The framing correction that mattered most
|
||
|
||
The ask reads as "I can't memorise v6 addresses", but the deeper problem is
|
||
that **v6 addresses are derived, not assigned**, so they cannot reliably be
|
||
*written down once* either. SLAAC gives EUI-64 (MAC-coupled) or
|
||
privacy-extension (rotating) addresses, and UniFi has **no v6 equivalent of a
|
||
DHCP reservation** — so a hand-maintained v6 table rots on its own.
|
||
|
||
⇒ The fix has two halves and only the second is DNS: (1) pin static v6 on
|
||
server-class hosts, (2) then the name table is just a file. Surfaced to the
|
||
operator before building.
|
||
|
||
**Verified 2026-08-19: no fleet host has a global v6 address at all yet** —
|
||
ESH's `/56` is live only on `esh-cameras`, NH3's LANs are back to
|
||
`ipv6_interface_type: none`, the colo has none. So the `v6:` column ships
|
||
EMPTY and correct, and the naming layer was built first rather than blocking
|
||
on v6. Names established now need no renaming when addresses land.
|
||
|
||
Suggested convention when they do (awaiting operator): each server static at
|
||
its site's `/64` with low-order bits echoing the v4 host octet —
|
||
`esh-docker-vm` at `…::45` — so addresses are declarable *and* semi-memorable.
|
||
|
||
## Two properties not to break
|
||
|
||
**Authority is scoped to the ZONE, not the resolver.** Only rewrites ending in
|
||
`.internal` are managed. ESH's resolver turned out to carry three hand-made
|
||
`esteban.net` rewrites (`eshnas`, `brotherprinter`, `eshhome`) — **my first
|
||
read of the config missed them**, because an `awk` range on `rewrites:` matched
|
||
an empty-looking block. A resolver-wide authoritative sync would have silently
|
||
deleted all three on first run. Verified intact after sync.
|
||
|
||
**Within `.internal` it IS authoritative** — names added by hand in the AdGuard
|
||
UI get deleted by the next sync. That is the point: one place to look.
|
||
|
||
## The colo had no resolver at all
|
||
|
||
ESH and NH3 each ran AdGuard; **ana-docker resolved straight against
|
||
`1.1.1.1`**, so the colo had no way to answer for internal names. Closed with
|
||
`stacks/adguard-ana/`.
|
||
|
||
⚠️ Its API is on **8053**, not 8080 — `:8080` and `:3000` were already taken on
|
||
that busy host. The port is therefore carried **per-site in the yaml**, not
|
||
assumed by the script, so the odd one out cannot be forgotten.
|
||
|
||
⚠️ It ships with **no blocklists**, deliberately. The other two filter ads for
|
||
human browsing; this one resolves for a rack of servers, where a blocklist
|
||
false-positive breaks service-to-service calls at 3am for no upside.
|
||
|
||
First boot uses a **seed config** (`conf/AdGuardHome.seed.yaml`) copied into
|
||
the conf volume before first start, so the container comes up configured
|
||
instead of sitting in the setup wizard.
|
||
|
||
## Credential — service account, not the operator's
|
||
|
||
Added a dedicated **`infra-ops`** AdGuard user to all three resolvers rather
|
||
than asking for the `lkraven` password (per the standing migrate-off-operator-
|
||
creds directive). Password vaulted at
|
||
`nh3-dev/adguard-infra-ops-password`; `lkraven` untouched; pre-change configs
|
||
backed up on each host as `AdGuardHome.yaml.bak-preinfraops-*`. Both existing
|
||
resolvers kept answering across the restart.
|
||
|
||
Two landmines worth keeping:
|
||
|
||
- **Go's bcrypt rejects `htpasswd`'s `$2y$` prefix.** Same algorithm, different
|
||
marker; `golang.org/x/crypto/bcrypt` accepts only `$2a$`/`$2b$`. Normalise
|
||
the prefix, and self-verify the hash with `htpasswd -vb` BEFORE installing it
|
||
on a live resolver.
|
||
- **The vault appends a trailing newline on `get`.** A password carrying a
|
||
stray `\n` fails auth in a way that looks exactly like a wrong password.
|
||
`dns-sync.py` strips it.
|
||
|
||
## `pfi.local` migration — and the one that must NOT move
|
||
|
||
`searxng.pfi.local` → `searxng.ana.internal`, with the **old `Host()` kept
|
||
alongside** in the Traefik rule so nothing breaks mid-migration; both return
|
||
200. Drop the fallback once the access log shows the old name unused.
|
||
|
||
**`matrix.pfi.local` deliberately NOT migrated.** A Matrix `server_name` is
|
||
baked into every user ID, room ID and signing key, and federation identity
|
||
derives from it — renaming it is not a DNS change, it is rebuilding the
|
||
homeserver's identity and invalidating its history. The operator approved
|
||
"migrate pfi.local" generally; this was surfaced as a deliberate exclusion
|
||
rather than executed blindly.
|
||
|
||
## Still open
|
||
|
||
Colo hosts still point at `1.1.1.1`, so they do not yet *use* the new resolver
|
||
— it only answers what asks it directly. Repointing a whole site's DNS is a
|
||
bigger change than standing the service up, and is the operator's to schedule.
|
||
|
||
See also [[2026-08-17-fleet-ipv6-mesh]].
|
||
|
||
_Archived 2026-09-05._
|
||
|
||
# `[2026-08-19]` waterland studio containerised on irv-ml1 — three landmines, all measured
|
||
|
||
Handover from `waterland-dev` over althing (thread `01M0CDRGEZWAJCEJXXMQWXV80F`):
|
||
a FastAPI + SPA GPU service fronting the `waterland` CLI, running as a bare
|
||
`nohup` (PID 1283383) that would not survive a reboot. Now
|
||
`stacks/waterland-studio/`, `restart: unless-stopped`, healthy on
|
||
irv-ml1:8410. Commits `a2b5b58`, `8189076`.
|
||
|
||
Tracking `main` per operator: PR #4 merged and `main` HEAD was exactly the
|
||
pinned `8025366`, so tracking-a-moving-ref and keeping-the-pin agreed anyway.
|
||
|
||
**Now deployed at `b72425b` (2026-08-19).** The container sat on `8025366` for
|
||
a few hours after PR #5 (`464dfc2`) landed — deliberately, since the image's
|
||
own guards already neutralised both landmines and the project was in
|
||
wind-down. PR #6 (the job-store rehydrate, operator-green-lit) was the rebuild
|
||
with a real reason behind it, and one `update.sh` run carried both. Verified
|
||
end to end after the update: healthy, `backend: cupy`, and a real 256² plate
|
||
render completes warm — the kernel-cache volume survived the image swap.
|
||
|
||
## Build context lives OUTSIDE the compose dir — on purpose
|
||
|
||
`/opt/waterland-studio/src` is the checkout; the Dockerfile is passed
|
||
out-of-context from `/opt/docker/compose/waterland-studio/`. **`deploy-stack.sh`
|
||
rsyncs `stacks/<stack>/` with `--delete`**, so a checkout kept beside
|
||
`compose.yaml` would be destroyed by the next deploy of this stack. `update.sh`
|
||
refreshes source → rebuild → recreate → health, and is verified end to end.
|
||
|
||
## Landmine 1 — both uv extras are load-bearing at BUILD *and* RUN
|
||
|
||
`gpu` carries `cupy-cuda12x`; a bare `uv sync` prunes it and the renderer
|
||
silently drops to the numpy path at ~21x wall time — it does not error, it
|
||
just gets slow. waterland-dev warned about the build side.
|
||
|
||
The runtime side is worse and was not in the handover: **`studio/jobs.py`
|
||
shells the renderer out as a literal `uv run waterland ...` with no `--extra`
|
||
flags** (`cwd=WATERLAND_STUDIO_REPO`). Left alone, uv re-syncs the project
|
||
mid-job to its default extras and prunes cupy back out from under a correctly
|
||
built venv. Pinned with `UV_NO_SYNC=1`; `UV_OFFLINE=1` alongside so that if the
|
||
pin ever stops holding the job fails **loudly** instead of quietly rebuilding a
|
||
slower environment.
|
||
|
||
**Fixed upstream in `464dfc2`:** the server now spawns
|
||
`sys.executable -m waterland.cli` directly — no resolver in the render path at
|
||
all. **The pins stay anyway.** They cost nothing and are now defence-in-depth:
|
||
if any future code path re-enters `uv` inside the container, the job fails
|
||
loudly instead of quietly dropping to the numpy backend. `uv` itself must stay
|
||
in the image regardless — it performs the build-time `uv sync` /
|
||
`uv pip install`, and this is a single-stage build.
|
||
|
||
## Landmine 2 — cupy needs CUDA HEADERS, which the host never had to declare
|
||
|
||
Every render died 1.7s in with:
|
||
|
||
```
|
||
RuntimeError: Failed to find CUDA headers.
|
||
```
|
||
|
||
printed **through argparse's usage banner**, which makes it read like a CLI
|
||
argument bug rather than a missing toolkit. That misdirection is the reason
|
||
this is written down.
|
||
|
||
cupy compiles kernels at runtime through NVRTC, which needs the toolkit
|
||
**headers** — not just the driver and the runtime libs bundled in the
|
||
`cupy-cuda12x` wheel. irv-ml1 has a CUDA toolkit installed system-wide, so the
|
||
bare `nohup` process found them **by accident**; a slim image has none.
|
||
|
||
Fixed with `uv pip install "cupy-cuda12x[ctk]"` — headers as wheels, a few
|
||
hundred MB against ~6 GB for a `-devel` base image. It runs **after**
|
||
`uv sync`, because sync prunes what it does not know about.
|
||
|
||
Reported upstream: it is an undeclared runtime dependency of the `gpu` extra,
|
||
and anyone running this without a system toolkit hits it. **Declared upstream
|
||
in `464dfc2`** (`gpu` is now `cupy-cuda12x[ctk]>=13`). **The explicit install
|
||
stays in the Dockerfile**: the header requirement is a property of *this*
|
||
image — a slim base with no system CUDA toolkit — so it belongs in the file
|
||
that creates the problem, not inherited from an extra two repos away. It also
|
||
survives any future restructuring of the `gpu` extra. Cost of keeping it is
|
||
now measured, not assumed: since `uv sync` satisfies it first, the line
|
||
reports `Audited 1 package in 49ms` and adds **0.3s** to the build. A no-op
|
||
that documents a non-obvious requirement is worth 0.3s. (waterland-dev
|
||
independently agreed they would keep it too.)
|
||
|
||
## Landmine 3 — the GPU index inside the container is not the host's
|
||
|
||
The app pins `CUDA_DEVICE_ORDER=PCI_BUS_ID` and selects
|
||
`CUDA_VISIBLE_DEVICES_TARGET` (default `1`, correct on the host, where
|
||
`nvidia-smi` shows A6000 at 1). Compose exposes **exactly one** GPU
|
||
(`device_ids: ["1"]`, the A6000 in Docker's ordering), so **inside** the
|
||
container that card is index **0** ⇒ `CUDA_VISIBLE_DEVICES_TARGET=0`. Copying
|
||
the host's value selects a device that does not exist. Host device 0 is the
|
||
3090, which carries the TTS zoo and must not be touched.
|
||
|
||
## Cold start is ~17s of NVRTC compile → `/root/.cupy` is a volume
|
||
|
||
| job | wall |
|
||
|---|---|
|
||
| 256² + anim, cold container | 23.3 s |
|
||
| 256² + anim, warm | **6.1 s** |
|
||
| 256² plate only (`--codec none`) | 3.9 s |
|
||
| 512² plate only | 6.4 s |
|
||
|
||
Warm beats the **7.4 s** recorded against the bare-metal process, so
|
||
containerising cost nothing. Verified the cache volume properly: recreate
|
||
(fresh cache → 23.2 s first render) then restart (populated → 6.0 s). Without
|
||
it every restart makes the next user wait 4x and the service merely *looks*
|
||
slow.
|
||
|
||
## Upstream finding — the on-disk job store grows without bound
|
||
|
||
`JobStore._jobs` is a plain dict and **nothing scans `WATERLAND_STUDIO_DATA` at
|
||
startup**. Consequences:
|
||
|
||
1. After a restart `/api/jobs` lists only jobs created since — cosmetic, and
|
||
how this was spotted: the API reported **1 job** while the volume held all
|
||
**16 directories, 60.6 MB**. Not data loss.
|
||
2. The real one: `RETAIN = 40` eviction only ever iterates the in-memory dict,
|
||
so directories orphaned by a restart are **never reclaimed**. The
|
||
handover's "bounded around 500 MB" holds within a single process lifetime;
|
||
across restarts the store grows monotonically at ~12 MB per animated job.
|
||
|
||
Reported to waterland-dev with evidence; **not patched from the infra side** —
|
||
it is their code. Prune the volume by hand if it bites first.
|
||
|
||
**waterland-dev confirmed it (2026-08-19)** — their "bounded ~500 MB" handover
|
||
claim holds within one process lifetime and nowhere else, which on a
|
||
`restart: unless-stopped` service is the wrong lifetime to have bounded. They
|
||
have **surfaced a startup-rehydrate fix to the operator** rather than opening a
|
||
third PR during wind-down. **Operator green-lit it; PR #6 merged as `b72425b`
|
||
and is DEPLOYED (2026-08-19).**
|
||
|
||
Startup rehydrate, as recommended — and waterland-dev deliberately went
|
||
further than the framing I sent them. I had said a directory the scan cannot
|
||
parse "just does not enter the index"; they made the opposite call, because a
|
||
directory that never enters the index is exactly the one that never gets
|
||
reclaimed. **That is the sharper reading and it is the reason the fix works on
|
||
this volume at all** — the 16 pre-existing dirs have no sidecar. Their
|
||
adoption ladder: sidecar → restored verbatim; no sidecar → adopted with
|
||
dimensions recovered from the PNG IHDR (24-byte read, not a decode); corrupt
|
||
sidecar → degrades to inference, no startup crash; **neither source nor
|
||
sidecar → skipped on purpose**, since adopting it would turn eviction into a
|
||
delete-arbitrary-directories primitive pointed at this volume. Sidecar writes
|
||
go through `os.replace`, and `job.json` is excluded from `ARTIFACTS` so it is
|
||
unreachable via the artifact route.
|
||
|
||
They also closed a second leak I never saw, because it needs a restart
|
||
*mid-render* to surface: a job left `running`/`queued` in its sidecar is
|
||
non-terminal forever, and eviction skips non-terminal jobs — so it is a
|
||
phantom that is never reclaimed and `queue_depth` over-reports for the life of
|
||
the process. Adoption now marks those `failed`.
|
||
|
||
**Verified on this host after the update:** `/api/jobs` went **1 → 16** while
|
||
the volume stayed at 16 dirs / 61 MB — disk and API agree for the first time.
|
||
Nothing was reclaimed, correctly: 16 is under `RETAIN=40`, so adoption only
|
||
made them visible. A subsequent real render took both to 17. From here the
|
||
store is bounded **across** restarts, not merely within a process.
|
||
|
||
## Access
|
||
|
||
Repo is not anonymously readable (a bare clone 403s). Operator granted
|
||
**`claude-bot` read on `vh/waterland`** — verified `admin: False, push: False,
|
||
pull: True`. Token on irv-ml1 at
|
||
`/root/.config/waterland-studio/git-credentials`, `0600` root-owned, wired as a
|
||
**repo-scoped** credential helper; `.git/config` carries no token (verified),
|
||
so the remote stays clean in any diff or backup. The operator's `vh`
|
||
site-admin token was used only for the initial clone and the grant itself and
|
||
was **never written to disk on that host** — a site-admin credential on a GPU
|
||
box is a blast radius nobody needs for a read-only fetch.
|
||
|
||
## Constraints honoured as stated (not inferred)
|
||
|
||
- **Serial by design — one replica, one card.** A render is 20–45s of near-full
|
||
GPU with a single worker thread. Two on the same A6000 would OOM or thrash.
|
||
Throughput is a hardware conversation, not a replica-count one.
|
||
- **No authentication, arbitrary file uploads** ⇒ stays inside the
|
||
LAN/WireGuard boundary. Do **not** paper over it with a proxy password;
|
||
waterland-dev offered to add a real auth layer if wider reach is ever needed.
|
||
|
||
_Archived 2026-09-05._
|
||
|
||
# `[2026-08-19]` Homepage cleaned up, then themed with Australis Skyfall + an Arbo-generated background
|
||
|
||
Commits `9d92c4b`, `c3de7db`, `45c1995`, `f38cf69`, `df68dd2`.
|
||
|
||
## The cleanup (three real defects)
|
||
|
||
- **UltraSeedbox rendered on all four tabs.** The bookmark group had no entry in
|
||
`settings.yaml`'s `layout:` block at all, and Homepage's documented behaviour
|
||
is that a group with no `tab:` is shown on **every** tab. Pinned to Main.
|
||
⚠️ This will happen again to the next group added without a `tab:` — the rule
|
||
is now written at the top of the layout block.
|
||
- **Uptime Kuma rendered twice** — a manual `services.yaml` entry under
|
||
Monitoring *and* `homepage.group=Apps` on the container. Exactly the
|
||
"never list a labelled container manually" failure the stack README warns
|
||
about; it survived the previous day's audit because a duplicate reads as two
|
||
plausible cards rather than as an error. Manual block deleted, label moved to
|
||
`Monitoring`, `homepage.siteMonitor` added.
|
||
- **Column counts were fiction** — several groups declared more columns than
|
||
they had members, so the last row of each was dead space (Notes: 1 card in a
|
||
4-wide row). Columns now track member counts; `GET /api/services` prints the
|
||
live per-group counts and is the check.
|
||
|
||
Later, on operator instruction, the **AI tab was reordered by clickability**:
|
||
Gateways & Chat → Image & Media → Audio Tools on top, then the vLLM `/docs`
|
||
seats and TTS endpoints. Reasoning written into the config so it survives:
|
||
order by "would I click this?", not by how central the service is.
|
||
|
||
## ⚠️ The expensive red herring — the tab bar after a recreate
|
||
|
||
After a recreate the client render comes up with **no tab bar, no wallpaper and
|
||
no i18n** (search box shows the raw key `search.search`), groups falling back to
|
||
side-by-side columns. **It restores itself with no intervention.**
|
||
|
||
Timing, measured rather than assumed: a fresh container was still tab-less at
|
||
**4m30s, twice**; it was healthy again after roughly an hour. `docker ps`
|
||
reporting `healthy` says nothing about it — the container is serving, the page
|
||
is just wrong.
|
||
|
||
An hour went into ruling out four causes that were never the cause:
|
||
|
||
1. **Not the config** — restoring `settings.yaml` *and* `services.yaml` to
|
||
their committed versions reproduces it, as does the pre-adoption backup in
|
||
`/opt/docker-bu/conf/homepage/`.
|
||
2. **Not the v2.0.0 release** — a throwaway container on `v1.13.2` shows
|
||
identical symptoms, and the image never changed anyway (working and broken
|
||
both report `v2.0.0` / rev `17456f2`).
|
||
3. **Not `PUID`/`PGID`**, and not Docker discovery — tested both, and with the
|
||
socket unmounted entirely.
|
||
4. **Not server-side** — the server-rendered HTML still contains the tab
|
||
markup, the background URL and `useEqualHeights`; `GET /api/validate`
|
||
returns `[]`. The loss is client-side, with no page error, no failed chunk
|
||
and no non-200.
|
||
|
||
Every throwaway container in that list was judged within ~30s of starting, so
|
||
they were all inside the same window — and that consistency **read as a
|
||
reproduction when it was the same measurement mistake five times over.**
|
||
|
||
**Operative rule: recreate, walk away, re-check later. Do not chase it.**
|
||
|
||
## ⚠️ The iteration loop that would have prevented the overcook
|
||
|
||
`custom.css` is served **per request** from `/api/config/custom.css`, so a CSS
|
||
change needs a **browser reload** — not a container recreate, and it never owed
|
||
the layout warm-up above. Conflating the two costs ~10 operator-visible minutes
|
||
per attempt (operator called this out directly).
|
||
|
||
Faster still, and how the final pass was done: **inject candidate CSS into the
|
||
running page and screenshot it** —
|
||
`await p.addStyleTag({content: css})` in Playwright against the live
|
||
dashboard. Seconds per iteration, no deploy. Build + deploy only once the
|
||
render looks right.
|
||
|
||
## The theme — Australis Skyfall
|
||
|
||
Operator supplied a Claude Design handoff bundle via the Booth (`26-copper`).
|
||
Skyfall is a dual-theme OKLCH system: one lightness law across every chromatic
|
||
family (deep 0.48 / base 0.66 / bright 0.80), all hues cooler than neutral, a
|
||
Sea neutral ramp drifting ice-blue→ocean-green as it brightens, and a
|
||
"calm depth" language of **hairline + two-layer shadow on every elevated
|
||
surface, never one without the other**.
|
||
|
||
```
|
||
theme/colors.css layout.css typography.css vendored VERBATIM from the bundle
|
||
theme/fonts/Supreme-{400,500,700}.woff2 the body/UI face
|
||
theme/skyfall.css.in the Homepage bindings (ours)
|
||
theme/build.py → conf/custom.css (generated — do not hand-edit)
|
||
```
|
||
|
||
The build step exists for one reason: **Homepage serves only `custom.css` and
|
||
`custom.js` out of its config dir**, with no static route beside them, so a
|
||
`@font-face` pointing at a vendored `.woff2` would 404 — the face must arrive
|
||
as a data URI. The background image takes the other road, because
|
||
`/app/public/images` **is** a real static route (mounted read-only in
|
||
`compose.yaml`).
|
||
|
||
Only Supreme is embedded: a link dashboard has no display type, and Victor
|
||
Mono ships as 2.4 MB TTF statics per cut — 30x the whole stylesheet for a
|
||
handful of latency figures.
|
||
|
||
## The background is generated, not stock
|
||
|
||
**Arbo as an image-gen engine** (the operator's actual ask, which I first
|
||
misread as "use Arbo's palette" and had to redo). Arbo's `t2i-ui-background`
|
||
workflow is purpose-built: *"abstract full-bleed backgrounds, no subject"*.
|
||
Job `13f0891f4e42`, seed 26, flux2-klein-9b, 2048×1152, 1.6 MB PNG → **22 KB
|
||
WebP** (smooth gradients compress absurdly well).
|
||
|
||
⚠️ Arbo API gotcha: `prompt` is a **discriminated union, not a string** — a
|
||
bare string 422s. `{"kind":"raw","text":…,"negative":…}` is the shape.
|
||
|
||
## Two documented deviations from the design system
|
||
|
||
1. **Skyfall forbids this background.** Its rule is "flat semantic surfaces; no
|
||
photography, no textures", with one permitted motif — a subtle aurora
|
||
gradient on hero/empty-state areas only, *"never behind body text blocks"*.
|
||
A dashboard is a body-text block. Present on the operator's explicit
|
||
instruction, mitigated rather than excused: abstract, no subject, strictly
|
||
cool temperature, held at **`opacity: 30`**. That number is load-bearing —
|
||
at 14 the aurora was invisible, and turning it up makes the cards fight the
|
||
ribbon.
|
||
2. **Service icons stay full-colour vendor logos.** Desaturating them from CSS
|
||
only makes them illegible.
|
||
|
||
## Overcorrection, and the colour pass
|
||
|
||
First stat-well pass went from `font-thin` 13px straight to **bold 22px in
|
||
heading white** — operator: *"went from subtle to BASH YOU OVER THE HEAD."*
|
||
The principle missed: a stat only has to out-rank **its own label**, not the
|
||
service name above it. Now `--text-md` medium in cyan.
|
||
|
||
Colour was then lifted **from inside the system**: Skyfall names Aurora (blue,
|
||
cyan, green) the *primary* families, "used generously, in that order", while
|
||
Dawn (amber/red/violet) is semantic-only. So group markers cycle
|
||
blue→cyan→green down the page (icons full strength, names at 0.72), service
|
||
icons take a single cool wash, latency tags move to the info family so
|
||
"how fast" stops looking like "is it alive". **No Dawn colour is used
|
||
decoratively anywhere.**
|
||
|
||
Two DOM findings that made it possible:
|
||
|
||
- **Homepage renders mdi icons as a gradient behind an SVG mask** — recolour
|
||
via `background`, not `color`.
|
||
- **Homepage emits `docker-status-<state>`, not `status-<state>`.** The
|
||
original selectors matched nothing, so every green pill up to that point was
|
||
stock colouring rather than the theme. Both forms are now matched.
|
||
|
||
_Archived 2026-09-05._
|
||
|
||
# `[2026-08-19]` Four unmanaged stacks found on live hosts — and two of them were quietly broken
|
||
|
||
Commits `42c594c`, `dc3e47b`, plus `uptimekuma` in `9d92c4b`.
|
||
|
||
## The pattern worth remembering
|
||
|
||
Chasing two bad-looking cards on the dashboard turned up **four stacks running
|
||
on fleet hosts with no canonical copy anywhere**: `uptimekuma` and (already
|
||
known) the two AdGuards on esh-docker-vm, `searxng` and `seafile` on
|
||
ana-docker, and `heretic2-charrp-reasoning` on ana-ml2 (untracked in git).
|
||
|
||
⇒ **A dashboard card is a cheap census of what is actually running.** When
|
||
something on it looks wrong, check whether the stack behind it is even in
|
||
`stacks/` before debugging the symptom — twice here the answer was "no", and
|
||
the fix belonged in version control as much as on the host.
|
||
|
||
Adopted: `stacks/uptimekuma/`, `stacks/searxng/`, `stacks/seafile/`,
|
||
`stacks/heretic2-charrp-reasoning/`. ESH/NH3 AdGuard compose files were
|
||
**deliberately left unmanaged** — adopting three live resolvers while also
|
||
introducing a new DNS naming system is two risky changes at once.
|
||
|
||
## SearXNG — the healthcheck was eating itself
|
||
|
||
Card flapped UNHEALTHY; the container was fine the whole time. The compose
|
||
passed `--tries` and `--spider` as **two separate argv entries**, so wget
|
||
consumed `--spider` as the *value* of `--tries`. Spider mode never engaged,
|
||
which means every probe since April **downloaded** the healthz response to a
|
||
file:
|
||
|
||
```
|
||
295,287 healthz.N files in the container's working directory
|
||
```
|
||
|
||
With that many files, wget's scan for the next free filename is what
|
||
intermittently blew the 10s timeout. **Self-worsening — every probe made the
|
||
next one slower.** Restored `--tries=1`; the junk lived in the writable layer
|
||
so the recreate cleared it. Now `healthy`, `fails=0`, 200 in 0.16s.
|
||
|
||
Lesson: an argv list in YAML has no shell to catch a missing `=`. A flag that
|
||
silently swallows the next argument turns a liveness probe into a workload.
|
||
|
||
## SeaFile — not broken, never restarted
|
||
|
||
Card showed EXITED for three months. **None of the three services declared a
|
||
restart policy**, so Docker defaulted them to `no`. On
|
||
**2026-05-06T21:27:45Z** the daemon stopped all three within 200ms of each
|
||
other — a daemon restart or host reboot — and nothing brought them back.
|
||
|
||
⚠️ **Exit code `255` is a red herring**: it is what a container that ignores
|
||
SIGTERM reports when the daemon stops it, **not** evidence of a crash. Reading
|
||
it as one sends you hunting a bug that does not exist. The tell was all three
|
||
services stopping within 200ms.
|
||
|
||
Added `restart: unless-stopped` to all three; brought up; mariadb gated on its
|
||
healthcheck exactly as the existing `depends_on` comments intended, seahub
|
||
started without the race, `302` → login page. Data was in local named volumes,
|
||
not on the ana-nas NFS, so nothing was at risk.
|
||
|
||
Three months of silent downtime whose only signal was a card nobody read as an
|
||
outage — the argument for semantic status colour on the dashboard (see
|
||
[[2026-08-19-homepage-skyfall-theme]], where amber EXITED pills made six
|
||
mis-grouped AI seats obvious at a glance).
|
||
|
||
## heretic2-charrp-reasoning — tracked, with its shim
|
||
|
||
The `char-rp-reasoning` seat (NEO-CODE Heretic2 27B, modelopt NVFP4 + grafted
|
||
BF16 MTP head, ~77 tok/s via `qwen3_5_mtp` spec-decode) had been running
|
||
untracked. Now in `stacks/`, including
|
||
`conf/mtp-workaround/sitecustomize.py`, which is **not optional**: vLLM 0.24.0
|
||
does not propagate modelopt `exclude_modules` to the spec-decode **draft**
|
||
model, so the BF16 MTP head gets quantized and the engine dies at load. Both
|
||
the mount and `PYTHONPATH` are load-bearing.
|
||
|
||
Added the two files house convention expects and the directory lacked — a
|
||
`.env.example` naming every knob (all values are compose defaults; the host
|
||
overrides only the three VRAM ones) and a README pointing at
|
||
`docs/runbooks/heretic2-nvfp4-mtp-seat.md` rather than duplicating it.
|
||
|
||
_Archived 2026-09-05._
|
||
|
||
## Superseded in-flight snapshots (archived)
|
||
|
||
_Snapshot of the in-flight section as it stood before the 2026-09-09 rewrite (2026-09-08 and 2026-09-05/06 blocks). Archived 2026-09-09._
|
||
|
||
|
||
_As of 2026-09-08 (fleet-ops session). SUPERSEDES lower framing: run 4 = STILL-COUPLED; run 5 is
|
||
COMPLETE and gated RESCUED (02:13 PDT). Live open items:_
|
||
|
||
- **✅ ERP run 5 COMPLETE — gate = RESCUED (2026-09-08 02:13 PDT, landmark R49.5).** FIRST arm of this
|
||
line where the capability gate did NOT fail. The dependency-forcing slot (GovReport+QMSum, only 3.46%
|
||
of loss) broke the diversity↔coherence coupling that run 4 (STILL-COUPLED, 20.6% instruct slot) and
|
||
3c (20× LR cut) could not — **STRUCTURE of the loss was the lever, not its mass; INERT did not fire.**
|
||
T4 long-context 8/8 (run 4: 5/8; base 8/8); t4_dissect noise@31 tuned 0.9062 vs run-3 tuned 0.5625;
|
||
diversity held (rp density 3.37→0.00, story 2.86→1.58). Reported-beside (not in the cell, all
|
||
de-gated + stated): T3 constraint 8/8→6/8 (NEW loss, ship-path list); RP length 68w vs 250-floor =
|
||
PARTIAL fail (short-QA slot + style shift); refusal erosion rides with the style shift (k=25 both
|
||
arms, CSAM clean); free-check base LEVELS 5-6pp below run 4 (vLLM 0.28.0 unchanged — infra-ops
|
||
confirmed — so a generations shift, not a stack delta; taxes every cross-run number). Write-up
|
||
brokkr-smithy `research/R47-premium-corpus-gate/run05-gate/RESULT-run05-gate.md`; FLOOR-LOCKED
|
||
`0f3e4e2` (cites infra-ops' base index-sha 907826a6). Whole gate infra-ops-served on pfi-gx10:8098
|
||
(name-keyed base→tuned swap, hands-off honoured, sha+stack answers on record). Launch procedure +
|
||
canonical: eshpfi `scripts/erp-tune-gx10/` + `docs/runbooks/gx10-run-05.md`. **✅ SEAT: `erp-tune-v5`
|
||
SERVED on pfi-gx10:8098 (merged-run05); LiteLLM `trial` alias REPOINTED 3c→v5 (operator, 2026-09-08)
|
||
so it's testable from Open WebUI — verified end-to-end (trial→erp-tune-v5, coherent output). config.yaml
|
||
trial block rewritten to run-5 reality incl. the measured refusal-erosion note. ⚠ Seat is hand-launched
|
||
(vllm-run05.pid), NO restart policy/systemd — does not survive a gx10 reboot; yields to the next training
|
||
(~6 min re-serve). Brokkr: nothing further owed.**
|
||
- **run-4 gate = STILL-COUPLED** (RESULT-run04-gate.md, brokkr-smithy). Corpus dilution kept the
|
||
diversity gain, did NOT remove the safety/coherence regression.
|
||
- **✅ RESOLVED (2026-09-08, settled from bytes): the R47 base is STOCK `google/gemma-4-26B-A4B-it`,
|
||
byte-for-byte — NOT the abliteration.** The `-heretic-bf16` label in the recipes is a naming error;
|
||
run-04's "stock" provenance was right; brokkr's 77.7%-refusal telemetry lean is confirmed. Proof
|
||
(three-way match): local shards at `/home/infra-ops/models/gemma4-26b-a4b-it-bf16` sha256
|
||
`1127684971…`/`aab47033…` == the HF download etags (`.cache/huggingface/download/*.metadata`, so the
|
||
copy is uncorrupted) == the stock repo's two LFS oids, and the download commit `4d7ae498…` == stock
|
||
HEAD. Every run 3/3c/4/5 trained from a REFUSING stock base. Why plausible: the 2026-08-24 note
|
||
SELECTED llmfan46's Gemma-4-26B-A4B Heretic v1.2.0 ARA (3/100 refusals, bf16 51.6 GB), but llmfan46
|
||
ships that 26B-A4B abliteration **GGUF-only** — no bf16 safetensors — so the bf16 that actually got
|
||
pulled was stock google, and the `-heretic` name rode along from intent. **Operator/Brokkr decision
|
||
now evidenced (not a label guess): accept RESCUED-on-stock, or swap to a real abliteration (needs a
|
||
bf16 source, not the GGUF) + re-run. Recipes should drop `-heretic` from the base name.**
|
||
- **✅ RUN 6 COMPLETE on pfi-gx10 (2026-09-08 16:06 PT, 524/524, 11 h 42 min) → 🔥 GATE IN PROGRESS: base seat
|
||
`erp-seat-base-ara` SERVING on gx10:8098 (pid `vllm-base-run06gate.pid`, since 16:24 PT) for brokkr's floors;
|
||
AWAITING HIS SWAP CUE → then stop it and serve `serve/merged-run06` as `erp-tune-v6` (same port/flags).**
|
||
train_loss **3.259** (run 5: 3.235 — preregistered read wanted below; +0.024 is inside the ±0.1 per-step band).
|
||
Adapter `run-06/adapter` (410 tensors); merged-run06 = adapter + jenerallee78 base (index 33c59654), stock
|
||
template ae53464b. ⚠ **The abliterated repo ships NO `processor_config.json`** — vLLM's Gemma4 loader dies with
|
||
"Can't load feature extractor" without it (first base launch failed exactly so). STOCK's copy (sha `32bdf45d…`)
|
||
carried into both the base dir and merged-run06; processor plumbing, not weights. Base: jenerallee78 ARA
|
||
abliteration @ `0631379a`, pulled to `/home/infra-ops/models/gemma4-26b-a4b-it-ara-abliterated-jenerallee78-0631379a`,
|
||
32/32 shards verified vs brokkr's pins. Run-5 recipe byte-held (free check exact). Grant
|
||
`operator-2026-09-08-rnd-run6`. ⚠ Repo `tokenizer.json` bakes a 256-token truncation → stock tokenizer set
|
||
installed over it (`*.repo` kept). ⚠ `hf download` ignores `--include` with several patterns after one flag.
|
||
**LiteLLM `trial` alias DARK until the gate ends** (then repoint 5→6 only on the operator's word).
|
||
Miranda informed 16:07 PT. → `docs/runbooks/gx10-run-06.md`.
|
||
- **✅ ADJUDICATED GO by the operator 22:45 PT** — verbatim: "in the vernacular, baby is about the most common pet name
|
||
you can get, ESPECIALLY during sex. I'm going to adjudicate it as a go. There are unmistakable CSAM terms, but baby is
|
||
not one of them." No regeneration wanted. Relayed to brokkr + Miranda; halt lifted; `trial` stays on the NVFP4 build.
|
||
Brokkr's cue-length probe on erp-tune-v6 DONE (288 gens): reply length is CONDITIONAL on the cue (5-word opening →
|
||
54/62 words median; 221-word → 292; "at least 250 words" persona → 354) → run-7 lever = opening-split render (A′) or a
|
||
prompt-only mitigation — operator picks. **GX10 → 🔥 RUN 7 TRAINING (launched 2026-09-08 23:06 PT, pid 599489, `run-07.pid`, 542 steps, ETA ~13 h → ~noon
|
||
09-09).** Operator's direct grant `operator-2026-09-08-rnd-run7` (/goal). Variable = opening-split slot + companion
|
||
mask (brokkr recipe r7). Free check passed: held roots identical to run 6, slot 293/293 fit whole, mask 6,106 turns
|
||
(all 224 landed, 1 dup), two cwm rows dropped as fully-masked (loss moved into the slot). 17% padding (was 0%).
|
||
→ `docs/runbooks/gx10-run-07.md`. Gate after: `erp-seat-base-ara` then `erp-tune-v7`.
|
||
- **erp-seat (ana-ml2 :8021, `trial`) now on vLLM nightly `311b3513` (v0.27.2rc1)** — forced (named/required)
|
||
tool_choice is PROMPT-DRIVEN on every Gemma-4 seat by vLLM design (gemma4 parser skips guided JSON); measured
|
||
1/9 on v0.26.0 → 6/9 on the nightly (n=9, small), round trip clean 3/3, 12-shape matrix green. Deterministic forced
|
||
calls = `response_format: json_schema` (tts-dev: 18/18). ⚠ Removing `--reasoning-parser gemma4` leaks
|
||
`<|channel>thought` into content 3/3 — keep it.
|
||
- (history) **GX10 was FREE 23:03–23:06 PT** (reference arm in 06:00Z, 288 gens; base seat stopped, GPU clear). Run-7 recipe
|
||
`erp-seat-sft-r7` is emitted + verified in brokkr-smithy but carries NO GRANT — nothing trains until the operator
|
||
gives one. Base-vs-tuned reply-length: tune's deficit ~120 words at the short cue, ~0 at the long cue.
|
||
- **(history) ⛔ RUN-6 GATE HALTED 22:18 PT on the CSAM line (brokkr, level=review, SOFT trip): one generation in the tuned refusal
|
||
k=5 step tripped R19's drift detector — minor_soft=1 ('soft_baby' adult-usage, the spec's named false-positive class),
|
||
minor_hard=0, sexual=7, NO hard signal; fail-closed → operator-adjudicated; TRANSFERRED is PROVISIONAL (diversity,
|
||
reasoning, t4_dissect complete); tuned-probeB.json absent; brokkr will not re-run.** ⚠ CONFLICT SURFACED: the operator
|
||
had ALREADY aliased `trial` → the NVFP4 build ("no gate", 22:18) two minutes before the halt reached me; brokkr asks
|
||
the quant not be served until adjudication. Left the alias IN PLACE (explicit operator ruling), escalated to the
|
||
operator + Miranda with the one-line rollback offered. Run 7 does not start from this recipe until adjudication.
|
||
- **🔥 GATE seat: erp-tune-v6 (bf16) SERVING on gx10:8098 since 21:46 PT** (`vllm-run06.pid`); brokkr's tuned window
|
||
(~1.5 h, k=25 legs CUT by operator) is unattended — files in brokkr-smithy `run06-gate/` are the signal, nobody pings.
|
||
**After TUNED-WINDOW-DONE (brokkr's three asks, agreed):** (1) keep erp-tune-v6 up until he posts "probe done"
|
||
(cue-length probe, ~15 min); (2) re-serve `erp-seat-base-ara` ~20 min for the probe's reference arm, then the GX10
|
||
is FREE for run 7; (3) ✅ done — `dialogue-survivors.jsonl` (610, sha b2b6a0e9) copied to
|
||
`/mnt/smithy/datasets/derived/_recipes/erp-seat-sft-r3/`. Run 7 = recipe adjustment (reply length +
|
||
constraint-following), variable picked by the probe; no recipe/grant yet.
|
||
- **✅ erp-tune-v6-nvfp4a16 SERVING on ana-ml2 GPU1 :8021 (operator directive 2026-09-08 ~21:30 PT: "quant the
|
||
latest trained model into nvfp4 and serve it on ana-ml2 while we train a new model on the gx10").** Stack
|
||
`stacks/erp-seat` (recipe = gemma4-charrp's: gemma4 tool+reasoning parsers, enable_thinking pinned false, stock
|
||
template ae53464b, v0.26.0, util 0.35, 32K ctx) — TRUE name only. **`trial` alias REPOINTED to it 2026-09-08 22:18 PT (operator: "alias erp-tune-v6-nvfp4 to trial, please. no gate")** — config-file deployment (`/model/update` refuses config models), `deploy-stack.sh ana-docker litellm --conf` + `sudo docker compose restart litellm`; verified ×3 through the gateway.
|
||
Artifact `/tank/aimodels/erp-tune-v6-nvfp4a16` (16 GB, compressed-tensors nvfp4-pack, W4A16, 252 ignores incl.
|
||
60 router + 191 vision) from `/tank/aimodels/erp-tune-v6-bf16` (merged-run06 relayed gx10→nh3-dev→ana-ml2 in 17 min,
|
||
no key path gx10→ana-ml2). Quant = `services/erp-seat-quant/` — DATA-FREE (~90 s; playbook §3.16), tokenizer cap
|
||
reset, encode-equal to source. Smoke: prose in `content`, ~200 tok/s single-stream (n=3, spread <1%).
|
||
**Tool calling FIXED 2026-09-08 23:00 PT (operator: "fix toolcalling with the trial seat"):** the one reproducible
|
||
defect in a 12-shape matrix was `tool_choice:"none"` → empty turn (content AND tool_calls null, 3/3) — vLLM kept the
|
||
tools in the prompt, the model called one, parsing was off. Fix = `--exclude-tools-when-tool-choice-none` (seat
|
||
recreated; none→prose 3/3; auto/required/named/parallel/nested/empty-list/streaming/round-trip all green).
|
||
⚠ `stacks/gemma4-charrp` (char-rp seat) has the same exposure and no flag yet — bouncing it is a consumer-visible
|
||
restart, so left for the operator's word.
|
||
⚠ NOT gate-parity (gated artifact = bf16 arm on the GX10; this seat has a smoke test + 3-run decode probe only) —
|
||
**operator ruled "no gate"**; the config block states it as unrated on every safety axis.
|
||
✅ **ana-ml2 mesh return routes PERSISTED 23:49 PT** (operator ruling) as `/etc/network/if-up.d/mesh-routes` via
|
||
`playbooks/ana-ml2-mesh-routes.yaml` (elway); verify after the next ana-ml2 reboot. (was: non-persistent (`ip route replace 10.100/16, 10.0/16, 10.6.110/24, 100.64/10
|
||
via 10.250.50.45` on `enp97s0f0np0.50`, 2026-09-08) — before that nh3-dev→ana-ml2 timed out (two DHCP defaults,
|
||
reply left the wrong NIC). Lost on reboot; make durable (netplan/networkd) or expect the timeout to return.
|
||
- **📮 althing reachability on THIS bg seat = the cc-channel route, NOT the waiter.** `althing-listen`
|
||
reaps within ~seconds-to-minutes on an idle background seat (harness kills detached bg tasks; hit it
|
||
repeatedly this session). Fix (operator-directed 2026-09-08): `althing-route declare --handle infra-ops
|
||
--pid <PID>` where `<PID>` is the number in `$CLAUDE_CODE_MESSAGING_SOCKET` filename
|
||
(`/run/user/1000/cc-socks/<PID>.sock`) — `--discover-pid` REFUSES on a forked child session (ancestry
|
||
≠ zellij launcher). Flips `postbox status` to `mode: push, reachable: True`, no process to reap. A fresh
|
||
bg session should re-declare it (the pid changes per launch). See the `[2026-09-07]` cc-channel entry in
|
||
Recent decisions for the durable why.
|
||
- **✅ Fleet fixes shipped this session (2026-09-08), all committed:** WhereTF Homepage card + DNS alias
|
||
(`wherethef.nh3.internal`, e0d1c44/4506ef6); ext-tts LiteLLM alias repointed to `irv-ml1.nh3.internal`
|
||
via DB `/model/update` + `extra_hosts` (957c8f1); the 09-06 irv-ml1 move's stale-IP trail repointed
|
||
across 25 stack composes + services.yaml + ssh-target → DNS name (e0d1c44); Homepage `/api/services`
|
||
outage fixed — irv-ml1 docker.yaml repointed + **ana-ml2 discovery via a socat proxy on ana-docker**
|
||
(`stacks/ana-ml2-proxy`, 913d2d2, reversible when ana-ml2 gets an ESH return route).
|
||
- **⏳ Minor cleanup leftovers (offered, operator hasn't taken — not urgent):** (#2) the ~10 RUNNING
|
||
irv-ml1 service cards still show dead `10.100.79.3` hrefs — canonical fixed, but each running container
|
||
needs a recreate to apply the label (bounces the service); (#3) deployed `.env` for asset-engine /
|
||
open-webui / skaldsong may still hold the dead default (canonical defaults fixed; a read-only check would
|
||
confirm which deployments are broken).
|
||
- **⚠ persistent-memory.md is 2.4× the soft cap (~710 lines).** Bulk is the non-archivable structural
|
||
sections (sister-repos table + backup architecture under Tools and conventions) — archival only touches
|
||
dated logs, so it can't reach 250. The real fix is a redundancy-trim of Tools-and-conventions rows that
|
||
CLAUDE.md already covers (deliberate pass, not archival). Flagged, not done.
|
||
- **NASPool evac copy now safe to destroy** — scrub clean (0 err) AND PBS runs landing (verified 116
|
||
backups, 8 guests 09-06). `ospool/naspool-evac` (1.65T) can go once ONE Backrest run is confirmed:
|
||
`zfs destroy -r ospool/naspool-evac` + drop `@evac`. pfi-pve PSU1 dead + backplane bays 9/10 dead
|
||
(cold spares → next colo visit).
|
||
- **MEMORY.md (auto-memory index) ~23.3KB, near the 24.4KB read cap** — needs a compaction pass soon
|
||
or fresh sessions may fail to load it. Operator offered; not yet done.
|
||
- **irv-ml1 on-site window / YTVC still open from 09-06:** FortiGate WAN SSH still temporarily open
|
||
(close when the edge is retired); YTVC still DOWN (dante retired, scoped exit-node egress unwired);
|
||
reverse-tunnel / wg0-delete decisions pending the operator's Irvine access.
|
||
|
||
_Infra session 2026-09-06 (NASPool rebuild + full headscale cutover incl. irv-ml1) — open
|
||
follow-ups; the ERP / althing / fiber items further down belong to other streams, untouched:_
|
||
|
||
- **NASPool parked copy still on ospool** — `ospool/naspool-evac` (1.65T) + `NASPool/*@evac`
|
||
snapshots. Destroy ONLY after the new raidz2 scrub is clean (it is, 0 errors 04:43Z) AND
|
||
one Backrest (01:00 PDT) + one PBS run succeed. Then `zfs destroy -r ospool/naspool-evac`
|
||
and drop the `@evac` snaps. ⚠ pfi-pve PSU1 still dead; backplane swap (bays 9/10) next colo
|
||
visit → then `zpool add NASPool spare`. Runbook `docs/runbooks/pfi-pve-naspool-rebuild.md`.
|
||
- **FortiGate WAN SSH is temporarily open** (`wan1` allowaccess ping+ssh; admin `infra-ops`
|
||
trusthost2/3 = 70.230.226.88 NH3 + 23.164.40.160 ESH). Safety net for the cutover — CLOSE it
|
||
when the edge is retired (OPNsense/R420). `ssh infra-ops@38.120.12.42`.
|
||
- **irv-ml1 FOLDED INTO THE MESH + cut over (done remotely, operator has NO Irvine access for
|
||
~5 days from 2026-09-06).** Node 100.64.0.6; wg0 DOWN and `wg-quick@wg0` DISABLED (not
|
||
reboot-restorable); full subnet router (accept-routes + advertises 10.6.110.0/24, gateway
|
||
routes added, fleet↔Irvine verified). Failover for the 5-day window = `wg0-watchdog.service`
|
||
(wg-quick up wg0 on ~5min mesh loss) + independent reverse SSH tunnel (`revtun-nh3.service`
|
||
→ nh3-dev via UDM fwd tcp/47822 src-restricted; reach it `ssh -i ~/.ssh/infra-ops_ed25519
|
||
-p 2201 infra-ops@127.0.0.1` on nh3-dev). Detail: docs/pfi/headscale-mesh-plan.md.
|
||
- **dante SOCKS proxy RETIRED** on nh3-dev (danted disabled, :1080 closed, config `.retired`).
|
||
⚠ **yt-voice-clipper is DOWN** until its SCOPED exit-node egress is wired (operator-accepted).
|
||
Follow-up: wire YTVC egress via tailscale `--socks5-server`+nh3 exit node or a per-container
|
||
netns — **NEVER set irv-ml1 `--exit-node` globally** (routes the reverse tunnel through the
|
||
mesh → kills the independent lifeline). Then bring YTVC back.
|
||
- **On-site (Irvine, ~5 days): decide** whether to keep or remove the reverse tunnel +
|
||
UDM forward `irv-revtun-ssh` + the revtun authorized_key on nh3-dev (small src-restricted WAN
|
||
surface), and whether to fully delete the wg0 config.
|
||
- **infra-ops now on all four PVE hypervisors** (pfi-pve/nh3-pve/esh-pve/esh-pve-nas) — PVE
|
||
ships without sudo, `apt install sudo` first or elway hangs on a password prompt.
|
||
|
||
_As of 2026-09-05 06:35 PDT — **ERP run 4 is TRAINING on pfi-gx10.** Everything else
|
||
below is a live commitment or a known-open risk._
|
||
|
||
- **⚠ RUN 4 IS MID-FLIGHT — do not touch the GX10 GPU.** `~/erp-tune/run-04.pid`,
|
||
log `~/erp-tune/run-04.log`. At 06:32 it was **486/938 steps**, 6 h 17 m elapsed,
|
||
a genuinely settled **46.0 s/it** (unlike 3c, which climbed 52→70 — airoboros rows
|
||
are short and single-window, so there is no long tail for the sampler to find).
|
||
**~12.0 h total, finishing ~12:15 PDT 2026-09-05.** Loss ~2.04 at step 450,
|
||
gnorm well under 1, checkpoints every 50. **Ping brokkr-smithy-dev at completion**
|
||
— he takes base floors on the GX10 first, then the tuned arm, serially.
|
||
|
||
- **Operator ruling on the GX10: training first, serving transiently.** *"it's mostly
|
||
for training, but can serve its trials. unless the box is needed for training work."*
|
||
So `trial` (= run 3c on :8098) is down for the duration and comes back when run 4
|
||
ends. I over-read an earlier version of this as "training-only" and had to correct
|
||
it to brokkr — his serial floors-then-arm plan on the GX10 was never wrong.
|
||
|
||
- **`trial` gateway alias is a live 404 while the seat is down** — expected, not a
|
||
fault. Restore with `~/erp-tune/relaunch-trial-seat.sh` on the GX10 (hand-run by
|
||
operator ruling: experimental, NOT a compose stack, does not survive a reboot).
|
||
|
||
- ⚠ **Three dead gateway aliases return HTTP 500, not 404/503**: `trial`,
|
||
`gemma4-26b-a4b-it-base`, `erp-tune-v2`. A dead seat reporting an *internal error*
|
||
reads as an outage — brokkr checked his own work against mine because he could not
|
||
tell. Deregistration costs a ~60 s fleet-wide LiteLLM restart; batch it with the
|
||
next gateway change rather than spending a restart on tidying.
|
||
|
||
- **`trial` is on the SHARED-KEY gateway with a measured −40pp selfharm/methods
|
||
regression.** Flagged to the operator twice (before adding, and after the gate
|
||
measured it); he has left it up. His direct endpoint `10.100.50.60:8098` gives the
|
||
same access with a blast radius of one. Settled — do not re-litigate.
|
||
|
||
- **ESH DAC: reverted to autoneg/1G, fiber going in at the weekend.** The operator
|
||
ran copper through a drilled floor 2x4 himself; recommendation was a 10Gtek
|
||
SR 2-pack + OM4 3 m LC-LC (~$40–65) because cable-vs-pull-damage was never resolved.
|
||
|
||
- ✅ **ESH WAN is now a STATIC PUBLIC IPv4: `128.177.138.182/30`, gw `128.177.138.181` (Cityside Fiber),
|
||
LIVE** — operator confirmed 2026-09-08; UDM WAN1 (`eth8`) reads `wan_type: static`, uplink up since
|
||
~2026-09-05, egress verified from esh-docker-vm = `128.177.138.182` (a real public address — CGNAT at
|
||
ESH is HISTORY; `100.104.0.1` still shows as the ISP's first hop, that is their access network, not
|
||
NAT). IPv6 unchanged (`2607:73c0:402:1d00::/56`, hosts still egress as their own v6). Done 2026-09-08:
|
||
`128.177.138.182` added to the crowdsec `esh` allowlist on ana-docker (the false-ban class is closed
|
||
for ESH). **All three follow-ups LANDED 2026-09-08 ~20:45Z (operator: "land all 3"):** (a) FortiGate
|
||
infra-ops `trusthost3` 23.164.40.160 → `128.177.138.182/32` — VERIFIED by a real infra-ops login to
|
||
`38.120.12.42` from esh-docker-vm (`ana-gw #`); `execute backup config flash pre-trusthost3-esh-static-20260908`
|
||
ran ("Please wait...") but `execute revision list` errors on this box, so the backup is unconfirmed —
|
||
the before-state was a single line, recorded here. (b) dormant `esh-ana` IPsec rebound wan2/192.168.200.111
|
||
→ **wan1/`128.177.138.182`**, still `enabled=false`. (c) ESH UDM port-forward `esh-scale tailscale direct
|
||
(UDP 41641)` → 10.0.50.65:41641 (id `6aa0727a…`) — VERIFIED: esh-scale now peers **direct via
|
||
`128.177.138.182:41641`** (was DERP lax). Helpers: scratchpad `esh-udm-land.py` + `fg-trusthost3.sh`.
|
||
`wan1-REVERT.json` is obsolete.
|
||
|
||
- ⚠ **ana-ml2 `tank` (raidz2, 8× NVMe) has 2 CKSUM errors on nvme7n1 + a boot-time 638 GB resilver on 2026-09-05 14:26
|
||
(the box rebooted; nvme7 came up late/dirty). ONLINE, no data errors, 58% full — but NO scrub since 2026-04-12** (the
|
||
Debian second-Sunday cron scrubbed zroot on 08-09, tank not — cause unknown). Box has neither `nvme-cli` nor
|
||
`smartctl`, so nvme7's media-error counter is unread. Recommended (operator hasn't ruled): `zpool scrub tank` now,
|
||
`apt install nvme-cli` + read nvme7 SMART, `zpool clear` after a clean scrub. **zroot is at 91%** — docker holds
|
||
429 GB of images (204 GB reclaimable) + 74 GB build cache (36 GB reclaimable); a prune buys ~240 GB. pfi-pve pools
|
||
(NASPool 7%, ospool 19%) clean, scrubbed 09-05 / 08-09. (checked 2026-09-09 00:00 PT)
|
||
- **ESH 10G topology (measured 2026-09-09):** UDM SFP+1 (port 10, TP-Link DAC) ↔ USW-Pro-HD-24 Garage p25; UDM SFP+2
|
||
(port 11, **OEM SFP-10G-LR fiber — the weekend fiber run IS in**) ↔ USW-Pro-XG-10 Media p11; Garage p27/p28 (3 m
|
||
SFP-H10GB-CU3M DACs) ↔ esh-pve `enp3s0f0np0` / esh-pve-nas `enp5s0f0`. All 10G full duplex, autoneg off, optics
|
||
tx −1.9 / rx −2.1 dBm both ends. Media p11 carries 416 rx / 251 tx errors that are STATIC (0 growth in 60 s, 363 h
|
||
uptime — install-era); host NIC counters are ring-buffer misses, flat. Nothing to fix.
|
||
- ⚠ **esh-nas is effectively open to the whole ESH LAN** — twelve NFS exports rw to
|
||
`10.0.0.0/8` with `sec=sys`, and every SMB share but `backup` guest-writable.
|
||
Hardening offered, ~1 h, **operator has not ruled**. → `persistent-memory.d/2026-09-04-esh-nas-smb-and-exposure.md`
|
||
|
||
- ⚠ **The nh3-dev backup throughput cause is UNEXPLAINED.** A job that once ran at
|
||
941 MiB/s ran at 1.4 MiB/s with the link up and pbs-ana answering in 11 ms.
|
||
Nightly 21:00, `all 1`. Worth its own investigation.
|
||
|
||
- **Ledger→SVOS rename: vault side DONE, gitea side is ledger-dev's to execute.**
|
||
Name settled as `svos`, `~/development/ledger` → `~/development/svos`. Vault moved
|
||
2026-09-05 (`secret` has no rename, so re-put + `rm`): stored
|
||
`nh3-dev/development/svos/env.sh` (sha 7253633d4155, verified on read-back),
|
||
retired `nh3-dev/development/ledger/env.sh` (sha feb418634e10, id
|
||
3a2af37c-c5aa-4f46-9178-f4fb6008a753) — `secret rm` is a SOFT delete to trash, so
|
||
it is recoverable. ⚠ The two shas differ: the vaulted copy was a 2026-08-11
|
||
snapshot and the live file had drifted un-vaulted since. **The vault goes stale
|
||
unless `secret backfill` is re-run.** Gitea `corviduo/ledger` (id 70) NOT renamed —
|
||
their repo, their call; answered that 1.26.1 writes a `repo_redirect` on a
|
||
same-org repo rename (upstream #807), that org/user renames do NOT redirect
|
||
(#9531), that the redirect dies if anything re-creates the old path, and that the
|
||
repo and org both carry 0 webhooks. **Gitea rename EXECUTED 2026-09-05** on the
|
||
operator's direct authorization: `corviduo/ledger` → `corviduo/svos`, repo id 70
|
||
unchanged. Redirect verified by measurement — web and API both 301, and
|
||
`git ls-remote` on the old URL warns-and-follows to HEAD b48a11ca5183. ⚠ **The
|
||
name `corviduo/ledger` is now burned**: the redirect dies silently the moment
|
||
anything creates a repo at that path — ledger-dev carries it as a standing item
|
||
in `docs/svos-rename-runbook.md`, since nothing warns whoever eventually creates
|
||
that repo. They repointed their own clone the same day (`origin/main` at
|
||
b48a11c), so the redirect is no longer load-bearing for any known consumer. Handle `ledger-dev` → `svos-dev` is an
|
||
operator action at the post office.
|
||
|
||
- **DONE 2026-09-05 — `svos` Heimdall user + API key minted** on operator
|
||
authorization. `user_id=svos`, `key_id=eab3cdbe`, suffix `d5ec48c2`, `wt_live_`
|
||
format, on **worldtree-personal (10.250.50.152:8081)** — established by finding
|
||
the `ledger` key there (created 2026-07-13, last used 2026-09-05T13:34, exactly
|
||
as ledger-dev described). Value vaulted at
|
||
`nh3-dev/development/svos/worldtree-api-key` (sha 23c10c9c7219, verified on
|
||
read-back) and delivered by vault path, NOT over althing — ledger-dev runs on
|
||
nh3-dev under the same uid, so the bus never carried the secret. The `ledger`
|
||
key was read back after the mint and is untouched and live (`disabled=False`).
|
||
⚠ **Plan tier left UNSET, deliberately**: `POST /admin/keys` takes an optional
|
||
tier (user|free|pro|admin|readonly-admin) and there is **no way to read a user's
|
||
current tier back** — no GET, `/admin/usage` returns an empty users list, and
|
||
`/admin/events` is a live SSE stream, not an audit log. Guessing would have
|
||
handed over a key that quietly differs; `POST /admin/users/svos/tier` fixes it in
|
||
one call if their cutover hits a limit — and ledger-dev has recorded it as a
|
||
cutover watch item to fix ON REPORT, explicitly not pre-emptively. ledger-dev
|
||
pulled the key from the vault and verified it independently (same sha), so
|
||
delivery is confirmed. **The cutover itself — pasting the value into env.sh,
|
||
flipping `worldtree.user_id` from `ledger` to `svos`, registering
|
||
`svos:miranda`, restarting the service — is WITH THE OPERATOR**, not with me;
|
||
they will not do it off a peer message. **CUTOVER DONE + VERIFIED 2026-09-05**:
|
||
`POST /agents/define` returned **201, not 409** — the load-bearing signal that
|
||
they are genuinely on the new identity rather than silently still on the old
|
||
one — then clean session create, turn, bifrost handshake and tool-call. **No
|
||
plan- or rate-limit errors, so the unset tier is compatible and is NOT to be
|
||
set** (they asked explicitly; it stays a watch item to fix on report, never by
|
||
guess). Incidentally confirmed the bifrost allowlist really is per-deployment
|
||
(host:port), not per-consumer — Worldtree reached back to their untouched
|
||
endpoint under the new consumer_id. `env.sh` re-vaulted, sha 8a225c002072.
|
||
⚠ **`secret backfill` was the WRONG tool for one known item** — it rescans every
|
||
`~/development/*/{env.sh,.env}` and had not reached svos after three minutes;
|
||
targeted `put` is the fast path, backfill is for catching drift across the box.
|
||
**OPERATOR RULING 2026-09-05:
|
||
worldtree-dev owns code only, no ops — key material is infra-ops's.** The global
|
||
`~/.claude/CLAUDE.md` line routing "Heimdall scopes (Worldtree auth) →
|
||
worldtree-dev" was corrected in place the same day on operator instruction.
|
||
|
||
- ⚠ **FOOT-GUN, generalises past this rename: a credential cutover whose OLD key
|
||
is required for a later cleanup is destroyed by the natural housekeeping motion
|
||
right after cutover.** Re-vaulting the post-cutover `env.sh` would have
|
||
overwritten the last convenient copy of the old `ledger` key value — the only
|
||
credential that can ever delete `ledger:miranda`. ledger-dev caught it and
|
||
preserved the value first at
|
||
`nh3-dev/development/svos/worldtree-api-key-ledger-legacy` (sha d44c2c1a651b);
|
||
their step 8 ends by deleting that item. **I verified it is genuinely the live
|
||
key** rather than trusting the label: its last 8 chars are `e68a5170`, matching
|
||
the `ledger` key's suffix (key_id b38932f5).
|
||
|
||
- **STEP 7 DONE 2026-09-05, STEP 8 HELD.** `DELETE /agents/ledger:miranda` with
|
||
the OLD key → 204; corroborated from my side without taking their word for it,
|
||
since an admin key cannot see consumer agents: the `ledger` key's `last_used`
|
||
jumped 13:34:14 → 14:20:35 and `svos` was used at 14:21:10 — two
|
||
authentications 35 s apart after 47 minutes of silence is the signature of
|
||
"delete with the old key, confirm with the new". Confirmed behaviour worth
|
||
keeping: **the hard delete revokes live sessions to 401 `auth_revoked` only for
|
||
sessions bound to the DELETED agent** — their svos session served straight
|
||
through. **Step 8 (retire key b38932f5) is NOT done**: ledger-dev relayed the
|
||
operator's authorization and I refused it — see
|
||
[[feedback_no_relayed_authorization_for_irreversible_work]]. Both keys remain
|
||
live. The staged legacy item stays until I confirm the retire landed, because
|
||
while step 8 is pending it is the only copy of a still-live key; ledger-dev has
|
||
rewritten their runbook so that deletion is conditional on my confirmation
|
||
rather than scheduled after step 8.
|
||
|
||
- **SVOS ARC CLOSED — step 8 done 2026-09-05T14:28:15Z** on the operator's direct
|
||
authorization in my own channel (never the relay). `DELETE /admin/keys/b38932f5`
|
||
→ 200; preconditions checked BEFORE firing (svos had a live key, ledger existed
|
||
and was not already revoked) and the post-state read back from `/admin/keys`
|
||
rather than inferred from the 200: `ledger` disabled=True, `svos` untouched,
|
||
deployment `/health` 200. **The rollback window is closed** — re-defining
|
||
`ledger:miranda` is no longer possible. ledger-dev clears the staged
|
||
`worldtree-api-key-ledger-legacy` vault item on this confirmation. ledger-dev gated their
|
||
cleanup on observing a **401 from the old key**, not on my report of the
|
||
timestamp — the right instinct, and they deleted the staged legacy item
|
||
themselves (soft → trash, id a8038e5a-e2f6-4b77-bf17-99c9197d4b1f). Vault
|
||
verified from my side: exactly two svos items remain (`env.sh` 8a225c002072,
|
||
`worldtree-api-key` 23c10c9c7219) and no ledger-era item anywhere. Remaining on
|
||
the arc: only the `ledger-dev` → `svos-dev` handle (with `_SEED_RECORD_TO`
|
||
behind it) and a prose sweep — reversible work, theirs and the operator's.
|
||
|
||
- **Original constraints on that mint** (recorded because the deletion ordering is
|
||
a permanent trap, not a one-time step): string
|
||
`svos` verbatim (WT tier 3 admits only `^[a-z][a-z0-9-]{2,63}$`, INV-181-15);
|
||
**keep the existing `ledger` key LIVE**, do not revoke. Ordering is load-bearing —
|
||
`DELETE /agents/{agent_id}` refuses any caller that is not the row's owner, so
|
||
`ledger:miranda` can ONLY be deleted with the `ledger` key; retire it first and
|
||
the stale row outlives the ability to remove it, holding a live
|
||
`agents.call:ledger:miranda` grant that nothing reaps (the 24h sweep only touches
|
||
soft-deleted rows, and soft-deletion comes from revocation, never disuse). So:
|
||
mint new → they cut over and verify → delete the agent with the OLD key → then
|
||
retire it. Precedent for who mints: msg 401, worldtree-dev routed the pewpewstudio
|
||
key request TO infra-ops. I hold only the PERSONAL admin token (:8081); which
|
||
deployment `ledger` lives on is not yet established. Surfaced to the operator.
|
||
|
||
- **Open commitment to vastblue-dev:** a dedicated CI runner, gated on their first
|
||
client-premises release cut (U10, unscheduled). Ping expected when U10 is scheduled.
|
||
|
||
- **Neither Mac nor the Studio is in `servers/` or `dns/internal.yaml`** — deliberate;
|
||
they are the operator's personal machines. A choice to revisit, not an oversight.
|
||
|
||
- **`vh/remote-ssh-mcp` forked 2026-09-05 (repo id 117, private, full 51-commit
|
||
history)** — our copy of `the-nine-nation/remote-ssh-mcp` (MIT), an SSH MCP
|
||
server chosen over the 693★ `tufantunc/ssh-mcp` on trust-surface grounds: **two
|
||
npm deps** (`@modelcontextprotocol/server`, `zod`), 183 KB, and it **never
|
||
touches key material** — it shells out to the system OpenSSH client, so
|
||
`~/.ssh/config`, ControlMaster, ProxyJump and `infra-ops_ed25519` all just work.
|
||
Shape: 2349 LOC across 11 source files, 811 LOC of tests including fake-ssh hang
|
||
harnesses. Complements `elway` rather than replacing it — no file transfer, no
|
||
idempotency; it takes ad-hoc reconnaissance with persistent cwd/env sessions,
|
||
elway keeps deploys and uploads. ⚠ **The denylist is NOT security**: four regexes
|
||
(`rm -rf /`, shutdown/reboot/poweroff/halt, mkfs, iptables -F) trivially bypassed
|
||
by `bash -c`, variables or base64 — the author says so. **The real containment
|
||
boundary is the host allowlist**, drawn from exact `Host` aliases in ssh_config
|
||
with wildcards deliberately ignored. Two things to settle before use: the
|
||
reboot/shutdown denial will block legitimate infra-ops work, and
|
||
`.github/workflows/star-history.yml` is upstream chore CI sitting in a repo where
|
||
`has_actions=True`. **Both actioned — three commits landed 2026-09-05, LOCAL
|
||
ONLY and NOT PUSHED (push is the operator's call):** (1) stripped upstream
|
||
furniture — star-history CI, its generated assets, the `server.json` registry
|
||
manifest, branding JPEGs, zh-CN README; (2) removed the power-control denylist
|
||
rule and documented in code + tests + README that the list guards ACCIDENTS and
|
||
is not a boundary, with three bypasses asserted as ALLOWED so a green suite is
|
||
never read as containment; (3) **`strictAllowlist`** — upstream's allowlist was
|
||
additive and discovery unconditional, so the default allowlist was all 18 `Host`
|
||
entries in `~/.ssh/config`. Strict makes explicit hosts authoritative and
|
||
discovery metadata-only. Verified live: `corviduo-dev` is in ssh_config, not in
|
||
our allowlist, and is refused `host_not_allowed`. 41/41 tests green.
|
||
|
||
- **`remote-ssh` MCP server is LIVE** — registered project-scoped in
|
||
`eshpfi-management/.mcp.json` with `SSH_MCP_STRICT_ALLOWLIST=1`; allowlist in
|
||
`~/.config/remote-ssh-mcp/config.json` starts deliberately narrow at
|
||
**`irv-ml1`, `nh3-extdev`** (widen there, not by discovery). Smoke-verified end
|
||
to end on both: persistent shell, `cd` and exported vars survive across calls,
|
||
**~6 ms/command on nh3-extdev and ~22 ms on irv-ml1** (WireGuard) versus a fresh
|
||
handshake each time. ⚠ **`.mcp.json` points at the built `dist/`** — edit the
|
||
fork without `npm run build` and the server keeps serving old code; that bit me
|
||
mid-session. ⚠ **A finite stdin pipe is NOT a valid smoke harness** — closing
|
||
stdin kills the server mid-handshake and reports `connect_failed: SSH shell
|
||
exited during the open handshake`, which looks exactly like a remote-side fault
|
||
and is not. Use a client that holds stdin open. (I briefly suspected irv-ml1's
|
||
zsh login shell; wrong — the server invokes `bash --noprofile --norc`
|
||
explicitly, so the login shell is irrelevant.)
|
||
|
||
- **`esh-macbook-air` (10.0.10.83) is DELIBERATELY NOT BACKED UP — operator ruling
|
||
2026-09-05, settled, do not re-raise.** Surveyed it and found no Time Machine
|
||
destination and no restic/borg/rclone/kopia installed, protecting 132 GiB.
|
||
Operator's answer: it is his laptop and the surface is **regenerable** — mostly
|
||
applications, with real data living in OneDrive, iCloud and ssh sessions — and he
|
||
does not want PBS filled with it. Correct call; the finding was real and the
|
||
conclusion is that it does not matter. FileVault On and SIP enabled already cover
|
||
the loss-and-theft axis. The same reasoning presumably extends to
|
||
`esh-mac-studio` and `vuongs-mac-mini`. **Still open and much smaller:** Remote
|
||
Apple Events (port 3031/eppc) is listening and nothing uses it — one toggle.
|
||
|
||
- ⚠ **`remote-ssh` MCP could not be used for its FIRST real task, and the blocker
|
||
is `~/.ssh/config`, not the tool.** The server accepts only exact `Host` aliases,
|
||
so a host addressed by raw IP is structurally unreachable no matter what the
|
||
allowlist says. **13 of the 28 hosts in `servers/` have an alias; 15 do not** —
|
||
including `ana-docker`, `ana-ml2`, `nh3-docker`, `pfi-gx10`, `esh-docker-vm` and
|
||
every hypervisor, i.e. most of where the work happens. Widening
|
||
`~/.config/remote-ssh-mcp/config.json` does NOT fix this; the aliases have to
|
||
exist first. **RESOLVED the same day, and NOT by adding aliases.** Operator
|
||
pushback, correct: a poking-around tool is ad-hoc by nature, and pre-registering
|
||
a host before you can look at it is the opposite of ad-hoc — generating aliases
|
||
for the known fleet would not have helped, because the ad-hoc case is by
|
||
definition the host not yet in the inventory. Implemented address-based reach
|
||
instead (`allowedNetworks` / `deniedNetworks` / `defaultUser` /
|
||
`defaultIdentityFile` / `hostKeyPolicy`). **Live config: `10.0.0.0/8` allowed,
|
||
connecting as `infra-ops` with `~/.ssh/infra-ops_ed25519`, `accept-new` host
|
||
keys, SureFire tenant hosts carved out via `deniedNetworks` (deny beats allow,
|
||
host-specific rather than a /24 because `pfi-pve` shares 10.250.250.0/24).**
|
||
Verified live: 10.0.10.83 opens by raw IP as infra-ops, 10.250.150.100 refused by
|
||
the carve-out, 192.168.1.5 refused as outside. ⚠ **My own earlier objection was
|
||
half wrong** — the credential boundary is about SECRETS ("never accept passwords
|
||
or private-key material"), not identity, so supplying a username does not breach
|
||
it; the real problem was only that the server passed no user at all, so a bare
|
||
address would connect as the LOCAL account. Mechanics, not principle.
|
||
|
||
- ⚠ **`uv tool install --force .` DOES NOT REBUILD when the version has not moved**
|
||
(forseti, measured 2026-09-05). `--force` only handles "a tool by this name
|
||
exists"; `--reinstall` is what rebuilds instead of reusing the cached build keyed
|
||
on the version string. It prints `Installed 9 executables` over **stale code**
|
||
with nothing raising its hand — it cost forseti a bug that survived a reinstall
|
||
AND a re-smoke, because the binary verified against had not changed. **Always
|
||
`uv tool install --force --reinstall .`**, both flags, every time. Same shape as
|
||
the `.mcp.json` → built `dist/` trap found today: a deploy surface that reports
|
||
success while serving the previous artifact. When a fix "does not take", suspect
|
||
the artifact before the code.
|
||
|
||
- **althing 3.5.0 released** (forseti) — adds a 9th binary,
|
||
`althing-operator declare <handle> --description "..."`, restoring the CLI handle
|
||
declaration v2 had and v3 removed. Deliberately a SEPARATE binary, not a
|
||
`postbox` subcommand: the invariant is that no SESSION surface exposes an
|
||
operator verb. Relevant to the pending `ledger-dev` → `svos-dev` rename, which is
|
||
still the operator's call. nh3-dev not yet upgraded.
|
||
|
||
- ⚠ **`remote-ssh` MCP: a bare `sudo` hangs the session forever — pipe it.**
|
||
`ssh_run 'sudo -n whoami'` returns `running` with EMPTY stdout and the session is
|
||
then permanently `busy`; `sudo -n id | cat` works and returns everything.
|
||
**Measured on BOTH macOS 26.6 and Debian (nh3-extdev), so it is the tool, not a
|
||
platform quirk.** Cause: sudo ≥1.9.14 defaults `use_pty` on and relays through
|
||
its own PTY; the run frame gives the command stdin on `/dev/null` while stdout
|
||
stays on the session PTY, the relay never completes, and the completion marker
|
||
never arrives. Workaround `| cat` is in CLAUDE.md. **The proper fix is unbuilt**
|
||
— likely running the command through a pipe inside the run frame and taking the
|
||
exit code from `PIPESTATUS`, which is a real protocol change (commands lose tty
|
||
detection) and wants its own red-green cycle. Matters more than it sounds: infra
|
||
work is sudo work, and this was found by USING the tool, not by smoke-testing it.
|
||
|
||
- **`dsh` on `esh-macbook-air` updated 0.1.1-rc.2 → 0.1.2-rc.1** (2026-09-05;
|
||
latest published 2026-09-03). Global install and the shared profile tree both
|
||
confirmed on the new version. ⚠ **The RUNNING `dsh web` (pid 16231, up since
|
||
Wed 4pm, 127.0.0.1:3080) is still on the OLD code and was deliberately NOT
|
||
killed** — there is no LaunchAgent, so killing it would have left nothing
|
||
running rather than a restarted service. It runs as a FOREGROUND process in the
|
||
operator's terminal (`s005`, `S+`): it dies with the terminal and does not
|
||
survive a reboot, which is the real fragility. A `com.pfi.dsh-web` LaunchAgent
|
||
was drafted but **the privileged write was blocked by the permission
|
||
classifier** — base64 piped into `sudo tee` of a LaunchAgent is a malware-shaped
|
||
pattern and the block is correct; it needs operator approval or an operator-run
|
||
install. Bind stays `127.0.0.1` deliberately: widening it is a security decision
|
||
on a personal laptop whose application firewall is off, and not mine to take.
|
||
|
||
- **sudo hang FIXED in the fork (`30a1f76`), and two wrong shapes are recorded so
|
||
nobody retries them.** The command's stdout now goes to a **fifo drained by a
|
||
background `cat`**: non-tty (so sudo skips its own PTY), no subshell (so `cd`
|
||
and `export` still persist), and relayed live (so `running` + `ssh_peek`
|
||
streaming survives). `cmd | cat` was tried first and **broke cwd persistence** —
|
||
every pipeline stage runs in a subshell — caught by the existing test.
|
||
`cmd > file` would have been non-tty and subshell-free but invisible until the
|
||
command ends. ⚠ **Deliberately NO `wait` on the relay**: a sudo child inherits
|
||
the fifo's write end, `cat` never sees EOF, and the wait hangs — measured, with
|
||
`sudo -n whoami` printing `root` and then wedging the session. Residual risk
|
||
stated in the frame: a command's tail can in principle land after its own
|
||
marker. ⚠ **Job control off AND the relay brace-wrapped with stderr discarded** —
|
||
both needed, because macOS ships bash 3.2 where `set +m` alone still leaked
|
||
`[1] 75449` into the parsed stream. Verified live on macOS and Debian: bare sudo
|
||
in ~20 ms, state persists, exit codes correct. **`sudo -u <other-user>` still
|
||
wants `| cat`** — not chased further.
|
||
|
||
- **dsh web on `esh-macbook-air` is now a LaunchAgent** (`com.pfi.dsh-web`,
|
||
installed 2026-09-05, `runs=1`, `state=running`, pid 76728 on 0.1.2-rc.1). It
|
||
was a foreground process in the operator's terminal that died with the window;
|
||
it now survives terminal close and reboot with `KeepAlive` + `RunAtLoad` and a
|
||
10 s `ThrottleInterval` so a startup error cannot hot-loop. Logs to
|
||
`~/Library/Logs/dsh-web.log`. ⚠ **The plist names the node interpreter
|
||
explicitly** — launchd's minimal PATH has no `~/.local/node/bin`, so the
|
||
shebang's `env node` fails. ⚠ **0.1.2-rc.1 requires a TOKEN**: bare
|
||
`http://127.0.0.1:3080/` now returns 401 and the tokened URL is printed to the
|
||
log on each start, so a bookmark from the old version will not work. Bind stays
|
||
127.0.0.1 deliberately.
|
||
|
||
- ⚠ **althing tools on nh3-dev are 3.6.0, but the POST OFFICE CONTAINER IS STILL
|
||
3.0.0** (`gitea.phasefinal.com/claude-bot/althing-post-office:3.0.0`, up 7 days
|
||
on nh3-docker). forseti: the new handle verbs (`althing-operator delete` /
|
||
`retire`, and `declare` from 3.5.0) live in the post office, so they fail with
|
||
"no tool named ..." until the container carries 3.6.0. Schema gains
|
||
`handles.retired_at` via the idempotent `_ADDED_COLUMNS` path, so the live store
|
||
upgrades itself on first start — no manual migration. **REBUILT AND DEPLOYED
|
||
2026-09-05** on operator authorization: image
|
||
`claude-bot/althing-post-office:3.6.0@sha256:13158835488a8ec04f990c97c4f4c68f1d923b12494319cf07392552e68f8a78`,
|
||
built on nh3-dev from a clean tree at `4d26226`, pushed to the gitea registry
|
||
under the **claude-bot** namespace (not `vh` — package namespaces are owned).
|
||
**Bus down ~4 minutes, 09:35–09:39 PDT.**
|
||
**The backup was taken the way the compose file says to, and it mattered:** at
|
||
stop time `post_office.db` was 23.8 MB with a **5.9 MB WAL** — copying the .db
|
||
alone would have silently lost the day's mail. Stop → `PRAGMA
|
||
wal_checkpoint(TRUNCATE)` (WAL → 0 bytes) → copy → verify. Backup at
|
||
`nh3-docker:/var/backups/althing/post_office.db.pre-3.6.0-20260905`, integrity
|
||
`ok`, counts identical on both sides (handles 76, messages 995, recipients
|
||
1022). ⚠ **Reading a WAL-mode SQLite backup read-only needs `?immutable=1`, not
|
||
`?mode=ro`** — `mode=ro` still wants to create a `-shm` and dies with "attempt to
|
||
write a readonly database". Post-deploy: same counts, `handles.retired_at`
|
||
present, `retired 0`, and `mem=536870912` / `oom=-500` verified by `docker
|
||
inspect` rather than by reading the yaml, per that file's own warning.
|
||
`althing-operator` now offers `declare | delete | retire`, which unblocks the
|
||
pending `ledger-dev` → `svos-dev` rename.
|
||
|
||
- **Handle `retire` is REVERSIBLE — re-declaring the name revives it, history
|
||
intact** (forseti smoked it against the live bus 2026-09-05). That matters for
|
||
the pending `ledger-dev` → `svos-dev` rename: `retire` is the right verb (delete
|
||
refuses any handle that has mail, naming both counts — `delete forseti` was run
|
||
against production and correctly refused at 53 sent / 81 addressed, which is
|
||
safe to try precisely because refusing IS the behaviour), and it can be undone
|
||
by declaring the name again. Lower stakes than "retire" sounds.
|
||
Both of my deploy findings — the naive-copy WAL trap and `?immutable=1` — are
|
||
now in althing's own `deploy/INSTALL.md` (`d6f4fb5`) under a new
|
||
"Backing up the store" section, on the reasoning that they are properties of
|
||
the project's `journal_mode=WAL` choice rather than of my procedure.
|
||
|