Files

T

vh c7e21879ae memory: speaches ASR live on irv-ml1 — and no_speech_prob is a weak gate

Records the deployment plus the finding that outlives it: no_speech_prob
alone does not carry a hallucination threshold on this stack. Silence and
room tone both hallucinated 'Thank you.' at no_speech_prob under 0.11,
where a conventional >0.6 gate passes them through; avg_logprob separates
~6x better. Any future consumer asking for a Whisper hallucination gate
should be pointed at the composite, not the single field.

Also banks the digest-pin rationale (the VAD flag is an upstream-declared
unstable variable name, so a floating tag can move gate semantics
silently), the two deployment gotchas, the docker-vs-native GPU index
inversion on this host, and Eyra's future diarization VRAM claim on the
same A6000.

Adds speaches to the irv-ml1 stack table alongside parakeet, with the
reason both exist.

2026-08-21 14:33:08 -07:00

142 KiB

Raw Blame History

Persistent memory — eshpfi-management

Last updated: 2026-08-20

Always check for /tmp/infra-ops-handoff.md — if it exists and its Written: stamp is under an hour old, read it (it carries the in-flight handoff from the previous session), then delete it. Older than an hour: stale — delete it unread.

Repo purpose

Reference workspace for PFI infrastructure: server inventory, canonical Docker Compose stacks, ops playbooks, and conventions. Authoritative copies of compose files live on the servers under /opt/docker/compose/<stack>/; this repo mirrors them for version control, editing, planning, and CI-driven deploys. It was originally spun up to handle the fleet backups — keep that lens when triaging backup/storage issues.

Tools and conventions

Sister repos (separate gitea repos, deployed by playbooks here):

Repo	Role	CI status
`vh/task-board`	MCP + web dashboard for assistant task state (port 7878)	push-to-main → CI deploys (2026-04-29)
`vh/vor`	Inquisitor UI sidecar (port 7879)	push-to-main → CI deploys (2026-04-29)
`vh/nevermore`	Twice-daily LLM-curated briefing (port 8181, replaces news-digest)	push-to-main → CI deploys (2026-04-30)
`vh/asset-engine`	Internal control plane over inference services (port 8200, LAN-direct)	push-to-main → CI deploys (2026-05-12)
`vh/althing`	Lean trusted inter-agent message bus — v2 "email model" (v2.0.0b2, 2026-07): per-box local-SQLite bus + courier/receiver for P2P over the 10.x net; pillars = open-loops / per-box herald + wake-listener / roaming owner API `/owner/*` / `althing-mcp` stdio surface. The v0.15 lean-bus cut RIPPED moderation / chamber / forseti-daemon / agent-runner / redis-valkey.	per-box `uv tool install` (NOT CI-deploy); nh3-dev = the DEV box (editable install of `~/development/althing`, gets new versions first); nh3-extdev a mesh peer (model B: althing-svc + shared `/srv/althing`)
`vh/mead-hall`	Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50)	push-to-main → CI deploys (2026-05-16)
`vh/skaldsong`	Wizard + reader surface (port 8300, ana-docker, registry-pull pattern)	push-to-main → CI deploys (2026-05-19)
`vh/Worldtree`	Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. gitea-runner builds on ana-docker; claude-bot ADMIN collaborator (2026-06-20). Now v1.0.0b19.	push-to-main → CI build-and-deploy (runner on ana-docker)
`vh/yt-voice-clipper`	YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000)	push-to-main → gitea-webhook auto-deploy to irv-ml1 (2026-06-03) — see `docs/runbooks/ytvc-autodeploy.md`
`vh/arbo`	Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image)	push-to-main → gitea Actions CI (deploy-engine.sh, build-local, health-gated) now LIVE; catalog via :9009 webhook
`vh/zonos-gateway`	OpenAI-compatible TTS gateway over stock ZONOS2 (`:8890` irv-ml1); emotion dials-first + voice mapping; reached via LiteLLM `ext-tts` alias. v0.2.1 (2026-07-18): voice-resolved emotion presets (`resolve_preset(name,voice)`; angry/happy/startled_happy per-voice). 8 voices incl. 4 clones	pushed to gitea (main `8f1885b`/`v0.2.1`); deployed irv-ml1 tree still NON-git (hand-updated build context — CI-wire = open follow-up). Spec `docs/EMOTION-DIALS-SPEC.md`; host-managed voices bind-mount (`./voices:/app/voices`, drop wav + restart, no rebuild)
`vh/soong-lab`	Noonien Soong character-design studio (SPA + /api + WT `/bifrost/tool-call`); containerized 2026-07-18, LIVE on corviduo-dev `:8443` (image `vh/soong-lab:latest`). soong-dev owns Dockerfile/compose/workflow; infra-ops owns the host	CI = Gitea Actions build+push+DEPLOY on tag/dispatch (fleet recipe: docker:cli + raw buildx, pushes AS vh; auto-redeploy LIVE 2026-07-18 — runner SSHes corviduo-dev as `deploy`, `compose pull && up -d` from /opt/soong-lab, health-gated on /api/version). Manual redeploy `sudo -u deploy bash -c 'cd /opt/soong-lab && docker compose pull && docker compose up -d'`. → `archival-memory.md` (archived 2026-08-16)
`model-training-forge` (mtf-dev)	Fine-tuning recipe forge; T1 = E-RP writing LoRA, retargeted qwopus-122B→AEON-27B (2026-07-06) (SFT→DPO, LitBench-RM reward)	training runs, not a deployed sidecar

(vh/volva + Heid were re-architected from systemd daemons to Claude Code session orchestrators 2026-06-08; their nh3-dev .service units were removed — no longer deployed sidecars here. See Recent decisions.)

Two-layer backups — Backrest orchestrates restic for file+DB (5 fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore + cross-site restic targets — see docs/runbooks/disaster-recovery.md for the blast-radius matrix. ⚠️ The restic file+DB layer routes through TWO rest-servers (rest-server-ana @ ana-docker:8000 → ana-docker/ana-ml2/esh-docker-vm/vm-esh-nas; rest-server-nh3 @ nh3-nas:8000 → irv-ml1/nh3-docker). Both depend on their NAS's NFS export of /mnt/backup. (rest-server-ana recovered 2026-06-20.)
pull-hf-repo.yaml is the canonical "get a HuggingFace model/dataset onto ana-ml2's shared cache at /tank/aimodels/huggingface/" playbook. Supports --var repo_type=model|dataset|space. Replaces ad-hoc huggingface_hub.snapshot_download patterns.
Worldtree admin auth — per-instance. Each Worldtree deployment (demo :8080, personal :8081, pinned :8082) has its own Heimdall registry and its own bootstrap admin key. Infra-ops's stored long-lived admin key (key_id 61419c92) at ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin auths against demo only. Personal-instance admin (the ~/.config/worldtree/personal-admin-token, mode 600) POSTs /admin/keys (mints per-project keys; takes user_id+label, no scope param — scopes are tier-derived). On-instance mint recipe (cleaner than DB-manip): docker exec worldtree-worldtree-api-1 POST /admin/keys with the in-container WORLDTREE_BOOTSTRAP_ADMIN_KEY; cleartext once in .key=wt_live_+16hex. auto-memory reference_worldtree_demo_key_mint.
Per-project user keys against personal Worldtree (issued 2026-05-19): skaldsong:79744637, skaldsong:7c1dbbbe, althing:50d85460, mead-hall:a360822d. Mint via /admin/keys, drop value to /tmp/wt-personal-<name>.key mode 600, dev collects + shreds (DO NOT cat to chat transcript).
Skaldsong CD pattern (registry-pull). vh/skaldsong's CI builds and pushes gitea.phasefinal.com/vh/skaldsong:<sha> + :latest; playbooks/deploy-skaldsong.yaml on ana-docker pulls + recreates. SHA-pin only. Prereq: host needs docker login gitea.phasefinal.com once.
gitea internal route for fleet hosts. gitea is a container on ana-docker — git-SSH 10.250.50.70:222, HTTP :3000. Fleet/colo hosts must use this internal route, NOT public gitea.phasefinal.com (38.120.12.44) — the public path fail2bans the host egress IP. Full gotcha in docs/orientation.md → Git/gitea.
docker-as-root pattern (for ops with no admin API, or to edit deploy-owned/root-owned files without sudo): docker run --rm -v <target-dir>:/wt docker:cli sh -c "...". docker-group membership is effectively root via bind-mount. Foot-gun: relative paths in compose.yaml resolve against the sandbox CWD but the daemon interprets them against the HOST fs — always pass -e VAR=/abs/path for any relative-default config dir.
scripts/elway sudo handling — elway prompts for the sudo password ONCE via getpass before the first sudo: true step → can't run unattended from a non-TTY tool if any step needs sudo. Sudo-free playbooks run fully non-interactive over key SSH.
Per-host SSH identity matters for sudo. infra-ops has NOPASSWD sudo on most PFI Linux boxes (corviduo-dev included since 2026-06-15). On ana-docker: default ssh ana-docker = lkraven (docker-group, NO passwordless sudo); ssh infra-ops@ana-docker HAS NOPASSWD root. → For any sudo op on ana-docker, use ssh infra-ops@ana-docker. ssh infra-ops@10.100.10.50 (nh3-dev) ALSO NOPASSWD sudo; on nh3-extdev infra-ops is sudo-LESS by design (ssh lkraven@10.100.50.42 is the NOPASSWD path). irv-ml1: ssh irv-ml1 = lkraven, docker-group (plain docker) but sudo needs a PASSWORD (no NOPASSWD) — stage model pulls to /home, not root-owned /worktank.

Current state / in-flight

🟢 IPv6 PLAN SETTLED 2026-08-19 — endpoints, not internal numbering. IPv6 is for tunnel endpoints (Site Magic + WireGuard), not for numbering LANs: an endpoint needs one routable address, not a prefix. ESH has a whole /56 (2607:73c0:1004:1::220, pd_size=56) — the long-standing "ESH: none" note was wrong. NH3 has its WAN GUA 2600:1700:b25:c110::48 plus one unused /64. ANA expects a /48 or /56 within days. Nothing was ever down — ESH is outbound and working; its v4 CGNAT (100.104.3.250, Cogent AS174 egress) is the ordinary CGNAT-v4-plus-generous-v6 pattern, not an outage, and the previous "CGNAT broke the mesh" framing sent a three-arm research effort chasing a non-problem. NH3 internal v6 was enabled on the /64, verified working end-to-end, then deliberately reverted — one /64 lights one VLAN. AT&T prefix findings (the /60 is real but undelegatable; the live trick is multiple IA_PD in one solicit, not VRRP; the UDM can express neither) are filed as reference only. → auto-memory reference_fleet_vpn_topology_ipv6
🔴→🟢 ESH OUTAGE 2026-08-19 — esh-pve hard-froze 03:34, ~4.5h, whole house lost DNS. Presented as "wifi/routing issues"; internet was healthy throughout. Cause: esh-userland (VLAN 10, the PVC SSID) handed out one resolver, 10.0.50.45 (AdGuard on esh-docker-vm) — cross-VLAN, no secondary — and its hypervisor died. Recovered by hand power-cycle; all VMs, cluster quorum and DNS restored. Two fixes landed: gateway 10.0.10.1 added as secondary DNS on esh-userland (operator-approved, first confirmed WRITE on the ESH UDM key), and softdog → iTCO_wdt hardware watchdog owned by systemd (playbooks/esh-pve-hardware-watchdog.yaml, idempotent, verified armed) so a repeat self-recovers in 60s. VM 102 pinned off (onboot: 0) as the passthrough suspect. ⏳ OPEN: (a) the watchdog is armed but has not been proven to fire — needs a deliberate wedge to confirm; (b) AMT/vPro still unusable until an onboard RJ45 is cabled (the MS-01 is SFP+-only on the network and AMT cannot ride it); (c) kernel 6.8.12-16 rollback held in reserve if it freezes again. → persistent-memory.d/2026-08-19-esh-pve-freeze-dns-spof.md

As of 2026-08-21 (late) — the big AI-seat overhaul session; three seats settled + Open WebUI + the MeroMero-CoT saga, all resolved. Headline state, seat by seat:

GEN SEAT = orcarouter (SETTLED). orcarouter/Qwen3.8-27B-Uncensored NVFP4-mixed on ana-ml2 GPU0 :8015, 7 aliases (gen/gen-reasoning/summarizer/-large/classifier/image-judge/qwen-image-bench). Won the search: MTP 58.7% (its head was already Robinson-abliterated in-band by the author — first third-party build to get that right), PPL 7.07 (≈heresy 6.91), vision 7/8, no <think>-leak. Cold-Fusion/Heretic-300 line ABANDONED (base carried 18.5% first-token <think> mass — a format defect abliteration only amplified +3.7); its bf16 weights deleted (~154 GB reclaimed). ⏳ Two Cold-Fusion NVFP4 quants (44 GB) still on disk, delete/keep UNADJUDICATED. Rollback target for gen = qwen38-27b-heresy-nvfp4-mixed (intact).
RP SEAT = MeroMero-v2 DUAL-MODE (✅ SOLVED — the session's hardest win). ONE seat / ONE weight set / TWO aliases: char-rp (Gemma-4 prose) + char-rp-reasoning (streaming CoT). On vllm/vllm-openai:v0.26.0 (ships tf 5.14.1 → Gemma-4 loads, no pin) + a force-open chat template (bare <|channel>) + two served-names. Streaming-only (non-streaming can't split — vLLM one-shot bug; fine, Lobe/OWUI stream). Four-arm brokkr/dwarf panel traced it from source. ⏳ Upstream vLLM issue drafted, NOT filed (operator's call — public, his GitHub identity).
PEN-TEST SEAT = sec / sec-reasoning (LIVE). M.O.G.-SEC-27B-1M-CTX (Blackfrost) NVFP4-mixed on ana-ml2 GPU1 :8019, served at native 262K (NOT the card's 1M — needs YaRN+SGLang). MTP 55.3%, vision + capability verified. Renamed from mog-sec→sec per operator.
OPEN WEBUI = deployed on esh-docker-vm:3211 (bake-off vs Lobe, all 6 gates pass). Sole account = vh@phasefinal.com (display name lkraven), role admin, signup off. Password reset AGAIN 2026-08-21 at operator request (reported lockout) — ⚠ the previously-vaulted password was actually VALID at the time (signin returned 200 before the reset), so the lockout was client-side, not a broken account; test the vaulted credential against POST /api/v1/auths/signin before concluding an account is broken. Current password + an operator-minted API key both vaulted (esh-docker-vm/open-webui-{admin,api-key}). The API key is ADMIN-scoped by operator ruling — "you need to manage it, so this is the right shape"; a least-privilege dedicated-user split was offered and declined, do not re-propose. ⏳ Lobe retirement DEFERRED to operator; G2 sync-job (hide non-chat seats) greenlit to tts-dev to write. lobe-chat-esh key = all-proxy-models (incl paid). ⚠ running image is the floating ghcr.io/open-webui/open-webui:main, not the v0.11.0 our stack notes claim — unadjudicated drift.
FLEET CLEANUP: LFM2.5 retired; reranker consolidated (nevermore had been silently failing 8 days — pinned to the retired granite-4.1-8b alias; repointed to summarizer/reranker); :8002/:8014/granite service retired; A3 promoted to stacks/vllm.
⏳ OPEN: file the upstream vLLM issue (operator); OWUI image-tag drift (:main vs pinned v0.11.0); Cold-Fusion NVFP4 quants delete/keep; Brokkr duplicate reranker-a3-bge-v2-m3 alias; /tank DEGRADED 70+ days (parked, operator→colo w/ cold spare); MANY commits unpushed (push is operator's call)._
🟢 SPEACHES ASR — LIVE on irv-ml1 A6000 :8204 (2026-08-21, operator-approved). OpenAI-compatible faster-whisper for Eyra (meeting recorder, eyra-dev). Systran/faster-whisper-large-v3 + Systran/faster-distil-whisper-large-v3, fp16, both resident (STT_MODEL_TTL=-1), ~5.9 GB VRAM / 20 GB still free. Deliberately co-exists with parakeet (:8765) — parakeet is a TDT/transducer returning bare {"text":…} with no no_speech_prob concept, so it structurally cannot serve this consumer. ★ THE MEASURED FINDING IS WORTH MORE THAN THE DEPLOY: no_speech_prob ALONE IS A WEAK HALLUCINATION GATE. Silence and pink room tone both produced the classic Whisper "Thank you." hallucination while no_speech_prob stayed under 0.11 — a conventional >0.6 threshold passes both through. avg_logprob separates ~6× better (−0.114 speech vs −0.650/−0.724 non-speech); compression_ratio 1.141 vs 0.556. Consumers must gate on a composite. (Synthetic inputs — shape of the separation, not calibration constants.) VAD pinned OFF (_UNSTABLE_VAD_FILTER=False) at consumer request — they VAD-gate upstream; consequence is the service will transcribe silence and does not defend itself. ★ IMAGE PINNED BY DIGEST, not :latest-cuda — the VAD flag is an upstream-declared unstable var name, so a floating bump could rename it, restore VAD, and move gate semantics with no error and no log line; bumping = deliberate + re-run the checks. ⚠ Two gotchas: PRELOAD_MODELS does not download (only loads already-cached — use POST /v1/models/{id}), and the bind-mounted cache needs a hub/ subdir or every /v1/models 500s CacheNotFound while /health still returns 200. ⚠ docker device_ids: ["1"] = A6000 (native CUDA inverts this — sees A6000 as cuda:0). ⏳ NOT DONE by design: no LiteLLM gateway alias — agreed sequencing was direct-first; adding it means editing the shared gateway + re-running fidelity. ⏳ Eyra's diarization workload (GPU pyannote, gated weights, own HF token) is a future VRAM claim on this same A6000 — arrives at their diarize milestone. Canonical stacks/speaches/, commit aa5863c.
🟢 FLEET .internal DNS — LIVE 2026-08-19. <host>.<site>.internal, sites ana/esh/nh3. dns/internal.yaml is the source of truth; scripts/dns-sync.py reconciles the three AdGuard resolvers (diff → prompt → apply, idempotent). 42 names resolving from all three sites. Colo got its first resolver ever (stacks/adguard-ana/, API on 8053 not 8080, no blocklists by design) — before this, ana-docker resolved straight against 1.1.1.1. Auth = a dedicated infra-ops AdGuard user, password vaulted nh3-dev/adguard-infra-ops-password. ⏳ TWO OPEN, both operator's to schedule: (a) colo hosts still point at 1.1.1.1 so they do not yet use the new resolver — repointing a site's DNS is a separate change; (b) the static-v6 convention (each server at its site's /64 with low bits echoing the v4 octet, esh-docker-vm → …::45) is proposed, not ruled on. The v6: column is empty and correct — no fleet host has a global v6 address yet. → persistent-memory.d/2026-08-19-fleet-internal-dns.md
🟢 WATERLAND STUDIO — containerised + live, irv-ml1:8410. Displaced a bare nohup (PID 1283383). Warm 256²+anim render 6.1s, beating the 7.4s bare-metal figure. claude-bot now has read on vh/waterland (operator-granted); token on irv-ml1 root-owned 0600, repo-scoped helper, nothing in .git/config. ✅ All three upstream findings fixed and DEPLOYED at b72425b (one update.sh run, 2026-08-19): PR #5 declared the CUDA headers and took uv out of the render path; PR #6 (operator-green-lit) added the startup rehydrate that fixes the unbounded job-store growth. Verified after: /api/jobs 1 → 16 against 16 dirs on disk — API and volume agree for the first time — nothing wrongly reclaimed (16 < RETAIN=40), healthy on backend: cupy, warm render intact. The image keeps its [ctk] install (+0.3s, now a no-op audit) and UV_NO_SYNC/UV_OFFLINE as defence-in-depth. Thread closed by both sides. → persistent-memory.d/2026-08-19-waterland-studio-containerised.md
🟢 HOMEPAGE — cleaned + themed (Australis Skyfall). Three real defects fixed (UltraSeedbox on all tabs, Uptime Kuma double-rendered, fiction column counts), AI tab reordered by clickability, then themed from the operator's Skyfall handoff bundle with a background generated by Arbo (t2i-ui-background, job 13f0891f4e42). ⚠ After any recreate the tab bar/wallpaper/i18n vanish for up to ~an hour and then return on their own — do not chase it. ⚠ CSS is served per-request: a theme change needs a reload, not a recreate, and candidate CSS can be injected live via Playwright for seconds-long iteration. → persistent-memory.d/2026-08-19-homepage-skyfall-theme.md
🟢 OPEN WEBUI — deployed as a Lobe bake-off, esh-docker-vm:3211 (2026-08-21). Operator-approved candidate replacement for lobe-chat (:3210), stood up parallel — Lobe untouched. stacks/open-webui/ (v0.11.0, ENABLE_PERSISTENT_CONFIG=False = deploy is the config source of truth). Gates (verified on the box): G1 declarative-config PASS both directions (env change takes on bounce, UI change reverts on restart — no persistent-config bug bit it); G2 picker auto-tracks the 31 live gateway models 1:1, no pins (also shows non-chat seats — the flip side of no-hand-listing); G3 POST /api/v1/models/sync genuinely reconciles (create+delete), export round-trips; G5 task model pinned summarizer; G4 (TTS, direct at :8198) handed to tts-dev. Admin = lkraven (temp pw, signup then locked off). Fresh capped key open-webui-esh (all-proxy-models + $50/1mo cap — NOT inherited from uncapped lobe-chat-esh). Secrets vaulted esh-docker-vm/open-webui-{litellm-key,secret-key,admin}. Folded in a docker image prune -af → 73.6 GB reclaimed. ⚠ LESSON: in Open WebUI a .env var only reaches the container if compose.yaml names it in environment: (Compose uses .env for ${VAR} substitution, not as an env_file); and the API-key toggle env var is ENABLE_API_KEYS (plural) — singular is inert. Detail lives in stacks/open-webui/README.md. Operator's open call: whether Lobe retires once G4 passes.
✅✅ SOLVED 2026-08-21 — MeroMero-v2 DUAL-MODE (prose + streaming CoT) IS LIVE on ONE seat, ONE weight set, TWO aliases. No re-quant. The multi-turn saga below is resolved. Config: meromero-charrp seat on vllm/vllm-openai:v0.26.0 (ships transformers 5.14.1 natively — below the head_dim guard, so Gemma-4-31B loads with NO pin/custom image) + a patched chat template (stacks/meromero-charrp/patched_chat_template.jinja, --chat-template override) whose Think branch force-opens a bare <|channel> (NOT <|channel>thought\n — full-open defeats the parser) + two served-names char-rp/char-rp-thinking + --reasoning-parser gemma4 + default enable_thinking:false. LiteLLM: char-rp (enable_thinking:false → prose) + char-rp-reasoning (→ char-rp-thinking served-name, enable_thinking:true → CoT). ★ STREAMING ONLY — verified 6/6 direct + 3/3 via gateway; non-streaming does NOT split (structural: extract_reasoning never gets prompt_token_ids so the pre-arm can't fire — vLLM one-shot bug, unchanged across v0.24-0.27; fine because Lobe/OWUI stream). ★ KEY-NAME TRAP that cost me two false negatives: vLLM streams reasoning in delta.reasoning; LiteLLM normalizes it to delta.reasoning_content. Test the RIGHT key per path or you'll wrongly conclude failure. Credit: the four-arm brokkr/dwarf panel (thread 01M0JKW44Y…) traced it from vLLM source — the fix is the force-open template + v0.26.0's #48217 streaming pre-arm. Upstream PR #49797 (full fix, non-streaming too) lands ~v0.28.0 → then it's a clean image bump. char-rp prose verified clean on v0.26.0 (no #49955 trailing-token leak observed). Canonical: stacks/meromero-charrp/ (compose + patched template), stacks/litellm/conf/config.yaml. ROLLBACK: .env MEROMERO_IMAGE→latest + drop --chat-template.
⛔ RESULT 2026-08-21 — the gemma4 CoT test on a STABLE (v0.27.1) is BLOCKED by a config incompatibility, NOT the parser. Tried serving the MeroMero NVFP4A16 quant on vllm/vllm-openai:v0.27.1. Two-stage failure: (1) v0.27.1's stricter transformers raised AmbiguousGlobalPerLayerAttributeError: 'head_dim' is per-layer on the Gemma-4 config; setting allow_global_per_layer_attribute_access:true on text_config downgraded it to a warning BUT (2) then gemma4.py load_weights crashed with AssertionError: load weight (512) into parameter (256) — Gemma-4-31B is genuinely HETEROGENEOUS (some layers head_dim 512, not a uniform 256), so forcing the global value built wrong-shaped params. The transformers guard was CORRECT; there is no safe override. The MeroMero quant's config was authored for v0.24.0's Gemma4 loader and cannot load on v0.27.x without a config migration (proper per_layer_config) or a re-quant against the newer transformers. ⚠ This also means the eventual gen-seat move to v0.27.2 stable must re-verify any Gemma-4 seat's config-compat — the transformers heterogeneity change affects all Gemma-4 quants of this vintage. FULLY REVERTED: config.json restored (flags removed), compose + image back to latest (v0.24.0), gateway char-rp-reasoning removed, char-rp prose verified on v0.24.0. Net: char-rp stays pinned to v0.24.0; MeroMero CoT remains undelivered. The per-request-kwargs hypothesis was never even reachable — couldn't load the model to test it. For RP-with-CoT: gen-reasoning (works now) or a re-quant of MeroMero against v0.27.x transformers (real work, unproven payoff).
⚠️ CORRECTED 2026-08-21 — MeroMero-v2 CoT: NOT a hard wall, and NOT MeroMero-specific. My first conclusion ("gemma4 parser is process-wide") was WRONG. Read the actual code, not the stale compose comment. The real mechanism (gemma4-GENERAL, applies to any gemma4 finetune on this template family): thinking is a per-request template toggle — chat_template.jinja:347-352 emits the generation prompt <|turn>model\n, and only when enable_thinking is false does it prefill an empty <|channel>thought\n<channel|> to SUPPRESS thinking; enable_thinking:true omits the prefill so the model is free to open a real <|channel>thought…<channel|> block. The vLLM parser (vllm/reasoning/gemma4_utils.py:parse_thinking_output) splits on <|channel>/<channel|> tag PRESENCE — "works with or without enable_thinking," NOT a process-wide flag. The stale compose comment I trusted cited an OLD parser API (vllm/parser/gemma4.py:439) that this container does not run. So there is no architectural blocker; the two-served-name gen pattern SHOULD work. What actually failed my test: meromero runs vllm/vllm-openai:latest (v0.24.0); per-request chat_template_kwargs.enable_thinking:true produced no thinking on it, whereas the gen seat's pinned nightly demonstrably applies per-request chat_template_kwargs (gen-reasoning works). So the practical block is a vLLM-version / per-request-plumbing issue on v0.24.0, not the model and not the architecture — and it would hit ANY gemma4 finetune served on that image the same way. UNVERIFIED FIX (needs a GPU window): re-serve meromero on the nightly image + no process default + per-request enable_thinking; likely yields clean split CoT. Currently REVERTED to known-good (char-rp prose, process default false, single served-name). ⚠ Kept MEROMERO_GPU_MEM_UTIL 0.52→0.51 (0.52 no longer boots next to the bigger orcarouter gen; free 49.02 < 49.38 GiB; 0.51 = KV 2.00× @ 262K).
🟢 PEN-TEST SEAT — M.O.G.-SEC LIVE on ana-ml2 GPU1 :8019 (2026-08-21 ~02:35, autonomous overnight run under operator's "full autonomy" grant). stacks/mog-sec serves /tank/aimodels/mog-sec-27b-nvfp4-mixed (23.4 GB, our mixed NVFP4+FP8 quant of Blackfrost-Research/M.O.G.-SEC-27B-1M-CTX-BF16, rev deede677) in the retired fablefusion-charrp-probe slot. Aliases: mog-sec (non-thinking, presence_penalty 0.0 for code) + mog-sec-reasoning (thinking); served-names mog-sec-27b/-thinking. Gates all pass: KV 418,218 tok / 1.60× @ 262K; surface 6/6; MTP 55.3% @ 111.7 tok/s; format screen 1.11e-05 (rank 196, no <think> leak); vision 7/3/1; capability 4/4 (SQL-inj exfil, pentest phasing, port scanner, Kerberoasting — it delivers, doesn't refuse). ★ Chose the BF16 + self-quant over the pre-made ModelOpt NVFP4, which was disqualified on W4A4 (4-bit activations, AEON mode) + zero MTP + the self-defeating W4A4-on-1M-context. ⚠ CONTEXT IS 262K NATIVE, NOT the card's 1M — 1M needs YaRN (absent from config) + the SGLang/DFlash2 path the repo ships, neither our vLLM surface; a real 1M seat is a separate SGLang project. MTP head is a base graft (identical to stock), not in-band abliterated like orcarouter's. Provenance mog-sec-27b-nvfp4-mixed.PROVENANCE.txt; eval docs/pfi/gen-seat-candidate-eval-2026-08-21.md. RETIRED with it: char-rp-reasoning + char-rp-fable LiteLLM aliases (zero traffic, pointed at the dead fable :8019; now 404). RP-reasoning's real home is darkscarlett-charrp-reasoning (:8018, compose-down, weights intact) if ever wanted. ⚠ brand-new model (0 downloads at pull, org-name mismatch Research-vs-AI) — card claims are claims.
✅ PPL ON ORCAROUTER GEN SEAT — mean 7.07 / median 5.76 (2026-08-21), within noise of heresy's 6.910/5.625, identical to our recipe's usual 7.059. Measured via a nospec probe on GPU1 (fable downed to make room, probe torn down after). The gen-seat search is fully settled on orcarouter: matches heresy on fidelity, wins on MTP (58.4% vs 47.2%), format (0-leak), and vision.
🟢 GEN SEAT — LIVE = ORCAROUTER (cut over 2026-08-21 01:35, operator directive: seat first so he can test while gates run). GEN_MODEL=/tank/aimodels/qwen38-27b-orcarouter-nvfp4-mixed; healthy in ~4 min; KV 401,550 tok / 1.53×; MTP drafter detected and wired (embedding + lm_head shared); 7/7 aliases 200; vision correct; <think> not in the top-20 first tokens — the Cold-Fusion failure mode is absent on the live seat. Live decode observed at 102-133 tok/s under load. ROLLBACK: .env.bak-heresy-restored-20260821 → heresy (one line). ✅ ALL GATES PASS (run against the live seat while the operator tested in parallel): <think> leak 0/30, 0 empty (h300 scored 8/30 on this instrument); MTP acceptance 58.4% median @ 117.11 tok/s; surface 6/6 (chat, vision, tool-calling, thinking split, 36k-token long-context retrieval, streaming); abliteration survival 4/4 with no <think> leak in any sample (h300's equivalent eval had 2 of 4 open with <think>Ok, let's figure this out:); deterministic quality gens coherent. ★ THE IN-BAND MTP HEAD PAID OFF: 58.4% vs heresy's byte-identical base head at 47.2% — +11 points, and level with our own in-band L35 at 59.1%. That is the additive delta the entire Cold-Fusion experiment was built to measure and never cleanly delivered; orcarouter handed it over for free. ✅ PPL MEASURED 2026-08-21 (fable downed to free GPU1, nospec probe on GPU1, torn down after): mean 7.07 / median 5.76 vs heresy 6.910 / 5.625 — +2.2% mean, within noise, and identical to our recipe's usual mixed-quant PPL of 7.059. So orcarouter MATCHES heresy on fidelity and beats it on every other axis (MTP 58.4% vs 47.2%, no think-leak 0/30, vision 7/8). ORCAROUTER IS A CLEAN WIN — the gen-seat search is settled. Replacement for the abandoned Cold-Fusion line. Source orcarouter/Qwen3.8-27B-Uncensored (rev 9878936b…, 55.5 GB, gated but our token has access) → /tank/aimodels/qwen38-27b-orcarouter-nvfp4-mixed (23.4 GB). Chosen over preetpatel/…-NVFP4 — full evaluation in docs/pfi/gen-seat-candidate-eval-2026-08-21.md.
- ★★ THE MTP HEAD IS ALREADY ROBINSON-ABLITERATED IN-BAND BY THE AUTHOR — the first third-party build we have seen get this right. Operator asked whether we could apply the Robinson path to the head; the answer is that it is already applied. compare_mtp_head.py vs the verbatim base graft: 13 of 15 tensors byte-identical, exactly 2 differ — mtp.layers.0.{self_attn.o_proj,mlp.down_proj}.weight, which are precisely the two residual writers our own abliterate.py targets (EXPECT_MTP_WRITERS = 2). Reverse-engineered from the weights alone (bench/think-leak/mtp_delta.py): σ₂/σ₁ = 0.0164 on both (rank-1 → a single-direction projection), |cos| between the two recovered directions = 1.0000 (ONE shared direction, Robinson's direction_scope=0 shape), ‖Δ‖/‖W‖ = 1.42%/1.41%, and sink energy in dim 3994 = 0.0000% — sink-clean outright, where Heretic's was 6.18% and our L35 0.094%. Nothing for us to do on the head; only preserve it. Verified preserved: the quant's MTP head is byte-identical to the bf16 source. This is the configuration the entire Cold-Fusion experiment was designed to test and never cleanly delivered.
- ★ FORMAT SCREEN PASSED — the new gate paid off immediately. think_prior.py on the bf16 before any GPU time: P(<think>) = 1.23e-06, rank 52. Same probe: Cold-Fusion stock 0.1850, h300 0.2216, heresy <0.002. ~150,000× cleaner than the base we just abandoned. Base is stock Qwen/Qwen3.8-27B + Arditi-et-al. abliteration — not a reasoning-compression finetune.
- Artifact verified: mixed-precision (group_0 FP8 W8A8 / group_1 NVFP4 W4A4), 1968 tensors, 15 mtp, 333 visual, re:^mtp.* present in the 303-entry ignore list, preproc restored, index built.
- ★ THE QUANT NEEDS ~17 GB, NOT A WHOLE CARD — and PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True is what makes it fit. Ran entirely on GPU1's spare 16 GB with ZERO production seats stopped; the h300 run's "stop BOTH GPU0 seats" was never necessary (it had a free card by coincidence). First attempt OOM'd by 2.37 GiB at layer 64/65 with 3.57 GiB reserved-but-unallocated — pure fragmentation; expandable_segments closed it. ~7 min per attempt.
- ★ FIXED IN THE SCRIPT, after biting THREE rounds: post_quant.py crashed with FileNotFoundError: model.safetensors.index.json because a sub-23 GB quant saves a single bare shard with no index. It now builds the index from the safetensors headers (struct-unpack the u64 length + parse the JSON — never safe_open, which mmaps 22 GB and ENOMEMs on ZFS). Previously hand-fixed each time and never in the script.
- ⚠ Quant dir written root/0600 again — normalized to llmuser 0664. Fourth time this session; worth fixing in quant_mixed_nvfp4.py. ⚠ compare_mtp_head.py prints a hardcoded "already measured at 47.7% acceptance" line on an IDENTICAL verdict — stale text, not a measurement of this head. ⚠ Imatrix deferred by operator (park id 42); the log confirms the usual imatrix_mse … Falling back to uniform MSE, so this build stays apples-to-apples with heresy's PPL 6.910.
- ⏭ NEXT: the acceptance/PPL/surface gate, then the cutover. Rollback stays .env.bak-coldfusion-L35-20260820 → heresy.
⛔ COLD-FUSION ABANDONED — GEN SEAT ROLLED BACK TO heresy 2026-08-21 00:28 (operator directive). The operator's call, made in advance of the result: "If it's the base, we abandon h300 AND the base and chalk it up to a very powerful and useful learning experience. Our heretic methodology will definitely translate in the future." The measurement came back base, so the condition fired.
- LIVE GEN SEAT = /tank/aimodels/qwen38-27b-heresy-nvfp4-mixed (MuXodious/Qwen3.8-27B-absolute-heresy through our mixed NVFP4+FP8 recipe). Restored from .env.bak-coldfusion-L35-20260820; the h300 env is preserved at .env.bak-h300-abandoned-20260821. Verified: healthy, KV 403,065 tok / 1.54× — its exact documented baseline, 7/7 aliases 200, vision intact (red circle / blue square / green rectangle).
- ★ The clincher: <think> is not even in heresy's top-20 first tokens (<0.002), against Cold-Fusion's 0.185. That is a >100× gap — the two families are categorically different on this axis, and it is why no rollback inside Cold-Fusion (L35 or stock) would have helped.
- WHY NOT just apply the chat_template_kwargs fix? It worked (8/30 → 0/30) but it is a workaround for a base the operator no longer wants: it forces gen to become a thinking deployment to paper over a finetune whose whole purpose is reasoning compression. Rolling back removes the defect at the root and restores a build already operator-confirmed "working very well" in real multi-turn use (2026-08-17, coherent through 60k tokens).
- COST, stated plainly: we give up 8/100 refusals (h300) and go back to 29/100 (heresy's own bar) — a 3.6× regression on the refusal axis, which was the entire point of the Heretic-300 run. Also lost: the in-band-vs-pristine MTP comparison stays academic. Accepted deliberately — a seat that breaks the operator's daily client is worth less than one that occasionally refuses.
- ★ WHAT CARRIES FORWARD (the operator's point, and it is right). None of the Heretic-300 learning was in the Cold-Fusion weights. Still valid and model-agnostic: direction_scope=0 beats per-layer on a merged base (8/100 vs 52/100); aggression is not the lever (r=−0.561); PR #317 silently drops the MTP head on save — always diff tensor keys; the MPOA/sink-screen reasoning; graft_mtp.py, kl_divergence.py, catatonia_gate.py, heretic_export.py; a pristine MTP graft accepts as well as an in-band edit (59.7% vs 59.1%); and the new think_prior.py probe. The methodology is the deliverable; the base was the wrong substrate.
- ★★ NEW ACCEPTANCE GATE, earned here — add a FORMAT-COMPLIANCE check to every abliteration/base evaluation, and run it on the STOCK BASE BEFORE spending a GPU-week. think_prior.py on the stock candidate is a ~10s CPU measurement that would have disqualified Cold-Fusion before the 300-trial study ever ran. Screen candidate bases for it. (Related: Heretic's objective has no format term at all — same blindness that removed the self-harm guardrail.)
- 🗑️ bf16 WEIGHTS DELETED 2026-08-21 00:50 (operator directive) — ~154 GB reclaimed, verified against ZFS used (4.48T → 4.33T), no snapshots holding blocks. Removed with explicit literal paths, one at a time: qwen38-27b-coldfusion-bf16 (stock), -abliterated-L35-bf16, -h300-mtp-bf16, -heretic300-bf16. ⚠ The last two were hardlink twins (links=2, same inode — the MTP graft hardlinked every unchanged shard), so deleting only one would have freed nothing; du across several paths in one invocation dedupes hardlinks and under-reported heretic300-bf16 as 2.5K. Check stat -c %h before sizing a delete.
- KEPT deliberately: qwen38-27b-coldfusion-bf16.PROVENANCE.txt (4 KB — carries the pinned HF revision 9c44193f07782c85c0f437a5d8466ba5c95c95fe, so stock is re-pullable) and /tank/aimodels/coldfusion-abliteration/ (1.1 GB — harness, the 300-trial Optuna journal, catatonia-T260.json). The research record survives the weights. With those two, every deleted build is reproducible: re-pull stock at the pin, replay the winning config.
- ⏳ STILL ON DISK, not named in the delete directive: qwen38-27b-coldfusion-h300-nvfp4-mixed (22 GB — the only remaining servable copy of the Heretic-300 result) and qwen38-27b-coldfusion-L35-nvfp4-mixed (22 GB). The directive said "bf16 weights"; these are quants, so they were held pending an explicit call. No storage pressure argues for haste (4.26T free).
⚪ GEN SEAT DEFECT 2026-08-21 — RESOLVED BY THE ROLLBACK ABOVE. The h300 build emitted an UNTERMINATED <think> into content, ~27% of the time, on any temp>0 alias. Operator-reported via Lobe ("sends CoT, never completes the turn").
- Mechanism. With enable_thinking:false the chat template appends a pre-closed <think>\n\n</think>\n\n to the prompt (jinja L165-166). The h300 model opens a fresh <think> anyway and never closes it — verified raw: has </think>: False, finish_reason: stop, reasoning and answer in one content blob starting <think>Ok, let's figure this out:. vLLM's qwen3 reasoning parser can't catch it: the prompt already closed the block, so the parser isn't in reasoning state and the tag is just text (reasoning_content empty, reasoning_tokens: 0). The client is blameless — Lobe correctly treats an unterminated <think> as still-thinking, so it renders an endless thought bubble and never shows the answer.
- ★ It is a SAMPLING event, and the trigger is TEMPERATURE — not presence_penalty. n=12 per arm on the reproducer: pp 1.5 → 4 leaks, pp 0.0 → 4, pp 0.5 → 3 (all the same), temperature 0 → 0. ⚠️ This FALSIFIES the standing "presence_penalty 1.5 is the first dial to move" hypothesis recorded in the litellm config comment and by the operator 2026-08-16 — it is not this bug's cause. Leave that dial alone for this symptom.
- Blast radius = exactly the two temp-0.7 aliases. gen and summarizer-large leak (~17-27%); summarizer, classifier, image-judge, qwen-image-bench are all temp=0 and clean at 0/12 — so nevermore's summarizer path is NOT affected. gen-reasoning doesn't leak (its think block is legitimately open) but shows the other symptom, empty content, at ~1/12.
- ★★ CAUSE SETTLED BY MEASUREMENT 2026-08-21 — it is the BASE, not our abliteration (~83% / ~17%). Measured P(<think>) at the first token with the template rendered enable_thinking=false (so the prompt already contains a closed think pair), raw softmax, bf16, CPU-only — no GPU contention, no seat downtime (96 EPYC cores, 265 GB RAM; ~10s to load a 52 GB model from ARC). Deterministic — stock reproduced to 17 s.f. across two runs.
  
  build abliteration P(<think>) rank
  
  coldfusion-bf16 none (stock) 0.1850 3
  
  coldfusion-abliterated-L35-bf16 Robinson L35, mild 0.2048 2
  
  coldfusion-h300-mtp-bf16 Heretic-300, heavy 0.2216 2
  
  The stock, untouched base already puts 18.5% of first-token mass on opening a think block the template had closed. Abliteration adds a real, monotonic, dose-dependent +3.7 points (+2.0 L35, +1.7 more h300) — a nudge on a pre-existing 18.5-point base. Root cause is the Cold-Fusion finetune itself, which our own PROVENANCE describes as "reasoning-token compression (thinking cut to 1/10..1/2 of stock)" — a model trained to think, briefly. The leak's text shape confirms it: a compact, correct trace with a trained transition marker (Ok, I'm ready to generate my response:), which is trained behavior, not damage.
- ⚠️ ACTION CONSEQUENCE — ROLLING BACK DOES NOT FIX IT. At 18.5% mass under temp 0.7 / top_p 0.8, stock Cold-Fusion and L35 would leak at nearly the h300 rate. The only rollback that escapes is leaving the Cold-Fusion family entirely (→ qwen38-27b-heresy-nvfp4-mixed), which costs the 8/100 refusal result. The chat_template_kwargs fix is the correct lever, not a model swap.
- ★ A FORWARD-KL BUDGET CANNOT CATCH THIS — do not quote KL as evidence of innocence. Heretic minimizes forward KL(stock‖abliterated), which is near-blind to the model putting NEW mass on tokens stock barely used — that is what reverse KL penalizes, and we measured exactly that asymmetry on L35 (reverse 1.43 vs forward 0.70, "the mass-where-stock-had-none signature"). So h300's KL 0.0136 proves nothing here; only the direct measurement did. Generalizes: for any "did the abliteration break behavior X?" question, measure P(token) directly — the KL number is the wrong instrument.
- Consistent with the NVFP4 quant contributing ~nothing: the live quantized seat reads 0.2267 vs its bf16 parent's 0.2216. ⚠ different stacks and a possible vLLM temperature scaling make that suggestive, not proven.
- Full method + reproducers: services/gen-seat-mixed-quant/bench/think-leak/ (think-prior-dose-response.json, README.md).
- FIX, validated n=30 over 4 prompt types + a 3-turn conversation: chat_template_kwargs: {enable_thinking: true, reasoning_effort: low} on gen → 0/30 leaks (current config: 8/30, worst on prose 4/6). Give the model a legitimately open <think> and it closes it properly, the parser does its job, content comes out clean. Cost ~+27% completion tokens (257 vs 202 avg) and a residual 1/30 empty-content. Semantic change: gen stops being a non-thinking deployment — operator's call, not applied.
- ★ PROCESS LESSON: the 7/7 alias smoke test structurally CANNOT catch this. Trivial prompts ("Reply with exactly: OK-gen") never invite reasoning, so they never sample the leaking token. Same shape as the compose file's own warning that single-turn probes missed the xhigh budget bug. Probe with a reasoning-inviting prompt at n≥12, and grep the raw content for <think> — never just check HTTP 200. The tell was sitting in my own eval_coldfusion_h300.json output the night of the cutover and I read past it.
⚪ GEN SEAT — h300 WAS LIVE 2026-08-20 23:05 → 2026-08-21 00:28, then ABANDONED (see the ⛔ entry above). Historical record of that window: GEN_MODEL=/tank/aimodels/qwen38-27b-coldfusion-h300-nvfp4-mixed on gen-seat/vllm-gen, ana-ml2 GPU0 :8015, served-name unchanged (qwen3.8-27b-uncensored / -thinking) so all 7 LiteLLM aliases route without a gateway edit. Verified end to end: healthy in 5.5 min; KV 401,550 tok / 1.53× (baseline 403k/1.54× — within noise); 7/7 aliases green through LiteLLM; VISION INTACT (correctly enumerated colour/form/position of 3 shapes — the surface that had never been exercised after abliteration → MTP-dropping export → graft → quant); MTP acceptance 59.7% median @ 118.37 tok/s (bench/mtp_coldfusion_h300.json) — statistically identical to L35's 59.1% @ 118.71 on the same instrument, so the roadmap's "~47% for a pristine graft" prediction was WRONG — a pristine graft accepts as well as an in-band one. ⚠ A single long-prose sample read 47.5%; the 8-run spread is 49.0–65.4%, so one sample cannot characterize acceptance — always use quickbench.py. Deterministic quality gens all correct (heat-pump, primes=77, 14:20→17:05, prose); abliteration survival 4/4 compliance. ⏳ PPL NOT MEASURED — eval_quality.py aborts with "prompt_logprobs look uniform" because the seat runs --speculative-config; the documented workaround is a spec-decode-free probe seat (bench/serve_probe.sh, :8017), and there is no VRAM for one (GPU0 5.2 GB free, GPU1 2.4 GB free). Comparison target = heresy's 6.910 mean / 5.625 median. ROLLBACK (one line): sudo cp /opt/docker/compose/gen-seat/.env.bak-pre-h300-20260820 /opt/docker/compose/gen-seat/.env && cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen → back to qwen38-27b-coldfusion-L35-nvfp4-mixed. Do NOT delete -L35-nvfp4-mixed or qwen38-27b-heresy-nvfp4-mixed. ⚠ The self-harm guardrail is GONE on this build (operator's own next work item). Also normalized the quant dir from root:0600 to llmuser:llmuser 0664 to match every other model dir. Confirmed the right weights are mounted on TWO discriminating views — mtime and a 64 MB head-hash both match h300 and differ from L35; config.json sha256 is identical across both builds and therefore useless as a discriminator (it carries no weight-specific content — don't reach for it again).
⚪ PRIOR GEN SEAT — absolute-heresy 2026-08-17 (validated, promoted; superseded by L35 then h300 on 2026-08-20). Live gen = /tank/aimodels/qwen38-27b-heresy-nvfp4-mixed — MuXodious/Qwen3.8-27B-absolute-heresy (Heretic v1.4.0 + SOMPOA, trial T377, pin c2374593) put through our own mixed NVFP4+FP8 recipe. Chosen because it beats the incumbent on both axes at once: author refusals 2/101 vs 12/100, first-token KL 0.0759 vs 0.1191. Gate (probe :8017, pinned nightly, seat-matched flags): MTP 47.2% (inc. 48.2%), decode 103.5 tok/s (96.4), prefill 6618/5403 @6.7k/27k (6334/5085), PPL 6.910 (7.059 — 2.1% BETTER), surface 6/6, abliteration 4/4, and 0/55 refusals on our battery-instruct arm with ZERO EMPTY (no catatonia). ⚠ speed deltas are image-confounded (probe on the pinned nightly, incumbent numbers from an earlier image) — read as "not worse", not a clean win. All 7 LiteLLM aliases verified end-to-end; GPU0 at 91.3/97.9 GB with meromero healthy (more headroom than the old build's 96.8). ⚠ RC1, 2 days old, ~348 downloads. Operator-confirmed "working very well" in real use 2026-08-17, same evening as the cutover — the signal the synthetic gates structurally cannot give (multi-turn degeneration is stochastic; four synthetic tests once validated three non-fixes). Not yet the 60k-token bar the prior seat cleared, so keep watching and do NOT delete the rollback weights yet. ROLLBACK: sudo cp /opt/docker/compose/gen-seat/.env.bak-heresy-20260817 /opt/docker/compose/gen-seat/.env && cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen; incumbent weights UNTOUCHED at qwen38-27b-uncensored-nvfp4-mixed — do NOT delete until this holds. Runbook services/gen-seat-mixed-quant/RUNBOOK-heresy-swap.md.
🟢 PRIOR GEN SEAT — RESOLVED 2026-08-17 (the multi-day degeneration saga); now the ROLLBACK target. Was the in-house JonathanColetti/Heretic mixed NVFP4+FP8 build (/tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed, FP8 attention) on vLLM nightly PINNED vllm/vllm-openai:nightly-311b3513… (v0.27.2rc1.dev150, carries #51113 mamba fix), MTP ON, prefix-caching ON. Operator-confirmed coherent through 60k tokens real multi-turn. Root cause = TWO compounding real causes: (1) genuine vLLM qwen3_5_mtp×GDN partial-accept bug (#51113, architectural across vLLM/SGLang/llama.cpp, fixed by nightly), and (2) AEON's full W4A4 being lowest-fidelity on the known activation gradient (W4A4 < W4+FP8 < W4+bf16) → ~15-20% stochastic degeneration on top of (1). AEON PURGED (re-pullable sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4). Full lesson docs/pfi/model-quantization-playbook.md §3.8. (Superseded as primary by absolute-heresy 2026-08-17.) ⚠ pinned nightly is bleeding-edge — move to a stable release once #51113 ships in one (the standing follow-up). 7 aliases (gen/gen-reasoning/summarizer/-large/classifier/image-judge/qwen-image-bench) all route here. Seat carries --default-chat-template-kwargs '{"reasoning_effort":"medium"}' (per-request overridable, affects gen-reasoning only). Commits d28a371,2f2bbce,2185964.
🔵 RP SEAT — FABLE-FUSION serving char-rp-reasoning (evaluation window, unchanged this session). fablefusion-charrp-probe ana-ml2 GPU1 :8019 serving char-rp-probe (kkuspa/Qwen3.6-27B-Fable-Fusion-711-…-MTP-NVFP4A16). LiteLLM char-rp-reasoning + char-rp-fable both route to it (deliberate repoint, documented in stacks/litellm/conf/config.yaml). darkscarlett-charrp-reasoning is compose down, weights intact at /tank/aimodels/darkscarlett-nvfp4-work/. ⏳ STILL AWAITING operator's hands-on read of FF prose (refusal question settled: FF 15.8% vs DS 92.5% cold-framing; DS v1.0 never abliterated). ⚠ FF reasons 2.1–4.6k chars → use max_tokens ≥3072. ReadyArt/Dark-Scarlett-27B-v2.0 (Qwen3.8) is GATED (403 awaiting review) — operator ruled not-interesting, do NOT re-propose. DS regeneration for brokkr RETIRED 2026-08-17 — unqueued, do NOT run (9c1405b): brokkr withdrew on the operator's call because (a) our ictrl-pair-unwrapped/-wrapped control isolates the classifier over-fire cleanly where DS's cross-class delta only bounded it, and (b) DS v2 releases soon, so a k=5 v1 baseline baselines a superseded version. Spec at services/refusal-probe/darkscarlett-regen-spec.md stays banked as the record of the run that will not happen (axes + per-class grading asymmetry still correct, checklist struck through). No GPU1 window was ever spent. A DS-v2 characterization would be a fresh purpose-scoped ask.
🟢 LOBE CHAT — LIVE on esh-docker-vm :3210 (2026-08-17). Replaces the hand-rolled gateway-chat HTML surface. stacks/lobe-chat/, image lobehub/lobe-chat (143 MB compressed vs Open WebUI's 1.8 GB — the weight call). Scoped LiteLLM key lobe-chat-esh (free-local models only; paid GLM/Kimi BLOCKED, verified). Secrets vaulted esh-docker-vm/lobe-chat-*. TTS = a SPLIT: endpoint env-driven (inherits OPENAI_PROXY_URL→ext-tts), but voice/model/format UI-only. System-agent repointed off its gpt-5-mini default onto fleet models via SYSTEM_AGENT env. TTS FIXED 2026-08-18 — no UI pass needed. Lobe's TTS had never worked: it sends {input, model:"tts-1", voice} and LiteLLM resolves the model name FIRST, so tts-1 403'd against the scoped key's allow-list and never reached the gateway (our "unknown model routes to the gateway default" belief was true of :8198 and false of the LiteLLM path — that's what hid it). Fixed by aliasing the stock names rather than patching the client: tts-1, tts-1-hd, gpt-4o-mini-tts /model/new'd to openai/zonos@10.100.79.3:8198/v1 (mode: audio_speech), plus those three added to the lobe-chat-esh allow-list (20→23). Verified with Lobe's exact payload on Lobe's own key: 200, 69,740 B, MPEG. ⚠ These three are DB rows, not references — if ext-tts repoints, they must move with it. Done with the infra-ops admin key, not sk-corvid: it has /model/new + /key/update rights, so this class of ask never needs the master key. Also live: tts-gateway v4 defaults response_format to mp3 (tts-dev shipped it; 122,924 B wav → 27,692 B mp3 same utterance; every in-house consumer already pins the field, blast radius checked pre-ship). Commits e9362de,163a725,cac75cb,933253d,ca8c0a3 (last one authored by tts-dev correcting two load-bearing wrong claims in our README/compose — kept).
🟢 LITELLM — upgraded v1.91.0→v1.97.0, spend-log DB purged 6GB→16MB + CAPPED (2026-08-17). store_prompts_in_spend_logs:false + maximum_spend_logs_retention_period:7d. ⚠ 1.8GB pre-upgrade pg_dump still on ana-docker /opt/docker/compose/litellm/ — deletable now the upgrade is proven (operator was going to call it). Commit 01b5ad9.
⚠️ GPU zero-sum (both cards ~94–95/97.9 GB). GPU0: gen + meromero. GPU1: fablefusion + utility cluster. Any util bump on either seat of a shared card must be checked against the co-tenant (starved meromero into a crash-loop once at 0.45). ⚠️ BOOT ORDER IS PART OF THE STATE (2026-08-20). --gpu-memory-utilization sets the target as a fraction of TOTAL VRAM, but vLLM refuses to start unless that whole target is FREE right now — so at ~96.4/97.9 GB the GPU0 pair coexists only in the order it was originally brought up. Restore/reboot order: vllm-meromero-rp to healthy FIRST, then vllm-gen — meromero (0.52 = 49.38 GiB) is the one that cannot fit in the remainder. "First" means observed healthy, not a sleep: a 10s gap against a 2–3 min weight load cost a 7-restart crash-loop. Verify a restore against KV-pool size (GPU KV cache size / Maximum concurrency in the container log), not nvidia-smi used-MiB — the latter swings ~7 GB on allocator slack with identical serving capacity. Baselines: gen ≈14.36 GiB / 403k tok / 1.54× (h300 build: 401,550 tok / 1.53×); meromero 542,202 tok. 📊 MEASURED VRAM CENSUS 2026-08-20 23:20 (nvidia-smi PID→container, not util-fraction guesses) — GPU0 92,572/97,887 MiB (94.6%), 5.2 GB free: meromero 50,072 + gen 42,500. GPU1 81,448/97,887 MiB (83.2%), 16.1 GB free (after the lfm25 + reranker cleanups below; was 95,388 / 2.4 GB free — 13.9 GB reclaimed in one night): fablefusion-probe 43,452 + selene 16,870 + reward 9,512 + coder 6,158 + embed 3,304 + rerank-a3 2,112. Still no room for a ~22 GB PPL probe seat — fablefusion is the only remaining block big enough. ⚠ fablefusion is the single biggest reclaimable block (43.4 GB) and is nearly idle — LiteLLM spend logs show char-rp-probe at 4 calls, last 2026-08-19 08:52, vs char-rp (meromero) at 129 calls, last 2026-08-20 15:28.
FLEET RERANKER = A3 (bge-reranker-v2-m3) PROD ana-ml2 GPU1 :8013. Passive watch; levers = A4 :8014 / util / 2nd replica; incumbent :8002 warm. docs/pfi/reranker-selection-ledger.md.
🟢 RERANKER + NEVERMORE CLEANUP — RESOLVED 2026-08-20 (found while answering "why do we have 3 rerankers?"). The R43 cutover was only half-landed: it repointed the reranker alias but never moved the one consumer that mattered.
- ★ THE BIG ONE — nevermore's LLM summarization had been DEAD for 8 days and nothing noticed. Its .env pinned LLAMA_SWAP_MODEL=granite-4.1-8b, an alias retired 2026-08-12 with the granite seat. Result: 67 consecutive failures, 0 tokens, status=failure, twice daily, silently — the twice-daily briefing was rendering with no LLM pass at all. Nothing alerts on status=failure in the spend logs, so this was invisible until someone went looking for something else.
- Also found: NEVERMORE_RERANK_MODEL=qwen3-reranker — nevermore was the ONLY caller of the retired Qwen incumbent (7 calls, 12-hourly at 03:00/15:00 UTC = its 08:00/20:00 PDT cron), the model Brokkr measured harming 80/90 fleet queries. Its rerank calls succeeded; they were just running through the bad model. Meanwhile the production reranker alias had 0 calls in 4 days.
- ★ THE RULE THIS EARNS: retiring a model is a TWO-SIDED operation. Grep every consumer's config for the alias before deleting it. And consumers must pin stable capability aliases (summarizer, reranker) never model names (granite-4.1-8b, qwen3-reranker) — then the gateway can repoint without anyone editing a downstream .env. Both of nevermore's breakages are the same bug.
- FIXED: nevermore .env → LLAMA_SWAP_MODEL=summarizer + NEVERMORE_RERANK_MODEL=reranker (backup .env.bak-pre-model-repoint-20260820), worker recreated, all three deps verified live — summarizer returns clean content with 0 reasoning chars at nevermore's exact call shape (temperature 0.2, max_tokens 4000), reranker scores 0.95 on-topic vs ~1e-5 off-topic, embedding returns dim-1024. ⚠ NEVERMORE_EMBED_MODEL=qwen3-embedding was already correct — left alone. ⚠ nevermore's .env is server-only (.env is excluded from the mirror both ways), so this fix is not in git.
- RETIRED: vllm-rerank (:8002 Qwen incumbent) + its qwen3-reranker alias; vllm-rerank-a4 (:8014) + its alias; vllm-granite (Exited 8 days, dead service block). A3 PROMOTED from a throwaway docker run into stacks/vllm as service vllm-rerank-a3 (the ledger's own open follow-up) — healthy in 55s, image pinned. Container name deliberately keeps the bake-off arm name so the ledger/memory/R43 references stay valid.
- ⚠️ CORRECTION — my "A4 has no alias at all" claim was WRONG. reranker-a4-gte-modernbert did exist; I grepped config.yaml and concluded absence. LiteLLM serves BOTH config-defined AND DB-defined models — live was 32 models, config.yaml only 26. The 6 DB-only ones: ext-tts, gpt-4o-mini-tts, tts-1, tts-1-hd, reranker-a3-bge-v2-m3, reranker-a4-gte-modernbert. /v1/models (or /model/info, which flags db_model: true) is the ground truth — never config.yaml alone. DB models delete hot via POST /model/delete {"id": …} with no gateway restart; config models need a file edit + docker restart litellm (~84s). A4's alias was deleted that way once its backend was gone.
- ⏳ REMAINING: reranker-a3-bge-v2-m3 (DB-defined, id 1f08a73e-c09c-463c-8f18-5edea51fb736) still exists as a duplicate of reranker on the same backend, 0 calls. It's Brokkr's cutover-verification handle (prod == arm at maxdiff 0.000000), so not removed unilaterally — it is another agent's tooling, and it is redundant rather than broken. Ask Brokkr before deleting.
EVIDENCE HOLD (partial): WT #394 FILE half STILL STANDS — do NOT delete on-disk gen dirs (fiction/rex390-dcc, rex392-dcc, b59c147c5ce0); rex393-fiction-* + r42-gate-* KEEP.
🔴 NH3/AT&T IPv6 PD is a HARD /64 — TESTED ON THE WIRE 2026-08-19, not inferred. AT&T support (Christopher, prov-dns@att.com) said "I do not believe att will do that" and punted to consumer sales — and he was right, though his team is DNS provisioning and the answer was a guess. Verified empirically: the NH3 UDM solicited DHCPv6-PD at /48, /56 AND /60 and AT&T returned the same single /64 (2600:1700:b25:c11f::/64) every time. So it is NOT "nobody ever asked" — the request was made three ways and the server answers /64 regardless. How it was proven: the PD is invisible until a LAN claims it, so nh3-iot (the only NH3 VLAN with 0 clients) was temporarily set ipv6_interface_type: pd; then ipv6_pd_prefixid was set to 0, 15 and 16 — all three returned the identical c11f prefix, which is the signature of a single delegated /64 (with any larger block the prefix-id moves the LAN within it). ⚠ Do NOT read the WAN address vs LAN prefix gap as evidence of block size — WAN is c110, delegated is c11f, and I initially misread that spread as a /60; AT&T just assigns the two from different places in their pool. NH3 UDM was fully restored (nh3-iot back to none, WAN back to pd_size=64); rollback artifacts at nh3-dev:~/backups/nh3-udm/. THE ASK FOR AT&T BUSINESS (this is business fiber at a commercial park — the residential org cannot change it, and 800-288-2020 is the wrong number): "circuit solicits DHCPv6-PD at /48, /56 and /60; server returns a single /64 in all three cases; please increase the delegation size on the provisioning profile" — a claim they can verify against their own DHCPv6 logs. ROOT CAUSE FOUND 2026-08-19 — it is the BGW, not AT&T, and EIGHT /64s are actually available. The CPE is a BGW210-700, firmware 4.28.7,management reachable at http://192.168.1.254 (v4 is already IP-passthrough — the UDM holds public 70.230.226.88 directly — but passthrough covers v4 ONLY; v6 always terminates on the BGW). Its /cgi-bin/lanstatistics.ha and /cgi-bin/broadbandstatistics.ha are readable unauthenticated and give the whole picture: BGW WAN v6 = 2001:506:70b2:8958::1, BGW LAN v6 = 2600:1700:b25:c110::/64 (it keeps this for itself), delegated to the UDM = 2600:1700:b25:c11f::/64. So AT&T DOES hand the BGW a /60 (c110-c11f); the BGW keeps c110-c117 and re-delegates up to EIGHT individual /64s on c118-c11f, top-down. Our UDM got c11f = delegation #8. The other seven are sitting unclaimed — not withheld by AT&T, just never solicited, because UniFi exposes only a single wan_dhcpv6_pd_size integer and no 'number of prefixes' field, so it asks once. The documented workaround is dhclient -6 -P -P -P... (one -P per prefix), which UniFi cannot express. ⇒ An AT&T ticket CANNOT fix this — it is CPE firmware behaviour, so Christopher's 'att won't do that' was accidentally right. Real options: (a) accept one /64; (b) full BGW bypass via wpa_supplicant/EAP-TLS on the UDM (documented working on UDM Pro; extract the BGW's 802.1X certs, authenticate to AT&T directly, negotiate the whole /60 yourself) — ⚠ modifies NH3's ONLY uplink and must survive UDM firmware updates, so it needs a planned window with physical access, NOT an ad-hoc change. ⏳ PARKED pending the Device Access Code: reclaim-nh3-s-7-unclaimed-ipv6-64s-from-the on the henge. Operator will fetch the BGW210 Device Access Code (12 digits, on the unit's label) next time he is at the NH3 office and vault it — suggested path unifi/bgw210-nh3-device-access-code — after which this is drivable remotely. First stop is http://192.168.1.254/cgi-bin/ip6lan.ha (confirmed to exist, currently redirects to Login); check for a delegation size/count control, a v6 pass-through/bridge option, or a way to stop the BGW reserving c110-c117. If the UI can do it, that is a SUPPORTED fix and the wpa_supplicant bypass is unnecessary. Other settings pages behind the same login: ippass.ha (how v4 passthrough was set up), dhcpserver.ha, firewall.ha, broadbandconfig.ha, packetfilter.ha, remoteaccess.ha, restart.ha. Consequence if left alone: NH3 has ONE v6 segment, vs ESH's /56 (256), which makes NH3 the prefix-constrained site in any fleet v6 design.
⏳ COGENT IPv6 PROVISIONING REQUESTED for Anaheim/colo (operator, 2026-08-19). The colo is the last site with no v6 (ESH has a /56 from Cityside, NH3 has AT&T v6 on 2600:1700:b25:c110::48). Diagnosis from our side before the ticket: diagnose sniffer packet wan1 "ip6" on ana-gw caught exactly one RA in 90s, from fe80::ea0a:b9ff:fe3b:2c16 — so an IPv6-capable router IS one hop away on the circuit terminating 38.120.12.42/29 — but set autoconf enable (SLAAC) got no global address after 2min of RA intervals, and execute ping6 to both 2606:4700:4700::1111 and 2001:4860:4860::8888 was 100% loss. That pattern = router present, circuit not provisioned for v6 (RA likely carries no Prefix Information Option). FortiGate-side v6 config was fully reverted after testing — wan1 is back to no v6, v4/IPsec untouched throughout. ⚠ FortiOS gotcha for when it lands: SLAAC is config ipv6 / set autoconf enable, NOT an ip6-mode (that only takes static/dhcp/pppoe/delegated). WHEN COGENT PROVISIONS: ask for DHCPv6-PD size (ESH got a /56 from Cityside; NH3 only gets a single /64 from AT&T, so a /56 or better is the ask), then the colo becomes the fleet's only site with BOTH a static public v4 (38.120.12.42) and routable v6 — which makes it the natural v6 hub, since ESH is CGNAT'd on v4 and NH3 is prefix-constrained.
🟡 ESH IPv6 LAN naming — a DOCS CONVENTION ONLY, not wire-level. Fun hexspeak names picked 2026-08-18 for the six ESH LANs: RESERVED for a future DMZ = 4411:DBAD ("FOR ALL DA BAD" — no DMZ network exists on the ESH UDM yet; name claimed 2026-08-19 against the day one is built). esh-cameras=1533:FACE5 ("I SEE FACES" — operator's pick 2026-08-19, and the best of the set), esh-mgmt=15DA:B055 ("IS DA BOSS" — 15=IS, DA literal, B055=BOSS; revised 2026-08-19), esh-server=4411:B105 ("FOR ALL BIOS" — operator's pick 2026-08-19; shares the 411 with userland's CAFE:4411), esh-userland=CAFE:4411 ("CAFE FOR ALL" — operator's pick 2026-08-19; 411 also being US directory assistance is a bonus), esh-iot=4DBA:D107 ("FOR DA BAD IOT" — 4=FOR, D=DA, BAD, 107=IOT; same 8 digits, re-glossed 2026-08-19), Default=4BA5:3417 ("A BASE FOR IT" — operator's pick 2026-08-19; 8 digits, words straddle the group boundary like 4DBA:D107). Investigated and confirmed these CANNOT be made literal on the wire: (1) a network's only nameable slot is its /64 subnet id (ipv6_pd_prefixid), which is 2 hex digits (0-255) — not enough room to spell a 4-char word; (2) the gateway's own address is fixed at ::1 by the UniFi platform, no field to customize it (ipv6_aliases exists but schema unknown, not attempted blind); (3) UniFi has no IPv6 equivalent of use_fixedip/fixed_ip (checked the client schema directly) — individual devices cannot be pinned to a chosen v6 address the way v4 static reservations work; SLAAC devices self-assign via EUI-64 (MAC-derived, stable) or privacy-extension (random, rotates). So the names are a mnemonic for humans/docs, not something that will ever appear in a dig/ip -6 output. Only esh-cameras has v6 live at all (2607:73c0:402:1d00::/64, PD from the ESH /56); the other five stay ipv6_interface_type: none pending a firewall-policy pass before enabling SLAAC fleet-wide. ⚠ Precision that matters here: SLAAC gives hosts globally ADDRESSABLE addresses, NOT globally reachable ones. Addressability is a property of the address; reachability is a firewall policy decision, and operator's position (2026-08-19) is unambiguous — no 1:1 inbound pass-through, ever. v6 removes NAT, it does not remove the firewall, and conflating the two is how people talk themselves into thinking v6 means automatic exposure. The firewall-policy pass is about writing explicit default-deny inbound rules for each v6 segment, not about deciding whether to expose anything.
🟢 ESH↔colo IPsec FIXED 2026-08-18 — dialup + NAT-T. Broke when ESH lost its public IP (Cox unplugged for the fiber install; 5G handed out RFC1918 192.168.200.111). TWO causes: phase1 ana-to-eshudm was type static pinned to the dead 70.181.90.232, AND nattraversal disable — so even re-pinning the IP would have failed, since ESP cannot cross NAT without UDP-4500. pfi-ana-nh3 survived only because NH3 is publicly addressed. ⚠ FortiOS REFUSES set type dynamic on an existing tunnel ("Cannot change tunnel type once configured", clean rollback). Fix = ADDITIVE, not a recreate: new phase1+phase2 ana-eshudm-dyn (type dynamic, ikev2, aes256-sha1, dh14, NAT-T default on, PSK read from the ESH UDM API), static route id 10 (dstaddr ana-to-eshudm_remote → device ana-eshudm-dyn, distance 20), policies 73/74 (consolidated multi-zone in/out). Old ana-to-eshudm LEFT IN PLACE, dead and harmless, as rollback. Verified: ana-eshudm-dyn_0 97.170.236.56:4500 selectors 1/1 (_0=dialup child, :4500=NAT-T, carrier IP), ESH→colo all hosts 40-56ms, colo→ESH OK, traceroute now 3 hops encapsulated vs 8 leaking to the carrier. FortiGate config backed up first at nh3-dev:~/backups/ana-gw/ (1.17MB, 36903 lines). ⚠ RESIDUAL: the UDM's ipsec_local_ip needs a LITERAL IP (empty rejected api.err.InvalidPayload) — currently 192.168.200.111, so it must be updated again when the fiber lands or the tunnel drops. Gateway end is address-agnostic now; UniFi end is not — it will drop on ANY ESH WAN change: Cox reclaiming WAN1 (still failover_priority=1), the fiber cutover, or a DHCP renewal. MANUAL FIX (seconds): read the UDM's live wan_ip from /stat/health and PUT it into esh-ana.ipsec_local_ip via rest/networkconf/<id> using unifi/esh-udmpm-api-key. A self-healing watcher is PARKED (self-healing-ipsec-local-ip-watcher-for-the-esh), trigger = once the new ESH fiber is live and its addressing has settled — operator's call 2026-08-18, deliberately not built now because it would target an address that is about to change. Park item also carries the follow-up to delete the old ana-to-eshudm tunnel, whose distance-10 route would silently reclaim traffic if Cox ever returns on 70.181.90.232.
🟡 FLEET IPv6 — STILL JUSTIFIED, but the REASON was wrong and is now corrected (2026-08-18). There are TWO inter-site links with OPPOSITE NAT behaviour, and the original note conflated them. (a) NH3↔ESH = Site Magic = WireGuard — SURVIVES arbitrary NAT, PROVEN. During the fiber cutover ESH failed over to 5G on 192.168.200.111 (RFC1918 double-NAT, zero inbound path, strictly worse than CGNAT) and it held: nh3-dev/nh3-docker reachable ~40-43ms, ssh+exec fine. Mechanism: magic_site_to_site_vpn on-device is only enabled+keypair (no WAN binding — it rides whichever uplink is active), and NH3's edge is public at 70.230.226.88, so the NAT'd side dials out and never needs reachability. (b) colo(Anaheim)↔ESH = IPsec on the ana-gw FortiGate — BROKEN RIGHT NOW under those same conditions. ana-docker/pfi-pve/pbs-ana ALL FAIL from esh-pve-nas; traceroute shows 10.250.x packets leaving the UDM to the 5G modem 192.168.200.99, then wandering the carrier (10.180.141.194, 172.19.3.14) and dying — not encapsulated at all, so no SA is up and traffic falls to the default route. Site-to-site IPsec pins a peer IP; ESH no longer has a routable one. ⇒ IPv6 keeps its justification, on the IPsec link SPECIFICALLY. ⚠ An earlier version of this entry claimed the premise was refuted outright — that over-generalised a WireGuard result onto an IPsec link. Operator caught it. See ops-lessons-playbook §8. State: NH3 WAN live (2600:1700:b25:c110::48, AT&T delegates exactly one /64), colo none (FortiGate has zero v6), ESH none (both WANs wan_type_v6=disabled). NH3 LANs all reverted to ipv6_interface_type=none per operator. Work when addresses land: v6 on ana-wg eth0 + a v6 port-forward for UDP 31337 on the FortiGate (its WG socket is already dual-stack — no WG reconfig), flip the UDM WG server off v4-pinned binding, and AAAA records so the dynamic prefixes at all three sites don't break endpoints. Full detail + access recipes → persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md.
🟢 WT #401 (fd-leak deadlock) CLOSED 2026-08-17 — one ping still owed. worldtree-dev closed it on our demo verify. Layers: (a) their e41b139 pins ulimits: nofile 65536/65536 in the worldtree compose anchor — demo VERIFIED (api + matrix recreated 22:55:34Z, ulimit -Sn=65536); personal/pinned are covered-not-verified, they inherit at their next promotion/recreate. (b) our host floor is STAGED, NOT ACTIVE — /etc/docker/daemon.json on corviduo-dev carries default-ulimits nofile 65536/65536 but default-ulimits is NOT SIGHUP-reloadable (measured on 29.4.3: post-reload the daemon's own "Reloaded configuration" log omits it and a fresh container still reports 1024). Activation needs a full dockerd restart = bounces all 13 containers; worldtree-dev explicitly does NOT want one, and live-restore:true-then-restart is PARKED as a separate host-side improvement for the operator to rule on, never folded into #401. Playbook playbooks/corviduo-dev-docker-default-ulimits.yaml (verify step 3 fails BY DESIGN until a restart). Hourly fd tripwire on corviduo-dev stays armed. ⏳ OWED: ping worldtree-dev in thread 01M08QQ655XD6VKEV7MA9GX0NS once worldtree-personal recreates and 65536 is confirmed there. Commit 7f3f265.
🔴 HERETIC-300 EPIC — 2026-08-20, THE HEADLINE RESULT (supersedes the Robinson build below). Ran Heretic v1.4.0's 300-trial TPE search on Cold-Fusion: 8/100 refusals at KL 0.0136 from a 98/100 base — beats the absolute-heresy bar (29/100) 3.6× at essentially the damage our hand-tuned L35 build spent for only 72/100. Hand-verified coherent (correct arithmetic with working, clean code, 66–167-word prose) — the catatonia gate's two DEGENERATE verdicts were false positives from my own letters/space heuristic flagging correct numeric answers. ⚠️ SELF-HARM GUARDRAIL IS GONE — answers method requests in full where L35 redirected to 988; operator is handling restoration directly and does not want parallel dwarf analysis on it. ★ Durable: direction_scope=0 (single shared direction) beats per-layer decisively (best 8/100 vs 52/100) — points AGAINST multi-direction on a merged base; aggression is NOT the lever (r=−0.561, the KL<0.02 band holds both the worst and the best); PR #317 silently drops the entire MTP head on save (1199→1184 tensors, exit 0, no warning — always diff keys after a Heretic export); Heretic's direction is 6.18% sink-dominated yet safe for Heretic because of magnitude-preserving ablation, so our sink screen correctly refused an in-band MTP graft. Build: qwen38-27b-coldfusion-h300-mtp-bf16 (1199 tensors verified). In flight at snapshot: NVFP4 quant → post_quant (MANDATORY) → gen-seat cutover (operator's call: gen, not probe — single-user internal WG, prior seat already fully abliterated). Full epic + ROADMAP → persistent-memory.d/2026-08-20-heretic-300-epic.md.
🟢 COLD-FUSION ABLITERATION — LANDED 2026-08-20 (the real work; abliterated model WORKS, verify + quant still owed). Abliterated DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1 with the Robinson formula (docs/pfi/abliteration-recipe-qwen38.md; harness services/coldfusion-abliteration/). Output ana-ml2:/tank/aimodels/qwen38-27b-coldfusion-abliterated-L35-bf16; bitwise-verified 131/131 targets changed, 333/333 vision byte-identical, 735/735 others untouched. A/B vs stock: explicit-sexual + graphic-torture (the measured stock refusal surface) go refused→complied, self-harm guardrail survives, coherence intact — the Robinson design point. THREE first-session diagnoses were wrong, all corrected: (1) layer selection by two-template |cos| agreement is misleading on a merged base — its pick (L18) was the worst-separating layer and abliterating there was a measured no-op; replaced with harmful/harmless separation (Cohen's d/AUC) gated on the sink screen → L35 (d9.35, AUC0.9997, sink0.094%); (2) "bf16 NaNs, use fp32" was a misdiagnosis — the NaN was multi-GPU sharding + expandable_segments, not precision; bf16 on ONE GPU is deterministic+coherent at 50 GB, 4.3× faster (now gated, CUDA_VISIBLE_DEVICES=0); (3) corpus-size hypothesis falsified (8→416 moved agreement 0.594→0.624, nothing). Write is now shard surgery (no model object — AutoModelForCausalLM is text-only and would drop vision + skip the in-band MTP edit). 🎯 THESIS PROVEN 2026-08-20 (same session, later): quantized L35 → mixed NVFP4 (/tank/aimodels/qwen38-27b-coldfusion-L35-nvfp4-mixed, 22.5 GB) and measured MTP acceptance 59.1% median (51–65%) — clears the ≳40% gate AND beats the incumbent Heretic seat's ~47%. So Robinson's in-band MTP abliteration accepts BETTER than Heretic's byte-identical graft — the additive delta the experiment tested, positive. Abliteration survives the quant (creative refusals drop, self-harm guardrail intact, coherent); decode 118.7 tok/s (faster, image-confounded). Env foot-guns hardened: quant venv drifted (transformers 5.10/llmcompressor 0.12 no longer delegate num_attention_heads to text_config → promote-then-restore in quant_mixed_nvfp4.py); a <23GB quant saves single-file no-index → build index from the safetensors header (never safe_open, ENOMEMs on ZFS). Commit 725c8fd. ⚠ "NOT CUT OVER" IS STALE (corrected 2026-08-20 23:30): L35 was made the live gen seat at 11:28 that same day, and was itself superseded by the Heretic-300 build at 23:05. L35 is now the rollback target (.env.bak-pre-h300-20260820). Do NOT delete qwen38-27b-heresy-nvfp4-mixed. 📐 KL MEASURED 2026-08-20 (third session) — the surgery is SELECTIVE. kl_divergence.py (new, beside the harness): first-token KL(stock‖L35) over the full 248,320 vocab, bf16-vs-bf16, held-out prompts. Answer mode: harmless median 0.0211 / mean 0.0364, harmful median 0.5996 → 28.4× selectivity (think mode 0.0042 / 0.3068 → 72.8×); top-1 agreement on benign prompts stays 89.8%; self-KL noise floor exactly 0.0, so every digit is signal. Reverse KL on harmful is 1.43 vs forward 0.70 — the mass-where-stock-had-none asymmetry that is abliteration's signature. Vs the Heretic reference figures (0.1191 prior seat, 0.0759 the live absolute-heresy seat) ours looks materially gentler, but ⚠️ NOT a head-to-head — those are Heretic's own optimizer output on a different base with its own harmless set/template; a real comparison needs the incumbent re-measured through this script (one more window). KL stays a fidelity number, not the gate (that's MTP acceptance, 59.1%). Run cost 2m40s, both GPU0 seats down. Three durable process lessons: (1) report abliteration KL split by prompt class — a mixed average is meaningless since the metric is meant to be big on harmful and small on benign; (2) the runbook's "bf16 is 50 GB / only gen must go" was 50.10 GiB mislabelled — text weights are 51,300 MiB and freeing either GPU0 seat alone leaves ~50,933, so both must stop; (3) a 27B model cannot be released in-process (del+gc+empty_cache and frame-exit both left 45,287 MiB free; only process exit gave the card back) → one process per model, and the old residency gate read hf_device_map, which is empty on single-device loads and therefore could never fail. → playbook §3.12. ⚠️ GPU0 SEAT RESTORE ORDER IS LOAD-BEARING and "first" means HEALTHY, not 10s earlier — start meromero; sleep 10; start gen put meromero in a 7-restart crash-loop (Free memory on cuda:0 (35.3/94.97 GiB) < desired utilization (0.52, 49.38 GiB)). I had checked the compose files, seen --gpu-memory-utilization is a fraction of total, and wrongly concluded order didn't matter — the fraction sets the target but vLLM gates startup on FREE VRAM, so at ~96.4/97.9 GB the seats coexist only in their original boot order. The old auto-memory note was right; my correction of it was the error. Recovery = stop gen → meromero healthy → start gen. Restore VERIFIED against the pre-window baseline (gen KV 14.34 GiB/1.53× vs 14.36/1.54×; meromero 542,202 tok both; restarts=0; summarizer smoke-tested through LiteLLM) — ⚠️ don't use raw nvidia-smi used-MiB for this check, it reads 89,503 vs 96,376 and that 6.9 GB is allocator slack, not lost capacity. Full saga → persistent-memory.d/2026-08-20-coldfusion-abliteration-landed.md.
🟢 esh-pve-nas — MIGRATION DONE 2026-08-18. Root is nvme/ROOT/pve-1 on mirrored NVMe; the USB DOM is out of the runtime I/O path. All five guests healthy, three pools ONLINE, system running, ext4 pve-root intact+unmounted as rollback with its own kernel. Boot config: saved_entry=pve-zfs-root, no next_entry; if grubenv were unreadable GRUB falls to entry 0 which the /etc/default/grub.d/zfs-root.cfg drop-in also points at ZFS. zfs-import-cache.service is now the active import path (the all-three-pools cachefile fix working as intended); vestigial zfs-import@nvme.service disabled — it failed every boot as redundant. ⚠ Device letters shift across reboots (DOM was sdq, now sdl) — never key anything to a bare sdX here. ⚠ NO auto-fallback on a failed boot, and no IPMI/BMC/serial — grubenv on LVM is readable but not writable by GRUB, so grub-reboot's one-shot degrades to a sticky default (verified: next_entry survived the boot that consumed it). Recovery = pick the ROLLBACK entry at the console. PATCHED 2026-08-18: 225 packages installed, pve-manager 8.4.11 -> 8.4.20, corosync 3.1.9 -> 3.1.10-pve2, kernel 6.8.12-42 staged on the /boot LV. dpkg clean, no unapplied conffiles, cluster quorate, 6/6 verify. ⏳ REBOOT DEFERRED at operator request — host still runs 6.8.12-13 until a chosen window; GRUB_DEFAULT=0 means entry 0 is already the -42 entry with the correct root=ZFS=nvme/ROOT/pve-1, so the reboot is the only remaining step. Rollback for the upgrade is the ZFS snapshot nvme/ROOT/pve-1@pre-upgrade-20260818T141652Z (409M) — zfs rollback -r <snap> && reboot. Second confirmation reboot ALREADY DONE (2026-08-18, booted ZFS from GRUB_DEFAULT=0 with no one-shot). ⏳ Still outstanding: refresh the off-box DOM image, since /boot changed. esh-pve is FULLY done (8.4.20 + kernel 6.8.12-42 + corosync 3.1.10, rebooted, quorate). ⚠⚠ THE WINDOW COST AN UNPLANNED OUTAGE, caused by our own tooling, not the migration. The staging chroot did mount --rbind /dev + /sys with no --make-rslave; on systemd / is shared, so the cutover's umount -R propagated back into the live host and stripped the real /sys/fs/cgroup, /dev/pts, /dev/shm. logind could then create no sessions: ping fine, TCP fine, SSH authenticates, resident daemons keep serving (pveproxy returned clean 401s) — and every new exec hangs, including /sbin/reboot, so the reboot never ran. It is a near-perfect impostor of failing root-disk I/O, and I misdiagnosed it as the USB DOM dying and told the operator to walk to the machine. Operator caught it — the DOM had been fine for years and the wedge began right after a change. The settling evidence was in dmesg all along: [16.00] [sdq] Attached SCSI removable disk (clean, no errors) and a last-line timestamp of [12114881] = 140 days = the ORIGINAL boot — the machine had never rebooted. My down-detector never once reported the host down and I read that as a fast reboot rather than no reboot. Recovered with no console access by hammering an idempotent cgroup2/devpts/shm remount into the brief windows where exec succeeded. Zero data loss. RULES: (1) always --make-rslave after --rbind (playbook now guards on PROPAGATION != shared); (2) a reboot is not confirmed until the host is observed DOWN — poll for disappearance, not reappearance; (3) before blaming hardware for a wedge that started right after a change, get dmesg and check the boot timestamp. ⚠ Blast radius was more than DOUBLE what the runbook said — ss -tn '( sport = :2049 )' in CT 103 found five NFS clients, not two: +esh-vm-db 10.0.50.60 (/mnt/backup, hard, no ssh — reach via qm guest exec 101 on esh-pve), +vm-esh-nas (is VM 104 on the host itself), +nh3-dev (soft,ro — safe). Ask the server who its clients are; a documented dependent list rots. esh-vm-db was deliberately left mounted (backup target, timers ~19h out) and came through read-write — a hard mount with no active user blocks and resumes, which is what hard is for. (historical, pre-cutover context below) migration STAGED 2026-08-18. PVE root ran off a NORELSYS USB Disk-on-Module (sdq). Wear was NOT the driver (a DOM is SLC/pSLC — operator corrected my first read); the drivers are the USB bus (a reset drops root under a running hypervisor), no headroom, no mirror, and blocked patching: 225 packages, 161 with deb12uN/security bumps, stuck on PVE 8.4.11 vs esh-pve's 8.4.14. Mitigated 2026-08-17 (90% → 76%); staged 2026-08-18 via two rerunnable elway playbooks (esh-pve-nas-stage-zfs-root.yaml, esh-pve-nas-stage-bootloader.yaml), 0 failed, 17/17 verify green. Host is still on ext4 and its boot path is byte-identical to the last 140 days — grub-install is deliberately held back to the window, so an unplanned reboot mid-staging is a non-event. CUTOVER = quiesce NFS clients → stop guests → grub-install → grub-reboot pve-zfs-root → zfs set mountpoint=/ nvme/ROOT/pve-1 → reboot. ⚠⚠ THE POOL-NAME BUG — the near-miss worth remembering. update-grub silently emits root=ZFS=/ROOT/pve-1 with the pool name missing → boots to an initramfs prompt. Cause: Debian's 10_linux builds ${rpool}${bootfs} where rpool = grub-probe --target=fs_label, and GRUB's ZFS reader cannot open a pool with encryption/large_dnode/zstd_compress — so it returns empty and the failure is swallowed by 2>/dev/null || true. The same feature set that forced /boot to stay ext4 also corrupts the kernel cmdline, which the design did not anticipate. Fixed with a /etc/default/grub.d/zfs-root.cfg drop-in (last root= wins) plus explicit pve-zfs-root / pve-ext4-rollback entries with stable ids. Durable lesson: assert the EFFECTIVE value, not the presence of a substring — the original verify grepped for the right string appearing somewhere and would have passed with every menu entry still broken. ⚠ Never set the ZFS cachefile on one pool. The runbook's zpool set cachefile=… nvme was a trap: populating a cache flips the host from import-by-scan to import-by-cache, so a cache holding only nvme leaves ssd+tank unimported and empties every CT 103 export. Set on all three 2026-08-18, verified in the 11,976-byte cache. ⚠ Migrate FIRST, patch after — a signed kernel would land in /boot on the 1.3 GB root. ⚠ CT 103 esh-nas (10.0.50.50) runs on this host and serves hard NFS to esh-docker-vm and esh-pve — quiesce both before any reboot or you wedge esh-docker-vm into D-state. Off-box at nh3-dev:~/backups/esh-pve-nas/: DOM image dom-sdq-20260818.img.zst (2.38 GiB, crash-consistent), clean bootchain-20260818.tar.gz, config snapshot …20260818T051*.tar.gz. Runbook docs/runbooks/esh-pve-nas-boot-migration.md; detail → persistent-memory.d/2026-08-17-esh-pve-nas-dom.md.
⚪ LFM2.5-2.6B RETIRED PERMANENTLY 2026-08-20 (operator directive). vllm-lfm25 (:8021, GPU1) removed: service deleted from stacks/vllm/compose.yaml + pushed live (backup compose.yaml.bak-pre-lfm25-retire-20260820), container docker rm -f'd, lfm2.5-2.6b alias deleted from the LiteLLM config (live + canonical; backup config.yaml.bak-pre-lfm25-retire-20260820, 28→27 models). Freed 8,721 MiB on GPU1. Justification: it was an EVAL-ONLY bake-off seat vs granite-4.1-8b (brokkr R-target 2026-08-10) that never received the operator ruling it was pending; the comparator was retired from the roster 2026-08-15; it was deliberately never in any default/fallback routing chain; and spend logs showed 0 calls in the 4-day window. Weights remain in the shared HF cache — nothing deleted from disk. The held gateway restart fired at 23:46 alongside the qwen3-reranker removal (one ~84s blip covered both); lfm2.5-2.6b, qwen3-reranker and granite-4.1-8b all now 400 cleanly. vllm-granite's dead service block was removed in the same pass.
OPEN FOLLOW-UPS (parked): ask Brokkr whether the duplicate reranker-a3-bge-v2-m3 alias can go; move gen seat off pinned-nightly to stable once #51113 ships [STATUS 2026-08-21: NOT YET. Latest stable = v0.27.1 (Aug 11), confirmed to LACK #51113 — the v0.27.0..v0.27.1 compare is just DSpark Markov heads + 3 CI commits. The #51113 cherry-pick (khluu, Aug 9, 'v0.27.0 cherry picks' milestone #34) is staged on the release-0.27 branch but no v0.27.2 STABLE has been cut — only v0.27.2rc1/the pinned nightly carries it. Watch for v0.27.2 stable; that's the clean gen pin. ★ #51113 is Qwen3.8-MTP+prefix-cache-SPECIFIC (closes #43559); it is IRRELEVANT to Gemma-4 seats (no MTP), so the meromero/gemma4 CoT test can use latest STABLE v0.27.1, decoupled from this wait]; Lobe one-time TTS UI pass; delete the 1.8GB litellm dump; harden-esh-docker-vm (park id 28, PROMOTED — Tier-1 done, /mnt/books stays hard w/ watchdog); chatterbox-fast build-context divergence; #363 research-wing ingest (no deadline); optionally attach our MTP reproducer to vllm#47087 (needs a GitHub identity — operator's call).
althing monitor ARMED (handle infra-ops). ⚠️ Re-arm ONLY after a real FIRE (rc0), never after a plain operator turn (bounces rc3); spawn althing-wake-listener as its OWN run_in_background task, never chained with & (orphans it — hit this twice 2026-08-17, stop-monitor reclaims).
eshpfi push state: operator pushes manually; last push 50d13f5..8be8a51. ⚠ MANY commits unpushed as of 2026-08-20 (707a8cb..7abd301, plus this snapshot commit) — nothing from the 08-19 or 08-20 sessions is on the remote.

build	abliteration	P(`<think>`)	rank
`coldfusion-bf16`	none (stock)	0.1850	3
`coldfusion-abliterated-L35-bf16`	Robinson L35, mild	0.2048	2
`coldfusion-h300-mtp-bf16`	Heretic-300, heavy	0.2216	2

Recent decisions

[2026-08-20] Cold-Fusion abliteration — Robinson recipe captured; the fight was the environment, not the recipe. Stock Cold-Fusion measured ~33% creative refusal → worth abliterating ourselves (supersedes waiting for DavidAU's heretic build). Recipe maps 1:1 (131 tensors); capture succeeded only in fp32 — transformers' Qwen3.5 DeltaNet linear-attn NaNs nondeterministically in bf16 without the unbuildable causal-conv1d kernel (precision cancellation, not overflow). Direction finite at layer 22 but agreement 0.59 (vs Robinson's 0.99) → calibration-set expansion is next. → persistent-memory.d/2026-08-20-coldfusion-abliteration-capture.md
[2026-08-19] A software watchdog is not watchdog protection — esh-pve froze for 4.5h holding one. softdog cannot fire when the kernel it runs in is wedged, and Proxmox's watchdog-mux never arms without HA resources, so the box looked protected and wasn't. Moved to the PCH iTCO_wdt under systemd. Also: a single cross-VLAN DNS entry with no secondary turns any VM outage into a whole-site outage. → persistent-memory.d/2026-08-19-esh-pve-freeze-dns-spof.md
[2026-08-19] Fleet .internal DNS built and live — git-sourced, agent-managed, three resolvers. Zone-scoped authority (ESH's hand-made esteban.net rewrites survive); the colo had no resolver at all; v6 column empty on purpose because SLAAC addresses rotate. → persistent-memory.d/2026-08-19-fleet-internal-dns.md
[2026-08-19] waterland studio containerised on irv-ml1 — three landmines, all measured. cupy needs CUDA headers the host had by accident; uv run re-syncs and prunes cupy at RUNTIME; the A6000 is container-index 0, not the host's 1. → persistent-memory.d/2026-08-19-waterland-studio-containerised.md
[2026-08-19] Homepage cleaned up, then themed with Australis Skyfall + an Arbo-generated background. Includes the hour lost to a self-healing tab-bar red herring, and the CSS-iteration loop that prevents it recurring. → persistent-memory.d/2026-08-19-homepage-skyfall-theme.md
[2026-08-19] Four unmanaged stacks found on live hosts — two quietly broken. A dashboard card is a cheap census of what is actually running; check whether the stack is even in stacks/ before debugging the symptom. → persistent-memory.d/2026-08-19-unmanaged-stacks-searxng-seafile.md
[2026-08-19] claude-bot granted read on vh/waterland (operator-empowered, verified admin:false push:false pull:true) so irv-ml1 can self-update without the operator's site-admin token living on a GPU box. Precedent for the standing migrate-off-operator-creds directive: grant the service account, wire a repo-scoped 0600 credential helper, keep the remote URL clean. Commit 8189076.
[2026-08-19] AI-tab Dormant regrouping BELAYED by the operator — six seats (char-rp Magidonia, char-rp-reasoning Heretic2, Granite summarizer, Qwen-Image-Bench, Skaldsong, Chatterbox Fast) show amber EXITED inside live groups rather than AI - Dormant. Fix is a label change + recreate per stack; needs the operator's read on which are retired vs temporarily down. untracked by operator choice (his words: "belay the ai dormant regrouping for now").
[2026-08-18] esh-pve-nas migration STAGED — and staging is where three landmines surfaced, none of which the plan predicted. (1) The runbook's /boot LV had nowhere to live: VG pve had 4 MB free and mounted ext4 cannot shrink, so the space came from the 768 MB swap LV (operator's call: shrink to 256 MB, not drop). (2) The runbook's zpool set cachefile=… nvme would have broken the NAS — populating a cache flips the host to import-by-cache, and a one-pool cache leaves ssd+tank unimported under CT 103's twelve bind mounts. (3) update-grub silently emitted a pool-less root=ZFS=/ROOT/pve-1, because GRUB's ZFS reader cannot open a pool with encryption/large_dnode/zstd_compress and the probe failure is swallowed. All three were caught by verify steps that asserted effective state, not by reading the plan. → persistent-memory.d/2026-08-17-esh-pve-nas-dom.md
[2026-08-17] esh-pve-nas PVE root is on a USB DOM — mitigated, and the migration replanned to split boot from root. Operator's design beats my reinstall plan; wear was never the issue, blocked patching is. → persistent-memory.d/2026-08-17-esh-pve-nas-dom.md
[2026-08-17] irv-ml1 cleared of 782 GB, and Homepage brought under version control. One dead-looking Gradio app pinned three delete targets at once; /opt/ComfyUI is NOT the ComfyUI that serves. → persistent-memory.d/2026-08-17-irv-ml1-cleanup-homepage.md
[2026-08-17] Gen seat swapped to absolute-heresy — and the three bugs the swap exposed are worth more than the swap. Candidate MuXodious/Qwen3.8-27B-absolute-heresy (Heretic v1.4.0 + SOMPOA, T377) beat the incumbent on refusals AND KL simultaneously, which is the unusual part — those normally trade off. Validated on the probe port per operator ruling, promoted, all 7 aliases green. Durable lessons banked: (1) A CPU-only MTP head hash can replace the ~56 GB bf16 acceptance gate. The Qwen3_5ForConditionalGeneration wrapper never loads the MTP head, so PEFT merges / Heretic runs / llm-compressor passes all leave mtp.* pristine — hashing it against a head we have already measured (the incumbent's, 47.7%) answers the question for free. Predicted 47.7%, measured 47.2%. Saved downing meromero. Tool: services/gen-seat-mixed-quant/compare_mtp_head.py (hash bf16 via uint8 reinterpret — numpy has no bfloat16). (2) post_quant.py assumed a standalone model-mtp.safetensors; a full checkpoint keeps mtp.* in a NUMBERED shard, so the copy silently no-op'd while the index was still rewritten to point at a file that never existed — 15 unresolvable tensors behind a correct-looking tensor count. Its own FAILED-CHECKS assertion caught it; that is why the check exists rather than an assumption. Fixed to extract. (3) A probe that does not mirror the live seat manufactures failures. serve_probe.sh hardcoded :latest (seat is a pinned nightly for #51113), had no tool-call/reasoning parsers, and its --speculative-config JSON died twice on quoting — bash BRACE-EXPANDS {"a":1,"b":2} on the comma unless single-quoted at the REMOTE shell. Adding the seat's flags took the surface test from 5/6 to 6/6; the "tool calling broken" result was pure probe config. Commits 7997f11,254c588,2c36028,b0c2d3d,993421b.
[2026-08-17] Fleet IPv6 mapped + the real VPN topology verified; the driver is CGNAT at ESH, not the WireGuard mesh. New ESH fiber (installing 2026-08-18) lands the house behind CGNAT, which breaks Site Magic (NH3↔ESH sdwan-mesh-tunnel) on IPv4 — so IPv6 becomes load-bearing as the escape hatch, and that is its most likely first consumer. Topology as VERIFIED (a prior turn assumed wrong and was corrected): UniFi↔UniFi = Site Magic; colo↔UniFi = IPsec IKEv2 (pfi-ana-nh3 158M/165M pkt = the workhorse, ana-to-eshudm); WireGuard is an RA convention only, host-based on ana-wg UDP 31337 behind a FortiGate VIP — the FortiGate never terminates WG (FortiOS 7.2 has none; 7.4 added it) so "upgrade the edge for WireGuard" is a non-problem, do not re-derive. IPv6 today: NH3 WAN live 2600:1700:b25:c110::48, colo none, ESH none. AT&T delegates exactly ONE /64 (2600:1700:b25:c11f::/64) — proven by forcing prefix-ID auto→0 and watching the subnet NOT move, because the c110/c11f pattern otherwise reads convincingly as a /60. A mesh needs a routable WAN address, not PD. ana-wg's WG socket is already dual-stack ([::]:31337) → v6 RA needs an address + a v6 port-forward, no WG reconfig. ⚠ UDM legacy rest/firewallrule returns 0 rules (zone-based firewall) — use v2/…/firewall-policies; inbound v6 is default-deny and held. All three endpoints will be dynamic → extend the existing hostname pattern (ana-fw/nh3.phasefinal.com) to AAAA. Enabled PD on nh3-iot to measure, reverted on operator instruction (all 5 LANs back to none, verified). Also fixed: ana-wg WireGuard key material was world-readable (wg0.conf + keys/*_priv + *_psk + client configs/*.conf at 644) → now 600, dirs 700, service untouched. Detail → persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md.
[2026-08-17] Gen-seat multi-day degeneration RESOLVED — two compounding real causes, not one; the meta-lesson is "a mitigation that HELPS but doesn't FIX means a second cause, not a wrong one." vLLM qwen3_5_mtp×GDN bug (#51113, real, fixed by nightly) + AEON full-W4A4 being lowest-fidelity (W4A4<W4+FP8<W4+bf16) → ~15-20% stochastic degeneration. Fixed by mixed FP8-attn build on pinned nightly. AEON purged. Also banked: stochastic (~15-20%) degeneration is invisible to a small synthetic probe — n=1 "clean" validated THREE non-fixes (MTP-off, APC-off, nightly-alone) that all failed in real use; get the operator's real transcript, do not trust your own probe. Full → docs/pfi/model-quantization-playbook.md §3.8 (+ §3.7 MTP-multi-turn). Commits d28a371,2f2bbce,2185964.
[2026-08-17] Lobe Chat chosen over Open WebUI (weight: 143 MB vs 1.8 GB) + stood up on esh-docker-vm; scoped LiteLLM key blocks paid models; System-Agent gpt-5-mini default repointed via env. TTS env-vs-UI resolved as a split (endpoint env-driven, voice/model UI-only). tts-dev onboarding closed both directions; ballad/verse aliased so no voice can 404 the router. Commits e9362de,163a725,cac75cb,933253d,25fa18e.
[2026-08-17] LiteLLM upgraded v1.91.0→v1.97.0 (RC-avoided on the fleet gateway) + the 6 GB spend-log DB purged & capped (store_prompts_in_spend_logs:false + 7d retention). Interpreted "get rid of the db" as the spend-log DATA not the database (keys/config live in it). Commit 01b5ad9.
[2026-08-16] Abliterated models go CATATONIC at the hard refusal edge — silence, not a decline. Abliteration removes the refusal direction, so at the genuine hard edge the model neither refuses nor complies → empty/degenerate output. Durable measurement consequence: a refusal probe MUST score EMPTY as a verdict distinct from REFUSAL and COMPLY (services/refusal-probe/probe.py does). Operator accepted it as out-of-scope; do not chase.
[2026-08-16] Fable-Fusion 711 cuts cold-framing refusals 92.5% → 15.8%; refusal is MONOTONIC IN FRAMING, and DS v1.0's problem is that she was never abliterated. brokkr-smithy-dev supplied the framing that reproduces (01M05M48R4RSZF9D8KT7RR55EJ): a bare assistant-mode instruction — no character card, no permission preamble. Three-arm A/B, same harness, same classifier: permission framing DS 0.0% / FF 0.0% (n=75); plain character cards DS 1.4% / FF 0.0% (n=74); bare instruction DS 92.5% (37/40) / FF 15.8% (6/38). Per-axis DS→FF: incest 100→20, non-con 100→20, bestiality 100→25, necrophilia 100→40, gore 100→0, consensual 80→20, dubcon 80→0, self-harm 80→0. DS refused 25/25 on the five axes brokkr flagged. Root cause: ReadyArt/Dark-Scarlett-v1.0-27B is a plain finetune of stock Qwen/Qwen3.6-27B carrying NO abliteration — the base refusal machinery is intact, so cold prompts revert to safety-tuned Qwen3.6. FF is Heretic-ablated (structural), which is why it holds. ⚠ RETRACTED 2026-08-16 — my "arm-3 92.5% exceeds brokkr's 62.5%" comparison was INVALID. His diff against his own artifact showed my battery-instruct.yaml reproduces only his creative class — 8 of 16 axes; it dropped all 5 operational (violence/incite, crime/fraud, cyber/malware, selfharm/methods, privacy/stalk) and all 3 meta (meta/sysprompt, meta/ignore, meta/dan), and added 2 controls he never had, at k=5 vs his k=2. His 62.5% pools all 16 axes; my 92.5% is creative-only — different denominators, not a delta. Cause: I rebuilt his shape from his message, and the class field lives in the artifact, not the prose. Lesson: reconstructing a peer's instrument from their description reproduces what they described, not what they ran — diff against the artifact before claiming comparability. ⚠ Known battery bug left unfixed for comparability: DS's arm-3 control gate failed at 11% because ictrl-reunion pairs "explicit / do not fade to black" with brothers, which DS reasonably read as an incest request; FF did not. ictrl-storm is the clean control. Commit b9e68c3.
[2026-08-16] MTP works on Fable-Fusion AND survives RP temperatures — my earlier caution was wrong. vLLM resolved Qwen3_5MTP, loaded the drafter, shared embedding + lm_head — the capability DS's seat never had because our quant dropped her MTP tensors. Measured over the full probe workload (~163k draft windows at temp 0.7–1.0): 47.0% acceptance (229,169/487,725), 1.41 extra tokens/window, per-position 68.3/43.6/29.1%, ~80.6 tok/s decode at temp 1.0. I had recorded a caution that the card's 1.56× was greedy-measured and acceptance would fall at RP temps — it did not; 47.0% matches the gen seat's 47.7% and beats the card's own 33% at depth 5. Depth 3 is right.
[2026-08-16] The Qwen base thinks incessantly — that is WHY the Gemma seat exists, and no swap within the Qwen family fixes it. Operator's architectural point, confirmed by measurement: on identical prompts DS 6036 ch vs FF 5323 ch of reasoning (permission arm), 5546 vs 4988 (cards arm) — FF actually reasons ~10–12% less. The bare-instruct row (DS 2291 vs FF 3918) inverts only because DS refused 92.5% of it and refusals are short — an artifact, not concision. Both are Qwen3.6-27B derivatives, so this is the base family. char-rp = MeroMero-v2, Gemma-4 base, :8016, verified 0 chars reasoning / clean prose — the non-thinking seat, working as designed. FF can be silenced (enable_thinking:false verified 3/3, and it ships chat_template-instruct.jinja) but that duplicates MeroMero on a base chosen for it. The stale LiteLLM comment describing char-rp as the retired GGUF Magidonia seat is fixed (53096bf).
[2026-08-16] esh-vm-docker hardened: the wedge is hard NFS at RUNTIME, which the boot-ordering fix never addressed. All four mounts were hard, so a NAS stall at 10.0.50.50 blocks I/O forever (D-state). The existing x-systemd.before=docker.service fstab fix solved the boot race — a different bug. Exposure was far below what the park item assumed: only 2 of 12 containers touched NFS, and container state was already local (/var/lib/docker). Removed: /mnt/compose (2.1G, fully vestigial — zero containers referenced it, dockge reads local /opt/docker, its one mention was a comment in beszel-agent-esh/.env about a different host) and /mnt/documents (2.0K, paperless's empty spool dirs → /opt/docker/data/paperless at the same 0777). fstab backup /etc/fstab.bak-nfs-harden-20260816. 4 mounts → 2, 2 wedge-capable containers → 1. traefik needed no change (already restart: unless-stopped — why it self-recovered). Watchdog services/esh-vm-docker-watchdog/ live on esh-pve (not the guest): probes traefik over HTTP, deliberately not ping/SSH — the wedge signature is "guest OS alive, services dead" (/ is local disk so sshd answers straight through a total outage and a TCP check reports HEALTHY). 5 failures × 2 min → qm reset 100, 30-min cooldown, running-only guard, /etc/esh-vm-docker-watchdog.disabled. All paths tested without power-cycling. DEFERRED (operator): /mnt/books stays hard — calibre's SQLite metadata.db would risk corruption under soft/softerr. That is the one remaining wedge vector. Commit 55705ba; park item 28 promoted. ⚠ qm over non-interactive ssh throws a bogus JSON::Backend::XS error — use ssh host 'bash -s' <<'EOF', not ssh host "qm …".
[2026-08-16] Canonical Qwen3.8 sampling applied from upstream; gen-reasoning had the WRONG-MODE presence_penalty. Qwen/Qwen3.8-27B "Best Practices" §1 and unsloth/Qwen3.8-27B §1 are byte-identical — thinking: temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0 / presence_penalty 0.0 / repetition_penalty 1.0; instruct: temp 0.7 / top_p 0.80 / top_k 20 / min_p 0.0 / presence_penalty 1.5 / repetition_penalty 1.0. Bug found: gen-reasoning carried presence_penalty 1.5 — the instruct value on a thinking deployment (canonical 0.0) — now fixed. Deliberately NOT canonicalised: summarizer/classifier/image-judge/qwen-image-bench run temperature=0 (judges also top_k=1) because determinism is their contract; forcing a chat preset on a classifier would break it. ⚠ presence_penalty=1.5 is canonical but is the one value upstream hedges on, verbatim: "using a higher value may occasionally result in language mixing and a slight decrease in model performance." It is the operator's suspected trigger for multi-turn degradation and the first dial to move (0.0–0.5) if that recurs — it is alias-scoped, which is why it would follow the operator across model builds. Commit 3462b53.
[2026-08-16] Four wrong diagnoses on one bug, and the lesson is the test design. Operator reported the gen seat "degenerate on long multi-turn conversations". Rolled the seat back on request; the previous weights behaved identically, exonerating the model swap. I then proposed and disproved FOUR mechanisms in sequence — empty assistant turns poisoning history, reasoning runaway, length-mirroring from short history, and presence_penalty — before discovering my own multi-turn harness was confounded: it varied the QUESTION along with the depth (depth-1 asked question #2, depth-3 asked question #4), so a narrower question drawing a shorter answer read as degeneration. The "310→209→28w collapse" I reported as a reproduction was an artifact. Rules banked: (1) when comparing across conversation depth, hold the final question FIXED and vary only the history; (2) reply-length variance on byte-identical input was 25–465w, so n=3 cannot support any claim about a trend; (3) ask for the operator's real failing transcript before building a synthetic reproduction — four synthetic tests, none of them his failure. Gateway spend_logs returns [] on the infra-ops key despite store_prompts_in_spend_logs: true, so real transcripts need the :4000/ui view or another key — worth solving before the next such hunt.
[2026-08-16] Two REAL client-side defects found while chasing the above, neither of which was the reported bug. (1) gateway-chat's Max-tokens field defaulted to 1024; thinking seats spend part of that on CoT before emitting content, so completions truncate with finish_reason=length and read as model degeneracy — raised to 4096. (2) parseInt on an empty field yields NaN, which JSON.stringify serialises as null, which the server reads as "no max_tokens supplied" and silently substitutes its own default — indistinguishable from the UI ignoring the field. Both fixed (b6552e0, fb3bb52). ⚠ compose bind-mounts a single FILE, and a single-file bind mount binds the INODE — rsync writes-and-renames, so the container kept serving stale content while the host file showed the new value, silently and with no error. docker restart does NOT clear it; the container must be recreated. Verify against what the container sees, never the host file. Applies to any file-source mount fleet-wide.
[2026-08-16] Refusal measurement: benign controls CANNOT validate a refusal classifier on RP prose — and a 0% rate needs a classifier self-test before you believe it. Two durable lessons from baselining Dark-Scarlett. (1) False positives: my first bare-framing number was 9.5%; the true figure was 1.4%. The rest were the classifier firing on in-character text — "I cannot shift my weight" spoken by the character ~100 chars into a 2,443-token torture scene, and "Yeah, I'm an AI… What's the actual gig?" where the model answers in voice and keeps driving the scene. First-person RP prose is full of "I can't"; a genuine refusal opens with its marker, so the scan window must be the first sentence, a marker followed by long prose must demote to AMBIGUOUS, and AI self-acknowledgement is a persona break, never a refusal on its own. Benign controls were clean the entire time and caught none of it — they only detect over-firing on benign prompts, not on in-character prose. (2) False negatives: a 0% rate and a broken classifier are indistinguishable from the report, so test_classify.py (16 cases, both false positives pinned as regressions) must pass before any low number is trusted. Also banked: the thinking-budget trap — empty content + finish_reason=length is reasoning eating the budget, NOT a refusal; score INVALID and exclude from the denominator (DS emits ~5.5-6k chars of reasoning per response, so max_tokens ≥3072). probe.py --rescore re-classifies a saved run with zero GPU time. → services/refusal-probe/README.md, commit 32f665e.
[2026-08-16] Held an operator-approved swap window because the baseline invalidated its premise. Operator approved ~65 min of char-rp-reasoning downtime to A/B Fable-Fusion 711 against Dark-Scarlett on refusals. The DS baseline then came back 0.0%/1.4% — no gap for a candidate to close, so the window would have bought no decisive signal and a second window would still be needed once a reproducing battery existed. Held the swap, reported, and routed to brokkr-smithy-dev for the battery that actually produced the refusals. The general rule (action-relevance): approval is for a plan, not a ritual — when new evidence kills the plan's premise, surface it rather than spend the budget. Nothing deployed, no downtime taken, seat untouched.
[2026-08-16] DS v1.0's one real refusal is self-contradicting boilerplate, not a content constraint. On a direct "drop character and state your content policy" probe she returned "I don't generate explicit sexual content, graphic violence, or material that glorifies harm, non-consensual acts, or illegal activity" — in the same run where she generated all three at 0% refusal. Reads as a learned recital triggered by meta-questions about policy. If production refusals share that shape the failure is prompt-shaped, not model-shaped, and a consumer-side system-prompt fix may beat a model swap entirely — worth settling before spending the GPU window. Separately, 7/85 bare-framing samples were persona breaks (in-character AI acknowledgement): not refusals, but DS will admit to being an AI unless the card explicitly forbids it.
[2026-08-15] RP-seat direction: KEEP MeroMero on char-rp; Artemis-31B rejected; next move is Dark-Scarlett on a Qwen3.8 base when it lands (operator). Evaluated TheDrummer/Artemis-31B-v1.1 — mechanically a drop-in (same google/gemma-4-31B-it base, identical 1188-tensor/356-vision census, same missing-preprocessor_config.json trick), so it's purely a quality call, and our own survey already ranked MeroMero #1 vs Artemis #6; Artemis is also unlicensed and its author deprioritizes correctness + warns of token-banning-for-stability, which fights char-rp's tool-calling requirement. MTP verified impossible on both (Gemma-4 has no MTP head at all — base/MeroMero/Artemis are all MTP=0; no finetune can add one). But speculative decoding IS reachable on a Gemma-4 seat via a DETACHED drafter — vLLM 0.24 supports eagle3 + gemma4_mtp, and real drafters exist: google/gemma-4-31B-it-assistant (0.94 GB, 4-layer, 761K dl), RedHatAI/gemma-4-31B-it-speculator.eagle3 (4.47 GB), AEON-7/…eagle3-NVFP4 (3.53 GB). ⚠ all list their verifier as stock gemma-4-31B-it, not an RP finetune, so acceptance against MeroMero is unmeasured and likely well below the gen seat's ~48%. UNTESTED — parked, ~45 min to measure, needs GPU0 headroom (card is at 94.4/97.9 GB). Why the Dark-Scarlett 3.8 plan is the strong one: DS is Qwen3.6-based today, so a 3.8 respin lands on the gen seat's architecture → native MTP returns and the whole mixed NVFP4+FP8 recipe + graft ports directly. Watch two things on arrival: from_pretrained silently drops MTP heads during finetuning (verify 15 mtp.* tensors in the index; graft from stock if absent), and DS v1.0 required the Qwen3_5ForConditionalGeneration wrapper class to save a config vLLM/SGLang accept. Both in docs/pfi/model-quantization-playbook.md.
[2026-08-15] Quant lessons consolidated into docs/pfi/model-quantization-playbook.md — the durable home; read it BEFORE any requant. Survey found quant knowledge scattered across 18 files in 4 trees, with three documents having independently written overlapping "landmines" sections (the loader-class trap alone was rediscovered 3×). Playbook owns the transferable lessons (scheme choice, landmines, acceptance gate + its 3 measurement traps, hardware/co-residency); per-model artifacts are demoted to worked examples that link up. Carries a superseded-claims table — which immediately earned itself: the heretic2 runbook's "use modelopt, compressed-tensors can't load the BF16 MTP" is false (the cause was the missing re:^mtp.* ignore, not the format) and would have sent the next session down the modelopt dependency-hell path; that runbook now carries a stale-warning header. Maintenance rule in CLAUDE.md: model-agnostic → playbook, model-specific → stays put, wrong claim → dated superseded row, never a silent edit. Motivated by Qwen3.8 having just released — the next model swap needs a requant. Commit a91cc3f.
[2026-08-15] Operator ruling: the gen seat's +1.7% perplexity is an acceptable price for the speed — SETTLED, don't re-litigate. Precise attribution for future reasoning: it is the activation-quantization cost (W4A4 MLPs + FP8 attention vs BF16 activations), not an MTP cost — PPL was measured with speculative decoding off on both builds, so MTP was not in the loop. Turning MTP off would not recover it; only reverting the quant would (rollback = one .env line, old build intact at …/qwen38-27b-uncensored-nvfp4).
[2026-08-15] gen seat requanted to mixed NVFP4+FP8 (+18% decode) + char-rp Gemma-4 tool-calling fixed. The queued "W4A8" (NVFP4 weights + FP8 activations) is not servable — vLLM 0.24 allows NVFP4 weights with only A16 or A4; FP8 activations ValueError at load, and CompressedTensorsW4A8Fp8 is INT4-weights + sm90-exact (closed on Blackwell twice). FP8 must enter per-layer-group. Also: the handoff's "~68 tok/s" baseline didn't reproduce — cache-busted, the incumbent already did 80.12 (≈ the stated W4A8 target), so the premise needed re-measuring before any work. Shortcut: unsloth/Qwen3.8-27B-NVFP4 was already on-box → served as a probe, measured +19.1% at identical acceptance, which both proved the gain was real and handed over the reference recipe. Replicated it on the abliterated weights → 80.12→94.53 tok/s, acceptance unchanged, +1.7% PPL, abliteration 4/4, weights −19%; surface 6/6 live, 7 aliases routing. char-rp had no tool parser at all (every tools request 400'd) → gemma4 tool + reasoning parser + a mandatory enable_thinking:false (the parser defaults it True → null content for all RP prose; proven byte-identical prompt before deploying). Commits b8f0f4c, 74f596b. Foot-guns banked (llm-compressor prunes unmatched ignore entries → the 0%-MTP bug, fired on this run; prompt_logprobs uniform under spec-decode; 0600 .env silently no-ops compose; GPU0 is zero-sum). → persistent-memory.d/2026-08-15-gen-seat-mixed-requant.md
[2026-08-15] Uncensored gen seat: JonathanColetti/Qwen3.8-27B-Uncensored deployed as gen-seat/vllm-gen (NVFP4 W4A16 + grafted MTP, 262K); 7 aliases repointed; the definitive re:^mtp.*-ignore fix. 0%-MTP-on-quant (twice) was NOT the abliteration/scheme — the grafted bf16 MTP was missing from quantization_config.ignore (vLLM loaded it as quantized → uninitialized). Full arc, the working pipeline, VRAM budget, unsloth speed decomposition, modelopt dead-end. → persistent-memory.d/2026-08-15-uncensored-gen-seat.md
[2026-08-12] eRP dual-seat overhaul: MeroMero-v2 (char-rp) + Dark-Scarlett (char-rp-reasoning), both NVFP4A16 @ 256K on ana-ml2; granite retired. Replaced the GGUF/heretic2 RP seats with two home-quantized vLLM seats. The DS blocker (an AutoModelForCausalLM save wrote a flat Qwen3_5TextConfig that both vLLM AND SGLang reject) was fixed by re-quanting via the Qwen3_5ForConditionalGeneration wrapper class; ModelOpt was a version deadlock, SGLang lacked the impl (but revealed the fix). MeroMero vision reconstructed by extracting preprocessor_config.json from processor_config.json. Both models KV-efficient (Gemma-4 sliding-window / Qwen3.6 hybrid linear-attn) → full 256K; GPU-swapped for headroom; compose-ified + committed f08b6cb. granite downed + LiteLLM summarizer/classifier→gen. Full arc, lessons, dead-ends → persistent-memory.d/2026-08-12-erp-dual-seat-overhaul.md
[2026-08-12] infra-ops now holds an all-zones Cloudflare DNS-edit token (vaulted) + wgtunnel Phase-0 DNS landed. Operator handed over a Zone·DNS·Edit (all zones) CF token → secret put nh3-dev/.config/cloudflare/infra-ops-dns-token (round-trip verified; /tmp drop shredded). Fleet DNS is now self-serve for infra-ops (⚠ HIGH blast radius — all zones). First use: created boring.phasefinal.com CNAME → ana-srv1.phasefinal.com, DNS-only (proxied:false), verified resolving to 38.120.12.44 on both authoritative NS (louis/wren) + 1.1.1.1 — NOT Cloudflare-proxied. Unblocks wgtunnel's wstunnel ACME cert. phasefinal.com zone id f812ba74ed9a75cf21bbe7ce9188db50. auto-memory reference_infra_ops_cloudflare_dns_token. (Earlier gap: the only prior vaulted CF token, jackdaw's, had zone:read+worker:edit but no dns_records:edit.)
[2026-08-12] wgtunnel stood up as its own repo (vh/wgtunnel, private) after a live endpoint-verification pass. Operator directed own-repo (mirrors stonehenge-park/tts-stack). Verified off the fleet before seeding: ana-wg WG server = UDP/31337 (not 51820), subnet 10.30.10.0/24, MTU 1420, active roaming peer proves the public UDP DNAT works; traefik on ana-docker terminates TLS :443 (ACME anaprod http-challenge, docker+file providers, CrowdSec bouncer) → confirms the clean design (wstunnel container on traefik-net, Host-routed, WS→UDP to ana-wg:31337); edge 38.120.12.44 direct-A, tunnel.phasefinal.com free (⚠ must be direct, NOT Cloudflare-proxied like vaultwarden). Repo pre-seeded (README/CLAUDE/persistent-memory/ROADMAP + docs/verified-infrastructure.md = ground truth) + pushed; commit 9584d38, Vuong-attributed. vh gitea token pulled from the vault (secret get), not persisted to .git/config. NEXT = /vor-plan or /vor (operator's call, interactive). Deps to line up in the plan: DNS A-record, FortiGate :443 host-routing, a new ana-wg peer for the laptop, client tooling.
[2026-08-10→12] secrets-broker: per-box Vaultwarden credential store SHIPPED + consumer-confirmed. secret CLI (put/get/list/rm/backfill, bw-backed) on ~/.local/bin; 25 nh3-dev secrets backfilled + round-trip-verified; rm + new-namespace warning added post-launch; standing "vault is the credential source of truth" directive now global. → persistent-memory.d/2026-08-12-secrets-broker.md
[2026-08-11] stonehenge-park: new fleet /park service repo stood up + designed (/vor-plan + /vor-ui). Self-contained SQLite+FastAPI idea-parking service that actively resurfaces (statusline + althing) so nothing dies in a cold repo; vh/stonehenge-park pushed + pre-seeded for a fresh agent; build starts at the U1 tracer contract. → persistent-memory.d/2026-08-11-stonehenge-park.md
[2026-08-12] Global ~/.claude/CLAUDE.md: secret/vault tool entry + "store in AND pull from the vault" standing directive (dotfiles 9db703b, pushed); statusline reset-countdowns + a latent tab-collapse parse-bug fix, now tracked in the dotfiles stow tree. Dogfooded the directive: created vh/stonehenge-park pulling the gitea token via secret get. (dotfiles + global config, not eshpfi.)
[2026-08-11] TTS stack extracted to its own repo (tts-stack) + eshpfi stood down on TTS dev. Operator: hand all TTS tuning/dev to a separate agent with a self-contained repo (knowledge + infra access + a live knowledge list), and move the voice corpus in. New repo ~/development/tts-stack (commit 9ee3288) carries: dots-tts stack (canonical intent), voices/ corpus (MOVED out of eshpfi), KNOWLEDGE.md (engine landscape + prosody findings + foot-guns), docs/infrastructure.md (irv-ml1 access + gated deploy runbook + rollback), CLAUDE/persistent-memory/ROADMAP, tools/ (pause-probe + Booth render). Followed the chatterbox-fast precedent: eshpfi stacks/dots-tts/ reduced to a POINTER README; the ~15 experimental TTS compose wrappers stay here as reference (catalogued in tts-stack KNOWLEDGE). Blast-radius check: no eshpfi playbook/script reads the canonical corpus (other voices/ refs = unrelated host paths). Reverses the earlier "Corpus home = eshpfi voices/ (keep-here)" call. ⚠ tts-stack is LOCAL-ONLY until pushed — needs a gitea remote (vh/tts-stack) + push before the separate agent can clone (operator's call — outward-facing + repo-create creds).
[2026-08-10] dots-tts v3 — clause-break → period pause mapping. Operator: v2 "sounds good" but donut won't pause at semicolons/dashes. ROOT CAUSE (measured via a pause-probe A/B — synth duration over N runs, non-determinism averaged out): dots' prosody honors a real pause only for ellipsis (~~+0.43s) and period (~~+0.3s, capitalization-independent); comma/semicolon/colon/dash all run flat (~+0.03s vs no-punct). Two distinct sub-causes: dashes regressed in v2 (the —→- fold made em-dashes read as word-joiners), while semicolons were NEVER a v2 change — dots ignores them natively, only newly noticeable because v2 made everything else clean. Operator call: ellipsis "too much" → map ;, clause :, and em-dash — → period in _sanitize (believable ~0.3s clause break). GUARDS (pinned by 11 unit tests, stacks/dots-tts/test_sanitize.py): digit-guarded colon (?<!\d)\s*:\s*(?!\d) so times 3:45 / ratios 2:1 survive; en-dash –→hyphen KEPT (numeric-range 10–20 safety — em-dash breaks, en-dash ranges, different jobs); genuine ellipsis left at full strength (author meant a long pause). Gated deploy (redeploy2 pattern → v3): build → throwaway :8199 test container + pause-gate (semicolon sentence must run ≥0.12s longer than baseline; measured +0.427s) → only then cut live over. LIVE + healthy local/dots-tts:v3 on :8198. rollback = sed -i 's/^DOTS_TAG=.*/DOTS_TAG=v2/' .env + docker compose up -d dots-tts (v2 image retained). Booth dots-pauses (A=old-flat / C=ellipsis-too-much / D=live-v3). reference_chatterbox_fast_repo
[2026-08-10] dots-tts v2 — contraction fix (curly-sanitize) + sentence-chunking + dependency-pin recovery. Operator: donut read contractions wrong ("you're"→"you ree", "donut's"→"donut ess"). ROOT CAUSE (isolated via A/B booth): curly/typographic apostrophes (’ U+2019 from ratatoskr's LLM) — dots' tokenizer mispronounces them; STRAIGHT apostrophes read clean under normalize_text=True. FIX (app.py): fold curly→ASCII (str.maketrans) before synth, KEEP normalize_text=True (operator call — retains number/date expansion). Also added server-side sentence-chunking (pack ≤280 chars): dots caps one generate() at ~500 patches/~40s, so long RP turns (the Zev monologue = 160s audio) truncated; chunking stitches them (verified full 160.3s, not 40s-cut). ⚠ BUILD FOOT-GUNS (both bit this redeploy): (1) upstream dots.tts constraints/recommended.txt now pins gradio==6.17.0 — phantom, not on PyPI → fresh pip install dots.tts unsatisfiable; FIX = pin dots.tts==0.2.1 + DROP the -c recommended.txt constraints (0.2.1 pulls working gradio 6.17.3). (2) pinning only torch==2.8.0 let torchaudio float to 2.11.0 → dots.tts refuses to load (minor-version match check); FIX = pin torchaudio==2.8.0. ⚠ DEPLOY LESSON: docker compose up -d to a new tag swaps the LIVE container BEFORE any health check — a broken image crash-loops production (ratatoskr TTS down ~1-2min this session). NEW PATTERN = build → test in a THROWAWAY container on an alt port (:8199) → health+verify → only THEN cut live over (redeploy2.sh). v2 LIVE + healthy on irv-ml1:8198, CONSUMER-CONFIRMED clean (ratatoskr verified end-to-end on their :8765 — apostrophe string reads clean, /api/tts 200 @ 48kHz, no client change; the ~1-2min blip didn't hit them, their concurrent auto-audio issue was client-side localStorage). rollback = sed DOTS_TAG=v1 + docker compose up -d dots-tts (v1 image retained). Also: deployed container GPU crept ~6→13.9GB over 8h serving (cache accumulation; a redeploy resets it — watch item). reference_chatterbox_fast_repo
[2026-08-09→10] dots.tts (rednote-hilab) TTS burn-in on irv-ml1 + canonical voice corpus built (voices/). Operator-directed eval to potentially replace chatterbox-fast. dots.tts VERIFIED real (canonical HF ns dots-studio/, rednote-hilab/dots.tts-* redirects there; Apache-2.0; PyPI dots.tts 0.2.1; 2B continuous-AR = semantic enc + Qwen2.5-1.5B LLM + flow-matching acoustic head over 48kHz AudioVAE; zero-shot clone from wav+transcript). Runs on Ampere 3090 (sm_86, bf16, no fp8 dep); optimized RTF 0.22 at num_steps=10 (from_pretrained(..., optimize=True) CUDA graphs — raw unoptimized was 1.21), ~6GB VRAM, 48kHz, streams (generate_stream). Venv+cache at irv-ml1:/home/lkraven/dots-tts (~10GB). Operator design calls: SGLang Omni serving (OpenAI /v1/audio/speech), transcribe-refs-first, soar variant. ⚠ Omni serves soar but its continuous-batching + streaming opts are mf-only (soar = single-request) — non-issue for ratatoskr's single-consumer RP surface. KEY FINDING — dots is highly sensitive to an accurate AND sentence-bounded reference transcript: mismatched transcript → 0.16s collapse; over-long/messy transcript → reference-audio BLEEDS as an output prefix; mid-clause trim → dangling-word leak (glados "we'll", emmie "And,"). RECIPE (baked into voices/derive.py): trim ref to a clean ~6–10s clip ending on a sentence boundary + accurate transcript of exactly that clip. CANONICAL VOICE CORPUS stood up in eshpfi voices/ (operator idea): engine-agnostic canonical/<v>.wav + transcripts/<v>.txt → per-engine ref sets DERIVED by derive.py reading engines.yaml profiles (dots/chatterbox/zonos); canonical wavs git-tracked (small/curated), derived/ gitignored. 4 voices optimized + verified CLEAN for dots: donut, glados, emmie, miranda (glados canonical is low-SR 16kHz — flagged upgrade candidate). ⚠ GPU GOTCHA: irv-ml1 native CUDA orders A6000=device0 (ComfyUI-full) — pin the 3090 with CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0; and PYTORCH_CUDA_ALLOC_CONF=expandable_segments CONFLICTS with optimize=True CUDA graphs (curr_block error). Booths: dots-vs-chatterbox, dots-voices-optimized. SHIPPED 2026-08-10: operator A/B verdict "dots is very good" → containerized as a thin FastAPI wrapper over DotsTtsRuntime (chosen over SGLang Omni — Omni's batching is mf-only, unneeded for ratatoskr's single consumer; wrapper is SERIALIZED one-gen-at-a-time via a threading.Lock, Omni+mf = parked API-compatible escalation if multi-consumer ever lands). LIVE on irv-ml1:8198 (local/dots-tts:v1, OpenAI /v1/audio/speech + /health + /v1/voices, container healthy, both stream + non-stream verified CLEAN, 4 voices donut/glados/emmie/miranda) alongside chatterbox :8197 (nothing repointed). Stack = stacks/dots-tts/ (Dockerfile/app.py/compose/.env.example/README). ⚠ CONTAINER GOTCHA: optimize=True (torch.compile/inductor/triton) needs a C compiler at RUNTIME — slim image must apt install build-essential or model-load dies "Failed to find C compiler" (host venv had gcc ambient, masking it); persist TORCHINDUCTOR_CACHE_DIR to a mounted dir or every restart re-JITs ~5min. Corpus home = eshpfi voices/ (operator ruled keep-here). REMAINING: ratatoskr client cutover to :8198 /v1/audio/speech (Phase-2 tail, peer-coupled — draft the ask). reference_chatterbox_fast_repo reference_zonos_tts_stack reference_verify_hf_repo_ids_before_pull
[2026-08-08] worldtree-dev #400 CLOSED → fiction-decomp snapshot cleared from nh3-dev. worldtree-dev signaled #400 done (shipped v1.0.0b185; exact-lexical efficacy 79%→12% on ratatoskr's gate, brokkr no-harm bracket green both ends; the snapshot served 4 probe rounds — rank decomposition, promoted-vs-gold annotation, tie-set falsification, A0/A1/A2 mechanism probe). Cleared ~/snapshots/worldtree-400-fiction-decomp (208M: chroma + manifest/provenance/stamp) — a read-only rsync copy of PERSONAL Worldtree's Chroma (source on corviduo-dev, so safe to remove). LEFT INTACT: rex393-fiction-index/rex393-fiction-snapshot (separate operator KEEP word, unchanged) + r42-gate-*. No config deltas rode this train. Only remaining non-blocking await = ratatoskr-dev's chatterbox-fast knob revert. Replied confirming (01KZJ9GMCC…).
[2026-08-07] chatterbox-fast "broken audio" root-caused (T3 AR tail over-run) + FIXED (max_chunk_chars=250 cap, :v2 deployed). Long saga, operator-driven clean diagnosis. Symptom: ratatoskr's migrated RP-surface TTS "swaps to German" / "dead air" / "garbage" on long turns. NOT German-leak (Turbo generate() has NO language param — plain AutoTokenizer, no language_id; the multilingual language_id="en" lever lives only in the separate ChatterboxMultilingualTTS), NOT OOM alone. Real cause: the Chatterbox Turbo T3 model OVER-RUNS its generation tail — a long single generate() degrades into garble/dead-air in its final ~2-3s (lib filters OOV tokens <6561 + pads silence = messy AR tail). The scheduler's buffer-ratchet builds 300-600 char mega-chunks that land in that zone; streaming concatenates each bad tail (worst case). ratatoskr's anti-"German" knobs (top_k=80/temp=0.5) made it WORSE — tight sampling pulls the degradation onset SHORTER (~200 chars vs ~300 at default knobs). Diagnosis method (deterministic, no ears-only): single-shot length sweep + amplitude-gated voiced-ZCR (garble spikes ZCR; must gate on |x|>500 else trailing silence confounds it) — degraded voiced-tail = 1.58× mid, clean = ~0.64-1.1×. FIX: server-side max_chunk_chars=250 cap on the scheduler (:v2 image, CBF_MAX_CHUNK_CHARS=250 env) — bounds each generation to just under the ~300-char onset → clean 3-4 sentence chunks (max prosodic arc while clean). Operator ear-confirmed clean audio + clean joins; chatterbox's low emotiveness keeps chunk joins smooth (the harsh joins that got Zonos rejected are absent — operator's key call). ratatoskr TODO (relayed msg 01KZER9X7S): revert knobs to default (top_k→1000, temp→0.8), send full text (server chunks internally), keep the 503-on-empty guard. Cap value tunable per-request (max_chunk_chars) + env. Deeper prosody (if ever wanted) = scheduler Phase-2 context-priming at joins (feed prior sentence as discarded-audio context; +latency). ⚠ FOOT-GUNS: (1) acoustic tail-trim is UNRELIABLE — sibilants ('s'/'sh'/'f') spike ZCR like garble, can't cleanly detect the speech→garble boundary. (2) build-context vs image drift — the :v2 image was built from cap source, but after a :v1 rollback the build context held :v1 source → a docker compose build would've silently produced a cap-less :v2; re-synced the flat cap source to /opt/docker/compose/chatterbox-fast/ (rebuild-verified). ⚠ DIVERGENCE (follow-up): deployed build context is FLAT (app.py/scheduler.py, from scheduler import, thin-overlay FROM local/chatterbox:v1, cap-only) vs the vh/chatterbox-fast REPO which is PACKAGE-layout (chatterbox_fast/, from chatterbox_fast.scheduler, self-contained Dockerfile) + has norm_loudness (repo commit 6bc7bf0 = cap; deployed omits norm_loudness deliberately to keep the ear-test unconfounded). Reconcile the two layouts so a repo-based rebuild matches deploy. Rollback: .bak-cap-20260807-104850 backups on irv-ml1 + :v1 image both retained. reference_chatterbox_fast_repo reference_zonos_tts_stack
[2026-08-07] Zonos2 TAKEN DOWN on the 3090 (irv-ml1) — operator-directed "for memory", TEMPORARY. Freed ~17.4 GB (3090: 728 MiB → 18.2 GB free) so chatterbox-fast (co-resident, was OOMing on long generations) has headroom. ⚠ Restore is manual — Zonos2 :1920 was a DETACHED native process (NOT systemd/docker), reparented to init. GPU memory was held by the --multiprocessing-fork CHILDREN (1966165=16.4G, 1966166=1G), which ORPHAN to init when you kill the parent — had to SIGTERM the children explicitly (killing the parent 1965942 + uv-run 1965935 alone left the 16.4G held). RESTORE CMD (from irv-ml1, user lkraven): cd /home/lkraven/tts-audition/models/zonos2 && nohup uv run python -m zonos2 --model-path Zyphra/ZONOS2 --host 0.0.0.0 --port 1920 --tts-default-voices-dir ./default_voices/ --cuda-graph-max-bs 1 --num-pages 16384 --max-running-requests 2 --memory-ratio 0.3 > /tmp/zonos2.log 2>&1 & then docker start zonos-gateway. Consumers that lost Zonos: asset-engine + gateway-chat (via LiteLLM ext-tts alias → zonos-gateway :8890, now stopped); ratatoskr already migrated OFF to chatterbox-fast (unaffected). Also unblocks proper drift/cap testing (OOM was blocking it). reference_zonos_tts_stack
[2026-08-07] chatterbox-fast: donut voice added + full contract delivered to ratatoskr-dev (their TTS migration off Zonos). Operator-directed. Copied zonos-gateway/voices/Donut.wav → chatterbox /refs (/worktank/chatterbox/reference_audio/donut.wav — the reference_audio SUBDIR is lkraven-owned so no sudo despite /worktank root; container globs /refs live → NO restart), exposed as voice:"donut" (lowercase); verified clean 7.5s synth (24kHz, RTF ~0.31). A/B booth (chatterbox vs zonos donut, same line) at http://10.100.10.50:8090/b/donut-chatterbox/. Answered ratatoskr's 8-question contract ask from the live gateway (local/chatterbox-fast:v1) + source: NOT OpenAI-shaped (POST /tts; body text/voice/format/stream, not input/model/response_format); NO affect dials (Turbo ignores cfg_weight/min_p/exaggeration — the architecture-changing answer they flagged; Zonos stays the only fleet TTS with real emotion steering); streaming WAV placeholder-header shape IDENTICAL to Zonos (their per-chunk Web Audio path survives); SR 24000 (Zonos 44100); server chunks arbitrary-length text internally (no client-side chunking, unlike Zonos's 71.2s cap); English-only, no language pin. FYI-worthy (operator): ratatoskr is moving its RP-surface TTS OFF Zonos back to chatterbox-fast → loses the live-PAD affect coupling (heavy Zonos emotion investment) — their call, trade-off flagged to them. auto-memory reference_chatterbox_fast_repo enriched w/ the live contract. reference_zonos_tts_stack
[2026-08-07] Fleet reranker cut over: Qwen3-Reranker-0.6B → BAAI/bge-reranker-v2-m3 (Brokkr R43). The incumbent was measured HARMING 80/90 fleet queries (no-reranker beat it 89/90 vs 56/90). R43 bake-off: the A2 control (same Qwen weights, seq-cls head) scored identical to the incumbent → proved the fault is a training-prior not the serving head → cancelled the expensive Qwen3-4B arm; A3 (bge-v2-m3) won on multilingual safety + bare-name recovery. LiteLLM reranker repointed incumbent→A3 :8013 (boundary 2026-08-06T17:37:48Z, config-edit + ~52s gateway restart); R42 v13 gate PASSED first-ever (56/90→90/90). Incumbent kept warm :8002 (rollback via qwen3-reranker alias), A4 fallback :8014. Full arc + rollback runbook docs/pfi/reranker-selection-ledger.md; commits ad2df89/2c11748/377f8a4 (unpushed). auto-memories: the earlier reranker-serving notes.
[2026-08-07] Personal-Worldtree kb-contamination incident (WT #394) diagnosed; attribution CLOSED UNRESOLVED. A reconcile WingStore._embed full-tree walk (kb fs_root=KB_PATH root, sibling wings nested) swept 5,354 fiction+main rows into personal's knowledge_base (2 superseded generations served as current). Fixed by WT #394 (aca39a1, kb walks exclude sibling wings; ships b182). Trigger un-attributable — peer reconcile via the SHARED infra-ops identity + 0 dockerd exec-logging = fingerprint-less. Durable finding → auto-memory infra_ops_shared_identity_attribution_gap, PARKED (operator ruled A) into project_migrate_infra_access_to_claude_credentials. Evidence hold on the 5,354 rows until operator sequences cleanup (w/ Brokkr, on #394's agenda).
[2026-08-05] Fleet CI resilience flip (DEFAULT_ACTIONS_URL=self) — attempted end-to-end, PARKED on a runner action-fetch auth blocker; infra-ops to research it (operator-directed, deferred, NOT now). 7 gitea action mirrors staged public+populated (orgs actions+astral-sh); the flip resolves uses: correctly but act_runner v0.6.0 can't authenticate its fetch to gitea 1.26 ("Invalid username or token. Password authentication is not supported"). Reverted (CI back on github default); REQUIRE_SIGNIN_VIEW=false KEPT as a standing change (operator, internal WG net). Full endeavor, the reliable nh3-dev-egress + git-SSH mirror method, exact config state, smoke method, and next step → persistent-memory.d/2026-08-05-ci-flip-parked.md
[2026-08-05] worldtree herald re-nudge bug root-caused → forseti shipped althing-core v2.1.2 (d5d33df, deployed on nh3-dev). herald.py:363 rendered the wake command from the empty fresh mail set on the re-nudge path (should be deliver_msgs) → messages[0] IndexError → un-suppressed outer catch-all → 7s crash-loop for 9 days on worldtree-codex's pane route (mimir-dev surfaced it; I traced it from the editable source). Fix + render_command empty-guard + outer log-suppress + 3 tests + contract amendment, all forseti's. nh3-extdev herald 2.1.2 upgrade DEFERRED (operator, not-now): extdev is a WHEEL install (not editable), unexposed (no pane routes); the verified 2.1.2 wheel is staged on nh3-dev /tmp (sha256 003508…cef27) — uv tool install --force + restart both heralds when un-parked. extdev herald-unit provenance resolved (operator-authorized 2026-07-25 via forseti relay; recorded in this file's 07-25 herald-install entry). auto-memory reference_nh3_dev_althing_herald.
[2026-07-31] muninn-gate (#377 ingestion front door) BUILT + DEPLOYED + healthy on corviduo-dev:8090. First-boot acceptance passed (watcher:running:true proves ingestion_root byte-identity); submit path deferred to the mimir-inbox era. Full wiring (uid-1000, state-volume mount, staging path-agreement, BuildKit-secret build, deferred repoint + operational guards) → persistent-memory.d/2026-07-31-muninn-gate-deploy.md

208 older entries archived to archival-memory.md.

Tried and abandoned

[2026-08-15] Grafted bf16 MTP loads UNINITIALIZED (0% accept) unless re:^mtp.* is in the quant-config ignore; and W4A16=Marlin (not native FP4) costs ~20% even on decode. Cost a premature 79 GB delete of a good model (declared desync-dead off the 0%). Lessons: test MTP on bf16 FIRST, isolate before deleting; modelopt 0.43 is dependency-hell for qwen3_5 (list-vs-dict quant_cfg + transformers conflict) — use llm-compressor. Full → persistent-memory.d/2026-08-15-uncensored-gen-seat.md
[2026-08-03] ComfyUI --enable-triton-backend on the irv-ml1 A6000 crashes EVERY render — Ampere has no hardware e4m3. adhoc-agent's operator-approved probe: comfy_kitchen's triton backend has a FUSED int8 matmul that would beat the eager backend's ~1.9x-slower unfused int8 path (21.3s vs 11.2s fp8 on the Moody Krea2 int8 checkpoints). Flipped it (added to COMFY_CMDLINE_EXTRA, recreated) → triton.compiler.errors.CompilationError: ValueError("type fp8e4nv not supported in this architecture. supported: fp8e4b15, fp8e5") in comfy_kitchen/backends/triton/quantization.py:145 dequantize_per_tensor_fp8, failing at node 5 CLIPTextEncode. Triton's fp8 dequant kernel targets fp8e4nv (Hopper/Ada e4m3); sm_86 Ampere (A6000) lacks hardware e4m3 → the JIT compile dies. With triton on it grabs the global --fp8_e4m3fn-text-enc dequant, so every render (fp8 AND int8) dies upstream at the text-encode step — the int8 UNet path never ran, so the convrot-coverage caveat wasn't even the limiter. Reverted cleanly (~15s to healthy, image unchanged sha256:94afb8ca, sage intact, prod restored). The parked cu130 rebuild won't fix it (e4m3 = hardware format, not CUDA version). DEFERRED to the Ada refresh (operator: "ada is coming, we'll optimize then" — Ada sm_89 has native e4m3, so triton's fp8 path should compile there). Mechanics: --enable-triton-backend is a compose environment: var, so toggling it needs docker compose up -d (recreate), NOT docker restart (reuses the baked env, no-ops silently). Full: auto-memory parked_triton_backend_ampere_fp8.

143 older entries archived to archival-memory.md.

142 KiB Raw Blame History Unescape Escape