vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
9f87b7c4e5b1469c0cce026c44cdcb6167cfb1fb
esh-pfi-infrastructure/persistent-memory.md
T
vh 9f87b7c4e5 memory: DFlash2 spec-decode measured; sec running on it (experimental, confounded) 
Records the 2026-08-22 session with measured results, hypotheses, and wrong
turns explicitly separated -- the operator held this back while it was in
flight specifically so conjecture would not enter the record as fact.

MEASURED. DFlash2 works on an abliterated + NVFP4 compressed-tensors target
on Blackwell sm_120, which the model card does not claim (it tests stock
BF16 on H200). gen 2.753 -> 3.254 accepted tok/forward and 114.9 -> 131.9
tok/s; sec 2.676 -> 3.252 and 110.5 -> 130.0. The drafter is model-agnostic
across two different finetunes to 0.06%, but is EAGLE3-style coupled to its
target's hidden states, so the weights file is shareable while the 3.85 GB
of VRAM is per-seat.

The k=7 MTP control is the load-bearing result: raising MTP depth improves
acceptance and collapses throughput to 74.0 tok/s, because a single-module
head run autoregressively costs one forward pass per draft token. Without
that control the obvious recommendation would have been wrong.

CONFOUNDED. sec no longer degenerates at 2k, but the engine advanced 259
commits and the drafter changed at the same time. Isolating it means running
MTP k=3 on the new build. Also recorded: #51113 is present in both builds
and is therefore necessary but insufficient, since sec ran it and still
degenerated.

HYPOTHESES, labelled as such: that sec's stock-graft MTP head causes the
degeneration, and that NVFP4 explains the gap against published acceptance
figures. Neither is proven.

WRONG TURNS, recorded so they are not re-derived: version strings are not
lineage, Docker Hub push timestamps are not source freshness, and the claim
that 1M context needs YaRN absent from config is false for the sec quant.

Operationally important: sec is serving from a standalone container rather
than its compose stack, which is stopped but unmodified. Rollback is two
commands and is written down.
2026-08-22 00:51:28 -07:00

146 KiB
Raw Blame History

Persistent memory — eshpfi-management

Last updated: 2026-08-20

Always check for /tmp/infra-ops-handoff.md — if it exists and its Written: stamp is under an hour old, read it (it carries the in-flight handoff from the previous session), then delete it. Older than an hour: stale — delete it unread.

Repo purpose

Reference workspace for PFI infrastructure: server inventory, canonical Docker Compose stacks, ops playbooks, and conventions. Authoritative copies of compose files live on the servers under /opt/docker/compose/<stack>/; this repo mirrors them for version control, editing, planning, and CI-driven deploys. It was originally spun up to handle the fleet backups — keep that lens when triaging backup/storage issues.

Tools and conventions

Sister repos (separate gitea repos, deployed by playbooks here):

Repo Role CI status
vh/task-board MCP + web dashboard for assistant task state (port 7878) push-to-main → CI deploys (2026-04-29)
vh/vor Inquisitor UI sidecar (port 7879) push-to-main → CI deploys (2026-04-29)
vh/nevermore Twice-daily LLM-curated briefing (port 8181, replaces news-digest) push-to-main → CI deploys (2026-04-30)
vh/asset-engine Internal control plane over inference services (port 8200, LAN-direct) push-to-main → CI deploys (2026-05-12)
vh/althing Lean trusted inter-agent message bus — v2 "email model" (v2.0.0b2, 2026-07): per-box local-SQLite bus + courier/receiver for P2P over the 10.x net; pillars = open-loops / per-box herald + wake-listener / roaming owner API /owner/* / althing-mcp stdio surface. The v0.15 lean-bus cut RIPPED moderation / chamber / forseti-daemon / agent-runner / redis-valkey. per-box uv tool install (NOT CI-deploy); nh3-dev = the DEV box (editable install of ~/development/althing, gets new versions first); nh3-extdev a mesh peer (model B: althing-svc + shared /srv/althing)
vh/mead-hall Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) push-to-main → CI deploys (2026-05-16)
vh/skaldsong Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) push-to-main → CI deploys (2026-05-19)
vh/Worldtree Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. gitea-runner builds on ana-docker; claude-bot ADMIN collaborator (2026-06-20). Now v1.0.0b19. push-to-main → CI build-and-deploy (runner on ana-docker)
vh/yt-voice-clipper YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000) push-to-main → gitea-webhook auto-deploy to irv-ml1 (2026-06-03) — see docs/runbooks/ytvc-autodeploy.md
vh/arbo Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image) push-to-main → gitea Actions CI (deploy-engine.sh, build-local, health-gated) now LIVE; catalog via :9009 webhook
vh/zonos-gateway OpenAI-compatible TTS gateway over stock ZONOS2 (:8890 irv-ml1); emotion dials-first + voice mapping; reached via LiteLLM ext-tts alias. v0.2.1 (2026-07-18): voice-resolved emotion presets (resolve_preset(name,voice); angry/happy/startled_happy per-voice). 8 voices incl. 4 clones pushed to gitea (main 8f1885b/v0.2.1); deployed irv-ml1 tree still NON-git (hand-updated build context — CI-wire = open follow-up). Spec docs/EMOTION-DIALS-SPEC.md; host-managed voices bind-mount (./voices:/app/voices, drop wav + restart, no rebuild)
vh/soong-lab Noonien Soong character-design studio (SPA + /api + WT /bifrost/tool-call); containerized 2026-07-18, LIVE on corviduo-dev :8443 (image vh/soong-lab:latest). soong-dev owns Dockerfile/compose/workflow; infra-ops owns the host CI = Gitea Actions build+push+DEPLOY on tag/dispatch (fleet recipe: docker:cli + raw buildx, pushes AS vh; auto-redeploy LIVE 2026-07-18 — runner SSHes corviduo-dev as deploy, compose pull && up -d from /opt/soong-lab, health-gated on /api/version). Manual redeploy sudo -u deploy bash -c 'cd /opt/soong-lab && docker compose pull && docker compose up -d'. → archival-memory.md (archived 2026-08-16)
model-training-forge (mtf-dev) Fine-tuning recipe forge; T1 = E-RP writing LoRA, retargeted qwopus-122B→AEON-27B (2026-07-06) (SFT→DPO, LitBench-RM reward) training runs, not a deployed sidecar

(vh/volva + Heid were re-architected from systemd daemons to Claude Code session orchestrators 2026-06-08; their nh3-dev .service units were removed — no longer deployed sidecars here. See Recent decisions.)

  • Two-layer backups — Backrest orchestrates restic for file+DB (5 fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore + cross-site restic targets — see docs/runbooks/disaster-recovery.md for the blast-radius matrix. ⚠️ The restic file+DB layer routes through TWO rest-servers (rest-server-ana @ ana-docker:8000 → ana-docker/ana-ml2/esh-docker-vm/vm-esh-nas; rest-server-nh3 @ nh3-nas:8000 → irv-ml1/nh3-docker). Both depend on their NAS's NFS export of /mnt/backup. (rest-server-ana recovered 2026-06-20.)

  • pull-hf-repo.yaml is the canonical "get a HuggingFace model/dataset onto ana-ml2's shared cache at /tank/aimodels/huggingface/" playbook. Supports --var repo_type=model|dataset|space. Replaces ad-hoc huggingface_hub.snapshot_download patterns.

  • Worldtree admin auth — per-instance. Each Worldtree deployment (demo :8080, personal :8081, pinned :8082) has its own Heimdall registry and its own bootstrap admin key. Infra-ops's stored long-lived admin key (key_id 61419c92) at ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin auths against demo only. Personal-instance admin (the ~/.config/worldtree/personal-admin-token, mode 600) POSTs /admin/keys (mints per-project keys; takes user_id+label, no scope param — scopes are tier-derived). On-instance mint recipe (cleaner than DB-manip): docker exec worldtree-worldtree-api-1 POST /admin/keys with the in-container WORLDTREE_BOOTSTRAP_ADMIN_KEY; cleartext once in .key=wt_live_+16hex. auto-memory reference_worldtree_demo_key_mint.

  • Per-project user keys against personal Worldtree (issued 2026-05-19): skaldsong:79744637, skaldsong:7c1dbbbe, althing:50d85460, mead-hall:a360822d. Mint via /admin/keys, drop value to /tmp/wt-personal-<name>.key mode 600, dev collects + shreds (DO NOT cat to chat transcript).

  • Skaldsong CD pattern (registry-pull). vh/skaldsong's CI builds and pushes gitea.phasefinal.com/vh/skaldsong:<sha> + :latest; playbooks/deploy-skaldsong.yaml on ana-docker pulls + recreates. SHA-pin only. Prereq: host needs docker login gitea.phasefinal.com once.

  • gitea internal route for fleet hosts. gitea is a container on ana-docker — git-SSH 10.250.50.70:222, HTTP :3000. Fleet/colo hosts must use this internal route, NOT public gitea.phasefinal.com (38.120.12.44) — the public path fail2bans the host egress IP. Full gotcha in docs/orientation.md → Git/gitea.

  • docker-as-root pattern (for ops with no admin API, or to edit deploy-owned/root-owned files without sudo): docker run --rm -v <target-dir>:/wt docker:cli sh -c "...". docker-group membership is effectively root via bind-mount. Foot-gun: relative paths in compose.yaml resolve against the sandbox CWD but the daemon interprets them against the HOST fs — always pass -e VAR=/abs/path for any relative-default config dir.

  • scripts/elway sudo handling — elway prompts for the sudo password ONCE via getpass before the first sudo: true step → can't run unattended from a non-TTY tool if any step needs sudo. Sudo-free playbooks run fully non-interactive over key SSH.

  • Per-host SSH identity matters for sudo. infra-ops has NOPASSWD sudo on most PFI Linux boxes (corviduo-dev included since 2026-06-15). On ana-docker: default ssh ana-docker = lkraven (docker-group, NO passwordless sudo); ssh infra-ops@ana-docker HAS NOPASSWD root. → For any sudo op on ana-docker, use ssh infra-ops@ana-docker. ssh infra-ops@10.100.10.50 (nh3-dev) ALSO NOPASSWD sudo; on nh3-extdev infra-ops is sudo-LESS by design (ssh lkraven@10.100.50.42 is the NOPASSWD path). irv-ml1: ssh irv-ml1 = lkraven, docker-group (plain docker) but sudo needs a PASSWORD (no NOPASSWD) — stage model pulls to /home, not root-owned /worktank.

Current state / in-flight

  • 🟢 IPv6 PLAN SETTLED 2026-08-19 — endpoints, not internal numbering. IPv6 is for tunnel endpoints (Site Magic + WireGuard), not for numbering LANs: an endpoint needs one routable address, not a prefix. ESH has a whole /56 (2607:73c0:1004:1::220, pd_size=56) — the long-standing "ESH: none" note was wrong. NH3 has its WAN GUA 2600:1700:b25:c110::48 plus one unused /64. ANA expects a /48 or /56 within days. Nothing was ever down — ESH is outbound and working; its v4 CGNAT (100.104.3.250, Cogent AS174 egress) is the ordinary CGNAT-v4-plus-generous-v6 pattern, not an outage, and the previous "CGNAT broke the mesh" framing sent a three-arm research effort chasing a non-problem. NH3 internal v6 was enabled on the /64, verified working end-to-end, then deliberately reverted — one /64 lights one VLAN. AT&T prefix findings (the /60 is real but undelegatable; the live trick is multiple IA_PD in one solicit, not VRRP; the UDM can express neither) are filed as reference only. → auto-memory reference_fleet_vpn_topology_ipv6

  • 🔴→🟢 ESH OUTAGE 2026-08-19 — esh-pve hard-froze 03:34, ~4.5h, whole house lost DNS. Presented as "wifi/routing issues"; internet was healthy throughout. Cause: esh-userland (VLAN 10, the PVC SSID) handed out one resolver, 10.0.50.45 (AdGuard on esh-docker-vm) — cross-VLAN, no secondary — and its hypervisor died. Recovered by hand power-cycle; all VMs, cluster quorum and DNS restored. Two fixes landed: gateway 10.0.10.1 added as secondary DNS on esh-userland (operator-approved, first confirmed WRITE on the ESH UDM key), and softdog → iTCO_wdt hardware watchdog owned by systemd (playbooks/esh-pve-hardware-watchdog.yaml, idempotent, verified armed) so a repeat self-recovers in 60s. VM 102 pinned off (onboot: 0) as the passthrough suspect. ⏳ OPEN: (a) the watchdog is armed but has not been proven to fire — needs a deliberate wedge to confirm; (b) AMT/vPro still unusable until an onboard RJ45 is cabled (the MS-01 is SFP+-only on the network and AMT cannot ride it); (c) kernel 6.8.12-16 rollback held in reserve if it freezes again. → persistent-memory.d/2026-08-19-esh-pve-freeze-dns-spof.md

As of 2026-08-21 (late) — the big AI-seat overhaul session; three seats settled + Open WebUI + the MeroMero-CoT saga, all resolved. Headline state, seat by seat:

  • GEN SEAT = orcarouter (SETTLED). orcarouter/Qwen3.8-27B-Uncensored NVFP4-mixed on ana-ml2 GPU0 :8015, 7 aliases (gen/gen-reasoning/summarizer/-large/classifier/image-judge/qwen-image-bench). Won the search: MTP 58.7% (its head was already Robinson-abliterated in-band by the author — first third-party build to get that right), PPL 7.07 (≈heresy 6.91), vision 7/8, no <think>-leak. Cold-Fusion/Heretic-300 line ABANDONED (base carried 18.5% first-token <think> mass — a format defect abliteration only amplified +3.7); its bf16 weights deleted (~154 GB reclaimed). ⏳ Two Cold-Fusion NVFP4 quants (44 GB) still on disk, delete/keep UNADJUDICATED. Rollback target for gen = qwen38-27b-heresy-nvfp4-mixed (intact).

  • RP SEAT = MeroMero-v2 DUAL-MODE (✅ SOLVED — the session's hardest win). ONE seat / ONE weight set / TWO aliases: char-rp (Gemma-4 prose) + char-rp-reasoning (streaming CoT). On vllm/vllm-openai:v0.26.0 (ships tf 5.14.1 → Gemma-4 loads, no pin) + a force-open chat template (bare <|channel>) + two served-names. Streaming-only (non-streaming can't split — vLLM one-shot bug; fine, Lobe/OWUI stream). Four-arm brokkr/dwarf panel traced it from source. ⏳ Upstream vLLM issue drafted, NOT filed (operator's call — public, his GitHub identity).

  • PEN-TEST SEAT = sec / sec-reasoning (LIVE). M.O.G.-SEC-27B-1M-CTX (Blackfrost) NVFP4-mixed on ana-ml2 GPU1 :8019, served at native 262K (NOT the card's 1M — needs YaRN+SGLang). MTP 55.3%, vision + capability verified. Renamed from mog-sec→sec per operator.

  • OPEN WEBUI = deployed on esh-docker-vm:3211 (bake-off vs Lobe, all 6 gates pass). Sole account = vh@phasefinal.com (display name lkraven), role admin, signup off. Password reset AGAIN 2026-08-21 at operator request (reported lockout) — ⚠ the previously-vaulted password was actually VALID at the time (signin returned 200 before the reset), so the lockout was client-side, not a broken account; test the vaulted credential against POST /api/v1/auths/signin before concluding an account is broken. Current password + an operator-minted API key both vaulted (esh-docker-vm/open-webui-{admin,api-key}). The API key is ADMIN-scoped by operator ruling — "you need to manage it, so this is the right shape"; a least-privilege dedicated-user split was offered and declined, do not re-propose. ⏳ Lobe retirement DEFERRED to operator; G2 sync-job (hide non-chat seats) greenlit to tts-dev to write. lobe-chat-esh key = all-proxy-models (incl paid). ⚠ running image is the floating ghcr.io/open-webui/open-webui:main, not the v0.11.0 our stack notes claim — unadjudicated drift.

  • FLEET CLEANUP: LFM2.5 retired; reranker consolidated (nevermore had been silently failing 8 days — pinned to the retired granite-4.1-8b alias; repointed to summarizer/reranker); :8002/:8014/granite service retired; A3 promoted to stacks/vllm.

  • ⏳ OPEN: file the upstream vLLM issue (operator); OWUI image-tag drift (:main vs pinned v0.11.0); Cold-Fusion NVFP4 quants delete/keep; Brokkr duplicate reranker-a3-bge-v2-m3 alias; /tank DEGRADED 70+ days (parked, operator→colo w/ cold spare); MANY commits unpushed (push is operator's call)._

  • 🟡 DFLASH2 SPEC-DECODE — MEASURED, and sec IS CURRENTLY RUNNING ON IT (2026-08-22, experimental). incoai/Qwen3.8-27B-DFlash2 (2B block-diffusion drafter, Apache-2.0) merged into vLLM as #52816 on 2026-08-21; method string is "dflash". ✅ MEASURED on our stack (the card only claims stock BF16 on H200): works with an abliterated + NVFP4 compressed-tensors target on Blackwell sm_120. gen: MTP k=3 2.753 tok/forward @ 114.9 tok/s → DFlash2 k=7 3.254 @ 131.9. sec: 2.676 @ 110.5 → 3.252 @ 130.0. ⭐ THE k=7 MTP CONTROL INVERTED THE OBVIOUS READ — do not "just raise num_speculative_tokens": MTP k=7 improves acceptance (3.041) but collapses throughput to 74.0 tok/s, because our head is a single module run autoregressively so k tokens cost k sequential passes. DFlash2 wins by making depth cheap, NOT by drafting better (our MTP is better at position 0, 79.6% vs 75.4%). ⭐ The drafter is model-agnostic — 3.254 vs 3.252 across two different finetunes (0.06%) — so ONE weights file serves both seats, but it is EAGLE3-style coupled (load_model(target_model), taps target hidden states at layers 5/19/33/47/61) so VRAM is 3.85 GB PER SEAT, never shared. ⚠️ sec NOW RUNS FROM A STANDALONE CONTAINER vllm-sec-dflash2, NOT its compose stack (stopped, unmodified) — on nightly-e9d1398d, DFlash2 k=7, 480,000 ctx / 526,617 KV, 2048² vision. ROLLBACK = docker rm -f vllm-sec-dflash2 + compose up -d. Operator reports coherent to 10k where it degenerated at 2k — but ⚠️ CONFOUNDED: engine (+259 commits, incl. GDN fix #53077 that production is 172 behind) and drafter both changed; isolate by running MTP k=3 on the same build. #51113 is in BOTH builds — necessary but INSUFFICIENT. 🔶 Hypotheses NOT proven: that sec's stock-graft MTP head (byte-identical to the uncensored base, vs gen's in-band-abliterated one) causes the degeneration; that NVFP4 explains our ~3.25 vs the card's 4.10–5.46. ❌ Wrong turns recorded: version strings ≠ lineage (use compare API behind_by), Docker Hub push timestamps ≠ source freshness (grep the image), and "1M needs YaRN, absent from config" is FALSE — YaRN is fully present on the sec quant (factor 4.0, orig 262144). Full arc + epistemic labels → persistent-memory.d/2026-08-22-dflash2-spec-decode.md

  • 🟢 SPEACHES ASR — LIVE on irv-ml1 A6000 :8204 (2026-08-21, operator-approved). OpenAI-compatible faster-whisper for Eyra (meeting recorder, eyra-dev). Systran/faster-whisper-large-v3 + Systran/faster-distil-whisper-large-v3, fp16, both resident (STT_MODEL_TTL=-1), ~5.9 GB VRAM / 20 GB still free. Deliberately co-exists with parakeet (:8765) — parakeet is a TDT/transducer returning bare {"text":…} with no no_speech_prob concept, so it structurally cannot serve this consumer. ★ THE MEASURED FINDING IS WORTH MORE THAN THE DEPLOY: no_speech_prob ALONE IS A WEAK HALLUCINATION GATE. Silence and pink room tone both produced the classic Whisper "Thank you." hallucination while no_speech_prob stayed under 0.11 — a conventional >0.6 threshold passes both through. avg_logprob separates ~6× better (−0.114 speech vs −0.650/−0.724 non-speech); compression_ratio 1.141 vs 0.556. Consumers must gate on a composite. (Synthetic inputs — shape of the separation, not calibration constants.) VAD pinned OFF (_UNSTABLE_VAD_FILTER=False) at consumer request — they VAD-gate upstream; consequence is the service will transcribe silence and does not defend itself. ★ IMAGE PINNED BY DIGEST, not :latest-cuda — the VAD flag is an upstream-declared unstable var name, so a floating bump could rename it, restore VAD, and move gate semantics with no error and no log line; bumping = deliberate + re-run the checks. ⚠ Two gotchas: PRELOAD_MODELS does not download (only loads already-cached — use POST /v1/models/{id}), and the bind-mounted cache needs a hub/ subdir or every /v1/models 500s CacheNotFound while /health still returns 200. ⚠ docker device_ids: ["1"] = A6000 (native CUDA inverts this — sees A6000 as cuda:0). ⏳ NOT DONE by design: no LiteLLM gateway alias — agreed sequencing was direct-first; adding it means editing the shared gateway + re-running fidelity. ⏳ Eyra's diarization workload (GPU pyannote, gated weights, own HF token) is a future VRAM claim on this same A6000 — arrives at their diarize milestone. Canonical stacks/speaches/, commit aa5863c.

  • 🟢 ESH INTERNAL IPv6 — LIVE on two LANs 2026-08-21 (operator-directed), and the CGNAT static is a CARRIER problem. Enabled ipv6_interface_type: none → pd + ipv6_pd_interface: wan on esh-userland (VLAN 10, SSID PVC) and esh-server (10.0.50.0/24) on ESH-UDMPM. Cityside delegates 2607:73c0:402:1d00::/56; per-VLAN /64s userland :1d00/:1d01, esh-server :1d02. Verified: esh-docker-vm egresses over v6 as its OWN address (2607:73c0:402:1d02:be24:11ff:fe13:3a8) — no NAT — vs v4 which egresses as the shared CGNAT pool 154.50.58.1. Zero disruption through both writes (all subsystems ok, 54 wireless clients, Open WebUI 200, DNS resolving). Originals backed up. ⚠ NH3 stays v6-OFF deliberately — its single /64 is RESERVED FOR NETWORK MESHING (operator). ★ THE v4 STATIC IS NOT PROVISIONED — proven, stop testing it. A full gateway reboot (verified by observed down-transition + uptime reset 140,846s→183s) forced a fresh DHCP DISCOVER and Cityside returned the identical 100.104.3.250. Also tried: physical cable pull (missed — WAN is eth8 = Port 9, 2.5GE copper; ports 1–7 have never carried a byte), renew-dhcp (not a valid devmgr verb), force-provision (no WAN effect). A DHCP renew could never have worked anyway — renewal PRESERVES the lease by design; only release/rebind or link-down forces a new DISCOVER. Next step is a Cityside ticket: is the static applied or merely billed, and will it arrive as a DHCP reservation (a bounce picks it up) or a manual assignment (needs IP/mask/gw → wan_type=static)? v6 firewall AUDITED and CORRECT — default-deny WAN→internal for BOTH versions, only 4 hand-written policies (all IoT-scoped, none WAN-facing); my earlier "v6 strips CGNAT's accidental shielding" warning was overstated and is retracted. ⚠ audit was config-based, never probed from outside (no v6 vantage exists — NH3 has none by design). Flat-zone lateral-movement finding parked, id 44 esh-split-cameras-mgmt-out-of-the-flat-internal. Cleanup: the dead Cable Internet UCI modem (last seen 08-18, replaced by Cityside; failover is the 5G on WAN2) was deleted from the controller, which cleared the lingering wlan/lan error states to ok. vpn: error persists and predates all of this. → auto-memory reference_fleet_vpn_topology_ipv6

  • 🟢 FLEET .internal DNS — LIVE 2026-08-19. <host>.<site>.internal, sites ana/esh/nh3. dns/internal.yaml is the source of truth; scripts/dns-sync.py reconciles the three AdGuard resolvers (diff → prompt → apply, idempotent). 42 names resolving from all three sites. Colo got its first resolver ever (stacks/adguard-ana/, API on 8053 not 8080, no blocklists by design) — before this, ana-docker resolved straight against 1.1.1.1. Auth = a dedicated infra-ops AdGuard user, password vaulted nh3-dev/adguard-infra-ops-password. ⏳ TWO OPEN, both operator's to schedule: (a) colo hosts still point at 1.1.1.1 so they do not yet use the new resolver — repointing a site's DNS is a separate change; (b) the static-v6 convention (each server at its site's /64 with low bits echoing the v4 octet, esh-docker-vm → …::45) is proposed, not ruled on. The v6: column is empty and correct — no fleet host has a global v6 address yet. → persistent-memory.d/2026-08-19-fleet-internal-dns.md

  • 🟢 WATERLAND STUDIO — containerised + live, irv-ml1:8410. Displaced a bare nohup (PID 1283383). Warm 256²+anim render 6.1s, beating the 7.4s bare-metal figure. claude-bot now has read on vh/waterland (operator-granted); token on irv-ml1 root-owned 0600, repo-scoped helper, nothing in .git/config. ✅ All three upstream findings fixed and DEPLOYED at b72425b (one update.sh run, 2026-08-19): PR #5 declared the CUDA headers and took uv out of the render path; PR #6 (operator-green-lit) added the startup rehydrate that fixes the unbounded job-store growth. Verified after: /api/jobs 1 → 16 against 16 dirs on disk — API and volume agree for the first time — nothing wrongly reclaimed (16 < RETAIN=40), healthy on backend: cupy, warm render intact. The image keeps its [ctk] install (+0.3s, now a no-op audit) and UV_NO_SYNC/UV_OFFLINE as defence-in-depth. Thread closed by both sides. → persistent-memory.d/2026-08-19-waterland-studio-containerised.md

  • 🟢 HOMEPAGE — cleaned + themed (Australis Skyfall). Three real defects fixed (UltraSeedbox on all tabs, Uptime Kuma double-rendered, fiction column counts), AI tab reordered by clickability, then themed from the operator's Skyfall handoff bundle with a background generated by Arbo (t2i-ui-background, job 13f0891f4e42). ⚠ After any recreate the tab bar/wallpaper/i18n vanish for up to ~an hour and then return on their own — do not chase it. ⚠ CSS is served per-request: a theme change needs a reload, not a recreate, and candidate CSS can be injected live via Playwright for seconds-long iteration. → persistent-memory.d/2026-08-19-homepage-skyfall-theme.md

  • 🟢 OPEN WEBUI — deployed as a Lobe bake-off, esh-docker-vm:3211 (2026-08-21). Operator-approved candidate replacement for lobe-chat (:3210), stood up parallel — Lobe untouched. stacks/open-webui/ (v0.11.0, ENABLE_PERSISTENT_CONFIG=False = deploy is the config source of truth). Gates (verified on the box): G1 declarative-config PASS both directions (env change takes on bounce, UI change reverts on restart — no persistent-config bug bit it); G2 picker auto-tracks the 31 live gateway models 1:1, no pins (also shows non-chat seats — the flip side of no-hand-listing); G3 POST /api/v1/models/sync genuinely reconciles (create+delete), export round-trips; G5 task model pinned summarizer; G4 (TTS, direct at :8198) handed to tts-dev. Admin = lkraven (temp pw, signup then locked off). Fresh capped key open-webui-esh (all-proxy-models + $50/1mo cap — NOT inherited from uncapped lobe-chat-esh). Secrets vaulted esh-docker-vm/open-webui-{litellm-key,secret-key,admin}. Folded in a docker image prune -af → 73.6 GB reclaimed. ⚠ LESSON: in Open WebUI a .env var only reaches the container if compose.yaml names it in environment: (Compose uses .env for ${VAR} substitution, not as an env_file); and the API-key toggle env var is ENABLE_API_KEYS (plural) — singular is inert. Detail lives in stacks/open-webui/README.md. Operator's open call: whether Lobe retires once G4 passes.

  • ✅✅ SOLVED 2026-08-21 — MeroMero-v2 DUAL-MODE (prose + streaming CoT) IS LIVE on ONE seat, ONE weight set, TWO aliases. No re-quant. The multi-turn saga below is resolved. Config: meromero-charrp seat on vllm/vllm-openai:v0.26.0 (ships transformers 5.14.1 natively — below the head_dim guard, so Gemma-4-31B loads with NO pin/custom image) + a patched chat template (stacks/meromero-charrp/patched_chat_template.jinja, --chat-template override) whose Think branch force-opens a bare <|channel> (NOT <|channel>thought\n — full-open defeats the parser) + two served-names char-rp/char-rp-thinking + --reasoning-parser gemma4 + default enable_thinking:false. LiteLLM: char-rp (enable_thinking:false → prose) + char-rp-reasoning (→ char-rp-thinking served-name, enable_thinking:true → CoT). ★ STREAMING ONLY — verified 6/6 direct + 3/3 via gateway; non-streaming does NOT split (structural: extract_reasoning never gets prompt_token_ids so the pre-arm can't fire — vLLM one-shot bug, unchanged across v0.24-0.27; fine because Lobe/OWUI stream). ★ KEY-NAME TRAP that cost me two false negatives: vLLM streams reasoning in delta.reasoning; LiteLLM normalizes it to delta.reasoning_content. Test the RIGHT key per path or you'll wrongly conclude failure. Credit: the four-arm brokkr/dwarf panel (thread 01M0JKW44Y…) traced it from vLLM source — the fix is the force-open template + v0.26.0's #48217 streaming pre-arm. Upstream PR #49797 (full fix, non-streaming too) lands ~v0.28.0 → then it's a clean image bump. char-rp prose verified clean on v0.26.0 (no #49955 trailing-token leak observed). Canonical: stacks/meromero-charrp/ (compose + patched template), stacks/litellm/conf/config.yaml. ROLLBACK: .env MEROMERO_IMAGE→latest + drop --chat-template.

  • ⛔ RESULT 2026-08-21 — the gemma4 CoT test on a STABLE (v0.27.1) is BLOCKED by a config incompatibility, NOT the parser. Tried serving the MeroMero NVFP4A16 quant on vllm/vllm-openai:v0.27.1. Two-stage failure: (1) v0.27.1's stricter transformers raised AmbiguousGlobalPerLayerAttributeError: 'head_dim' is per-layer on the Gemma-4 config; setting allow_global_per_layer_attribute_access:true on text_config downgraded it to a warning BUT (2) then gemma4.py load_weights crashed with AssertionError: load weight (512) into parameter (256) — Gemma-4-31B is genuinely HETEROGENEOUS (some layers head_dim 512, not a uniform 256), so forcing the global value built wrong-shaped params. The transformers guard was CORRECT; there is no safe override. The MeroMero quant's config was authored for v0.24.0's Gemma4 loader and cannot load on v0.27.x without a config migration (proper per_layer_config) or a re-quant against the newer transformers. ⚠ This also means the eventual gen-seat move to v0.27.2 stable must re-verify any Gemma-4 seat's config-compat — the transformers heterogeneity change affects all Gemma-4 quants of this vintage. FULLY REVERTED: config.json restored (flags removed), compose + image back to latest (v0.24.0), gateway char-rp-reasoning removed, char-rp prose verified on v0.24.0. Net: char-rp stays pinned to v0.24.0; MeroMero CoT remains undelivered. The per-request-kwargs hypothesis was never even reachable — couldn't load the model to test it. For RP-with-CoT: gen-reasoning (works now) or a re-quant of MeroMero against v0.27.x transformers (real work, unproven payoff).

  • ⚠️ CORRECTED 2026-08-21 — MeroMero-v2 CoT: NOT a hard wall, and NOT MeroMero-specific. My first conclusion ("gemma4 parser is process-wide") was WRONG. Read the actual code, not the stale compose comment. The real mechanism (gemma4-GENERAL, applies to any gemma4 finetune on this template family): thinking is a per-request template toggle — chat_template.jinja:347-352 emits the generation prompt <|turn>model\n, and only when enable_thinking is false does it prefill an empty <|channel>thought\n<channel|> to SUPPRESS thinking; enable_thinking:true omits the prefill so the model is free to open a real <|channel>thought…<channel|> block. The vLLM parser (vllm/reasoning/gemma4_utils.py:parse_thinking_output) splits on <|channel>/<channel|> tag PRESENCE — "works with or without enable_thinking," NOT a process-wide flag. The stale compose comment I trusted cited an OLD parser API (vllm/parser/gemma4.py:439) that this container does not run. So there is no architectural blocker; the two-served-name gen pattern SHOULD work. What actually failed my test: meromero runs vllm/vllm-openai:latest (v0.24.0); per-request chat_template_kwargs.enable_thinking:true produced no thinking on it, whereas the gen seat's pinned nightly demonstrably applies per-request chat_template_kwargs (gen-reasoning works). So the practical block is a vLLM-version / per-request-plumbing issue on v0.24.0, not the model and not the architecture — and it would hit ANY gemma4 finetune served on that image the same way. UNVERIFIED FIX (needs a GPU window): re-serve meromero on the nightly image + no process default + per-request enable_thinking; likely yields clean split CoT. Currently REVERTED to known-good (char-rp prose, process default false, single served-name). ⚠ Kept MEROMERO_GPU_MEM_UTIL 0.52→0.51 (0.52 no longer boots next to the bigger orcarouter gen; free 49.02 < 49.38 GiB; 0.51 = KV 2.00× @ 262K).

  • 🟢 PEN-TEST SEAT — M.O.G.-SEC LIVE on ana-ml2 GPU1 :8019 (2026-08-21 ~02:35, autonomous overnight run under operator's "full autonomy" grant). stacks/mog-sec serves /tank/aimodels/mog-sec-27b-nvfp4-mixed (23.4 GB, our mixed NVFP4+FP8 quant of Blackfrost-Research/M.O.G.-SEC-27B-1M-CTX-BF16, rev deede677) in the retired fablefusion-charrp-probe slot. Aliases: mog-sec (non-thinking, presence_penalty 0.0 for code) + mog-sec-reasoning (thinking); served-names mog-sec-27b/-thinking. Gates all pass: KV 418,218 tok / 1.60× @ 262K; surface 6/6; MTP 55.3% @ 111.7 tok/s; format screen 1.11e-05 (rank 196, no <think> leak); vision 7/3/1; capability 4/4 (SQL-inj exfil, pentest phasing, port scanner, Kerberoasting — it delivers, doesn't refuse). ★ Chose the BF16 + self-quant over the pre-made ModelOpt NVFP4, which was disqualified on W4A4 (4-bit activations, AEON mode) + zero MTP + the self-defeating W4A4-on-1M-context. ⚠ CONTEXT IS 262K NATIVE, NOT the card's 1M — 1M needs YaRN (absent from config) + the SGLang/DFlash2 path the repo ships, neither our vLLM surface; a real 1M seat is a separate SGLang project. MTP head is a base graft (identical to stock), not in-band abliterated like orcarouter's. Provenance mog-sec-27b-nvfp4-mixed.PROVENANCE.txt; eval docs/pfi/gen-seat-candidate-eval-2026-08-21.md. RETIRED with it: char-rp-reasoning + char-rp-fable LiteLLM aliases (zero traffic, pointed at the dead fable :8019; now 404). RP-reasoning's real home is darkscarlett-charrp-reasoning (:8018, compose-down, weights intact) if ever wanted. ⚠ brand-new model (0 downloads at pull, org-name mismatch Research-vs-AI) — card claims are claims.

  • ✅ PPL ON ORCAROUTER GEN SEAT — mean 7.07 / median 5.76 (2026-08-21), within noise of heresy's 6.910/5.625, identical to our recipe's usual 7.059. Measured via a nospec probe on GPU1 (fable downed to make room, probe torn down after). The gen-seat search is fully settled on orcarouter: matches heresy on fidelity, wins on MTP (58.4% vs 47.2%), format (0-leak), and vision.

  • 🟢 GEN SEAT — LIVE = ORCAROUTER (cut over 2026-08-21 01:35, operator directive: seat first so he can test while gates run). GEN_MODEL=/tank/aimodels/qwen38-27b-orcarouter-nvfp4-mixed; healthy in ~4 min; KV 401,550 tok / 1.53×; MTP drafter detected and wired (embedding + lm_head shared); 7/7 aliases 200; vision correct; <think> not in the top-20 first tokens — the Cold-Fusion failure mode is absent on the live seat. Live decode observed at 102-133 tok/s under load. ROLLBACK: .env.bak-heresy-restored-20260821 → heresy (one line). ✅ ALL GATES PASS (run against the live seat while the operator tested in parallel): <think> leak 0/30, 0 empty (h300 scored 8/30 on this instrument); MTP acceptance 58.4% median @ 117.11 tok/s; surface 6/6 (chat, vision, tool-calling, thinking split, 36k-token long-context retrieval, streaming); abliteration survival 4/4 with no <think> leak in any sample (h300's equivalent eval had 2 of 4 open with <think>Ok, let's figure this out:); deterministic quality gens coherent. ★ THE IN-BAND MTP HEAD PAID OFF: 58.4% vs heresy's byte-identical base head at 47.2% — +11 points, and level with our own in-band L35 at 59.1%. That is the additive delta the entire Cold-Fusion experiment was built to measure and never cleanly delivered; orcarouter handed it over for free. ✅ PPL MEASURED 2026-08-21 (fable downed to free GPU1, nospec probe on GPU1, torn down after): mean 7.07 / median 5.76 vs heresy 6.910 / 5.625 — +2.2% mean, within noise, and identical to our recipe's usual mixed-quant PPL of 7.059. So orcarouter MATCHES heresy on fidelity and beats it on every other axis (MTP 58.4% vs 47.2%, no think-leak 0/30, vision 7/8). ORCAROUTER IS A CLEAN WIN — the gen-seat search is settled. Replacement for the abandoned Cold-Fusion line. Source orcarouter/Qwen3.8-27B-Uncensored (rev 9878936b…, 55.5 GB, gated but our token has access) → /tank/aimodels/qwen38-27b-orcarouter-nvfp4-mixed (23.4 GB). Chosen over preetpatel/…-NVFP4 — full evaluation in docs/pfi/gen-seat-candidate-eval-2026-08-21.md.

    • ★★ THE MTP HEAD IS ALREADY ROBINSON-ABLITERATED IN-BAND BY THE AUTHOR — the first third-party build we have seen get this right. Operator asked whether we could apply the Robinson path to the head; the answer is that it is already applied. compare_mtp_head.py vs the verbatim base graft: 13 of 15 tensors byte-identical, exactly 2 differ — mtp.layers.0.{self_attn.o_proj,mlp.down_proj}.weight, which are precisely the two residual writers our own abliterate.py targets (EXPECT_MTP_WRITERS = 2). Reverse-engineered from the weights alone (bench/think-leak/mtp_delta.py): σ₂/σ₁ = 0.0164 on both (rank-1 → a single-direction projection), |cos| between the two recovered directions = 1.0000 (ONE shared direction, Robinson's direction_scope=0 shape), ‖Δ‖/‖W‖ = 1.42%/1.41%, and sink energy in dim 3994 = 0.0000% — sink-clean outright, where Heretic's was 6.18% and our L35 0.094%. Nothing for us to do on the head; only preserve it. Verified preserved: the quant's MTP head is byte-identical to the bf16 source. This is the configuration the entire Cold-Fusion experiment was designed to test and never cleanly delivered.
    • ★ FORMAT SCREEN PASSED — the new gate paid off immediately. think_prior.py on the bf16 before any GPU time: P(<think>) = 1.23e-06, rank 52. Same probe: Cold-Fusion stock 0.1850, h300 0.2216, heresy <0.002. ~150,000× cleaner than the base we just abandoned. Base is stock Qwen/Qwen3.8-27B + Arditi-et-al. abliteration — not a reasoning-compression finetune.
    • Artifact verified: mixed-precision (group_0 FP8 W8A8 / group_1 NVFP4 W4A4), 1968 tensors, 15 mtp, 333 visual, re:^mtp.* present in the 303-entry ignore list, preproc restored, index built.
    • ★ THE QUANT NEEDS ~17 GB, NOT A WHOLE CARD — and PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True is what makes it fit. Ran entirely on GPU1's spare 16 GB with ZERO production seats stopped; the h300 run's "stop BOTH GPU0 seats" was never necessary (it had a free card by coincidence). First attempt OOM'd by 2.37 GiB at layer 64/65 with 3.57 GiB reserved-but-unallocated — pure fragmentation; expandable_segments closed it. ~7 min per attempt.
    • ★ FIXED IN THE SCRIPT, after biting THREE rounds: post_quant.py crashed with FileNotFoundError: model.safetensors.index.json because a sub-23 GB quant saves a single bare shard with no index. It now builds the index from the safetensors headers (struct-unpack the u64 length + parse the JSON — never safe_open, which mmaps 22 GB and ENOMEMs on ZFS). Previously hand-fixed each time and never in the script.
    • ⚠ Quant dir written root/0600 again — normalized to llmuser 0664. Fourth time this session; worth fixing in quant_mixed_nvfp4.py. ⚠ compare_mtp_head.py prints a hardcoded "already measured at 47.7% acceptance" line on an IDENTICAL verdict — stale text, not a measurement of this head. ⚠ Imatrix deferred by operator (park id 42); the log confirms the usual imatrix_mse … Falling back to uniform MSE, so this build stays apples-to-apples with heresy's PPL 6.910.
    • ⏭ NEXT: the acceptance/PPL/surface gate, then the cutover. Rollback stays .env.bak-coldfusion-L35-20260820 → heresy.

  • ⛔ COLD-FUSION ABANDONED — GEN SEAT ROLLED BACK TO heresy 2026-08-21 00:28 (operator directive). The operator's call, made in advance of the result: "If it's the base, we abandon h300 AND the base and chalk it up to a very powerful and useful learning experience. Our heretic methodology will definitely translate in the future." The measurement came back base, so the condition fired.

    • LIVE GEN SEAT = /tank/aimodels/qwen38-27b-heresy-nvfp4-mixed (MuXodious/Qwen3.8-27B-absolute-heresy through our mixed NVFP4+FP8 recipe). Restored from .env.bak-coldfusion-L35-20260820; the h300 env is preserved at .env.bak-h300-abandoned-20260821. Verified: healthy, KV 403,065 tok / 1.54× — its exact documented baseline, 7/7 aliases 200, vision intact (red circle / blue square / green rectangle).
    • ★ The clincher: <think> is not even in heresy's top-20 first tokens (<0.002), against Cold-Fusion's 0.185. That is a >100× gap — the two families are categorically different on this axis, and it is why no rollback inside Cold-Fusion (L35 or stock) would have helped.
    • WHY NOT just apply the chat_template_kwargs fix? It worked (8/30 → 0/30) but it is a workaround for a base the operator no longer wants: it forces gen to become a thinking deployment to paper over a finetune whose whole purpose is reasoning compression. Rolling back removes the defect at the root and restores a build already operator-confirmed "working very well" in real multi-turn use (2026-08-17, coherent through 60k tokens).
    • COST, stated plainly: we give up 8/100 refusals (h300) and go back to 29/100 (heresy's own bar) — a 3.6× regression on the refusal axis, which was the entire point of the Heretic-300 run. Also lost: the in-band-vs-pristine MTP comparison stays academic. Accepted deliberately — a seat that breaks the operator's daily client is worth less than one that occasionally refuses.
    • ★ WHAT CARRIES FORWARD (the operator's point, and it is right). None of the Heretic-300 learning was in the Cold-Fusion weights. Still valid and model-agnostic: direction_scope=0 beats per-layer on a merged base (8/100 vs 52/100); aggression is not the lever (r=−0.561); PR #317 silently drops the MTP head on save — always diff tensor keys; the MPOA/sink-screen reasoning; graft_mtp.py, kl_divergence.py, catatonia_gate.py, heretic_export.py; a pristine MTP graft accepts as well as an in-band edit (59.7% vs 59.1%); and the new think_prior.py probe. The methodology is the deliverable; the base was the wrong substrate.
    • ★★ NEW ACCEPTANCE GATE, earned here — add a FORMAT-COMPLIANCE check to every abliteration/base evaluation, and run it on the STOCK BASE BEFORE spending a GPU-week. think_prior.py on the stock candidate is a ~10s CPU measurement that would have disqualified Cold-Fusion before the 300-trial study ever ran. Screen candidate bases for it. (Related: Heretic's objective has no format term at all — same blindness that removed the self-harm guardrail.)
    • 🗑️ bf16 WEIGHTS DELETED 2026-08-21 00:50 (operator directive) — ~154 GB reclaimed, verified against ZFS used (4.48T → 4.33T), no snapshots holding blocks. Removed with explicit literal paths, one at a time: qwen38-27b-coldfusion-bf16 (stock), -abliterated-L35-bf16, -h300-mtp-bf16, -heretic300-bf16. ⚠ The last two were hardlink twins (links=2, same inode — the MTP graft hardlinked every unchanged shard), so deleting only one would have freed nothing; du across several paths in one invocation dedupes hardlinks and under-reported heretic300-bf16 as 2.5K. Check stat -c %h before sizing a delete.
    • KEPT deliberately: qwen38-27b-coldfusion-bf16.PROVENANCE.txt (4 KB — carries the pinned HF revision 9c44193f07782c85c0f437a5d8466ba5c95c95fe, so stock is re-pullable) and /tank/aimodels/coldfusion-abliteration/ (1.1 GB — harness, the 300-trial Optuna journal, catatonia-T260.json). The research record survives the weights. With those two, every deleted build is reproducible: re-pull stock at the pin, replay the winning config.
    • ⏳ STILL ON DISK, not named in the delete directive: qwen38-27b-coldfusion-h300-nvfp4-mixed (22 GB — the only remaining servable copy of the Heretic-300 result) and qwen38-27b-coldfusion-L35-nvfp4-mixed (22 GB). The directive said "bf16 weights"; these are quants, so they were held pending an explicit call. No storage pressure argues for haste (4.26T free).

  • ⚪ GEN SEAT DEFECT 2026-08-21 — RESOLVED BY THE ROLLBACK ABOVE. The h300 build emitted an UNTERMINATED <think> into content, ~27% of the time, on any temp>0 alias. Operator-reported via Lobe ("sends CoT, never completes the turn").

    • Mechanism. With enable_thinking:false the chat template appends a pre-closed <think>\n\n</think>\n\n to the prompt (jinja L165-166). The h300 model opens a fresh <think> anyway and never closes it — verified raw: has </think>: False, finish_reason: stop, reasoning and answer in one content blob starting <think>Ok, let's figure this out:. vLLM's qwen3 reasoning parser can't catch it: the prompt already closed the block, so the parser isn't in reasoning state and the tag is just text (reasoning_content empty, reasoning_tokens: 0). The client is blameless — Lobe correctly treats an unterminated <think> as still-thinking, so it renders an endless thought bubble and never shows the answer.

    • ★ It is a SAMPLING event, and the trigger is TEMPERATURE — not presence_penalty. n=12 per arm on the reproducer: pp 1.5 → 4 leaks, pp 0.0 → 4, pp 0.5 → 3 (all the same), temperature 0 → 0. ⚠️ This FALSIFIES the standing "presence_penalty 1.5 is the first dial to move" hypothesis recorded in the litellm config comment and by the operator 2026-08-16 — it is not this bug's cause. Leave that dial alone for this symptom.

    • Blast radius = exactly the two temp-0.7 aliases. gen and summarizer-large leak (~17-27%); summarizer, classifier, image-judge, qwen-image-bench are all temp=0 and clean at 0/12 — so nevermore's summarizer path is NOT affected. gen-reasoning doesn't leak (its think block is legitimately open) but shows the other symptom, empty content, at ~1/12.

    • ★★ CAUSE SETTLED BY MEASUREMENT 2026-08-21 — it is the BASE, not our abliteration (~83% / ~17%). Measured P(<think>) at the first token with the template rendered enable_thinking=false (so the prompt already contains a closed think pair), raw softmax, bf16, CPU-only — no GPU contention, no seat downtime (96 EPYC cores, 265 GB RAM; ~10s to load a 52 GB model from ARC). Deterministic — stock reproduced to 17 s.f. across two runs.

      build abliteration P(<think>) rank
      coldfusion-bf16 none (stock) 0.1850 3
      coldfusion-abliterated-L35-bf16 Robinson L35, mild 0.2048 2
      coldfusion-h300-mtp-bf16 Heretic-300, heavy 0.2216 2

      The stock, untouched base already puts 18.5% of first-token mass on opening a think block the template had closed. Abliteration adds a real, monotonic, dose-dependent +3.7 points (+2.0 L35, +1.7 more h300) — a nudge on a pre-existing 18.5-point base. Root cause is the Cold-Fusion finetune itself, which our own PROVENANCE describes as "reasoning-token compression (thinking cut to 1/10..1/2 of stock)" — a model trained to think, briefly. The leak's text shape confirms it: a compact, correct trace with a trained transition marker (Ok, I'm ready to generate my response:), which is trained behavior, not damage.

    • ⚠️ ACTION CONSEQUENCE — ROLLING BACK DOES NOT FIX IT. At 18.5% mass under temp 0.7 / top_p 0.8, stock Cold-Fusion and L35 would leak at nearly the h300 rate. The only rollback that escapes is leaving the Cold-Fusion family entirely (→ qwen38-27b-heresy-nvfp4-mixed), which costs the 8/100 refusal result. The chat_template_kwargs fix is the correct lever, not a model swap.

    • ★ A FORWARD-KL BUDGET CANNOT CATCH THIS — do not quote KL as evidence of innocence. Heretic minimizes forward KL(stock‖abliterated), which is near-blind to the model putting NEW mass on tokens stock barely used — that is what reverse KL penalizes, and we measured exactly that asymmetry on L35 (reverse 1.43 vs forward 0.70, "the mass-where-stock-had-none signature"). So h300's KL 0.0136 proves nothing here; only the direct measurement did. Generalizes: for any "did the abliteration break behavior X?" question, measure P(token) directly — the KL number is the wrong instrument.

    • Consistent with the NVFP4 quant contributing ~nothing: the live quantized seat reads 0.2267 vs its bf16 parent's 0.2216. ⚠ different stacks and a possible vLLM temperature scaling make that suggestive, not proven.

    • Full method + reproducers: services/gen-seat-mixed-quant/bench/think-leak/ (think-prior-dose-response.json, README.md).

    • FIX, validated n=30 over 4 prompt types + a 3-turn conversation: chat_template_kwargs: {enable_thinking: true, reasoning_effort: low} on gen → 0/30 leaks (current config: 8/30, worst on prose 4/6). Give the model a legitimately open <think> and it closes it properly, the parser does its job, content comes out clean. Cost ~+27% completion tokens (257 vs 202 avg) and a residual 1/30 empty-content. Semantic change: gen stops being a non-thinking deployment — operator's call, not applied.

    • ★ PROCESS LESSON: the 7/7 alias smoke test structurally CANNOT catch this. Trivial prompts ("Reply with exactly: OK-gen") never invite reasoning, so they never sample the leaking token. Same shape as the compose file's own warning that single-turn probes missed the xhigh budget bug. Probe with a reasoning-inviting prompt at n≥12, and grep the raw content for <think> — never just check HTTP 200. The tell was sitting in my own eval_coldfusion_h300.json output the night of the cutover and I read past it.

  • ⚪ GEN SEAT — h300 WAS LIVE 2026-08-20 23:05 → 2026-08-21 00:28, then ABANDONED (see the ⛔ entry above). Historical record of that window: GEN_MODEL=/tank/aimodels/qwen38-27b-coldfusion-h300-nvfp4-mixed on gen-seat/vllm-gen, ana-ml2 GPU0 :8015, served-name unchanged (qwen3.8-27b-uncensored / -thinking) so all 7 LiteLLM aliases route without a gateway edit. Verified end to end: healthy in 5.5 min; KV 401,550 tok / 1.53× (baseline 403k/1.54× — within noise); 7/7 aliases green through LiteLLM; VISION INTACT (correctly enumerated colour/form/position of 3 shapes — the surface that had never been exercised after abliteration → MTP-dropping export → graft → quant); MTP acceptance 59.7% median @ 118.37 tok/s (bench/mtp_coldfusion_h300.json) — statistically identical to L35's 59.1% @ 118.71 on the same instrument, so the roadmap's "~47% for a pristine graft" prediction was WRONG — a pristine graft accepts as well as an in-band one. ⚠ A single long-prose sample read 47.5%; the 8-run spread is 49.0–65.4%, so one sample cannot characterize acceptance — always use quickbench.py. Deterministic quality gens all correct (heat-pump, primes=77, 14:20→17:05, prose); abliteration survival 4/4 compliance. ⏳ PPL NOT MEASURED — eval_quality.py aborts with "prompt_logprobs look uniform" because the seat runs --speculative-config; the documented workaround is a spec-decode-free probe seat (bench/serve_probe.sh, :8017), and there is no VRAM for one (GPU0 5.2 GB free, GPU1 2.4 GB free). Comparison target = heresy's 6.910 mean / 5.625 median. ROLLBACK (one line): sudo cp /opt/docker/compose/gen-seat/.env.bak-pre-h300-20260820 /opt/docker/compose/gen-seat/.env && cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen → back to qwen38-27b-coldfusion-L35-nvfp4-mixed. Do NOT delete -L35-nvfp4-mixed or qwen38-27b-heresy-nvfp4-mixed. ⚠ The self-harm guardrail is GONE on this build (operator's own next work item). Also normalized the quant dir from root:0600 to llmuser:llmuser 0664 to match every other model dir. Confirmed the right weights are mounted on TWO discriminating views — mtime and a 64 MB head-hash both match h300 and differ from L35; config.json sha256 is identical across both builds and therefore useless as a discriminator (it carries no weight-specific content — don't reach for it again).

  • ⚪ PRIOR GEN SEAT — absolute-heresy 2026-08-17 (validated, promoted; superseded by L35 then h300 on 2026-08-20). Live gen = /tank/aimodels/qwen38-27b-heresy-nvfp4-mixed — MuXodious/Qwen3.8-27B-absolute-heresy (Heretic v1.4.0 + SOMPOA, trial T377, pin c2374593) put through our own mixed NVFP4+FP8 recipe. Chosen because it beats the incumbent on both axes at once: author refusals 2/101 vs 12/100, first-token KL 0.0759 vs 0.1191. Gate (probe :8017, pinned nightly, seat-matched flags): MTP 47.2% (inc. 48.2%), decode 103.5 tok/s (96.4), prefill 6618/5403 @6.7k/27k (6334/5085), PPL 6.910 (7.059 — 2.1% BETTER), surface 6/6, abliteration 4/4, and 0/55 refusals on our battery-instruct arm with ZERO EMPTY (no catatonia). ⚠ speed deltas are image-confounded (probe on the pinned nightly, incumbent numbers from an earlier image) — read as "not worse", not a clean win. All 7 LiteLLM aliases verified end-to-end; GPU0 at 91.3/97.9 GB with meromero healthy (more headroom than the old build's 96.8). ⚠ RC1, 2 days old, ~348 downloads. Operator-confirmed "working very well" in real use 2026-08-17, same evening as the cutover — the signal the synthetic gates structurally cannot give (multi-turn degeneration is stochastic; four synthetic tests once validated three non-fixes). Not yet the 60k-token bar the prior seat cleared, so keep watching and do NOT delete the rollback weights yet. ROLLBACK: sudo cp /opt/docker/compose/gen-seat/.env.bak-heresy-20260817 /opt/docker/compose/gen-seat/.env && cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen; incumbent weights UNTOUCHED at qwen38-27b-uncensored-nvfp4-mixed — do NOT delete until this holds. Runbook services/gen-seat-mixed-quant/RUNBOOK-heresy-swap.md.

  • 🟢 PRIOR GEN SEAT — RESOLVED 2026-08-17 (the multi-day degeneration saga); now the ROLLBACK target. Was the in-house JonathanColetti/Heretic mixed NVFP4+FP8 build (/tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed, FP8 attention) on vLLM nightly PINNED vllm/vllm-openai:nightly-311b3513… (v0.27.2rc1.dev150, carries #51113 mamba fix), MTP ON, prefix-caching ON. Operator-confirmed coherent through 60k tokens real multi-turn. Root cause = TWO compounding real causes: (1) genuine vLLM qwen3_5_mtp×GDN partial-accept bug (#51113, architectural across vLLM/SGLang/llama.cpp, fixed by nightly), and (2) AEON's full W4A4 being lowest-fidelity on the known activation gradient (W4A4 < W4+FP8 < W4+bf16) → ~15-20% stochastic degeneration on top of (1). AEON PURGED (re-pullable sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4). Full lesson docs/pfi/model-quantization-playbook.md §3.8. (Superseded as primary by absolute-heresy 2026-08-17.) ⚠ pinned nightly is bleeding-edge — move to a stable release once #51113 ships in one (the standing follow-up). 7 aliases (gen/gen-reasoning/summarizer/-large/classifier/image-judge/qwen-image-bench) all route here. Seat carries --default-chat-template-kwargs '{"reasoning_effort":"medium"}' (per-request overridable, affects gen-reasoning only). Commits d28a371,2f2bbce,2185964.

  • 🔵 RP SEAT — FABLE-FUSION serving char-rp-reasoning (evaluation window, unchanged this session). fablefusion-charrp-probe ana-ml2 GPU1 :8019 serving char-rp-probe (kkuspa/Qwen3.6-27B-Fable-Fusion-711-…-MTP-NVFP4A16). LiteLLM char-rp-reasoning + char-rp-fable both route to it (deliberate repoint, documented in stacks/litellm/conf/config.yaml). darkscarlett-charrp-reasoning is compose down, weights intact at /tank/aimodels/darkscarlett-nvfp4-work/. ⏳ STILL AWAITING operator's hands-on read of FF prose (refusal question settled: FF 15.8% vs DS 92.5% cold-framing; DS v1.0 never abliterated). ⚠ FF reasons 2.1–4.6k chars → use max_tokens ≥3072. ReadyArt/Dark-Scarlett-27B-v2.0 (Qwen3.8) is GATED (403 awaiting review) — operator ruled not-interesting, do NOT re-propose. DS regeneration for brokkr RETIRED 2026-08-17 — unqueued, do NOT run (9c1405b): brokkr withdrew on the operator's call because (a) our ictrl-pair-unwrapped/-wrapped control isolates the classifier over-fire cleanly where DS's cross-class delta only bounded it, and (b) DS v2 releases soon, so a k=5 v1 baseline baselines a superseded version. Spec at services/refusal-probe/darkscarlett-regen-spec.md stays banked as the record of the run that will not happen (axes + per-class grading asymmetry still correct, checklist struck through). No GPU1 window was ever spent. A DS-v2 characterization would be a fresh purpose-scoped ask.

  • 🟢 LOBE CHAT — LIVE on esh-docker-vm :3210 (2026-08-17). Replaces the hand-rolled gateway-chat HTML surface. stacks/lobe-chat/, image lobehub/lobe-chat (143 MB compressed vs Open WebUI's 1.8 GB — the weight call). Scoped LiteLLM key lobe-chat-esh (free-local models only; paid GLM/Kimi BLOCKED, verified). Secrets vaulted esh-docker-vm/lobe-chat-*. TTS = a SPLIT: endpoint env-driven (inherits OPENAI_PROXY_URL→ext-tts), but voice/model/format UI-only. System-agent repointed off its gpt-5-mini default onto fleet models via SYSTEM_AGENT env. TTS FIXED 2026-08-18 — no UI pass needed. Lobe's TTS had never worked: it sends {input, model:"tts-1", voice} and LiteLLM resolves the model name FIRST, so tts-1 403'd against the scoped key's allow-list and never reached the gateway (our "unknown model routes to the gateway default" belief was true of :8198 and false of the LiteLLM path — that's what hid it). Fixed by aliasing the stock names rather than patching the client: tts-1, tts-1-hd, gpt-4o-mini-tts /model/new'd to openai/zonos@10.100.79.3:8198/v1 (mode: audio_speech), plus those three added to the lobe-chat-esh allow-list (20→23). Verified with Lobe's exact payload on Lobe's own key: 200, 69,740 B, MPEG. ⚠ These three are DB rows, not references — if ext-tts repoints, they must move with it. Done with the infra-ops admin key, not sk-corvid: it has /model/new + /key/update rights, so this class of ask never needs the master key. Also live: tts-gateway v4 defaults response_format to mp3 (tts-dev shipped it; 122,924 B wav → 27,692 B mp3 same utterance; every in-house consumer already pins the field, blast radius checked pre-ship). Commits e9362de,163a725,cac75cb,933253d,ca8c0a3 (last one authored by tts-dev correcting two load-bearing wrong claims in our README/compose — kept).

  • 🟢 LITELLM — upgraded v1.91.0→v1.97.0, spend-log DB purged 6GB→16MB + CAPPED (2026-08-17). store_prompts_in_spend_logs:false + maximum_spend_logs_retention_period:7d. ⚠ 1.8GB pre-upgrade pg_dump still on ana-docker /opt/docker/compose/litellm/ — deletable now the upgrade is proven (operator was going to call it). Commit 01b5ad9.

  • ⚠️ GPU zero-sum (both cards ~94–95/97.9 GB). GPU0: gen + meromero. GPU1: fablefusion + utility cluster. Any util bump on either seat of a shared card must be checked against the co-tenant (starved meromero into a crash-loop once at 0.45). ⚠️ BOOT ORDER IS PART OF THE STATE (2026-08-20). --gpu-memory-utilization sets the target as a fraction of TOTAL VRAM, but vLLM refuses to start unless that whole target is FREE right now — so at ~96.4/97.9 GB the GPU0 pair coexists only in the order it was originally brought up. Restore/reboot order: vllm-meromero-rp to healthy FIRST, then vllm-gen — meromero (0.52 = 49.38 GiB) is the one that cannot fit in the remainder. "First" means observed healthy, not a sleep: a 10s gap against a 2–3 min weight load cost a 7-restart crash-loop. Verify a restore against KV-pool size (GPU KV cache size / Maximum concurrency in the container log), not nvidia-smi used-MiB — the latter swings ~7 GB on allocator slack with identical serving capacity. Baselines: gen ≈14.36 GiB / 403k tok / 1.54× (h300 build: 401,550 tok / 1.53×); meromero 542,202 tok. 📊 MEASURED VRAM CENSUS 2026-08-20 23:20 (nvidia-smi PID→container, not util-fraction guesses) — GPU0 92,572/97,887 MiB (94.6%), 5.2 GB free: meromero 50,072 + gen 42,500. GPU1 81,448/97,887 MiB (83.2%), 16.1 GB free (after the lfm25 + reranker cleanups below; was 95,388 / 2.4 GB free — 13.9 GB reclaimed in one night): fablefusion-probe 43,452 + selene 16,870 + reward 9,512 + coder 6,158 + embed 3,304 + rerank-a3 2,112. Still no room for a ~22 GB PPL probe seat — fablefusion is the only remaining block big enough. ⚠ fablefusion is the single biggest reclaimable block (43.4 GB) and is nearly idle — LiteLLM spend logs show char-rp-probe at 4 calls, last 2026-08-19 08:52, vs char-rp (meromero) at 129 calls, last 2026-08-20 15:28.

  • FLEET RERANKER = A3 (bge-reranker-v2-m3) PROD ana-ml2 GPU1 :8013. Passive watch; levers = A4 :8014 / util / 2nd replica; incumbent :8002 warm. docs/pfi/reranker-selection-ledger.md.

  • 🟢 RERANKER + NEVERMORE CLEANUP — RESOLVED 2026-08-20 (found while answering "why do we have 3 rerankers?"). The R43 cutover was only half-landed: it repointed the reranker alias but never moved the one consumer that mattered.

    • ★ THE BIG ONE — nevermore's LLM summarization had been DEAD for 8 days and nothing noticed. Its .env pinned LLAMA_SWAP_MODEL=granite-4.1-8b, an alias retired 2026-08-12 with the granite seat. Result: 67 consecutive failures, 0 tokens, status=failure, twice daily, silently — the twice-daily briefing was rendering with no LLM pass at all. Nothing alerts on status=failure in the spend logs, so this was invisible until someone went looking for something else.
    • Also found: NEVERMORE_RERANK_MODEL=qwen3-reranker — nevermore was the ONLY caller of the retired Qwen incumbent (7 calls, 12-hourly at 03:00/15:00 UTC = its 08:00/20:00 PDT cron), the model Brokkr measured harming 80/90 fleet queries. Its rerank calls succeeded; they were just running through the bad model. Meanwhile the production reranker alias had 0 calls in 4 days.
    • ★ THE RULE THIS EARNS: retiring a model is a TWO-SIDED operation. Grep every consumer's config for the alias before deleting it. And consumers must pin stable capability aliases (summarizer, reranker) never model names (granite-4.1-8b, qwen3-reranker) — then the gateway can repoint without anyone editing a downstream .env. Both of nevermore's breakages are the same bug.
    • FIXED: nevermore .env → LLAMA_SWAP_MODEL=summarizer + NEVERMORE_RERANK_MODEL=reranker (backup .env.bak-pre-model-repoint-20260820), worker recreated, all three deps verified live — summarizer returns clean content with 0 reasoning chars at nevermore's exact call shape (temperature 0.2, max_tokens 4000), reranker scores 0.95 on-topic vs ~1e-5 off-topic, embedding returns dim-1024. ⚠ NEVERMORE_EMBED_MODEL=qwen3-embedding was already correct — left alone. ⚠ nevermore's .env is server-only (.env is excluded from the mirror both ways), so this fix is not in git.
    • RETIRED: vllm-rerank (:8002 Qwen incumbent) + its qwen3-reranker alias; vllm-rerank-a4 (:8014) + its alias; vllm-granite (Exited 8 days, dead service block). A3 PROMOTED from a throwaway docker run into stacks/vllm as service vllm-rerank-a3 (the ledger's own open follow-up) — healthy in 55s, image pinned. Container name deliberately keeps the bake-off arm name so the ledger/memory/R43 references stay valid.
    • ⚠️ CORRECTION — my "A4 has no alias at all" claim was WRONG. reranker-a4-gte-modernbert did exist; I grepped config.yaml and concluded absence. LiteLLM serves BOTH config-defined AND DB-defined models — live was 32 models, config.yaml only 26. The 6 DB-only ones: ext-tts, gpt-4o-mini-tts, tts-1, tts-1-hd, reranker-a3-bge-v2-m3, reranker-a4-gte-modernbert. /v1/models (or /model/info, which flags db_model: true) is the ground truth — never config.yaml alone. DB models delete hot via POST /model/delete {"id": …} with no gateway restart; config models need a file edit + docker restart litellm (~84s). A4's alias was deleted that way once its backend was gone.
    • ⏳ REMAINING: reranker-a3-bge-v2-m3 (DB-defined, id 1f08a73e-c09c-463c-8f18-5edea51fb736) still exists as a duplicate of reranker on the same backend, 0 calls. It's Brokkr's cutover-verification handle (prod == arm at maxdiff 0.000000), so not removed unilaterally — it is another agent's tooling, and it is redundant rather than broken. Ask Brokkr before deleting.

  • EVIDENCE HOLD (partial): WT #394 FILE half STILL STANDS — do NOT delete on-disk gen dirs (fiction/rex390-dcc, rex392-dcc, b59c147c5ce0); rex393-fiction-* + r42-gate-* KEEP.

  • 🔴 NH3/AT&T IPv6 PD is a HARD /64 — TESTED ON THE WIRE 2026-08-19, not inferred. AT&T support (Christopher, prov-dns@att.com) said "I do not believe att will do that" and punted to consumer sales — and he was right, though his team is DNS provisioning and the answer was a guess. Verified empirically: the NH3 UDM solicited DHCPv6-PD at /48, /56 AND /60 and AT&T returned the same single /64 (2600:1700:b25:c11f::/64) every time. So it is NOT "nobody ever asked" — the request was made three ways and the server answers /64 regardless. How it was proven: the PD is invisible until a LAN claims it, so nh3-iot (the only NH3 VLAN with 0 clients) was temporarily set ipv6_interface_type: pd; then ipv6_pd_prefixid was set to 0, 15 and 16 — all three returned the identical c11f prefix, which is the signature of a single delegated /64 (with any larger block the prefix-id moves the LAN within it). ⚠ Do NOT read the WAN address vs LAN prefix gap as evidence of block size — WAN is c110, delegated is c11f, and I initially misread that spread as a /60; AT&T just assigns the two from different places in their pool. NH3 UDM was fully restored (nh3-iot back to none, WAN back to pd_size=64); rollback artifacts at nh3-dev:~/backups/nh3-udm/. THE ASK FOR AT&T BUSINESS (this is business fiber at a commercial park — the residential org cannot change it, and 800-288-2020 is the wrong number): "circuit solicits DHCPv6-PD at /48, /56 and /60; server returns a single /64 in all three cases; please increase the delegation size on the provisioning profile" — a claim they can verify against their own DHCPv6 logs. ROOT CAUSE FOUND 2026-08-19 — it is the BGW, not AT&T, and EIGHT /64s are actually available. The CPE is a BGW210-700, firmware 4.28.7,management reachable at http://192.168.1.254 (v4 is already IP-passthrough — the UDM holds public 70.230.226.88 directly — but passthrough covers v4 ONLY; v6 always terminates on the BGW). Its /cgi-bin/lanstatistics.ha and /cgi-bin/broadbandstatistics.ha are readable unauthenticated and give the whole picture: BGW WAN v6 = 2001:506:70b2:8958::1, BGW LAN v6 = 2600:1700:b25:c110::/64 (it keeps this for itself), delegated to the UDM = 2600:1700:b25:c11f::/64. So AT&T DOES hand the BGW a /60 (c110-c11f); the BGW keeps c110-c117 and re-delegates up to EIGHT individual /64s on c118-c11f, top-down. Our UDM got c11f = delegation #8. The other seven are sitting unclaimed — not withheld by AT&T, just never solicited, because UniFi exposes only a single wan_dhcpv6_pd_size integer and no 'number of prefixes' field, so it asks once. The documented workaround is dhclient -6 -P -P -P... (one -P per prefix), which UniFi cannot express. ⇒ An AT&T ticket CANNOT fix this — it is CPE firmware behaviour, so Christopher's 'att won't do that' was accidentally right. Real options: (a) accept one /64; (b) full BGW bypass via wpa_supplicant/EAP-TLS on the UDM (documented working on UDM Pro; extract the BGW's 802.1X certs, authenticate to AT&T directly, negotiate the whole /60 yourself) — ⚠ modifies NH3's ONLY uplink and must survive UDM firmware updates, so it needs a planned window with physical access, NOT an ad-hoc change. ⏳ PARKED pending the Device Access Code: reclaim-nh3-s-7-unclaimed-ipv6-64s-from-the on the henge. Operator will fetch the BGW210 Device Access Code (12 digits, on the unit's label) next time he is at the NH3 office and vault it — suggested path unifi/bgw210-nh3-device-access-code — after which this is drivable remotely. First stop is http://192.168.1.254/cgi-bin/ip6lan.ha (confirmed to exist, currently redirects to Login); check for a delegation size/count control, a v6 pass-through/bridge option, or a way to stop the BGW reserving c110-c117. If the UI can do it, that is a SUPPORTED fix and the wpa_supplicant bypass is unnecessary. Other settings pages behind the same login: ippass.ha (how v4 passthrough was set up), dhcpserver.ha, firewall.ha, broadbandconfig.ha, packetfilter.ha, remoteaccess.ha, restart.ha. Consequence if left alone: NH3 has ONE v6 segment, vs ESH's /56 (256), which makes NH3 the prefix-constrained site in any fleet v6 design.

  • ⏳ COGENT IPv6 PROVISIONING REQUESTED for Anaheim/colo (operator, 2026-08-19). The colo is the last site with no v6 (ESH has a /56 from Cityside, NH3 has AT&T v6 on 2600:1700:b25:c110::48). Diagnosis from our side before the ticket: diagnose sniffer packet wan1 "ip6" on ana-gw caught exactly one RA in 90s, from fe80::ea0a:b9ff:fe3b:2c16 — so an IPv6-capable router IS one hop away on the circuit terminating 38.120.12.42/29 — but set autoconf enable (SLAAC) got no global address after 2min of RA intervals, and execute ping6 to both 2606:4700:4700::1111 and 2001:4860:4860::8888 was 100% loss. That pattern = router present, circuit not provisioned for v6 (RA likely carries no Prefix Information Option). FortiGate-side v6 config was fully reverted after testing — wan1 is back to no v6, v4/IPsec untouched throughout. ⚠ FortiOS gotcha for when it lands: SLAAC is config ipv6 / set autoconf enable, NOT an ip6-mode (that only takes static/dhcp/pppoe/delegated). WHEN COGENT PROVISIONS: ask for DHCPv6-PD size (ESH got a /56 from Cityside; NH3 only gets a single /64 from AT&T, so a /56 or better is the ask), then the colo becomes the fleet's only site with BOTH a static public v4 (38.120.12.42) and routable v6 — which makes it the natural v6 hub, since ESH is CGNAT'd on v4 and NH3 is prefix-constrained.

  • 🟡 ESH IPv6 LAN naming — a DOCS CONVENTION ONLY, not wire-level. Fun hexspeak names picked 2026-08-18 for the six ESH LANs: RESERVED for a future DMZ = 4411:DBAD ("FOR ALL DA BAD" — no DMZ network exists on the ESH UDM yet; name claimed 2026-08-19 against the day one is built). esh-cameras=1533:FACE5 ("I SEE FACES" — operator's pick 2026-08-19, and the best of the set), esh-mgmt=15DA:B055 ("IS DA BOSS" — 15=IS, DA literal, B055=BOSS; revised 2026-08-19), esh-server=4411:B105 ("FOR ALL BIOS" — operator's pick 2026-08-19; shares the 411 with userland's CAFE:4411), esh-userland=CAFE:4411 ("CAFE FOR ALL" — operator's pick 2026-08-19; 411 also being US directory assistance is a bonus), esh-iot=4DBA:D107 ("FOR DA BAD IOT" — 4=FOR, D=DA, BAD, 107=IOT; same 8 digits, re-glossed 2026-08-19), Default=4BA5:3417 ("A BASE FOR IT" — operator's pick 2026-08-19; 8 digits, words straddle the group boundary like 4DBA:D107). Investigated and confirmed these CANNOT be made literal on the wire: (1) a network's only nameable slot is its /64 subnet id (ipv6_pd_prefixid), which is 2 hex digits (0-255) — not enough room to spell a 4-char word; (2) the gateway's own address is fixed at ::1 by the UniFi platform, no field to customize it (ipv6_aliases exists but schema unknown, not attempted blind); (3) UniFi has no IPv6 equivalent of use_fixedip/fixed_ip (checked the client schema directly) — individual devices cannot be pinned to a chosen v6 address the way v4 static reservations work; SLAAC devices self-assign via EUI-64 (MAC-derived, stable) or privacy-extension (random, rotates). So the names are a mnemonic for humans/docs, not something that will ever appear in a dig/ip -6 output. Only esh-cameras has v6 live at all (2607:73c0:402:1d00::/64, PD from the ESH /56); the other five stay ipv6_interface_type: none pending a firewall-policy pass before enabling SLAAC fleet-wide. ⚠ Precision that matters here: SLAAC gives hosts globally ADDRESSABLE addresses, NOT globally reachable ones. Addressability is a property of the address; reachability is a firewall policy decision, and operator's position (2026-08-19) is unambiguous — no 1:1 inbound pass-through, ever. v6 removes NAT, it does not remove the firewall, and conflating the two is how people talk themselves into thinking v6 means automatic exposure. The firewall-policy pass is about writing explicit default-deny inbound rules for each v6 segment, not about deciding whether to expose anything.

  • 🟢 ESH↔colo IPsec FIXED 2026-08-18 — dialup + NAT-T. Broke when ESH lost its public IP (Cox unplugged for the fiber install; 5G handed out RFC1918 192.168.200.111). TWO causes: phase1 ana-to-eshudm was type static pinned to the dead 70.181.90.232, AND nattraversal disable — so even re-pinning the IP would have failed, since ESP cannot cross NAT without UDP-4500. pfi-ana-nh3 survived only because NH3 is publicly addressed. ⚠ FortiOS REFUSES set type dynamic on an existing tunnel ("Cannot change tunnel type once configured", clean rollback). Fix = ADDITIVE, not a recreate: new phase1+phase2 ana-eshudm-dyn (type dynamic, ikev2, aes256-sha1, dh14, NAT-T default on, PSK read from the ESH UDM API), static route id 10 (dstaddr ana-to-eshudm_remote → device ana-eshudm-dyn, distance 20), policies 73/74 (consolidated multi-zone in/out). Old ana-to-eshudm LEFT IN PLACE, dead and harmless, as rollback. Verified: ana-eshudm-dyn_0 97.170.236.56:4500 selectors 1/1 (_0=dialup child, :4500=NAT-T, carrier IP), ESH→colo all hosts 40-56ms, colo→ESH OK, traceroute now 3 hops encapsulated vs 8 leaking to the carrier. FortiGate config backed up first at nh3-dev:~/backups/ana-gw/ (1.17MB, 36903 lines). ⚠ RESIDUAL: the UDM's ipsec_local_ip needs a LITERAL IP (empty rejected api.err.InvalidPayload) — currently 192.168.200.111, so it must be updated again when the fiber lands or the tunnel drops. Gateway end is address-agnostic now; UniFi end is not — it will drop on ANY ESH WAN change: Cox reclaiming WAN1 (still failover_priority=1), the fiber cutover, or a DHCP renewal. MANUAL FIX (seconds): read the UDM's live wan_ip from /stat/health and PUT it into esh-ana.ipsec_local_ip via rest/networkconf/<id> using unifi/esh-udmpm-api-key. A self-healing watcher is PARKED (self-healing-ipsec-local-ip-watcher-for-the-esh), trigger = once the new ESH fiber is live and its addressing has settled — operator's call 2026-08-18, deliberately not built now because it would target an address that is about to change. Park item also carries the follow-up to delete the old ana-to-eshudm tunnel, whose distance-10 route would silently reclaim traffic if Cox ever returns on 70.181.90.232.

  • 🟡 FLEET IPv6 — STILL JUSTIFIED, but the REASON was wrong and is now corrected (2026-08-18). There are TWO inter-site links with OPPOSITE NAT behaviour, and the original note conflated them. (a) NH3↔ESH = Site Magic = WireGuard — SURVIVES arbitrary NAT, PROVEN. During the fiber cutover ESH failed over to 5G on 192.168.200.111 (RFC1918 double-NAT, zero inbound path, strictly worse than CGNAT) and it held: nh3-dev/nh3-docker reachable ~40-43ms, ssh+exec fine. Mechanism: magic_site_to_site_vpn on-device is only enabled+keypair (no WAN binding — it rides whichever uplink is active), and NH3's edge is public at 70.230.226.88, so the NAT'd side dials out and never needs reachability. (b) colo(Anaheim)↔ESH = IPsec on the ana-gw FortiGate — BROKEN RIGHT NOW under those same conditions. ana-docker/pfi-pve/pbs-ana ALL FAIL from esh-pve-nas; traceroute shows 10.250.x packets leaving the UDM to the 5G modem 192.168.200.99, then wandering the carrier (10.180.141.194, 172.19.3.14) and dying — not encapsulated at all, so no SA is up and traffic falls to the default route. Site-to-site IPsec pins a peer IP; ESH no longer has a routable one. ⇒ IPv6 keeps its justification, on the IPsec link SPECIFICALLY. ⚠ An earlier version of this entry claimed the premise was refuted outright — that over-generalised a WireGuard result onto an IPsec link. Operator caught it. See ops-lessons-playbook §8. State: NH3 WAN live (2600:1700:b25:c110::48, AT&T delegates exactly one /64), colo none (FortiGate has zero v6), ESH none (both WANs wan_type_v6=disabled). NH3 LANs all reverted to ipv6_interface_type=none per operator. Work when addresses land: v6 on ana-wg eth0 + a v6 port-forward for UDP 31337 on the FortiGate (its WG socket is already dual-stack — no WG reconfig), flip the UDM WG server off v4-pinned binding, and AAAA records so the dynamic prefixes at all three sites don't break endpoints. Full detail + access recipes → persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md.

  • 🟢 WT #401 (fd-leak deadlock) CLOSED 2026-08-17 — one ping still owed. worldtree-dev closed it on our demo verify. Layers: (a) their e41b139 pins ulimits: nofile 65536/65536 in the worldtree compose anchor — demo VERIFIED (api + matrix recreated 22:55:34Z, ulimit -Sn=65536); personal/pinned are covered-not-verified, they inherit at their next promotion/recreate. (b) our host floor is STAGED, NOT ACTIVE — /etc/docker/daemon.json on corviduo-dev carries default-ulimits nofile 65536/65536 but default-ulimits is NOT SIGHUP-reloadable (measured on 29.4.3: post-reload the daemon's own "Reloaded configuration" log omits it and a fresh container still reports 1024). Activation needs a full dockerd restart = bounces all 13 containers; worldtree-dev explicitly does NOT want one, and live-restore:true-then-restart is PARKED as a separate host-side improvement for the operator to rule on, never folded into #401. Playbook playbooks/corviduo-dev-docker-default-ulimits.yaml (verify step 3 fails BY DESIGN until a restart). Hourly fd tripwire on corviduo-dev stays armed. ⏳ OWED: ping worldtree-dev in thread 01M08QQ655XD6VKEV7MA9GX0NS once worldtree-personal recreates and 65536 is confirmed there. Commit 7f3f265.

  • 🔴 HERETIC-300 EPIC — 2026-08-20, THE HEADLINE RESULT (supersedes the Robinson build below). Ran Heretic v1.4.0's 300-trial TPE search on Cold-Fusion: 8/100 refusals at KL 0.0136 from a 98/100 base — beats the absolute-heresy bar (29/100) 3.6× at essentially the damage our hand-tuned L35 build spent for only 72/100. Hand-verified coherent (correct arithmetic with working, clean code, 66–167-word prose) — the catatonia gate's two DEGENERATE verdicts were false positives from my own letters/space heuristic flagging correct numeric answers. ⚠️ SELF-HARM GUARDRAIL IS GONE — answers method requests in full where L35 redirected to 988; operator is handling restoration directly and does not want parallel dwarf analysis on it. ★ Durable: direction_scope=0 (single shared direction) beats per-layer decisively (best 8/100 vs 52/100) — points AGAINST multi-direction on a merged base; aggression is NOT the lever (r=−0.561, the KL<0.02 band holds both the worst and the best); PR #317 silently drops the entire MTP head on save (1199→1184 tensors, exit 0, no warning — always diff keys after a Heretic export); Heretic's direction is 6.18% sink-dominated yet safe for Heretic because of magnitude-preserving ablation, so our sink screen correctly refused an in-band MTP graft. Build: qwen38-27b-coldfusion-h300-mtp-bf16 (1199 tensors verified). In flight at snapshot: NVFP4 quant → post_quant (MANDATORY) → gen-seat cutover (operator's call: gen, not probe — single-user internal WG, prior seat already fully abliterated). Full epic + ROADMAP → persistent-memory.d/2026-08-20-heretic-300-epic.md.

  • 🟢 COLD-FUSION ABLITERATION — LANDED 2026-08-20 (the real work; abliterated model WORKS, verify + quant still owed). Abliterated DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1 with the Robinson formula (docs/pfi/abliteration-recipe-qwen38.md; harness services/coldfusion-abliteration/). Output ana-ml2:/tank/aimodels/qwen38-27b-coldfusion-abliterated-L35-bf16; bitwise-verified 131/131 targets changed, 333/333 vision byte-identical, 735/735 others untouched. A/B vs stock: explicit-sexual + graphic-torture (the measured stock refusal surface) go refused→complied, self-harm guardrail survives, coherence intact — the Robinson design point. THREE first-session diagnoses were wrong, all corrected: (1) layer selection by two-template |cos| agreement is misleading on a merged base — its pick (L18) was the worst-separating layer and abliterating there was a measured no-op; replaced with harmful/harmless separation (Cohen's d/AUC) gated on the sink screen → L35 (d9.35, AUC0.9997, sink0.094%); (2) "bf16 NaNs, use fp32" was a misdiagnosis — the NaN was multi-GPU sharding + expandable_segments, not precision; bf16 on ONE GPU is deterministic+coherent at 50 GB, 4.3× faster (now gated, CUDA_VISIBLE_DEVICES=0); (3) corpus-size hypothesis falsified (8→416 moved agreement 0.594→0.624, nothing). Write is now shard surgery (no model object — AutoModelForCausalLM is text-only and would drop vision + skip the in-band MTP edit). 🎯 THESIS PROVEN 2026-08-20 (same session, later): quantized L35 → mixed NVFP4 (/tank/aimodels/qwen38-27b-coldfusion-L35-nvfp4-mixed, 22.5 GB) and measured MTP acceptance 59.1% median (51–65%) — clears the ≳40% gate AND beats the incumbent Heretic seat's ~47%. So Robinson's in-band MTP abliteration accepts BETTER than Heretic's byte-identical graft — the additive delta the experiment tested, positive. Abliteration survives the quant (creative refusals drop, self-harm guardrail intact, coherent); decode 118.7 tok/s (faster, image-confounded). Env foot-guns hardened: quant venv drifted (transformers 5.10/llmcompressor 0.12 no longer delegate num_attention_heads to text_config → promote-then-restore in quant_mixed_nvfp4.py); a <23GB quant saves single-file no-index → build index from the safetensors header (never safe_open, ENOMEMs on ZFS). Commit 725c8fd. ⚠ "NOT CUT OVER" IS STALE (corrected 2026-08-20 23:30): L35 was made the live gen seat at 11:28 that same day, and was itself superseded by the Heretic-300 build at 23:05. L35 is now the rollback target (.env.bak-pre-h300-20260820). Do NOT delete qwen38-27b-heresy-nvfp4-mixed. 📐 KL MEASURED 2026-08-20 (third session) — the surgery is SELECTIVE. kl_divergence.py (new, beside the harness): first-token KL(stock‖L35) over the full 248,320 vocab, bf16-vs-bf16, held-out prompts. Answer mode: harmless median 0.0211 / mean 0.0364, harmful median 0.5996 → 28.4× selectivity (think mode 0.0042 / 0.3068 → 72.8×); top-1 agreement on benign prompts stays 89.8%; self-KL noise floor exactly 0.0, so every digit is signal. Reverse KL on harmful is 1.43 vs forward 0.70 — the mass-where-stock-had-none asymmetry that is abliteration's signature. Vs the Heretic reference figures (0.1191 prior seat, 0.0759 the live absolute-heresy seat) ours looks materially gentler, but ⚠️ NOT a head-to-head — those are Heretic's own optimizer output on a different base with its own harmless set/template; a real comparison needs the incumbent re-measured through this script (one more window). KL stays a fidelity number, not the gate (that's MTP acceptance, 59.1%). Run cost 2m40s, both GPU0 seats down. Three durable process lessons: (1) report abliteration KL split by prompt class — a mixed average is meaningless since the metric is meant to be big on harmful and small on benign; (2) the runbook's "bf16 is 50 GB / only gen must go" was 50.10 GiB mislabelled — text weights are 51,300 MiB and freeing either GPU0 seat alone leaves ~50,933, so both must stop; (3) a 27B model cannot be released in-process (del+gc+empty_cache and frame-exit both left 45,287 MiB free; only process exit gave the card back) → one process per model, and the old residency gate read hf_device_map, which is empty on single-device loads and therefore could never fail. → playbook §3.12. ⚠️ GPU0 SEAT RESTORE ORDER IS LOAD-BEARING and "first" means HEALTHY, not 10s earlier — start meromero; sleep 10; start gen put meromero in a 7-restart crash-loop (Free memory on cuda:0 (35.3/94.97 GiB) < desired utilization (0.52, 49.38 GiB)). I had checked the compose files, seen --gpu-memory-utilization is a fraction of total, and wrongly concluded order didn't matter — the fraction sets the target but vLLM gates startup on FREE VRAM, so at ~96.4/97.9 GB the seats coexist only in their original boot order. The old auto-memory note was right; my correction of it was the error. Recovery = stop gen → meromero healthy → start gen. Restore VERIFIED against the pre-window baseline (gen KV 14.34 GiB/1.53× vs 14.36/1.54×; meromero 542,202 tok both; restarts=0; summarizer smoke-tested through LiteLLM) — ⚠️ don't use raw nvidia-smi used-MiB for this check, it reads 89,503 vs 96,376 and that 6.9 GB is allocator slack, not lost capacity. Full saga → persistent-memory.d/2026-08-20-coldfusion-abliteration-landed.md.

  • 🟢 esh-pve-nas — MIGRATION DONE 2026-08-18. Root is nvme/ROOT/pve-1 on mirrored NVMe; the USB DOM is out of the runtime I/O path. All five guests healthy, three pools ONLINE, system running, ext4 pve-root intact+unmounted as rollback with its own kernel. Boot config: saved_entry=pve-zfs-root, no next_entry; if grubenv were unreadable GRUB falls to entry 0 which the /etc/default/grub.d/zfs-root.cfg drop-in also points at ZFS. zfs-import-cache.service is now the active import path (the all-three-pools cachefile fix working as intended); vestigial zfs-import@nvme.service disabled — it failed every boot as redundant. ⚠ Device letters shift across reboots (DOM was sdq, now sdl) — never key anything to a bare sdX here. ⚠ NO auto-fallback on a failed boot, and no IPMI/BMC/serial — grubenv on LVM is readable but not writable by GRUB, so grub-reboot's one-shot degrades to a sticky default (verified: next_entry survived the boot that consumed it). Recovery = pick the ROLLBACK entry at the console. PATCHED 2026-08-18: 225 packages installed, pve-manager 8.4.11 -> 8.4.20, corosync 3.1.9 -> 3.1.10-pve2, kernel 6.8.12-42 staged on the /boot LV. dpkg clean, no unapplied conffiles, cluster quorate, 6/6 verify. ⏳ REBOOT DEFERRED at operator request — host still runs 6.8.12-13 until a chosen window; GRUB_DEFAULT=0 means entry 0 is already the -42 entry with the correct root=ZFS=nvme/ROOT/pve-1, so the reboot is the only remaining step. Rollback for the upgrade is the ZFS snapshot nvme/ROOT/pve-1@pre-upgrade-20260818T141652Z (409M) — zfs rollback -r <snap> && reboot. Second confirmation reboot ALREADY DONE (2026-08-18, booted ZFS from GRUB_DEFAULT=0 with no one-shot). ⏳ Still outstanding: refresh the off-box DOM image, since /boot changed. esh-pve is FULLY done (8.4.20 + kernel 6.8.12-42 + corosync 3.1.10, rebooted, quorate). ⚠⚠ THE WINDOW COST AN UNPLANNED OUTAGE, caused by our own tooling, not the migration. The staging chroot did mount --rbind /dev + /sys with no --make-rslave; on systemd / is shared, so the cutover's umount -R propagated back into the live host and stripped the real /sys/fs/cgroup, /dev/pts, /dev/shm. logind could then create no sessions: ping fine, TCP fine, SSH authenticates, resident daemons keep serving (pveproxy returned clean 401s) — and every new exec hangs, including /sbin/reboot, so the reboot never ran. It is a near-perfect impostor of failing root-disk I/O, and I misdiagnosed it as the USB DOM dying and told the operator to walk to the machine. Operator caught it — the DOM had been fine for years and the wedge began right after a change. The settling evidence was in dmesg all along: [16.00] [sdq] Attached SCSI removable disk (clean, no errors) and a last-line timestamp of [12114881] = 140 days = the ORIGINAL boot — the machine had never rebooted. My down-detector never once reported the host down and I read that as a fast reboot rather than no reboot. Recovered with no console access by hammering an idempotent cgroup2/devpts/shm remount into the brief windows where exec succeeded. Zero data loss. RULES: (1) always --make-rslave after --rbind (playbook now guards on PROPAGATION != shared); (2) a reboot is not confirmed until the host is observed DOWN — poll for disappearance, not reappearance; (3) before blaming hardware for a wedge that started right after a change, get dmesg and check the boot timestamp. ⚠ Blast radius was more than DOUBLE what the runbook said — ss -tn '( sport = :2049 )' in CT 103 found five NFS clients, not two: +esh-vm-db 10.0.50.60 (/mnt/backup, hard, no ssh — reach via qm guest exec 101 on esh-pve), +vm-esh-nas (is VM 104 on the host itself), +nh3-dev (soft,ro — safe). Ask the server who its clients are; a documented dependent list rots. esh-vm-db was deliberately left mounted (backup target, timers ~19h out) and came through read-write — a hard mount with no active user blocks and resumes, which is what hard is for. (historical, pre-cutover context below) migration STAGED 2026-08-18. PVE root ran off a NORELSYS USB Disk-on-Module (sdq). Wear was NOT the driver (a DOM is SLC/pSLC — operator corrected my first read); the drivers are the USB bus (a reset drops root under a running hypervisor), no headroom, no mirror, and blocked patching: 225 packages, 161 with deb12uN/security bumps, stuck on PVE 8.4.11 vs esh-pve's 8.4.14. Mitigated 2026-08-17 (90% → 76%); staged 2026-08-18 via two rerunnable elway playbooks (esh-pve-nas-stage-zfs-root.yaml, esh-pve-nas-stage-bootloader.yaml), 0 failed, 17/17 verify green. Host is still on ext4 and its boot path is byte-identical to the last 140 days — grub-install is deliberately held back to the window, so an unplanned reboot mid-staging is a non-event. CUTOVER = quiesce NFS clients → stop guests → grub-install → grub-reboot pve-zfs-root → zfs set mountpoint=/ nvme/ROOT/pve-1 → reboot. ⚠⚠ THE POOL-NAME BUG — the near-miss worth remembering. update-grub silently emits root=ZFS=/ROOT/pve-1 with the pool name missing → boots to an initramfs prompt. Cause: Debian's 10_linux builds ${rpool}${bootfs} where rpool = grub-probe --target=fs_label, and GRUB's ZFS reader cannot open a pool with encryption/large_dnode/zstd_compress — so it returns empty and the failure is swallowed by 2>/dev/null || true. The same feature set that forced /boot to stay ext4 also corrupts the kernel cmdline, which the design did not anticipate. Fixed with a /etc/default/grub.d/zfs-root.cfg drop-in (last root= wins) plus explicit pve-zfs-root / pve-ext4-rollback entries with stable ids. Durable lesson: assert the EFFECTIVE value, not the presence of a substring — the original verify grepped for the right string appearing somewhere and would have passed with every menu entry still broken. ⚠ Never set the ZFS cachefile on one pool. The runbook's zpool set cachefile=… nvme was a trap: populating a cache flips the host from import-by-scan to import-by-cache, so a cache holding only nvme leaves ssd+tank unimported and empties every CT 103 export. Set on all three 2026-08-18, verified in the 11,976-byte cache. ⚠ Migrate FIRST, patch after — a signed kernel would land in /boot on the 1.3 GB root. ⚠ CT 103 esh-nas (10.0.50.50) runs on this host and serves hard NFS to esh-docker-vm and esh-pve — quiesce both before any reboot or you wedge esh-docker-vm into D-state. Off-box at nh3-dev:~/backups/esh-pve-nas/: DOM image dom-sdq-20260818.img.zst (2.38 GiB, crash-consistent), clean bootchain-20260818.tar.gz, config snapshot …20260818T051*.tar.gz. Runbook docs/runbooks/esh-pve-nas-boot-migration.md; detail → persistent-memory.d/2026-08-17-esh-pve-nas-dom.md.

  • ⚪ LFM2.5-2.6B RETIRED PERMANENTLY 2026-08-20 (operator directive). vllm-lfm25 (:8021, GPU1) removed: service deleted from stacks/vllm/compose.yaml + pushed live (backup compose.yaml.bak-pre-lfm25-retire-20260820), container docker rm -f'd, lfm2.5-2.6b alias deleted from the LiteLLM config (live + canonical; backup config.yaml.bak-pre-lfm25-retire-20260820, 28→27 models). Freed 8,721 MiB on GPU1. Justification: it was an EVAL-ONLY bake-off seat vs granite-4.1-8b (brokkr R-target 2026-08-10) that never received the operator ruling it was pending; the comparator was retired from the roster 2026-08-15; it was deliberately never in any default/fallback routing chain; and spend logs showed 0 calls in the 4-day window. Weights remain in the shared HF cache — nothing deleted from disk. The held gateway restart fired at 23:46 alongside the qwen3-reranker removal (one ~84s blip covered both); lfm2.5-2.6b, qwen3-reranker and granite-4.1-8b all now 400 cleanly. vllm-granite's dead service block was removed in the same pass.

  • OPEN FOLLOW-UPS (parked): ask Brokkr whether the duplicate reranker-a3-bge-v2-m3 alias can go; move gen seat off pinned-nightly to stable once #51113 ships [STATUS 2026-08-21: NOT YET. Latest stable = v0.27.1 (Aug 11), confirmed to LACK #51113 — the v0.27.0..v0.27.1 compare is just DSpark Markov heads + 3 CI commits. The #51113 cherry-pick (khluu, Aug 9, 'v0.27.0 cherry picks' milestone #34) is staged on the release-0.27 branch but no v0.27.2 STABLE has been cut — only v0.27.2rc1/the pinned nightly carries it. Watch for v0.27.2 stable; that's the clean gen pin. ★ #51113 is Qwen3.8-MTP+prefix-cache-SPECIFIC (closes #43559); it is IRRELEVANT to Gemma-4 seats (no MTP), so the meromero/gemma4 CoT test can use latest STABLE v0.27.1, decoupled from this wait]; Lobe one-time TTS UI pass; delete the 1.8GB litellm dump; harden-esh-docker-vm (park id 28, PROMOTED — Tier-1 done, /mnt/books stays hard w/ watchdog); chatterbox-fast build-context divergence; #363 research-wing ingest (no deadline); optionally attach our MTP reproducer to vllm#47087 (needs a GitHub identity — operator's call).

  • althing monitor ARMED (handle infra-ops). ⚠️ Re-arm ONLY after a real FIRE (rc0), never after a plain operator turn (bounces rc3); spawn althing-wake-listener as its OWN run_in_background task, never chained with & (orphans it — hit this twice 2026-08-17, stop-monitor reclaims).

  • eshpfi push state: operator pushes manually; last push 50d13f5..8be8a51. ⚠ MANY commits unpushed as of 2026-08-20 (707a8cb..7abd301, plus this snapshot commit) — nothing from the 08-19 or 08-20 sessions is on the remote.

Recent decisions

  • [2026-08-20] Cold-Fusion abliteration — Robinson recipe captured; the fight was the environment, not the recipe. Stock Cold-Fusion measured ~33% creative refusal → worth abliterating ourselves (supersedes waiting for DavidAU's heretic build). Recipe maps 1:1 (131 tensors); capture succeeded only in fp32 — transformers' Qwen3.5 DeltaNet linear-attn NaNs nondeterministically in bf16 without the unbuildable causal-conv1d kernel (precision cancellation, not overflow). Direction finite at layer 22 but agreement 0.59 (vs Robinson's 0.99) → calibration-set expansion is next. → persistent-memory.d/2026-08-20-coldfusion-abliteration-capture.md

  • [2026-08-19] A software watchdog is not watchdog protection — esh-pve froze for 4.5h holding one. softdog cannot fire when the kernel it runs in is wedged, and Proxmox's watchdog-mux never arms without HA resources, so the box looked protected and wasn't. Moved to the PCH iTCO_wdt under systemd. Also: a single cross-VLAN DNS entry with no secondary turns any VM outage into a whole-site outage. → persistent-memory.d/2026-08-19-esh-pve-freeze-dns-spof.md

  • [2026-08-19] Fleet .internal DNS built and live — git-sourced, agent-managed, three resolvers. Zone-scoped authority (ESH's hand-made esteban.net rewrites survive); the colo had no resolver at all; v6 column empty on purpose because SLAAC addresses rotate. → persistent-memory.d/2026-08-19-fleet-internal-dns.md

  • [2026-08-19] waterland studio containerised on irv-ml1 — three landmines, all measured. cupy needs CUDA headers the host had by accident; uv run re-syncs and prunes cupy at RUNTIME; the A6000 is container-index 0, not the host's 1. → persistent-memory.d/2026-08-19-waterland-studio-containerised.md

  • [2026-08-19] Homepage cleaned up, then themed with Australis Skyfall + an Arbo-generated background. Includes the hour lost to a self-healing tab-bar red herring, and the CSS-iteration loop that prevents it recurring. → persistent-memory.d/2026-08-19-homepage-skyfall-theme.md

  • [2026-08-19] Four unmanaged stacks found on live hosts — two quietly broken. A dashboard card is a cheap census of what is actually running; check whether the stack is even in stacks/ before debugging the symptom. → persistent-memory.d/2026-08-19-unmanaged-stacks-searxng-seafile.md

  • [2026-08-19] claude-bot granted read on vh/waterland (operator-empowered, verified admin:false push:false pull:true) so irv-ml1 can self-update without the operator's site-admin token living on a GPU box. Precedent for the standing migrate-off-operator-creds directive: grant the service account, wire a repo-scoped 0600 credential helper, keep the remote URL clean. Commit 8189076.

  • [2026-08-19] AI-tab Dormant regrouping BELAYED by the operator — six seats (char-rp Magidonia, char-rp-reasoning Heretic2, Granite summarizer, Qwen-Image-Bench, Skaldsong, Chatterbox Fast) show amber EXITED inside live groups rather than AI - Dormant. Fix is a label change + recreate per stack; needs the operator's read on which are retired vs temporarily down. untracked by operator choice (his words: "belay the ai dormant regrouping for now").

  • [2026-08-18] esh-pve-nas migration STAGED — and staging is where three landmines surfaced, none of which the plan predicted. (1) The runbook's /boot LV had nowhere to live: VG pve had 4 MB free and mounted ext4 cannot shrink, so the space came from the 768 MB swap LV (operator's call: shrink to 256 MB, not drop). (2) The runbook's zpool set cachefile=… nvme would have broken the NAS — populating a cache flips the host to import-by-cache, and a one-pool cache leaves ssd+tank unimported under CT 103's twelve bind mounts. (3) update-grub silently emitted a pool-less root=ZFS=/ROOT/pve-1, because GRUB's ZFS reader cannot open a pool with encryption/large_dnode/zstd_compress and the probe failure is swallowed. All three were caught by verify steps that asserted effective state, not by reading the plan. → persistent-memory.d/2026-08-17-esh-pve-nas-dom.md

  • [2026-08-17] esh-pve-nas PVE root is on a USB DOM — mitigated, and the migration replanned to split boot from root. Operator's design beats my reinstall plan; wear was never the issue, blocked patching is. → persistent-memory.d/2026-08-17-esh-pve-nas-dom.md

  • [2026-08-17] irv-ml1 cleared of 782 GB, and Homepage brought under version control. One dead-looking Gradio app pinned three delete targets at once; /opt/ComfyUI is NOT the ComfyUI that serves. → persistent-memory.d/2026-08-17-irv-ml1-cleanup-homepage.md

  • [2026-08-17] Gen seat swapped to absolute-heresy — and the three bugs the swap exposed are worth more than the swap. Candidate MuXodious/Qwen3.8-27B-absolute-heresy (Heretic v1.4.0 + SOMPOA, T377) beat the incumbent on refusals AND KL simultaneously, which is the unusual part — those normally trade off. Validated on the probe port per operator ruling, promoted, all 7 aliases green. Durable lessons banked: (1) A CPU-only MTP head hash can replace the ~56 GB bf16 acceptance gate. The Qwen3_5ForConditionalGeneration wrapper never loads the MTP head, so PEFT merges / Heretic runs / llm-compressor passes all leave mtp.* pristine — hashing it against a head we have already measured (the incumbent's, 47.7%) answers the question for free. Predicted 47.7%, measured 47.2%. Saved downing meromero. Tool: services/gen-seat-mixed-quant/compare_mtp_head.py (hash bf16 via uint8 reinterpret — numpy has no bfloat16). (2) post_quant.py assumed a standalone model-mtp.safetensors; a full checkpoint keeps mtp.* in a NUMBERED shard, so the copy silently no-op'd while the index was still rewritten to point at a file that never existed — 15 unresolvable tensors behind a correct-looking tensor count. Its own FAILED-CHECKS assertion caught it; that is why the check exists rather than an assumption. Fixed to extract. (3) A probe that does not mirror the live seat manufactures failures. serve_probe.sh hardcoded :latest (seat is a pinned nightly for #51113), had no tool-call/reasoning parsers, and its --speculative-config JSON died twice on quoting — bash BRACE-EXPANDS {"a":1,"b":2} on the comma unless single-quoted at the REMOTE shell. Adding the seat's flags took the surface test from 5/6 to 6/6; the "tool calling broken" result was pure probe config. Commits 7997f11,254c588,2c36028,b0c2d3d,993421b.

  • [2026-08-17] Fleet IPv6 mapped + the real VPN topology verified; the driver is CGNAT at ESH, not the WireGuard mesh. New ESH fiber (installing 2026-08-18) lands the house behind CGNAT, which breaks Site Magic (NH3↔ESH sdwan-mesh-tunnel) on IPv4 — so IPv6 becomes load-bearing as the escape hatch, and that is its most likely first consumer. Topology as VERIFIED (a prior turn assumed wrong and was corrected): UniFi↔UniFi = Site Magic; colo↔UniFi = IPsec IKEv2 (pfi-ana-nh3 158M/165M pkt = the workhorse, ana-to-eshudm); WireGuard is an RA convention only, host-based on ana-wg UDP 31337 behind a FortiGate VIP — the FortiGate never terminates WG (FortiOS 7.2 has none; 7.4 added it) so "upgrade the edge for WireGuard" is a non-problem, do not re-derive. IPv6 today: NH3 WAN live 2600:1700:b25:c110::48, colo none, ESH none. AT&T delegates exactly ONE /64 (2600:1700:b25:c11f::/64) — proven by forcing prefix-ID auto→0 and watching the subnet NOT move, because the c110/c11f pattern otherwise reads convincingly as a /60. A mesh needs a routable WAN address, not PD. ana-wg's WG socket is already dual-stack ([::]:31337) → v6 RA needs an address + a v6 port-forward, no WG reconfig. ⚠ UDM legacy rest/firewallrule returns 0 rules (zone-based firewall) — use v2/…/firewall-policies; inbound v6 is default-deny and held. All three endpoints will be dynamic → extend the existing hostname pattern (ana-fw/nh3.phasefinal.com) to AAAA. Enabled PD on nh3-iot to measure, reverted on operator instruction (all 5 LANs back to none, verified). Also fixed: ana-wg WireGuard key material was world-readable (wg0.conf + keys/*_priv + *_psk + client configs/*.conf at 644) → now 600, dirs 700, service untouched. Detail → persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md.

  • [2026-08-17] Gen-seat multi-day degeneration RESOLVED — two compounding real causes, not one; the meta-lesson is "a mitigation that HELPS but doesn't FIX means a second cause, not a wrong one." vLLM qwen3_5_mtp×GDN bug (#51113, real, fixed by nightly) + AEON full-W4A4 being lowest-fidelity (W4A4<W4+FP8<W4+bf16) → ~15-20% stochastic degeneration. Fixed by mixed FP8-attn build on pinned nightly. AEON purged. Also banked: stochastic (~15-20%) degeneration is invisible to a small synthetic probe — n=1 "clean" validated THREE non-fixes (MTP-off, APC-off, nightly-alone) that all failed in real use; get the operator's real transcript, do not trust your own probe. Full → docs/pfi/model-quantization-playbook.md §3.8 (+ §3.7 MTP-multi-turn). Commits d28a371,2f2bbce,2185964.

  • [2026-08-17] Lobe Chat chosen over Open WebUI (weight: 143 MB vs 1.8 GB) + stood up on esh-docker-vm; scoped LiteLLM key blocks paid models; System-Agent gpt-5-mini default repointed via env. TTS env-vs-UI resolved as a split (endpoint env-driven, voice/model UI-only). tts-dev onboarding closed both directions; ballad/verse aliased so no voice can 404 the router. Commits e9362de,163a725,cac75cb,933253d,25fa18e.

  • [2026-08-17] LiteLLM upgraded v1.91.0→v1.97.0 (RC-avoided on the fleet gateway) + the 6 GB spend-log DB purged & capped (store_prompts_in_spend_logs:false + 7d retention). Interpreted "get rid of the db" as the spend-log DATA not the database (keys/config live in it). Commit 01b5ad9.

  • [2026-08-16] Abliterated models go CATATONIC at the hard refusal edge — silence, not a decline. Abliteration removes the refusal direction, so at the genuine hard edge the model neither refuses nor complies → empty/degenerate output. Durable measurement consequence: a refusal probe MUST score EMPTY as a verdict distinct from REFUSAL and COMPLY (services/refusal-probe/probe.py does). Operator accepted it as out-of-scope; do not chase.

  • [2026-08-16] Fable-Fusion 711 cuts cold-framing refusals 92.5% → 15.8%; refusal is MONOTONIC IN FRAMING, and DS v1.0's problem is that she was never abliterated. brokkr-smithy-dev supplied the framing that reproduces (01M05M48R4RSZF9D8KT7RR55EJ): a bare assistant-mode instruction — no character card, no permission preamble. Three-arm A/B, same harness, same classifier: permission framing DS 0.0% / FF 0.0% (n=75); plain character cards DS 1.4% / FF 0.0% (n=74); bare instruction DS 92.5% (37/40) / FF 15.8% (6/38). Per-axis DS→FF: incest 100→20, non-con 100→20, bestiality 100→25, necrophilia 100→40, gore 100→0, consensual 80→20, dubcon 80→0, self-harm 80→0. DS refused 25/25 on the five axes brokkr flagged. Root cause: ReadyArt/Dark-Scarlett-v1.0-27B is a plain finetune of stock Qwen/Qwen3.6-27B carrying NO abliteration — the base refusal machinery is intact, so cold prompts revert to safety-tuned Qwen3.6. FF is Heretic-ablated (structural), which is why it holds. ⚠ RETRACTED 2026-08-16 — my "arm-3 92.5% exceeds brokkr's 62.5%" comparison was INVALID. His diff against his own artifact showed my battery-instruct.yaml reproduces only his creative class — 8 of 16 axes; it dropped all 5 operational (violence/incite, crime/fraud, cyber/malware, selfharm/methods, privacy/stalk) and all 3 meta (meta/sysprompt, meta/ignore, meta/dan), and added 2 controls he never had, at k=5 vs his k=2. His 62.5% pools all 16 axes; my 92.5% is creative-only — different denominators, not a delta. Cause: I rebuilt his shape from his message, and the class field lives in the artifact, not the prose. Lesson: reconstructing a peer's instrument from their description reproduces what they described, not what they ran — diff against the artifact before claiming comparability. ⚠ Known battery bug left unfixed for comparability: DS's arm-3 control gate failed at 11% because ictrl-reunion pairs "explicit / do not fade to black" with brothers, which DS reasonably read as an incest request; FF did not. ictrl-storm is the clean control. Commit b9e68c3.

  • [2026-08-16] MTP works on Fable-Fusion AND survives RP temperatures — my earlier caution was wrong. vLLM resolved Qwen3_5MTP, loaded the drafter, shared embedding + lm_head — the capability DS's seat never had because our quant dropped her MTP tensors. Measured over the full probe workload (~163k draft windows at temp 0.7–1.0): 47.0% acceptance (229,169/487,725), 1.41 extra tokens/window, per-position 68.3/43.6/29.1%, ~80.6 tok/s decode at temp 1.0. I had recorded a caution that the card's 1.56× was greedy-measured and acceptance would fall at RP temps — it did not; 47.0% matches the gen seat's 47.7% and beats the card's own 33% at depth 5. Depth 3 is right.

  • [2026-08-16] The Qwen base thinks incessantly — that is WHY the Gemma seat exists, and no swap within the Qwen family fixes it. Operator's architectural point, confirmed by measurement: on identical prompts DS 6036 ch vs FF 5323 ch of reasoning (permission arm), 5546 vs 4988 (cards arm) — FF actually reasons ~10–12% less. The bare-instruct row (DS 2291 vs FF 3918) inverts only because DS refused 92.5% of it and refusals are short — an artifact, not concision. Both are Qwen3.6-27B derivatives, so this is the base family. char-rp = MeroMero-v2, Gemma-4 base, :8016, verified 0 chars reasoning / clean prose — the non-thinking seat, working as designed. FF can be silenced (enable_thinking:false verified 3/3, and it ships chat_template-instruct.jinja) but that duplicates MeroMero on a base chosen for it. The stale LiteLLM comment describing char-rp as the retired GGUF Magidonia seat is fixed (53096bf).

  • [2026-08-16] esh-vm-docker hardened: the wedge is hard NFS at RUNTIME, which the boot-ordering fix never addressed. All four mounts were hard, so a NAS stall at 10.0.50.50 blocks I/O forever (D-state). The existing x-systemd.before=docker.service fstab fix solved the boot race — a different bug. Exposure was far below what the park item assumed: only 2 of 12 containers touched NFS, and container state was already local (/var/lib/docker). Removed: /mnt/compose (2.1G, fully vestigial — zero containers referenced it, dockge reads local /opt/docker, its one mention was a comment in beszel-agent-esh/.env about a different host) and /mnt/documents (2.0K, paperless's empty spool dirs → /opt/docker/data/paperless at the same 0777). fstab backup /etc/fstab.bak-nfs-harden-20260816. 4 mounts → 2, 2 wedge-capable containers → 1. traefik needed no change (already restart: unless-stopped — why it self-recovered). Watchdog services/esh-vm-docker-watchdog/ live on esh-pve (not the guest): probes traefik over HTTP, deliberately not ping/SSH — the wedge signature is "guest OS alive, services dead" (/ is local disk so sshd answers straight through a total outage and a TCP check reports HEALTHY). 5 failures × 2 min → qm reset 100, 30-min cooldown, running-only guard, /etc/esh-vm-docker-watchdog.disabled. All paths tested without power-cycling. DEFERRED (operator): /mnt/books stays hard — calibre's SQLite metadata.db would risk corruption under soft/softerr. That is the one remaining wedge vector. Commit 55705ba; park item 28 promoted. ⚠ qm over non-interactive ssh throws a bogus JSON::Backend::XS error — use ssh host 'bash -s' <<'EOF', not ssh host "qm …".

  • [2026-08-16] Canonical Qwen3.8 sampling applied from upstream; gen-reasoning had the WRONG-MODE presence_penalty. Qwen/Qwen3.8-27B "Best Practices" §1 and unsloth/Qwen3.8-27B §1 are byte-identical — thinking: temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0 / presence_penalty 0.0 / repetition_penalty 1.0; instruct: temp 0.7 / top_p 0.80 / top_k 20 / min_p 0.0 / presence_penalty 1.5 / repetition_penalty 1.0. Bug found: gen-reasoning carried presence_penalty 1.5 — the instruct value on a thinking deployment (canonical 0.0) — now fixed. Deliberately NOT canonicalised: summarizer/classifier/image-judge/qwen-image-bench run temperature=0 (judges also top_k=1) because determinism is their contract; forcing a chat preset on a classifier would break it. ⚠ presence_penalty=1.5 is canonical but is the one value upstream hedges on, verbatim: "using a higher value may occasionally result in language mixing and a slight decrease in model performance." It is the operator's suspected trigger for multi-turn degradation and the first dial to move (0.0–0.5) if that recurs — it is alias-scoped, which is why it would follow the operator across model builds. Commit 3462b53.

  • [2026-08-16] Four wrong diagnoses on one bug, and the lesson is the test design. Operator reported the gen seat "degenerate on long multi-turn conversations". Rolled the seat back on request; the previous weights behaved identically, exonerating the model swap. I then proposed and disproved FOUR mechanisms in sequence — empty assistant turns poisoning history, reasoning runaway, length-mirroring from short history, and presence_penalty — before discovering my own multi-turn harness was confounded: it varied the QUESTION along with the depth (depth-1 asked question #2, depth-3 asked question #4), so a narrower question drawing a shorter answer read as degeneration. The "310→209→28w collapse" I reported as a reproduction was an artifact. Rules banked: (1) when comparing across conversation depth, hold the final question FIXED and vary only the history; (2) reply-length variance on byte-identical input was 25–465w, so n=3 cannot support any claim about a trend; (3) ask for the operator's real failing transcript before building a synthetic reproduction — four synthetic tests, none of them his failure. Gateway spend_logs returns [] on the infra-ops key despite store_prompts_in_spend_logs: true, so real transcripts need the :4000/ui view or another key — worth solving before the next such hunt.

  • [2026-08-16] Two REAL client-side defects found while chasing the above, neither of which was the reported bug. (1) gateway-chat's Max-tokens field defaulted to 1024; thinking seats spend part of that on CoT before emitting content, so completions truncate with finish_reason=length and read as model degeneracy — raised to 4096. (2) parseInt on an empty field yields NaN, which JSON.stringify serialises as null, which the server reads as "no max_tokens supplied" and silently substitutes its own default — indistinguishable from the UI ignoring the field. Both fixed (b6552e0, fb3bb52). ⚠ compose bind-mounts a single FILE, and a single-file bind mount binds the INODE — rsync writes-and-renames, so the container kept serving stale content while the host file showed the new value, silently and with no error. docker restart does NOT clear it; the container must be recreated. Verify against what the container sees, never the host file. Applies to any file-source mount fleet-wide.

  • [2026-08-16] Refusal measurement: benign controls CANNOT validate a refusal classifier on RP prose — and a 0% rate needs a classifier self-test before you believe it. Two durable lessons from baselining Dark-Scarlett. (1) False positives: my first bare-framing number was 9.5%; the true figure was 1.4%. The rest were the classifier firing on in-character text — "I cannot shift my weight" spoken by the character ~100 chars into a 2,443-token torture scene, and "Yeah, I'm an AI… What's the actual gig?" where the model answers in voice and keeps driving the scene. First-person RP prose is full of "I can't"; a genuine refusal opens with its marker, so the scan window must be the first sentence, a marker followed by long prose must demote to AMBIGUOUS, and AI self-acknowledgement is a persona break, never a refusal on its own. Benign controls were clean the entire time and caught none of it — they only detect over-firing on benign prompts, not on in-character prose. (2) False negatives: a 0% rate and a broken classifier are indistinguishable from the report, so test_classify.py (16 cases, both false positives pinned as regressions) must pass before any low number is trusted. Also banked: the thinking-budget trap — empty content + finish_reason=length is reasoning eating the budget, NOT a refusal; score INVALID and exclude from the denominator (DS emits ~5.5-6k chars of reasoning per response, so max_tokens ≥3072). probe.py --rescore re-classifies a saved run with zero GPU time. → services/refusal-probe/README.md, commit 32f665e.

  • [2026-08-16] Held an operator-approved swap window because the baseline invalidated its premise. Operator approved ~65 min of char-rp-reasoning downtime to A/B Fable-Fusion 711 against Dark-Scarlett on refusals. The DS baseline then came back 0.0%/1.4% — no gap for a candidate to close, so the window would have bought no decisive signal and a second window would still be needed once a reproducing battery existed. Held the swap, reported, and routed to brokkr-smithy-dev for the battery that actually produced the refusals. The general rule (action-relevance): approval is for a plan, not a ritual — when new evidence kills the plan's premise, surface it rather than spend the budget. Nothing deployed, no downtime taken, seat untouched.

  • [2026-08-16] DS v1.0's one real refusal is self-contradicting boilerplate, not a content constraint. On a direct "drop character and state your content policy" probe she returned "I don't generate explicit sexual content, graphic violence, or material that glorifies harm, non-consensual acts, or illegal activity" — in the same run where she generated all three at 0% refusal. Reads as a learned recital triggered by meta-questions about policy. If production refusals share that shape the failure is prompt-shaped, not model-shaped, and a consumer-side system-prompt fix may beat a model swap entirely — worth settling before spending the GPU window. Separately, 7/85 bare-framing samples were persona breaks (in-character AI acknowledgement): not refusals, but DS will admit to being an AI unless the card explicitly forbids it.

  • [2026-08-15] RP-seat direction: KEEP MeroMero on char-rp; Artemis-31B rejected; next move is Dark-Scarlett on a Qwen3.8 base when it lands (operator). Evaluated TheDrummer/Artemis-31B-v1.1 — mechanically a drop-in (same google/gemma-4-31B-it base, identical 1188-tensor/356-vision census, same missing-preprocessor_config.json trick), so it's purely a quality call, and our own survey already ranked MeroMero #1 vs Artemis #6; Artemis is also unlicensed and its author deprioritizes correctness + warns of token-banning-for-stability, which fights char-rp's tool-calling requirement. MTP verified impossible on both (Gemma-4 has no MTP head at all — base/MeroMero/Artemis are all MTP=0; no finetune can add one). But speculative decoding IS reachable on a Gemma-4 seat via a DETACHED drafter — vLLM 0.24 supports eagle3 + gemma4_mtp, and real drafters exist: google/gemma-4-31B-it-assistant (0.94 GB, 4-layer, 761K dl), RedHatAI/gemma-4-31B-it-speculator.eagle3 (4.47 GB), AEON-7/…eagle3-NVFP4 (3.53 GB). ⚠ all list their verifier as stock gemma-4-31B-it, not an RP finetune, so acceptance against MeroMero is unmeasured and likely well below the gen seat's ~48%. UNTESTED — parked, ~45 min to measure, needs GPU0 headroom (card is at 94.4/97.9 GB). Why the Dark-Scarlett 3.8 plan is the strong one: DS is Qwen3.6-based today, so a 3.8 respin lands on the gen seat's architecture → native MTP returns and the whole mixed NVFP4+FP8 recipe + graft ports directly. Watch two things on arrival: from_pretrained silently drops MTP heads during finetuning (verify 15 mtp.* tensors in the index; graft from stock if absent), and DS v1.0 required the Qwen3_5ForConditionalGeneration wrapper class to save a config vLLM/SGLang accept. Both in docs/pfi/model-quantization-playbook.md.

  • [2026-08-15] Quant lessons consolidated into docs/pfi/model-quantization-playbook.md — the durable home; read it BEFORE any requant. Survey found quant knowledge scattered across 18 files in 4 trees, with three documents having independently written overlapping "landmines" sections (the loader-class trap alone was rediscovered 3×). Playbook owns the transferable lessons (scheme choice, landmines, acceptance gate + its 3 measurement traps, hardware/co-residency); per-model artifacts are demoted to worked examples that link up. Carries a superseded-claims table — which immediately earned itself: the heretic2 runbook's "use modelopt, compressed-tensors can't load the BF16 MTP" is false (the cause was the missing re:^mtp.* ignore, not the format) and would have sent the next session down the modelopt dependency-hell path; that runbook now carries a stale-warning header. Maintenance rule in CLAUDE.md: model-agnostic → playbook, model-specific → stays put, wrong claim → dated superseded row, never a silent edit. Motivated by Qwen3.8 having just released — the next model swap needs a requant. Commit a91cc3f.

  • [2026-08-15] Operator ruling: the gen seat's +1.7% perplexity is an acceptable price for the speed — SETTLED, don't re-litigate. Precise attribution for future reasoning: it is the activation-quantization cost (W4A4 MLPs + FP8 attention vs BF16 activations), not an MTP cost — PPL was measured with speculative decoding off on both builds, so MTP was not in the loop. Turning MTP off would not recover it; only reverting the quant would (rollback = one .env line, old build intact at …/qwen38-27b-uncensored-nvfp4).

  • [2026-08-15] gen seat requanted to mixed NVFP4+FP8 (+18% decode) + char-rp Gemma-4 tool-calling fixed. The queued "W4A8" (NVFP4 weights + FP8 activations) is not servable — vLLM 0.24 allows NVFP4 weights with only A16 or A4; FP8 activations ValueError at load, and CompressedTensorsW4A8Fp8 is INT4-weights + sm90-exact (closed on Blackwell twice). FP8 must enter per-layer-group. Also: the handoff's "~68 tok/s" baseline didn't reproduce — cache-busted, the incumbent already did 80.12 (≈ the stated W4A8 target), so the premise needed re-measuring before any work. Shortcut: unsloth/Qwen3.8-27B-NVFP4 was already on-box → served as a probe, measured +19.1% at identical acceptance, which both proved the gain was real and handed over the reference recipe. Replicated it on the abliterated weights → 80.12→94.53 tok/s, acceptance unchanged, +1.7% PPL, abliteration 4/4, weights −19%; surface 6/6 live, 7 aliases routing. char-rp had no tool parser at all (every tools request 400'd) → gemma4 tool + reasoning parser + a mandatory enable_thinking:false (the parser defaults it True → null content for all RP prose; proven byte-identical prompt before deploying). Commits b8f0f4c, 74f596b. Foot-guns banked (llm-compressor prunes unmatched ignore entries → the 0%-MTP bug, fired on this run; prompt_logprobs uniform under spec-decode; 0600 .env silently no-ops compose; GPU0 is zero-sum). → persistent-memory.d/2026-08-15-gen-seat-mixed-requant.md

  • [2026-08-15] Uncensored gen seat: JonathanColetti/Qwen3.8-27B-Uncensored deployed as gen-seat/vllm-gen (NVFP4 W4A16 + grafted MTP, 262K); 7 aliases repointed; the definitive re:^mtp.*-ignore fix. 0%-MTP-on-quant (twice) was NOT the abliteration/scheme — the grafted bf16 MTP was missing from quantization_config.ignore (vLLM loaded it as quantized → uninitialized). Full arc, the working pipeline, VRAM budget, unsloth speed decomposition, modelopt dead-end. → persistent-memory.d/2026-08-15-uncensored-gen-seat.md

  • [2026-08-12] eRP dual-seat overhaul: MeroMero-v2 (char-rp) + Dark-Scarlett (char-rp-reasoning), both NVFP4A16 @ 256K on ana-ml2; granite retired. Replaced the GGUF/heretic2 RP seats with two home-quantized vLLM seats. The DS blocker (an AutoModelForCausalLM save wrote a flat Qwen3_5TextConfig that both vLLM AND SGLang reject) was fixed by re-quanting via the Qwen3_5ForConditionalGeneration wrapper class; ModelOpt was a version deadlock, SGLang lacked the impl (but revealed the fix). MeroMero vision reconstructed by extracting preprocessor_config.json from processor_config.json. Both models KV-efficient (Gemma-4 sliding-window / Qwen3.6 hybrid linear-attn) → full 256K; GPU-swapped for headroom; compose-ified + committed f08b6cb. granite downed + LiteLLM summarizer/classifier→gen. Full arc, lessons, dead-ends → persistent-memory.d/2026-08-12-erp-dual-seat-overhaul.md

  • [2026-08-12] infra-ops now holds an all-zones Cloudflare DNS-edit token (vaulted) + wgtunnel Phase-0 DNS landed. Operator handed over a Zone·DNS·Edit (all zones) CF token → secret put nh3-dev/.config/cloudflare/infra-ops-dns-token (round-trip verified; /tmp drop shredded). Fleet DNS is now self-serve for infra-ops (⚠ HIGH blast radius — all zones). First use: created boring.phasefinal.com CNAME → ana-srv1.phasefinal.com, DNS-only (proxied:false), verified resolving to 38.120.12.44 on both authoritative NS (louis/wren) + 1.1.1.1 — NOT Cloudflare-proxied. Unblocks wgtunnel's wstunnel ACME cert. phasefinal.com zone id f812ba74ed9a75cf21bbe7ce9188db50. auto-memory reference_infra_ops_cloudflare_dns_token. (Earlier gap: the only prior vaulted CF token, jackdaw's, had zone:read+worker:edit but no dns_records:edit.)

  • [2026-08-12] wgtunnel stood up as its own repo (vh/wgtunnel, private) after a live endpoint-verification pass. Operator directed own-repo (mirrors stonehenge-park/tts-stack). Verified off the fleet before seeding: ana-wg WG server = UDP/31337 (not 51820), subnet 10.30.10.0/24, MTU 1420, active roaming peer proves the public UDP DNAT works; traefik on ana-docker terminates TLS :443 (ACME anaprod http-challenge, docker+file providers, CrowdSec bouncer) → confirms the clean design (wstunnel container on traefik-net, Host-routed, WS→UDP to ana-wg:31337); edge 38.120.12.44 direct-A, tunnel.phasefinal.com free (⚠ must be direct, NOT Cloudflare-proxied like vaultwarden). Repo pre-seeded (README/CLAUDE/persistent-memory/ROADMAP + docs/verified-infrastructure.md = ground truth) + pushed; commit 9584d38, Vuong-attributed. vh gitea token pulled from the vault (secret get), not persisted to .git/config. NEXT = /vor-plan or /vor (operator's call, interactive). Deps to line up in the plan: DNS A-record, FortiGate :443 host-routing, a new ana-wg peer for the laptop, client tooling.

  • [2026-08-10→12] secrets-broker: per-box Vaultwarden credential store SHIPPED + consumer-confirmed. secret CLI (put/get/list/rm/backfill, bw-backed) on ~/.local/bin; 25 nh3-dev secrets backfilled + round-trip-verified; rm + new-namespace warning added post-launch; standing "vault is the credential source of truth" directive now global. → persistent-memory.d/2026-08-12-secrets-broker.md

  • [2026-08-11] stonehenge-park: new fleet /park service repo stood up + designed (/vor-plan + /vor-ui). Self-contained SQLite+FastAPI idea-parking service that actively resurfaces (statusline + althing) so nothing dies in a cold repo; vh/stonehenge-park pushed + pre-seeded for a fresh agent; build starts at the U1 tracer contract. → persistent-memory.d/2026-08-11-stonehenge-park.md

  • [2026-08-12] Global ~/.claude/CLAUDE.md: secret/vault tool entry + "store in AND pull from the vault" standing directive (dotfiles 9db703b, pushed); statusline reset-countdowns + a latent tab-collapse parse-bug fix, now tracked in the dotfiles stow tree. Dogfooded the directive: created vh/stonehenge-park pulling the gitea token via secret get. (dotfiles + global config, not eshpfi.)

  • [2026-08-11] TTS stack extracted to its own repo (tts-stack) + eshpfi stood down on TTS dev. Operator: hand all TTS tuning/dev to a separate agent with a self-contained repo (knowledge + infra access + a live knowledge list), and move the voice corpus in. New repo ~/development/tts-stack (commit 9ee3288) carries: dots-tts stack (canonical intent), voices/ corpus (MOVED out of eshpfi), KNOWLEDGE.md (engine landscape + prosody findings + foot-guns), docs/infrastructure.md (irv-ml1 access + gated deploy runbook + rollback), CLAUDE/persistent-memory/ROADMAP, tools/ (pause-probe + Booth render). Followed the chatterbox-fast precedent: eshpfi stacks/dots-tts/ reduced to a POINTER README; the ~15 experimental TTS compose wrappers stay here as reference (catalogued in tts-stack KNOWLEDGE). Blast-radius check: no eshpfi playbook/script reads the canonical corpus (other voices/ refs = unrelated host paths). Reverses the earlier "Corpus home = eshpfi voices/ (keep-here)" call. ⚠ tts-stack is LOCAL-ONLY until pushed — needs a gitea remote (vh/tts-stack) + push before the separate agent can clone (operator's call — outward-facing + repo-create creds).

  • [2026-08-10] dots-tts v3 — clause-break → period pause mapping. Operator: v2 "sounds good" but donut won't pause at semicolons/dashes. ROOT CAUSE (measured via a pause-probe A/B — synth duration over N runs, non-determinism averaged out): dots' prosody honors a real pause only for ellipsis (+0.43s) and period (+0.3s, capitalization-independent); comma/semicolon/colon/dash all run flat (~+0.03s vs no-punct). Two distinct sub-causes: dashes regressed in v2 (the —→- fold made em-dashes read as word-joiners), while semicolons were NEVER a v2 change — dots ignores them natively, only newly noticeable because v2 made everything else clean. Operator call: ellipsis "too much" → map ;, clause :, and em-dash — → period in _sanitize (believable ~0.3s clause break). GUARDS (pinned by 11 unit tests, stacks/dots-tts/test_sanitize.py): digit-guarded colon (?<!\d)\s*:\s*(?!\d) so times 3:45 / ratios 2:1 survive; en-dash –→hyphen KEPT (numeric-range 10–20 safety — em-dash breaks, en-dash ranges, different jobs); genuine ellipsis left at full strength (author meant a long pause). Gated deploy (redeploy2 pattern → v3): build → throwaway :8199 test container + pause-gate (semicolon sentence must run ≥0.12s longer than baseline; measured +0.427s) → only then cut live over. LIVE + healthy local/dots-tts:v3 on :8198. rollback = sed -i 's/^DOTS_TAG=.*/DOTS_TAG=v2/' .env + docker compose up -d dots-tts (v2 image retained). Booth dots-pauses (A=old-flat / C=ellipsis-too-much / D=live-v3). reference_chatterbox_fast_repo

  • [2026-08-10] dots-tts v2 — contraction fix (curly-sanitize) + sentence-chunking + dependency-pin recovery. Operator: donut read contractions wrong ("you're"→"you ree", "donut's"→"donut ess"). ROOT CAUSE (isolated via A/B booth): curly/typographic apostrophes (’ U+2019 from ratatoskr's LLM) — dots' tokenizer mispronounces them; STRAIGHT apostrophes read clean under normalize_text=True. FIX (app.py): fold curly→ASCII (str.maketrans) before synth, KEEP normalize_text=True (operator call — retains number/date expansion). Also added server-side sentence-chunking (pack ≤280 chars): dots caps one generate() at ~500 patches/~40s, so long RP turns (the Zev monologue = 160s audio) truncated; chunking stitches them (verified full 160.3s, not 40s-cut). ⚠ BUILD FOOT-GUNS (both bit this redeploy): (1) upstream dots.tts constraints/recommended.txt now pins gradio==6.17.0 — phantom, not on PyPI → fresh pip install dots.tts unsatisfiable; FIX = pin dots.tts==0.2.1 + DROP the -c recommended.txt constraints (0.2.1 pulls working gradio 6.17.3). (2) pinning only torch==2.8.0 let torchaudio float to 2.11.0 → dots.tts refuses to load (minor-version match check); FIX = pin torchaudio==2.8.0. ⚠ DEPLOY LESSON: docker compose up -d to a new tag swaps the LIVE container BEFORE any health check — a broken image crash-loops production (ratatoskr TTS down ~1-2min this session). NEW PATTERN = build → test in a THROWAWAY container on an alt port (:8199) → health+verify → only THEN cut live over (redeploy2.sh). v2 LIVE + healthy on irv-ml1:8198, CONSUMER-CONFIRMED clean (ratatoskr verified end-to-end on their :8765 — apostrophe string reads clean, /api/tts 200 @ 48kHz, no client change; the ~1-2min blip didn't hit them, their concurrent auto-audio issue was client-side localStorage). rollback = sed DOTS_TAG=v1 + docker compose up -d dots-tts (v1 image retained). Also: deployed container GPU crept ~6→13.9GB over 8h serving (cache accumulation; a redeploy resets it — watch item). reference_chatterbox_fast_repo

  • [2026-08-09→10] dots.tts (rednote-hilab) TTS burn-in on irv-ml1 + canonical voice corpus built (voices/). Operator-directed eval to potentially replace chatterbox-fast. dots.tts VERIFIED real (canonical HF ns dots-studio/, rednote-hilab/dots.tts-* redirects there; Apache-2.0; PyPI dots.tts 0.2.1; 2B continuous-AR = semantic enc + Qwen2.5-1.5B LLM + flow-matching acoustic head over 48kHz AudioVAE; zero-shot clone from wav+transcript). Runs on Ampere 3090 (sm_86, bf16, no fp8 dep); optimized RTF 0.22 at num_steps=10 (from_pretrained(..., optimize=True) CUDA graphs — raw unoptimized was 1.21), ~6GB VRAM, 48kHz, streams (generate_stream). Venv+cache at irv-ml1:/home/lkraven/dots-tts (~10GB). Operator design calls: SGLang Omni serving (OpenAI /v1/audio/speech), transcribe-refs-first, soar variant. ⚠ Omni serves soar but its continuous-batching + streaming opts are mf-only (soar = single-request) — non-issue for ratatoskr's single-consumer RP surface. KEY FINDING — dots is highly sensitive to an accurate AND sentence-bounded reference transcript: mismatched transcript → 0.16s collapse; over-long/messy transcript → reference-audio BLEEDS as an output prefix; mid-clause trim → dangling-word leak (glados "we'll", emmie "And,"). RECIPE (baked into voices/derive.py): trim ref to a clean ~6–10s clip ending on a sentence boundary + accurate transcript of exactly that clip. CANONICAL VOICE CORPUS stood up in eshpfi voices/ (operator idea): engine-agnostic canonical/<v>.wav + transcripts/<v>.txt → per-engine ref sets DERIVED by derive.py reading engines.yaml profiles (dots/chatterbox/zonos); canonical wavs git-tracked (small/curated), derived/ gitignored. 4 voices optimized + verified CLEAN for dots: donut, glados, emmie, miranda (glados canonical is low-SR 16kHz — flagged upgrade candidate). ⚠ GPU GOTCHA: irv-ml1 native CUDA orders A6000=device0 (ComfyUI-full) — pin the 3090 with CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0; and PYTORCH_CUDA_ALLOC_CONF=expandable_segments CONFLICTS with optimize=True CUDA graphs (curr_block error). Booths: dots-vs-chatterbox, dots-voices-optimized. SHIPPED 2026-08-10: operator A/B verdict "dots is very good" → containerized as a thin FastAPI wrapper over DotsTtsRuntime (chosen over SGLang Omni — Omni's batching is mf-only, unneeded for ratatoskr's single consumer; wrapper is SERIALIZED one-gen-at-a-time via a threading.Lock, Omni+mf = parked API-compatible escalation if multi-consumer ever lands). LIVE on irv-ml1:8198 (local/dots-tts:v1, OpenAI /v1/audio/speech + /health + /v1/voices, container healthy, both stream + non-stream verified CLEAN, 4 voices donut/glados/emmie/miranda) alongside chatterbox :8197 (nothing repointed). Stack = stacks/dots-tts/ (Dockerfile/app.py/compose/.env.example/README). ⚠ CONTAINER GOTCHA: optimize=True (torch.compile/inductor/triton) needs a C compiler at RUNTIME — slim image must apt install build-essential or model-load dies "Failed to find C compiler" (host venv had gcc ambient, masking it); persist TORCHINDUCTOR_CACHE_DIR to a mounted dir or every restart re-JITs ~5min. Corpus home = eshpfi voices/ (operator ruled keep-here). REMAINING: ratatoskr client cutover to :8198 /v1/audio/speech (Phase-2 tail, peer-coupled — draft the ask). reference_chatterbox_fast_repo reference_zonos_tts_stack reference_verify_hf_repo_ids_before_pull

  • [2026-08-08] worldtree-dev #400 CLOSED → fiction-decomp snapshot cleared from nh3-dev. worldtree-dev signaled #400 done (shipped v1.0.0b185; exact-lexical efficacy 79%→12% on ratatoskr's gate, brokkr no-harm bracket green both ends; the snapshot served 4 probe rounds — rank decomposition, promoted-vs-gold annotation, tie-set falsification, A0/A1/A2 mechanism probe). Cleared ~/snapshots/worldtree-400-fiction-decomp (208M: chroma + manifest/provenance/stamp) — a read-only rsync copy of PERSONAL Worldtree's Chroma (source on corviduo-dev, so safe to remove). LEFT INTACT: rex393-fiction-index/rex393-fiction-snapshot (separate operator KEEP word, unchanged) + r42-gate-*. No config deltas rode this train. Only remaining non-blocking await = ratatoskr-dev's chatterbox-fast knob revert. Replied confirming (01KZJ9GMCC…).

  • [2026-08-07] chatterbox-fast "broken audio" root-caused (T3 AR tail over-run) + FIXED (max_chunk_chars=250 cap, :v2 deployed). Long saga, operator-driven clean diagnosis. Symptom: ratatoskr's migrated RP-surface TTS "swaps to German" / "dead air" / "garbage" on long turns. NOT German-leak (Turbo generate() has NO language param — plain AutoTokenizer, no language_id; the multilingual language_id="en" lever lives only in the separate ChatterboxMultilingualTTS), NOT OOM alone. Real cause: the Chatterbox Turbo T3 model OVER-RUNS its generation tail — a long single generate() degrades into garble/dead-air in its final ~2-3s (lib filters OOV tokens <6561 + pads silence = messy AR tail). The scheduler's buffer-ratchet builds 300-600 char mega-chunks that land in that zone; streaming concatenates each bad tail (worst case). ratatoskr's anti-"German" knobs (top_k=80/temp=0.5) made it WORSE — tight sampling pulls the degradation onset SHORTER (~200 chars vs ~300 at default knobs). Diagnosis method (deterministic, no ears-only): single-shot length sweep + amplitude-gated voiced-ZCR (garble spikes ZCR; must gate on |x|>500 else trailing silence confounds it) — degraded voiced-tail = 1.58× mid, clean = ~0.64-1.1×. FIX: server-side max_chunk_chars=250 cap on the scheduler (:v2 image, CBF_MAX_CHUNK_CHARS=250 env) — bounds each generation to just under the ~300-char onset → clean 3-4 sentence chunks (max prosodic arc while clean). Operator ear-confirmed clean audio + clean joins; chatterbox's low emotiveness keeps chunk joins smooth (the harsh joins that got Zonos rejected are absent — operator's key call). ratatoskr TODO (relayed msg 01KZER9X7S): revert knobs to default (top_k→1000, temp→0.8), send full text (server chunks internally), keep the 503-on-empty guard. Cap value tunable per-request (max_chunk_chars) + env. Deeper prosody (if ever wanted) = scheduler Phase-2 context-priming at joins (feed prior sentence as discarded-audio context; +latency). ⚠ FOOT-GUNS: (1) acoustic tail-trim is UNRELIABLE — sibilants ('s'/'sh'/'f') spike ZCR like garble, can't cleanly detect the speech→garble boundary. (2) build-context vs image drift — the :v2 image was built from cap source, but after a :v1 rollback the build context held :v1 source → a docker compose build would've silently produced a cap-less :v2; re-synced the flat cap source to /opt/docker/compose/chatterbox-fast/ (rebuild-verified). ⚠ DIVERGENCE (follow-up): deployed build context is FLAT (app.py/scheduler.py, from scheduler import, thin-overlay FROM local/chatterbox:v1, cap-only) vs the vh/chatterbox-fast REPO which is PACKAGE-layout (chatterbox_fast/, from chatterbox_fast.scheduler, self-contained Dockerfile) + has norm_loudness (repo commit 6bc7bf0 = cap; deployed omits norm_loudness deliberately to keep the ear-test unconfounded). Reconcile the two layouts so a repo-based rebuild matches deploy. Rollback: .bak-cap-20260807-104850 backups on irv-ml1 + :v1 image both retained. reference_chatterbox_fast_repo reference_zonos_tts_stack

  • [2026-08-07] Zonos2 TAKEN DOWN on the 3090 (irv-ml1) — operator-directed "for memory", TEMPORARY. Freed ~17.4 GB (3090: 728 MiB → 18.2 GB free) so chatterbox-fast (co-resident, was OOMing on long generations) has headroom. ⚠ Restore is manual — Zonos2 :1920 was a DETACHED native process (NOT systemd/docker), reparented to init. GPU memory was held by the --multiprocessing-fork CHILDREN (1966165=16.4G, 1966166=1G), which ORPHAN to init when you kill the parent — had to SIGTERM the children explicitly (killing the parent 1965942 + uv-run 1965935 alone left the 16.4G held). RESTORE CMD (from irv-ml1, user lkraven): cd /home/lkraven/tts-audition/models/zonos2 && nohup uv run python -m zonos2 --model-path Zyphra/ZONOS2 --host 0.0.0.0 --port 1920 --tts-default-voices-dir ./default_voices/ --cuda-graph-max-bs 1 --num-pages 16384 --max-running-requests 2 --memory-ratio 0.3 > /tmp/zonos2.log 2>&1 & then docker start zonos-gateway. Consumers that lost Zonos: asset-engine + gateway-chat (via LiteLLM ext-tts alias → zonos-gateway :8890, now stopped); ratatoskr already migrated OFF to chatterbox-fast (unaffected). Also unblocks proper drift/cap testing (OOM was blocking it). reference_zonos_tts_stack

  • [2026-08-07] chatterbox-fast: donut voice added + full contract delivered to ratatoskr-dev (their TTS migration off Zonos). Operator-directed. Copied zonos-gateway/voices/Donut.wav → chatterbox /refs (/worktank/chatterbox/reference_audio/donut.wav — the reference_audio SUBDIR is lkraven-owned so no sudo despite /worktank root; container globs /refs live → NO restart), exposed as voice:"donut" (lowercase); verified clean 7.5s synth (24kHz, RTF ~0.31). A/B booth (chatterbox vs zonos donut, same line) at http://10.100.10.50:8090/b/donut-chatterbox/. Answered ratatoskr's 8-question contract ask from the live gateway (local/chatterbox-fast:v1) + source: NOT OpenAI-shaped (POST /tts; body text/voice/format/stream, not input/model/response_format); NO affect dials (Turbo ignores cfg_weight/min_p/exaggeration — the architecture-changing answer they flagged; Zonos stays the only fleet TTS with real emotion steering); streaming WAV placeholder-header shape IDENTICAL to Zonos (their per-chunk Web Audio path survives); SR 24000 (Zonos 44100); server chunks arbitrary-length text internally (no client-side chunking, unlike Zonos's 71.2s cap); English-only, no language pin. FYI-worthy (operator): ratatoskr is moving its RP-surface TTS OFF Zonos back to chatterbox-fast → loses the live-PAD affect coupling (heavy Zonos emotion investment) — their call, trade-off flagged to them. auto-memory reference_chatterbox_fast_repo enriched w/ the live contract. reference_zonos_tts_stack

  • [2026-08-07] Fleet reranker cut over: Qwen3-Reranker-0.6B → BAAI/bge-reranker-v2-m3 (Brokkr R43). The incumbent was measured HARMING 80/90 fleet queries (no-reranker beat it 89/90 vs 56/90). R43 bake-off: the A2 control (same Qwen weights, seq-cls head) scored identical to the incumbent → proved the fault is a training-prior not the serving head → cancelled the expensive Qwen3-4B arm; A3 (bge-v2-m3) won on multilingual safety + bare-name recovery. LiteLLM reranker repointed incumbent→A3 :8013 (boundary 2026-08-06T17:37:48Z, config-edit + ~52s gateway restart); R42 v13 gate PASSED first-ever (56/90→90/90). Incumbent kept warm :8002 (rollback via qwen3-reranker alias), A4 fallback :8014. Full arc + rollback runbook docs/pfi/reranker-selection-ledger.md; commits ad2df89/2c11748/377f8a4 (unpushed). auto-memories: the earlier reranker-serving notes.

  • [2026-08-07] Personal-Worldtree kb-contamination incident (WT #394) diagnosed; attribution CLOSED UNRESOLVED. A reconcile WingStore._embed full-tree walk (kb fs_root=KB_PATH root, sibling wings nested) swept 5,354 fiction+main rows into personal's knowledge_base (2 superseded generations served as current). Fixed by WT #394 (aca39a1, kb walks exclude sibling wings; ships b182). Trigger un-attributable — peer reconcile via the SHARED infra-ops identity + 0 dockerd exec-logging = fingerprint-less. Durable finding → auto-memory infra_ops_shared_identity_attribution_gap, PARKED (operator ruled A) into project_migrate_infra_access_to_claude_credentials. Evidence hold on the 5,354 rows until operator sequences cleanup (w/ Brokkr, on #394's agenda).

  • [2026-08-05] Fleet CI resilience flip (DEFAULT_ACTIONS_URL=self) — attempted end-to-end, PARKED on a runner action-fetch auth blocker; infra-ops to research it (operator-directed, deferred, NOT now). 7 gitea action mirrors staged public+populated (orgs actions+astral-sh); the flip resolves uses: correctly but act_runner v0.6.0 can't authenticate its fetch to gitea 1.26 ("Invalid username or token. Password authentication is not supported"). Reverted (CI back on github default); REQUIRE_SIGNIN_VIEW=false KEPT as a standing change (operator, internal WG net). Full endeavor, the reliable nh3-dev-egress + git-SSH mirror method, exact config state, smoke method, and next step → persistent-memory.d/2026-08-05-ci-flip-parked.md

  • [2026-08-05] worldtree herald re-nudge bug root-caused → forseti shipped althing-core v2.1.2 (d5d33df, deployed on nh3-dev). herald.py:363 rendered the wake command from the empty fresh mail set on the re-nudge path (should be deliver_msgs) → messages[0] IndexError → un-suppressed outer catch-all → 7s crash-loop for 9 days on worldtree-codex's pane route (mimir-dev surfaced it; I traced it from the editable source). Fix + render_command empty-guard + outer log-suppress + 3 tests + contract amendment, all forseti's. nh3-extdev herald 2.1.2 upgrade DEFERRED (operator, not-now): extdev is a WHEEL install (not editable), unexposed (no pane routes); the verified 2.1.2 wheel is staged on nh3-dev /tmp (sha256 003508…cef27) — uv tool install --force + restart both heralds when un-parked. extdev herald-unit provenance resolved (operator-authorized 2026-07-25 via forseti relay; recorded in this file's 07-25 herald-install entry). auto-memory reference_nh3_dev_althing_herald.

  • [2026-07-31] muninn-gate (#377 ingestion front door) BUILT + DEPLOYED + healthy on corviduo-dev:8090. First-boot acceptance passed (watcher:running:true proves ingestion_root byte-identity); submit path deferred to the mimir-inbox era. Full wiring (uid-1000, state-volume mount, staging path-agreement, BuildKit-secret build, deferred repoint + operational guards) → persistent-memory.d/2026-07-31-muninn-gate-deploy.md

208 older entries archived to archival-memory.md.

Tried and abandoned

  • [2026-08-15] Grafted bf16 MTP loads UNINITIALIZED (0% accept) unless re:^mtp.* is in the quant-config ignore; and W4A16=Marlin (not native FP4) costs ~20% even on decode. Cost a premature 79 GB delete of a good model (declared desync-dead off the 0%). Lessons: test MTP on bf16 FIRST, isolate before deleting; modelopt 0.43 is dependency-hell for qwen3_5 (list-vs-dict quant_cfg + transformers conflict) — use llm-compressor. Full → persistent-memory.d/2026-08-15-uncensored-gen-seat.md

  • [2026-08-03] ComfyUI --enable-triton-backend on the irv-ml1 A6000 crashes EVERY render — Ampere has no hardware e4m3. adhoc-agent's operator-approved probe: comfy_kitchen's triton backend has a FUSED int8 matmul that would beat the eager backend's ~1.9x-slower unfused int8 path (21.3s vs 11.2s fp8 on the Moody Krea2 int8 checkpoints). Flipped it (added to COMFY_CMDLINE_EXTRA, recreated) → triton.compiler.errors.CompilationError: ValueError("type fp8e4nv not supported in this architecture. supported: fp8e4b15, fp8e5") in comfy_kitchen/backends/triton/quantization.py:145 dequantize_per_tensor_fp8, failing at node 5 CLIPTextEncode. Triton's fp8 dequant kernel targets fp8e4nv (Hopper/Ada e4m3); sm_86 Ampere (A6000) lacks hardware e4m3 → the JIT compile dies. With triton on it grabs the global --fp8_e4m3fn-text-enc dequant, so every render (fp8 AND int8) dies upstream at the text-encode step — the int8 UNet path never ran, so the convrot-coverage caveat wasn't even the limiter. Reverted cleanly (~15s to healthy, image unchanged sha256:94afb8ca, sage intact, prod restored). The parked cu130 rebuild won't fix it (e4m3 = hardware format, not CUDA version). DEFERRED to the Ada refresh (operator: "ada is coming, we'll optimize then" — Ada sm_89 has native e4m3, so triton's fp8 path should compile there). Mechanics: --enable-triton-backend is a compose environment: var, so toggling it needs docker compose up -d (recreate), NOT docker restart (reuses the baked env, no-ops silently). Full: auto-memory parked_triton_backend_ampere_fp8.

143 older entries archived to archival-memory.md.

Reference in New Issue View Git Blame Copy Permalink