Files

T

vh 40257247b0 memory: IPv6 plan settled — endpoints not internal numbering; ESH has a /56

Corrects three claims that had been standing in the fleet IPv6 notes and that
sent a three-arm research effort after a problem that did not exist:

- ESH was recorded as having no IPv6. It has a /56 delegated and a routable
  WAN GUA -- substantially more prefix than NH3's single /64.
- The mesh was recorded as broken by ESH's CGNAT. It is not and was not down;
  ESH is outbound and working. CGNAT on v4 alongside generous v6 is just the
  modern ISP pattern, not an outage.
- IPv6 was framed as the escape hatch for that outage. The actual plan is that
  IPv6 carries tunnel ENDPOINTS for Site Magic and WireGuard, and LANs are not
  numbered in v6 at all.

NH3 internal v6 was brought up on the delegated /64 and verified end-to-end
(global GUA on nh3-docker, zero loss to Cloudflare and Google v6, un-NATed
source address seen from outside), then reverted on operator direction: one
/64 lights exactly one VLAN and that is not worth the split-brain.

The AT&T prefix research is kept as reference rather than deleted -- the /60
is real but undelegatable, the living multi-prefix mechanism is multiple
IA_PD in one solicit rather than the VRRP/multi-MAC recipe we were handed,
and the UDM SE can express neither. That is the answer if NH3 LAN-side v6
ever earns a maintenance window; it is not on any critical path now.

2026-08-19 10:44:56 -07:00

103 KiB

Raw Blame History

Persistent memory — eshpfi-management

Last updated: 2026-08-19

Always check for /tmp/infra-ops-handoff.md — if it exists and its Written: stamp is under an hour old, read it (it carries the in-flight handoff from the previous session), then delete it. Older than an hour: stale — delete it unread.

Repo purpose

Reference workspace for PFI infrastructure: server inventory, canonical Docker Compose stacks, ops playbooks, and conventions. Authoritative copies of compose files live on the servers under /opt/docker/compose/<stack>/; this repo mirrors them for version control, editing, planning, and CI-driven deploys. It was originally spun up to handle the fleet backups — keep that lens when triaging backup/storage issues.

Tools and conventions

Sister repos (separate gitea repos, deployed by playbooks here):

Repo	Role	CI status
`vh/task-board`	MCP + web dashboard for assistant task state (port 7878)	push-to-main → CI deploys (2026-04-29)
`vh/vor`	Inquisitor UI sidecar (port 7879)	push-to-main → CI deploys (2026-04-29)
`vh/nevermore`	Twice-daily LLM-curated briefing (port 8181, replaces news-digest)	push-to-main → CI deploys (2026-04-30)
`vh/asset-engine`	Internal control plane over inference services (port 8200, LAN-direct)	push-to-main → CI deploys (2026-05-12)
`vh/althing`	Lean trusted inter-agent message bus — v2 "email model" (v2.0.0b2, 2026-07): per-box local-SQLite bus + courier/receiver for P2P over the 10.x net; pillars = open-loops / per-box herald + wake-listener / roaming owner API `/owner/*` / `althing-mcp` stdio surface. The v0.15 lean-bus cut RIPPED moderation / chamber / forseti-daemon / agent-runner / redis-valkey.	per-box `uv tool install` (NOT CI-deploy); nh3-dev = the DEV box (editable install of `~/development/althing`, gets new versions first); nh3-extdev a mesh peer (model B: althing-svc + shared `/srv/althing`)
`vh/mead-hall`	Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50)	push-to-main → CI deploys (2026-05-16)
`vh/skaldsong`	Wizard + reader surface (port 8300, ana-docker, registry-pull pattern)	push-to-main → CI deploys (2026-05-19)
`vh/Worldtree`	Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. gitea-runner builds on ana-docker; claude-bot ADMIN collaborator (2026-06-20). Now v1.0.0b19.	push-to-main → CI build-and-deploy (runner on ana-docker)
`vh/yt-voice-clipper`	YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000)	push-to-main → gitea-webhook auto-deploy to irv-ml1 (2026-06-03) — see `docs/runbooks/ytvc-autodeploy.md`
`vh/arbo`	Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image)	push-to-main → gitea Actions CI (deploy-engine.sh, build-local, health-gated) now LIVE; catalog via :9009 webhook
`vh/zonos-gateway`	OpenAI-compatible TTS gateway over stock ZONOS2 (`:8890` irv-ml1); emotion dials-first + voice mapping; reached via LiteLLM `ext-tts` alias. v0.2.1 (2026-07-18): voice-resolved emotion presets (`resolve_preset(name,voice)`; angry/happy/startled_happy per-voice). 8 voices incl. 4 clones	pushed to gitea (main `8f1885b`/`v0.2.1`); deployed irv-ml1 tree still NON-git (hand-updated build context — CI-wire = open follow-up). Spec `docs/EMOTION-DIALS-SPEC.md`; host-managed voices bind-mount (`./voices:/app/voices`, drop wav + restart, no rebuild)
`vh/soong-lab`	Noonien Soong character-design studio (SPA + /api + WT `/bifrost/tool-call`); containerized 2026-07-18, LIVE on corviduo-dev `:8443` (image `vh/soong-lab:latest`). soong-dev owns Dockerfile/compose/workflow; infra-ops owns the host	CI = Gitea Actions build+push+DEPLOY on tag/dispatch (fleet recipe: docker:cli + raw buildx, pushes AS vh; auto-redeploy LIVE 2026-07-18 — runner SSHes corviduo-dev as `deploy`, `compose pull && up -d` from /opt/soong-lab, health-gated on /api/version). Manual redeploy `sudo -u deploy bash -c 'cd /opt/soong-lab && docker compose pull && docker compose up -d'`. → `archival-memory.md` (archived 2026-08-16)
`model-training-forge` (mtf-dev)	Fine-tuning recipe forge; T1 = E-RP writing LoRA, retargeted qwopus-122B→AEON-27B (2026-07-06) (SFT→DPO, LitBench-RM reward)	training runs, not a deployed sidecar

(vh/volva + Heid were re-architected from systemd daemons to Claude Code session orchestrators 2026-06-08; their nh3-dev .service units were removed — no longer deployed sidecars here. See Recent decisions.)

Two-layer backups — Backrest orchestrates restic for file+DB (5 fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore + cross-site restic targets — see docs/runbooks/disaster-recovery.md for the blast-radius matrix. ⚠️ The restic file+DB layer routes through TWO rest-servers (rest-server-ana @ ana-docker:8000 → ana-docker/ana-ml2/esh-docker-vm/vm-esh-nas; rest-server-nh3 @ nh3-nas:8000 → irv-ml1/nh3-docker). Both depend on their NAS's NFS export of /mnt/backup. (rest-server-ana recovered 2026-06-20.)
pull-hf-repo.yaml is the canonical "get a HuggingFace model/dataset onto ana-ml2's shared cache at /tank/aimodels/huggingface/" playbook. Supports --var repo_type=model|dataset|space. Replaces ad-hoc huggingface_hub.snapshot_download patterns.
Worldtree admin auth — per-instance. Each Worldtree deployment (demo :8080, personal :8081, pinned :8082) has its own Heimdall registry and its own bootstrap admin key. Infra-ops's stored long-lived admin key (key_id 61419c92) at ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin auths against demo only. Personal-instance admin (the ~/.config/worldtree/personal-admin-token, mode 600) POSTs /admin/keys (mints per-project keys; takes user_id+label, no scope param — scopes are tier-derived). On-instance mint recipe (cleaner than DB-manip): docker exec worldtree-worldtree-api-1 POST /admin/keys with the in-container WORLDTREE_BOOTSTRAP_ADMIN_KEY; cleartext once in .key=wt_live_+16hex. auto-memory reference_worldtree_demo_key_mint.
Per-project user keys against personal Worldtree (issued 2026-05-19): skaldsong:79744637, skaldsong:7c1dbbbe, althing:50d85460, mead-hall:a360822d. Mint via /admin/keys, drop value to /tmp/wt-personal-<name>.key mode 600, dev collects + shreds (DO NOT cat to chat transcript).
Skaldsong CD pattern (registry-pull). vh/skaldsong's CI builds and pushes gitea.phasefinal.com/vh/skaldsong:<sha> + :latest; playbooks/deploy-skaldsong.yaml on ana-docker pulls + recreates. SHA-pin only. Prereq: host needs docker login gitea.phasefinal.com once.
gitea internal route for fleet hosts. gitea is a container on ana-docker — git-SSH 10.250.50.70:222, HTTP :3000. Fleet/colo hosts must use this internal route, NOT public gitea.phasefinal.com (38.120.12.44) — the public path fail2bans the host egress IP. Full gotcha in docs/orientation.md → Git/gitea.
docker-as-root pattern (for ops with no admin API, or to edit deploy-owned/root-owned files without sudo): docker run --rm -v <target-dir>:/wt docker:cli sh -c "...". docker-group membership is effectively root via bind-mount. Foot-gun: relative paths in compose.yaml resolve against the sandbox CWD but the daemon interprets them against the HOST fs — always pass -e VAR=/abs/path for any relative-default config dir.
scripts/elway sudo handling — elway prompts for the sudo password ONCE via getpass before the first sudo: true step → can't run unattended from a non-TTY tool if any step needs sudo. Sudo-free playbooks run fully non-interactive over key SSH.
Per-host SSH identity matters for sudo. infra-ops has NOPASSWD sudo on most PFI Linux boxes (corviduo-dev included since 2026-06-15). On ana-docker: default ssh ana-docker = lkraven (docker-group, NO passwordless sudo); ssh infra-ops@ana-docker HAS NOPASSWD root. → For any sudo op on ana-docker, use ssh infra-ops@ana-docker. ssh infra-ops@10.100.10.50 (nh3-dev) ALSO NOPASSWD sudo; on nh3-extdev infra-ops is sudo-LESS by design (ssh lkraven@10.100.50.42 is the NOPASSWD path). irv-ml1: ssh irv-ml1 = lkraven, docker-group (plain docker) but sudo needs a PASSWORD (no NOPASSWD) — stage model pulls to /home, not root-owned /worktank.

Current state / in-flight

🟢 IPv6 PLAN SETTLED 2026-08-19 — endpoints, not internal numbering. IPv6 is for tunnel endpoints (Site Magic + WireGuard), not for numbering LANs: an endpoint needs one routable address, not a prefix. ESH has a whole /56 (2607:73c0:1004:1::220, pd_size=56) — the long-standing "ESH: none" note was wrong. NH3 has its WAN GUA 2600:1700:b25:c110::48 plus one unused /64. ANA expects a /48 or /56 within days. Nothing was ever down — ESH is outbound and working; its v4 CGNAT (100.104.3.250, Cogent AS174 egress) is the ordinary CGNAT-v4-plus-generous-v6 pattern, not an outage, and the previous "CGNAT broke the mesh" framing sent a three-arm research effort chasing a non-problem. NH3 internal v6 was enabled on the /64, verified working end-to-end, then deliberately reverted — one /64 lights one VLAN. AT&T prefix findings (the /60 is real but undelegatable; the live trick is multiple IA_PD in one solicit, not VRRP; the UDM can express neither) are filed as reference only. → auto-memory reference_fleet_vpn_topology_ipv6
🔴→🟢 ESH OUTAGE 2026-08-19 — esh-pve hard-froze 03:34, ~4.5h, whole house lost DNS. Presented as "wifi/routing issues"; internet was healthy throughout. Cause: esh-userland (VLAN 10, the PVC SSID) handed out one resolver, 10.0.50.45 (AdGuard on esh-docker-vm) — cross-VLAN, no secondary — and its hypervisor died. Recovered by hand power-cycle; all VMs, cluster quorum and DNS restored. Two fixes landed: gateway 10.0.10.1 added as secondary DNS on esh-userland (operator-approved, first confirmed WRITE on the ESH UDM key), and softdog → iTCO_wdt hardware watchdog owned by systemd (playbooks/esh-pve-hardware-watchdog.yaml, idempotent, verified armed) so a repeat self-recovers in 60s. VM 102 pinned off (onboot: 0) as the passthrough suspect. ⏳ OPEN: (a) the watchdog is armed but has not been proven to fire — needs a deliberate wedge to confirm; (b) AMT/vPro still unusable until an onboard RJ45 is cabled (the MS-01 is SFP+-only on the network and AMT cannot ride it); (c) kernel 6.8.12-16 rollback held in reserve if it freezes again. → persistent-memory.d/2026-08-19-esh-pve-freeze-dns-spof.md

As of 2026-08-19 (late) — a long infra session, all of it landed and committed: fleet *.internal DNS is live (git-sourced, 42 names, three resolvers incl. a NEW colo one), waterland studio containerised on irv-ml1, Homepage cleaned up and themed with Australis Skyfall + an Arbo-generated background, and four unmanaged stacks adopted into stacks/ (two of which turned out to be quietly broken — SearXNG's healthcheck, SeaFile down 3 months). ⚠ 17 commits unpushed. ~~⚠ 2 unread althing messages from waterland-dev left deliberately undrained~~ — drained 2026-08-19 post-/clear; thread closed, see below.

🟢 FLEET .internal DNS — LIVE 2026-08-19. <host>.<site>.internal, sites ana/esh/nh3. dns/internal.yaml is the source of truth; scripts/dns-sync.py reconciles the three AdGuard resolvers (diff → prompt → apply, idempotent). 42 names resolving from all three sites. Colo got its first resolver ever (stacks/adguard-ana/, API on 8053 not 8080, no blocklists by design) — before this, ana-docker resolved straight against 1.1.1.1. Auth = a dedicated infra-ops AdGuard user, password vaulted nh3-dev/adguard-infra-ops-password. ⏳ TWO OPEN, both operator's to schedule: (a) colo hosts still point at 1.1.1.1 so they do not yet use the new resolver — repointing a site's DNS is a separate change; (b) the static-v6 convention (each server at its site's /64 with low bits echoing the v4 octet, esh-docker-vm → …::45) is proposed, not ruled on. The v6: column is empty and correct — no fleet host has a global v6 address yet. → persistent-memory.d/2026-08-19-fleet-internal-dns.md
🟢 WATERLAND STUDIO — containerised + live, irv-ml1:8410. Displaced a bare nohup (PID 1283383). Warm 256²+anim render 6.1s, beating the 7.4s bare-metal figure. claude-bot now has read on vh/waterland (operator-granted); token on irv-ml1 root-owned 0600, repo-scoped helper, nothing in .git/config. ✅ All three upstream findings fixed and DEPLOYED at b72425b (one update.sh run, 2026-08-19): PR #5 declared the CUDA headers and took uv out of the render path; PR #6 (operator-green-lit) added the startup rehydrate that fixes the unbounded job-store growth. Verified after: /api/jobs 1 → 16 against 16 dirs on disk — API and volume agree for the first time — nothing wrongly reclaimed (16 < RETAIN=40), healthy on backend: cupy, warm render intact. The image keeps its [ctk] install (+0.3s, now a no-op audit) and UV_NO_SYNC/UV_OFFLINE as defence-in-depth. Thread closed by both sides. → persistent-memory.d/2026-08-19-waterland-studio-containerised.md
🟢 HOMEPAGE — cleaned + themed (Australis Skyfall). Three real defects fixed (UltraSeedbox on all tabs, Uptime Kuma double-rendered, fiction column counts), AI tab reordered by clickability, then themed from the operator's Skyfall handoff bundle with a background generated by Arbo (t2i-ui-background, job 13f0891f4e42). ⚠ After any recreate the tab bar/wallpaper/i18n vanish for up to ~an hour and then return on their own — do not chase it. ⚠ CSS is served per-request: a theme change needs a reload, not a recreate, and candidate CSS can be injected live via Playwright for seconds-long iteration. → persistent-memory.d/2026-08-19-homepage-skyfall-theme.md
🟢 GEN SEAT — SWAPPED to absolute-heresy 2026-08-17 (validated, promoted). Live gen = /tank/aimodels/qwen38-27b-heresy-nvfp4-mixed — MuXodious/Qwen3.8-27B-absolute-heresy (Heretic v1.4.0 + SOMPOA, trial T377, pin c2374593) put through our own mixed NVFP4+FP8 recipe. Chosen because it beats the incumbent on both axes at once: author refusals 2/101 vs 12/100, first-token KL 0.0759 vs 0.1191. Gate (probe :8017, pinned nightly, seat-matched flags): MTP 47.2% (inc. 48.2%), decode 103.5 tok/s (96.4), prefill 6618/5403 @6.7k/27k (6334/5085), PPL 6.910 (7.059 — 2.1% BETTER), surface 6/6, abliteration 4/4, and 0/55 refusals on our battery-instruct arm with ZERO EMPTY (no catatonia). ⚠ speed deltas are image-confounded (probe on the pinned nightly, incumbent numbers from an earlier image) — read as "not worse", not a clean win. All 7 LiteLLM aliases verified end-to-end; GPU0 at 91.3/97.9 GB with meromero healthy (more headroom than the old build's 96.8). ⚠ RC1, 2 days old, ~348 downloads. Operator-confirmed "working very well" in real use 2026-08-17, same evening as the cutover — the signal the synthetic gates structurally cannot give (multi-turn degeneration is stochastic; four synthetic tests once validated three non-fixes). Not yet the 60k-token bar the prior seat cleared, so keep watching and do NOT delete the rollback weights yet. ROLLBACK: sudo cp /opt/docker/compose/gen-seat/.env.bak-heresy-20260817 /opt/docker/compose/gen-seat/.env && cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen; incumbent weights UNTOUCHED at qwen38-27b-uncensored-nvfp4-mixed — do NOT delete until this holds. Runbook services/gen-seat-mixed-quant/RUNBOOK-heresy-swap.md.
🟢 PRIOR GEN SEAT — RESOLVED 2026-08-17 (the multi-day degeneration saga); now the ROLLBACK target. Was the in-house JonathanColetti/Heretic mixed NVFP4+FP8 build (/tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed, FP8 attention) on vLLM nightly PINNED vllm/vllm-openai:nightly-311b3513… (v0.27.2rc1.dev150, carries #51113 mamba fix), MTP ON, prefix-caching ON. Operator-confirmed coherent through 60k tokens real multi-turn. Root cause = TWO compounding real causes: (1) genuine vLLM qwen3_5_mtp×GDN partial-accept bug (#51113, architectural across vLLM/SGLang/llama.cpp, fixed by nightly), and (2) AEON's full W4A4 being lowest-fidelity on the known activation gradient (W4A4 < W4+FP8 < W4+bf16) → ~15-20% stochastic degeneration on top of (1). AEON PURGED (re-pullable sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4). Full lesson docs/pfi/model-quantization-playbook.md §3.8. (Superseded as primary by absolute-heresy 2026-08-17.) ⚠ pinned nightly is bleeding-edge — move to a stable release once #51113 ships in one (the standing follow-up). 7 aliases (gen/gen-reasoning/summarizer/-large/classifier/image-judge/qwen-image-bench) all route here. Seat carries --default-chat-template-kwargs '{"reasoning_effort":"medium"}' (per-request overridable, affects gen-reasoning only). Commits d28a371,2f2bbce,2185964.
🔵 RP SEAT — FABLE-FUSION serving char-rp-reasoning (evaluation window, unchanged this session). fablefusion-charrp-probe ana-ml2 GPU1 :8019 serving char-rp-probe (kkuspa/Qwen3.6-27B-Fable-Fusion-711-…-MTP-NVFP4A16). LiteLLM char-rp-reasoning + char-rp-fable both route to it (deliberate repoint, documented in stacks/litellm/conf/config.yaml). darkscarlett-charrp-reasoning is compose down, weights intact at /tank/aimodels/darkscarlett-nvfp4-work/. ⏳ STILL AWAITING operator's hands-on read of FF prose (refusal question settled: FF 15.8% vs DS 92.5% cold-framing; DS v1.0 never abliterated). ⚠ FF reasons 2.1–4.6k chars → use max_tokens ≥3072. ReadyArt/Dark-Scarlett-27B-v2.0 (Qwen3.8) is GATED (403 awaiting review) — operator ruled not-interesting, do NOT re-propose. DS regeneration for brokkr RETIRED 2026-08-17 — unqueued, do NOT run (9c1405b): brokkr withdrew on the operator's call because (a) our ictrl-pair-unwrapped/-wrapped control isolates the classifier over-fire cleanly where DS's cross-class delta only bounded it, and (b) DS v2 releases soon, so a k=5 v1 baseline baselines a superseded version. Spec at services/refusal-probe/darkscarlett-regen-spec.md stays banked as the record of the run that will not happen (axes + per-class grading asymmetry still correct, checklist struck through). No GPU1 window was ever spent. A DS-v2 characterization would be a fresh purpose-scoped ask.
🟢 LOBE CHAT — LIVE on esh-docker-vm :3210 (2026-08-17). Replaces the hand-rolled gateway-chat HTML surface. stacks/lobe-chat/, image lobehub/lobe-chat (143 MB compressed vs Open WebUI's 1.8 GB — the weight call). Scoped LiteLLM key lobe-chat-esh (free-local models only; paid GLM/Kimi BLOCKED, verified). Secrets vaulted esh-docker-vm/lobe-chat-*. TTS = a SPLIT: endpoint env-driven (inherits OPENAI_PROXY_URL→ext-tts), but voice/model/format UI-only. System-agent repointed off its gpt-5-mini default onto fleet models via SYSTEM_AGENT env. TTS FIXED 2026-08-18 — no UI pass needed. Lobe's TTS had never worked: it sends {input, model:"tts-1", voice} and LiteLLM resolves the model name FIRST, so tts-1 403'd against the scoped key's allow-list and never reached the gateway (our "unknown model routes to the gateway default" belief was true of :8198 and false of the LiteLLM path — that's what hid it). Fixed by aliasing the stock names rather than patching the client: tts-1, tts-1-hd, gpt-4o-mini-tts /model/new'd to openai/zonos@10.100.79.3:8198/v1 (mode: audio_speech), plus those three added to the lobe-chat-esh allow-list (20→23). Verified with Lobe's exact payload on Lobe's own key: 200, 69,740 B, MPEG. ⚠ These three are DB rows, not references — if ext-tts repoints, they must move with it. Done with the infra-ops admin key, not sk-corvid: it has /model/new + /key/update rights, so this class of ask never needs the master key. Also live: tts-gateway v4 defaults response_format to mp3 (tts-dev shipped it; 122,924 B wav → 27,692 B mp3 same utterance; every in-house consumer already pins the field, blast radius checked pre-ship). Commits e9362de,163a725,cac75cb,933253d,ca8c0a3 (last one authored by tts-dev correcting two load-bearing wrong claims in our README/compose — kept).
🟢 LITELLM — upgraded v1.91.0→v1.97.0, spend-log DB purged 6GB→16MB + CAPPED (2026-08-17). store_prompts_in_spend_logs:false + maximum_spend_logs_retention_period:7d. ⚠ 1.8GB pre-upgrade pg_dump still on ana-docker /opt/docker/compose/litellm/ — deletable now the upgrade is proven (operator was going to call it). Commit 01b5ad9.
⚠️ GPU zero-sum (both cards ~94–95/97.9 GB). GPU0: gen + meromero. GPU1: fablefusion + utility cluster. Any util bump on either seat of a shared card must be checked against the co-tenant (starved meromero into a crash-loop once at 0.45).
FLEET RERANKER = A3 (bge-reranker-v2-m3) PROD ana-ml2 GPU1 :8013. Passive watch; levers = A4 :8014 / util / 2nd replica; incumbent :8002 warm. docs/pfi/reranker-selection-ledger.md.
EVIDENCE HOLD (partial): WT #394 FILE half STILL STANDS — do NOT delete on-disk gen dirs (fiction/rex390-dcc, rex392-dcc, b59c147c5ce0); rex393-fiction-* + r42-gate-* KEEP.
🔴 NH3/AT&T IPv6 PD is a HARD /64 — TESTED ON THE WIRE 2026-08-19, not inferred. AT&T support (Christopher, prov-dns@att.com) said "I do not believe att will do that" and punted to consumer sales — and he was right, though his team is DNS provisioning and the answer was a guess. Verified empirically: the NH3 UDM solicited DHCPv6-PD at /48, /56 AND /60 and AT&T returned the same single /64 (2600:1700:b25:c11f::/64) every time. So it is NOT "nobody ever asked" — the request was made three ways and the server answers /64 regardless. How it was proven: the PD is invisible until a LAN claims it, so nh3-iot (the only NH3 VLAN with 0 clients) was temporarily set ipv6_interface_type: pd; then ipv6_pd_prefixid was set to 0, 15 and 16 — all three returned the identical c11f prefix, which is the signature of a single delegated /64 (with any larger block the prefix-id moves the LAN within it). ⚠ Do NOT read the WAN address vs LAN prefix gap as evidence of block size — WAN is c110, delegated is c11f, and I initially misread that spread as a /60; AT&T just assigns the two from different places in their pool. NH3 UDM was fully restored (nh3-iot back to none, WAN back to pd_size=64); rollback artifacts at nh3-dev:~/backups/nh3-udm/. THE ASK FOR AT&T BUSINESS (this is business fiber at a commercial park — the residential org cannot change it, and 800-288-2020 is the wrong number): "circuit solicits DHCPv6-PD at /48, /56 and /60; server returns a single /64 in all three cases; please increase the delegation size on the provisioning profile" — a claim they can verify against their own DHCPv6 logs. ROOT CAUSE FOUND 2026-08-19 — it is the BGW, not AT&T, and EIGHT /64s are actually available. The CPE is a BGW210-700, firmware 4.28.7,management reachable at http://192.168.1.254 (v4 is already IP-passthrough — the UDM holds public 70.230.226.88 directly — but passthrough covers v4 ONLY; v6 always terminates on the BGW). Its /cgi-bin/lanstatistics.ha and /cgi-bin/broadbandstatistics.ha are readable unauthenticated and give the whole picture: BGW WAN v6 = 2001:506:70b2:8958::1, BGW LAN v6 = 2600:1700:b25:c110::/64 (it keeps this for itself), delegated to the UDM = 2600:1700:b25:c11f::/64. So AT&T DOES hand the BGW a /60 (c110-c11f); the BGW keeps c110-c117 and re-delegates up to EIGHT individual /64s on c118-c11f, top-down. Our UDM got c11f = delegation #8. The other seven are sitting unclaimed — not withheld by AT&T, just never solicited, because UniFi exposes only a single wan_dhcpv6_pd_size integer and no 'number of prefixes' field, so it asks once. The documented workaround is dhclient -6 -P -P -P... (one -P per prefix), which UniFi cannot express. ⇒ An AT&T ticket CANNOT fix this — it is CPE firmware behaviour, so Christopher's 'att won't do that' was accidentally right. Real options: (a) accept one /64; (b) full BGW bypass via wpa_supplicant/EAP-TLS on the UDM (documented working on UDM Pro; extract the BGW's 802.1X certs, authenticate to AT&T directly, negotiate the whole /60 yourself) — ⚠ modifies NH3's ONLY uplink and must survive UDM firmware updates, so it needs a planned window with physical access, NOT an ad-hoc change. ⏳ PARKED pending the Device Access Code: reclaim-nh3-s-7-unclaimed-ipv6-64s-from-the on the henge. Operator will fetch the BGW210 Device Access Code (12 digits, on the unit's label) next time he is at the NH3 office and vault it — suggested path unifi/bgw210-nh3-device-access-code — after which this is drivable remotely. First stop is http://192.168.1.254/cgi-bin/ip6lan.ha (confirmed to exist, currently redirects to Login); check for a delegation size/count control, a v6 pass-through/bridge option, or a way to stop the BGW reserving c110-c117. If the UI can do it, that is a SUPPORTED fix and the wpa_supplicant bypass is unnecessary. Other settings pages behind the same login: ippass.ha (how v4 passthrough was set up), dhcpserver.ha, firewall.ha, broadbandconfig.ha, packetfilter.ha, remoteaccess.ha, restart.ha. Consequence if left alone: NH3 has ONE v6 segment, vs ESH's /56 (256), which makes NH3 the prefix-constrained site in any fleet v6 design.
⏳ COGENT IPv6 PROVISIONING REQUESTED for Anaheim/colo (operator, 2026-08-19). The colo is the last site with no v6 (ESH has a /56 from Cityside, NH3 has AT&T v6 on 2600:1700:b25:c110::48). Diagnosis from our side before the ticket: diagnose sniffer packet wan1 "ip6" on ana-gw caught exactly one RA in 90s, from fe80::ea0a:b9ff:fe3b:2c16 — so an IPv6-capable router IS one hop away on the circuit terminating 38.120.12.42/29 — but set autoconf enable (SLAAC) got no global address after 2min of RA intervals, and execute ping6 to both 2606:4700:4700::1111 and 2001:4860:4860::8888 was 100% loss. That pattern = router present, circuit not provisioned for v6 (RA likely carries no Prefix Information Option). FortiGate-side v6 config was fully reverted after testing — wan1 is back to no v6, v4/IPsec untouched throughout. ⚠ FortiOS gotcha for when it lands: SLAAC is config ipv6 / set autoconf enable, NOT an ip6-mode (that only takes static/dhcp/pppoe/delegated). WHEN COGENT PROVISIONS: ask for DHCPv6-PD size (ESH got a /56 from Cityside; NH3 only gets a single /64 from AT&T, so a /56 or better is the ask), then the colo becomes the fleet's only site with BOTH a static public v4 (38.120.12.42) and routable v6 — which makes it the natural v6 hub, since ESH is CGNAT'd on v4 and NH3 is prefix-constrained.
🟡 ESH IPv6 LAN naming — a DOCS CONVENTION ONLY, not wire-level. Fun hexspeak names picked 2026-08-18 for the six ESH LANs: RESERVED for a future DMZ = 4411:DBAD ("FOR ALL DA BAD" — no DMZ network exists on the ESH UDM yet; name claimed 2026-08-19 against the day one is built). esh-cameras=1533:FACE5 ("I SEE FACES" — operator's pick 2026-08-19, and the best of the set), esh-mgmt=15DA:B055 ("IS DA BOSS" — 15=IS, DA literal, B055=BOSS; revised 2026-08-19), esh-server=4411:B105 ("FOR ALL BIOS" — operator's pick 2026-08-19; shares the 411 with userland's CAFE:4411), esh-userland=CAFE:4411 ("CAFE FOR ALL" — operator's pick 2026-08-19; 411 also being US directory assistance is a bonus), esh-iot=4DBA:D107 ("FOR DA BAD IOT" — 4=FOR, D=DA, BAD, 107=IOT; same 8 digits, re-glossed 2026-08-19), Default=4BA5:3417 ("A BASE FOR IT" — operator's pick 2026-08-19; 8 digits, words straddle the group boundary like 4DBA:D107). Investigated and confirmed these CANNOT be made literal on the wire: (1) a network's only nameable slot is its /64 subnet id (ipv6_pd_prefixid), which is 2 hex digits (0-255) — not enough room to spell a 4-char word; (2) the gateway's own address is fixed at ::1 by the UniFi platform, no field to customize it (ipv6_aliases exists but schema unknown, not attempted blind); (3) UniFi has no IPv6 equivalent of use_fixedip/fixed_ip (checked the client schema directly) — individual devices cannot be pinned to a chosen v6 address the way v4 static reservations work; SLAAC devices self-assign via EUI-64 (MAC-derived, stable) or privacy-extension (random, rotates). So the names are a mnemonic for humans/docs, not something that will ever appear in a dig/ip -6 output. Only esh-cameras has v6 live at all (2607:73c0:402:1d00::/64, PD from the ESH /56); the other five stay ipv6_interface_type: none pending a firewall-policy pass before enabling SLAAC fleet-wide. ⚠ Precision that matters here: SLAAC gives hosts globally ADDRESSABLE addresses, NOT globally reachable ones. Addressability is a property of the address; reachability is a firewall policy decision, and operator's position (2026-08-19) is unambiguous — no 1:1 inbound pass-through, ever. v6 removes NAT, it does not remove the firewall, and conflating the two is how people talk themselves into thinking v6 means automatic exposure. The firewall-policy pass is about writing explicit default-deny inbound rules for each v6 segment, not about deciding whether to expose anything.
🟢 ESH↔colo IPsec FIXED 2026-08-18 — dialup + NAT-T. Broke when ESH lost its public IP (Cox unplugged for the fiber install; 5G handed out RFC1918 192.168.200.111). TWO causes: phase1 ana-to-eshudm was type static pinned to the dead 70.181.90.232, AND nattraversal disable — so even re-pinning the IP would have failed, since ESP cannot cross NAT without UDP-4500. pfi-ana-nh3 survived only because NH3 is publicly addressed. ⚠ FortiOS REFUSES set type dynamic on an existing tunnel ("Cannot change tunnel type once configured", clean rollback). Fix = ADDITIVE, not a recreate: new phase1+phase2 ana-eshudm-dyn (type dynamic, ikev2, aes256-sha1, dh14, NAT-T default on, PSK read from the ESH UDM API), static route id 10 (dstaddr ana-to-eshudm_remote → device ana-eshudm-dyn, distance 20), policies 73/74 (consolidated multi-zone in/out). Old ana-to-eshudm LEFT IN PLACE, dead and harmless, as rollback. Verified: ana-eshudm-dyn_0 97.170.236.56:4500 selectors 1/1 (_0=dialup child, :4500=NAT-T, carrier IP), ESH→colo all hosts 40-56ms, colo→ESH OK, traceroute now 3 hops encapsulated vs 8 leaking to the carrier. FortiGate config backed up first at nh3-dev:~/backups/ana-gw/ (1.17MB, 36903 lines). ⚠ RESIDUAL: the UDM's ipsec_local_ip needs a LITERAL IP (empty rejected api.err.InvalidPayload) — currently 192.168.200.111, so it must be updated again when the fiber lands or the tunnel drops. Gateway end is address-agnostic now; UniFi end is not — it will drop on ANY ESH WAN change: Cox reclaiming WAN1 (still failover_priority=1), the fiber cutover, or a DHCP renewal. MANUAL FIX (seconds): read the UDM's live wan_ip from /stat/health and PUT it into esh-ana.ipsec_local_ip via rest/networkconf/<id> using unifi/esh-udmpm-api-key. A self-healing watcher is PARKED (self-healing-ipsec-local-ip-watcher-for-the-esh), trigger = once the new ESH fiber is live and its addressing has settled — operator's call 2026-08-18, deliberately not built now because it would target an address that is about to change. Park item also carries the follow-up to delete the old ana-to-eshudm tunnel, whose distance-10 route would silently reclaim traffic if Cox ever returns on 70.181.90.232.
🟡 FLEET IPv6 — STILL JUSTIFIED, but the REASON was wrong and is now corrected (2026-08-18). There are TWO inter-site links with OPPOSITE NAT behaviour, and the original note conflated them. (a) NH3↔ESH = Site Magic = WireGuard — SURVIVES arbitrary NAT, PROVEN. During the fiber cutover ESH failed over to 5G on 192.168.200.111 (RFC1918 double-NAT, zero inbound path, strictly worse than CGNAT) and it held: nh3-dev/nh3-docker reachable ~40-43ms, ssh+exec fine. Mechanism: magic_site_to_site_vpn on-device is only enabled+keypair (no WAN binding — it rides whichever uplink is active), and NH3's edge is public at 70.230.226.88, so the NAT'd side dials out and never needs reachability. (b) colo(Anaheim)↔ESH = IPsec on the ana-gw FortiGate — BROKEN RIGHT NOW under those same conditions. ana-docker/pfi-pve/pbs-ana ALL FAIL from esh-pve-nas; traceroute shows 10.250.x packets leaving the UDM to the 5G modem 192.168.200.99, then wandering the carrier (10.180.141.194, 172.19.3.14) and dying — not encapsulated at all, so no SA is up and traffic falls to the default route. Site-to-site IPsec pins a peer IP; ESH no longer has a routable one. ⇒ IPv6 keeps its justification, on the IPsec link SPECIFICALLY. ⚠ An earlier version of this entry claimed the premise was refuted outright — that over-generalised a WireGuard result onto an IPsec link. Operator caught it. See ops-lessons-playbook §8. State: NH3 WAN live (2600:1700:b25:c110::48, AT&T delegates exactly one /64), colo none (FortiGate has zero v6), ESH none (both WANs wan_type_v6=disabled). NH3 LANs all reverted to ipv6_interface_type=none per operator. Work when addresses land: v6 on ana-wg eth0 + a v6 port-forward for UDP 31337 on the FortiGate (its WG socket is already dual-stack — no WG reconfig), flip the UDM WG server off v4-pinned binding, and AAAA records so the dynamic prefixes at all three sites don't break endpoints. Full detail + access recipes → persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md.
🟢 WT #401 (fd-leak deadlock) CLOSED 2026-08-17 — one ping still owed. worldtree-dev closed it on our demo verify. Layers: (a) their e41b139 pins ulimits: nofile 65536/65536 in the worldtree compose anchor — demo VERIFIED (api + matrix recreated 22:55:34Z, ulimit -Sn=65536); personal/pinned are covered-not-verified, they inherit at their next promotion/recreate. (b) our host floor is STAGED, NOT ACTIVE — /etc/docker/daemon.json on corviduo-dev carries default-ulimits nofile 65536/65536 but default-ulimits is NOT SIGHUP-reloadable (measured on 29.4.3: post-reload the daemon's own "Reloaded configuration" log omits it and a fresh container still reports 1024). Activation needs a full dockerd restart = bounces all 13 containers; worldtree-dev explicitly does NOT want one, and live-restore:true-then-restart is PARKED as a separate host-side improvement for the operator to rule on, never folded into #401. Playbook playbooks/corviduo-dev-docker-default-ulimits.yaml (verify step 3 fails BY DESIGN until a restart). Hourly fd tripwire on corviduo-dev stays armed. ⏳ OWED: ping worldtree-dev in thread 01M08QQ655XD6VKEV7MA9GX0NS once worldtree-personal recreates and 65536 is confirmed there. Commit 7f3f265.
⏳ WATCHING: DavidAU's HERETIC build of Qwen3.8-27B — the one worth waiting for. DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1 examined 2026-08-17 and NOT adopted: it is a capability/efficiency finetune of stock Qwen3.8 and every bench row on its card is labelled [non heretic] — adopting it would reintroduce base refusals the current seat does not have. ⚠ Easy to misread as uncensored (operator did, and it is a fair mistake): DavidAU's back catalog is almost all Uncensored-Heretic builds — Fable-Fusion 711, Qwen3.5-9B Cold-Fusion — so the naming pattern implies it. This one simply has not had that stage run yet; the card's roadmap says the HERETIC version is IN PROGRESS from base. That is the release to watch, not this one. What makes it worth watching: third-party benches (Nightmedia, mxfp8) beat stock Qwen3.8 by +0.064 arc/c, +0.056 arc/e, +0.056 obkqa; claimed MTP acceptance 55.7% (record 59.9%) vs our measured 47.2%; thinking tokens cut to 1/10–1/2; PPL dropped vs base. Same GAIN/Cold-Fusion pipeline that produced Fable-Fusion 711, which we already serve on char-rp-reasoning — proven in-fleet, not just claimed. Structurally clean (1199 tensors, 15 mtp in shard 18, 333 visual). ⚠ MTP/speed figures are GGUF/llama.cpp on a 5090, not vLLM — may not transfer; and because its MTP head was likely trained, the free CPU-hash shortcut would NOT apply (it won't match base) so a real acceptance gate would be needed.
🟢 esh-pve-nas — MIGRATION DONE 2026-08-18. Root is nvme/ROOT/pve-1 on mirrored NVMe; the USB DOM is out of the runtime I/O path. All five guests healthy, three pools ONLINE, system running, ext4 pve-root intact+unmounted as rollback with its own kernel. Boot config: saved_entry=pve-zfs-root, no next_entry; if grubenv were unreadable GRUB falls to entry 0 which the /etc/default/grub.d/zfs-root.cfg drop-in also points at ZFS. zfs-import-cache.service is now the active import path (the all-three-pools cachefile fix working as intended); vestigial zfs-import@nvme.service disabled — it failed every boot as redundant. ⚠ Device letters shift across reboots (DOM was sdq, now sdl) — never key anything to a bare sdX here. ⚠ NO auto-fallback on a failed boot, and no IPMI/BMC/serial — grubenv on LVM is readable but not writable by GRUB, so grub-reboot's one-shot degrades to a sticky default (verified: next_entry survived the boot that consumed it). Recovery = pick the ROLLBACK entry at the console. PATCHED 2026-08-18: 225 packages installed, pve-manager 8.4.11 -> 8.4.20, corosync 3.1.9 -> 3.1.10-pve2, kernel 6.8.12-42 staged on the /boot LV. dpkg clean, no unapplied conffiles, cluster quorate, 6/6 verify. ⏳ REBOOT DEFERRED at operator request — host still runs 6.8.12-13 until a chosen window; GRUB_DEFAULT=0 means entry 0 is already the -42 entry with the correct root=ZFS=nvme/ROOT/pve-1, so the reboot is the only remaining step. Rollback for the upgrade is the ZFS snapshot nvme/ROOT/pve-1@pre-upgrade-20260818T141652Z (409M) — zfs rollback -r <snap> && reboot. Second confirmation reboot ALREADY DONE (2026-08-18, booted ZFS from GRUB_DEFAULT=0 with no one-shot). ⏳ Still outstanding: refresh the off-box DOM image, since /boot changed. esh-pve is FULLY done (8.4.20 + kernel 6.8.12-42 + corosync 3.1.10, rebooted, quorate). ⚠⚠ THE WINDOW COST AN UNPLANNED OUTAGE, caused by our own tooling, not the migration. The staging chroot did mount --rbind /dev + /sys with no --make-rslave; on systemd / is shared, so the cutover's umount -R propagated back into the live host and stripped the real /sys/fs/cgroup, /dev/pts, /dev/shm. logind could then create no sessions: ping fine, TCP fine, SSH authenticates, resident daemons keep serving (pveproxy returned clean 401s) — and every new exec hangs, including /sbin/reboot, so the reboot never ran. It is a near-perfect impostor of failing root-disk I/O, and I misdiagnosed it as the USB DOM dying and told the operator to walk to the machine. Operator caught it — the DOM had been fine for years and the wedge began right after a change. The settling evidence was in dmesg all along: [16.00] [sdq] Attached SCSI removable disk (clean, no errors) and a last-line timestamp of [12114881] = 140 days = the ORIGINAL boot — the machine had never rebooted. My down-detector never once reported the host down and I read that as a fast reboot rather than no reboot. Recovered with no console access by hammering an idempotent cgroup2/devpts/shm remount into the brief windows where exec succeeded. Zero data loss. RULES: (1) always --make-rslave after --rbind (playbook now guards on PROPAGATION != shared); (2) a reboot is not confirmed until the host is observed DOWN — poll for disappearance, not reappearance; (3) before blaming hardware for a wedge that started right after a change, get dmesg and check the boot timestamp. ⚠ Blast radius was more than DOUBLE what the runbook said — ss -tn '( sport = :2049 )' in CT 103 found five NFS clients, not two: +esh-vm-db 10.0.50.60 (/mnt/backup, hard, no ssh — reach via qm guest exec 101 on esh-pve), +vm-esh-nas (is VM 104 on the host itself), +nh3-dev (soft,ro — safe). Ask the server who its clients are; a documented dependent list rots. esh-vm-db was deliberately left mounted (backup target, timers ~19h out) and came through read-write — a hard mount with no active user blocks and resumes, which is what hard is for. (historical, pre-cutover context below) migration STAGED 2026-08-18. PVE root ran off a NORELSYS USB Disk-on-Module (sdq). Wear was NOT the driver (a DOM is SLC/pSLC — operator corrected my first read); the drivers are the USB bus (a reset drops root under a running hypervisor), no headroom, no mirror, and blocked patching: 225 packages, 161 with deb12uN/security bumps, stuck on PVE 8.4.11 vs esh-pve's 8.4.14. Mitigated 2026-08-17 (90% → 76%); staged 2026-08-18 via two rerunnable elway playbooks (esh-pve-nas-stage-zfs-root.yaml, esh-pve-nas-stage-bootloader.yaml), 0 failed, 17/17 verify green. Host is still on ext4 and its boot path is byte-identical to the last 140 days — grub-install is deliberately held back to the window, so an unplanned reboot mid-staging is a non-event. CUTOVER = quiesce NFS clients → stop guests → grub-install → grub-reboot pve-zfs-root → zfs set mountpoint=/ nvme/ROOT/pve-1 → reboot. ⚠⚠ THE POOL-NAME BUG — the near-miss worth remembering. update-grub silently emits root=ZFS=/ROOT/pve-1 with the pool name missing → boots to an initramfs prompt. Cause: Debian's 10_linux builds ${rpool}${bootfs} where rpool = grub-probe --target=fs_label, and GRUB's ZFS reader cannot open a pool with encryption/large_dnode/zstd_compress — so it returns empty and the failure is swallowed by 2>/dev/null || true. The same feature set that forced /boot to stay ext4 also corrupts the kernel cmdline, which the design did not anticipate. Fixed with a /etc/default/grub.d/zfs-root.cfg drop-in (last root= wins) plus explicit pve-zfs-root / pve-ext4-rollback entries with stable ids. Durable lesson: assert the EFFECTIVE value, not the presence of a substring — the original verify grepped for the right string appearing somewhere and would have passed with every menu entry still broken. ⚠ Never set the ZFS cachefile on one pool. The runbook's zpool set cachefile=… nvme was a trap: populating a cache flips the host from import-by-scan to import-by-cache, so a cache holding only nvme leaves ssd+tank unimported and empties every CT 103 export. Set on all three 2026-08-18, verified in the 11,976-byte cache. ⚠ Migrate FIRST, patch after — a signed kernel would land in /boot on the 1.3 GB root. ⚠ CT 103 esh-nas (10.0.50.50) runs on this host and serves hard NFS to esh-docker-vm and esh-pve — quiesce both before any reboot or you wedge esh-docker-vm into D-state. Off-box at nh3-dev:~/backups/esh-pve-nas/: DOM image dom-sdq-20260818.img.zst (2.38 GiB, crash-consistent), clean bootchain-20260818.tar.gz, config snapshot …20260818T051*.tar.gz. Runbook docs/runbooks/esh-pve-nas-boot-migration.md; detail → persistent-memory.d/2026-08-17-esh-pve-nas-dom.md.
OPEN FOLLOW-UPS (parked): move gen seat off pinned-nightly to stable once #51113 ships; Lobe one-time TTS UI pass; delete the 1.8GB litellm dump; harden-esh-docker-vm (park id 28, PROMOTED — Tier-1 done, /mnt/books stays hard w/ watchdog); chatterbox-fast build-context divergence; #363 research-wing ingest (no deadline); optionally attach our MTP reproducer to vllm#47087 (needs a GitHub identity — operator's call).
althing monitor ARMED (handle infra-ops). ⚠️ Re-arm ONLY after a real FIRE (rc0), never after a plain operator turn (bounces rc3); spawn althing-wake-listener as its OWN run_in_background task, never chained with & (orphans it — hit this twice 2026-08-17, stop-monitor reclaims).
eshpfi push state: operator pushes manually; last push 50d13f5..8be8a51. ⚠ 17 commits unpushed as of 2026-08-19 (707a8cb..b8003c7, plus this snapshot commit) — nothing is on the remote from this session.

Recent decisions

[2026-08-19] A software watchdog is not watchdog protection — esh-pve froze for 4.5h holding one. softdog cannot fire when the kernel it runs in is wedged, and Proxmox's watchdog-mux never arms without HA resources, so the box looked protected and wasn't. Moved to the PCH iTCO_wdt under systemd. Also: a single cross-VLAN DNS entry with no secondary turns any VM outage into a whole-site outage. → persistent-memory.d/2026-08-19-esh-pve-freeze-dns-spof.md
[2026-08-19] Fleet .internal DNS built and live — git-sourced, agent-managed, three resolvers. Zone-scoped authority (ESH's hand-made esteban.net rewrites survive); the colo had no resolver at all; v6 column empty on purpose because SLAAC addresses rotate. → persistent-memory.d/2026-08-19-fleet-internal-dns.md
[2026-08-19] waterland studio containerised on irv-ml1 — three landmines, all measured. cupy needs CUDA headers the host had by accident; uv run re-syncs and prunes cupy at RUNTIME; the A6000 is container-index 0, not the host's 1. → persistent-memory.d/2026-08-19-waterland-studio-containerised.md
[2026-08-19] Homepage cleaned up, then themed with Australis Skyfall + an Arbo-generated background. Includes the hour lost to a self-healing tab-bar red herring, and the CSS-iteration loop that prevents it recurring. → persistent-memory.d/2026-08-19-homepage-skyfall-theme.md
[2026-08-19] Four unmanaged stacks found on live hosts — two quietly broken. A dashboard card is a cheap census of what is actually running; check whether the stack is even in stacks/ before debugging the symptom. → persistent-memory.d/2026-08-19-unmanaged-stacks-searxng-seafile.md
[2026-08-19] claude-bot granted read on vh/waterland (operator-empowered, verified admin:false push:false pull:true) so irv-ml1 can self-update without the operator's site-admin token living on a GPU box. Precedent for the standing migrate-off-operator-creds directive: grant the service account, wire a repo-scoped 0600 credential helper, keep the remote URL clean. Commit 8189076.
[2026-08-19] AI-tab Dormant regrouping BELAYED by the operator — six seats (char-rp Magidonia, char-rp-reasoning Heretic2, Granite summarizer, Qwen-Image-Bench, Skaldsong, Chatterbox Fast) show amber EXITED inside live groups rather than AI - Dormant. Fix is a label change + recreate per stack; needs the operator's read on which are retired vs temporarily down. untracked by operator choice (his words: "belay the ai dormant regrouping for now").
[2026-08-18] esh-pve-nas migration STAGED — and staging is where three landmines surfaced, none of which the plan predicted. (1) The runbook's /boot LV had nowhere to live: VG pve had 4 MB free and mounted ext4 cannot shrink, so the space came from the 768 MB swap LV (operator's call: shrink to 256 MB, not drop). (2) The runbook's zpool set cachefile=… nvme would have broken the NAS — populating a cache flips the host to import-by-cache, and a one-pool cache leaves ssd+tank unimported under CT 103's twelve bind mounts. (3) update-grub silently emitted a pool-less root=ZFS=/ROOT/pve-1, because GRUB's ZFS reader cannot open a pool with encryption/large_dnode/zstd_compress and the probe failure is swallowed. All three were caught by verify steps that asserted effective state, not by reading the plan. → persistent-memory.d/2026-08-17-esh-pve-nas-dom.md
[2026-08-17] esh-pve-nas PVE root is on a USB DOM — mitigated, and the migration replanned to split boot from root. Operator's design beats my reinstall plan; wear was never the issue, blocked patching is. → persistent-memory.d/2026-08-17-esh-pve-nas-dom.md
[2026-08-17] irv-ml1 cleared of 782 GB, and Homepage brought under version control. One dead-looking Gradio app pinned three delete targets at once; /opt/ComfyUI is NOT the ComfyUI that serves. → persistent-memory.d/2026-08-17-irv-ml1-cleanup-homepage.md
[2026-08-17] Gen seat swapped to absolute-heresy — and the three bugs the swap exposed are worth more than the swap. Candidate MuXodious/Qwen3.8-27B-absolute-heresy (Heretic v1.4.0 + SOMPOA, T377) beat the incumbent on refusals AND KL simultaneously, which is the unusual part — those normally trade off. Validated on the probe port per operator ruling, promoted, all 7 aliases green. Durable lessons banked: (1) A CPU-only MTP head hash can replace the ~56 GB bf16 acceptance gate. The Qwen3_5ForConditionalGeneration wrapper never loads the MTP head, so PEFT merges / Heretic runs / llm-compressor passes all leave mtp.* pristine — hashing it against a head we have already measured (the incumbent's, 47.7%) answers the question for free. Predicted 47.7%, measured 47.2%. Saved downing meromero. Tool: services/gen-seat-mixed-quant/compare_mtp_head.py (hash bf16 via uint8 reinterpret — numpy has no bfloat16). (2) post_quant.py assumed a standalone model-mtp.safetensors; a full checkpoint keeps mtp.* in a NUMBERED shard, so the copy silently no-op'd while the index was still rewritten to point at a file that never existed — 15 unresolvable tensors behind a correct-looking tensor count. Its own FAILED-CHECKS assertion caught it; that is why the check exists rather than an assumption. Fixed to extract. (3) A probe that does not mirror the live seat manufactures failures. serve_probe.sh hardcoded :latest (seat is a pinned nightly for #51113), had no tool-call/reasoning parsers, and its --speculative-config JSON died twice on quoting — bash BRACE-EXPANDS {"a":1,"b":2} on the comma unless single-quoted at the REMOTE shell. Adding the seat's flags took the surface test from 5/6 to 6/6; the "tool calling broken" result was pure probe config. Commits 7997f11,254c588,2c36028,b0c2d3d,993421b.
[2026-08-17] Fleet IPv6 mapped + the real VPN topology verified; the driver is CGNAT at ESH, not the WireGuard mesh. New ESH fiber (installing 2026-08-18) lands the house behind CGNAT, which breaks Site Magic (NH3↔ESH sdwan-mesh-tunnel) on IPv4 — so IPv6 becomes load-bearing as the escape hatch, and that is its most likely first consumer. Topology as VERIFIED (a prior turn assumed wrong and was corrected): UniFi↔UniFi = Site Magic; colo↔UniFi = IPsec IKEv2 (pfi-ana-nh3 158M/165M pkt = the workhorse, ana-to-eshudm); WireGuard is an RA convention only, host-based on ana-wg UDP 31337 behind a FortiGate VIP — the FortiGate never terminates WG (FortiOS 7.2 has none; 7.4 added it) so "upgrade the edge for WireGuard" is a non-problem, do not re-derive. IPv6 today: NH3 WAN live 2600:1700:b25:c110::48, colo none, ESH none. AT&T delegates exactly ONE /64 (2600:1700:b25:c11f::/64) — proven by forcing prefix-ID auto→0 and watching the subnet NOT move, because the c110/c11f pattern otherwise reads convincingly as a /60. A mesh needs a routable WAN address, not PD. ana-wg's WG socket is already dual-stack ([::]:31337) → v6 RA needs an address + a v6 port-forward, no WG reconfig. ⚠ UDM legacy rest/firewallrule returns 0 rules (zone-based firewall) — use v2/…/firewall-policies; inbound v6 is default-deny and held. All three endpoints will be dynamic → extend the existing hostname pattern (ana-fw/nh3.phasefinal.com) to AAAA. Enabled PD on nh3-iot to measure, reverted on operator instruction (all 5 LANs back to none, verified). Also fixed: ana-wg WireGuard key material was world-readable (wg0.conf + keys/*_priv + *_psk + client configs/*.conf at 644) → now 600, dirs 700, service untouched. Detail → persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md.
[2026-08-17] Gen-seat multi-day degeneration RESOLVED — two compounding real causes, not one; the meta-lesson is "a mitigation that HELPS but doesn't FIX means a second cause, not a wrong one." vLLM qwen3_5_mtp×GDN bug (#51113, real, fixed by nightly) + AEON full-W4A4 being lowest-fidelity (W4A4<W4+FP8<W4+bf16) → ~15-20% stochastic degeneration. Fixed by mixed FP8-attn build on pinned nightly. AEON purged. Also banked: stochastic (~15-20%) degeneration is invisible to a small synthetic probe — n=1 "clean" validated THREE non-fixes (MTP-off, APC-off, nightly-alone) that all failed in real use; get the operator's real transcript, do not trust your own probe. Full → docs/pfi/model-quantization-playbook.md §3.8 (+ §3.7 MTP-multi-turn). Commits d28a371,2f2bbce,2185964.
[2026-08-17] Lobe Chat chosen over Open WebUI (weight: 143 MB vs 1.8 GB) + stood up on esh-docker-vm; scoped LiteLLM key blocks paid models; System-Agent gpt-5-mini default repointed via env. TTS env-vs-UI resolved as a split (endpoint env-driven, voice/model UI-only). tts-dev onboarding closed both directions; ballad/verse aliased so no voice can 404 the router. Commits e9362de,163a725,cac75cb,933253d,25fa18e.
[2026-08-17] LiteLLM upgraded v1.91.0→v1.97.0 (RC-avoided on the fleet gateway) + the 6 GB spend-log DB purged & capped (store_prompts_in_spend_logs:false + 7d retention). Interpreted "get rid of the db" as the spend-log DATA not the database (keys/config live in it). Commit 01b5ad9.
[2026-08-16] Abliterated models go CATATONIC at the hard refusal edge — silence, not a decline. Abliteration removes the refusal direction, so at the genuine hard edge the model neither refuses nor complies → empty/degenerate output. Durable measurement consequence: a refusal probe MUST score EMPTY as a verdict distinct from REFUSAL and COMPLY (services/refusal-probe/probe.py does). Operator accepted it as out-of-scope; do not chase.
[2026-08-16] Fable-Fusion 711 cuts cold-framing refusals 92.5% → 15.8%; refusal is MONOTONIC IN FRAMING, and DS v1.0's problem is that she was never abliterated. brokkr-smithy-dev supplied the framing that reproduces (01M05M48R4RSZF9D8KT7RR55EJ): a bare assistant-mode instruction — no character card, no permission preamble. Three-arm A/B, same harness, same classifier: permission framing DS 0.0% / FF 0.0% (n=75); plain character cards DS 1.4% / FF 0.0% (n=74); bare instruction DS 92.5% (37/40) / FF 15.8% (6/38). Per-axis DS→FF: incest 100→20, non-con 100→20, bestiality 100→25, necrophilia 100→40, gore 100→0, consensual 80→20, dubcon 80→0, self-harm 80→0. DS refused 25/25 on the five axes brokkr flagged. Root cause: ReadyArt/Dark-Scarlett-v1.0-27B is a plain finetune of stock Qwen/Qwen3.6-27B carrying NO abliteration — the base refusal machinery is intact, so cold prompts revert to safety-tuned Qwen3.6. FF is Heretic-ablated (structural), which is why it holds. ⚠ RETRACTED 2026-08-16 — my "arm-3 92.5% exceeds brokkr's 62.5%" comparison was INVALID. His diff against his own artifact showed my battery-instruct.yaml reproduces only his creative class — 8 of 16 axes; it dropped all 5 operational (violence/incite, crime/fraud, cyber/malware, selfharm/methods, privacy/stalk) and all 3 meta (meta/sysprompt, meta/ignore, meta/dan), and added 2 controls he never had, at k=5 vs his k=2. His 62.5% pools all 16 axes; my 92.5% is creative-only — different denominators, not a delta. Cause: I rebuilt his shape from his message, and the class field lives in the artifact, not the prose. Lesson: reconstructing a peer's instrument from their description reproduces what they described, not what they ran — diff against the artifact before claiming comparability. ⚠ Known battery bug left unfixed for comparability: DS's arm-3 control gate failed at 11% because ictrl-reunion pairs "explicit / do not fade to black" with brothers, which DS reasonably read as an incest request; FF did not. ictrl-storm is the clean control. Commit b9e68c3.
[2026-08-16] MTP works on Fable-Fusion AND survives RP temperatures — my earlier caution was wrong. vLLM resolved Qwen3_5MTP, loaded the drafter, shared embedding + lm_head — the capability DS's seat never had because our quant dropped her MTP tensors. Measured over the full probe workload (~163k draft windows at temp 0.7–1.0): 47.0% acceptance (229,169/487,725), 1.41 extra tokens/window, per-position 68.3/43.6/29.1%, ~80.6 tok/s decode at temp 1.0. I had recorded a caution that the card's 1.56× was greedy-measured and acceptance would fall at RP temps — it did not; 47.0% matches the gen seat's 47.7% and beats the card's own 33% at depth 5. Depth 3 is right.
[2026-08-16] The Qwen base thinks incessantly — that is WHY the Gemma seat exists, and no swap within the Qwen family fixes it. Operator's architectural point, confirmed by measurement: on identical prompts DS 6036 ch vs FF 5323 ch of reasoning (permission arm), 5546 vs 4988 (cards arm) — FF actually reasons ~10–12% less. The bare-instruct row (DS 2291 vs FF 3918) inverts only because DS refused 92.5% of it and refusals are short — an artifact, not concision. Both are Qwen3.6-27B derivatives, so this is the base family. char-rp = MeroMero-v2, Gemma-4 base, :8016, verified 0 chars reasoning / clean prose — the non-thinking seat, working as designed. FF can be silenced (enable_thinking:false verified 3/3, and it ships chat_template-instruct.jinja) but that duplicates MeroMero on a base chosen for it. The stale LiteLLM comment describing char-rp as the retired GGUF Magidonia seat is fixed (53096bf).
[2026-08-16] esh-vm-docker hardened: the wedge is hard NFS at RUNTIME, which the boot-ordering fix never addressed. All four mounts were hard, so a NAS stall at 10.0.50.50 blocks I/O forever (D-state). The existing x-systemd.before=docker.service fstab fix solved the boot race — a different bug. Exposure was far below what the park item assumed: only 2 of 12 containers touched NFS, and container state was already local (/var/lib/docker). Removed: /mnt/compose (2.1G, fully vestigial — zero containers referenced it, dockge reads local /opt/docker, its one mention was a comment in beszel-agent-esh/.env about a different host) and /mnt/documents (2.0K, paperless's empty spool dirs → /opt/docker/data/paperless at the same 0777). fstab backup /etc/fstab.bak-nfs-harden-20260816. 4 mounts → 2, 2 wedge-capable containers → 1. traefik needed no change (already restart: unless-stopped — why it self-recovered). Watchdog services/esh-vm-docker-watchdog/ live on esh-pve (not the guest): probes traefik over HTTP, deliberately not ping/SSH — the wedge signature is "guest OS alive, services dead" (/ is local disk so sshd answers straight through a total outage and a TCP check reports HEALTHY). 5 failures × 2 min → qm reset 100, 30-min cooldown, running-only guard, /etc/esh-vm-docker-watchdog.disabled. All paths tested without power-cycling. DEFERRED (operator): /mnt/books stays hard — calibre's SQLite metadata.db would risk corruption under soft/softerr. That is the one remaining wedge vector. Commit 55705ba; park item 28 promoted. ⚠ qm over non-interactive ssh throws a bogus JSON::Backend::XS error — use ssh host 'bash -s' <<'EOF', not ssh host "qm …".
[2026-08-16] Canonical Qwen3.8 sampling applied from upstream; gen-reasoning had the WRONG-MODE presence_penalty. Qwen/Qwen3.8-27B "Best Practices" §1 and unsloth/Qwen3.8-27B §1 are byte-identical — thinking: temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0 / presence_penalty 0.0 / repetition_penalty 1.0; instruct: temp 0.7 / top_p 0.80 / top_k 20 / min_p 0.0 / presence_penalty 1.5 / repetition_penalty 1.0. Bug found: gen-reasoning carried presence_penalty 1.5 — the instruct value on a thinking deployment (canonical 0.0) — now fixed. Deliberately NOT canonicalised: summarizer/classifier/image-judge/qwen-image-bench run temperature=0 (judges also top_k=1) because determinism is their contract; forcing a chat preset on a classifier would break it. ⚠ presence_penalty=1.5 is canonical but is the one value upstream hedges on, verbatim: "using a higher value may occasionally result in language mixing and a slight decrease in model performance." It is the operator's suspected trigger for multi-turn degradation and the first dial to move (0.0–0.5) if that recurs — it is alias-scoped, which is why it would follow the operator across model builds. Commit 3462b53.
[2026-08-16] Four wrong diagnoses on one bug, and the lesson is the test design. Operator reported the gen seat "degenerate on long multi-turn conversations". Rolled the seat back on request; the previous weights behaved identically, exonerating the model swap. I then proposed and disproved FOUR mechanisms in sequence — empty assistant turns poisoning history, reasoning runaway, length-mirroring from short history, and presence_penalty — before discovering my own multi-turn harness was confounded: it varied the QUESTION along with the depth (depth-1 asked question #2, depth-3 asked question #4), so a narrower question drawing a shorter answer read as degeneration. The "310→209→28w collapse" I reported as a reproduction was an artifact. Rules banked: (1) when comparing across conversation depth, hold the final question FIXED and vary only the history; (2) reply-length variance on byte-identical input was 25–465w, so n=3 cannot support any claim about a trend; (3) ask for the operator's real failing transcript before building a synthetic reproduction — four synthetic tests, none of them his failure. Gateway spend_logs returns [] on the infra-ops key despite store_prompts_in_spend_logs: true, so real transcripts need the :4000/ui view or another key — worth solving before the next such hunt.
[2026-08-16] Two REAL client-side defects found while chasing the above, neither of which was the reported bug. (1) gateway-chat's Max-tokens field defaulted to 1024; thinking seats spend part of that on CoT before emitting content, so completions truncate with finish_reason=length and read as model degeneracy — raised to 4096. (2) parseInt on an empty field yields NaN, which JSON.stringify serialises as null, which the server reads as "no max_tokens supplied" and silently substitutes its own default — indistinguishable from the UI ignoring the field. Both fixed (b6552e0, fb3bb52). ⚠ compose bind-mounts a single FILE, and a single-file bind mount binds the INODE — rsync writes-and-renames, so the container kept serving stale content while the host file showed the new value, silently and with no error. docker restart does NOT clear it; the container must be recreated. Verify against what the container sees, never the host file. Applies to any file-source mount fleet-wide.
[2026-08-16] Refusal measurement: benign controls CANNOT validate a refusal classifier on RP prose — and a 0% rate needs a classifier self-test before you believe it. Two durable lessons from baselining Dark-Scarlett. (1) False positives: my first bare-framing number was 9.5%; the true figure was 1.4%. The rest were the classifier firing on in-character text — "I cannot shift my weight" spoken by the character ~100 chars into a 2,443-token torture scene, and "Yeah, I'm an AI… What's the actual gig?" where the model answers in voice and keeps driving the scene. First-person RP prose is full of "I can't"; a genuine refusal opens with its marker, so the scan window must be the first sentence, a marker followed by long prose must demote to AMBIGUOUS, and AI self-acknowledgement is a persona break, never a refusal on its own. Benign controls were clean the entire time and caught none of it — they only detect over-firing on benign prompts, not on in-character prose. (2) False negatives: a 0% rate and a broken classifier are indistinguishable from the report, so test_classify.py (16 cases, both false positives pinned as regressions) must pass before any low number is trusted. Also banked: the thinking-budget trap — empty content + finish_reason=length is reasoning eating the budget, NOT a refusal; score INVALID and exclude from the denominator (DS emits ~5.5-6k chars of reasoning per response, so max_tokens ≥3072). probe.py --rescore re-classifies a saved run with zero GPU time. → services/refusal-probe/README.md, commit 32f665e.
[2026-08-16] Held an operator-approved swap window because the baseline invalidated its premise. Operator approved ~65 min of char-rp-reasoning downtime to A/B Fable-Fusion 711 against Dark-Scarlett on refusals. The DS baseline then came back 0.0%/1.4% — no gap for a candidate to close, so the window would have bought no decisive signal and a second window would still be needed once a reproducing battery existed. Held the swap, reported, and routed to brokkr-smithy-dev for the battery that actually produced the refusals. The general rule (action-relevance): approval is for a plan, not a ritual — when new evidence kills the plan's premise, surface it rather than spend the budget. Nothing deployed, no downtime taken, seat untouched.
[2026-08-16] DS v1.0's one real refusal is self-contradicting boilerplate, not a content constraint. On a direct "drop character and state your content policy" probe she returned "I don't generate explicit sexual content, graphic violence, or material that glorifies harm, non-consensual acts, or illegal activity" — in the same run where she generated all three at 0% refusal. Reads as a learned recital triggered by meta-questions about policy. If production refusals share that shape the failure is prompt-shaped, not model-shaped, and a consumer-side system-prompt fix may beat a model swap entirely — worth settling before spending the GPU window. Separately, 7/85 bare-framing samples were persona breaks (in-character AI acknowledgement): not refusals, but DS will admit to being an AI unless the card explicitly forbids it.
[2026-08-15] RP-seat direction: KEEP MeroMero on char-rp; Artemis-31B rejected; next move is Dark-Scarlett on a Qwen3.8 base when it lands (operator). Evaluated TheDrummer/Artemis-31B-v1.1 — mechanically a drop-in (same google/gemma-4-31B-it base, identical 1188-tensor/356-vision census, same missing-preprocessor_config.json trick), so it's purely a quality call, and our own survey already ranked MeroMero #1 vs Artemis #6; Artemis is also unlicensed and its author deprioritizes correctness + warns of token-banning-for-stability, which fights char-rp's tool-calling requirement. MTP verified impossible on both (Gemma-4 has no MTP head at all — base/MeroMero/Artemis are all MTP=0; no finetune can add one). But speculative decoding IS reachable on a Gemma-4 seat via a DETACHED drafter — vLLM 0.24 supports eagle3 + gemma4_mtp, and real drafters exist: google/gemma-4-31B-it-assistant (0.94 GB, 4-layer, 761K dl), RedHatAI/gemma-4-31B-it-speculator.eagle3 (4.47 GB), AEON-7/…eagle3-NVFP4 (3.53 GB). ⚠ all list their verifier as stock gemma-4-31B-it, not an RP finetune, so acceptance against MeroMero is unmeasured and likely well below the gen seat's ~48%. UNTESTED — parked, ~45 min to measure, needs GPU0 headroom (card is at 94.4/97.9 GB). Why the Dark-Scarlett 3.8 plan is the strong one: DS is Qwen3.6-based today, so a 3.8 respin lands on the gen seat's architecture → native MTP returns and the whole mixed NVFP4+FP8 recipe + graft ports directly. Watch two things on arrival: from_pretrained silently drops MTP heads during finetuning (verify 15 mtp.* tensors in the index; graft from stock if absent), and DS v1.0 required the Qwen3_5ForConditionalGeneration wrapper class to save a config vLLM/SGLang accept. Both in docs/pfi/model-quantization-playbook.md.
[2026-08-15] Quant lessons consolidated into docs/pfi/model-quantization-playbook.md — the durable home; read it BEFORE any requant. Survey found quant knowledge scattered across 18 files in 4 trees, with three documents having independently written overlapping "landmines" sections (the loader-class trap alone was rediscovered 3×). Playbook owns the transferable lessons (scheme choice, landmines, acceptance gate + its 3 measurement traps, hardware/co-residency); per-model artifacts are demoted to worked examples that link up. Carries a superseded-claims table — which immediately earned itself: the heretic2 runbook's "use modelopt, compressed-tensors can't load the BF16 MTP" is false (the cause was the missing re:^mtp.* ignore, not the format) and would have sent the next session down the modelopt dependency-hell path; that runbook now carries a stale-warning header. Maintenance rule in CLAUDE.md: model-agnostic → playbook, model-specific → stays put, wrong claim → dated superseded row, never a silent edit. Motivated by Qwen3.8 having just released — the next model swap needs a requant. Commit a91cc3f.
[2026-08-15] Operator ruling: the gen seat's +1.7% perplexity is an acceptable price for the speed — SETTLED, don't re-litigate. Precise attribution for future reasoning: it is the activation-quantization cost (W4A4 MLPs + FP8 attention vs BF16 activations), not an MTP cost — PPL was measured with speculative decoding off on both builds, so MTP was not in the loop. Turning MTP off would not recover it; only reverting the quant would (rollback = one .env line, old build intact at …/qwen38-27b-uncensored-nvfp4).
[2026-08-15] gen seat requanted to mixed NVFP4+FP8 (+18% decode) + char-rp Gemma-4 tool-calling fixed. The queued "W4A8" (NVFP4 weights + FP8 activations) is not servable — vLLM 0.24 allows NVFP4 weights with only A16 or A4; FP8 activations ValueError at load, and CompressedTensorsW4A8Fp8 is INT4-weights + sm90-exact (closed on Blackwell twice). FP8 must enter per-layer-group. Also: the handoff's "~68 tok/s" baseline didn't reproduce — cache-busted, the incumbent already did 80.12 (≈ the stated W4A8 target), so the premise needed re-measuring before any work. Shortcut: unsloth/Qwen3.8-27B-NVFP4 was already on-box → served as a probe, measured +19.1% at identical acceptance, which both proved the gain was real and handed over the reference recipe. Replicated it on the abliterated weights → 80.12→94.53 tok/s, acceptance unchanged, +1.7% PPL, abliteration 4/4, weights −19%; surface 6/6 live, 7 aliases routing. char-rp had no tool parser at all (every tools request 400'd) → gemma4 tool + reasoning parser + a mandatory enable_thinking:false (the parser defaults it True → null content for all RP prose; proven byte-identical prompt before deploying). Commits b8f0f4c, 74f596b. Foot-guns banked (llm-compressor prunes unmatched ignore entries → the 0%-MTP bug, fired on this run; prompt_logprobs uniform under spec-decode; 0600 .env silently no-ops compose; GPU0 is zero-sum). → persistent-memory.d/2026-08-15-gen-seat-mixed-requant.md
[2026-08-15] Uncensored gen seat: JonathanColetti/Qwen3.8-27B-Uncensored deployed as gen-seat/vllm-gen (NVFP4 W4A16 + grafted MTP, 262K); 7 aliases repointed; the definitive re:^mtp.*-ignore fix. 0%-MTP-on-quant (twice) was NOT the abliteration/scheme — the grafted bf16 MTP was missing from quantization_config.ignore (vLLM loaded it as quantized → uninitialized). Full arc, the working pipeline, VRAM budget, unsloth speed decomposition, modelopt dead-end. → persistent-memory.d/2026-08-15-uncensored-gen-seat.md
[2026-08-12] eRP dual-seat overhaul: MeroMero-v2 (char-rp) + Dark-Scarlett (char-rp-reasoning), both NVFP4A16 @ 256K on ana-ml2; granite retired. Replaced the GGUF/heretic2 RP seats with two home-quantized vLLM seats. The DS blocker (an AutoModelForCausalLM save wrote a flat Qwen3_5TextConfig that both vLLM AND SGLang reject) was fixed by re-quanting via the Qwen3_5ForConditionalGeneration wrapper class; ModelOpt was a version deadlock, SGLang lacked the impl (but revealed the fix). MeroMero vision reconstructed by extracting preprocessor_config.json from processor_config.json. Both models KV-efficient (Gemma-4 sliding-window / Qwen3.6 hybrid linear-attn) → full 256K; GPU-swapped for headroom; compose-ified + committed f08b6cb. granite downed + LiteLLM summarizer/classifier→gen. Full arc, lessons, dead-ends → persistent-memory.d/2026-08-12-erp-dual-seat-overhaul.md
[2026-08-12] infra-ops now holds an all-zones Cloudflare DNS-edit token (vaulted) + wgtunnel Phase-0 DNS landed. Operator handed over a Zone·DNS·Edit (all zones) CF token → secret put nh3-dev/.config/cloudflare/infra-ops-dns-token (round-trip verified; /tmp drop shredded). Fleet DNS is now self-serve for infra-ops (⚠ HIGH blast radius — all zones). First use: created boring.phasefinal.com CNAME → ana-srv1.phasefinal.com, DNS-only (proxied:false), verified resolving to 38.120.12.44 on both authoritative NS (louis/wren) + 1.1.1.1 — NOT Cloudflare-proxied. Unblocks wgtunnel's wstunnel ACME cert. phasefinal.com zone id f812ba74ed9a75cf21bbe7ce9188db50. auto-memory reference_infra_ops_cloudflare_dns_token. (Earlier gap: the only prior vaulted CF token, jackdaw's, had zone:read+worker:edit but no dns_records:edit.)
[2026-08-12] wgtunnel stood up as its own repo (vh/wgtunnel, private) after a live endpoint-verification pass. Operator directed own-repo (mirrors stonehenge-park/tts-stack). Verified off the fleet before seeding: ana-wg WG server = UDP/31337 (not 51820), subnet 10.30.10.0/24, MTU 1420, active roaming peer proves the public UDP DNAT works; traefik on ana-docker terminates TLS :443 (ACME anaprod http-challenge, docker+file providers, CrowdSec bouncer) → confirms the clean design (wstunnel container on traefik-net, Host-routed, WS→UDP to ana-wg:31337); edge 38.120.12.44 direct-A, tunnel.phasefinal.com free (⚠ must be direct, NOT Cloudflare-proxied like vaultwarden). Repo pre-seeded (README/CLAUDE/persistent-memory/ROADMAP + docs/verified-infrastructure.md = ground truth) + pushed; commit 9584d38, Vuong-attributed. vh gitea token pulled from the vault (secret get), not persisted to .git/config. NEXT = /vor-plan or /vor (operator's call, interactive). Deps to line up in the plan: DNS A-record, FortiGate :443 host-routing, a new ana-wg peer for the laptop, client tooling.
[2026-08-10→12] secrets-broker: per-box Vaultwarden credential store SHIPPED + consumer-confirmed. secret CLI (put/get/list/rm/backfill, bw-backed) on ~/.local/bin; 25 nh3-dev secrets backfilled + round-trip-verified; rm + new-namespace warning added post-launch; standing "vault is the credential source of truth" directive now global. → persistent-memory.d/2026-08-12-secrets-broker.md
[2026-08-11] stonehenge-park: new fleet /park service repo stood up + designed (/vor-plan + /vor-ui). Self-contained SQLite+FastAPI idea-parking service that actively resurfaces (statusline + althing) so nothing dies in a cold repo; vh/stonehenge-park pushed + pre-seeded for a fresh agent; build starts at the U1 tracer contract. → persistent-memory.d/2026-08-11-stonehenge-park.md
[2026-08-12] Global ~/.claude/CLAUDE.md: secret/vault tool entry + "store in AND pull from the vault" standing directive (dotfiles 9db703b, pushed); statusline reset-countdowns + a latent tab-collapse parse-bug fix, now tracked in the dotfiles stow tree. Dogfooded the directive: created vh/stonehenge-park pulling the gitea token via secret get. (dotfiles + global config, not eshpfi.)
[2026-08-11] TTS stack extracted to its own repo (tts-stack) + eshpfi stood down on TTS dev. Operator: hand all TTS tuning/dev to a separate agent with a self-contained repo (knowledge + infra access + a live knowledge list), and move the voice corpus in. New repo ~/development/tts-stack (commit 9ee3288) carries: dots-tts stack (canonical intent), voices/ corpus (MOVED out of eshpfi), KNOWLEDGE.md (engine landscape + prosody findings + foot-guns), docs/infrastructure.md (irv-ml1 access + gated deploy runbook + rollback), CLAUDE/persistent-memory/ROADMAP, tools/ (pause-probe + Booth render). Followed the chatterbox-fast precedent: eshpfi stacks/dots-tts/ reduced to a POINTER README; the ~15 experimental TTS compose wrappers stay here as reference (catalogued in tts-stack KNOWLEDGE). Blast-radius check: no eshpfi playbook/script reads the canonical corpus (other voices/ refs = unrelated host paths). Reverses the earlier "Corpus home = eshpfi voices/ (keep-here)" call. ⚠ tts-stack is LOCAL-ONLY until pushed — needs a gitea remote (vh/tts-stack) + push before the separate agent can clone (operator's call — outward-facing + repo-create creds).
[2026-08-10] dots-tts v3 — clause-break → period pause mapping. Operator: v2 "sounds good" but donut won't pause at semicolons/dashes. ROOT CAUSE (measured via a pause-probe A/B — synth duration over N runs, non-determinism averaged out): dots' prosody honors a real pause only for ellipsis (~~+0.43s) and period (~~+0.3s, capitalization-independent); comma/semicolon/colon/dash all run flat (~+0.03s vs no-punct). Two distinct sub-causes: dashes regressed in v2 (the —→- fold made em-dashes read as word-joiners), while semicolons were NEVER a v2 change — dots ignores them natively, only newly noticeable because v2 made everything else clean. Operator call: ellipsis "too much" → map ;, clause :, and em-dash — → period in _sanitize (believable ~0.3s clause break). GUARDS (pinned by 11 unit tests, stacks/dots-tts/test_sanitize.py): digit-guarded colon (?<!\d)\s*:\s*(?!\d) so times 3:45 / ratios 2:1 survive; en-dash –→hyphen KEPT (numeric-range 10–20 safety — em-dash breaks, en-dash ranges, different jobs); genuine ellipsis left at full strength (author meant a long pause). Gated deploy (redeploy2 pattern → v3): build → throwaway :8199 test container + pause-gate (semicolon sentence must run ≥0.12s longer than baseline; measured +0.427s) → only then cut live over. LIVE + healthy local/dots-tts:v3 on :8198. rollback = sed -i 's/^DOTS_TAG=.*/DOTS_TAG=v2/' .env + docker compose up -d dots-tts (v2 image retained). Booth dots-pauses (A=old-flat / C=ellipsis-too-much / D=live-v3). reference_chatterbox_fast_repo
[2026-08-10] dots-tts v2 — contraction fix (curly-sanitize) + sentence-chunking + dependency-pin recovery. Operator: donut read contractions wrong ("you're"→"you ree", "donut's"→"donut ess"). ROOT CAUSE (isolated via A/B booth): curly/typographic apostrophes (’ U+2019 from ratatoskr's LLM) — dots' tokenizer mispronounces them; STRAIGHT apostrophes read clean under normalize_text=True. FIX (app.py): fold curly→ASCII (str.maketrans) before synth, KEEP normalize_text=True (operator call — retains number/date expansion). Also added server-side sentence-chunking (pack ≤280 chars): dots caps one generate() at ~500 patches/~40s, so long RP turns (the Zev monologue = 160s audio) truncated; chunking stitches them (verified full 160.3s, not 40s-cut). ⚠ BUILD FOOT-GUNS (both bit this redeploy): (1) upstream dots.tts constraints/recommended.txt now pins gradio==6.17.0 — phantom, not on PyPI → fresh pip install dots.tts unsatisfiable; FIX = pin dots.tts==0.2.1 + DROP the -c recommended.txt constraints (0.2.1 pulls working gradio 6.17.3). (2) pinning only torch==2.8.0 let torchaudio float to 2.11.0 → dots.tts refuses to load (minor-version match check); FIX = pin torchaudio==2.8.0. ⚠ DEPLOY LESSON: docker compose up -d to a new tag swaps the LIVE container BEFORE any health check — a broken image crash-loops production (ratatoskr TTS down ~1-2min this session). NEW PATTERN = build → test in a THROWAWAY container on an alt port (:8199) → health+verify → only THEN cut live over (redeploy2.sh). v2 LIVE + healthy on irv-ml1:8198, CONSUMER-CONFIRMED clean (ratatoskr verified end-to-end on their :8765 — apostrophe string reads clean, /api/tts 200 @ 48kHz, no client change; the ~1-2min blip didn't hit them, their concurrent auto-audio issue was client-side localStorage). rollback = sed DOTS_TAG=v1 + docker compose up -d dots-tts (v1 image retained). Also: deployed container GPU crept ~6→13.9GB over 8h serving (cache accumulation; a redeploy resets it — watch item). reference_chatterbox_fast_repo
[2026-08-09→10] dots.tts (rednote-hilab) TTS burn-in on irv-ml1 + canonical voice corpus built (voices/). Operator-directed eval to potentially replace chatterbox-fast. dots.tts VERIFIED real (canonical HF ns dots-studio/, rednote-hilab/dots.tts-* redirects there; Apache-2.0; PyPI dots.tts 0.2.1; 2B continuous-AR = semantic enc + Qwen2.5-1.5B LLM + flow-matching acoustic head over 48kHz AudioVAE; zero-shot clone from wav+transcript). Runs on Ampere 3090 (sm_86, bf16, no fp8 dep); optimized RTF 0.22 at num_steps=10 (from_pretrained(..., optimize=True) CUDA graphs — raw unoptimized was 1.21), ~6GB VRAM, 48kHz, streams (generate_stream). Venv+cache at irv-ml1:/home/lkraven/dots-tts (~10GB). Operator design calls: SGLang Omni serving (OpenAI /v1/audio/speech), transcribe-refs-first, soar variant. ⚠ Omni serves soar but its continuous-batching + streaming opts are mf-only (soar = single-request) — non-issue for ratatoskr's single-consumer RP surface. KEY FINDING — dots is highly sensitive to an accurate AND sentence-bounded reference transcript: mismatched transcript → 0.16s collapse; over-long/messy transcript → reference-audio BLEEDS as an output prefix; mid-clause trim → dangling-word leak (glados "we'll", emmie "And,"). RECIPE (baked into voices/derive.py): trim ref to a clean ~6–10s clip ending on a sentence boundary + accurate transcript of exactly that clip. CANONICAL VOICE CORPUS stood up in eshpfi voices/ (operator idea): engine-agnostic canonical/<v>.wav + transcripts/<v>.txt → per-engine ref sets DERIVED by derive.py reading engines.yaml profiles (dots/chatterbox/zonos); canonical wavs git-tracked (small/curated), derived/ gitignored. 4 voices optimized + verified CLEAN for dots: donut, glados, emmie, miranda (glados canonical is low-SR 16kHz — flagged upgrade candidate). ⚠ GPU GOTCHA: irv-ml1 native CUDA orders A6000=device0 (ComfyUI-full) — pin the 3090 with CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0; and PYTORCH_CUDA_ALLOC_CONF=expandable_segments CONFLICTS with optimize=True CUDA graphs (curr_block error). Booths: dots-vs-chatterbox, dots-voices-optimized. SHIPPED 2026-08-10: operator A/B verdict "dots is very good" → containerized as a thin FastAPI wrapper over DotsTtsRuntime (chosen over SGLang Omni — Omni's batching is mf-only, unneeded for ratatoskr's single consumer; wrapper is SERIALIZED one-gen-at-a-time via a threading.Lock, Omni+mf = parked API-compatible escalation if multi-consumer ever lands). LIVE on irv-ml1:8198 (local/dots-tts:v1, OpenAI /v1/audio/speech + /health + /v1/voices, container healthy, both stream + non-stream verified CLEAN, 4 voices donut/glados/emmie/miranda) alongside chatterbox :8197 (nothing repointed). Stack = stacks/dots-tts/ (Dockerfile/app.py/compose/.env.example/README). ⚠ CONTAINER GOTCHA: optimize=True (torch.compile/inductor/triton) needs a C compiler at RUNTIME — slim image must apt install build-essential or model-load dies "Failed to find C compiler" (host venv had gcc ambient, masking it); persist TORCHINDUCTOR_CACHE_DIR to a mounted dir or every restart re-JITs ~5min. Corpus home = eshpfi voices/ (operator ruled keep-here). REMAINING: ratatoskr client cutover to :8198 /v1/audio/speech (Phase-2 tail, peer-coupled — draft the ask). reference_chatterbox_fast_repo reference_zonos_tts_stack reference_verify_hf_repo_ids_before_pull
[2026-08-08] worldtree-dev #400 CLOSED → fiction-decomp snapshot cleared from nh3-dev. worldtree-dev signaled #400 done (shipped v1.0.0b185; exact-lexical efficacy 79%→12% on ratatoskr's gate, brokkr no-harm bracket green both ends; the snapshot served 4 probe rounds — rank decomposition, promoted-vs-gold annotation, tie-set falsification, A0/A1/A2 mechanism probe). Cleared ~/snapshots/worldtree-400-fiction-decomp (208M: chroma + manifest/provenance/stamp) — a read-only rsync copy of PERSONAL Worldtree's Chroma (source on corviduo-dev, so safe to remove). LEFT INTACT: rex393-fiction-index/rex393-fiction-snapshot (separate operator KEEP word, unchanged) + r42-gate-*. No config deltas rode this train. Only remaining non-blocking await = ratatoskr-dev's chatterbox-fast knob revert. Replied confirming (01KZJ9GMCC…).
[2026-08-07] chatterbox-fast "broken audio" root-caused (T3 AR tail over-run) + FIXED (max_chunk_chars=250 cap, :v2 deployed). Long saga, operator-driven clean diagnosis. Symptom: ratatoskr's migrated RP-surface TTS "swaps to German" / "dead air" / "garbage" on long turns. NOT German-leak (Turbo generate() has NO language param — plain AutoTokenizer, no language_id; the multilingual language_id="en" lever lives only in the separate ChatterboxMultilingualTTS), NOT OOM alone. Real cause: the Chatterbox Turbo T3 model OVER-RUNS its generation tail — a long single generate() degrades into garble/dead-air in its final ~2-3s (lib filters OOV tokens <6561 + pads silence = messy AR tail). The scheduler's buffer-ratchet builds 300-600 char mega-chunks that land in that zone; streaming concatenates each bad tail (worst case). ratatoskr's anti-"German" knobs (top_k=80/temp=0.5) made it WORSE — tight sampling pulls the degradation onset SHORTER (~200 chars vs ~300 at default knobs). Diagnosis method (deterministic, no ears-only): single-shot length sweep + amplitude-gated voiced-ZCR (garble spikes ZCR; must gate on |x|>500 else trailing silence confounds it) — degraded voiced-tail = 1.58× mid, clean = ~0.64-1.1×. FIX: server-side max_chunk_chars=250 cap on the scheduler (:v2 image, CBF_MAX_CHUNK_CHARS=250 env) — bounds each generation to just under the ~300-char onset → clean 3-4 sentence chunks (max prosodic arc while clean). Operator ear-confirmed clean audio + clean joins; chatterbox's low emotiveness keeps chunk joins smooth (the harsh joins that got Zonos rejected are absent — operator's key call). ratatoskr TODO (relayed msg 01KZER9X7S): revert knobs to default (top_k→1000, temp→0.8), send full text (server chunks internally), keep the 503-on-empty guard. Cap value tunable per-request (max_chunk_chars) + env. Deeper prosody (if ever wanted) = scheduler Phase-2 context-priming at joins (feed prior sentence as discarded-audio context; +latency). ⚠ FOOT-GUNS: (1) acoustic tail-trim is UNRELIABLE — sibilants ('s'/'sh'/'f') spike ZCR like garble, can't cleanly detect the speech→garble boundary. (2) build-context vs image drift — the :v2 image was built from cap source, but after a :v1 rollback the build context held :v1 source → a docker compose build would've silently produced a cap-less :v2; re-synced the flat cap source to /opt/docker/compose/chatterbox-fast/ (rebuild-verified). ⚠ DIVERGENCE (follow-up): deployed build context is FLAT (app.py/scheduler.py, from scheduler import, thin-overlay FROM local/chatterbox:v1, cap-only) vs the vh/chatterbox-fast REPO which is PACKAGE-layout (chatterbox_fast/, from chatterbox_fast.scheduler, self-contained Dockerfile) + has norm_loudness (repo commit 6bc7bf0 = cap; deployed omits norm_loudness deliberately to keep the ear-test unconfounded). Reconcile the two layouts so a repo-based rebuild matches deploy. Rollback: .bak-cap-20260807-104850 backups on irv-ml1 + :v1 image both retained. reference_chatterbox_fast_repo reference_zonos_tts_stack
[2026-08-07] Zonos2 TAKEN DOWN on the 3090 (irv-ml1) — operator-directed "for memory", TEMPORARY. Freed ~17.4 GB (3090: 728 MiB → 18.2 GB free) so chatterbox-fast (co-resident, was OOMing on long generations) has headroom. ⚠ Restore is manual — Zonos2 :1920 was a DETACHED native process (NOT systemd/docker), reparented to init. GPU memory was held by the --multiprocessing-fork CHILDREN (1966165=16.4G, 1966166=1G), which ORPHAN to init when you kill the parent — had to SIGTERM the children explicitly (killing the parent 1965942 + uv-run 1965935 alone left the 16.4G held). RESTORE CMD (from irv-ml1, user lkraven): cd /home/lkraven/tts-audition/models/zonos2 && nohup uv run python -m zonos2 --model-path Zyphra/ZONOS2 --host 0.0.0.0 --port 1920 --tts-default-voices-dir ./default_voices/ --cuda-graph-max-bs 1 --num-pages 16384 --max-running-requests 2 --memory-ratio 0.3 > /tmp/zonos2.log 2>&1 & then docker start zonos-gateway. Consumers that lost Zonos: asset-engine + gateway-chat (via LiteLLM ext-tts alias → zonos-gateway :8890, now stopped); ratatoskr already migrated OFF to chatterbox-fast (unaffected). Also unblocks proper drift/cap testing (OOM was blocking it). reference_zonos_tts_stack
[2026-08-07] chatterbox-fast: donut voice added + full contract delivered to ratatoskr-dev (their TTS migration off Zonos). Operator-directed. Copied zonos-gateway/voices/Donut.wav → chatterbox /refs (/worktank/chatterbox/reference_audio/donut.wav — the reference_audio SUBDIR is lkraven-owned so no sudo despite /worktank root; container globs /refs live → NO restart), exposed as voice:"donut" (lowercase); verified clean 7.5s synth (24kHz, RTF ~0.31). A/B booth (chatterbox vs zonos donut, same line) at http://10.100.10.50:8090/b/donut-chatterbox/. Answered ratatoskr's 8-question contract ask from the live gateway (local/chatterbox-fast:v1) + source: NOT OpenAI-shaped (POST /tts; body text/voice/format/stream, not input/model/response_format); NO affect dials (Turbo ignores cfg_weight/min_p/exaggeration — the architecture-changing answer they flagged; Zonos stays the only fleet TTS with real emotion steering); streaming WAV placeholder-header shape IDENTICAL to Zonos (their per-chunk Web Audio path survives); SR 24000 (Zonos 44100); server chunks arbitrary-length text internally (no client-side chunking, unlike Zonos's 71.2s cap); English-only, no language pin. FYI-worthy (operator): ratatoskr is moving its RP-surface TTS OFF Zonos back to chatterbox-fast → loses the live-PAD affect coupling (heavy Zonos emotion investment) — their call, trade-off flagged to them. auto-memory reference_chatterbox_fast_repo enriched w/ the live contract. reference_zonos_tts_stack
[2026-08-07] Fleet reranker cut over: Qwen3-Reranker-0.6B → BAAI/bge-reranker-v2-m3 (Brokkr R43). The incumbent was measured HARMING 80/90 fleet queries (no-reranker beat it 89/90 vs 56/90). R43 bake-off: the A2 control (same Qwen weights, seq-cls head) scored identical to the incumbent → proved the fault is a training-prior not the serving head → cancelled the expensive Qwen3-4B arm; A3 (bge-v2-m3) won on multilingual safety + bare-name recovery. LiteLLM reranker repointed incumbent→A3 :8013 (boundary 2026-08-06T17:37:48Z, config-edit + ~52s gateway restart); R42 v13 gate PASSED first-ever (56/90→90/90). Incumbent kept warm :8002 (rollback via qwen3-reranker alias), A4 fallback :8014. Full arc + rollback runbook docs/pfi/reranker-selection-ledger.md; commits ad2df89/2c11748/377f8a4 (unpushed). auto-memories: the earlier reranker-serving notes.
[2026-08-07] Personal-Worldtree kb-contamination incident (WT #394) diagnosed; attribution CLOSED UNRESOLVED. A reconcile WingStore._embed full-tree walk (kb fs_root=KB_PATH root, sibling wings nested) swept 5,354 fiction+main rows into personal's knowledge_base (2 superseded generations served as current). Fixed by WT #394 (aca39a1, kb walks exclude sibling wings; ships b182). Trigger un-attributable — peer reconcile via the SHARED infra-ops identity + 0 dockerd exec-logging = fingerprint-less. Durable finding → auto-memory infra_ops_shared_identity_attribution_gap, PARKED (operator ruled A) into project_migrate_infra_access_to_claude_credentials. Evidence hold on the 5,354 rows until operator sequences cleanup (w/ Brokkr, on #394's agenda).
[2026-08-05] Fleet CI resilience flip (DEFAULT_ACTIONS_URL=self) — attempted end-to-end, PARKED on a runner action-fetch auth blocker; infra-ops to research it (operator-directed, deferred, NOT now). 7 gitea action mirrors staged public+populated (orgs actions+astral-sh); the flip resolves uses: correctly but act_runner v0.6.0 can't authenticate its fetch to gitea 1.26 ("Invalid username or token. Password authentication is not supported"). Reverted (CI back on github default); REQUIRE_SIGNIN_VIEW=false KEPT as a standing change (operator, internal WG net). Full endeavor, the reliable nh3-dev-egress + git-SSH mirror method, exact config state, smoke method, and next step → persistent-memory.d/2026-08-05-ci-flip-parked.md
[2026-08-05] Booth — 3 features shipped, live on :8090 + tagged. (1) verbatim-index.html booths get a floating top-right "‹ all booths" chip + inherited favicon, doctype/charset-safe byte-injection (booth-v0.1.5, 8577e7e); (2) .md renders + .txt/.log view in-booth without downloading via the /b/<n>/view route + a markdown dep + doc.html (booth-v0.1.6, 315faac); (3) prev/next arrows in the image zoom viewer — wrap-around + keyboard ←/→, hidden for single-image booths (booth-v0.1.7, c37a425). Canonical services/booth/; deploy = systemctl --user restart booth.service on nh3-dev (runs from the checkout's .venv; uv pip install new deps into it first); 47 tests. uv.lock gitignored (348c5c1).
[2026-08-05] worldtree herald re-nudge bug root-caused → forseti shipped althing-core v2.1.2 (d5d33df, deployed on nh3-dev). herald.py:363 rendered the wake command from the empty fresh mail set on the re-nudge path (should be deliver_msgs) → messages[0] IndexError → un-suppressed outer catch-all → 7s crash-loop for 9 days on worldtree-codex's pane route (mimir-dev surfaced it; I traced it from the editable source). Fix + render_command empty-guard + outer log-suppress + 3 tests + contract amendment, all forseti's. nh3-extdev herald 2.1.2 upgrade DEFERRED (operator, not-now): extdev is a WHEEL install (not editable), unexposed (no pane routes); the verified 2.1.2 wheel is staged on nh3-dev /tmp (sha256 003508…cef27) — uv tool install --force + restart both heralds when un-parked. extdev herald-unit provenance resolved (operator-authorized 2026-07-25 via forseti relay; recorded in this file's 07-25 herald-install entry). auto-memory reference_nh3_dev_althing_herald.
[2026-07-31] muninn-gate (#377 ingestion front door) BUILT + DEPLOYED + healthy on corviduo-dev:8090. First-boot acceptance passed (watcher:running:true proves ingestion_root byte-identity); submit path deferred to the mimir-inbox era. Full wiring (uid-1000, state-volume mount, staging path-agreement, BuildKit-secret build, deferred repoint + operational guards) → persistent-memory.d/2026-07-31-muninn-gate-deploy.md
[2026-07-31] worldtree-sdk 1.1.0 (Python) published to vh Gitea PyPI + a durable infra-ops publish cred. memory_context pass-through; unblocked wyrd-dev. claude-bot now a write-collaborator on vh/worldtree-sdk (source pulled via the Gitea API archive — git-HTTP 403s on that repo); publishing to the vh USER namespace can't be delegated (401 reqPackageAccess even with write:package) so it needs an owner token — operator saved a FULL vh site-admin token at ~/.config/gitea/vh-token (0600) for it (⚠️ high blast radius, kept over a scoped one; org-namespace migration is the only real de-personalization, parked by wtsdk-dev). auto-memory reference_infra_ops_vh_gitea_token_and_sdk_publish.

206 older entries archived to archival-memory.md.

Tried and abandoned

[2026-08-15] Grafted bf16 MTP loads UNINITIALIZED (0% accept) unless re:^mtp.* is in the quant-config ignore; and W4A16=Marlin (not native FP4) costs ~20% even on decode. Cost a premature 79 GB delete of a good model (declared desync-dead off the 0%). Lessons: test MTP on bf16 FIRST, isolate before deleting; modelopt 0.43 is dependency-hell for qwen3_5 (list-vs-dict quant_cfg + transformers conflict) — use llm-compressor. Full → persistent-memory.d/2026-08-15-uncensored-gen-seat.md
[2026-08-03] ComfyUI --enable-triton-backend on the irv-ml1 A6000 crashes EVERY render — Ampere has no hardware e4m3. adhoc-agent's operator-approved probe: comfy_kitchen's triton backend has a FUSED int8 matmul that would beat the eager backend's ~1.9x-slower unfused int8 path (21.3s vs 11.2s fp8 on the Moody Krea2 int8 checkpoints). Flipped it (added to COMFY_CMDLINE_EXTRA, recreated) → triton.compiler.errors.CompilationError: ValueError("type fp8e4nv not supported in this architecture. supported: fp8e4b15, fp8e5") in comfy_kitchen/backends/triton/quantization.py:145 dequantize_per_tensor_fp8, failing at node 5 CLIPTextEncode. Triton's fp8 dequant kernel targets fp8e4nv (Hopper/Ada e4m3); sm_86 Ampere (A6000) lacks hardware e4m3 → the JIT compile dies. With triton on it grabs the global --fp8_e4m3fn-text-enc dequant, so every render (fp8 AND int8) dies upstream at the text-encode step — the int8 UNet path never ran, so the convrot-coverage caveat wasn't even the limiter. Reverted cleanly (~15s to healthy, image unchanged sha256:94afb8ca, sage intact, prod restored). The parked cu130 rebuild won't fix it (e4m3 = hardware format, not CUDA version). DEFERRED to the Ada refresh (operator: "ada is coming, we'll optimize then" — Ada sm_89 has native e4m3, so triton's fp8 path should compile there). Mechanics: --enable-triton-backend is a compose environment: var, so toggling it needs docker compose up -d (recreate), NOT docker restart (reuses the baked env, no-ops silently). Full: auto-memory parked_triton_backend_ampere_fp8.
[2026-08-03] corviduo-dev shared containerd: a concurrent-pull race fails ONE instance's deploy; DON'T "prune to fix" — the image is in-use by the instance that won the race. b169 personal deploy failed at docker compose pull (Lchown … no such file or directory on the big torch layer → looked like disk pressure / corrupt snapshot). ACTUAL: NOT disk (56G free, inodes 7%). demo + personal + pinned share ONE /var/lib/containerd on corviduo-dev; demo (from main) and personal (from staging tag) extracted b169's shared torch layer simultaneously → personal's hit a partial snapshot mid-race and aborted while demo's completed. The image 6e34a87 was FULLY VALID — demo was RUNNING it healthy. Fix = just re-run the failed deploy (image already materialized; compose pull finds it present). NEAR-MISS: worldtree-dev's suggested "prune unused images/snapshots" would have rmi'd 6e34a87 = the image the running demo depends on → demo outage. Lesson: before any prune/rmi "cleanup," docker ps the running images — an "unused" image may be a co-tenant's live one; and verify the failure's REAL cause (disk? inode? in-use? race?) before applying the suggested remedy. (Pipeline fix, deferred: serialize demo-from-main + personal-from-staging, or a per-image pull lock, to avoid the shared-layer extraction race.)
[2026-08-02] mv <job> complete/ → failed/ RENAMED the job to failed because failed/ didn't exist. worldtree-dev's round-2 unblock command (mv /data/state/ingestion/complete/<job> /data/state/ingestion/failed/) assumed failed/ existed; on PERSONAL muninn it did NOT (fresh instance — root was active/ complete/ pending/ sources/, no failed/). mv src nonexistent/ renames src→nonexistent, so job1 became the failed dir and job2 nested inside it. Caught on post-move ls (failed/ held job contents, not two subdirs), reconstructed via complete/ as watcher-safe scratch + rebuilt failed/ (worldtree:worldtree 755) — NO data loss. Lessons: (1) before mv X into-dir/, verify the dir EXISTS ([ -d dir ]) — an empty ls dir/ 2>/dev/null is AMBIGUOUS (missing vs empty), which was the preflight miss that let it through; (2) the correct guard is mv -t <targetdir> <src> (--target-directory): it refuses a MISSING target loudly (rc=1, "No such file or directory", nothing moved) — this is the house convention for queue/state moves now. TESTED by muninn-dev on coreutils 9.1: a trailing slash does NOT protect — mv src failed/ with failed/ missing STILL silently renames to failed (rc=0); "just add the slash" is a false guard. (mkdir -p failed/ first also works, but mv -t inverts the failure from silent-wrong to loud-safe in one flag.) Container sh is dash — no ( in echo strings. SILENT failure mode (muninn-dev carry-forward): a misplaced ingestion-state move doesn't crash anything — list_jobs() stays OK, loose files are inert; the ONLY symptom is the job quietly absent from the board (job_row→None, requeue→not_found/404, looks IDENTICAL to the original block). So after ANY state move, verify the job is actually ON THE BOARD (job_row found + guards pass), don't trust mv exit codes — and confirm job.dispatch.json survived (requeue refuses a dispatch-less job with the same not_requeueable symptom). Cross-checked + all-clear'd by muninn-dev, who correctly refused to mutate ingestion_root (INV-MG-1) and flagged instead. DON'T TIDY (round-2 pending): both DCC + P&P jobs currently REST in personal failed/ with manifests reading state: complete until round-2 requeue runs — deliberate + load-bearing (requeue keys on DIRECTORY PLACEMENT, not manifest state); looks wrong to anyone cold, leave it exactly as-is. Round-2 sequencing: the requeue is mimir-dev's browser flow (pending their operator's board-vs-API ruling); muninn-dev is the gate confirmer (runs the post-move board-check inside its custody — the right split, don't reach across INV-MG-1); infra-ops = the #381 restart after both jobs go terminal, then later the supervised main-collection sweep. Guard-verified HOLD LIFTED by muninn-dev 02:36Z. ARC COMPLETE (2026-08-03 ~05:49): both books terminal — DCC mimir-6351554e8e8f 705 concepts + P&P mimir-f3887c9b97b7 667, extracted AND indexed, 5/5 phases, 0 failures/truncations (validates the #385 budget fix vs April's 785 control); #381 restart-after-ingest FIRED (personal api, healthz/readyz 200 ~25s), retrieval-visibility confirmed (search_library returns DCC+P&P from fiction post-restart); handed ratatoskr-verify go to worldtree-dev. Delete-sweep precondition NOW MET — the stale DCC rows in main are genuine duplicates of live fiction rows, so worldtree-dev's supervised sweep of the ~785 April orphans is unblocked (still comes to me supervised: snapshot + operator-in-loop).

141 older entries archived to archival-memory.md.

103 KiB Raw Blame History Unescape Escape