From b8e5022a1aa093e76e759af849623f25f3959ba7 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Fri, 21 Aug 2026 13:55:35 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20snapshot=20=E2=80=94=20three=20AI=20s?= =?UTF-8?q?eats=20settled=20(orcarouter=20gen=20/=20MeroMero=20dual-mode?= =?UTF-8?q?=20RP=20/=20sec=20pen-test),=20Open=20WebUI=20deployed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- persistent-memory.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/persistent-memory.md b/persistent-memory.md index 8512c6e..abec300 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -114,7 +114,14 @@ no longer deployed sidecars here. See Recent decisions.) - **πŸ”΄β†’πŸŸ’ ESH OUTAGE 2026-08-19 β€” esh-pve hard-froze 03:34, ~4.5h, whole house lost DNS.** Presented as "wifi/routing issues"; internet was healthy throughout. Cause: `esh-userland` (VLAN 10, the `PVC` SSID) handed out **one** resolver, `10.0.50.45` (AdGuard on `esh-docker-vm`) β€” cross-VLAN, no secondary β€” and its hypervisor died. **Recovered by hand power-cycle; all VMs, cluster quorum and DNS restored.** Two fixes landed: gateway `10.0.10.1` added as secondary DNS on esh-userland (operator-approved, first confirmed WRITE on the ESH UDM key), and **`softdog` β†’ `iTCO_wdt` hardware watchdog owned by systemd** (`playbooks/esh-pve-hardware-watchdog.yaml`, idempotent, verified armed) so a repeat self-recovers in 60s. **VM 102 pinned off** (`onboot: 0`) as the passthrough suspect. **⏳ OPEN:** (a) the watchdog is armed but **has not been proven to fire** β€” needs a deliberate wedge to confirm; (b) AMT/vPro still unusable until an onboard **RJ45** is cabled (the MS-01 is SFP+-only on the network and AMT cannot ride it); (c) kernel `6.8.12-16` rollback held in reserve if it freezes again. β†’ `persistent-memory.d/2026-08-19-esh-pve-freeze-dns-spof.md` -_As of 2026-08-21 00:35 β€” **the Heretic-300 session, and its reversal** (see the πŸ”΄ entry above; full epic + roadmap in `persistent-memory.d/2026-08-20-heretic-300-epic.md`). Landed tonight: 300-trial Heretic search β†’ **8/100 refusals @ KL 0.0136**, hand-verified coherent, beating the `absolute-heresy` bar 3.6Γ—; MTP head grafted back after PR #317 silently dropped it; **NVFP4 quant COMPLETE and verified** (`qwen38-27b-coldfusion-h300-nvfp4-mixed`, 21 GB, 1968 tensors, `re:^mtp.*` re-injected); gen-seat cutover done 23:05 and every gate passed (7/7 aliases, vision intact, MTP 59.7%). **β›” THEN ROLLED BACK 00:28 ON OPERATOR DIRECTIVE.** Lobe surfaced an unterminated-`` leak; measurement showed the **Cold-Fusion base** carries 18.5% first-token mass on it and our abliteration only added +3.7 β€” so the operator called it: **abandon h300 AND the Cold-Fusion base**, since no rollback inside that family escapes the leak. Gen seat is back on **`qwen38-27b-heresy-nvfp4-mixed`** (KV 403,065 / 1.54Γ— = its exact documented baseline; 7/7 aliases; vision intact; `` not even in the top-20 at <0.002 vs Cold-Fusion's 0.185). **The methodology is the deliverable and it stands** β€” see the Heretic-300 entry. ⚠ **The self-harm guardrail is GONE on this build** β€” operator is handling restoration directly and does not want dwarf analysis on it; four-dwarf panel formally STOOD DOWN. Earlier in the day: KL harness (28.4Γ— selectivity on L35), esh-nas-pve corruption resolved (171 files recoverable, error scrub clean, pool ONLINE), `/tank` on ana-ml2 still DEGRADED 69 days (parked, operator going to colo). ⚠ MANY commits unpushed._ +_As of 2026-08-21 (late) β€” **the big AI-seat overhaul session; three seats settled + Open WebUI + the MeroMero-CoT saga, all resolved.** Headline state, seat by seat:_ + +- **GEN SEAT = `orcarouter` (SETTLED).** `orcarouter/Qwen3.8-27B-Uncensored` NVFP4-mixed on ana-ml2 GPU0 :8015, 7 aliases (gen/gen-reasoning/summarizer/-large/classifier/image-judge/qwen-image-bench). Won the search: MTP **58.7%** (its head was already Robinson-abliterated **in-band** by the author β€” first third-party build to get that right), PPL **7.07** (β‰ˆheresy 6.91), vision **7/8**, no ``-leak. **Cold-Fusion/Heretic-300 line ABANDONED** (base carried 18.5% first-token `` mass β€” a format defect abliteration only amplified +3.7); its bf16 weights **deleted (~154 GB reclaimed)**. ⏳ **Two Cold-Fusion NVFP4 quants (44 GB) still on disk, delete/keep UNADJUDICATED.** Rollback target for gen = `qwen38-27b-heresy-nvfp4-mixed` (intact). +- **RP SEAT = MeroMero-v2 DUAL-MODE (βœ… SOLVED β€” the session's hardest win).** ONE seat / ONE weight set / TWO aliases: `char-rp` (Gemma-4 prose) + `char-rp-reasoning` (**streaming** CoT). On **`vllm/vllm-openai:v0.26.0`** (ships tf 5.14.1 β†’ Gemma-4 loads, no pin) + a **force-open chat template** (bare `<|channel>`) + two served-names. **Streaming-only** (non-streaming can't split β€” vLLM one-shot bug; fine, Lobe/OWUI stream). Four-arm brokkr/dwarf panel traced it from source. ⏳ Upstream vLLM issue **drafted, NOT filed** (operator's call β€” public, his GitHub identity). +- **PEN-TEST SEAT = `sec` / `sec-reasoning` (LIVE).** `M.O.G.-SEC-27B-1M-CTX` (Blackfrost) NVFP4-mixed on ana-ml2 GPU1 :8019, served at native 262K (NOT the card's 1M β€” needs YaRN+SGLang). MTP 55.3%, vision + capability verified. Renamed from mog-secβ†’sec per operator. +- **OPEN WEBUI = deployed** on esh-docker-vm:3211 (bake-off vs Lobe, all 6 gates pass). Admin=lkraven; ⚠ **the deploy's original temp password was BROKEN β€” reset via bcrypt + verified, working credential in vault `esh-docker-vm/open-webui-admin`.** ⏳ **Lobe retirement DEFERRED to operator**; G2 sync-job (hide non-chat seats) greenlit to tts-dev to write. `lobe-chat-esh` key = all-proxy-models (incl paid). +- **FLEET CLEANUP:** LFM2.5 retired; reranker consolidated (**nevermore had been silently failing 8 days** β€” pinned to the retired `granite-4.1-8b` alias; repointed to `summarizer`/`reranker`); `:8002`/`:8014`/granite service retired; A3 promoted to `stacks/vllm`. +- ⏳ **OPEN:** file the upstream vLLM issue (operator); operator's OWUI password change; Cold-Fusion NVFP4 quants delete/keep; Brokkr duplicate `reranker-a3-bge-v2-m3` alias; `/tank` DEGRADED **70+ days** (parked, operatorβ†’colo w/ cold spare); **MANY commits unpushed** (push is operator's call)._ - **🟒 FLEET `.internal` DNS β€” LIVE 2026-08-19.** `..internal`, sites `ana`/`esh`/`nh3`. `dns/internal.yaml` is the source of truth; `scripts/dns-sync.py` reconciles the three AdGuard resolvers (diff β†’ prompt β†’ apply, idempotent). 42 names resolving from all three sites. **Colo got its first resolver ever** (`stacks/adguard-ana/`, API on **8053** not 8080, no blocklists by design) β€” before this, ana-docker resolved straight against `1.1.1.1`. Auth = a dedicated `infra-ops` AdGuard user, password vaulted `nh3-dev/adguard-infra-ops-password`. **⏳ TWO OPEN, both operator's to schedule:** (a) colo hosts still point at `1.1.1.1` so they do not yet *use* the new resolver β€” repointing a site's DNS is a separate change; (b) the static-v6 convention (each server at its site's `/64` with low bits echoing the v4 octet, `esh-docker-vm` β†’ `…::45`) is **proposed, not ruled on**. The `v6:` column is empty and correct β€” no fleet host has a global v6 address yet. β†’ `persistent-memory.d/2026-08-19-fleet-internal-dns.md`