memory: snapshot — Heretic-300 built, quantized and verified; gen-seat cutover is the next step
8/100 refusals at KL 0.0136, hand-verified coherent, beating the absolute-heresy bar 3.6x. NVFP4 quant complete: 21 GB, 1968 tensors, MTP head grafted back after PR #317 dropped it, and re:^mtp.* re-injected into quantization_config.ignore after llm-compressor pruned it. Self-harm guardrail is gone on this build and is the operator's own next work item; the four-dwarf panel is stood down.
This commit is contained in:
@@ -114,7 +114,7 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
- **🔴→🟢 ESH OUTAGE 2026-08-19 — esh-pve hard-froze 03:34, ~4.5h, whole house lost DNS.** Presented as "wifi/routing issues"; internet was healthy throughout. Cause: `esh-userland` (VLAN 10, the `PVC` SSID) handed out **one** resolver, `10.0.50.45` (AdGuard on `esh-docker-vm`) — cross-VLAN, no secondary — and its hypervisor died. **Recovered by hand power-cycle; all VMs, cluster quorum and DNS restored.** Two fixes landed: gateway `10.0.10.1` added as secondary DNS on esh-userland (operator-approved, first confirmed WRITE on the ESH UDM key), and **`softdog` → `iTCO_wdt` hardware watchdog owned by systemd** (`playbooks/esh-pve-hardware-watchdog.yaml`, idempotent, verified armed) so a repeat self-recovers in 60s. **VM 102 pinned off** (`onboot: 0`) as the passthrough suspect. **⏳ OPEN:** (a) the watchdog is armed but **has not been proven to fire** — needs a deliberate wedge to confirm; (b) AMT/vPro still unusable until an onboard **RJ45** is cabled (the MS-01 is SFP+-only on the network and AMT cannot ride it); (c) kernel `6.8.12-16` rollback held in reserve if it freezes again. → `persistent-memory.d/2026-08-19-esh-pve-freeze-dns-spof.md`
|
||||
|
||||
_As of 2026-08-20 — continued from the 08-19 infra session (DNS/.internal, waterland, homepage-theme, unmanaged-stacks — all landed). This session's adds, all committed: **waterland upstream fixes deployed** (`b72425b`, all 3 findings closed); **esh-pve froze 4.5h → hardware watchdog + secondary DNS** (SPOF closed); **Homepage theme REBUILT** on canonical Australis tokens (the Skyfall/Arbo version was ugly — pills/cards realigned, wallpaper killed, live at 10.0.50.45:5100); **Booth gained kept-boards (`.forever`) + a standing agent link board + inline .md/.txt rendering**; **fleet IPv6 plan settled** (endpoints not internal numbering; ESH has a /56); and **the Cold-Fusion abliteration** (above) — capture done, calibration expansion next. ⚠ MANY commits unpushed (through `7abd301`)._
|
||||
_As of 2026-08-20 22:55 — **the Heretic-300 session** (see the 🔴 entry above; full epic + roadmap in `persistent-memory.d/2026-08-20-heretic-300-epic.md`). Landed tonight: 300-trial Heretic search → **8/100 refusals @ KL 0.0136**, hand-verified coherent, beating the `absolute-heresy` bar 3.6×; MTP head grafted back after PR #317 silently dropped it; **NVFP4 quant COMPLETE and verified** (`qwen38-27b-coldfusion-h300-nvfp4-mixed`, 21 GB, 1968 tensors, `re:^mtp.*` re-injected). ⏭ **NEXT: gen-seat cutover** (operator's call — gen, not probe: single-user internal WG, prior seat already fully abliterated), then MTP acceptance + PPL + surface 6/6, **especially vision**. ⚠ **The self-harm guardrail is GONE on this build** — operator is handling restoration directly and does not want dwarf analysis on it; four-dwarf panel formally STOOD DOWN. Earlier in the day: KL harness (28.4× selectivity on L35), esh-nas-pve corruption resolved (171 files recoverable, error scrub clean, pool ONLINE), `/tank` on ana-ml2 still DEGRADED 69 days (parked, operator going to colo). ⚠ MANY commits unpushed._
|
||||
|
||||
- **🟢 FLEET `.internal` DNS — LIVE 2026-08-19.** `<host>.<site>.internal`, sites `ana`/`esh`/`nh3`. `dns/internal.yaml` is the source of truth; `scripts/dns-sync.py` reconciles the three AdGuard resolvers (diff → prompt → apply, idempotent). 42 names resolving from all three sites. **Colo got its first resolver ever** (`stacks/adguard-ana/`, API on **8053** not 8080, no blocklists by design) — before this, ana-docker resolved straight against `1.1.1.1`. Auth = a dedicated `infra-ops` AdGuard user, password vaulted `nh3-dev/adguard-infra-ops-password`. **⏳ TWO OPEN, both operator's to schedule:** (a) colo hosts still point at `1.1.1.1` so they do not yet *use* the new resolver — repointing a site's DNS is a separate change; (b) the static-v6 convention (each server at its site's `/64` with low bits echoing the v4 octet, `esh-docker-vm` → `…::45`) is **proposed, not ruled on**. The `v6:` column is empty and correct — no fleet host has a global v6 address yet. → `persistent-memory.d/2026-08-19-fleet-internal-dns.md`
|
||||
|
||||
|
||||
Reference in New Issue
Block a user