diff --git a/persistent-memory.md b/persistent-memory.md index a347599..691a28e 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -133,7 +133,7 @@ _As of 2026-08-18 — **gen seat SWAPPED to `absolute-heresy` and operator-confi - **⏳ WATCHING: DavidAU's HERETIC build of Qwen3.8-27B — the one worth waiting for.** `DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1` examined 2026-08-17 and **NOT adopted**: it is a capability/efficiency finetune of **stock** Qwen3.8 and every bench row on its card is labelled **`[non heretic]`** — adopting it would reintroduce base refusals the current seat does not have. ⚠ **Easy to misread as uncensored** (operator did, and it is a fair mistake): DavidAU's back catalog is almost all `Uncensored-Heretic` builds — Fable-Fusion 711, Qwen3.5-9B Cold-Fusion — so the naming pattern implies it. This one simply has not had that stage run yet; the card's roadmap says the HERETIC version is **IN PROGRESS from base**. **That** is the release to watch, not this one. What makes it worth watching: third-party benches (Nightmedia, mxfp8) beat stock Qwen3.8 by **+0.064 arc/c, +0.056 arc/e, +0.056 obkqa**; claimed **MTP acceptance 55.7% (record 59.9%)** vs our measured 47.2%; **thinking tokens cut to 1/10–1/2**; PPL *dropped* vs base. Same GAIN/Cold-Fusion pipeline that produced Fable-Fusion 711, which we already serve on `char-rp-reasoning` — proven in-fleet, not just claimed. Structurally clean (1199 tensors, 15 mtp in shard 18, 333 visual). ⚠ MTP/speed figures are **GGUF/llama.cpp on a 5090, not vLLM** — may not transfer; and because its MTP head was likely *trained*, the free CPU-hash shortcut would NOT apply (it won't match base) so a real acceptance gate would be needed. -- **🟢 esh-pve-nas — MIGRATION DONE 2026-08-18. Root is `nvme/ROOT/pve-1` on mirrored NVMe; the USB DOM is out of the runtime I/O path.** All five guests healthy, three pools ONLINE, system `running`, ext4 `pve-root` intact+unmounted as rollback with its own kernel. Boot config: `saved_entry=pve-zfs-root`, no `next_entry`; if grubenv were unreadable GRUB falls to entry 0 which the `/etc/default/grub.d/zfs-root.cfg` drop-in also points at ZFS. **`zfs-import-cache.service` is now the active import path** (the all-three-pools cachefile fix working as intended); vestigial `zfs-import@nvme.service` disabled — it failed every boot as redundant. ⚠ **Device letters shift across reboots** (DOM was `sdq`, now `sdl`) — never key anything to a bare `sdX` here. ⚠ **NO auto-fallback on a failed boot, and no IPMI/BMC/serial** — grubenv on LVM is readable but not writable by GRUB, so `grub-reboot`'s one-shot degrades to a sticky default (verified: `next_entry` survived the boot that consumed it). Recovery = pick the ROLLBACK entry at the console. **⏳ REMAINING: second reboot to prove repeatability (needs another quiesce window), refresh the DOM image since `/boot` changed, then patch the 225 pending packages — the whole point of the migration.** +- **🟢 esh-pve-nas — MIGRATION DONE 2026-08-18. Root is `nvme/ROOT/pve-1` on mirrored NVMe; the USB DOM is out of the runtime I/O path.** All five guests healthy, three pools ONLINE, system `running`, ext4 `pve-root` intact+unmounted as rollback with its own kernel. Boot config: `saved_entry=pve-zfs-root`, no `next_entry`; if grubenv were unreadable GRUB falls to entry 0 which the `/etc/default/grub.d/zfs-root.cfg` drop-in also points at ZFS. **`zfs-import-cache.service` is now the active import path** (the all-three-pools cachefile fix working as intended); vestigial `zfs-import@nvme.service` disabled — it failed every boot as redundant. ⚠ **Device letters shift across reboots** (DOM was `sdq`, now `sdl`) — never key anything to a bare `sdX` here. ⚠ **NO auto-fallback on a failed boot, and no IPMI/BMC/serial** — grubenv on LVM is readable but not writable by GRUB, so `grub-reboot`'s one-shot degrades to a sticky default (verified: `next_entry` survived the boot that consumed it). Recovery = pick the ROLLBACK entry at the console. **PATCHED 2026-08-18: 225 packages installed, pve-manager 8.4.11 -> 8.4.20, corosync 3.1.9 -> 3.1.10-pve2, kernel 6.8.12-42 staged on the /boot LV. dpkg clean, no unapplied conffiles, cluster quorate, 6/6 verify. ⏳ REBOOT DEFERRED at operator request — host still runs 6.8.12-13 until a chosen window; `GRUB_DEFAULT=0` means entry 0 is already the -42 entry with the correct `root=ZFS=nvme/ROOT/pve-1`, so the reboot is the only remaining step. Rollback for the upgrade is the ZFS snapshot `nvme/ROOT/pve-1@pre-upgrade-20260818T141652Z` (409M) — `zfs rollback -r && reboot`. Second confirmation reboot ALREADY DONE (2026-08-18, booted ZFS from GRUB_DEFAULT=0 with no one-shot). ⏳ Still outstanding: refresh the off-box DOM image, since `/boot` changed.** esh-pve is FULLY done (8.4.20 + kernel 6.8.12-42 + corosync 3.1.10, rebooted, quorate). ⚠⚠ **THE WINDOW COST AN UNPLANNED OUTAGE, caused by our own tooling, not the migration.** The staging chroot did `mount --rbind /dev` + `/sys` with **no `--make-rslave`**; on systemd `/` is *shared*, so the cutover's `umount -R` **propagated back into the live host** and stripped the real `/sys/fs/cgroup`, `/dev/pts`, `/dev/shm`. logind could then create no sessions: ping fine, TCP fine, **SSH authenticates**, resident daemons keep serving (pveproxy returned clean 401s) — and **every new exec hangs, including `/sbin/reboot`**, so the reboot never ran. **It is a near-perfect impostor of failing root-disk I/O**, and I misdiagnosed it as the USB DOM dying and told the operator to walk to the machine. **Operator caught it** — the DOM had been fine for years and the wedge began right after a change. The settling evidence was in `dmesg` all along: `[16.00] [sdq] Attached SCSI removable disk` (clean, no errors) and a last-line timestamp of **`[12114881]` = 140 days = the ORIGINAL boot** — the machine had never rebooted. My down-detector never once reported the host down and I read that as a fast reboot rather than *no* reboot. Recovered with **no console access** by hammering an idempotent cgroup2/devpts/shm remount into the brief windows where exec succeeded. Zero data loss. **RULES: (1) always `--make-rslave` after `--rbind` (playbook now guards on `PROPAGATION != shared`); (2) a reboot is not confirmed until the host is observed DOWN — poll for disappearance, not reappearance; (3) before blaming hardware for a wedge that started right after a change, get `dmesg` and check the boot timestamp.** ⚠ **Blast radius was more than DOUBLE what the runbook said** — `ss -tn '( sport = :2049 )'` in CT 103 found **five** NFS clients, not two: +**esh-vm-db `10.0.50.60`** (`/mnt/backup`, **hard**, no ssh — reach via `qm guest exec 101` on esh-pve), +vm-esh-nas (is VM 104 on the host itself), +nh3-dev (soft,ro — safe). **Ask the server who its clients are; a documented dependent list rots.** esh-vm-db was deliberately left mounted (backup target, timers ~19h out) and **came through read-write** — a `hard` mount with no active user blocks and resumes, which is what `hard` is for. _(historical, pre-cutover context below)_ **migration STAGED 2026-08-18.** PVE root ran off a **NORELSYS USB Disk-on-Module** (`sdq`). Wear was NOT the driver (a DOM is SLC/pSLC — operator corrected my first read); the drivers are the **USB bus** (a reset drops root under a running hypervisor), no headroom, no mirror, and **blocked patching: 225 packages, 161 with `deb12uN`/security bumps**, stuck on PVE 8.4.11 vs esh-pve's 8.4.14. Mitigated 2026-08-17 (90% → 76%); **staged 2026-08-18 via two rerunnable elway playbooks** (`esh-pve-nas-stage-zfs-root.yaml`, `esh-pve-nas-stage-bootloader.yaml`), 0 failed, 17/17 verify green. diff --git a/playbooks/pve-node-upgrade.yaml b/playbooks/pve-node-upgrade.yaml index 5f6aa7e..70e900c 100644 --- a/playbooks/pve-node-upgrade.yaml +++ b/playbooks/pve-node-upgrade.yaml @@ -15,6 +15,12 @@ # FIRST. Its boot default used to pin a single kernel, so installing a new one # would either break the default entry or silently keep booting the old kernel. # +# ⚠ The corosync bump RESTARTS corosync mid-upgrade, which on a 2-node cluster is +# a brief quorum event — both nodes' /etc/pve go read-only for a few seconds and +# then recover. Guests are unaffected and PVE does this routinely, but do not run +# it concurrently with anything that writes cluster config, and check +# `pvecm status` afterwards rather than assuming. +# # Conffile policy: --force-confdef + --force-confold, i.e. keep the on-disk # version wherever a package ships a changed conffile. That is the right default # for these hosts (hand-tuned /etc/default/grub, grub.d drop-ins, storage.cfg), @@ -58,6 +64,20 @@ steps: ls -la {{ backup_dir }}/ creates: "{{ backup_dir }}/pre-upgrade-backup.done" + # On a ZFS-root node this is the cheapest insurance available: an instant, + # space-free snapshot of the entire userspace before 200+ packages land. If the + # upgrade goes wrong, the recovery is a rollback and a reboot rather than an + # archaeology session in dpkg. Skipped automatically on non-ZFS roots. + - name: Snapshot the root dataset (ZFS-root nodes only) + shell: | + ds=$(findmnt -no SOURCE /) + snap="${ds}@pre-upgrade-$(date -u +%Y%m%dT%H%M%SZ)" + zfs snapshot "$snap" + echo " created $snap" + echo " rollback if needed: zfs rollback -r $snap && reboot" + zfs list -t snapshot -o name,used,creation -s creation "$ds" 2>/dev/null | tail -4 + when: "test \"$(findmnt -no FSTYPE /)\" = zfs" + - name: Refresh package lists shell: apt-get update -qq changed_when: "true"