vh
838132cd6b
Session captured: the FV outbound-NAT root cause and its diagnostic signature, the fv-ml1 dead man's switch, fleet identity/group/path conventions and the root:docker normalization, nh3-dev's ts-input reachability fix, ESPHome modernisation and the kb KB-search tool, and the Hermes bearer rotation release. Six new detail files. Tried-and-abandoned gains three: probing OPNsense endpoints by POSTing at them (which rebooted the FV firewall), advertising a /32 from nh3-dev, and the nh3-scale remote-site masquerade rules that fired but were not the fix. Housekeeping: 8 Recent-decisions entries archived to archival-memory.md, and 21 oversized inline entries split into detail files per the two-tier rule -- they had been sitting fully inline in the index, which is what the split exists to prevent. Two pointers to a detail file archived this run were repointed at archival-memory.md. The index is 389 lines, still over the ~300 soft cap. The archival guards stop it there: only 4 further entries are old enough to move and every one carries an open deferred-work pointer. An over-cap file that keeps live decisions beats a scannable one that lost a deferred call.
1.5 KiB
[2026-09-03] nh3-dev wedged for ~40 min and it was the BACKUP, not the disk — a stalled cross-site vzdump holding every gue
nh3-dev wedged for ~40 min and it was the BACKUP, not the disk — a stalled cross-site vzdump holding every guest write via copy-before-write. Symptoms screamed dying disk: 45 writes in flight completing zero, jbd2 + flush kworkers in D state 33 min, io pressure full 96%, load 26, virtio_ring in the stack. ⚠ The discriminator was the ABSENCE of errors — no SCSI/ATA/IO errors, rpool ONLINE 21%, guest fs 79%, memory fine, and Dirty only 3.8 MB (so nothing backed up in page cache; it was stuck BELOW the block layer). ⚠ The hypervisor was IDLE — load 0.63, io pressure 0.00, zpool ~0 writes: nothing was reaching the disk because the filter held it. Cause: vzdump of VM 102 → pbs-ana did 1% at 64 MiB/s then collapsed to 1.4 MiB/s for 35 min; Proxmox backups interpose a copy-before-write filter, so every guest write queues behind the backup's copy-out. FIX = cancel the task (pvesh delete /nodes/localhost/tasks/<UPID>); filter detached, inflight 45→0, D-states gone, 191 MB/s dsync restored. ⚠ fleecing 0 on the job is why a slow TARGET can stall a GUEST — fleecing routes copy-before-write to a fast local image instead. Job = backup-5d8f1221-8f71, daily 21:00, all 1, storage pbs-ana → recurs nightly until changed. A prior run of this VM managed 941 MiB/s read, so 1.4 MiB/s is degradation, not normal. → docs/runbooks/nh3-dev-io-stall.md