Commit Graph

1 Commits

Author SHA1 Message Date
vh 55705ba650 feat(esh): harden esh-vm-docker against the NFS D-state wedge (Tier 1 + watchdog)
Root cause: all four NFS mounts were `hard`, so a NAS stall at 10.0.50.50
blocks I/O in uninterruptible sleep forever. The existing
x-systemd.before=docker.service fstab fix addressed the BOOT RACE -- a
different bug -- and never touched the runtime stall that keeps wedging
the box (2026-07-15, 2026-08-16).

Investigation narrowed the exposure well below what the parked item
assumed: only 2 of 12 containers touched NFS at all, and container state
was already on local disk (/var/lib/docker, 143G free).

Removed, no data risk:
  /mnt/compose (2.1G)   fully vestigial -- zero containers running or
                        stopped referenced it, dockge reads local
                        /opt/docker, and its one surviving mention was a
                        comment in beszel-agent-esh/.env describing a
                        DIFFERENT host.
  /mnt/documents (2.0K) paperless's consume/export spool dirs, verified
                        empty, moved to /opt/docker/data/paperless at the
                        same 0777 the container already saw. Recreated,
                        healthy.

Both commented out in fstab (backup /etc/fstab.bak-nfs-harden-20260816)
and unmounted. Wedge surface halved: 4 mounts -> 2, 2 wedge-capable
containers -> 1.

traefik needed no change -- already restart: unless-stopped, which is why
it self-recovered after the reset.

Watchdog on esh-pve (NOT in the guest -- a watchdog inside the thing it
watches is no watchdog). It probes traefik over HTTP rather than ping or
SSH because the wedge signature is 'guest OS alive, services dead': / is
local disk, so sshd answers and ICMP replies straight through a total
outage, and a TCP check would report HEALTHY. The guest-agent ping is
recorded only to classify the failure, never to veto a reset. 5
consecutive failures at 2-min interval (~10 min) then qm reset 100;
30-min cooldown against reset loops, acts only when qm status is running,
and honours /etc/esh-vm-docker-watchdog.disabled.

All four paths validated on install without power-cycling anything:
healthy -> silent no-op, disable flag -> SKIP, simulated outage -> counts
and classifies as the D-state signature, recovery -> counter cleared.

DEFERRED by operator ruling: /mnt/books stays `hard`. It holds calibre's
SQLite metadata.db and soft/softerr risks corrupting it mid-write. That
is the one remaining wedge vector; revisit alongside moving the library
off NFS.

Park item 28 promoted with full provenance.
2026-08-16 10:23:24 -07:00