vh
55705ba650
Root cause: all four NFS mounts were `hard`, so a NAS stall at 10.0.50.50
blocks I/O in uninterruptible sleep forever. The existing
x-systemd.before=docker.service fstab fix addressed the BOOT RACE -- a
different bug -- and never touched the runtime stall that keeps wedging
the box (2026-07-15, 2026-08-16).
Investigation narrowed the exposure well below what the parked item
assumed: only 2 of 12 containers touched NFS at all, and container state
was already on local disk (/var/lib/docker, 143G free).
Removed, no data risk:
/mnt/compose (2.1G) fully vestigial -- zero containers running or
stopped referenced it, dockge reads local
/opt/docker, and its one surviving mention was a
comment in beszel-agent-esh/.env describing a
DIFFERENT host.
/mnt/documents (2.0K) paperless's consume/export spool dirs, verified
empty, moved to /opt/docker/data/paperless at the
same 0777 the container already saw. Recreated,
healthy.
Both commented out in fstab (backup /etc/fstab.bak-nfs-harden-20260816)
and unmounted. Wedge surface halved: 4 mounts -> 2, 2 wedge-capable
containers -> 1.
traefik needed no change -- already restart: unless-stopped, which is why
it self-recovered after the reset.
Watchdog on esh-pve (NOT in the guest -- a watchdog inside the thing it
watches is no watchdog). It probes traefik over HTTP rather than ping or
SSH because the wedge signature is 'guest OS alive, services dead': / is
local disk, so sshd answers and ICMP replies straight through a total
outage, and a TCP check would report HEALTHY. The guest-agent ping is
recorded only to classify the failure, never to veto a reset. 5
consecutive failures at 2-min interval (~10 min) then qm reset 100;
30-min cooldown against reset loops, acts only when qm status is running,
and honours /etc/esh-vm-docker-watchdog.disabled.
All four paths validated on install without power-cycling anything:
healthy -> silent no-op, disable flag -> SKIP, simulated outage -> counts
and classifies as the D-state signature, recovery -> counter cleared.
DEFERRED by operator ruling: /mnt/books stays `hard`. It holds calibre's
SQLite metadata.db and soft/softerr risks corrupting it mid-write. That
is the one remaining wedge vector; revisit alongside moving the library
off NFS.
Park item 28 promoted with full provenance.
4.0 KiB
esh-vm-docker-watchdog — bound the NFS D-state wedge
Runs on esh-pve (10.0.250.35), watching esh-vm-docker (VMID 100 / 10.0.50.45). A watchdog living inside the thing it watches is no watchdog — this one sits on the hypervisor, which is also the only place that can actually recover the VM.
The failure it exists for
esh-vm-docker mounts NFS from the NAS at 10.0.50.50 with hard semantics. When
the NAS stalls, I/O blocks in uninterruptible sleep — the D-state — and nothing
inside the guest clears it. Reconfirmed 2026-07-15: docker stop/rm -f,
ctr -n moby task delete, systemctl restart docker, and even docker exec
all fail. Only a host reset recovers. Most recent occurrence: 2026-08-16.
The pre-existing x-systemd.before=docker.service fstab fix addresses the boot
race — a different bug. It does nothing for a runtime stall, which is the
one that keeps biting.
This watchdog does not prevent the wedge. It bounds the outage from "until someone notices" to roughly ten minutes.
Why the probe is an HTTP check and not ping/SSH
The wedge signature is specifically "guest OS alive, services dead." The root filesystem is local disk, so sshd keeps accepting connections and ICMP keeps answering straight through a total service outage. A ping or TCP check reports HEALTHY while the box is unusable — which is exactly how the outage goes unnoticed in the first place.
So the trigger is an HTTP probe of traefik (the ingress every service sits behind). The qemu-guest-agent ping is recorded only to classify the failure in the log — agent alive + HTTP dead is the textbook wedge — and never vetoes a reset.
Behaviour
| knob | default | why |
|---|---|---|
| probe | http://10.0.50.45:80/ |
any HTTP response counts as alive; a 404 from traefik still proves it is serving |
PROBE_TIMEOUT |
10s | bounded so the probe itself can never hang the watchdog |
| interval | 2 min | systemd timer |
FAIL_THRESHOLD |
5 | ~10 min sustained outage before acting — a traefik redeploy cannot trigger a power cycle |
COOLDOWN_SEC |
1800 | a genuinely broken VM cannot be reset-looped |
Safety rails: resets only when qm status reports running (a deliberately
stopped VM is left alone), and honours a disable flag.
# suspend during planned maintenance
touch /etc/esh-vm-docker-watchdog.disabled
rm /etc/esh-vm-docker-watchdog.disabled # re-arm
Log: /var/log/esh-vm-docker-watchdog.log. State:
/var/lib/esh-vm-docker-watchdog/.
Install
scp esh-vm-docker-watchdog.sh root@10.0.250.35:/usr/local/sbin/
scp esh-vm-docker-watchdog.{service,timer} root@10.0.250.35:/etc/systemd/system/
ssh root@10.0.250.35 'chmod 0755 /usr/local/sbin/esh-vm-docker-watchdog.sh &&
systemctl daemon-reload && systemctl enable --now esh-vm-docker-watchdog.timer'
Verify without power-cycling anything
The failure path can be exercised safely by pointing the probe at a closed port and raising the threshold out of reach:
PROBE_URL=http://10.0.50.45:9/ FAIL_THRESHOLD=99 PROBE_TIMEOUT=3 \
/usr/local/sbin/esh-vm-docker-watchdog.sh
tail -3 /var/log/esh-vm-docker-watchdog.log
Validated this way on install (2026-08-16): healthy → silent no-op; disable flag
→ SKIP; simulated outage → counts and classifies as the D-state signature;
recovery → counter cleared. VM never reset during testing.
What it does NOT cover
/mnt/books is still a hard mount and is still able to wedge the box —
deliberately. Calibre's library holds a SQLite metadata.db, and soft /
softerr semantics risk corrupting it on a stall mid-write. Operator ruling
2026-08-16: keep hard, accept the wedge, bound it with this watchdog. Revisit
only alongside moving the calibre library off NFS.
/mnt/backup (restic, daily 01:00) is also still hard, but is touched only by
a scheduled job rather than a long-running container, so a stall there wedges
restic and not dockerd.
Related: playbooks/fix-esh-nfs-boot-ordering.yaml (the boot-race fix),
park item harden-esh-docker-vm-against-the-recurring-nfs (id 28).