vh
55705ba650
Root cause: all four NFS mounts were `hard`, so a NAS stall at 10.0.50.50
blocks I/O in uninterruptible sleep forever. The existing
x-systemd.before=docker.service fstab fix addressed the BOOT RACE -- a
different bug -- and never touched the runtime stall that keeps wedging
the box (2026-07-15, 2026-08-16).
Investigation narrowed the exposure well below what the parked item
assumed: only 2 of 12 containers touched NFS at all, and container state
was already on local disk (/var/lib/docker, 143G free).
Removed, no data risk:
/mnt/compose (2.1G) fully vestigial -- zero containers running or
stopped referenced it, dockge reads local
/opt/docker, and its one surviving mention was a
comment in beszel-agent-esh/.env describing a
DIFFERENT host.
/mnt/documents (2.0K) paperless's consume/export spool dirs, verified
empty, moved to /opt/docker/data/paperless at the
same 0777 the container already saw. Recreated,
healthy.
Both commented out in fstab (backup /etc/fstab.bak-nfs-harden-20260816)
and unmounted. Wedge surface halved: 4 mounts -> 2, 2 wedge-capable
containers -> 1.
traefik needed no change -- already restart: unless-stopped, which is why
it self-recovered after the reset.
Watchdog on esh-pve (NOT in the guest -- a watchdog inside the thing it
watches is no watchdog). It probes traefik over HTTP rather than ping or
SSH because the wedge signature is 'guest OS alive, services dead': / is
local disk, so sshd answers and ICMP replies straight through a total
outage, and a TCP check would report HEALTHY. The guest-agent ping is
recorded only to classify the failure, never to veto a reset. 5
consecutive failures at 2-min interval (~10 min) then qm reset 100;
30-min cooldown against reset loops, acts only when qm status is running,
and honours /etc/esh-vm-docker-watchdog.disabled.
All four paths validated on install without power-cycling anything:
healthy -> silent no-op, disable flag -> SKIP, simulated outage -> counts
and classifies as the D-state signature, recovery -> counter cleared.
DEFERRED by operator ruling: /mnt/books stays `hard`. It holds calibre's
SQLite metadata.db and soft/softerr risks corrupting it mid-write. That
is the one remaining wedge vector; revisit alongside moving the library
off NFS.
Park item 28 promoted with full provenance.
95 lines
4.0 KiB
Markdown
95 lines
4.0 KiB
Markdown
# esh-vm-docker-watchdog — bound the NFS D-state wedge
|
|
|
|
Runs on **esh-pve (10.0.250.35)**, watching **esh-vm-docker (VMID 100 / 10.0.50.45)**.
|
|
A watchdog living inside the thing it watches is no watchdog — this one sits on
|
|
the hypervisor, which is also the only place that can actually recover the VM.
|
|
|
|
## The failure it exists for
|
|
|
|
esh-vm-docker mounts NFS from the NAS at 10.0.50.50 with **`hard`** semantics. When
|
|
the NAS stalls, I/O blocks in uninterruptible sleep — the D-state — and nothing
|
|
inside the guest clears it. Reconfirmed 2026-07-15: `docker stop/rm -f`,
|
|
`ctr -n moby task delete`, `systemctl restart docker`, and even `docker exec`
|
|
all fail. Only a host reset recovers. Most recent occurrence: 2026-08-16.
|
|
|
|
The pre-existing `x-systemd.before=docker.service` fstab fix addresses the **boot
|
|
race** — a different bug. It does nothing for a **runtime** stall, which is the
|
|
one that keeps biting.
|
|
|
|
This watchdog does **not prevent** the wedge. It bounds the outage from "until
|
|
someone notices" to roughly ten minutes.
|
|
|
|
## Why the probe is an HTTP check and not ping/SSH
|
|
|
|
The wedge signature is specifically **"guest OS alive, services dead."** The root
|
|
filesystem is local disk, so sshd keeps accepting connections and ICMP keeps
|
|
answering straight through a total service outage. **A ping or TCP check reports
|
|
HEALTHY while the box is unusable** — which is exactly how the outage goes
|
|
unnoticed in the first place.
|
|
|
|
So the trigger is an HTTP probe of traefik (the ingress every service sits
|
|
behind). The qemu-guest-agent ping is recorded only to *classify* the failure in
|
|
the log — agent alive + HTTP dead is the textbook wedge — and never vetoes a
|
|
reset.
|
|
|
|
## Behaviour
|
|
|
|
| knob | default | why |
|
|
|---|---|---|
|
|
| probe | `http://10.0.50.45:80/` | any HTTP response counts as alive; a 404 from traefik still proves it is serving |
|
|
| `PROBE_TIMEOUT` | 10s | bounded so the probe itself can never hang the watchdog |
|
|
| interval | 2 min | systemd timer |
|
|
| `FAIL_THRESHOLD` | 5 | ~10 min sustained outage before acting — a traefik redeploy cannot trigger a power cycle |
|
|
| `COOLDOWN_SEC` | 1800 | a genuinely broken VM cannot be reset-looped |
|
|
|
|
Safety rails: resets only when `qm status` reports `running` (a deliberately
|
|
stopped VM is left alone), and honours a disable flag.
|
|
|
|
```bash
|
|
# suspend during planned maintenance
|
|
touch /etc/esh-vm-docker-watchdog.disabled
|
|
rm /etc/esh-vm-docker-watchdog.disabled # re-arm
|
|
```
|
|
|
|
Log: `/var/log/esh-vm-docker-watchdog.log`. State:
|
|
`/var/lib/esh-vm-docker-watchdog/`.
|
|
|
|
## Install
|
|
|
|
```bash
|
|
scp esh-vm-docker-watchdog.sh root@10.0.250.35:/usr/local/sbin/
|
|
scp esh-vm-docker-watchdog.{service,timer} root@10.0.250.35:/etc/systemd/system/
|
|
ssh root@10.0.250.35 'chmod 0755 /usr/local/sbin/esh-vm-docker-watchdog.sh &&
|
|
systemctl daemon-reload && systemctl enable --now esh-vm-docker-watchdog.timer'
|
|
```
|
|
|
|
## Verify without power-cycling anything
|
|
|
|
The failure path can be exercised safely by pointing the probe at a closed port
|
|
and raising the threshold out of reach:
|
|
|
|
```bash
|
|
PROBE_URL=http://10.0.50.45:9/ FAIL_THRESHOLD=99 PROBE_TIMEOUT=3 \
|
|
/usr/local/sbin/esh-vm-docker-watchdog.sh
|
|
tail -3 /var/log/esh-vm-docker-watchdog.log
|
|
```
|
|
|
|
Validated this way on install (2026-08-16): healthy → silent no-op; disable flag
|
|
→ `SKIP`; simulated outage → counts and classifies as the D-state signature;
|
|
recovery → counter cleared. VM never reset during testing.
|
|
|
|
## What it does NOT cover
|
|
|
|
`/mnt/books` is still a **`hard`** mount and is still able to wedge the box —
|
|
deliberately. Calibre's library holds a SQLite `metadata.db`, and `soft` /
|
|
`softerr` semantics risk corrupting it on a stall mid-write. Operator ruling
|
|
2026-08-16: keep `hard`, accept the wedge, bound it with this watchdog. Revisit
|
|
only alongside moving the calibre library off NFS.
|
|
|
|
`/mnt/backup` (restic, daily 01:00) is also still `hard`, but is touched only by
|
|
a scheduled job rather than a long-running container, so a stall there wedges
|
|
restic and not dockerd.
|
|
|
|
Related: `playbooks/fix-esh-nfs-boot-ordering.yaml` (the boot-race fix),
|
|
park item `harden-esh-docker-vm-against-the-recurring-nfs` (id 28).
|