# esh-vm-docker-watchdog — bound the NFS D-state wedge

Runs on **esh-pve (10.0.250.35)**, watching **esh-vm-docker (VMID 100 / 10.0.50.45)**.
A watchdog living inside the thing it watches is no watchdog — this one sits on
the hypervisor, which is also the only place that can actually recover the VM.

## The failure it exists for

esh-vm-docker mounts NFS from the NAS at 10.0.50.50 with **`hard`** semantics. When
the NAS stalls, I/O blocks in uninterruptible sleep — the D-state — and nothing
inside the guest clears it. Reconfirmed 2026-07-15: `docker stop/rm -f`,
`ctr -n moby task delete`, `systemctl restart docker`, and even `docker exec`
all fail. Only a host reset recovers. Most recent occurrence: 2026-08-16.

The pre-existing `x-systemd.before=docker.service` fstab fix addresses the **boot
race** — a different bug. It does nothing for a **runtime** stall, which is the
one that keeps biting.

This watchdog does **not prevent** the wedge. It bounds the outage from "until
someone notices" to roughly ten minutes.

## Why the probe is an HTTP check and not ping/SSH

The wedge signature is specifically **"guest OS alive, services dead."** The root
filesystem is local disk, so sshd keeps accepting connections and ICMP keeps
answering straight through a total service outage. **A ping or TCP check reports
HEALTHY while the box is unusable** — which is exactly how the outage goes
unnoticed in the first place.

So the trigger is an HTTP probe of traefik (the ingress every service sits
behind). The qemu-guest-agent ping is recorded only to *classify* the failure in
the log — agent alive + HTTP dead is the textbook wedge — and never vetoes a
reset.

## Behaviour

| knob | default | why |
|---|---|---|
| probe | `http://10.0.50.45:80/` | any HTTP response counts as alive; a 404 from traefik still proves it is serving |
| `PROBE_TIMEOUT` | 10s | bounded so the probe itself can never hang the watchdog |
| interval | 2 min | systemd timer |
| `FAIL_THRESHOLD` | 5 | ~10 min sustained outage before acting — a traefik redeploy cannot trigger a power cycle |
| `COOLDOWN_SEC` | 1800 | a genuinely broken VM cannot be reset-looped |

Safety rails: resets only when `qm status` reports `running` (a deliberately
stopped VM is left alone), and honours a disable flag.

```bash
# suspend during planned maintenance
touch /etc/esh-vm-docker-watchdog.disabled
rm    /etc/esh-vm-docker-watchdog.disabled   # re-arm
```

Log: `/var/log/esh-vm-docker-watchdog.log`. State:
`/var/lib/esh-vm-docker-watchdog/`.

## Install

```bash
scp esh-vm-docker-watchdog.sh root@10.0.250.35:/usr/local/sbin/
scp esh-vm-docker-watchdog.{service,timer} root@10.0.250.35:/etc/systemd/system/
ssh root@10.0.250.35 'chmod 0755 /usr/local/sbin/esh-vm-docker-watchdog.sh &&
  systemctl daemon-reload && systemctl enable --now esh-vm-docker-watchdog.timer'
```

## Verify without power-cycling anything

The failure path can be exercised safely by pointing the probe at a closed port
and raising the threshold out of reach:

```bash
PROBE_URL=http://10.0.50.45:9/ FAIL_THRESHOLD=99 PROBE_TIMEOUT=3 \
  /usr/local/sbin/esh-vm-docker-watchdog.sh
tail -3 /var/log/esh-vm-docker-watchdog.log
```

Validated this way on install (2026-08-16): healthy → silent no-op; disable flag
→ `SKIP`; simulated outage → counts and classifies as the D-state signature;
recovery → counter cleared. VM never reset during testing.

## What it does NOT cover

`/mnt/books` is still a **`hard`** mount and is still able to wedge the box —
deliberately. Calibre's library holds a SQLite `metadata.db`, and `soft` /
`softerr` semantics risk corrupting it on a stall mid-write. Operator ruling
2026-08-16: keep `hard`, accept the wedge, bound it with this watchdog. Revisit
only alongside moving the calibre library off NFS.

`/mnt/backup` (restic, daily 01:00) is also still `hard`, but is touched only by
a scheduled job rather than a long-running container, so a stall there wedges
restic and not dockerd.

Related: `playbooks/fix-esh-nfs-boot-ordering.yaml` (the boot-race fix),
park item `harden-esh-docker-vm-against-the-recurring-nfs` (id 28).