docs(incident): nh3-dev IO stall was a stalled backup, not the disk

hamr-dev reported /dev/sda stalled on nh3-dev: 45 writes in flight
completing zero, jbd2 and flush kworkers in D state for 33 minutes, io
pressure full at 96%, load 26. Every symptom pointed at a dying disk.

It was a vzdump to pbs-ana that collapsed from 64 MiB/s to 1.4 MiB/s and
sat there. Proxmox interposes a copy-before-write filter during a backup,
so every guest write must copy the original block to the target first — the
guest was throttled to the speed of a stalled cross-site backup.

Three things distinguished it, all cheap: there were no device errors
anywhere (a failing disk says so); Dirty was 3.8 MB while 45 requests were
in flight, so the writes were submitted and not completing rather than
backed up in page cache; and the hypervisor was idle at 0.00 io pressure,
which means the writes were not reaching it at all.

Cancelling the task detached the filter: inflight 45 -> 0, D-states gone,
191 MB/s dsync restored. Tonight's incremental is lost and re-runs cheaply,
which is the standing trade in "prefer no backup over one that can crash
the service".

Recurs nightly at 21:00 until changed, and the job has fleecing disabled —
which is why a slow TARGET can stall a GUEST at all.

docs/runbooks/nh3-dev-io-stall.md

@@ -0,0 +1,76 @@
+# A VM wedges on IO and the disk is innocent
+
+First seen 2026-09-03, nh3-dev (VM 102 on nh3-pve). Reported by hamr-dev as
+"/dev/sda stalled — writes hang". It was the backup.
+
+## The signature
+
+    /proc/pressure/io     full avg10=96  (everything is IO-blocked)
+    /sys/block/sda/inflight        0 45  (45 writes in flight, zero completing)
+    D state       jbd2/sda1-8, flush-8:0 kworkers, systemd-journald, any writer
+    dmesg         "blocked for more than 120 seconds", virtio_ring in the trace
+
+Looks exactly like a failing disk or controller. It is not.
+
+## What distinguishes it — and the checks are cheap
+
+**1. There are no device errors.** No SCSI/ATA errors, no `I/O error`, no bus
+resets, pool `ONLINE`, SMART quiet. **A failing disk says so.** Absence of
+errors is evidence, not a gap in it.
+
+**2. `Dirty` in /proc/meminfo is ~zero** while requests pile up in flight. If
+writeback were merely slow, dirty pages would be accumulating. Near-zero Dirty
+plus stuck in-flight requests means the writes were already submitted and are
+not being *completed* — the stall is below the block layer, not above it.
+
+**3. THE HYPERVISOR IS IDLE.** Check it before touching the guest:
+
+    ssh root@nh3-pve 'cat /proc/loadavg; cat /proc/pressure/io; zpool iostat rpool 1 2'
+
+Host load 0.63 and io pressure 0.00 while a guest is 96% IO-blocked means the
+guest's writes are not reaching the host at all. Something between them is
+holding the queue.
+
+**4. Then look at the block layer, which names the culprit outright:**
+
+    ssh root@nh3-pve 'qm monitor <VMID> <<< "info block"'
+    drive-scsi0: /dev/zvol/rpool/data/vm-102-disk-0 (copy-before-write)   ← HERE
+
+    ssh root@nh3-pve 'qm monitor <VMID> <<< "info block-jobs"'
+    Type backup, device drive-scsi0: Completed 545 MB of 19.3 GB
+
+`copy-before-write` is the filter Proxmox interposes during a backup: **every
+guest write must first copy the original block out to the backup target before
+it may proceed.** A slow or stalled target therefore throttles the guest to the
+target's speed. On 2026-09-03 that was **1.4 MiB/s, cross-site to pbs-ana**, for
+35 minutes.
+
+## Fix
+
+    ssh root@nh3-pve 'pvesh delete "/nodes/localhost/tasks/<UPID>"'
+
+Cancels the backup and detaches the filter. Verify: `info block` shows the disk
+as `(raw)` again, `info block-jobs` says `No active jobs`, guest `inflight`
+returns to `0 0`, D-states clear.
+
+⚠ **No reboot is needed and none helps.** Rebooting the guest leaves the filter
+attached and the next writes stall the same way.
+
+⚠ **Cancelling is the standing call**, per the fleet rule *prefer no backup over
+one that can crash the service*. An incremental backup with a dirty bitmap
+re-runs cheaply; an unusable dev box does not.
+
+## Why it can recur, and the real fix
+
+The job is `backup-5d8f1221-8f71`: **daily at 21:00, `all 1`, storage
+`pbs-ana`** — every VM on the host, cross-site to Anaheim.
+
+⚠ **The job has `fleecing 0`.** With fleecing enabled, copy-before-write writes
+land in a fast *local* fleecing image instead of going straight to the backup
+target, which decouples guest IO from target throughput. That is the difference
+between "the backup is slow tonight" and "the box is unusable tonight."
+
+A previous run of this VM managed **941 MiB/s read**, so 1.4 MiB/s is a
+degradation with its own cause — the cross-site link and pbs-ana were both
+responsive (11 ms) throughout, so the throughput cause is **not yet known** and
+enabling fleecing treats the blast radius, not the disease.