# A VM wedges on IO and the disk is innocent

First seen 2026-09-03, nh3-dev (VM 102 on nh3-pve). Reported by hamr-dev as
"/dev/sda stalled — writes hang". It was the backup.

## The signature

    /proc/pressure/io     full avg10=96  (everything is IO-blocked)
    /sys/block/sda/inflight        0 45  (45 writes in flight, zero completing)
    D state       jbd2/sda1-8, flush-8:0 kworkers, systemd-journald, any writer
    dmesg         "blocked for more than 120 seconds", virtio_ring in the trace

Looks exactly like a failing disk or controller. It is not.

## What distinguishes it — and the checks are cheap

**1. There are no device errors.** No SCSI/ATA errors, no `I/O error`, no bus
resets, pool `ONLINE`, SMART quiet. **A failing disk says so.** Absence of
errors is evidence, not a gap in it.

**2. `Dirty` in /proc/meminfo is ~zero** while requests pile up in flight. If
writeback were merely slow, dirty pages would be accumulating. Near-zero Dirty
plus stuck in-flight requests means the writes were already submitted and are
not being *completed* — the stall is below the block layer, not above it.

**3. THE HYPERVISOR IS IDLE.** Check it before touching the guest:

    ssh root@nh3-pve 'cat /proc/loadavg; cat /proc/pressure/io; zpool iostat rpool 1 2'

Host load 0.63 and io pressure 0.00 while a guest is 96% IO-blocked means the
guest's writes are not reaching the host at all. Something between them is
holding the queue.

**4. Then look at the block layer, which names the culprit outright:**

    ssh root@nh3-pve 'qm monitor <VMID> <<< "info block"'
    drive-scsi0: /dev/zvol/rpool/data/vm-102-disk-0 (copy-before-write)   ← HERE

    ssh root@nh3-pve 'qm monitor <VMID> <<< "info block-jobs"'
    Type backup, device drive-scsi0: Completed 545 MB of 19.3 GB

`copy-before-write` is the filter Proxmox interposes during a backup: **every
guest write must first copy the original block out to the backup target before
it may proceed.** A slow or stalled target therefore throttles the guest to the
target's speed. On 2026-09-03 that was **1.4 MiB/s, cross-site to pbs-ana**, for
35 minutes.

## Fix

    ssh root@nh3-pve 'pvesh delete "/nodes/localhost/tasks/<UPID>"'

Cancels the backup and detaches the filter. Verify: `info block` shows the disk
as `(raw)` again, `info block-jobs` says `No active jobs`, guest `inflight`
returns to `0 0`, D-states clear.

⚠ **No reboot is needed and none helps.** Rebooting the guest leaves the filter
attached and the next writes stall the same way.

⚠ **Cancelling is the standing call**, per the fleet rule *prefer no backup over
one that can crash the service*. An incremental backup with a dirty bitmap
re-runs cheaply; an unusable dev box does not.

## Why it can recur, and the real fix

The job is `backup-5d8f1221-8f71`: **daily at 21:00, `all 1`, storage
`pbs-ana`** — every VM on the host, cross-site to Anaheim.

⚠ **The job has `fleecing 0`.** With fleecing enabled, copy-before-write writes
land in a fast *local* fleecing image instead of going straight to the backup
target, which decouples guest IO from target throughput. That is the difference
between "the backup is slow tonight" and "the box is unusable tonight."

A previous run of this VM managed **941 MiB/s read**, so 1.4 MiB/s is a
degradation with its own cause — the cross-site link and pbs-ana were both
responsive (11 ms) throughout, so the throughput cause is **not yet known** and
enabling fleecing treats the blast radius, not the disease.

## Fleecing was enabled 2026-09-03, and verified by behaviour

    pvesh set /cluster/backup/backup-5d8f1221-8f71 \
      --fleecing 'enabled=1,storage=local-zfs'

Proxmox's own `man vzdump` states the mechanism and the fix outright:

> The guest write blocks until this operation is finished so guest IO to
> not-yet-backed-up sectors will be limited by the speed of the backup target.
> [...] With backup fleecing, such old data is cached in a fleecing image rather
> than sent directly to the backup target. This can help guest IO performance and
> even prevent hangs in certain scenarios, at the cost of requiring more storage
> space.

⚠ **Verify by behaviour, not by config.** The config read-back only proves the
key is set. Run a backup and measure the GUEST while it runs:

    # host: topology should show BOTH filters
    qm monitor <VMID> <<< "info block"
    drive-scsi0           ... (copy-before-write)
    drive-scsi0-fleecing  ... /dev/zvol/rpool/data/vm-<VMID>-fleece-0 (raw)   ← this

    # guest: must stay responsive
    cat /proc/pressure/io ; cat /sys/block/sda/inflight
    dd if=/dev/zero of=$(mktemp) bs=1M count=32 oflag=dsync

Measured during a live backup that was **still crawling at the same 1.4 MiB/s**:

| | during the stall | with fleecing |
|---|---|---|
| io pressure full avg10 | 96.5% | **0.01%** |
| in-flight writes | 45, completing none | **0 0** |
| D-state processes | 11 | **0** |
| 32 MiB dsync write | never completed | **0.24 s (133 MB/s)** |

The backup is exactly as slow as before. The guest no longer cares — which is
the whole point, and the reason this is a blast-radius fix rather than a cure.

⚠ **Fleecing costs disk.** The image holds pre-write copies of changed blocks
for the life of the job, on `local-zfs` (717 G free at the time). A long, slow
backup of a busy guest is the case that grows it — watch
`zfs list rpool/data/vm-<VMID>-fleece-0` if the target stays degraded.