vh
d4aa59a199
Operator-approved after last night's stall. Proxmox's own man vzdump names both the mechanism and the remedy: without fleecing, guest writes block on the backup target's speed; with it, pre-write copies land in a local image instead. Verified by behaviour rather than by config read-back, on a live backup that was still crawling at the same 1.4 MiB/s that caused the outage: io pressure full avg10 96.5% -> 0.01% in-flight writes 45, completing none -> 0 0 D-state processes 11 -> 0 32 MiB dsync write never completed -> 0.24s (133 MB/s) The backup is exactly as slow as before and the guest no longer cares. That is the point, and also the limit: this contains the blast radius and does not explain why a cross-site backup that once ran at 941 MiB/s now runs at 1.4, with the link up and pbs-ana answering in 11 ms. Job config backed up to /root/jobs.cfg.bak-* on nh3-pve before the change.
5.3 KiB
A VM wedges on IO and the disk is innocent
First seen 2026-09-03, nh3-dev (VM 102 on nh3-pve). Reported by hamr-dev as "/dev/sda stalled — writes hang". It was the backup.
The signature
/proc/pressure/io full avg10=96 (everything is IO-blocked)
/sys/block/sda/inflight 0 45 (45 writes in flight, zero completing)
D state jbd2/sda1-8, flush-8:0 kworkers, systemd-journald, any writer
dmesg "blocked for more than 120 seconds", virtio_ring in the trace
Looks exactly like a failing disk or controller. It is not.
What distinguishes it — and the checks are cheap
1. There are no device errors. No SCSI/ATA errors, no I/O error, no bus
resets, pool ONLINE, SMART quiet. A failing disk says so. Absence of
errors is evidence, not a gap in it.
2. Dirty in /proc/meminfo is ~zero while requests pile up in flight. If
writeback were merely slow, dirty pages would be accumulating. Near-zero Dirty
plus stuck in-flight requests means the writes were already submitted and are
not being completed — the stall is below the block layer, not above it.
3. THE HYPERVISOR IS IDLE. Check it before touching the guest:
ssh root@nh3-pve 'cat /proc/loadavg; cat /proc/pressure/io; zpool iostat rpool 1 2'
Host load 0.63 and io pressure 0.00 while a guest is 96% IO-blocked means the guest's writes are not reaching the host at all. Something between them is holding the queue.
4. Then look at the block layer, which names the culprit outright:
ssh root@nh3-pve 'qm monitor <VMID> <<< "info block"'
drive-scsi0: /dev/zvol/rpool/data/vm-102-disk-0 (copy-before-write) ← HERE
ssh root@nh3-pve 'qm monitor <VMID> <<< "info block-jobs"'
Type backup, device drive-scsi0: Completed 545 MB of 19.3 GB
copy-before-write is the filter Proxmox interposes during a backup: every
guest write must first copy the original block out to the backup target before
it may proceed. A slow or stalled target therefore throttles the guest to the
target's speed. On 2026-09-03 that was 1.4 MiB/s, cross-site to pbs-ana, for
35 minutes.
Fix
ssh root@nh3-pve 'pvesh delete "/nodes/localhost/tasks/<UPID>"'
Cancels the backup and detaches the filter. Verify: info block shows the disk
as (raw) again, info block-jobs says No active jobs, guest inflight
returns to 0 0, D-states clear.
⚠ No reboot is needed and none helps. Rebooting the guest leaves the filter attached and the next writes stall the same way.
⚠ Cancelling is the standing call, per the fleet rule prefer no backup over one that can crash the service. An incremental backup with a dirty bitmap re-runs cheaply; an unusable dev box does not.
Why it can recur, and the real fix
The job is backup-5d8f1221-8f71: daily at 21:00, all 1, storage
pbs-ana — every VM on the host, cross-site to Anaheim.
⚠ The job has fleecing 0. With fleecing enabled, copy-before-write writes
land in a fast local fleecing image instead of going straight to the backup
target, which decouples guest IO from target throughput. That is the difference
between "the backup is slow tonight" and "the box is unusable tonight."
A previous run of this VM managed 941 MiB/s read, so 1.4 MiB/s is a degradation with its own cause — the cross-site link and pbs-ana were both responsive (11 ms) throughout, so the throughput cause is not yet known and enabling fleecing treats the blast radius, not the disease.
Fleecing was enabled 2026-09-03, and verified by behaviour
pvesh set /cluster/backup/backup-5d8f1221-8f71 \
--fleecing 'enabled=1,storage=local-zfs'
Proxmox's own man vzdump states the mechanism and the fix outright:
The guest write blocks until this operation is finished so guest IO to not-yet-backed-up sectors will be limited by the speed of the backup target. [...] With backup fleecing, such old data is cached in a fleecing image rather than sent directly to the backup target. This can help guest IO performance and even prevent hangs in certain scenarios, at the cost of requiring more storage space.
⚠ Verify by behaviour, not by config. The config read-back only proves the key is set. Run a backup and measure the GUEST while it runs:
# host: topology should show BOTH filters
qm monitor <VMID> <<< "info block"
drive-scsi0 ... (copy-before-write)
drive-scsi0-fleecing ... /dev/zvol/rpool/data/vm-<VMID>-fleece-0 (raw) ← this
# guest: must stay responsive
cat /proc/pressure/io ; cat /sys/block/sda/inflight
dd if=/dev/zero of=$(mktemp) bs=1M count=32 oflag=dsync
Measured during a live backup that was still crawling at the same 1.4 MiB/s:
| during the stall | with fleecing | |
|---|---|---|
| io pressure full avg10 | 96.5% | 0.01% |
| in-flight writes | 45, completing none | 0 0 |
| D-state processes | 11 | 0 |
| 32 MiB dsync write | never completed | 0.24 s (133 MB/s) |
The backup is exactly as slow as before. The guest no longer cares — which is the whole point, and the reason this is a blast-radius fix rather than a cure.
⚠ Fleecing costs disk. The image holds pre-write copies of changed blocks
for the life of the job, on local-zfs (717 G free at the time). A long, slow
backup of a busy guest is the case that grows it — watch
zfs list rpool/data/vm-<VMID>-fleece-0 if the target stays degraded.