vh
d4aa59a199
Operator-approved after last night's stall. Proxmox's own man vzdump names both the mechanism and the remedy: without fleecing, guest writes block on the backup target's speed; with it, pre-write copies land in a local image instead. Verified by behaviour rather than by config read-back, on a live backup that was still crawling at the same 1.4 MiB/s that caused the outage: io pressure full avg10 96.5% -> 0.01% in-flight writes 45, completing none -> 0 0 D-state processes 11 -> 0 32 MiB dsync write never completed -> 0.24s (133 MB/s) The backup is exactly as slow as before and the guest no longer cares. That is the point, and also the limit: this contains the blast radius and does not explain why a cross-site backup that once ran at 941 MiB/s now runs at 1.4, with the link up and pbs-ana answering in 11 ms. Job config backed up to /root/jobs.cfg.bak-* on nh3-pve before the change.
120 lines
5.3 KiB
Markdown
120 lines
5.3 KiB
Markdown
# A VM wedges on IO and the disk is innocent
|
|
|
|
First seen 2026-09-03, nh3-dev (VM 102 on nh3-pve). Reported by hamr-dev as
|
|
"/dev/sda stalled — writes hang". It was the backup.
|
|
|
|
## The signature
|
|
|
|
/proc/pressure/io full avg10=96 (everything is IO-blocked)
|
|
/sys/block/sda/inflight 0 45 (45 writes in flight, zero completing)
|
|
D state jbd2/sda1-8, flush-8:0 kworkers, systemd-journald, any writer
|
|
dmesg "blocked for more than 120 seconds", virtio_ring in the trace
|
|
|
|
Looks exactly like a failing disk or controller. It is not.
|
|
|
|
## What distinguishes it — and the checks are cheap
|
|
|
|
**1. There are no device errors.** No SCSI/ATA errors, no `I/O error`, no bus
|
|
resets, pool `ONLINE`, SMART quiet. **A failing disk says so.** Absence of
|
|
errors is evidence, not a gap in it.
|
|
|
|
**2. `Dirty` in /proc/meminfo is ~zero** while requests pile up in flight. If
|
|
writeback were merely slow, dirty pages would be accumulating. Near-zero Dirty
|
|
plus stuck in-flight requests means the writes were already submitted and are
|
|
not being *completed* — the stall is below the block layer, not above it.
|
|
|
|
**3. THE HYPERVISOR IS IDLE.** Check it before touching the guest:
|
|
|
|
ssh root@nh3-pve 'cat /proc/loadavg; cat /proc/pressure/io; zpool iostat rpool 1 2'
|
|
|
|
Host load 0.63 and io pressure 0.00 while a guest is 96% IO-blocked means the
|
|
guest's writes are not reaching the host at all. Something between them is
|
|
holding the queue.
|
|
|
|
**4. Then look at the block layer, which names the culprit outright:**
|
|
|
|
ssh root@nh3-pve 'qm monitor <VMID> <<< "info block"'
|
|
drive-scsi0: /dev/zvol/rpool/data/vm-102-disk-0 (copy-before-write) ← HERE
|
|
|
|
ssh root@nh3-pve 'qm monitor <VMID> <<< "info block-jobs"'
|
|
Type backup, device drive-scsi0: Completed 545 MB of 19.3 GB
|
|
|
|
`copy-before-write` is the filter Proxmox interposes during a backup: **every
|
|
guest write must first copy the original block out to the backup target before
|
|
it may proceed.** A slow or stalled target therefore throttles the guest to the
|
|
target's speed. On 2026-09-03 that was **1.4 MiB/s, cross-site to pbs-ana**, for
|
|
35 minutes.
|
|
|
|
## Fix
|
|
|
|
ssh root@nh3-pve 'pvesh delete "/nodes/localhost/tasks/<UPID>"'
|
|
|
|
Cancels the backup and detaches the filter. Verify: `info block` shows the disk
|
|
as `(raw)` again, `info block-jobs` says `No active jobs`, guest `inflight`
|
|
returns to `0 0`, D-states clear.
|
|
|
|
⚠ **No reboot is needed and none helps.** Rebooting the guest leaves the filter
|
|
attached and the next writes stall the same way.
|
|
|
|
⚠ **Cancelling is the standing call**, per the fleet rule *prefer no backup over
|
|
one that can crash the service*. An incremental backup with a dirty bitmap
|
|
re-runs cheaply; an unusable dev box does not.
|
|
|
|
## Why it can recur, and the real fix
|
|
|
|
The job is `backup-5d8f1221-8f71`: **daily at 21:00, `all 1`, storage
|
|
`pbs-ana`** — every VM on the host, cross-site to Anaheim.
|
|
|
|
⚠ **The job has `fleecing 0`.** With fleecing enabled, copy-before-write writes
|
|
land in a fast *local* fleecing image instead of going straight to the backup
|
|
target, which decouples guest IO from target throughput. That is the difference
|
|
between "the backup is slow tonight" and "the box is unusable tonight."
|
|
|
|
A previous run of this VM managed **941 MiB/s read**, so 1.4 MiB/s is a
|
|
degradation with its own cause — the cross-site link and pbs-ana were both
|
|
responsive (11 ms) throughout, so the throughput cause is **not yet known** and
|
|
enabling fleecing treats the blast radius, not the disease.
|
|
|
|
## Fleecing was enabled 2026-09-03, and verified by behaviour
|
|
|
|
pvesh set /cluster/backup/backup-5d8f1221-8f71 \
|
|
--fleecing 'enabled=1,storage=local-zfs'
|
|
|
|
Proxmox's own `man vzdump` states the mechanism and the fix outright:
|
|
|
|
> The guest write blocks until this operation is finished so guest IO to
|
|
> not-yet-backed-up sectors will be limited by the speed of the backup target.
|
|
> [...] With backup fleecing, such old data is cached in a fleecing image rather
|
|
> than sent directly to the backup target. This can help guest IO performance and
|
|
> even prevent hangs in certain scenarios, at the cost of requiring more storage
|
|
> space.
|
|
|
|
⚠ **Verify by behaviour, not by config.** The config read-back only proves the
|
|
key is set. Run a backup and measure the GUEST while it runs:
|
|
|
|
# host: topology should show BOTH filters
|
|
qm monitor <VMID> <<< "info block"
|
|
drive-scsi0 ... (copy-before-write)
|
|
drive-scsi0-fleecing ... /dev/zvol/rpool/data/vm-<VMID>-fleece-0 (raw) ← this
|
|
|
|
# guest: must stay responsive
|
|
cat /proc/pressure/io ; cat /sys/block/sda/inflight
|
|
dd if=/dev/zero of=$(mktemp) bs=1M count=32 oflag=dsync
|
|
|
|
Measured during a live backup that was **still crawling at the same 1.4 MiB/s**:
|
|
|
|
| | during the stall | with fleecing |
|
|
|---|---|---|
|
|
| io pressure full avg10 | 96.5% | **0.01%** |
|
|
| in-flight writes | 45, completing none | **0 0** |
|
|
| D-state processes | 11 | **0** |
|
|
| 32 MiB dsync write | never completed | **0.24 s (133 MB/s)** |
|
|
|
|
The backup is exactly as slow as before. The guest no longer cares — which is
|
|
the whole point, and the reason this is a blast-radius fix rather than a cure.
|
|
|
|
⚠ **Fleecing costs disk.** The image holds pre-write copies of changed blocks
|
|
for the life of the job, on `local-zfs` (717 G free at the time). A long, slow
|
|
backup of a busy guest is the case that grows it — watch
|
|
`zfs list rpool/data/vm-<VMID>-fleece-0` if the target stays degraded.
|