vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(backup): enable fleecing so a slow target cannot stall a guest

Browse Source 
Operator-approved after last night's stall. Proxmox's own man vzdump names
both the mechanism and the remedy: without fleecing, guest writes block on
the backup target's speed; with it, pre-write copies land in a local image
instead.

Verified by behaviour rather than by config read-back, on a live backup that
was still crawling at the same 1.4 MiB/s that caused the outage:

  io pressure full avg10   96.5%  ->  0.01%
  in-flight writes         45, completing none  ->  0 0
  D-state processes        11  ->  0
  32 MiB dsync write       never completed  ->  0.24s (133 MB/s)

The backup is exactly as slow as before and the guest no longer cares. That
is the point, and also the limit: this contains the blast radius and does
not explain why a cross-site backup that once ran at 941 MiB/s now runs at
1.4, with the link up and pbs-ana answering in 11 ms.

Job config backed up to /root/jobs.cfg.bak-* on nh3-pve before the change.
This commit is contained in:
vh
2026-09-02 22:33:12 -07:00
parent bd9692c433
commit d4aa59a199
1 changed files with 43 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/runbooks/nh3-dev-io-stall.md
+43
View File
@@ -74,3 +74,46 @@ A previous run of this VM managed **941 MiB/s read**, so 1.4 MiB/s is a
degradation with its own cause — the cross-site link and pbs-ana were both
responsive (11 ms) throughout, so the throughput cause is **not yet known** and
enabling fleecing treats the blast radius, not the disease.
## Fleecing was enabled 2026-09-03, and verified by behaviour
pvesh set /cluster/backup/backup-5d8f1221-8f71 \
--fleecing 'enabled=1,storage=local-zfs'
Proxmox's own `man vzdump` states the mechanism and the fix outright:
> The guest write blocks until this operation is finished so guest IO to
> not-yet-backed-up sectors will be limited by the speed of the backup target.
> [...] With backup fleecing, such old data is cached in a fleecing image rather
> than sent directly to the backup target. This can help guest IO performance and
> even prevent hangs in certain scenarios, at the cost of requiring more storage
> space.
⚠ **Verify by behaviour, not by config.** The config read-back only proves the
key is set. Run a backup and measure the GUEST while it runs:
# host: topology should show BOTH filters
qm monitor <VMID> <<< "info block"
drive-scsi0 ... (copy-before-write)
drive-scsi0-fleecing ... /dev/zvol/rpool/data/vm-<VMID>-fleece-0 (raw) ← this
# guest: must stay responsive
cat /proc/pressure/io ; cat /sys/block/sda/inflight
dd if=/dev/zero of=$(mktemp) bs=1M count=32 oflag=dsync
Measured during a live backup that was **still crawling at the same 1.4 MiB/s**:
| | during the stall | with fleecing |
|---|---|---|
| io pressure full avg10 | 96.5% | **0.01%** |
| in-flight writes | 45, completing none | **0 0** |
| D-state processes | 11 | **0** |
| 32 MiB dsync write | never completed | **0.24 s (133 MB/s)** |
The backup is exactly as slow as before. The guest no longer cares — which is
the whole point, and the reason this is a blast-radius fix rather than a cure.
⚠ **Fleecing costs disk.** The image holds pre-write copies of changed blocks
for the life of the job, on `local-zfs` (717 G free at the time). A long, slow
backup of a busy guest is the case that grows it — watch
`zfs list rpool/data/vm-<VMID>-fleece-0` if the target stays degraded.