diff --git a/docs/runbooks/nh3-dev-io-stall.md b/docs/runbooks/nh3-dev-io-stall.md index d1dbc2f..192cf27 100644 --- a/docs/runbooks/nh3-dev-io-stall.md +++ b/docs/runbooks/nh3-dev-io-stall.md @@ -74,3 +74,46 @@ A previous run of this VM managed **941 MiB/s read**, so 1.4 MiB/s is a degradation with its own cause — the cross-site link and pbs-ana were both responsive (11 ms) throughout, so the throughput cause is **not yet known** and enabling fleecing treats the blast radius, not the disease. + +## Fleecing was enabled 2026-09-03, and verified by behaviour + + pvesh set /cluster/backup/backup-5d8f1221-8f71 \ + --fleecing 'enabled=1,storage=local-zfs' + +Proxmox's own `man vzdump` states the mechanism and the fix outright: + +> The guest write blocks until this operation is finished so guest IO to +> not-yet-backed-up sectors will be limited by the speed of the backup target. +> [...] With backup fleecing, such old data is cached in a fleecing image rather +> than sent directly to the backup target. This can help guest IO performance and +> even prevent hangs in certain scenarios, at the cost of requiring more storage +> space. + +⚠ **Verify by behaviour, not by config.** The config read-back only proves the +key is set. Run a backup and measure the GUEST while it runs: + + # host: topology should show BOTH filters + qm monitor <<< "info block" + drive-scsi0 ... (copy-before-write) + drive-scsi0-fleecing ... /dev/zvol/rpool/data/vm--fleece-0 (raw) ← this + + # guest: must stay responsive + cat /proc/pressure/io ; cat /sys/block/sda/inflight + dd if=/dev/zero of=$(mktemp) bs=1M count=32 oflag=dsync + +Measured during a live backup that was **still crawling at the same 1.4 MiB/s**: + +| | during the stall | with fleecing | +|---|---|---| +| io pressure full avg10 | 96.5% | **0.01%** | +| in-flight writes | 45, completing none | **0 0** | +| D-state processes | 11 | **0** | +| 32 MiB dsync write | never completed | **0.24 s (133 MB/s)** | + +The backup is exactly as slow as before. The guest no longer cares — which is +the whole point, and the reason this is a blast-radius fix rather than a cure. + +⚠ **Fleecing costs disk.** The image holds pre-write copies of changed blocks +for the life of the job, on `local-zfs` (717 G free at the time). A long, slow +backup of a busy guest is the case that grows it — watch +`zfs list rpool/data/vm--fleece-0` if the target stays degraded.