From bd9692c433caa71a7f83091d6b1372d5c38b272e Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Wed, 2 Sep 2026 21:37:25 -0700 Subject: [PATCH] docs(incident): nh3-dev IO stall was a stalled backup, not the disk MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit hamr-dev reported /dev/sda stalled on nh3-dev: 45 writes in flight completing zero, jbd2 and flush kworkers in D state for 33 minutes, io pressure full at 96%, load 26. Every symptom pointed at a dying disk. It was a vzdump to pbs-ana that collapsed from 64 MiB/s to 1.4 MiB/s and sat there. Proxmox interposes a copy-before-write filter during a backup, so every guest write must copy the original block to the target first — the guest was throttled to the speed of a stalled cross-site backup. Three things distinguished it, all cheap: there were no device errors anywhere (a failing disk says so); Dirty was 3.8 MB while 45 requests were in flight, so the writes were submitted and not completing rather than backed up in page cache; and the hypervisor was idle at 0.00 io pressure, which means the writes were not reaching it at all. Cancelling the task detached the filter: inflight 45 -> 0, D-states gone, 191 MB/s dsync restored. Tonight's incremental is lost and re-runs cheaply, which is the standing trade in "prefer no backup over one that can crash the service". Recurs nightly at 21:00 until changed, and the job has fleecing disabled — which is why a slow TARGET can stall a GUEST at all. --- docs/runbooks/nh3-dev-io-stall.md | 76 +++++++++++++++++++++++++++++++ persistent-memory.md | 1 + 2 files changed, 77 insertions(+) create mode 100644 docs/runbooks/nh3-dev-io-stall.md diff --git a/docs/runbooks/nh3-dev-io-stall.md b/docs/runbooks/nh3-dev-io-stall.md new file mode 100644 index 0000000..d1dbc2f --- /dev/null +++ b/docs/runbooks/nh3-dev-io-stall.md @@ -0,0 +1,76 @@ +# A VM wedges on IO and the disk is innocent + +First seen 2026-09-03, nh3-dev (VM 102 on nh3-pve). Reported by hamr-dev as +"/dev/sda stalled — writes hang". It was the backup. + +## The signature + + /proc/pressure/io full avg10=96 (everything is IO-blocked) + /sys/block/sda/inflight 0 45 (45 writes in flight, zero completing) + D state jbd2/sda1-8, flush-8:0 kworkers, systemd-journald, any writer + dmesg "blocked for more than 120 seconds", virtio_ring in the trace + +Looks exactly like a failing disk or controller. It is not. + +## What distinguishes it — and the checks are cheap + +**1. There are no device errors.** No SCSI/ATA errors, no `I/O error`, no bus +resets, pool `ONLINE`, SMART quiet. **A failing disk says so.** Absence of +errors is evidence, not a gap in it. + +**2. `Dirty` in /proc/meminfo is ~zero** while requests pile up in flight. If +writeback were merely slow, dirty pages would be accumulating. Near-zero Dirty +plus stuck in-flight requests means the writes were already submitted and are +not being *completed* — the stall is below the block layer, not above it. + +**3. THE HYPERVISOR IS IDLE.** Check it before touching the guest: + + ssh root@nh3-pve 'cat /proc/loadavg; cat /proc/pressure/io; zpool iostat rpool 1 2' + +Host load 0.63 and io pressure 0.00 while a guest is 96% IO-blocked means the +guest's writes are not reaching the host at all. Something between them is +holding the queue. + +**4. Then look at the block layer, which names the culprit outright:** + + ssh root@nh3-pve 'qm monitor <<< "info block"' + drive-scsi0: /dev/zvol/rpool/data/vm-102-disk-0 (copy-before-write) ← HERE + + ssh root@nh3-pve 'qm monitor <<< "info block-jobs"' + Type backup, device drive-scsi0: Completed 545 MB of 19.3 GB + +`copy-before-write` is the filter Proxmox interposes during a backup: **every +guest write must first copy the original block out to the backup target before +it may proceed.** A slow or stalled target therefore throttles the guest to the +target's speed. On 2026-09-03 that was **1.4 MiB/s, cross-site to pbs-ana**, for +35 minutes. + +## Fix + + ssh root@nh3-pve 'pvesh delete "/nodes/localhost/tasks/"' + +Cancels the backup and detaches the filter. Verify: `info block` shows the disk +as `(raw)` again, `info block-jobs` says `No active jobs`, guest `inflight` +returns to `0 0`, D-states clear. + +⚠ **No reboot is needed and none helps.** Rebooting the guest leaves the filter +attached and the next writes stall the same way. + +⚠ **Cancelling is the standing call**, per the fleet rule *prefer no backup over +one that can crash the service*. An incremental backup with a dirty bitmap +re-runs cheaply; an unusable dev box does not. + +## Why it can recur, and the real fix + +The job is `backup-5d8f1221-8f71`: **daily at 21:00, `all 1`, storage +`pbs-ana`** — every VM on the host, cross-site to Anaheim. + +⚠ **The job has `fleecing 0`.** With fleecing enabled, copy-before-write writes +land in a fast *local* fleecing image instead of going straight to the backup +target, which decouples guest IO from target throughput. That is the difference +between "the backup is slow tonight" and "the box is unusable tonight." + +A previous run of this VM managed **941 MiB/s read**, so 1.4 MiB/s is a +degradation with its own cause — the cross-site link and pbs-ana were both +responsive (11 ms) throughout, so the throughput cause is **not yet known** and +enabling fleecing treats the blast radius, not the disease. diff --git a/persistent-memory.md b/persistent-memory.md index 9060120..75ac0bf 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -151,6 +151,7 @@ not arrived._ ## Recent decisions +- `[2026-09-03]` **nh3-dev wedged for ~40 min and it was the BACKUP, not the disk — a stalled cross-site vzdump holding every guest write via `copy-before-write`.** Symptoms screamed dying disk: 45 writes in flight completing zero, jbd2 + flush kworkers in D state 33 min, io pressure full 96%, load 26, `virtio_ring` in the stack. ⚠ **The discriminator was the ABSENCE of errors** — no SCSI/ATA/IO errors, rpool ONLINE 21%, guest fs 79%, memory fine, and **Dirty only 3.8 MB** (so nothing backed up in page cache; it was stuck BELOW the block layer). ⚠ **The hypervisor was IDLE** — load 0.63, io pressure 0.00, zpool ~0 writes: nothing was reaching the disk because the filter held it. Cause: `vzdump` of VM 102 → **pbs-ana** did 1% at 64 MiB/s then collapsed to **1.4 MiB/s for 35 min**; Proxmox backups interpose a `copy-before-write` filter, so every guest write queues behind the backup's copy-out. FIX = cancel the task (`pvesh delete /nodes/localhost/tasks/`); filter detached, inflight 45→0, D-states gone, 191 MB/s dsync restored. ⚠ **`fleecing 0` on the job is why a slow TARGET can stall a GUEST** — fleecing routes copy-before-write to a fast local image instead. Job = `backup-5d8f1221-8f71`, **daily 21:00, `all 1`, storage pbs-ana** → recurs nightly until changed. A prior run of this VM managed 941 MiB/s read, so 1.4 MiB/s is degradation, not normal. → `docs/runbooks/nh3-dev-io-stall.md` - `[2026-09-02]` **althing deploy is SIX surfaces, and #6 is outside the althing repo: `~/.claude/settings.json` `crossSessionInbound: "accept"`.** Without it Claude Code HOLDS every cc poke — it auto-delivers only when the sender's permission-mode class matches, and the herald is a daemon that asserts none, so the notice goes to a human watching the pane instead of to the session. ⚠ **The seat reports declared, reachable and green throughout** — same failure shape as the SessionStart hook that was never deployed. Set on nh3-dev by forseti 09:28 with operator authorization (diff verified: one key, backup at `/tmp/settings.json.bak-20260902T092829`). Operator's reasoning: the herald reaches only local seats and a **pane poke already types+Enters into a session**, so the socket channel is strictly NARROWER than what it replaces — stating the existing trust boundary, not widening it. Cost without it is first-contact-only (in-memory correspondent record), not per-message. ⚠ **No attestation exists for the herald to send** — CC identifies a sender by verified pid against the session registry and reads that session's LIVE runtime mode; a daemon is not in it, and `from_mode` on a `type:"user"` frame is never consulted. **`deploy-althing.sh` reports surface 6 and deliberately never SETS it** — a deploy script that edits its own trust settings grants itself trust. → `docs/runbooks/althing-deploy.md` - `[2026-09-02]` **`vastblue` gitea org created (id 8, private, owner `vh`) with empty repo `vastblue/platform`** — third entity namespace alongside `corviduo` and `pfi`; most repos still live under `vh/`. Home of VastBlueDocumentAI + the anchor healthcare-billing SPA (signed 3-yr client contract). **Org scope was the decision**: org-level runner registration and secrets are inherited free by the DocumentAI repo when it splits out, and that is the only binding expensive to retrofit. Deliberately NOT set: org runner (instance-wide `ana-docker-runner` already serves it; org scope is for the DEDICATED runner, deferred to U10) and org secrets (none exist yet; a guessed secret looks bound). ⚠ **Dedicated runner is gated on the first client-premises release cut**, not on the first green pipeline — the risk is another repo's CI sharing a root-level daemon with a build that ships to a healthcare client, see the runner entry above. Push needs no credential: vastblue-dev is on nh3-dev and git-SSH there auths as `vh`. → `stacks/gitea-runner/README.md` - `[2026-09-02]` **althing 3.3.0 deployed — the `cc` channel, and a plugin-cache false green.** CC seats are now poked over their own message socket (`$XDG_RUNTIME_DIR/cc-socks/.sock`) instead of by typing into the pane: no process to reap, nothing near the input line. infra-ops moved to `channel=cc`; **the dwarves stay on pane and their guard-4 exposure is UNCHANGED** (`declare` prefers cc, falls back). ⚠ **An undocumented Claude Code interface, taken deliberately** (operator: the FIFO poker was also an unsanctioned hack — a better instance of a class we already had). Break mode = seat goes pull-only with a logged reason, mail still held. ⚠ **`claude plugin update` matches on the plugin VERSION and declines a content-only change** — 3.3.0 edited plugin content at an unchanged 0.1.1, so the CC cache stayed stale while every version check reported success (delta was docs-only, harmless this time). `deploy-althing.sh` now diffs marketplace vs live cache. ⚠ Ordering: **herald restart BEFORE anything declares cc**, or the seat goes silently pull-only. ⚠ This box was at **3.2.4, not 3.2.5** — rollback target here is 3.2.4. Follow-on 3.3.1: **the statusline bell measured a MECHANISM, not the property** — it read `wake-listener-.lock`, so a cc seat renders 🔕 while push/reachable. Both copies now ask the post office (`reachable` from the status payload) and add 📵 for an outage. ⚠ **TWO COPIES of that script now exist** — `scripts/claude-statusline-command.sh` here (the operator's wired one) and althing's `plugin/scripts/statusline.sh` — independently fixed to the same shape; a drift surface with a countdown, convergence not yet raised with the operator. → `docs/runbooks/althing-deploy.md`