docs(incident): nh3-dev IO stall was a stalled backup, not the disk

hamr-dev reported /dev/sda stalled on nh3-dev: 45 writes in flight
completing zero, jbd2 and flush kworkers in D state for 33 minutes, io
pressure full at 96%, load 26. Every symptom pointed at a dying disk.

It was a vzdump to pbs-ana that collapsed from 64 MiB/s to 1.4 MiB/s and
sat there. Proxmox interposes a copy-before-write filter during a backup,
so every guest write must copy the original block to the target first — the
guest was throttled to the speed of a stalled cross-site backup.

Three things distinguished it, all cheap: there were no device errors
anywhere (a failing disk says so); Dirty was 3.8 MB while 45 requests were
in flight, so the writes were submitted and not completing rather than
backed up in page cache; and the hypervisor was idle at 0.00 io pressure,
which means the writes were not reaching it at all.

Cancelling the task detached the filter: inflight 45 -> 0, D-states gone,
191 MB/s dsync restored. Tonight's incremental is lost and re-runs cheaply,
which is the standing trade in "prefer no backup over one that can crash
the service".

Recurs nightly at 21:00 until changed, and the job has fleecing disabled —
which is why a slow TARGET can stall a GUEST at all.

docs/runbooks/nh3-dev-io-stall.md

@@ -0,0 +1,76 @@
+# A VM wedges on IO and the disk is innocent
+
+First seen 2026-09-03, nh3-dev (VM 102 on nh3-pve). Reported by hamr-dev as
+"/dev/sda stalled — writes hang". It was the backup.
+
+## The signature
+
+    /proc/pressure/io     full avg10=96  (everything is IO-blocked)
+    /sys/block/sda/inflight        0 45  (45 writes in flight, zero completing)
+    D state       jbd2/sda1-8, flush-8:0 kworkers, systemd-journald, any writer
+    dmesg         "blocked for more than 120 seconds", virtio_ring in the trace
+
+Looks exactly like a failing disk or controller. It is not.
+
+## What distinguishes it — and the checks are cheap
+
+**1. There are no device errors.** No SCSI/ATA errors, no `I/O error`, no bus
+resets, pool `ONLINE`, SMART quiet. **A failing disk says so.** Absence of
+errors is evidence, not a gap in it.
+
+**2. `Dirty` in /proc/meminfo is ~zero** while requests pile up in flight. If
+writeback were merely slow, dirty pages would be accumulating. Near-zero Dirty
+plus stuck in-flight requests means the writes were already submitted and are
+not being *completed* — the stall is below the block layer, not above it.
+
+**3. THE HYPERVISOR IS IDLE.** Check it before touching the guest:
+
+    ssh root@nh3-pve 'cat /proc/loadavg; cat /proc/pressure/io; zpool iostat rpool 1 2'
+
+Host load 0.63 and io pressure 0.00 while a guest is 96% IO-blocked means the
+guest's writes are not reaching the host at all. Something between them is
+holding the queue.
+
+**4. Then look at the block layer, which names the culprit outright:**
+
+    ssh root@nh3-pve 'qm monitor <VMID> <<< "info block"'
+    drive-scsi0: /dev/zvol/rpool/data/vm-102-disk-0 (copy-before-write)   ← HERE
+
+    ssh root@nh3-pve 'qm monitor <VMID> <<< "info block-jobs"'
+    Type backup, device drive-scsi0: Completed 545 MB of 19.3 GB
+
+`copy-before-write` is the filter Proxmox interposes during a backup: **every
+guest write must first copy the original block out to the backup target before
+it may proceed.** A slow or stalled target therefore throttles the guest to the
+target's speed. On 2026-09-03 that was **1.4 MiB/s, cross-site to pbs-ana**, for
+35 minutes.
+
+## Fix
+
+    ssh root@nh3-pve 'pvesh delete "/nodes/localhost/tasks/<UPID>"'
+
+Cancels the backup and detaches the filter. Verify: `info block` shows the disk
+as `(raw)` again, `info block-jobs` says `No active jobs`, guest `inflight`
+returns to `0 0`, D-states clear.
+
+⚠ **No reboot is needed and none helps.** Rebooting the guest leaves the filter
+attached and the next writes stall the same way.
+
+⚠ **Cancelling is the standing call**, per the fleet rule *prefer no backup over
+one that can crash the service*. An incremental backup with a dirty bitmap
+re-runs cheaply; an unusable dev box does not.
+
+## Why it can recur, and the real fix
+
+The job is `backup-5d8f1221-8f71`: **daily at 21:00, `all 1`, storage
+`pbs-ana`** — every VM on the host, cross-site to Anaheim.
+
+⚠ **The job has `fleecing 0`.** With fleecing enabled, copy-before-write writes
+land in a fast *local* fleecing image instead of going straight to the backup
+target, which decouples guest IO from target throughput. That is the difference
+between "the backup is slow tonight" and "the box is unusable tonight."
+
+A previous run of this VM managed **941 MiB/s read**, so 1.4 MiB/s is a
+degradation with its own cause — the cross-site link and pbs-ana were both
+responsive (11 ms) throughout, so the throughput cause is **not yet known** and
+enabling fleecing treats the blast radius, not the disease.

persistent-memory.md

@@ -151,6 +151,7 @@ not arrived._
 
 ## Recent decisions
 
+- `[2026-09-03]` **nh3-dev wedged for ~40 min and it was the BACKUP, not the disk — a stalled cross-site vzdump holding every guest write via `copy-before-write`.** Symptoms screamed dying disk: 45 writes in flight completing zero, jbd2 + flush kworkers in D state 33 min, io pressure full 96%, load 26, `virtio_ring` in the stack. ⚠ **The discriminator was the ABSENCE of errors** — no SCSI/ATA/IO errors, rpool ONLINE 21%, guest fs 79%, memory fine, and **Dirty only 3.8 MB** (so nothing backed up in page cache; it was stuck BELOW the block layer). ⚠ **The hypervisor was IDLE** — load 0.63, io pressure 0.00, zpool ~0 writes: nothing was reaching the disk because the filter held it. Cause: `vzdump` of VM 102 → **pbs-ana** did 1% at 64 MiB/s then collapsed to **1.4 MiB/s for 35 min**; Proxmox backups interpose a `copy-before-write` filter, so every guest write queues behind the backup's copy-out. FIX = cancel the task (`pvesh delete /nodes/localhost/tasks/<UPID>`); filter detached, inflight 45→0, D-states gone, 191 MB/s dsync restored. ⚠ **`fleecing 0` on the job is why a slow TARGET can stall a GUEST** — fleecing routes copy-before-write to a fast local image instead. Job = `backup-5d8f1221-8f71`, **daily 21:00, `all 1`, storage pbs-ana** → recurs nightly until changed. A prior run of this VM managed 941 MiB/s read, so 1.4 MiB/s is degradation, not normal. → `docs/runbooks/nh3-dev-io-stall.md`
 - `[2026-09-02]` **althing deploy is SIX surfaces, and #6 is outside the althing repo: `~/.claude/settings.json` `crossSessionInbound: "accept"`.** Without it Claude Code HOLDS every cc poke — it auto-delivers only when the sender's permission-mode class matches, and the herald is a daemon that asserts none, so the notice goes to a human watching the pane instead of to the session. ⚠ **The seat reports declared, reachable and green throughout** — same failure shape as the SessionStart hook that was never deployed. Set on nh3-dev by forseti 09:28 with operator authorization (diff verified: one key, backup at `/tmp/settings.json.bak-20260902T092829`). Operator's reasoning: the herald reaches only local seats and a **pane poke already types+Enters into a session**, so the socket channel is strictly NARROWER than what it replaces — stating the existing trust boundary, not widening it. Cost without it is first-contact-only (in-memory correspondent record), not per-message. ⚠ **No attestation exists for the herald to send** — CC identifies a sender by verified pid against the session registry and reads that session's LIVE runtime mode; a daemon is not in it, and `from_mode` on a `type:"user"` frame is never consulted. **`deploy-althing.sh` reports surface 6 and deliberately never SETS it** — a deploy script that edits its own trust settings grants itself trust. → `docs/runbooks/althing-deploy.md`
 - `[2026-09-02]` **`vastblue` gitea org created (id 8, private, owner `vh`) with empty repo `vastblue/platform`** — third entity namespace alongside `corviduo` and `pfi`; most repos still live under `vh/`. Home of VastBlueDocumentAI + the anchor healthcare-billing SPA (signed 3-yr client contract). **Org scope was the decision**: org-level runner registration and secrets are inherited free by the DocumentAI repo when it splits out, and that is the only binding expensive to retrofit. Deliberately NOT set: org runner (instance-wide `ana-docker-runner` already serves it; org scope is for the DEDICATED runner, deferred to U10) and org secrets (none exist yet; a guessed secret looks bound). ⚠ **Dedicated runner is gated on the first client-premises release cut**, not on the first green pipeline — the risk is another repo's CI sharing a root-level daemon with a build that ships to a healthcare client, see the runner entry above. Push needs no credential: vastblue-dev is on nh3-dev and git-SSH there auths as `vh`. → `stacks/gitea-runner/README.md`
 - `[2026-09-02]` **althing 3.3.0 deployed — the `cc` channel, and a plugin-cache false green.** CC seats are now poked over their own message socket (`$XDG_RUNTIME_DIR/cc-socks/<pid>.sock`) instead of by typing into the pane: no process to reap, nothing near the input line. infra-ops moved to `channel=cc`; **the dwarves stay on pane and their guard-4 exposure is UNCHANGED** (`declare` prefers cc, falls back). ⚠ **An undocumented Claude Code interface, taken deliberately** (operator: the FIFO poker was also an unsanctioned hack — a better instance of a class we already had). Break mode = seat goes pull-only with a logged reason, mail still held. ⚠ **`claude plugin update` matches on the plugin VERSION and declines a content-only change** — 3.3.0 edited plugin content at an unchanged 0.1.1, so the CC cache stayed stale while every version check reported success (delta was docs-only, harmless this time). `deploy-althing.sh` now diffs marketplace vs live cache. ⚠ Ordering: **herald restart BEFORE anything declares cc**, or the seat goes silently pull-only. ⚠ This box was at **3.2.4, not 3.2.5** — rollback target here is 3.2.4. Follow-on 3.3.1: **the statusline bell measured a MECHANISM, not the property** — it read `wake-listener-<handle>.lock`, so a cc seat renders 🔕 while push/reachable. Both copies now ask the post office (`reachable` from the status payload) and add 📵 for an outage. ⚠ **TWO COPIES of that script now exist** — `scripts/claude-statusline-command.sh` here (the operator's wired one) and althing's `plugin/scripts/statusline.sh` — independently fixed to the same shape; a drift surface with a countdown, convergence not yet raised with the operator. → `docs/runbooks/althing-deploy.md`