vh
bd9692c433
hamr-dev reported /dev/sda stalled on nh3-dev: 45 writes in flight completing zero, jbd2 and flush kworkers in D state for 33 minutes, io pressure full at 96%, load 26. Every symptom pointed at a dying disk. It was a vzdump to pbs-ana that collapsed from 64 MiB/s to 1.4 MiB/s and sat there. Proxmox interposes a copy-before-write filter during a backup, so every guest write must copy the original block to the target first — the guest was throttled to the speed of a stalled cross-site backup. Three things distinguished it, all cheap: there were no device errors anywhere (a failing disk says so); Dirty was 3.8 MB while 45 requests were in flight, so the writes were submitted and not completing rather than backed up in page cache; and the hypervisor was idle at 0.00 io pressure, which means the writes were not reaching it at all. Cancelling the task detached the filter: inflight 45 -> 0, D-states gone, 191 MB/s dsync restored. Tonight's incremental is lost and re-runs cheaply, which is the standing trade in "prefer no backup over one that can crash the service". Recurs nightly at 21:00 until changed, and the job has fleecing disabled — which is why a slow TARGET can stall a GUEST at all.
docs/
Navigation map for the documentation tree. New session? Read
orientation.md first — it's the narrative overview
of the fleet, backup architecture, governing principles, and gotchas,
and it points at everything else.
Tree
docs/
├── orientation.md # start here — fleet overview + where-to-look guide
├── runbooks/ # ops runbooks (recovery, deployment phases)
│ ├── disaster-recovery.md
│ ├── nh3-prune-ritual.md
│ └── pbs-deployment.md
└── pfi/ # PFI-specific reference (services, models, VMs)
├── docker-stack.md
├── model-list.md
├── proxmox-vms.md
├── recommended-model-settings.md
├── vm-102-matrix-appservice.md
└── vm-102-matrix-synapse.md
What goes where
runbooks/— step-by-step ops procedures. Anything you'd reach for during an incident or while standing up new infrastructure. Examples: disaster recovery (blast-radius tiers + restoration steps), PBS deployment (9-phase rollout). New runbook → new file here.pfi/— PFI-specific reference material that's too narrow for the top-level CLAUDE.md but doesn't change incident response. AI model inventory, recommended inference settings, Matrix bridge config, Proxmox VM map. New stable reference → new file here.- Top-level (
docs/orientation.md,docs/README.md) — narrative guides about the workspace itself, not about specific infra.
Cross-references
- Fleet topology + servers table: top-level
CLAUDE.md. - Open work + recent milestones: top-level
STATUS.md. - Durable cross-session facts:
~/.claude/projects/-home-lkraven-development-eshpfi-management/memory/.
Conventions
- Markdown, GitHub-flavored. CommonMark renders fine in most viewers.
- File names are lowercase-kebab-case, descriptive. No dates in filenames — git history covers that.
- One topic per file. If a file grows past ~500 lines, look for a natural split before adding more.
- No checked-in binaries or checksums. Build/release artifacts belong
in a build pipeline or
tools/, notdocs/.