Files
esh-pfi-infrastructure/docs
vh 0559e12a2d docs(pfi): add an ops-lessons playbook for the transferable failures
Sibling to model-quantization-playbook.md, and it exists for the same
reason that one does: hard-won lessons were dying inside per-host
runbooks where nobody finds them until after repeating the mistake.

Six entries seeded from the esh-pve-nas migration, all of which would
bite identically on any other host:

1. mount --rbind into a chroot needs --make-rslave, and losing cgroup2
   impersonates failing root-disk I/O closely enough that it was
   misdiagnosed as exactly that.
2. A reboot is not confirmed until the host is observed DOWN; "never
   rebooted" and "rebooted fast" are indistinguishable otherwise.
3. Assert the effective value, not the presence of a substring. Grep
   proves presence; only evaluation proves effect.
4. Ask the server who its clients are -- documented dependent lists rot.
   Plus the corollary that an idle hard NFS mount blocks and resumes, so
   quiescing means stopping consumers, not always unmounting.
5. The scoped-looking command can be the dangerous one; setting a ZFS
   cachefile on one pool of three would have stopped the other two from
   importing at boot.
6. Long uptime hides breakage, and a forced look is worth more than it
   appears -- one migration surfaced an 82-day-dead pvestatd, a 126-day
   hung vzdump, a VM in prelaunch for four months, and an undocumented
   cluster, none of them caused by the work.

Carries a superseded-claims table so corrections are dated rather than
silently edited, same discipline as the quantization playbook. The ESH
runbook now links here so the general rules are reachable from the
specific story and vice versa.
2026-08-18 07:14:48 -07:00
..

docs/

Navigation map for the documentation tree. New session? Read orientation.md first — it's the narrative overview of the fleet, backup architecture, governing principles, and gotchas, and it points at everything else.

Tree

docs/
├── orientation.md            # start here — fleet overview + where-to-look guide
├── runbooks/                 # ops runbooks (recovery, deployment phases)
│   ├── disaster-recovery.md
│   ├── nh3-prune-ritual.md
│   └── pbs-deployment.md
└── pfi/                      # PFI-specific reference (services, models, VMs)
    ├── docker-stack.md
    ├── model-list.md
    ├── proxmox-vms.md
    ├── recommended-model-settings.md
    ├── vm-102-matrix-appservice.md
    └── vm-102-matrix-synapse.md

What goes where

  • runbooks/ — step-by-step ops procedures. Anything you'd reach for during an incident or while standing up new infrastructure. Examples: disaster recovery (blast-radius tiers + restoration steps), PBS deployment (9-phase rollout). New runbook → new file here.
  • pfi/ — PFI-specific reference material that's too narrow for the top-level CLAUDE.md but doesn't change incident response. AI model inventory, recommended inference settings, Matrix bridge config, Proxmox VM map. New stable reference → new file here.
  • Top-level (docs/orientation.md, docs/README.md) — narrative guides about the workspace itself, not about specific infra.

Cross-references

  • Fleet topology + servers table: top-level CLAUDE.md.
  • Open work + recent milestones: top-level STATUS.md.
  • Durable cross-session facts: ~/.claude/projects/-home-lkraven-development-eshpfi-management/memory/.

Conventions

  • Markdown, GitHub-flavored. CommonMark renders fine in most viewers.
  • File names are lowercase-kebab-case, descriptive. No dates in filenames — git history covers that.
  • One topic per file. If a file grows past ~500 lines, look for a natural split before adding more.
  • No checked-in binaries or checksums. Build/release artifacts belong in a build pipeline or tools/, not docs/.