# CLAUDE.md This workspace is for managing PFI infrastructure — servers, Docker stacks, and related configs. Spawn a dedicated Claude Code session here when working on infra so it doesn't clutter AIPA-MCP development context. ## Persistent memory `persistent-memory.md` at the repo root captures durable intent and supporting evidence (goals, decisions, foot-gun warnings, in-flight state) across context resets. Read it at session start; treat it as one input alongside this CLAUDE.md and the auto-memory system, not as the single source of truth. It is a lean **index**: the dated log sections (Recent decisions, Tried and abandoned) keep each over-threshold entry's full body in `persistent-memory.d/.md`. Read the index at session start; pull a detail file only when its index line is relevant to your work — never bulk-read `persistent-memory.d/`. When you commit, stage any pending `persistent-memory.md` and `persistent-memory.d/` updates in the same commit as the work that prompted them — durable memory that lags the code defeats its own purpose. **New session starting here?** Read [`docs/orientation.md`](docs/orientation.md) first — fleet topology, backup architecture, governing principles, and all the NFS/DSM/naming gotchas that have cost past sessions time. **For SSH-driven work: use `scripts/elway`.** Write a playbook under `playbooks/.yaml` and run `scripts/elway --playbook ...` instead of chaining `ssh -t host 'sudo …'` commands — handles sudo once lazily, structured pass/change/fail reporting, idempotency via `creates:` / `when:` / `changed_when:`. Template: [`playbooks/elway-smoke.yaml`](playbooks/elway-smoke.yaml). **Task visibility via task-board.** If the Claude Code session has the `task-board` plugin enabled (installed from `git@gitea.phasefinal.com:vh/task-board.git`), a card at `http://10.250.50.70:7878/` tracks work in progress. Hooks flip the card on turn boundaries automatically; call `task_start` / `task_update` / `task_wait` / `task_complete` MCP tools to set the activity subheader and post meaningful log entries. When you launch a `Bash` tool with `run_in_background: true` (or any long-running shell / monitor / poll loop), call `task_set_shells` with one short description per active background shell — and call it again with the updated list (or `[]`) when one completes. The board flips a waiting card to **orange** while the list is non-empty so the user can tell at a glance the session is parked on background work, not stalled on them. Hooks have no way to enumerate the bg-task list externally, so this is on the assistant. ## Model quantization Quants are hard-fought and we have repeatedly re-litigated the same lessons. **`docs/pfi/model-quantization-playbook.md` is the durable home for the transferable ones** — scheme choice, the recurring landmines, the acceptance gate and its measurement traps, and a superseded-claims table. Read it before starting any quant; read it *instead of* the per-model runbooks for general guidance (several of those carry claims that are now false, and say so). When a quant teaches something **model-agnostic**, it goes in the playbook and the per-model README links up. When it's **model-specific**, it stays in the per-model artifact. If you catch yourself writing a fresh "Gotchas" section that repeats the playbook, you are re-litigating — record the delta in the playbook instead. When a playbook claim turns out wrong, don't just fix it: add a dated row to its superseded-claims table so old docs stop misleading people. ## Training runs Same contract as quantization, different subject: **`docs/pfi/training-throughput-playbook.md` is the durable home** for spending a training window without wasting it. Two halves, and you want different ones at different moments: - **§1–§3, why a run is SLOW** — the 10-minute scaling triage that names the regime before you profile, the padding/masking landmines, the profiler traps, the serving-path and base-viability pre-flights. Read before hypothesising about kernels. - **§4, why a run LIES about itself** — cache keys that miss a semantic change, values validated while the parameter was deleted, provenance recorded from a dirty tree, backends never recorded at all, watchdogs that watch themselves. **Read §4 before you launch**, and run its §4.8 checklist. Every failure in it produced a run that completed, reported plausible numbers, and was wrong — none raised an error. (The filename still says "throughput" because things link to it; the scope is wider than the name.) The instruments are committed at [`scripts/training-probes/`](scripts/training-probes/) with raw output kept alongside, so the claims can be re-derived rather than taken on faith. ⚠ **Measure before you argue.** The playbook exists because a four-model frontier panel produced four self-retractions in ninety minutes on this question, and every one of them was a derivation while every survivor was a measurement. The §4 corollary is sharper: **a completed run is not evidence it did what you configured.** Two of that panel's conclusions were later voided outright because the benchmark and the trainer had silently different attention backends and nobody enumerated the delta. ## Purpose - Inventory of servers and their state - Canonical copies of Docker Compose stacks deployed on those servers - Scripts for inspecting and managing the infrastructure - Conventions so all stacks look the same This is a **reference workspace** — the authoritative copies of compose files and configs live **on the servers** under `/opt/docker/compose//` and `/opt/docker/conf//`. This workspace mirrors them for version control, editing, and planning. ## Conventions (enforce for every new stack) Observed and standardized across servers: - **Compose location on server:** `/opt/docker/compose//compose.yaml` - **Config mounts on server:** `/opt/docker/conf//...` - **Networks:** external `traefik-net`, aliased as `tnet` in compose ```yaml networks: tnet: name: traefik-net external: true ``` - **GPU reservation:** prefer `deploy.resources.reservations.devices` with explicit `device_ids` for pinning ```yaml deploy: resources: reservations: devices: - driver: nvidia device_ids: ["1"] capabilities: [gpu] ``` - **Tunables:** `.env` in the same directory as `compose.yaml` — keep the compose file constant, edit the `.env` - **Named volumes** for service state (pattern: `_`) - **Bind mounts** only for: model files (`/tank/aimodels/...`), config files (`/opt/docker/conf/...`), docker socket where required - **Restart policy:** `restart: unless-stopped` for daemons - **Homepage labels** on user-facing services. The dashboard runs on `esh-docker-vm` and reads the Docker API of **every** host in `stacks/homepage/conf/docker.yaml` (ana-docker, ana-ml2, nh3-docker, irv-ml1, esh-docker-vm), so a labelled container is discovered from wherever it runs — you do not add it to `services.yaml` as well. Doing both renders it twice. ```yaml labels: - homepage.group= - homepage.name= - homepage.icon=mdi- - homepage.description= - homepage.href=http://: ``` ⚠ **`homepage.group` must name a group that already exists in `stacks/homepage/conf/settings.yaml`'s `layout:` block.** A group the layout has never heard of gets no `tab:`, and Homepage renders an untabbed group on **all four tabs**. Inventing a group name here is how Scriberr's `AI Systems` ended up repeated at the bottom of every tab from 2026-08-23 (fixed 2026-08-24). If the service genuinely needs a new group, add the group to `layout:` **with a `tab:`** in the same change. Check with `curl -s http://10.0.50.45:5100/api/services | jq -r '.[].name'` — anything in that list that is not a key in `layout:` is leaking onto all tabs right now. Labels only apply at container **creation**, so a label edit needs `docker compose up -d `, not `restart`. - **Healthchecks** on services that expose HTTP ## Servers | Name | IP | Site | Role | Details | |------|-----|------|------|---------| | ana-ml2 | 10.250.50.54 | Anaheim (`10.250.0.0/16`) | GPU / AI inference (bare metal, dual RTX PRO 6000 Blackwell Max-Q, 96 GB each) | `servers/ana-ml2/README.md` | | irv-ml1 | 10.100.79.3 (WG) | Irvine — reachable only via WireGuard tunnel from NH3 | GPU / AI inference (bare metal, RTX 3090 + RTX A6000, native stacks) | `servers/irv-ml1/README.md` | | ana-docker | 10.250.50.70 | Anaheim | General-purpose Docker host (non-GPU VM on pfi-pve) | `servers/ana-docker/README.md` | | pfi-ana-webhost | 10.250.50.52 | Anaheim | VM on pfi-pve (VMID 110) — web workload | `servers/pfi-ana-webhost/README.md` | | ana-filebot | 10.250.50.53 | Anaheim | LXC on pfi-pve (CT 112) — file-task automation | `servers/ana-filebot/README.md` | | pfi-pteradactyl | 10.250.50.55 | Anaheim | VM on pfi-pve (VMID 107) — Pterodactyl game panel | `servers/pfi-pteradactyl/README.md` | | pfi-tacticalrmm | 10.250.50.57 | Anaheim | VM on pfi-pve (VMID 111) — TacticalRMM | `servers/pfi-tacticalrmm/README.md` | | pfi-postgres | 10.250.50.80 | Anaheim | VM on pfi-pve (VMID 105) — shared Postgres (vaultwarden/gitea/paperless) | `servers/pfi-postgres/README.md` | | ana-wg | 10.250.50.252 | Anaheim | LXC on pfi-pve (CT 113) — WireGuard | `servers/ana-wg/README.md` | | pfi-pve | 10.250.250.31 | Anaheim | Proxmox VE hypervisor | `servers/pfi-pve/README.md` | | pbs-ana | 10.250.50.90 | Anaheim | Proxmox Backup Server — fleet primary (VM on pfi-pve, NFS datastore on ana-nas) | `servers/pbs-ana/README.md` | | sfsrv-ana | 10.250.250.115 | Anaheim | **SureFire client (PFI-managed)** — Proxmox VE hypervisor | `servers/sfsrv-ana/README.md` | | sf-ana-container | 10.250.150.100 | Anaheim | **SureFire client (PFI-managed)** — container workload on sfsrv-ana | `servers/sf-ana-container/README.md` | | sf-r630 | iDRAC 10.250.250.110 | Anaheim | **SureFire client (PFI-managed)** — physical Dell R630, iDRAC-managed from PFI side | `servers/sf-r630/README.md` | | corviduo-dev | 10.250.50.152 | Anaheim | **Worldtree-team dev VM (PFI-hosted)** — runs the demo + personal + pinned Worldtree deployments vor/asset-engine talk to | `servers/corviduo-dev/README.md` | | nh3-docker | 10.100.50.40 | NH3 (`10.100.0.0/16`) | General-purpose Docker host (non-GPU VM on nh3-pve) | `servers/nh3-docker/README.md` | | nh3-dev | 10.100.10.50 | NH3 | Dev box — fleet sidecars (egress SOCKS5 proxy, ttyd seat, mead-hall, volva) + live Claude Code sessions; not a Docker-stack host | `servers/nh3-dev/README.md` | | nh3-extdev | 10.100.50.42 | NH3 | Manager / external-dev box (VM on nh3-pve, Debian 13); **sudo-less** infra-ops identity (user-level only, no Docker); successor to retired nh3-ansible | `servers/nh3-extdev/README.md` | | nh3-pve | 10.100.250.60 | NH3 | Proxmox VE hypervisor | `servers/nh3-pve/README.md` | | nh3-nas | 10.100.50.50 | NH3 | Synology RS2418+ — NFS exports, rest-server-nh3, PBS-NH3 datastore backend | `servers/nh3-nas/README.md` | | pbs-nh3 | 10.100.50.90 | NH3 | Proxmox Backup Server — DR mirror (VM on nh3-pve, NFS datastore on nh3-nas); syncs from pbs-ana | `servers/pbs-nh3/README.md` | | esh-docker-vm | 10.0.50.45 | ESH home lab (`esteban.net`, `10.0.50.0/24`) | Home-lab Docker host (VM on esh-pve) | `servers/esh-docker-vm/README.md` | | vm-esh-nas | 10.0.50.154 | ESH home lab | NAS-adjacent Docker host (VM on esh-pve-nas) | `servers/vm-esh-nas/README.md` | | esh-pve | 10.0.250.35 | ESH home lab | Proxmox VE hypervisor | `servers/esh-pve/README.md` | | esh-pve-nas | 10.0.50.55 | ESH home lab | Proxmox VE hypervisor (storage / media) | `servers/esh-pve-nas/README.md` | | esh-vm-db | 10.0.50.60 | ESH home lab | DB VM — PostgreSQL (paperless-ng) + MongoDB; bare-metal VM, no Docker | `servers/esh-vm-db/README.md` | **Placement rules:** - GPU-required stacks → `ana-ml2` (primary, Anaheim) or `irv-ml1` (secondary, Irvine — bigger VRAM ceiling at 72 GB total). Access to `irv-ml1` requires WireGuard. - Anaheim non-GPU services → `ana-docker`. - NH-site non-GPU services → `nh3-docker`. - ESH home-lab workloads (`esteban.net`) → `esh-docker-vm` (general) or `vm-esh-nas` (needs direct NFS mounts from 10.0.50.50). Not part of the PFI colo topology, but shares monitoring/backup tooling. - Cross-site services (e.g. Beszel hub, Dozzle hub) live on `ana-docker` and pull from agents on the other hosts. - **SureFire (SF) client hosts** (`sf-*`, `sfsrv-ana`) are PFI-managed under the hosting agreement — SSH, OS ops, backups are PFI's responsibility. Hardware and data belong to the client; coordinate anything that affects data with them. - **Worldtree-team dev VM** (`corviduo-dev`) is PFI-hosted (Anaheim subnet) but Worldtree-team-managed at the OS / application layer. PFI handles networking + emergency-ops backstop; OS configuration + deploy workflows + backup decisions live with the architect's team. Treat data-affecting work like SF hosts — coordinate before touching. - **Hypervisors** (`pfi-pve`, `nh3-pve`, `esh-pve`, `esh-pve-nas`) are tracked for inventory / capacity planning. Don't deploy Docker stacks directly on them; new workloads land as VMs. `server_inspect.sh` captures host-level detail only — VM/LXC/ZFS enumeration needs Proxmox-native tooling (`qm list`, `pvesh get …`, `zpool list`). ## How to refresh a server's state ```bash # Show help (no args) scripts/refresh-server-info.sh # Refresh every host discovered under servers/*/ scripts/refresh-server-info.sh all # Refresh a specific host (must match a servers// dir; ssh_config # entry or servers//ssh-target handles how to reach it) scripts/refresh-server-info.sh ana-docker ``` Fleet-wide runs require the literal `all` keyword — no-args prints help so you can't accidentally hit every host by forgetting a name. The script pipes `server_inspect.sh` over SSH via stdin (no scp, no remote cleanup) and writes each `servers//system-details.txt` atomically — a failed run never clobbers the previous snapshot. The inspect script itself is read-only. Each server dir can hold an `ssh-target` file (one line, `` or `@`) as a fallback for when the dir name doesn't resolve via DNS or `~/.ssh/config`. The script prefers whatever ssh would resolve normally and only consults the file when that fails. To register a new server: ```bash scripts/add-host.sh scripts/refresh-server-info.sh # pull the first snapshot ``` To audit discovery without touching the network (checks permissions, unresolvable names with no fallback, missing README / system-details, malformed `ssh-target`): ```bash scripts/refresh-server-info.sh --validate-only all scripts/refresh-server-info.sh --validate-only ``` ## Stack tree convention (canonical vs mirror) Two trees, distinct roles. **They are NOT interchangeable.** | tree | role | git | who writes | who reads | |---|---|---|---|---| | `stacks//` | **canonical / intent** — source of truth for what we want deployed | tracked | you / Claude | `deploy-stack.sh` | | `stacks-mirror///` | **snapshot / reality** — what's currently on each host | gitignored | `sync-stacks.sh` | drift inspection | **Why two:** keeps "intent" (committed, reviewable, deployed) cleanly separate from "reality on the server right now" (often drifts, useful to compare, not durable). Editing the mirror does NOT affect what gets deployed. ```bash # Edit the canonical, then push it to the host: # stacks// → /opt/docker/compose// # stacks//conf/ → /opt/docker/conf// $EDITOR stacks//compose.yaml scripts/deploy-stack.sh # diffs vs live, prompts y/N scripts/deploy-stack.sh --compose # skip conf side scripts/deploy-stack.sh --conf # skip compose side # Pull current host state into the gitignored snapshot tree (drift check): scripts/sync-stacks.sh # every host scripts/sync-stacks.sh ana-docker # one host scripts/sync-stacks.sh --dry-run # see what would change # Compare canonical (intent) vs mirror (reality) for one stack: diff -ru stacks// stacks-mirror/// ``` **Opt-out per stack** (mirror only — sync-stacks.sh skip): create `stacks-mirror///.no-sync` (skip both sides) or `stacks-mirror///conf/.no-sync` (skip conf only). **Always excluded in both directions** (secrets / runtime state): `.env`, `.env.*`, `acme.json`, `client_secrets.json`, `*.pem`, `*.key`, `*.crt`, `*.pfx`, `*.sqlite`, `*.sqlite3`, `*.db`, `*.log`, `*.log.*`, `*.pid`, `hub/`, `logs/`. Requires `rsync` installed on this workstation and every host you sync against (`apt install rsync`). ## Layout ``` eshpfi-management/ ├── CLAUDE.md # this file ├── README.md # human-facing overview ├── scripts/ │ └── server_inspect.sh # gather server state for compose planning ├── servers/ │ └── / │ ├── README.md │ └── system-details.txt # latest server_inspect output ├── stacks/ # canonical/intent — git-tracked source of truth │ └── / │ ├── compose.yaml # deployed to /opt/docker/compose// │ ├── conf/ # deployed to /opt/docker/conf// │ ├── .env.example # template; real .env lives on server │ └── README.md # what this stack does, how to deploy ├── stacks-mirror/ # gitignored snapshot of live host state (drift detection) │ └── // # populated by sync-stacks.sh, NOT a deploy source ├── dns/ # fleet internal DNS — *.internal names │ ├── internal.yaml # source of truth (hosts, sites, aliases) │ └── README.md # workflow, naming, IPv6 caveat └── docs/ └── pfi/ # general PFI infrastructure reference ``` ## Internal DNS (`*.internal`) Fleet hosts have names: `..internal`, sites `ana` / `esh` / `nh3`. `dns/internal.yaml` is the source of truth; the AdGuard resolvers are derived state. ```bash $EDITOR dns/internal.yaml scripts/dns-sync.py --dry-run # diff scripts/dns-sync.py # apply ``` The sync is authoritative **within `.internal` only** — names added by hand in the AdGuard UI get deleted, but rewrites in other zones (ESH's `esteban.net` entries) are left alone. See `dns/README.md`, especially the IPv6 note: v6 addresses only go in the file once they are pinned statically on the host, because SLAAC addresses rotate and a stale record is worse than none. ## Working rules - **Copies, not symlinks.** Files here reflect what's on the server at the time of the last sync. When you edit here, the server doesn't change until you deploy. - **Never commit secrets.** Use `.env.example` templates; real `.env` files (with tokens, passwords) live on the server and are gitignored if/when this becomes a git repo. - **Surgical edits.** When fixing one stack, don't touch unrelated ones. Follow AIPA-MCP's CLAUDE.md rules about scope discipline. - **Sanity-check before deploying.** Run `docker compose config` (dry parse) before `docker compose up -d` on the server.