esh-pfi-infrastructure/CLAUDE.md

CLAUDE.md

This workspace is for managing PFI infrastructure — servers, Docker stacks, and related configs. Spawn a dedicated Claude Code session here when working on infra so it doesn't clutter AIPA-MCP development context.

Persistent memory

persistent-memory.md at the repo root captures durable intent and supporting evidence (goals, decisions, foot-gun warnings, in-flight state) across context resets. Read it at session start; treat it as one input alongside this CLAUDE.md and the auto-memory system, not as the single source of truth.

It is a lean index: the dated log sections (Recent decisions, Tried and abandoned) keep each over-threshold entry's full body in persistent-memory.d/<slug>.md. Read the index at session start; pull a detail file only when its index line is relevant to your work — never bulk-read persistent-memory.d/. When you commit, stage any pending persistent-memory.md and persistent-memory.d/ updates in the same commit as the work that prompted them — durable memory that lags the code defeats its own purpose.

New session starting here? Read docs/orientation.md first — fleet topology, backup architecture, governing principles, and all the NFS/DSM/naming gotchas that have cost past sessions time.

SSH work splits two ways. Pick by whether you are CHANGING something or LOOKING at something — neither tool replaces the other.

Changing things → scripts/elway. Write a playbook under playbooks/<name>.yaml and run scripts/elway <host> --playbook ... instead of chaining ssh -t host 'sudo …' commands — handles sudo once lazily, structured pass/change/fail reporting, idempotency via creates: / when: / changed_when:, and file upload. Template: playbooks/elway-smoke.yaml.

Looking at things → the remote-ssh MCP server. Registered project-scoped in .mcp.json; tools ssh_hosts / ssh_open / ssh_run / ssh_peek / ssh_interrupt / ssh_list / ssh_close. It holds a persistent shell, so cd and exported variables survive between calls and each command costs ~6 ms instead of a fresh handshake. Reach for it for reconnaissance — the docker ps / grep / curl questions where authoring a playbook is ceremony. It deliberately has no file transfer and no idempotency; that is elway's half.

• Source: ~/development/remote-ssh-mcp — our fork of the-nine-nation/remote-ssh-mcp (MIT). .mcp.json points at the built dist/, so after editing the fork run npm run build or the server keeps serving the old code.
• ⚠ The host allowlist is the only real containment boundary. The denylist is three regexes and a bash -c walks past it. We run with SSH_MCP_STRICT_ALLOWLIST=1, which makes ~/.config/remote-ssh-mcp/config.json's allowedHosts authoritative rather than additive — without it, every one of the 18 Host entries in ~/.ssh/config is reachable. Widening that list is a deliberate act; do it in the config file, not by relying on discovery.
• Raw IPs work — no alias needed. Operator ruling 2026-09-05: requiring a host to be registered before you can poke at it is the opposite of ad-hoc, and the predictable result is that you use raw ssh instead. So ssh_open takes an address inside allowedNetworks (10.0.0.0/8) and connects as defaultUser=infra-ops with ~/.ssh/infra-ops_ed25519, hostKeyPolicy set to accept-new. Aliases still work and are still required outside those networks.
• ⚠ deniedNetworks carves out the SureFire tenant hosts — 10.250.150.0/24, 10.250.250.115, 10.250.250.110. Deny beats allow, so widening the allow list later cannot re-expose them. They are client property under the hosting agreement; coordinate before touching, which is a contractual posture and not a security one. pfi-pve (10.250.250.31) is deliberately NOT caught by this — the denies are host-specific, not a /24 over shared space.
• ⚠⚠ A bare sudo HANGS the session — always pipe it: sudo -n <cmd> | cat. ssh_run 'sudo -n whoami' returns running forever with zero output and the session is then permanently busy; the piped form returns everything. Measured on both macOS and Debian, so it is the harness and not a platform quirk. Cause: sudo ≥1.9.14 defaults use_pty on and relays through its own PTY; with the run frame's stdin on /dev/null and stdout on the session PTY the relay never completes, so the completion marker never arrives. A pipe makes stdout a non-tty and sudo skips the PTY. Not yet fixed in our fork.

Task visibility via task-board. If the Claude Code session has the task-board plugin enabled (installed from git@gitea.phasefinal.com:vh/task-board.git), a card at http://10.250.50.70:7878/ tracks work in progress. Hooks flip the card on turn boundaries automatically; call task_start / task_update / task_wait / task_complete MCP tools to set the activity subheader and post meaningful log entries.

When you launch a Bash tool with run_in_background: true (or any long-running shell / monitor / poll loop), call task_set_shells with one short description per active background shell — and call it again with the updated list (or []) when one completes. The board flips a waiting card to orange while the list is non-empty so the user can tell at a glance the session is parked on background work, not stalled on them. Hooks have no way to enumerate the bg-task list externally, so this is on the assistant.

Model quantization

Quants are hard-fought and we have repeatedly re-litigated the same lessons. docs/pfi/model-quantization-playbook.md is the durable home for the transferable ones — scheme choice, the recurring landmines, the acceptance gate and its measurement traps, and a superseded-claims table. Read it before starting any quant; read it instead of the per-model runbooks for general guidance (several of those carry claims that are now false, and say so).

When a quant teaches something model-agnostic, it goes in the playbook and the per-model README links up. When it's model-specific, it stays in the per-model artifact. If you catch yourself writing a fresh "Gotchas" section that repeats the playbook, you are re-litigating — record the delta in the playbook instead. When a playbook claim turns out wrong, don't just fix it: add a dated row to its superseded-claims table so old docs stop misleading people.

Training runs

Same contract as quantization, different subject: docs/pfi/training-throughput-playbook.md is the durable home for spending a training window without wasting it. Two halves, and you want different ones at different moments:

• §1–§3, why a run is SLOW — the 10-minute scaling triage that names the regime before you profile, the padding/masking landmines, the profiler traps, the serving-path and base-viability pre-flights. Read before hypothesising about kernels.
• §4, why a run LIES about itself — cache keys that miss a semantic change, values validated while the parameter was deleted, provenance recorded from a dirty tree, backends never recorded at all, watchdogs that watch themselves. Read §4 before you launch, and run its §4.8 checklist. Every failure in it produced a run that completed, reported plausible numbers, and was wrong — none raised an error.

(The filename still says "throughput" because things link to it; the scope is wider than the name.)

The instruments are committed at scripts/training-probes/ with raw output kept alongside, so the claims can be re-derived rather than taken on faith.

⚠ Measure before you argue. The playbook exists because a four-model frontier panel produced four self-retractions in ninety minutes on this question, and every one of them was a derivation while every survivor was a measurement. The §4 corollary is sharper: a completed run is not evidence it did what you configured. Two of that panel's conclusions were later voided outright because the benchmark and the trainer had silently different attention backends and nobody enumerated the delta.

Purpose

• Inventory of servers and their state
• Canonical copies of Docker Compose stacks deployed on those servers
• Scripts for inspecting and managing the infrastructure
• Conventions so all stacks look the same

This is a reference workspace — the authoritative copies of compose files and configs live on the servers under /opt/docker/compose/<stack>/ and /opt/docker/conf/<stack>/. This workspace mirrors them for version control, editing, and planning.

Conventions (enforce for every new stack)

Observed and standardized across servers:

• Compose location on server: /opt/docker/compose/<stack>/compose.yaml
• Config mounts on server: /opt/docker/conf/<stack>/...
• Networks: external traefik-net, aliased as tnet in compose

  networks:
    tnet:
      name: traefik-net
      external: true
  

• GPU reservation: prefer deploy.resources.reservations.devices with explicit device_ids for pinning

  deploy:
    resources:
      reservations:
        devices:
          - driver: nvidia
            device_ids: ["1"]
            capabilities: [gpu]
  

• Tunables: .env in the same directory as compose.yaml — keep the compose file constant, edit the .env
• Named volumes for service state (pattern: <stack>_<name>)
• Bind mounts only for: model files (/tank/aimodels/...), config files (/opt/docker/conf/...), docker socket where required
• Restart policy: restart: unless-stopped for daemons
• Homepage labels on user-facing services. The dashboard runs on esh-docker-vm and reads the Docker API of every host in stacks/homepage/conf/docker.yaml (ana-docker, ana-ml2, nh3-docker, irv-ml1, esh-docker-vm), so a labelled container is discovered from wherever it runs — you do not add it to services.yaml as well. Doing both renders it twice.

  labels:
    - homepage.group=<ExistingGroup>
    - homepage.name=<ServiceName>
    - homepage.icon=mdi-<icon>
    - homepage.description=<short>
    - homepage.href=http://<host-ip>:<port>
  

  ⚠ homepage.group must name a group that already exists in stacks/homepage/conf/settings.yaml's layout: block. A group the layout has never heard of gets no tab:, and Homepage renders an untabbed group on all four tabs. Inventing a group name here is how Scriberr's AI Systems ended up repeated at the bottom of every tab from 2026-08-23 (fixed 2026-08-24). If the service genuinely needs a new group, add the group to layout: with a tab: in the same change. Check with curl -s http://10.0.50.45:5100/api/services | jq -r '.[].name' — anything in that list that is not a key in layout: is leaking onto all tabs right now. Labels only apply at container creation, so a label edit needs docker compose up -d <service>, not restart.
• Healthchecks on services that expose HTTP

Servers

Name	IP	Site	Role	Details
ana-ml2	10.250.50.54	Anaheim (10.250.0.0/16)	GPU / AI inference (bare metal, dual RTX PRO 6000 Blackwell Max-Q, 96 GB each)	servers/ana-ml2/README.md
irv-ml1	100.64.0.6 (headscale mesh); 10.6.110.50 (Irvine LAN, routed via mesh)	Irvine — via headscale mesh; wg0 lifeline (10.100.79.3) dormant post-cutover	GPU / AI inference (bare metal, RTX 3090 + RTX A6000, native stacks)	servers/irv-ml1/README.md
ana-docker	10.250.50.70	Anaheim	General-purpose Docker host (non-GPU VM on pfi-pve)	servers/ana-docker/README.md
pfi-ana-webhost	10.250.50.52	Anaheim	VM on pfi-pve (VMID 110) — web workload	servers/pfi-ana-webhost/README.md
ana-filebot	10.250.50.53	Anaheim	LXC on pfi-pve (CT 112) — file-task automation	servers/ana-filebot/README.md
pfi-pteradactyl	10.250.50.55	Anaheim	VM on pfi-pve (VMID 107) — Pterodactyl game panel	servers/pfi-pteradactyl/README.md
pfi-tacticalrmm	10.250.50.57	Anaheim	VM on pfi-pve (VMID 111) — TacticalRMM	servers/pfi-tacticalrmm/README.md
pfi-postgres	10.250.50.80	Anaheim	VM on pfi-pve (VMID 105) — shared Postgres (vaultwarden/gitea/paperless)	servers/pfi-postgres/README.md
ana-wg	10.250.50.252	Anaheim	LXC on pfi-pve (CT 113) — WireGuard	servers/ana-wg/README.md
pfi-pve	10.250.250.31	Anaheim	Proxmox VE hypervisor	servers/pfi-pve/README.md
pbs-ana	10.250.50.90	Anaheim	Proxmox Backup Server — fleet primary (VM on pfi-pve, NFS datastore on ana-nas)	servers/pbs-ana/README.md
sfsrv-ana	10.250.250.115	Anaheim	SureFire client (PFI-managed) — Proxmox VE hypervisor	servers/sfsrv-ana/README.md
sf-ana-container	10.250.150.100	Anaheim	SureFire client (PFI-managed) — container workload on sfsrv-ana	servers/sf-ana-container/README.md
sf-r630	iDRAC 10.250.250.110	Anaheim	SureFire client (PFI-managed) — physical Dell R630, iDRAC-managed from PFI side	servers/sf-r630/README.md
corviduo-dev	10.250.50.152	Anaheim	Worldtree-team dev VM (PFI-hosted) — runs the demo + personal + pinned Worldtree deployments vor/asset-engine talk to	servers/corviduo-dev/README.md
nh3-docker	10.100.50.40	NH3 (10.100.0.0/16)	General-purpose Docker host (non-GPU VM on nh3-pve)	servers/nh3-docker/README.md
nh3-dev	10.100.10.50	NH3	Dev box — fleet sidecars (egress SOCKS5 proxy, ttyd seat, mead-hall, volva) + live Claude Code sessions; not a Docker-stack host	servers/nh3-dev/README.md
nh3-extdev	10.100.50.42	NH3	Manager / external-dev box (VM on nh3-pve, Debian 13); infra-ops is user-level and has no Docker, but does have NOPASSWD sudo — measured 2026-09-05, sudo -n whoami → root; this row previously said "sudo-less", which was wrong; successor to retired nh3-ansible	servers/nh3-extdev/README.md
nh3-pve	10.100.250.60	NH3	Proxmox VE hypervisor	servers/nh3-pve/README.md
nh3-nas	10.100.50.50	NH3	Synology RS2418+ — NFS exports, rest-server-nh3, PBS-NH3 datastore backend	servers/nh3-nas/README.md
pbs-nh3	10.100.50.90	NH3	Proxmox Backup Server — DR mirror (VM on nh3-pve, NFS datastore on nh3-nas); syncs from pbs-ana	servers/pbs-nh3/README.md
esh-docker-vm	10.0.50.45	ESH home lab (esteban.net, 10.0.50.0/24)	Home-lab Docker host (VM on esh-pve)	servers/esh-docker-vm/README.md
vm-esh-nas	10.0.50.154	ESH home lab	NAS-adjacent Docker host (VM on esh-pve-nas)	servers/vm-esh-nas/README.md
esh-pve	10.0.250.35	ESH home lab	Proxmox VE hypervisor	servers/esh-pve/README.md
esh-pve-nas	10.0.50.55	ESH home lab	Proxmox VE hypervisor (storage / media)	servers/esh-pve-nas/README.md
esh-vm-db	10.0.50.60	ESH home lab	DB VM — PostgreSQL (paperless-ng) + MongoDB; bare-metal VM, no Docker	servers/esh-vm-db/README.md

Placement rules:

• GPU-required stacks → ana-ml2 (primary, Anaheim) or irv-ml1 (secondary, Irvine — bigger VRAM ceiling at 72 GB total). Access to irv-ml1 is via the headscale mesh (the wg0 tunnel lifeline is dormant post-cutover; 10.100.79.3 no longer answers).
• Anaheim non-GPU services → ana-docker.
• NH-site non-GPU services → nh3-docker.
• ESH home-lab workloads (esteban.net) → esh-docker-vm (general) or vm-esh-nas (needs direct NFS mounts from 10.0.50.50). Not part of the PFI colo topology, but shares monitoring/backup tooling.
• Cross-site services (e.g. Beszel hub, Dozzle hub) live on ana-docker and pull from agents on the other hosts.
• SureFire (SF) client hosts (sf-*, sfsrv-ana) are PFI-managed under the hosting agreement — SSH, OS ops, backups are PFI's responsibility. Hardware and data belong to the client; coordinate anything that affects data with them.
• Worldtree-team dev VM (corviduo-dev) is PFI-hosted (Anaheim subnet) but Worldtree-team-managed at the OS / application layer. PFI handles networking + emergency-ops backstop; OS configuration + deploy workflows + backup decisions live with the architect's team. Treat data-affecting work like SF hosts — coordinate before touching.
• Hypervisors (pfi-pve, nh3-pve, esh-pve, esh-pve-nas) are tracked for inventory / capacity planning. Don't deploy Docker stacks directly on them; new workloads land as VMs. server_inspect.sh captures host-level detail only — VM/LXC/ZFS enumeration needs Proxmox-native tooling (qm list, pvesh get …, zpool list).

How to refresh a server's state

# Show help (no args)
scripts/refresh-server-info.sh

# Refresh every host discovered under servers/*/
scripts/refresh-server-info.sh all

# Refresh a specific host (must match a servers/<name>/ dir; ssh_config
# entry or servers/<name>/ssh-target handles how to reach it)
scripts/refresh-server-info.sh ana-docker

Fleet-wide runs require the literal all keyword — no-args prints help so you can't accidentally hit every host by forgetting a name.

The script pipes server_inspect.sh over SSH via stdin (no scp, no remote cleanup) and writes each servers/<host>/system-details.txt atomically — a failed run never clobbers the previous snapshot, and neither does a run that succeeds but captures nothing: ssh exiting 0 is not proof of a usable capture, so an empty result is refused, the previous snapshot is kept, and the host is counted as failed (2026-09-09 — the readers all test the snapshot with -s, so the writer must not produce something they would call invalid). A capture that collapses to under a quarter of the previous one still promotes but is flagged, because a host really can shed services. The inspect script itself is read-only.

Each server dir can hold an ssh-target file (one line, <ip> or <user>@<ip>) as a fallback for when the dir name doesn't resolve via DNS or ~/.ssh/config. The script prefers whatever ssh would resolve normally and only consults the file when that fails.

To register a new server:

scripts/add-host.sh <name> <ip-or-user@ip>
scripts/refresh-server-info.sh <name>    # pull the first snapshot

To audit discovery without touching the network (checks permissions, unresolvable names with no fallback, missing README / system-details, malformed ssh-target):

scripts/refresh-server-info.sh --validate-only all
scripts/refresh-server-info.sh --validate-only <host>

Stack tree convention (canonical vs mirror)

Two trees, distinct roles. They are NOT interchangeable.

tree	role	git	who writes	who reads
stacks/<stack>/	canonical / intent — source of truth for what we want deployed	tracked	you / Claude	deploy-stack.sh
stacks-mirror/<host>/<stack>/	snapshot / reality — what's currently on each host	gitignored	sync-stacks.sh	drift inspection

Why two: keeps "intent" (committed, reviewable, deployed) cleanly separate from "reality on the server right now" (often drifts, useful to compare, not durable). Editing the mirror does NOT affect what gets deployed.

# Edit the canonical, then push it to the host:
# stacks/<stack>/<file>      → /opt/docker/compose/<stack>/<file>
# stacks/<stack>/conf/<file> → /opt/docker/conf/<stack>/<file>
$EDITOR stacks/<stack>/compose.yaml
scripts/deploy-stack.sh <host> <stack>      # diffs vs live, prompts y/N
scripts/deploy-stack.sh <host> <stack> --compose   # skip conf side
scripts/deploy-stack.sh <host> <stack> --conf      # skip compose side

# Pull current host state into the gitignored snapshot tree (drift check):
scripts/sync-stacks.sh                     # every host
scripts/sync-stacks.sh ana-docker          # one host
scripts/sync-stacks.sh --dry-run           # see what would change

# Compare canonical (intent) vs mirror (reality) for one stack:
diff -ru stacks/<stack>/ stacks-mirror/<host>/<stack>/

Opt-out per stack (mirror only — sync-stacks.sh skip): create stacks-mirror/<host>/<stack>/.no-sync (skip both sides) or stacks-mirror/<host>/<stack>/conf/.no-sync (skip conf only).

Always excluded in both directions (secrets / runtime state): .env, .env.*, acme.json, client_secrets.json, *.pem, *.key, *.crt, *.pfx, *.sqlite, *.sqlite3, *.db, *.log, *.log.*, *.pid, hub/, logs/.

Requires rsync installed on this workstation and every host you sync against (apt install rsync).

Layout

eshpfi-management/
├── CLAUDE.md              # this file
├── README.md              # human-facing overview
├── scripts/
│   └── server_inspect.sh  # gather server state for compose planning
├── servers/
│   └── <name>/
│       ├── README.md
│       └── system-details.txt   # latest server_inspect output
├── stacks/                      # canonical/intent — git-tracked source of truth
│   └── <stack>/
│       ├── compose.yaml         # deployed to /opt/docker/compose/<stack>/
│       ├── conf/<file>          # deployed to /opt/docker/conf/<stack>/<file>
│       ├── .env.example         # template; real .env lives on server
│       └── README.md            # what this stack does, how to deploy
├── stacks-mirror/               # gitignored snapshot of live host state (drift detection)
│   └── <host>/<stack>/          # populated by sync-stacks.sh, NOT a deploy source
├── dns/                         # fleet internal DNS — *.internal names
│   ├── internal.yaml            # source of truth (hosts, sites, aliases)
│   └── README.md                # workflow, naming, IPv6 caveat
└── docs/
    └── pfi/                     # general PFI infrastructure reference

Internal DNS (*.internal)

Fleet hosts have names: <host>.<site>.internal, sites ana / esh / nh3. dns/internal.yaml is the source of truth; the AdGuard resolvers are derived state.

$EDITOR dns/internal.yaml
scripts/dns-sync.py --dry-run     # diff
scripts/dns-sync.py               # apply

The sync is authoritative within .internal only — names added by hand in the AdGuard UI get deleted, but rewrites in other zones (ESH's esteban.net entries) are left alone. See dns/README.md, especially the IPv6 note: v6 addresses only go in the file once they are pinned statically on the host, because SLAAC addresses rotate and a stale record is worse than none.

Working rules

• Copies, not symlinks. Files here reflect what's on the server at the time of the last sync. When you edit here, the server doesn't change until you deploy.
• Never commit secrets. Use .env.example templates; real .env files (with tokens, passwords) live on the server and are gitignored if/when this becomes a git repo.
• Surgical edits. When fixing one stack, don't touch unrelated ones. Follow AIPA-MCP's CLAUDE.md rules about scope discipline.
• Sanity-check before deploying. Run docker compose config (dry parse) before docker compose up -d on the server.