vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
58f2a229664440aed609028114d868791703a59e
esh-pfi-infrastructure/CLAUDE.md
T
vh 017f8f9b76 fleet: re-frame SureFire hosts from tenant-only to PFI-managed 
Initial framing was wrong. PFI runs these under a managed-hosting
agreement: SSH, OS ops, backups are all PFI's responsibility. Hardware
and data belong to the client.

Changes:
- ssh-target files added for sfsrv-ana (root@10.250.250.115 — same
  pattern as other PVE nodes) and sf-ana-container
  (lkraven@10.250.150.100 guess, adjust if different user).
- sf-r630 still lacks an ssh-target — the OS-side LAN IP isn't in
  FortiGate DHCP (static config somewhere). Will fill in once
  identified; README flags that gap.
- READMEs rewritten: dropped "tenant-scoped" / "not SSH-managed"
  language, added "client context" section that explains the
  managed-hosting relationship. Backup coverage now listed as
  planned rather than blocked on tenant coordination.
- CLAUDE.md fleet table: SF rows re-labeled "SureFire client
  (PFI-managed)". Placement-rules section updated to note that
  SF hosts are first-class PFI-ops targets, just client-owned.
- Memory (project_surefire_tenant.md) rewritten to reflect
  managed-services reality + hosts-file entries needed for name
  resolution since these aren't in PFI DNS.
2026-04-21 16:51:50 -07:00

9.2 KiB
Raw Blame History

CLAUDE.md

This workspace is for managing PFI infrastructure — servers, Docker stacks, and related configs. Spawn a dedicated Claude Code session here when working on infra so it doesn't clutter AIPA-MCP development context.

Purpose

  • Inventory of servers and their state
  • Canonical copies of Docker Compose stacks deployed on those servers
  • Scripts for inspecting and managing the infrastructure
  • Conventions so all stacks look the same

This is a reference workspace — the authoritative copies of compose files and configs live on the servers under /opt/docker/compose/<stack>/ and /opt/docker/conf/<stack>/. This workspace mirrors them for version control, editing, and planning.

Conventions (enforce for every new stack)

Observed and standardized across servers:

  • Compose location on server: /opt/docker/compose/<stack>/compose.yaml
  • Config mounts on server: /opt/docker/conf/<stack>/...
  • Networks: external traefik-net, aliased as tnet in compose 
    networks:
  tnet:
    name: traefik-net
    external: true
  • GPU reservation: prefer deploy.resources.reservations.devices with explicit device_ids for pinning 
    deploy:
  resources:
    reservations:
      devices:
        - driver: nvidia
          device_ids: ["1"]
          capabilities: [gpu]
  • Tunables: .env in the same directory as compose.yaml — keep the compose file constant, edit the .env
  • Named volumes for service state (pattern: <stack>_<name>)
  • Bind mounts only for: model files (/tank/aimodels/...), config files (/opt/docker/conf/...), docker socket where required
  • Restart policy: restart: unless-stopped for daemons
  • Homepage labels on user-facing services: 
    labels:
  - homepage.group=AI Systems
  - homepage.name=<ServiceName>
  - homepage.icon=mdi-<icon>
  - homepage.description=<short>
  - homepage.href=http://<host-ip>:<port>
  • Healthchecks on services that expose HTTP

Servers

Name IP Site Role Details
ana-ml2 10.250.50.54 Anaheim (10.250.0.0/16) GPU / AI inference (bare metal) servers/ana-ml2/README.md
ana-docker 10.250.50.70 Anaheim General-purpose Docker host (non-GPU VM on pfi-pve) servers/ana-docker/README.md
pfi-ana-webhost 10.250.50.52 Anaheim VM on pfi-pve (VMID 110) — web workload servers/pfi-ana-webhost/README.md
ana-filebot 10.250.50.53 Anaheim LXC on pfi-pve (CT 112) — file-task automation servers/ana-filebot/README.md
pfi-pteradactyl 10.250.50.55 Anaheim VM on pfi-pve (VMID 107) — Pterodactyl game panel servers/pfi-pteradactyl/README.md
pfi-tacticalrmm 10.250.50.57 Anaheim VM on pfi-pve (VMID 111) — TacticalRMM servers/pfi-tacticalrmm/README.md
pfi-postgres 10.250.50.80 Anaheim VM on pfi-pve (VMID 105) — shared Postgres (vaultwarden/gitea/paperless) servers/pfi-postgres/README.md
ana-wg 10.250.50.252 Anaheim LXC on pfi-pve (CT 113) — WireGuard servers/ana-wg/README.md
pfi-pve 10.250.250.31 Anaheim Proxmox VE hypervisor servers/pfi-pve/README.md
sfsrv-ana 10.250.250.115 Anaheim SureFire client (PFI-managed) — Proxmox VE hypervisor servers/sfsrv-ana/README.md
sf-ana-container 10.250.150.100 Anaheim SureFire client (PFI-managed) — container workload on sfsrv-ana servers/sf-ana-container/README.md
sf-r630 iDRAC 10.250.250.110 Anaheim SureFire client (PFI-managed) — physical Dell R630, iDRAC-managed from PFI side servers/sf-r630/README.md
nh3-docker 10.100.50.40 NH3 (10.100.0.0/16) General-purpose Docker host (non-GPU VM on nh3-pve) servers/nh3-docker/README.md
nh3-pve 10.100.250.60 NH3 Proxmox VE hypervisor servers/nh3-pve/README.md
esh-docker-vm 10.0.50.45 ESH home lab (esteban.net, 10.0.50.0/24) Home-lab Docker host (VM on esh-pve) servers/esh-docker-vm/README.md
vm-esh-nas 10.0.50.154 ESH home lab NAS-adjacent Docker host (VM on esh-pve-nas) servers/vm-esh-nas/README.md
esh-pve 10.0.250.35 ESH home lab Proxmox VE hypervisor servers/esh-pve/README.md
esh-pve-nas 10.0.50.55 ESH home lab Proxmox VE hypervisor (storage / media) servers/esh-pve-nas/README.md

Placement rules:

  • GPU-required stacks → ana-ml2.
  • Anaheim non-GPU services → ana-docker.
  • NH-site non-GPU services → nh3-docker.
  • ESH home-lab workloads (esteban.net) → esh-docker-vm (general) or vm-esh-nas (needs direct NFS mounts from 10.0.50.50). Not part of the PFI colo topology, but shares monitoring/backup tooling.
  • Cross-site services (e.g. Beszel hub, Dozzle hub) live on ana-docker and pull from agents on the other hosts.
  • SureFire (SF) client hosts (sf-*, sfsrv-ana) are PFI-managed under the hosting agreement — SSH, OS ops, backups are PFI's responsibility. Hardware and data belong to the client; coordinate anything that affects data with them.
  • Hypervisors (pfi-pve, nh3-pve, esh-pve, esh-pve-nas) are tracked for inventory / capacity planning. Don't deploy Docker stacks directly on them; new workloads land as VMs. server_inspect.sh captures host-level detail only — VM/LXC/ZFS enumeration needs Proxmox-native tooling (qm list, pvesh get …, zpool list).

How to refresh a server's state

# Show help (no args)
scripts/refresh-server-info.sh

# Refresh every host discovered under servers/*/
scripts/refresh-server-info.sh all

# Refresh a specific host (must match a servers/<name>/ dir; ssh_config
# entry or servers/<name>/ssh-target handles how to reach it)
scripts/refresh-server-info.sh ana-docker

Fleet-wide runs require the literal all keyword — no-args prints help so you can't accidentally hit every host by forgetting a name.

The script pipes server_inspect.sh over SSH via stdin (no scp, no remote cleanup) and writes each servers/<host>/system-details.txt atomically — a failed run never clobbers the previous snapshot. The inspect script itself is read-only.

Each server dir can hold an ssh-target file (one line, <ip> or <user>@<ip>) as a fallback for when the dir name doesn't resolve via DNS or ~/.ssh/config. The script prefers whatever ssh would resolve normally and only consults the file when that fails.

To register a new server:

scripts/add-host.sh <name> <ip-or-user@ip>
scripts/refresh-server-info.sh <name>    # pull the first snapshot

To audit discovery without touching the network (checks permissions, unresolvable names with no fallback, missing README / system-details, malformed ssh-target):

scripts/refresh-server-info.sh --validate-only all
scripts/refresh-server-info.sh --validate-only <host>

Stack mirror (pull / push)

Compose and config trees are mirrored into stacks-mirror/<host>/<stack>/ so they can be diffed and version-controlled. Pull is fleet-wide and safe; push is one stack at a time with a diff + prompt.

# Pull compose + conf from every host into stacks-mirror/
scripts/sync-stacks.sh
scripts/sync-stacks.sh --dry-run     # see what would change
scripts/sync-stacks.sh ana-docker    # one host

# Push a local stack back to the server (diffs each file, prompts y/N)
scripts/deploy-stack.sh <host> <stack>
scripts/deploy-stack.sh <host> <stack> --compose   # skip conf
scripts/deploy-stack.sh <host> <stack> --conf      # skip compose

Opt-out per stack: create stacks-mirror/<host>/<stack>/.no-sync (skip both sides) or stacks-mirror/<host>/<stack>/conf/.no-sync (skip conf only).

Always excluded in both directions (secrets / runtime state): .env, .env.*, acme.json, client_secrets.json, *.pem, *.key, *.crt, *.pfx, *.sqlite, *.sqlite3, *.db, *.log, *.log.*, *.pid, hub/, logs/.

Requires rsync installed on this workstation and every host you sync against (apt install rsync).

Layout

eshpfi-management/
├── CLAUDE.md              # this file
├── README.md              # human-facing overview
├── scripts/
│   └── server_inspect.sh  # gather server state for compose planning
├── servers/
│   └── <name>/
│       ├── README.md
│       └── system-details.txt   # latest server_inspect output
├── stacks/
│   └── <stack>/
│       ├── compose.yaml         # deployed to /opt/docker/compose/<stack>/
│       ├── .env.example         # template; real .env lives on server
│       └── README.md            # what this stack does, how to deploy
└── docs/
    └── pfi/                     # general PFI infrastructure reference

Working rules

  • Copies, not symlinks. Files here reflect what's on the server at the time of the last sync. When you edit here, the server doesn't change until you deploy.
  • Never commit secrets. Use .env.example templates; real .env files (with tokens, passwords) live on the server and are gitignored if/when this becomes a git repo.
  • Surgical edits. When fixing one stack, don't touch unrelated ones. Follow AIPA-MCP's CLAUDE.md rules about scope discipline.
  • Sanity-check before deploying. Run docker compose config (dry parse) before docker compose up -d on the server.
Reference in New Issue View Git Blame Copy Permalink