Files
esh-pfi-infrastructure/docs/fleettools/ssh-fleet.md
T
vh 53c3e8000e docs: add FleetTools — an agent-family-agnostic index of fleet capability
Every agent on nh3-dev — Claude, Codex, Grok, Aider — needs the same
answers: what runs here, how do I call it, what will bite me. Until now
that lived in ~/.claude/CLAUDE.md, which only Claude sessions load, and
it was interleaved with operator preferences that other families have
no use for.

Two-tier by design, matching the persistent-memory split: FLEETTOOLS.md
is a 135-line index an agent reads whole, and each entry links to a
detail file it opens only when it actually needs that tool. Reading the
index costs about a fifth of reading the tree.

Detail paths are absolute so they resolve from any working directory,
since a non-Claude agent will cat the path rather than follow a
markdown link. ~/FLEETTOOLS.md symlinks to the index for discovery.

Rule zero is that live inventories get queried, not transcribed:
Homepage /api/services, asset-engine /api/v1/services, LiteLLM
/v1/models, and every FastAPI seat's /openapi.json. A copied service
table would be stale within a month and this repo already has a
standing rule against second copies that drift.

Contents verified against the running fleet rather than copied from
existing docs: binaries resolved on PATH, seven endpoints probed live,
the LiteLLM roster counted at 40 models where the old note said ~30.
No credentials are included; the vault and its CLI are pointed at
instead.
2026-09-18 07:35:28 -07:00

2.2 KiB

Fleet SSH — the infra-ops identity and host map

Identity

ssh infra-ops@<host> — a dedicated NOPASSWD-sudo user on PFI-owned boxes, key ~/.ssh/infra-ops_ed25519. Covers every fleet host including all four Proxmox hypervisors, corviduo-dev and esh-docker-vm.

Excludes the SureFire tenant hosts only (sf-*, sfsrv-ana).

⚠ Test with sudo -n -v first rather than assuming. Not every box has been bootstrapped — pbs-nh3 (10.100.50.90), for instance, has no infra-ops user and is reachable only as lkraven@.

⚠ Remote sudo needs ssh -t. Read-only probes can use plain ssh.

Sites and hosts

site subnet gateway
Anaheim (colo) 10.250.0.0/16 FortiGate 10.250.0.1
NH3 10.100.0.0/16 UDM-SE 10.100.0.1
ESH home lab 10.0.50.0/24 (esteban.net) UDM-PM 10.0.0.1
Fountain Valley 10.251.0.0/16
Irvine via headscale mesh

Full host table with roles: ~/development/eshpfi-management/CLAUDE.md § Servers. Live per-host snapshots: servers/<name>/system-details.txt. Internal DNS names are <host>.<site>.internal, source of truth dns/internal.yaml.

Mesh

Headscale is primary since 2026-09-06 (IPsec tunnels dormant). Subnet routers are nh3-scale / ana-scale / esh-scale at 100.64.0.1 / .3 / .2.

⚠ Disable a site-to-site tunnel BEFORE adding a mesh route for the same prefix or they fight. ⚠ accept-routes needs a return path FIRST. On a dev box it can black-hole its own LAN (route table 52 beats main). Gateway routes or SNAT before, never on your own box.

Traps

  • SureFire tenant hosts are client property under the hosting agreement. Coordinate before touching anything that affects data. This is a contractual posture, not a security one.
  • Confirm a reboot by observing DOWN, then return. Poll for disappearance and cross-check the boot timestamp — "never rebooted" and "rebooted very fast" look identical otherwise.
  • chroot rbind needs --make-rslave. Without it, umount -R unmounts the LIVE host's cgroup/devpts, which looks exactly like disk failure.
  • pct snapshot exits 0 when it refuses (bind-mounted guest). Use zfs snapshot and read it back.