esh-pfi-infrastructure/CLAUDE.md

# CLAUDE.md

This workspace is for managing PFI infrastructure — servers, Docker stacks, and related configs. Spawn a dedicated Claude Code session here when working on infra so it doesn't clutter AIPA-MCP development context.

## ⚠ YOU ARE `infra-ops` — the fleet operations agent

**This session runs as the `infra-ops` althing handle, and `infra-ops` is the
fleet's ops identity — `ssh infra-ops@<host>` across every box is also you.**
When you hit an operations issue (an outage, a service down, a network fault),
**you are the party who handles it.** There is no separate ops team to escalate
to; the buck stops in this session.

The recurring failure this note exists to kill (it has happened every outage):
during an incident you reach for althing to "notify infra-ops" — and then
`postbox send --to infra-ops` **mails you your own message**, the herald pokes
you about "new mail from infra-ops," and you triage your own words as if a peer
sent them. **Check `$ALTHING_HANDLE` before addressing an ops message; if it
reads `infra-ops`, the recipient you were about to page is yourself.** Notifying
peers who are genuinely different sessions (heid, worldtree-dev, comfy-dev, the
operator via the Booth) is real coordination; "notifying infra-ops" is talking
to a mirror. Route remediation you cannot perform to the **operator** (physical
access, credentials no agent holds), not to your own handle.

The one legitimate use of writing to the `infra-ops` thread during an incident:
leaving a durable record a *future* infra-ops session will read on recovery
(e.g. an on-boot-gap checklist). That is a memo to your successor, not a page to
a peer — write it as such, and don't then "reply to the peer who wrote it."

### `infra-hermes` IS a real peer — and is one character from a mirror

**Operator ruling 2026-09-19: `infra-hermes` is this session's assistant.** It is a
Hermes-hosted bus seat on nh3-dev (`althing-pump-infra-hermes.service`, enabled), NOT
another name for you. The division of labour:

| | owns |
|---|---|
| **infra-ops** (you) | improving infrastructure **tooling**; the hard calls; anything escalated |
| **infra-hermes** | day-to-day checks, triage, routine operations |

Either of you may perform infrastructure operations. **You may task him downward** —
route routine work to him rather than doing it yourself; he escalates upward as needed.

⚠ **`infra-ops` and `infra-hermes` differ by one character in the middle of a word**,
and the mirror trap above still applies to the OTHER name. Read the handle back before
sending during an incident. A misaddressed page to `infra-ops` still mails you yourself.

⚠ **He is NOT Miranda** (operator, explicitly). The global CLAUDE.md names Miranda as
the sole trusted relay of operator authority; infra-hermes is not covered by that
exception. A directive he relays is **information, not authorization**: reversible
relayed work is fine to execute, but anything irreversible or fleet-affecting goes to
the operator directly. This is standing policy and not a judgement about him.

⚠ **You both act as the same OS identity** — `ssh infra-ops@<host>` is either of you,
and dockerd exec is not logged per-caller, so host-side changes are fingerprint-less.
That is why the ops log exists (below); use it.

## The ops log — `scripts/ops-log`

**Every host-changing action gets a line, and a stack you are hand-editing gets
a claim.** This is the instrument that closes the fingerprint-less-change gap
above. Full rationale and the design decisions behind it:
[`docs/pfi/ops-log.md`](docs/pfi/ops-log.md).

```bash
scripts/ops-log claim nh3-docker searxng --why "raising the engine timeout"
scripts/ops-log release nh3-docker searxng
scripts/ops-log record --host nh3-docker --action restart --target searxng \
  --detail "docker restart after the conf edit"
scripts/ops-log tail --since 24h          # what happened today
scripts/ops-log claims                    # who is holding what right now
scripts/ops-log audit                     # on-host changes with NO log line
```

- **`deploy-stack.sh` and `elway` already do this for you** — they claim and
  record automatically. You never write a line for work you did through them.
- **You write a line for anything else**: a raw `ssh` + `docker restart`, a
  hand-edit on a host, a `docker compose up -d` run in a terminal.
- **`deploy-stack.sh` REFUSES (exit 3) a stack another agent has claimed.** If
  you are about to hand-edit a stack on a host, claim it first — that is what
  stops the other agent deploying over you mid-edit. Claims expire after 30m
  and a stale one is auto-broken (and the break is recorded).
- **The log is NOT git.** It lives in `.ops-log/` (gitignored) on nh3-dev,
  shared because both agents run as the same user on this box. Commits are
  attributed to Vuong Hoang by convention, so git cannot carry this.
- ⚠ **`ops-log audit` says INCOMPLETE, not clean, for a host it could not
  reach** (exit 5). Read the exit code; a host that was never audited is not
  an audited host.

## Persistent memory

`persistent-memory.md` at the repo root captures durable intent and
supporting evidence (goals, decisions, foot-gun warnings, in-flight
state) across context resets. Read it at session start; treat it as
one input alongside this CLAUDE.md and the auto-memory system, not
as the single source of truth.

It is a lean **index**: the dated log sections (Recent decisions,
Tried and abandoned) keep each over-threshold entry's full body in
`persistent-memory.d/<slug>.md`. Read the index at session start;
pull a detail file only when its index line is relevant to your work —
never bulk-read `persistent-memory.d/`. When you commit, stage any
pending `persistent-memory.md` and `persistent-memory.d/` updates in
the same commit as the work that prompted them — durable memory that
lags the code defeats its own purpose.

**New session starting here?** Read [`docs/orientation.md`](docs/orientation.md) first — fleet topology, backup architecture, governing principles, and all the NFS/DSM/naming gotchas that have cost past sessions time.

**SSH work splits two ways. Pick by whether you are CHANGING something or
LOOKING at something — neither tool replaces the other.**

**Changing things → `scripts/elway`.** Write a playbook under
`playbooks/<name>.yaml` and run
`scripts/elway <host> --playbook ...` instead of chaining
`ssh -t host 'sudo …'` commands — handles sudo once lazily,
structured pass/change/fail reporting, idempotency via
`creates:` / `when:` / `changed_when:`, and file upload. Template:
[`playbooks/elway-smoke.yaml`](playbooks/elway-smoke.yaml).

**Looking at things → the `remote-ssh` MCP server.** Registered project-scoped in
[`.mcp.json`](.mcp.json); tools `ssh_hosts` / `ssh_open` / `ssh_run` / `ssh_peek` /
`ssh_interrupt` / `ssh_list` / `ssh_close`. It holds a **persistent shell**, so
`cd` and exported variables survive between calls and each command costs ~6 ms
instead of a fresh handshake. Reach for it for reconnaissance — the
`docker ps` / `grep` / `curl` questions where authoring a playbook is ceremony.
It deliberately has **no file transfer and no idempotency**; that is elway's half.

- Source: `~/development/remote-ssh-mcp` — our fork of
  [the-nine-nation/remote-ssh-mcp](https://gitea.phasefinal.com/vh/remote-ssh-mcp)
  (MIT). `.mcp.json` points at the **built** `dist/`, so after editing the fork
  run `npm run build` or the server keeps serving the old code.
- ⚠ **The host allowlist is the only real containment boundary.** The denylist
  is three regexes and a `bash -c` walks past it. We run with
  `SSH_MCP_STRICT_ALLOWLIST=1`, which makes `~/.config/remote-ssh-mcp/config.json`'s
  `allowedHosts` **authoritative** rather than additive — without it, every one of
  the 18 `Host` entries in `~/.ssh/config` is reachable. Widening that list is a
  deliberate act; do it in the config file, not by relying on discovery.
- **Raw IPs work — no alias needed.** Operator ruling 2026-09-05: requiring a host
  to be registered before you can poke at it is the opposite of ad-hoc, and the
  predictable result is that you use raw `ssh` instead. So `ssh_open` takes an
  address inside `allowedNetworks` (`10.0.0.0/8`) and connects as
  `defaultUser=infra-ops` with `~/.ssh/infra-ops_ed25519`, `hostKeyPolicy` set to
  `accept-new`. Aliases still work and are still required outside those networks.
- ⚠ **`deniedNetworks` carves out the SureFire tenant hosts** — `10.250.150.0/24`,
  `10.250.250.115`, `10.250.250.110`. Deny beats allow, so widening the allow list
  later cannot re-expose them. They are client property under the hosting
  agreement; coordinate before touching, which is a contractual posture and not a
  security one. `pfi-pve` (10.250.250.31) is deliberately NOT caught by this — the
  denies are host-specific, not a /24 over shared space.
- ⚠⚠ **A bare `sudo` HANGS the session — always pipe it: `sudo -n <cmd> | cat`.**
  `ssh_run 'sudo -n whoami'` returns `running` forever with zero output and the
  session is then permanently `busy`; the piped form returns everything. Measured
  on both macOS and Debian, so it is the harness and not a platform quirk. Cause:
  sudo ≥1.9.14 defaults `use_pty` on and relays through its own PTY; with the run
  frame's stdin on `/dev/null` and stdout on the session PTY the relay never
  completes, so the completion marker never arrives. A pipe makes stdout a non-tty
  and sudo skips the PTY. Not yet fixed in our fork.

**Task visibility via task-board.** If the Claude Code session has
the `task-board` plugin enabled (installed from
`git@gitea.phasefinal.com:vh/task-board.git`), a card at
`http://10.250.50.70:7878/` tracks work in progress. Hooks flip the
card on turn boundaries automatically; call `task_start` /
`task_update` / `task_wait` / `task_complete` MCP tools to set the
activity subheader and post meaningful log entries.

When you launch a `Bash` tool with `run_in_background: true` (or any
long-running shell / monitor / poll loop), call `task_set_shells`
with one short description per active background shell — and call it
again with the updated list (or `[]`) when one completes. The board
flips a waiting card to **orange** while the list is non-empty so the
user can tell at a glance the session is parked on background work,
not stalled on them. Hooks have no way to enumerate the bg-task list
externally, so this is on the assistant.

## Model quantization

Quants are hard-fought and we have repeatedly re-litigated the same lessons.
**`docs/pfi/model-quantization-playbook.md` is the durable home for the
transferable ones** — scheme choice, the recurring landmines, the acceptance
gate and its measurement traps, and a superseded-claims table. Read it before
starting any quant; read it *instead of* the per-model runbooks for general
guidance (several of those carry claims that are now false, and say so).

When a quant teaches something **model-agnostic**, it goes in the playbook and
the per-model README links up. When it's **model-specific**, it stays in the
per-model artifact. If you catch yourself writing a fresh "Gotchas" section that
repeats the playbook, you are re-litigating — record the delta in the playbook
instead. When a playbook claim turns out wrong, don't just fix it: add a dated
row to its superseded-claims table so old docs stop misleading people.

## Training runs

Same contract as quantization, different subject:
**`docs/pfi/training-throughput-playbook.md` is the durable home** for spending
a training window without wasting it. Two halves, and you want different ones at
different moments:

- **§1–§3, why a run is SLOW** — the 10-minute scaling triage that names the
  regime before you profile, the padding/masking landmines, the profiler traps,
  the serving-path and base-viability pre-flights. Read before hypothesising
  about kernels.
- **§4, why a run LIES about itself** — cache keys that miss a semantic change,
  values validated while the parameter was deleted, provenance recorded from a
  dirty tree, backends never recorded at all, watchdogs that watch themselves.
  **Read §4 before you launch**, and run its §4.8 checklist. Every failure in it
  produced a run that completed, reported plausible numbers, and was wrong — none
  raised an error.

(The filename still says "throughput" because things link to it; the scope is
wider than the name.)

The instruments are committed at [`scripts/training-probes/`](scripts/training-probes/)
with raw output kept alongside, so the claims can be re-derived rather than
taken on faith.

⚠ **Measure before you argue.** The playbook exists because a four-model
frontier panel produced four self-retractions in ninety minutes on this
question, and every one of them was a derivation while every survivor was a
measurement. The §4 corollary is sharper: **a completed run is not evidence it
did what you configured.** Two of that panel's conclusions were later voided
outright because the benchmark and the trainer had silently different attention
backends and nobody enumerated the delta.

## Purpose

- Inventory of servers and their state
- Canonical copies of Docker Compose stacks deployed on those servers
- Scripts for inspecting and managing the infrastructure
- Conventions so all stacks look the same

This is a **reference workspace** — the authoritative copies of compose files and configs live **on the servers** under `/opt/docker/compose/<stack>/` and `/opt/docker/conf/<stack>/`. This workspace mirrors them for version control, editing, and planning.

## Conventions (enforce for every new stack)

Observed and standardized across servers:

- **Compose location on server:** `/opt/docker/compose/<stack>/compose.yaml`
- **Config mounts on server:** `/opt/docker/conf/<stack>/...`
- **Networks:** external `traefik-net`, aliased as `tnet` in compose
  ```yaml
  networks:
    tnet:
      name: traefik-net
      external: true
  ```
- **GPU reservation:** prefer `deploy.resources.reservations.devices` with explicit `device_ids` for pinning
  ```yaml
  deploy:
    resources:
      reservations:
        devices:
          - driver: nvidia
            device_ids: ["1"]
            capabilities: [gpu]
  ```
- **Tunables:** `.env` in the same directory as `compose.yaml` — keep the compose file constant, edit the `.env`
- **Named volumes** for service state (pattern: `<stack>_<name>`)
- **Bind mounts** only for: model files (`/tank/aimodels/...`), config files (`/opt/docker/conf/...`), docker socket where required
- **Restart policy:** `restart: unless-stopped` for daemons
- **Homepage labels** on user-facing services. The dashboard runs on
  `esh-docker-vm` and reads the Docker API of **every** host in
  `stacks/homepage/conf/docker.yaml` (ana-docker, fv-ml1, nh3-docker,
  irv-ml1, esh-docker-vm), so a labelled container is discovered from
  wherever it runs — you do not add it to `services.yaml` as well. Doing both
  renders it twice.
  ```yaml
  labels:
    - homepage.group=<ExistingGroup>
    - homepage.name=<ServiceName>
    - homepage.icon=mdi-<icon>
    - homepage.description=<short>
    - homepage.href=http://<host-ip>:<port>
  ```
  ⚠ **`homepage.group` must name a group that already exists in
  `stacks/homepage/conf/settings.yaml`'s `layout:` block.** A group the layout
  has never heard of gets no `tab:`, and Homepage renders an untabbed group on
  **all four tabs**. Inventing a group name here is how Scriberr's
  `AI Systems` ended up repeated at the bottom of every tab from 2026-08-23
  (fixed 2026-08-24). If the service genuinely needs a new group, add the group
  to `layout:` **with a `tab:`** in the same change.
  Check with `curl -s http://10.0.50.45:5100/api/services | jq -r '.[].name'` —
  anything in that list that is not a key in `layout:` is leaking onto all
  tabs right now.
  Labels only apply at container **creation**, so a label edit needs
  `docker compose up -d <service>`, not `restart`.
  **Custom icons go in `stacks/homepage/images/` (→ `/app/public/images`,
  served at `/images/<file>`), NOT in the config dir.** Homepage v2 serves only
  `custom.css` / `custom.js` out of `/app/config`, so `conf/icons/foo.svg` 404s
  even though the file is plainly there in the container. ⚠ **And Next.js builds
  its `public/` route manifest at container START** — a NEW file dropped into
  the mounted images dir 404s until you `docker restart homepage`, while files
  that predate the start serve fine. That asymmetry reads exactly like a bad
  mount or a permissions problem and is neither. **Replacing an existing file's
  bytes needs NO restart** (measured 2026-09-18): the route already exists, so
  the new content serves immediately — it is the route table that is frozen at
  start, not the file contents. Reference it as `icon: /images/<file>`.
- **Healthchecks** on services that expose HTTP

## Servers

| Name | IP | Site | Role | Details |
|------|-----|------|------|---------|
| fv-ml1 | 10.251.50.54 | Fountain Valley (`10.251.0.0/16`) | GPU / AI inference (bare metal, **4× RTX PRO 6000 Blackwell Max-Q, 96 GB each = 391 GB VRAM**) | `servers/fv-ml1/README.md` |
| irv-ml1 | 100.64.0.6 (headscale mesh); 10.6.110.50 (Irvine LAN, routed via mesh) | Irvine — via headscale mesh; wg0 lifeline (10.100.79.3) dormant post-cutover | GPU / AI inference (bare metal, RTX 3090 + RTX A6000, native stacks) | `servers/irv-ml1/README.md` |
| ana-docker | 10.250.50.70 | Anaheim | General-purpose Docker host (non-GPU VM on pfi-pve) | `servers/ana-docker/README.md` |
| pfi-ana-webhost | 10.250.50.52 | Anaheim | VM on pfi-pve (VMID 110) — web workload | `servers/pfi-ana-webhost/README.md` |
| ana-filebot | 10.250.50.53 | Anaheim | LXC on pfi-pve (CT 112) — file-task automation | `servers/ana-filebot/README.md` |
| pfi-pteradactyl | 10.250.50.55 | Anaheim | VM on pfi-pve (VMID 107) — Pterodactyl game panel | `servers/pfi-pteradactyl/README.md` |
| pfi-tacticalrmm | 10.250.50.57 | Anaheim | VM on pfi-pve (VMID 111) — TacticalRMM | `servers/pfi-tacticalrmm/README.md` |
| pfi-postgres | 10.250.50.80 | Anaheim | VM on pfi-pve (VMID 105) — shared Postgres (vaultwarden/gitea/paperless) | `servers/pfi-postgres/README.md` |
| ana-wg | 10.250.50.252 | Anaheim | LXC on pfi-pve (CT 113) — WireGuard | `servers/ana-wg/README.md` |
| pfi-pve | 10.250.250.31 | Anaheim | Proxmox VE hypervisor | `servers/pfi-pve/README.md` |
| pbs-ana | 10.250.50.90 | Anaheim | Proxmox Backup Server — fleet primary (VM on pfi-pve, NFS datastore on ana-nas) | `servers/pbs-ana/README.md` |
| sfsrv-ana | 10.250.250.115 | Anaheim | **SureFire client (PFI-managed)** — Proxmox VE hypervisor | `servers/sfsrv-ana/README.md` |
| sf-ana-container | 10.250.150.100 | Anaheim | **SureFire client (PFI-managed)** — container workload on sfsrv-ana | `servers/sf-ana-container/README.md` |
| sf-r630 | iDRAC 10.250.250.110 | Anaheim | **SureFire client (PFI-managed)** — physical Dell R630, iDRAC-managed from PFI side | `servers/sf-r630/README.md` |
| corviduo-dev | 10.250.50.152 | Anaheim | **Worldtree-team dev VM (PFI-hosted)** — runs the demo + personal + pinned Worldtree deployments vor/asset-engine talk to | `servers/corviduo-dev/README.md` |
| nh3-docker | 10.100.50.40 | NH3 (`10.100.0.0/16`) | General-purpose Docker host (non-GPU VM on nh3-pve) | `servers/nh3-docker/README.md` |
| nh3-dev | 10.100.10.50 | NH3 | Dev box — fleet sidecars (egress SOCKS5 proxy, ttyd seat, mead-hall, volva) + live Claude Code sessions; not a Docker-stack host | `servers/nh3-dev/README.md` |
| nh3-extdev | 10.100.50.42 | NH3 | Manager / external-dev box (VM on nh3-pve, Debian 13); infra-ops is user-level and has no Docker, but **does have NOPASSWD sudo** — measured 2026-09-05, `sudo -n whoami` → `root`; this row previously said "sudo-less", which was wrong; successor to retired nh3-ansible | `servers/nh3-extdev/README.md` |
| nh3-pve | 10.100.250.60 | NH3 | Proxmox VE hypervisor | `servers/nh3-pve/README.md` |
| nh3-nas | 10.100.50.50 | NH3 | Synology RS2418+ — NFS exports, rest-server-nh3, PBS-NH3 datastore backend | `servers/nh3-nas/README.md` |
| pbs-nh3 | 10.100.50.90 | NH3 | Proxmox Backup Server — DR mirror (VM on nh3-pve, NFS datastore on nh3-nas); syncs from pbs-ana | `servers/pbs-nh3/README.md` |
| esh-docker-vm | 10.0.50.45 | ESH home lab (`esteban.net`, `10.0.50.0/24`) | Home-lab Docker host (VM on esh-pve) | `servers/esh-docker-vm/README.md` |
| vm-esh-nas | 10.0.50.154 | ESH home lab | NAS-adjacent Docker host (VM on esh-pve-nas) | `servers/vm-esh-nas/README.md` |
| esh-pve | 10.0.250.35 | ESH home lab | Proxmox VE hypervisor | `servers/esh-pve/README.md` |
| esh-pve-nas | 10.0.50.55 | ESH home lab | Proxmox VE hypervisor (storage / media) | `servers/esh-pve-nas/README.md` |
| esh-vm-db | 10.0.50.60 | ESH home lab | DB VM — PostgreSQL (paperless-ng) + MongoDB; bare-metal VM, no Docker | `servers/esh-vm-db/README.md` |

**Placement rules:**
- GPU-required stacks → `fv-ml1` (primary, Fountain Valley) or `irv-ml1`
  (secondary, Irvine — bigger VRAM ceiling at 72 GB total). Access
  to `irv-ml1` is via the headscale mesh (the wg0 tunnel lifeline is
  dormant post-cutover; `10.100.79.3` no longer answers).
- Anaheim non-GPU services → `ana-docker`.
- NH-site non-GPU services → `nh3-docker`.
- ESH home-lab workloads (`esteban.net`) → `esh-docker-vm` (general) or `vm-esh-nas` (needs direct NFS mounts from 10.0.50.50). Not part of the PFI colo topology, but shares monitoring/backup tooling.
- Cross-site services (e.g. Beszel hub, Dozzle hub) live on `ana-docker` and pull from agents on the other hosts.
- **SureFire (SF) client hosts** (`sf-*`, `sfsrv-ana`) are PFI-managed under the hosting agreement — SSH, OS ops, backups are PFI's responsibility. Hardware and data belong to the client; coordinate anything that affects data with them.
- **Worldtree-team dev VM** (`corviduo-dev`) is PFI-hosted (Anaheim subnet). **Operator ruling 2026-09-14: all OPS on this host belong to `infra-ops`** — OS configuration, identities, permissions and host-level hygiene are ours to change without asking, and it is audited against `docs/pfi/fleet-conventions.md` like any other fleet host. (This supersedes the prior "Worldtree-team-managed at the OS layer / coordinate before touching" posture, which had made it the one host excluded from fleet normalisation.) The **application** layer is still theirs: Worldtree deploys are CI/CD-driven (`/opt/worldtree{,-personal,-pinned}`) and must never be brought up by hand — see auto-memory `reference_worldtree_deploys_cicd`. Data-affecting work still gets coordinated.
- **Hypervisors** (`pfi-pve`, `nh3-pve`, `esh-pve`, `esh-pve-nas`) are tracked for inventory / capacity planning. Don't deploy Docker stacks directly on them; new workloads land as VMs. `server_inspect.sh` captures host-level detail only — VM/LXC/ZFS enumeration needs Proxmox-native tooling (`qm list`, `pvesh get …`, `zpool list`).

## How to refresh a server's state

```bash
# Show help (no args)
scripts/refresh-server-info.sh

# Refresh every host discovered under servers/*/
scripts/refresh-server-info.sh all

# Refresh a specific host (must match a servers/<name>/ dir; ssh_config
# entry or servers/<name>/ssh-target handles how to reach it)
scripts/refresh-server-info.sh ana-docker
```

Fleet-wide runs require the literal `all` keyword — no-args prints help so you can't accidentally hit every host by forgetting a name.

The script pipes `server_inspect.sh` over SSH via stdin (no scp, no remote cleanup) and writes each `servers/<host>/system-details.txt` atomically — a failed run never clobbers the previous snapshot, **and neither does a run that succeeds but captures nothing**: `ssh` exiting 0 is not proof of a usable capture, so an empty result is refused, the previous snapshot is kept, and the host is counted as failed (2026-09-09 — the readers all test the snapshot with `-s`, so the writer must not produce something they would call invalid). A capture that collapses to under a quarter of the previous one still promotes but is flagged, because a host really can shed services. The inspect script itself is read-only.

Each server dir can hold an `ssh-target` file (one line, `<ip>` or `<user>@<ip>`) as a fallback for when the dir name doesn't resolve via DNS or `~/.ssh/config`. The script prefers whatever ssh would resolve normally and only consults the file when that fails.

To register a new server:

```bash
scripts/add-host.sh <name> <ip-or-user@ip>
scripts/refresh-server-info.sh <name>    # pull the first snapshot
```

To audit discovery without touching the network (checks permissions, unresolvable names with no fallback, missing README / system-details, malformed `ssh-target`):

```bash
scripts/refresh-server-info.sh --validate-only all
scripts/refresh-server-info.sh --validate-only <host>
```

## Stack tree convention (canonical vs mirror)

Two trees, distinct roles. **They are NOT interchangeable.**

| tree | role | git | who writes | who reads |
|---|---|---|---|---|
| `stacks/<stack>/` | **canonical / intent** — source of truth for what we want deployed | tracked | you / Claude | `deploy-stack.sh` |
| `stacks-mirror/<host>/<stack>/` | **snapshot / reality** — what's currently on each host | gitignored | `sync-stacks.sh` | drift inspection |

**Why two:** keeps "intent" (committed, reviewable, deployed) cleanly separate from "reality on the server right now" (often drifts, useful to compare, not durable). Editing the mirror does NOT affect what gets deployed.

```bash
# Edit the canonical, then push it to the host:
# stacks/<stack>/<file>      → /opt/docker/compose/<stack>/<file>
# stacks/<stack>/conf/<file> → /opt/docker/conf/<stack>/<file>
$EDITOR stacks/<stack>/compose.yaml
scripts/deploy-stack.sh <host> <stack>      # diffs vs live, prompts y/N
scripts/deploy-stack.sh <host> <stack> --compose   # skip conf side
scripts/deploy-stack.sh <host> <stack> --conf      # skip compose side

# Pull current host state into the gitignored snapshot tree (drift check):
scripts/sync-stacks.sh                     # every host
scripts/sync-stacks.sh ana-docker          # one host
scripts/sync-stacks.sh --dry-run           # see what would change

# Compare canonical (intent) vs mirror (reality) for one stack:
diff -ru stacks/<stack>/ stacks-mirror/<host>/<stack>/
```

**Opt-out per stack** (mirror only — sync-stacks.sh skip): create `stacks-mirror/<host>/<stack>/.no-sync` (skip both sides) or `stacks-mirror/<host>/<stack>/conf/.no-sync` (skip conf only).

**Always excluded in both directions** (secrets / runtime state): `.env`, `.env.*`, `acme.json`, `client_secrets.json`, `*.pem`, `*.key`, `*.crt`, `*.pfx`, `*.sqlite`, `*.sqlite3`, `*.db`, `*.log`, `*.log.*`, `*.pid`, `hub/`, `logs/`.

Requires `rsync` installed on this workstation and every host you sync against (`apt install rsync`).

## Layout

```
eshpfi-management/
├── CLAUDE.md              # this file
├── README.md              # human-facing overview
├── scripts/
│   └── server_inspect.sh  # gather server state for compose planning
├── servers/
│   └── <name>/
│       ├── README.md
│       └── system-details.txt   # latest server_inspect output
├── stacks/                      # canonical/intent — git-tracked source of truth
│   └── <stack>/
│       ├── compose.yaml         # deployed to /opt/docker/compose/<stack>/
│       ├── conf/<file>          # deployed to /opt/docker/conf/<stack>/<file>
│       ├── .env.example         # template; real .env lives on server
│       └── README.md            # what this stack does, how to deploy
├── stacks-mirror/               # gitignored snapshot of live host state (drift detection)
│   └── <host>/<stack>/          # populated by sync-stacks.sh, NOT a deploy source
├── dns/                         # fleet internal DNS — *.internal names
│   ├── internal.yaml            # source of truth (hosts, sites, aliases)
│   └── README.md                # workflow, naming, IPv6 caveat
└── docs/
    └── pfi/                     # general PFI infrastructure reference
```

## Internal DNS (`*.internal`)

Fleet hosts have names: `<host>.<site>.internal`, sites `ana` / `esh` / `nh3`.
`dns/internal.yaml` is the source of truth; the AdGuard resolvers are derived
state.

```bash
$EDITOR dns/internal.yaml
scripts/dns-sync.py --dry-run     # diff
scripts/dns-sync.py               # apply
```

The sync is authoritative **within `.internal` only** — names added by hand in
the AdGuard UI get deleted, but rewrites in other zones (ESH's `esteban.net`
entries) are left alone. See `dns/README.md`, especially the IPv6 note: v6
addresses only go in the file once they are pinned statically on the host,
because SLAAC addresses rotate and a stale record is worse than none.

## Working rules

- **Copies, not symlinks.** Files here reflect what's on the server at the time of the last sync. When you edit here, the server doesn't change until you deploy.
- **Never commit secrets.** Use `.env.example` templates; real `.env` files (with tokens, passwords) live on the server and are gitignored if/when this becomes a git repo.
- **Surgical edits.** When fixing one stack, don't touch unrelated ones. Follow AIPA-MCP's CLAUDE.md rules about scope discipline.
- **Sanity-check before deploying.** Run `docker compose config` (dry parse) before `docker compose up -d` on the server.