esh-pfi-infrastructure/servers/nh3-dev/README.md

# nh3-dev

NH3-site **developer box** — `10.100.10.50` (WireGuard-reachable from the NH3
subnet). General-purpose dev VM that hosts agent-fleet sidecars and live Claude
Code sessions; **not** a Docker-stack host in the `stacks/` sense.

**Reach:** `ssh 10.100.10.50` (as `lkraven`), or the dedicated agent identity
`ssh -i ~/.ssh/infra-ops_ed25519 infra-ops@10.100.10.50` (NOPASSWD sudo).
`infra-ops` bootstrapped here 2026-06-04 (see [`reference_infra_ops_sudo_identity`]
in auto-memory). Note: Claude Code sessions often run **natively on this box**, so
local Bash already executes here — no SSH-to-self needed for non-privileged work.

## What runs here

- **NH3 egress proxy — RETIRED 2026-09-06** (replaced by headscale exit nodes; `danted` disabled, config `.retired`). Was: durable internal-only SOCKS5 `socks5h://10.100.10.50:1080`
  (dante, ACL'd to the WG net). Residential egress for colo services gated on their
  datacenter IP (e.g. YouTube bot-gate). Runbook + setup committed; consumers point
  `*_PROXY` at it.
- **Docker runtime — `docker-ce` since 2026-09-14.** Was Debian's `docker.io`
  20.10.24 + the Python `docker-compose` 1.29.2 v1 CLI + `containerd` 1.6.20,
  with **no `cli-plugins` directory at all** — so `docker compose` (space) was
  not a command: it printed a help blurb and **exited 0**, which a deploy script
  cannot distinguish from success. Migrated via
  `playbooks/upgrade-docker-ce.yaml` to docker-ce 29.8.0 / compose plugin v5.5.1
  / containerd.io 2.3.5 / buildx v0.37.1. **The v1 `docker-compose` (hyphen)
  binary is gone and no shim was installed** (operator ruling 2026-09-14) — fix
  callers, don't paper over them.
  - ⚠ `vastblue-u5-pg` (an empty `postgres:16` probe container, `restart: no`,
    plain `docker run`, anonymous volume) **and its volume were removed** when
    the old daemon stopped. The playbook is not the cause — it has no `rm`,
    `prune` or `purge`, and the other five containers survived, two of them
    long-exited. Almost certainly `--rm` / `AutoRemove=true`, unprovable after
    the fact because the container record is gone. Measured beforehand as **zero
    user tables in every database**, so no data was lost. Lesson: capture
    `AutoRemove` and `RestartPolicy` together when snapshotting a container you
    are about to bounce.
  - The playbook's restart loop runs as the deploy identity, not root, and a
    stack `.env` may legitimately be root-owned `0600`
    (`/opt/docker/compose/beszel/.env`) — compose bails before doing anything,
    so the stack reported FAILED while `restart: unless-stopped` had already
    brought it back healthy. **Fixed 2026-09-14**: the loop retries under
    `sudo -n` before calling it a failure. Verified against beszel — plain
    `rc=1 permission denied`, sudo retry `rc=0 Container beszel-agent Started`.
    A false FAILED in automation output is worse than a quiet one; it trains
    readers to skim the failure lines.
- **`/opt/docker` ownership — normalised fleet-wide 2026-09-14 to
  `root:docker 2775`** (setgid) via `playbooks/normalize-docker-tree.yaml`,
  operator ruling. Was a three-way split: `root:root 755` here, `root:root 777`
  on nh3-docker + ana-docker (world-writable, from a 2024 `chmod -R 777` to get
  a git clone working), `lkraven 755` elsewhere. Not a personal username
  (`lkraven` is one of three the operator uses) and not a new admin account —
  the `docker` group already existed on every host holding exactly `lkraven` +
  `infra-ops`. ⚠ This is **not** privilege separation: `docker` membership is
  root-equivalent. A future non-root deployer needs a dedicated `deploy` group.
  Stack `.env` files went to `root:docker 0640` — previously 31 of 74 were
  `0600` readable by only ONE of the two deploy identities (varying by file,
  which is what false-FAILED beszel during the docker-ce upgrade) and 43 were
  world-readable `0644`. **No containers were bounced**; these are inode
  metadata changes and `.env` is read only at `compose up`. `/opt/docker/compose/talk` was created
  `lkraven`-owned 2026-09-14 and **tts-dev migrated `talk` into it the same
  day** — it had been at `~/talk`, a convention violation that hid it from
  anything walking `/opt/docker/compose/*/`. Old path parked at
  `~/talk.migrated-20260914`; `version: "3.8"` dropped from its compose file
  (it existed only to satisfy the v1 CLI, which is now gone). Normalising the
  parent directory is **unresolved — operator's call**; `/opt/docker` itself is a separate three-way split (`755` root, `777`
  root on two hosts, `755` lkraven).
- **NOT a mesh node** (retired 2026-09-15). nh3-dev sits on the NH3 LAN and reaches
  every site through its own default gateway — `RouteAll: false` meant it never used
  the tunnel for routing anyway, so membership bought only a `100.64.0.4` address
  nothing referenced. ⭐ It also *cost* something: a host running Tailscale installs
  `-A ts-input -s 100.64.0.0/10 ! -i tailscale0 -j DROP`, and because the fleet's
  subnet routers preserve source rather than masquerading RFC1918, a mesh client's
  packet reached `ens18` still sourced `100.64.x` and was silently dropped — which is
  why `nh3-dev.nh3.internal` failed from the mesh while every non-Tailscale NH3 host
  worked. That needed a `-d 10.100.10.50/32 -j MASQUERADE` exception on nh3-scale;
  retiring the membership removed the anti-spoof rule and the exception with it.
  Verified after: reachable at `10.100.10.50` from ESH, Anaheim, FV, Irvine and NH3,
  and reaching all four sites plus the internet itself.
- **ttyd fleet driver-seat** — web/iPad seat into the zellij `Claude` session (ttyd
  behind Caddy; OSC52 clipboard shim). User systemd services under `~/.config`.
- **mead-hall** — Bifrost tool-provider sidecar (`:5173`), CI-deployed from
  `vh/mead-hall`.
- **Hermes Agent gateway** — OpenAI-compatible agent API on `127.0.0.1:8765`
  (`hermes-gateway.service`, user systemd, installed 2026-09-14 via `hermes
  gateway install`). Runs the per-user install at `~/.hermes/hermes-agent`
  (v0.21.1, `b88e677`); config in `~/.hermes/{.env,config.yaml}`. Stood up for
  **SVOS/Miranda**, which replaced Worldtree with Hermes on 2026-09-11 and
  cannot boot without it. Bearer auth is mandatory even on loopback — key
  vaulted as `nh3-dev/hermes/api-server-key`. ⚠ The gateway registers Hermes's
  **full** toolset by default — 28 toolsets, 14 enabled, `terminal` /
  `code_execution` / `file` / `browser` among them. SVOS's security model is that
  write-capable tools are never *registered*, not that they are refused at
  dispatch, so `platform_toolsets: {api_server: []}` is set in `config.yaml`
  (2026-09-14) and measured back as 28 rows / **0 enabled / 0 tools** on
  `/v1/toolsets`. The endpoint still reports all 28 rows with their flags, which
  is what SVOS's `_hermes_roster` derives its required-config line from —
  narrowing does not blind it. Becomes `[svos_miranda]` once SVOS's plugin lands
  in `$HERMES_HOME/plugins/`.
  ⚠ **A deliberate restart can exit 1, and that is a Hermes race, not a crash**
  (2026-09-23). `ExecStop=gateway.systemd_stop_mark` writes a planned-stop
  marker; the gateway's marker-watcher thread sometimes sees it before systemd's
  SIGTERM lands, runs the shutdown handler ("Received UNKNOWN as a planned
  gateway stop"), and consumes the marker. The SIGTERM then runs the handler a
  second time, finds no marker, and exits 1 ("signal-initiated shutdown without
  restart request"). `~/.hermes/logs/gateway.log` shows both lines 26 ms apart.
  Before the 09-21 Hermes update every stop exited 1; since then it is
  intermittent. The exit 1 used to page infra-ops through the `OnFailure` hook,
  so drop-in `hermes-gateway.service.d/20-planned-stop-exit1-is-clean.conf`
  sets `SuccessExitStatus=1`. That costs nothing: `Restart=always` restarts
  whatever the exit status, and `StartLimitIntervalSec=0` means the unit never
  reaches `failed` on a start failure, so the hook can't catch anything real on
  this unit anyway. Delete the drop-in once upstream makes the handler
  idempotent.
- **Hermes model backend → `gen-large` on the fleet LiteLLM gateway** (free local
  compute), set 2026-09-14 per operator ruling. Until then `model.default` said
  `anthropic/claude-opus-4.6` with `model.base_url` at openrouter, but
  `provider: auto` plus a lone `zai` credential in `auth.json` silently resolved
  Miranda to **GLM-5.3 on the paid z.ai Coding Plan** — three settings that had
  to be read together before the real answer fell out. Now
  `default: gen-large` / `provider: custom` / `base_url:
  http://10.250.50.70:4000/v1`, verified by a real turn (`hermes status` →
  `gen-large` / `Custom endpoint`, plus a 660-token completion through
  `/v1/chat/completions`). The openrouter/nous credit warnings cleared with it.
  ⚠ **`CUSTOM_API_KEY` / `HERMES_CUSTOM_API_KEY` are INERT for bare
  `provider: "custom"`** — they only bind a *named* `custom_providers:` entry via
  its `key_env`. Set `model.api_key` in `config.yaml` instead; its only env
  fallback is the legacy name `OPENROUTER_API_KEY`. Get this wrong and the
  request ships the placeholder `no-key-required`, LiteLLM 401s **inside the
  response body**, and `hermes status` still reports a perfectly healthy
  `gen-large` / `Custom endpoint` — so status alone cannot verify this change,
  only a real completion can. The `zai` credential is still in `auth.json`,
  present and unused; `provider: custom` is explicit so it is not a candidate,
  and `hermes fallback list` is empty, so there is no degraded-mode route that
  quietly re-bills z.ai. Miranda fails rather than fails over if LiteLLM is down.
- **`nh3-dev/hermes/api-server-key` is free to rotate** (hold released
  2026-09-14, svos main `7165272`). It had been locked: SVOS signed its own
  Bifrost wall's HS256 dispatch tokens with the same value, so a routine
  rotation would have 401'd every Miranda tool call — silently, since losing
  signature validity does not raise. SVOS now verifies against its own secret
  (`SVOS_BIFROST_DISPATCH_KEY`, vaulted at `nh3-dev/svos/bifrost-dispatch-key`,
  theirs — nothing here reads it), and the Bearer has exactly one job again.
  Rotating it now only means svos-dev repoints `SVOS_HERMES_API_KEY`.
  ⚠ The invariant did not rot, its *precondition* did: "one shared secret" was
  sound while SVOS owned both ends of the key and stopped being sound the moment
  the runtime became externally managed. Worth remembering when any other peer
  reuses a credential across a boundary we control.
  ⚠ When the `svos_miranda` plugin arrives for `hermes plugins validate`, it
  will reference the **dispatch key, not the Bearer** — expected, not a defect.
  Assert its per-row `tools` array has **seven or eight** entries (`repo_read` is
  config-conditional on SVOS's side); any other number is a real fault.
- **bloom_music dev** — `~/development/bloom_music`; its `web/` test harness uses
  Playwright headless Chromium for OSMD browser-geometry assertions.
- **The Booth** — ephemeral media drop board (`:8090`, `booth.service`), from
  `~/development/booth` (gitea `vh/booth`, extracted from eshpfi 2026-09-21). Lets CC sessions surface A/B renders + smoke results
  (and browser uploads for pickup) to the operator; 24h TTL, Homepage-linked.
  Since 2026-09-09 it also carries **asks** — a session poses a multiple-choice
  question in a booth, the operator answers a radio form + notes in the browser,
  and the pick lands as an answer sidecar the session reads (`booth ask` /
  `booth answer --wait`). ⚠ The **`booth` CLI is on PATH via
  `~/.local/bin/booth` → `~/development/booth/scripts/booth`**, symlinked 2026-09-09;
  before that it was on no PATH at all, so every session following the global
  link-board convention was hitting `command not found` unless it used the full
  path. `~/.zshenv` puts `~/.local/bin` in PATH for non-interactive `ssh nh3-dev
  '<cmd>'` too.
- **jackdaw-compose** — JackDAW AI Composer `/compose` backend (`:8787`,
  `jackdaw-compose.service`), a thin stateless `bun server/index.ts` from
  `~/development/jackdaw` → LiteLLM `gen`. Origin-gated (INV-BK04/BK05), reached
  same-origin via the `:4500` bench's `/compose` proxy. Hosted for jackdaw-dev
  (their code; the model endpoint + key live in server env only — unit is `0600`,
  not committed).

## Box-wide Playwright / Chromium (2026-06-04)

Available to **every user/project** on this box — no per-home `playwright install`:
- System shared-libs: apt-installed via `playwright install-deps chromium` (Debian-12
  set + xvfb), global.
- Browser binaries: **shared `/opt/ms-playwright`** (chromium-1223 + headless-shell +
  ffmpeg), root-owned, world-readable. Installed via `infra-ops`.
- Discovery: `PLAYWRIGHT_BROWSERS_PATH=/opt/ms-playwright` set globally in
  `/etc/environment` (PAM/all sessions) + `/etc/profile.d/playwright-browsers.sh`
  (login shells). A project just `npm i playwright` (skip-browser-download is fine) and
  resolves the shared binary; verified launching headless from `/opt` as a normal user.
- To add more browsers / bump: `ssh infra-ops@10.100.10.50 'sudo env
  PLAYWRIGHT_BROWSERS_PATH=/opt/ms-playwright npx -y playwright install <browser>'`.

## Notes

- **PFI-owned Linux** — in scope for `infra-ops` management (apt, systemctl, service
  lifecycle). Added to the fleet bootstrap's Tier 1.
- OS: Debian 12 (bookworm). See `system-details.txt` for the latest snapshot
  (`scripts/refresh-server-info.sh nh3-dev`).
- Not in the colo Docker-stack topology — no `/opt/docker/compose` deploy target;
  workloads are systemd services + dev checkouts.
- **Retired (2026-06-08):** `volva.service` + `heid.service` user systemd units
  removed. Heid/Volva were re-architected from Python systemd daemons (`volva run`
  / `heid run` pollers) into Claude Code session orchestrators (heid commit
  `12aa5a9`); the `~/development/volva` dir + venvs are gone. `volva.service` had
  been crash-looping `203/EXEC`. Cleanup done by infra-ops at heid's request.