esh-pfi-infrastructure/servers/nh3-dev/README.md

# nh3-dev

NH3-site **developer box** — `10.100.10.50` (WireGuard-reachable from the NH3
subnet). General-purpose dev VM that hosts agent-fleet sidecars and live Claude
Code sessions; **not** a Docker-stack host in the `stacks/` sense.

**Reach:** `ssh 10.100.10.50` (as `lkraven`), or the dedicated agent identity
`ssh -i ~/.ssh/infra-ops_ed25519 infra-ops@10.100.10.50` (NOPASSWD sudo).
`infra-ops` bootstrapped here 2026-06-04 (see [`reference_infra_ops_sudo_identity`]
in auto-memory). Note: Claude Code sessions often run **natively on this box**, so
local Bash already executes here — no SSH-to-self needed for non-privileged work.

## What runs here

- **NH3 egress proxy — RETIRED 2026-09-06** (replaced by headscale exit nodes; `danted` disabled, config `.retired`). Was: durable internal-only SOCKS5 `socks5h://10.100.10.50:1080`
  (dante, ACL'd to the WG net). Residential egress for colo services gated on their
  datacenter IP (e.g. YouTube bot-gate). Runbook + setup committed; consumers point
  `*_PROXY` at it.
- **Docker runtime — `docker-ce` since 2026-09-14.** Was Debian's `docker.io`
  20.10.24 + the Python `docker-compose` 1.29.2 v1 CLI + `containerd` 1.6.20,
  with **no `cli-plugins` directory at all** — so `docker compose` (space) was
  not a command: it printed a help blurb and **exited 0**, which a deploy script
  cannot distinguish from success. Migrated via
  `playbooks/upgrade-docker-ce.yaml` to docker-ce 29.8.0 / compose plugin v5.5.1
  / containerd.io 2.3.5 / buildx v0.37.1. **The v1 `docker-compose` (hyphen)
  binary is gone and no shim was installed** (operator ruling 2026-09-14) — fix
  callers, don't paper over them.
  - ⚠ `vastblue-u5-pg` (an empty `postgres:16` probe container, `restart: no`,
    plain `docker run`, anonymous volume) **and its volume were removed** when
    the old daemon stopped. The playbook is not the cause — it has no `rm`,
    `prune` or `purge`, and the other five containers survived, two of them
    long-exited. Almost certainly `--rm` / `AutoRemove=true`, unprovable after
    the fact because the container record is gone. Measured beforehand as **zero
    user tables in every database**, so no data was lost. Lesson: capture
    `AutoRemove` and `RestartPolicy` together when snapshotting a container you
    are about to bounce.
  - The playbook's restart loop runs as the deploy identity, not root, and a
    stack `.env` may legitimately be root-owned `0600`
    (`/opt/docker/compose/beszel/.env`) — compose bails before doing anything,
    so the stack reported FAILED while `restart: unless-stopped` had already
    brought it back healthy. **Fixed 2026-09-14**: the loop retries under
    `sudo -n` before calling it a failure. Verified against beszel — plain
    `rc=1 permission denied`, sudo retry `rc=0 Container beszel-agent Started`.
    A false FAILED in automation output is worse than a quiet one; it trains
    readers to skim the failure lines.
- **`/opt/docker` ownership — normalised fleet-wide 2026-09-14 to
  `root:docker 2775`** (setgid) via `playbooks/normalize-docker-tree.yaml`,
  operator ruling. Was a three-way split: `root:root 755` here, `root:root 777`
  on nh3-docker + ana-docker (world-writable, from a 2024 `chmod -R 777` to get
  a git clone working), `lkraven 755` elsewhere. Not a personal username
  (`lkraven` is one of three the operator uses) and not a new admin account —
  the `docker` group already existed on every host holding exactly `lkraven` +
  `infra-ops`. ⚠ This is **not** privilege separation: `docker` membership is
  root-equivalent. A future non-root deployer needs a dedicated `deploy` group.
  Stack `.env` files went to `root:docker 0640` — previously 31 of 74 were
  `0600` readable by only ONE of the two deploy identities (varying by file,
  which is what false-FAILED beszel during the docker-ce upgrade) and 43 were
  world-readable `0644`. **No containers were bounced**; these are inode
  metadata changes and `.env` is read only at `compose up`. `/opt/docker/compose/talk` was created
  `lkraven`-owned 2026-09-14 and **tts-dev migrated `talk` into it the same
  day** — it had been at `~/talk`, a convention violation that hid it from
  anything walking `/opt/docker/compose/*/`. Old path parked at
  `~/talk.migrated-20260914`; `version: "3.8"` dropped from its compose file
  (it existed only to satisfy the v1 CLI, which is now gone). Normalising the
  parent directory is **unresolved — operator's call**; `/opt/docker` itself is a separate three-way split (`755` root, `777`
  root on two hosts, `755` lkraven).
- **ttyd fleet driver-seat** — web/iPad seat into the zellij `Claude` session (ttyd
  behind Caddy; OSC52 clipboard shim). User systemd services under `~/.config`.
- **mead-hall** — Bifrost tool-provider sidecar (`:5173`), CI-deployed from
  `vh/mead-hall`.
- **Hermes Agent gateway** — OpenAI-compatible agent API on `127.0.0.1:8765`
  (`hermes-gateway.service`, user systemd, installed 2026-09-14 via `hermes
  gateway install`). Runs the per-user install at `~/.hermes/hermes-agent`
  (v0.21.1, `b88e677`); config in `~/.hermes/{.env,config.yaml}`. Stood up for
  **SVOS/Miranda**, which replaced Worldtree with Hermes on 2026-09-11 and
  cannot boot without it. Bearer auth is mandatory even on loopback — key
  vaulted as `nh3-dev/hermes/api-server-key`. ⚠ The gateway registers Hermes's
  **full** toolset by default — 28 toolsets, 14 enabled, `terminal` /
  `code_execution` / `file` / `browser` among them. SVOS's security model is that
  write-capable tools are never *registered*, not that they are refused at
  dispatch, so `platform_toolsets: {api_server: []}` is set in `config.yaml`
  (2026-09-14) and measured back as 28 rows / **0 enabled / 0 tools** on
  `/v1/toolsets`. The endpoint still reports all 28 rows with their flags, which
  is what SVOS's `_hermes_roster` derives its required-config line from —
  narrowing does not blind it. Becomes `[svos_miranda]` once SVOS's plugin lands
  in `$HERMES_HOME/plugins/`.
- **Hermes model backend → `gen-large` on the fleet LiteLLM gateway** (free local
  compute), set 2026-09-14 per operator ruling. Until then `model.default` said
  `anthropic/claude-opus-4.6` with `model.base_url` at openrouter, but
  `provider: auto` plus a lone `zai` credential in `auth.json` silently resolved
  Miranda to **GLM-5.3 on the paid z.ai Coding Plan** — three settings that had
  to be read together before the real answer fell out. Now
  `default: gen-large` / `provider: custom` / `base_url:
  http://10.250.50.70:4000/v1`, verified by a real turn (`hermes status` →
  `gen-large` / `Custom endpoint`, plus a 660-token completion through
  `/v1/chat/completions`). The openrouter/nous credit warnings cleared with it.
  ⚠ **`CUSTOM_API_KEY` / `HERMES_CUSTOM_API_KEY` are INERT for bare
  `provider: "custom"`** — they only bind a *named* `custom_providers:` entry via
  its `key_env`. Set `model.api_key` in `config.yaml` instead; its only env
  fallback is the legacy name `OPENROUTER_API_KEY`. Get this wrong and the
  request ships the placeholder `no-key-required`, LiteLLM 401s **inside the
  response body**, and `hermes status` still reports a perfectly healthy
  `gen-large` / `Custom endpoint` — so status alone cannot verify this change,
  only a real completion can. The `zai` credential is still in `auth.json`,
  present and unused; `provider: custom` is explicit so it is not a candidate,
  and `hermes fallback list` is empty, so there is no degraded-mode route that
  quietly re-bills z.ai. Miranda fails rather than fails over if LiteLLM is down.
- ⚠ **Do not rotate `nh3-dev/hermes/api-server-key` yet.** SVOS currently reuses
  that value as the HS256 signing key on its Bifrost wall, so a rotation would
  401 every Miranda tool call. svos-dev is splitting theirs off (operator-
  approved 2026-09-14) and will confirm when it lands; rotation is safe after
  that, not before.
- **bloom_music dev** — `~/development/bloom_music`; its `web/` test harness uses
  Playwright headless Chromium for OSMD browser-geometry assertions.
- **The Booth** — ephemeral media drop board (`:8090`, `booth.service`), from
  eshpfi `services/booth/`. Lets CC sessions surface A/B renders + smoke results
  (and browser uploads for pickup) to the operator; 24h TTL, Homepage-linked.
  Since 2026-09-09 it also carries **asks** — a session poses a multiple-choice
  question in a booth, the operator answers a radio form + notes in the browser,
  and the pick lands as an answer sidecar the session reads (`booth ask` /
  `booth answer --wait`). ⚠ The **`booth` CLI is on PATH via
  `~/.local/bin/booth` → `services/booth/scripts/booth`**, symlinked 2026-09-09;
  before that it was on no PATH at all, so every session following the global
  link-board convention was hitting `command not found` unless it used the full
  path. `~/.zshenv` puts `~/.local/bin` in PATH for non-interactive `ssh nh3-dev
  '<cmd>'` too.
- **jackdaw-compose** — JackDAW AI Composer `/compose` backend (`:8787`,
  `jackdaw-compose.service`), a thin stateless `bun server/index.ts` from
  `~/development/jackdaw` → LiteLLM `gen`. Origin-gated (INV-BK04/BK05), reached
  same-origin via the `:4500` bench's `/compose` proxy. Hosted for jackdaw-dev
  (their code; the model endpoint + key live in server env only — unit is `0600`,
  not committed).

## Box-wide Playwright / Chromium (2026-06-04)

Available to **every user/project** on this box — no per-home `playwright install`:
- System shared-libs: apt-installed via `playwright install-deps chromium` (Debian-12
  set + xvfb), global.
- Browser binaries: **shared `/opt/ms-playwright`** (chromium-1223 + headless-shell +
  ffmpeg), root-owned, world-readable. Installed via `infra-ops`.
- Discovery: `PLAYWRIGHT_BROWSERS_PATH=/opt/ms-playwright` set globally in
  `/etc/environment` (PAM/all sessions) + `/etc/profile.d/playwright-browsers.sh`
  (login shells). A project just `npm i playwright` (skip-browser-download is fine) and
  resolves the shared binary; verified launching headless from `/opt` as a normal user.
- To add more browsers / bump: `ssh infra-ops@10.100.10.50 'sudo env
  PLAYWRIGHT_BROWSERS_PATH=/opt/ms-playwright npx -y playwright install <browser>'`.

## Notes

- **PFI-owned Linux** — in scope for `infra-ops` management (apt, systemctl, service
  lifecycle). Added to the fleet bootstrap's Tier 1.
- OS: Debian 12 (bookworm). See `system-details.txt` for the latest snapshot
  (`scripts/refresh-server-info.sh nh3-dev`).
- Not in the colo Docker-stack topology — no `/opt/docker/compose` deploy target;
  workloads are systemd services + dev checkouts.
- **Retired (2026-06-08):** `volva.service` + `heid.service` user systemd units
  removed. Heid/Volva were re-architected from Python systemd daemons (`volva run`
  / `heid run` pollers) into Claude Code session orchestrators (heid commit
  `12aa5a9`); the `~/development/volva` dir + venvs are gone. `volva.service` had
  been crash-looping `203/EXEC`. Cleanup done by infra-ops at heid's request.