esh-pfi-infrastructure/servers/nh3-dev/README.md

nh3-dev

NH3-site developer box — 10.100.10.50 (WireGuard-reachable from the NH3 subnet). General-purpose dev VM that hosts agent-fleet sidecars and live Claude Code sessions; not a Docker-stack host in the stacks/ sense.

Reach: ssh 10.100.10.50 (as lkraven), or the dedicated agent identity ssh -i ~/.ssh/infra-ops_ed25519 infra-ops@10.100.10.50 (NOPASSWD sudo). infra-ops bootstrapped here 2026-06-04 (see [reference_infra_ops_sudo_identity] in auto-memory). Note: Claude Code sessions often run natively on this box, so local Bash already executes here — no SSH-to-self needed for non-privileged work.

What runs here

• NH3 egress proxy — RETIRED 2026-09-06 (replaced by headscale exit nodes; danted disabled, config .retired). Was: durable internal-only SOCKS5 socks5h://10.100.10.50:1080 (dante, ACL'd to the WG net). Residential egress for colo services gated on their datacenter IP (e.g. YouTube bot-gate). Runbook + setup committed; consumers point *_PROXY at it.
• Docker runtime — docker-ce since 2026-09-14. Was Debian's docker.io 20.10.24 + the Python docker-compose 1.29.2 v1 CLI + containerd 1.6.20, with no cli-plugins directory at all — so docker compose (space) was not a command: it printed a help blurb and exited 0, which a deploy script cannot distinguish from success. Migrated via playbooks/upgrade-docker-ce.yaml to docker-ce 29.8.0 / compose plugin v5.5.1 / containerd.io 2.3.5 / buildx v0.37.1. The v1 docker-compose (hyphen) binary is gone and no shim was installed (operator ruling 2026-09-14) — fix callers, don't paper over them.
  • ⚠ vastblue-u5-pg (an empty postgres:16 probe container, restart: no, plain docker run, anonymous volume) and its volume were removed when the old daemon stopped. The playbook is not the cause — it has no rm, prune or purge, and the other five containers survived, two of them long-exited. Almost certainly --rm / AutoRemove=true, unprovable after the fact because the container record is gone. Measured beforehand as zero user tables in every database, so no data was lost. Lesson: capture AutoRemove and RestartPolicy together when snapshotting a container you are about to bounce.
  • The playbook's restart loop runs as the deploy identity, not root, and a stack .env may legitimately be root-owned 0600 (/opt/docker/compose/beszel/.env) — compose bails before doing anything, so the stack reported FAILED while restart: unless-stopped had already brought it back healthy. Fixed 2026-09-14: the loop retries under sudo -n before calling it a failure. Verified against beszel — plain rc=1 permission denied, sudo retry rc=0 Container beszel-agent Started. A false FAILED in automation output is worse than a quiet one; it trains readers to skim the failure lines.
• /opt/docker ownership — normalised fleet-wide 2026-09-14 to root:docker 2775 (setgid) via playbooks/normalize-docker-tree.yaml, operator ruling. Was a three-way split: root:root 755 here, root:root 777 on nh3-docker + ana-docker (world-writable, from a 2024 chmod -R 777 to get a git clone working), lkraven 755 elsewhere. Not a personal username (lkraven is one of three the operator uses) and not a new admin account — the docker group already existed on every host holding exactly lkraven + infra-ops. ⚠ This is not privilege separation: docker membership is root-equivalent. A future non-root deployer needs a dedicated deploy group. Stack .env files went to root:docker 0640 — previously 31 of 74 were 0600 readable by only ONE of the two deploy identities (varying by file, which is what false-FAILED beszel during the docker-ce upgrade) and 43 were world-readable 0644. No containers were bounced; these are inode metadata changes and .env is read only at compose up. /opt/docker/compose/talk was created lkraven-owned 2026-09-14 and tts-dev migrated talk into it the same day — it had been at ~/talk, a convention violation that hid it from anything walking /opt/docker/compose/*/. Old path parked at ~/talk.migrated-20260914; version: "3.8" dropped from its compose file (it existed only to satisfy the v1 CLI, which is now gone). Normalising the parent directory is unresolved — operator's call; /opt/docker itself is a separate three-way split (755 root, 777 root on two hosts, 755 lkraven).
• ttyd fleet driver-seat — web/iPad seat into the zellij Claude session (ttyd behind Caddy; OSC52 clipboard shim). User systemd services under ~/.config.
• mead-hall — Bifrost tool-provider sidecar (:5173), CI-deployed from vh/mead-hall.
• Hermes Agent gateway — OpenAI-compatible agent API on 127.0.0.1:8765 (hermes-gateway.service, user systemd, installed 2026-09-14 via hermes gateway install). Runs the per-user install at ~/.hermes/hermes-agent (v0.21.1, b88e677); config in ~/.hermes/{.env,config.yaml}. Stood up for SVOS/Miranda, which replaced Worldtree with Hermes on 2026-09-11 and cannot boot without it. Bearer auth is mandatory even on loopback — key vaulted as nh3-dev/hermes/api-server-key. ⚠ The gateway registers Hermes's full toolset by default — 28 toolsets, 14 enabled, terminal / code_execution / file / browser among them. SVOS's security model is that write-capable tools are never registered, not that they are refused at dispatch, so platform_toolsets: {api_server: []} is set in config.yaml (2026-09-14) and measured back as 28 rows / 0 enabled / 0 tools on /v1/toolsets. The endpoint still reports all 28 rows with their flags, which is what SVOS's _hermes_roster derives its required-config line from — narrowing does not blind it. Becomes [svos_miranda] once SVOS's plugin lands in $HERMES_HOME/plugins/.
• Hermes model backend → gen-large on the fleet LiteLLM gateway (free local compute), set 2026-09-14 per operator ruling. Until then model.default said anthropic/claude-opus-4.6 with model.base_url at openrouter, but provider: auto plus a lone zai credential in auth.json silently resolved Miranda to GLM-5.3 on the paid z.ai Coding Plan — three settings that had to be read together before the real answer fell out. Now default: gen-large / provider: custom / base_url: http://10.250.50.70:4000/v1, verified by a real turn (hermes status → gen-large / Custom endpoint, plus a 660-token completion through /v1/chat/completions). The openrouter/nous credit warnings cleared with it. ⚠ CUSTOM_API_KEY / HERMES_CUSTOM_API_KEY are INERT for bare provider: "custom" — they only bind a named custom_providers: entry via its key_env. Set model.api_key in config.yaml instead; its only env fallback is the legacy name OPENROUTER_API_KEY. Get this wrong and the request ships the placeholder no-key-required, LiteLLM 401s inside the response body, and hermes status still reports a perfectly healthy gen-large / Custom endpoint — so status alone cannot verify this change, only a real completion can. The zai credential is still in auth.json, present and unused; provider: custom is explicit so it is not a candidate, and hermes fallback list is empty, so there is no degraded-mode route that quietly re-bills z.ai. Miranda fails rather than fails over if LiteLLM is down.
• ⚠ Do not rotate nh3-dev/hermes/api-server-key yet. SVOS currently reuses that value as the HS256 signing key on its Bifrost wall, so a rotation would 401 every Miranda tool call. svos-dev is splitting theirs off (operator- approved 2026-09-14) and will confirm when it lands; rotation is safe after that, not before.
• bloom_music dev — ~/development/bloom_music; its web/ test harness uses Playwright headless Chromium for OSMD browser-geometry assertions.
• The Booth — ephemeral media drop board (:8090, booth.service), from eshpfi services/booth/. Lets CC sessions surface A/B renders + smoke results (and browser uploads for pickup) to the operator; 24h TTL, Homepage-linked. Since 2026-09-09 it also carries asks — a session poses a multiple-choice question in a booth, the operator answers a radio form + notes in the browser, and the pick lands as an answer sidecar the session reads (booth ask / booth answer --wait). ⚠ The booth CLI is on PATH via ~/.local/bin/booth → services/booth/scripts/booth, symlinked 2026-09-09; before that it was on no PATH at all, so every session following the global link-board convention was hitting command not found unless it used the full path. ~/.zshenv puts ~/.local/bin in PATH for non-interactive ssh nh3-dev '<cmd>' too.
• jackdaw-compose — JackDAW AI Composer /compose backend (:8787, jackdaw-compose.service), a thin stateless bun server/index.ts from ~/development/jackdaw → LiteLLM gen. Origin-gated (INV-BK04/BK05), reached same-origin via the :4500 bench's /compose proxy. Hosted for jackdaw-dev (their code; the model endpoint + key live in server env only — unit is 0600, not committed).

Box-wide Playwright / Chromium (2026-06-04)

Available to every user/project on this box — no per-home playwright install:

• System shared-libs: apt-installed via playwright install-deps chromium (Debian-12 set + xvfb), global.
• Browser binaries: shared /opt/ms-playwright (chromium-1223 + headless-shell + ffmpeg), root-owned, world-readable. Installed via infra-ops.
• Discovery: PLAYWRIGHT_BROWSERS_PATH=/opt/ms-playwright set globally in /etc/environment (PAM/all sessions) + /etc/profile.d/playwright-browsers.sh (login shells). A project just npm i playwright (skip-browser-download is fine) and resolves the shared binary; verified launching headless from /opt as a normal user.
• To add more browsers / bump: ssh infra-ops@10.100.10.50 'sudo env PLAYWRIGHT_BROWSERS_PATH=/opt/ms-playwright npx -y playwright install <browser>'.

Notes

• PFI-owned Linux — in scope for infra-ops management (apt, systemctl, service lifecycle). Added to the fleet bootstrap's Tier 1.
• OS: Debian 12 (bookworm). See system-details.txt for the latest snapshot (scripts/refresh-server-info.sh nh3-dev).
• Not in the colo Docker-stack topology — no /opt/docker/compose deploy target; workloads are systemd services + dev checkouts.
• Retired (2026-06-08): volva.service + heid.service user systemd units removed. Heid/Volva were re-architected from Python systemd daemons (volva run / heid run pollers) into Claude Code session orchestrators (heid commit 12aa5a9); the ~/development/volva dir + venvs are gone. volva.service had been crash-looping 203/EXEC. Cleanup done by infra-ops at heid's request.