A deliberate `systemctl --user restart hermes-gateway` paged infra-ops as a
FAILED unit (msg 3716) while the unit was already back up. Cause is a Hermes
v0.21.1 race: the planned-stop marker watcher runs the shutdown handler before
systemd's SIGTERM, consumes the marker, and the SIGTERM re-runs the handler,
which then classifies the stop as unexpected and exits 1.
Corrects the README claim that OnFailure never fires on a deliberate restart:
that holds only when the main process exits with a success status. Measured on a
throwaway unit (3/3 paged without SuccessExitStatus, 0/3 with it, and crash
restarts are unaffected), and a survey of every stop on nh3-dev since 09-15
found hermes-gateway to be the only unit that does this.
The host-side fix is drop-in hermes-gateway.service.d/
20-planned-stop-exit1-is-clean.conf (SuccessExitStatus=1). No alarm coverage
is lost: Restart=always ignores the classification, and StartLimitIntervalSec=0
means the unit can never reach `failed` from a start failure.
The Booth is now one of the most-used fleet tools -- 17 agent handles post to
it daily -- and it is taking an information-architecture rework plus a
cross-agent SVOS design retrofit from design-dev. That work wants its own
ROADMAP, contracts and blast radius, not the fleet-infrastructure repo's.
All 29 commits moved with it via `git subtree split`; the history carries real
lessons (two shipped-dead controls, the verbatim-injection traps) that a
squashed import would have thrown away.
Live service repointed and verified: the user unit and the ~/.local/bin/booth
symlink now resolve into ~/development/booth, healthz answers, all 24 booths
intact. services/booth/ keeps a pointer README, same shape as the
chatterbox-fast and tts-stack extractions.
nh3-dev sits on the NH3 LAN and reaches every site through its own default
gateway; RouteAll was already false, so it never used the tunnel for routing.
Membership bought a 100.64.0.4 address nothing referenced -- grep across the
repo and ~/development found only docs and memory hits.
It also cost something concrete. A host running Tailscale installs
-A ts-input -s 100.64.0.0/10 ! -i tailscale0 -j DROP, and because the fleet's
subnet routers preserve source rather than masquerading RFC1918, a mesh client's
packet reached nh3-dev's ens18 still sourced 100.64.x and was dropped silently.
That is why nh3-dev.nh3.internal failed from the mesh while every NH3 host that
does not run Tailscale worked, and it needed a -d 10.100.10.50/32 -j MASQUERADE
exception on nh3-scale to paper over. Retiring the membership removed the
anti-spoof rule, so the exception went with it -- mesh-exit-masq.sh is back to
the two rules it had before yesterday.
Verified after: nh3-dev reachable at 10.100.10.50 from ESH, Anaheim, FV, Irvine
and NH3, and reaching all four sites plus the internet itself. fv-ml1
unaffected.
The mesh is now six nodes and every one has a job: three site routers,
vb-gateway, irv-ml1 (Irvine's own router, no separate scale node), and the
operator's MacBook Air. Nothing is enrolled just in case.
One rule on nh3-scale (CT 107): -d 10.100.10.50/32 -j MASQUERADE, above the
RFC1918 RETURNs in /usr/local/sbin/mesh-exit-masq.sh, so it survives a reboot
rather than living only in the running ruleset.
Cause. A host that runs Tailscale installs -A ts-input -s 100.64.0.0/10
! -i tailscale0 -j DROP. The fleet's subnet routers run NoSNAT: true with
RFC1918 explicitly exempted from masquerade — deliberate source preservation,
and a departure from Tailscale's own --snat-subnet-routes=true default — so a
mesh client's packet reached nh3-dev's ens18 still sourced 100.64.x and died
at the anti-spoof rule. Every NH3 host that does not run Tailscale was
unaffected, which is why this read as a DNS or routing fault rather than a
policy one. Masquerading just this destination makes it behave like every
other host and leaves source preservation absolute elsewhere.
Verified before and after against 13 targets from nh3-dev and 9 from the
MacBook Air, and again after restarting the service so the chain was rebuilt
from the script rather than from the manual insert. nh3-dev.nh3.internal now
resolves and connects from the mesh, ssh and the Booth port included, with no
script changes anywhere.
Records the failed approach prominently, because it is the attractive one:
advertising 10.100.10.50/32 from nh3-dev itself black-holed it from ESH,
Anaheim, FV and Irvine. ip rule there puts lookup 52 at priority 5270 ahead of
main at 32766, and becoming a subnet router let table 52 capture cross-site
traffic the node has no accepted route for. Its own LAN and the internet kept
working throughout, so a single-host check confirms a break it cannot see.
svos-dev split their Bifrost wall's HS256 signing key off the Hermes Bearer
(svos main 7165272), so nh3-dev/hermes/api-server-key is free to rotate again.
The previous note said do-not-rotate and would have made a future session
refuse a legitimate rotation on stale grounds.
Not rotating now: the key was minted today, is vaulted, and has never been
exposed — rotation is a hygiene action with a trigger, and none applies. What
changed is the capability, which is what the record needs to reflect.
Also records two things for when the svos_miranda plugin arrives: it will
reference the dispatch key rather than the Bearer (expected, not a defect), and
its tools array is legitimately seven or eight entries because repo_read is
conditional on a config block SVOS owns. A third number is a real fault.
Operator ruling: root:docker, not a personal username and not a new admin
account. lkraven is one of three names he uses, so baking it into shared
infrastructure guarantees a stale owner later; a dedicated deploy account
buys nothing the existing docker group doesn't, since that group already
exists on every host holding exactly lkraven + infra-ops.
Applied to nh3-dev, nh3-docker, esh-docker-vm, irv-ml1, ana-docker. All
five now 2775 root:docker on /opt/docker and /opt/docker/compose. Clears
the 0777 on nh3-docker and ana-docker. 55 stack .env files normalized to
root:docker 0640, tightening 43 world-readable ones and opening 31 that
were readable by only one of the two deploy identities. No containers
bounced — inode metadata only, and .env is read at compose up.
Deliberately not a recursive chmod. Three acme.json files and an ssh
private key are mode 0600 and traefik/ssh refuse to start if that widens,
which would have been a delayed failure surfacing at the next restart
rather than now. Protection is both mode-based (0600/0400 untouched) and
name-based (acme.json, *.key, *.pem, *.pfx, id_*); modes are symbolic so
the 53 executable files in these trees keep their exec bit.
Two defects found and fixed mid-rollout. The name list was initially
reported but not enforced, so a .key already at 0644 on esh-docker-vm was
widened to 0664 — reverted, and the list is now enforced in the chgrp and
widening steps. And the exec-bit verify asserted every .sh is executable,
which was never true and false-FAILED irv-ml1; it now compares the
executable-file count against a recorded baseline.
The restart loop runs as the deploy identity, not root, and a stack .env is
allowed to be root-owned 0600. compose bails on the unreadable file before
doing anything, so the stack was reported FAILED while restart=unless-stopped
had already brought it back healthy — a false failure, which is worse than a
quiet one because it trains readers to skim the failure lines.
Retry under sudo -n before calling it a failure, and print compose's own
output either way. Verified on nh3-dev against beszel: plain attempt rc=1
'open /opt/docker/compose/beszel/.env: permission denied', sudo retry rc=0
'Container beszel-agent Started', container back to healthy. The happy path
is unchanged — the sudo attempt only fires after a failure.
Also record that tts-dev migrated talk from ~/talk into
/opt/docker/compose/talk, which removes the one stack on this host that was
invisible to anything walking that path.
Operator cleared the swap and ruled out a docker-compose v1 shim. Ran
playbooks/upgrade-docker-ce.yaml: docker.io 20.10.24 -> docker-ce 29.8.0,
docker-compose 1.29.2 -> compose plugin v5.5.1, containerd 1.6.20 ->
containerd.io 2.3.5, buildx v0.37.1 added. 12 changed, 0 failed, verify
4/4. talk and beszel-agent back healthy on their restart policies.
The pre-state was worse than 'old': there was no cli-plugins directory, so
'docker compose' was not a command and exited 0 on a help blurb — a silent
no-op that reads as a successful deploy.
Records two things the run surfaced. vastblue-u5-pg and its anonymous
volume were removed when the old daemon stopped; the playbook has no rm,
prune or purge and five other containers survived, so the cause is almost
certainly --rm, unprovable now that the record is gone. It was measured
beforehand as zero user tables in every database, so nothing was lost. And
the playbook's restart loop runs as infra-ops and cannot read a root-owned
0600 stack .env, so it false-FAILs that stack.
Also notes that nh3-dev is the only host where /opt/docker/compose is
root-owned; the other four are lkraven. Created /opt/docker/compose/talk
as lkraven so tts-dev can move talk out of ~/talk. Normalising the parent
is left to the operator.
Operator ruled the repoint; Miranda moves off the paid z.ai Coding Plan
onto free local compute. model.default gen-large, provider custom,
base_url http://10.250.50.70:4000/v1. Verified by a real turn rather than
by config: hermes status reports gen-large / Custom endpoint and a
completion through /v1/chat/completions returns 660 tokens. The
openrouter/nous credit warnings cleared with it.
Records the landmine found on the way: CUSTOM_API_KEY and
HERMES_CUSTOM_API_KEY are inert for bare provider: custom — they bind only
a named custom_providers entry through its key_env. Without model.api_key
the request ships the placeholder no-key-required and LiteLLM 401s inside
the response body while hermes status still reports a healthy gen-large /
Custom endpoint, so status alone cannot verify this change.
Also notes that nh3-dev/hermes/api-server-key must not be rotated until
SVOS splits its HS256 signing key off the shared value.
Set platform_toolsets[api_server] = [] in ~/.hermes/config.yaml. Measured
back on /v1/toolsets: 28 rows before and after, enabled 14 -> 0, registered
tools 40 -> 0. The row count surviving matters — SVOS's _hermes_roster
derives its required-config line from the full report, so narrowing the
surface does not blind the diagnostic. terminal, code_execution, file,
browser and computer_use are now unregistered rather than merely
unreachable.
Separately, record an unresolved finding raised by svos-dev: model.default
names anthropic/claude-opus-4.6 and model.base_url names openrouter, but
provider: auto with only a zai credential resolves Miranda to GLM-5.3 on
the paid z.ai Coding Plan. Nothing under ~/.hermes references the LiteLLM
gateway. The operator believes Miranda runs on gen-large, which does exist
on LiteLLM. Not repointed — the premise is his to confirm.
Stood up hermes-gateway.service (user systemd, vendor 'hermes gateway
install') for SVOS/Miranda, which cut over from Worldtree to Hermes on
2026-09-11 and aborts at boot without it. Bearer key vaulted as
nh3-dev/hermes/api-server-key.
Notes the wide default tool surface: with no roster config the gateway
registers all 28 toolsets on the api_server platform, terminal and
code_execution included. Loopback bind plus the key are the only
containment until SVOS installs its svos_miranda plugin and the
platform_toolsets / disabled_toolsets roster is applied.
Two user-systemd services stood up on nh3-dev this session weren't in the
"What runs here" list:
- The Booth (:8090, booth.service) — eshpfi services/booth ephemeral media board.
- jackdaw-compose (:8787, jackdaw-compose.service) — JackDAW /compose AI backend
hosted for jackdaw-dev, Origin-gated, fronted by the :4500 bench proxy.
Keyless entries (the jackdaw unit's key stays server-side, uncommitted).
- migration-plan.md: workstation-only migration of nh3-dev's dev env to
the M2 Ultra Mac Pro Rack (on-subnet, sidecars stay on Linux).
- nh3-dev README: drop volva from live sidecars + add retired note.
volva.service/heid.service removed by infra-ops at heid's request
(re-architected daemon->CC-orchestrator; volva.service was crash-looping).
Reflects the shared install: browsers in /opt/ms-playwright (root-owned,
world-readable) + PLAYWRIGHT_BROWSERS_PATH wired globally via
/etc/environment + profile.d, so any user/project resolves the shared
binary without a per-home playwright install. Includes the add-a-browser
command for future bumps.
Adds the NH3 dev box (10.100.10.50) as a managed host: README, first
system-details snapshot, ssh-target. Adds it to the fleet bootstrap's
Tier 1 and the CLAUDE.md servers table. infra-ops identity bootstrapped
there (operator-run) — NOPASSWD sudo + key, so root-level infra work on
the box (it runs the egress proxy, ttyd seat, mead-hall, volva, and dev
checkouts) no longer needs a per-task password.
First use: installed Playwright headless-Chromium system deps + binary
for bloom_music's OSMD browser-geometry test harness; headless launch +
real SVG geometry verified.
vh