Operator ruling: root:docker, not a personal username and not a new admin
account. lkraven is one of three names he uses, so baking it into shared
infrastructure guarantees a stale owner later; a dedicated deploy account
buys nothing the existing docker group doesn't, since that group already
exists on every host holding exactly lkraven + infra-ops.
Applied to nh3-dev, nh3-docker, esh-docker-vm, irv-ml1, ana-docker. All
five now 2775 root:docker on /opt/docker and /opt/docker/compose. Clears
the 0777 on nh3-docker and ana-docker. 55 stack .env files normalized to
root:docker 0640, tightening 43 world-readable ones and opening 31 that
were readable by only one of the two deploy identities. No containers
bounced — inode metadata only, and .env is read at compose up.
Deliberately not a recursive chmod. Three acme.json files and an ssh
private key are mode 0600 and traefik/ssh refuse to start if that widens,
which would have been a delayed failure surfacing at the next restart
rather than now. Protection is both mode-based (0600/0400 untouched) and
name-based (acme.json, *.key, *.pem, *.pfx, id_*); modes are symbolic so
the 53 executable files in these trees keep their exec bit.
Two defects found and fixed mid-rollout. The name list was initially
reported but not enforced, so a .key already at 0644 on esh-docker-vm was
widened to 0664 — reverted, and the list is now enforced in the chgrp and
widening steps. And the exec-bit verify asserted every .sh is executable,
which was never true and false-FAILED irv-ml1; it now compares the
executable-file count against a recorded baseline.
The restart loop runs as the deploy identity, not root, and a stack .env is
allowed to be root-owned 0600. compose bails on the unreadable file before
doing anything, so the stack was reported FAILED while restart=unless-stopped
had already brought it back healthy — a false failure, which is worse than a
quiet one because it trains readers to skim the failure lines.
Retry under sudo -n before calling it a failure, and print compose's own
output either way. Verified on nh3-dev against beszel: plain attempt rc=1
'open /opt/docker/compose/beszel/.env: permission denied', sudo retry rc=0
'Container beszel-agent Started', container back to healthy. The happy path
is unchanged — the sudo attempt only fires after a failure.
Also record that tts-dev migrated talk from ~/talk into
/opt/docker/compose/talk, which removes the one stack on this host that was
invisible to anything walking that path.
Operator cleared the swap and ruled out a docker-compose v1 shim. Ran
playbooks/upgrade-docker-ce.yaml: docker.io 20.10.24 -> docker-ce 29.8.0,
docker-compose 1.29.2 -> compose plugin v5.5.1, containerd 1.6.20 ->
containerd.io 2.3.5, buildx v0.37.1 added. 12 changed, 0 failed, verify
4/4. talk and beszel-agent back healthy on their restart policies.
The pre-state was worse than 'old': there was no cli-plugins directory, so
'docker compose' was not a command and exited 0 on a help blurb — a silent
no-op that reads as a successful deploy.
Records two things the run surfaced. vastblue-u5-pg and its anonymous
volume were removed when the old daemon stopped; the playbook has no rm,
prune or purge and five other containers survived, so the cause is almost
certainly --rm, unprovable now that the record is gone. It was measured
beforehand as zero user tables in every database, so nothing was lost. And
the playbook's restart loop runs as infra-ops and cannot read a root-owned
0600 stack .env, so it false-FAILs that stack.
Also notes that nh3-dev is the only host where /opt/docker/compose is
root-owned; the other four are lkraven. Created /opt/docker/compose/talk
as lkraven so tts-dev can move talk out of ~/talk. Normalising the parent
is left to the operator.
Operator ruled the repoint; Miranda moves off the paid z.ai Coding Plan
onto free local compute. model.default gen-large, provider custom,
base_url http://10.250.50.70:4000/v1. Verified by a real turn rather than
by config: hermes status reports gen-large / Custom endpoint and a
completion through /v1/chat/completions returns 660 tokens. The
openrouter/nous credit warnings cleared with it.
Records the landmine found on the way: CUSTOM_API_KEY and
HERMES_CUSTOM_API_KEY are inert for bare provider: custom — they bind only
a named custom_providers entry through its key_env. Without model.api_key
the request ships the placeholder no-key-required and LiteLLM 401s inside
the response body while hermes status still reports a healthy gen-large /
Custom endpoint, so status alone cannot verify this change.
Also notes that nh3-dev/hermes/api-server-key must not be rotated until
SVOS splits its HS256 signing key off the shared value.
Set platform_toolsets[api_server] = [] in ~/.hermes/config.yaml. Measured
back on /v1/toolsets: 28 rows before and after, enabled 14 -> 0, registered
tools 40 -> 0. The row count surviving matters — SVOS's _hermes_roster
derives its required-config line from the full report, so narrowing the
surface does not blind the diagnostic. terminal, code_execution, file,
browser and computer_use are now unregistered rather than merely
unreachable.
Separately, record an unresolved finding raised by svos-dev: model.default
names anthropic/claude-opus-4.6 and model.base_url names openrouter, but
provider: auto with only a zai credential resolves Miranda to GLM-5.3 on
the paid z.ai Coding Plan. Nothing under ~/.hermes references the LiteLLM
gateway. The operator believes Miranda runs on gen-large, which does exist
on LiteLLM. Not repointed — the premise is his to confirm.
Stood up hermes-gateway.service (user systemd, vendor 'hermes gateway
install') for SVOS/Miranda, which cut over from Worldtree to Hermes on
2026-09-11 and aborts at boot without it. Bearer key vaulted as
nh3-dev/hermes/api-server-key.
Notes the wide default tool surface: with no roster config the gateway
registers all 28 toolsets on the api_server platform, terminal and
code_execution included. Loopback bind plus the key are the only
containment until SVOS installs its svos_miranda plugin and the
platform_toolsets / disabled_toolsets roster is applied.
Two user-systemd services stood up on nh3-dev this session weren't in the
"What runs here" list:
- The Booth (:8090, booth.service) — eshpfi services/booth ephemeral media board.
- jackdaw-compose (:8787, jackdaw-compose.service) — JackDAW /compose AI backend
hosted for jackdaw-dev, Origin-gated, fronted by the :4500 bench proxy.
Keyless entries (the jackdaw unit's key stays server-side, uncommitted).
- migration-plan.md: workstation-only migration of nh3-dev's dev env to
the M2 Ultra Mac Pro Rack (on-subnet, sidecars stay on Linux).
- nh3-dev README: drop volva from live sidecars + add retired note.
volva.service/heid.service removed by infra-ops at heid's request
(re-architected daemon->CC-orchestrator; volva.service was crash-looping).
Reflects the shared install: browsers in /opt/ms-playwright (root-owned,
world-readable) + PLAYWRIGHT_BROWSERS_PATH wired globally via
/etc/environment + profile.d, so any user/project resolves the shared
binary without a per-home playwright install. Includes the add-a-browser
command for future bumps.
Adds the NH3 dev box (10.100.10.50) as a managed host: README, first
system-details snapshot, ssh-target. Adds it to the fleet bootstrap's
Tier 1 and the CLAUDE.md servers table. infra-ops identity bootstrapped
there (operator-run) — NOPASSWD sudo + key, so root-level infra work on
the box (it runs the egress proxy, ttyd seat, mead-hall, volva, and dev
checkouts) no longer needs a per-task password.
First use: installed Playwright headless-Chromium system deps + binary
for bloom_music's OSMD browser-geometry test harness; headless launch +
real SVG geometry verified.
vh