NH3 site visit done: Secure Boot off, iGPU restored as boot VGA, AMT port cabled.
- nh3-pve: NVIDIA 580.178.04 (DKMS, open modules) via pve-nvidia-host.yaml.
- nh3-ml1 = CT 109 @ 10.100.50.80 via gpu-lxc.yaml; embed-rerank (TEI 1.9.4)
deployed with HOST_NAME/HOST_IP labels.
- Parity vs esh-ml1 (1,126 texts, 2 runs/host, controls): embed cosine min
0.999993 = own noise floor; overlap@10 1.000 vs MRL-256 positive control
0.684; rerank top-1 1.00, max diff 0.0014 vs floor 0.0020. On-box speed
identical within rep spread.
- gpu-lxc.yaml: first step upgrades lxc-pve to >= 6.0.0-2 (Proxmox fix#7006).
With 6.0.0-1 every docker run in a nesting CT failed on runc 1.5's sysctl
reopen; applied on nh3-pve (one package).
- pve-nvidia-host.yaml: document that the headers meta drags in the newest
kernel (nh3-pve went 6.8.12-11 -> -43 at the next reboot).
- Monitoring: Beszel NVIDIA agent + 5 alerts, Kuma #29/#30, Homepage
nh3-ml1-docker, Dozzle agent (hub 8 clients). DNS nh3-ml1.nh3.internal.
- nh3-pve README: SB/IGFX/driver/kernel state, btmtk oops on -4x kernels,
AMT cabled but unreachable on the network.
Gateway routing to nh3-ml1 is not changed.
- esh-pve-nvidia-host -> pve-nvidia-host: headers/dkms/build-essential step,
nouveau blacklist + guarded unload (refuses if nouveau bound a device)
- esh-ml1-lxc -> gpu-lxc: host vars have no defaults (elway aborts on undefined),
rootfs storage/startup order parameterized, CT kept out of all-guests vzdump jobs
- embed-rerank: Homepage labels take HOST_NAME/HOST_IP, defaults = esh-ml1
nh3-pve and esh-pve are the same Minisforum MS-01 (BIOS AHWSA.1.17). With a
card in the x16 slot its root port takes bus 01 and every NIC moves down a
bus (measured on esh-pve), so predictable names change (enp2s0f0np0 ->
enp3s0f0np0 etc.) and vmbr0 would boot with no uplink. systemd .link files
now pin all NICs by MAC, baked into every initramfs and synced to the ESP;
udev confirms the files apply. The AMT-capable I226-LM (enp88s0) leaves
vmbr0's bridge-ports in the file (next boot), so cabling it for AMT cannot
loop the STP-less bridge.
Also: documented the NanoKVM (https://10.100.250.171) as nh3-pve's console
OOB and that AMT is not wired; nh3-dev's Beszel agent no longer binds NAS
shares (it died on the last NH3 cold start); post-boot checklist in
persistent-memory.
- Beszel: NVIDIA agent (stacks/beszel hosts/esh-ml1.yaml), hub system
registered with Status/Disk/CPU/Memory/Temperature alerts; GPU samples
verified (RTX 2000E Ada util, VRAM, power, temp).
- Uptime Kuma: health monitors for the TEI embed (:8001) and rerank (:8013)
services — the sole backends behind the gateway, so the one exception to
"seats are out of Kuma's lane". Reward seat excluded (no consumer).
- Homepage: dockerd on tcp/2375 bound to 10.0.50.80 (playbook step);
esh-ml1-docker added to docker.yaml.
- Dozzle: agent v10.4.1 on esh-ml1. The hub's DOZZLE_REMOTE_AGENT still
named ana-ml2's 10.250.50.54 and irv-ml1's retired 10.100.79.3; repointed
to 10.251.50.54 / 10.6.110.50 (fv-ml1 and irv-ml1 logs were missing).
- Documented nh3-dev's Beszel agent failing on NH3 cold start (bind of an
automounted NAS share); revived by hand, fix still open.
Prime, 2026-09-25: TEI serves embedding + reranking for esh-ml1 and the fleet
from now on; fv-ml1 retires both once esh-ml1 is up.
- stacks/embed-rerank: vLLM -> TEI 1.9.4 (89- Ada build), same ports
8001/8013, fail-closed truncation (--auto-truncate false; embed
--max-batch-tokens 32768).
- litellm: qwen3-embedding -> esh-ml1 only (hosted_vllm/, unchanged address);
reranker -> huggingface/ provider at :8013 (hosted_vllm/ 422s on TEI's
`texts` body). DB alias reranker-a3-bge-v2-m3 patched to the same target.
- Verified via the gateway against the retiring fv-ml1 seats: embed cosine
median 0.999927 (n=203); rerank top-1/top-3 29/30.
- stacks/vllm: vllm-embed and vllm-rerank-a3 removed (containers retired on
fv-ml1, GPU 1 freed ~6.1 GB); reward + coder unchanged.
- Bake-off record moved to docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md;
CLAUDE.md gains the TEI convention.
esh-pve: NVIDIA 580.178.04 (open modules, DKMS) installed on the host from
NVIDIA's .run and loaded live, no reboot. nvidia-persistenced unit creates the
device nodes before pve-guests; the T400's vfio-pci ids and `blacklist nvidia`
retired. playbooks/esh-pve-nvidia-host.yaml.
esh-ml1: CT 110, unprivileged Debian 12, 10.0.50.80, GPU nodes via devN,
NVIDIA userspace from the same .run (--no-kernel-modules), docker-ce +
nvidia-container-toolkit (no-cgroups). playbooks/esh-ml1-lxc.yaml. Not in the
vzdump job on purpose. DNS esh-ml1.esh.internal.
stacks/embed-rerank: Qwen3-Embedding-0.6B :8001 + bge-reranker-v2-m3 :8013 on
the same vLLM v0.24.0 digest and flags as fv-ml1. Measured parity: embed
cosine FV-vs-ESH median 0.999908 (min 0.999772), inside both self-noise
floors; rerank max |delta| 0.000145 vs floor 0.000181, identical ranking.
litellm: qwen3-embedding and reranker gain an esh-ml1 deployment at order 2
behind fv-ml1 (order 1). Order fallback proven with throwaway groups:
refused primary +0.15 s, host-down primary ~18.7 s per call, dead-only 500.
Also: repaired the DB-only alias reranker-a3-bge-v2-m3, dead since the
fv-ml1 relocation (still named 10.250.50.54); documented the third
unkillable homepage wedge on esh-docker-vm.
After the 2026-09-24 NH3 power outage every NFS line from nh3-nas failed at
boot: nh3-docker's /mnt/compose and /mnt/backup, and nh3-dev's /mnt/backup.
The NAS is the slowest box to serve NFS, and the plain fstab mounts stayed
down until remounted by hand. nh3-dev's /mnt/smithy already used
x-systemd.automount, and the same boot shows it self-healing on its next
access. playbooks/nh3-nfs-automount.yaml brings the other lines to that
shape (_netdev,nofail,x-systemd.automount,x-systemd.mount-timeout=30).
Hard-mount semantics are unchanged; only WHEN the mount happens moves.
Applied and read back on nh3-docker and nh3-dev: every nh3-nas line carries
automount, each automount unit is active, and each share is NFS-mounted after
access. A second run on nh3-docker is a no-op. The fstab check judges only
errors the rewrite ADDS, because nh3-dev's installer cdrom line has a
pre-existing verify error. The first run caught that and aborted without
writing.
servers/nh3-pve/README.md now lists each guest's expected state after a
power loss: nh3-laser is on-demand and stays off (Prime), pbs-nh3 now has
onboot=1 (it had none and stayed down), and pfi-gx10 is bare metal that
does not auto-power-on.
elway's sudo upload did scp-as-user then `sudo mv`, and mv keeps the owner,
so every file it installed "as root" (systemd units, /etc configs, root-run
scripts) ended up owned by the SSH user. A sudoers drop-in installed that
way would be rejected by sudo outright.
elway:
- Sudo uploads now chown to root:root by default. Playbooks can override
with `upload.owner:` and ad-hoc runs with `--owner`. An owner is refused
on a non-sudo upload, and `--owner` outside an ad-hoc `--upload` is an
error rather than silently ignored.
- Ownership and mode are applied to the STAGED file, then a single mv
publishes it, so a failed chown can no longer leave the live path owned
by the SSH user. chown runs before chmod so setuid bits survive. A trap
removes the staged file on every exit path.
- A directory dest is refused before anything moves. mv would otherwise
drop the file inside the directory under a staging name.
- `mode` was spliced unquoted into the remote root shell line. It is now
validated as octal (^[0-7]{3,4}$) and shell-quoted, on both the sudo
and non-sudo paths.
- `mode` and `owner` must be quoted YAML strings. YAML 1.1 turns a bare
0644 into 420, a bare 1000:0 into 60000 and a bare yes into True, and
str() of each is a different value that looks valid. All 85 playbooks
already quote them.
- New preflight(): every step is resolved before any remote action. An
undefined var or a templated owner that resolves badly is now refused
up front, not after earlier steps have already changed the host.
scripts/fleet-ownership-audit.sh (new, read-only) finds files in root's
territory owned by a normal user. Tier A lists /etc, /usr/local, /root and
cron, judging symlinks by their target. Tier X covers files that root-run
units exec from /opt, /srv, /home or /volume1, reading drop-ins and quoted
paths. Tier B only summarises /opt app trees. A host that is unprivileged,
whose find fails, or whose run dies is reported INCOMPLETE (exit 5), never
clean. Completion is marked with a per-run nonce.
Verification: 24 unit tests in scripts/test_elway.py. Live checks on
esh-docker-vm: default lands root:root; the override lands infra-ops:adm at
4755 with setuid intact; a bad group fails with dest untouched and no
staging left; a directory dest is refused; a bad mode is refused before
any copy. Audit positive controls on nh3-dev (a drop-in-only root Exec, a
quoted path containing a space, a symlink to a user-owned target) were all
flagged; esh-docker-vm negative control was 0. Probes removed. Cross-model
bug-hunt (heid, Gróa arm + seat) findings folded.
Operator decision. paperless-ngx's database lives on esh-vm-db and is backed
up at the source by that host's fail-closed pg_dumpall. esh-docker-vm's
second copy had failed auth every night since 2026-04-24 behind a WARN. Its
`> paperless.pg_dump` redirect left a 0-byte file in every snapshot
(confirmed in snapshot 6ec9f74f), which looked like a dump but held nothing.
The block was the only consumer of /etc/restic/dbcreds.env, so the creds
loader is gone and the template dbcreds.env.example is deleted. The host
file was moved (not deleted) to /var/lib/restic/repair-20260923/.
Paperless's media volumes are still captured under /var/lib/docker/volumes.
Also fixes ownership. elway's sudo upload does scp-as-user then `sudo mv`,
so the hook deployed at 08:04 landed infra-ops:infra-ops even though root
executes it. Both esh-docker-vm playbooks now chown it back to root and
verify root:root 700.
Verified: the live hook hash matches canonical (e0d3ddcef1bddf43), and
the manual backup saved snapshot decfae71 with 3 staged dumps and no WARN
lines.
Uptime Kuma moved from esh-docker-vm to ana-docker on 2026-09-22. The
pre-backup hook's fallback lookup, `docker ps | grep -E "uptime.kuma"`,
then matched nothing and exited 1. Under set -euo pipefail that aborted the
hook, and resticprofile treats a failed run-before as fatal, so no snapshot
was taken from 2026-09-22 01:00 until this fix (backup-freshness: 54h stale).
The block is removed rather than guarded because there is nothing on this
host left for it to back up. The header now records the invariant the
"blocks only WARN" promise depends on: every optional-service lookup must sit
inside an `if` test or end in `|| true`. The remaining blocks were checked
and all do.
Deployed with playbooks/esh-docker-vm-restic-drop-kuma-block.yaml (the
pre-fix hook is kept in /var/lib/restic/repair-20260923/). The live hook
hash matches the canonical copy (43e6bea8b8569602). The manual backup saved
snapshot 6ec9f74f, and backup-freshness now reports all backups fresh.
TAILSCALE IPv6. `tailscale status` had been reporting, continuously:
2 add route failures; first was: permission denied
adding address fd7a:115c:a1e0::6/128 from tunnel interface: permission denied
with tailscale0 carrying only 100.64.0.6/32 while headscale had assigned it
an IPv6 address it could not use.
Not a capability problem -- tailscaled runs as root with the full bounding
set. /etc/sysctl.conf:59 sets net.ipv6.conf.default.disable_ipv6=1, and
`default` is inherited by NEWLY CREATED interfaces; tailscale0 is created at
daemon start, inherits it, and the kernel returns EPERM for every attempt.
Fixed with a scoped systemd drop-in rather than flipping the global default.
That line carries no comment, but IPv6-off-by-default on a host with ~26
docker bridges reads as deliberate, and changing it would hand IPv6 to every
future bridge as a side effect of fixing Tailscale.
⚠ It must be ExecStartPost, not /etc/sysctl.d. A sysctl.d entry for a
per-interface key is applied at boot, BEFORE tailscale0 exists, and is
silently ignored -- the setting would look present and do nothing.
Also learned: setting the sysctl on the LIVE interface is not enough.
tailscaled only attempts the address at startup or on a netmap change, so
the verify failed for 60s until the daemon was restarted. Restart is part
of the operation, not an afterthought.
Verified: fd7a:115c:a1e0::6/128 present on tailscale0, health clean, mesh
and services (arbo, ytvc) up.
README. It documented the pre-headscale topology as current -- "Reachable
IP: 10.100.79.3 (WireGuard tunnel endpoint)", "No direct LAN access", and a
refresh caveat telling you to bring WG up. That sends anyone triaging this
host to the wrong layer, which is the exact tax the file exists to prevent.
Now: mesh primary at 100.64.0.6, LAN 10.6.110.50, and wg0 documented as
STILL UP with a live peer -- tailscale uses that address as its direct
endpoint, so it is load-bearing, not vestigial.
Recorded with it, because these cost hours tonight and will cost them again:
- Irvine is a TENANCY behind a Fortinet PFI does not control. Its TLS
inspection breaks Tailscale's relay and control channels (41 cert
warnings/week, 4 control-plane episodes in 14 days). Usually invisible
because direct peer paths carry the data. No fix on our side.
- Diagnose reachability with `tailscale ping`, NOT the status output:
headscale said "online" and status said "active, 19.7 GB" while nothing
on the host answered. Both are last-known state; only a round trip is
liveness.
- The ~26 docker bridges make tailscaled report captive portals.
Two stale claims corrected: the hostname rename it called "pending" is done,
and `ollama` is listed as running on :11434 when it is gone -- verified, no
unit file, nothing listening, no process. It is banned fleet-wide.
THE BRIDGE. `beszel-althing` hardcoded a "[Beszel] " subject prefix and a
Beszel hub footer from when Beszel was its only caller. Routing Uptime Kuma
through it unchanged would have delivered Kuma outages labelled [Beszel],
pointing the reader at the wrong dashboard -- an alert that lies about its own
source is worse than no alert.
Now a route registry: /beszel and /kuma, each with its own prefix, footer and
payload parser, because the tools do not agree on a shape (Beszel sends
{title, message}; Kuma sends {heartbeat, monitor, msg}). Generalising cost a
dict; a sibling service would have cost a second unit, a second port and a
second thing to notice had died.
Renamed beszel-althing -> althing-alert-bridge with it. A service named after
one consumer that carries two is the invisible coupling that sends a future
session looking in the wrong place.
⚠ /beszel IS FROZEN and this refactor proves it rather than claiming it. The
three original tests were kept BYTE-UNCHANGED -- including the one asserting
the exact postbox argv -- and deliver() still defaults to the Beszel route so
they exercise it. A new test asserts the Kuma footer never leaks into a Beszel
body or vice versa. Verified live after the rename: a real POST to /beszel
landed as "[Beszel] BRIDGE RENAME CHECK" with the correct hub footer, read back
from the thread rather than trusted from the receipt.
Payload shapes are parsed HERE, not via Kuma's custom-webhook-body feature,
because Kuma's notification config lives in its own database -- and that
database was destroyed and rebuilt from scratch hours ago. Anything living only
in a tool's DB is lost on the next rebuild; format knowledge belongs in git,
next to a test.
parse_kuma also handles the monitorless case. testNotification and cert-expiry
alerts carry no monitor and no heartbeat, and the first cut fabricated "unknown
monitor is ?" from them -- caught by sending a real one and reading the subject,
not by the suite. Fixed, pinned, and the earlier test asserting the bad
behaviour was corrected rather than worked around.
KUMA IS NOW WIRED. scripts/kuma gained notification support and the channel is
in monitors.yaml, seeded BEFORE the monitors and with applyExisting, so a
rebuild restores alerting and not just detection. Ground truth from the DB:
13 of 13 monitors carry the channel.
⚠ A THIRD instance of the same class of bug, worth naming: notifications() is
pushed as `notificationList` at LOGIN ONLY -- there is no event to ask with. The
first cut cleared the captured value before waiting, discarding the only copy it
would ever be sent, then blocked for the full timeout and reported an empty
list. That reads exactly like "no channels configured" and is a lie. Same family
as the getMonitorList ack-vs-push trap, different shape.
End-to-end, both shapes, read back from the inbox:
[Uptime Kuma] Homepage is DOWN + target + board link
[Uptime Kuma] althing (infra-ops) Testing (no fabricated subject)
[Beszel] BRIDGE RENAME CHECK + hub footer, unchanged
ALTHING CHAMBER RETIRED (operator). Three of its four containers had never
started -- created 2026-09-19, StartedAt epoch-zero, 0 restarts -- so :7881
refused, and nothing was watching it. Only its valkey was running, on the
project's own network with no external consumer. Stack, compose/build/conf dirs
and the local image removed; the Homepage card went with the label.
Operator-authorized. build123d 0.12.0 + OCP, numpy 2.4.6, trimesh 5.1.0 in a
3.11 venv, FreeCAD 1.0.0 headless as an AppImage, artifact root on smithy.
The acceptance test is the point and it PASSES: brokkr-smithy-dev shipped a
contrastive control pair whose every property is construction-known, so the
install is checked against arithmetic rather than against a render. Both parts
build, both bounding boxes are exactly 40x40x5, and both volumes match to float
tolerance. These scripts had never been executed by anyone -- build123d is
deliberately absent from the authoring box -- so this was their first run.
FreeCAD is an AppImage BY CHOICE: the distro package drags a GUI/Qt dependency
tree onto a host running twelve GPU and audio services, where an AppImage is
self-contained, pinned and removable with one rm.
Slicer DEFERRED rather than guessed. PrusaSlicer's latest release ships no
Linux AppImage at all, and a guessed release filename had already produced a
4 KB error page in this same session. It is the one item they said could lag.
Two things worth keeping in the header: irv-ml1's ssh config pins the gitea
host to gitea_brokkr_smithy with IdentitiesOnly, whose comment is
irv-ml1-r14-deploy and NOT the arbo key that sorts first in ~/.ssh/*.pub -- I
registered the wrong one first, and the failure names the right one. And
`ssh irv-ml1` is lkraven with no NOPASSWD sudo while `ssh infra-ops@irv-ml1`
has it, so only the artifact-root step needs the other identity.
The deploy key was registered with claude-bot's own token rather than the
operator's site-admin token -- the org grant from earlier today doing real work
on its first use.
Canonical pin bump for the 3.7.0 rollout forseti approved and released
(althing 6db955f, tag v3.7.0). Image built from that tree and pushed as
sha256:df0709b3f532557aebb47d6ece486ae32e851776f732862094608d55eefcd4e7;
previous pin was 3.6.3@sha256:978f8553 and remains the rollback target.
Content-verified rather than tag-verified: `postbox --version` inside the
built image reports 3.7.0 before it was pushed, and again inside the running
container after. A tag is a claim.
Also adds playbooks/nh3-extdev-althing-upgrade.yaml -- a version-parameterised
upgrade for the box a git pull cannot move. The existing
nh3-extdev-althing-v3.yaml is the historical v2->v3 CUTOVER and is left alone
rather than mutated per release; it still carries v2-retirement steps that
document what happened.
Every check in the new playbook asserts what the installed entry point
reports, including one that the herald restarted within the last 5 minutes
rather than surviving from before -- "active" does not mean "running the code
you just installed".
/etc/restic is 0700 root, so before elway 754db4b this step's creates: guard
evaluated unprivileged, always reported the file absent, and the step re-ran
on every invocation. A plain cp -p would then have overwritten the preserved
ORIGINAL with the current post-repair file -- destroying the one thing the
step exists to keep.
Verified it never fired: the preserved copy still holds the original
(md5 7b42324c, mtime 2026-04-21) and differs from the live hook (c79bef9a,
2026-09-15). The guard is fixed now, but a destructive step should not depend
on its guard being right.
esh-vm-workstation is a Windows 11 Parsec/RDP sandbox -- no password, no state
anyone needs recovered (operator ruling, 2026-09-19). Its nightly vzdump has
failed since ~2026-09-06 with "timeout waiting on systemd", and the ESH job has
reported `job errors` every night as a result, so a guest nobody needs backed
up was manufacturing a real job failure daily.
The job uses an explicit `vmid` INCLUDE list, so exclusion means removal from
that list -- there is no `exclude` directive here, which is the same
distinction that makes esh-scale CT 108 invisible in any exclude list. The
reason now lives in the job's own `comment`, and check-backup-freshness.sh
reports 102 under "NOT BACKED UP BY POLICY" rather than going quiet about it.
This is a policy decision, not a fix: the systemd-scope failure on stopped
guests is NOT diagnosed, merely no longer reached on this node. A stopped
guest that DOES matter would hit the same wall.
The playbook carries two hard-won warnings in its header. Read the job through
the API, never `grep -A<n>` on jobs.cfg -- `vmid` sorts last in the block, so
adding the comment line pushed it outside the window and every condition
silently inverted. And use YAML block scalars for the JSON-extraction
expressions; double-quoted ones did not survive YAML unescaping plus elway's
bash -c quoting.
Verified: vmid 100,101,103,104,105,106,107; the other seven guests still
covered; job still enabled; a rerun skips the change rather than repeating it.
Picks up uncommitted searxng changes left by another session and makes them
truthful rather than committing them as written. The stack itself verifies
clean: canonical and live are byte-identical for both compose.yaml and
searxng-settings.yml, the container is running with zero restarts, and live
queries return 51-54 results from 5-6 engines with braveapi contributing 20
each time.
compose.yaml gains SEARXNG_BRAVE_API_KEY, which NOTHING READS. It was added on
the belief that settings.yml could pull it via `!ENV SEARXNG_BRAVE_API_KEY`;
this build has no !ENV YAML constructor, so that attempt made the file
unparseable and crash-looped the container ten times with fleet search down.
The comment claiming the variable is "consumed by settings.yml" is replaced
with what is actually true. The variable is kept, unused, in case upstream ever
gains env interpolation — a comment that lies is worse than a variable that
does nothing.
The Tier A playbook is marked superseded FOR THE SETTINGS FILE ONLY, and
scoped deliberately: its v4 design uploads a settings file carrying the !ENV
tag, which would re-break the container, so settings deployment goes through
scripts/deploy-stack.sh like every other stack. Its .env merge and
up-d-not-restart steps remain useful, as do its two warnings recording real
bugs it hit — a wholesale .env overwrite that clobbered SEARXNG_SECRET, and a
sed that inserted literal backslash-n into compose.yaml. An unscoped
"superseded" banner would have buried those; that failure mode cost an outage
earlier today.
Also folds in the regenerated graphify report.
Measured 3/3 probes: duckduckgo/startpage CAPTCHA, brave rate-ban, wikidata
403 from this egress. Mojeek tried and also 403'd. Notes on keyed-engine
path to restore breadth recorded in the settings header.
Adds playbooks/update-stack-image.yaml — pull the newest image for one
compose stack service and recreate it, with a verify phase that asserts
the container's image id equals what the tag now resolves to rather than
trusting a 'Up' line from docker ps.
Scoped to a single service on purpose: the recreate is 'up -d <service>',
never a bare 'up -d', which would recreate every service in the project.
Go template format strings are written bare; elway's {{ identifier }}
substitution leaves them alone, but {{end}} / {{else}} would match and
die as undefined variables, so the health read uses {{json .State.Health}}
instead of an if/else.
First use: drawio on esh-docker-vm, 28.1.2 -> 31.4.6.
Operator ratified four conventions on 2026-09-14. docs/pfi/fleet-conventions.md
is the pin; playbooks/audit-host-conventions.yaml is its instrument.
Pinned, verified free on all eight surveyed hosts (dynamically-allocated
system accounts cluster in 989-999 and descend, so 800-899 is safe):
800-849 svc-* service accounts
850 infra-ops uid+gid
851 docker gid
852-899 reserved for fleet-wide groups
1000 the human account (vh)
Deliberately a pin for NEW hosts, not a migration mandate. The UID drift
(infra-ops is 1001/1002/1003/2001) is tolerable because there is no central
identity anywhere and a UID only has to agree where files cross hosts. They
do on /mnt/smithy — but that export is owned by Synology UIDs that resolve on
neither host and is 0777 throughout, so cross-host sharing works today
BECAUSE permissions are wide open. Aligning UIDs does not fix something
broken; it earns the right to drop that 777. Recorded as such rather than as
an urgent defect.
The audit playbook reports and never enforces, so a standard cannot quietly
become a flag day. Verified against nh3-dev, ana-docker, corviduo-dev and
nh3-extdev; it immediately surfaced two things the survey had missed —
llmuser holds sudo AND docker on ana-docker, and seven stacks on corviduo-dev
run from outside /opt/docker/compose (three under /home/vh, four under /opt,
including the three CI/CD-driven Worldtree deployments that must not be
moved).
Also supersedes the CLAUDE.md posture that made corviduo-dev the one host
excluded from fleet normalisation: the operator ruled all ops on it belong to
infra-ops. Its application layer stays CI/CD-owned.
Operator ruling: root:docker, not a personal username and not a new admin
account. lkraven is one of three names he uses, so baking it into shared
infrastructure guarantees a stale owner later; a dedicated deploy account
buys nothing the existing docker group doesn't, since that group already
exists on every host holding exactly lkraven + infra-ops.
Applied to nh3-dev, nh3-docker, esh-docker-vm, irv-ml1, ana-docker. All
five now 2775 root:docker on /opt/docker and /opt/docker/compose. Clears
the 0777 on nh3-docker and ana-docker. 55 stack .env files normalized to
root:docker 0640, tightening 43 world-readable ones and opening 31 that
were readable by only one of the two deploy identities. No containers
bounced — inode metadata only, and .env is read at compose up.
Deliberately not a recursive chmod. Three acme.json files and an ssh
private key are mode 0600 and traefik/ssh refuse to start if that widens,
which would have been a delayed failure surfacing at the next restart
rather than now. Protection is both mode-based (0600/0400 untouched) and
name-based (acme.json, *.key, *.pem, *.pfx, id_*); modes are symbolic so
the 53 executable files in these trees keep their exec bit.
Two defects found and fixed mid-rollout. The name list was initially
reported but not enforced, so a .key already at 0644 on esh-docker-vm was
widened to 0664 — reverted, and the list is now enforced in the chgrp and
widening steps. And the exec-bit verify asserted every .sh is executable,
which was never true and false-FAILED irv-ml1; it now compares the
executable-file count against a recorded baseline.
The restart loop runs as the deploy identity, not root, and a stack .env is
allowed to be root-owned 0600. compose bails on the unreadable file before
doing anything, so the stack was reported FAILED while restart=unless-stopped
had already brought it back healthy — a false failure, which is worse than a
quiet one because it trains readers to skim the failure lines.
Retry under sudo -n before calling it a failure, and print compose's own
output either way. Verified on nh3-dev against beszel: plain attempt rc=1
'open /opt/docker/compose/beszel/.env: permission denied', sudo retry rc=0
'Container beszel-agent Started', container back to healthy. The happy path
is unchanged — the sudo attempt only fires after a failure.
Also record that tts-dev migrated talk from ~/talk into
/opt/docker/compose/talk, which removes the one stack on this host that was
invisible to anything walking that path.
homepage-regroup, mog-sec-move-to-gpu0, pull-hf-repo and serve-qwen3.5-122b
all carried runnable 'scripts/elway ana-ml2 --playbook ...' instructions or
the old 10.250.50.54 address. Each would fail today against a dead name and a
dead IP, so these are corrections rather than cosmetics. homepage-regroup is
renamed to match; the other three keep their names, which never carried the
host.
A zsh glob failure in the staging step of 91bda3c meant the five renamed
playbooks went in as pure renames with their bodies still saying ana-ml2 and
10.250.50.54, and the homepage docker.yaml direct-connection edit from 17586ab
did not stage either. Same content, just the half that got dropped.
The box is physically at Fountain Valley, renamed, renumbered onto 10.251/16,
and serving inference again. This lands the repo half of that.
Host: hostname ana-ml2 -> fv-ml1, pinned to 10.251.50.54 by a dnsmasq
reservation so the address the runbook, DNS and LiteLLM all assume is the
address it actually has. Its headscale node is renamed too.
The sweep ran from scripts/fv-ml1-rename-sweep.sh, whose allowlist is the
reason this diff touches current-state files and not the record. Dated
persistent-memory entries, archival-memory and incident notes still say
ana-ml2 in 31 and 62 places respectively, because that is what the box was
when those things happened. Rewriting them would make the history lie.
LiteLLM was the load-bearing piece and needed more than the api_base sed the
runbook describes. Twenty api_base entries repointed, but a grep-and-verify
pass also caught a LIVE pass_through_endpoints target for the scalar-judge
reward route still on the old address -- an api_base-only substitution would
have left it dead. Four prose references describing current state were
repointed as well; one historical note recording where a hand-test was run
is deliberately left pointing at 10.250.50.54.
Two facts in the server tables were wrong and are corrected here. The site is
Fountain Valley, not Anaheim. And the box has FOUR RTX PRO 6000 Blackwell
Max-Q, not two -- verified by nvidia-smi -L and independently by PCI
enumeration of four GB202GL devices. That is 391 GB of VRAM rather than 196,
which changes what fits on it.
DNS: fv-ml1, fv-ml1-bmc and fv-gw added under the fv site via the piggyback
approach, scriberr re-homed, and the ana-ml2 records removed. Applied to all
three resolvers. The BMC record carries a warning that its 802.1q VLAN tag
must stay disabled -- it shipped tagging VLAN 250 into an untagged port,
which made it invisible to every network-side diagnostic and is the reason
it appeared dead through several cable changes.
Verified end to end: summarizer and sec both answer through the Anaheim
gateway across the mesh to FV seats on different ports.
Beszel agents are installed and verified across the fleet but the artifacts that
produced them were never committed, so the deployment existed only on the hosts.
Adds the per-host agent environment files (PORT, NICS, EXTRA_FILESYSTEMS and the
hub's PUBLIC key), the systemd unit, the guest install script, the Synology
compose, and the elway playbooks for native, guest-stage, guest-install and
Synology paths. The two dated memory detail files covering the priority-1 and
priority-2 waves ship alongside, per the convention that memory lands with the
work it describes.
No credentials here. The KEY= value in every host env is the Beszel hub's public
ed25519 key, identical across all nine and public by design; the agent README
says so explicitly. The nh3-nas sudo password referenced in the runbook prose
lives in Vaultwarden and the helper scripts named there never contained it.
⚠ Overlapping VMIDs across hypervisors are a standing trap and are recorded in
the priority-2 notes: pfi-pve 105=postgres and 100=pbs-ana, nh3-pve 105=pbs-nh3.
⚠ PBS-NH3's export was ~75.5% used at capture; resource checks are not job
success monitoring and should not be read as such.
pfi-gx10 moved from desk Wi-Fi (10.100.10.226, VLAN 10) to the rack on
10.100.50.60 (nh3-servers, VLAN 50), reachable as pfi-gx10.nh3.internal.
The address is a DHCP RESERVATION on the UDM, not a host static. Operator
ruling during the move, and the better design: a host-side static works
until the box moves and is then a stale netplan file on a machine whose
address you no longer know. The pre-written playbook wrote a host static;
it is kept for its safety ordering and annotated as retired.
The port arrived on the native VLAN, not the server VLAN, so switch port 22
was repointed first. port_overrides is a whole-array PUT — two unrelated
overrides on ports 21 and 23 were read, preserved and written back, with the
original array backed up to a file before the change.
Wi-Fi stayed up as the escape hatch until the wired path was proven from
outside, and was downed last as its own step. The step worth keeping: while
Wi-Fi was up, traffic to nh3-dev still preferred wlP9s9 because that
interface sits directly on the userland subnet, so reachability proved
nothing about the wired path. `ping -I enP7s7` across the VLAN boundary is
what actually settled it before the hatch came down.
Triton builds its CUDA-utils shim with gcc the first time a kernel runs,
and needs Python.h to do it. Without python3-dev the box looks entirely
healthy: torch imports, the 49 GB base loads, LoRA attaches with the
right parameter count, and then the first training step dies with a bare
CalledProcessError naming a gcc invocation and an exit code.
The real message -- "fatal error: Python.h: No such file or directory" --
is discarded, because Triton sends the compiler's stdout to DEVNULL. It
cost a probe run and a model load to find something a one-line manual
compile answered immediately.
Same shape as the dots-tts container needing a C compiler at runtime: a
JIT dependency invisible at install time that only surfaces under load.
The verify step runs the compile rather than checking the package is
present, because `dpkg -l python3-dev` would pass while the compile still
failed on a missing library path or header directory.
The userspace half is done and needed no root. This covers what does:
NOPASSWD sudo for infra-ops (ending the gx10 fleet exception), the docker
group, the CUDA toolkit for nvcc, and the NVIDIA Container Toolkit wired
into dockerd.
Unrun -- it needs one interactive invocation to supply the sudo password,
because no gx10 credential is vaulted and elway prompts via getpass. After
its first step lands, the box stops being an exception and later runs are
unattended.
Guards worth noting: the sudoers drop-in is validated with visudo -cf
before install, since a malformed one locks every sudo user out of a box
sitting on a desk with no out-of-band access; and cuda-toolkit is
installed rather than the cuda metapackage, so the working 580.173.02
driver is not replaced.
Target settled: nh3-servers VLAN 50, static 10.100.50.60. Clear of the four
existing statics and below the .150 DHCP pool where fleet statics live.
The playbook never leaves itself one path back. Wi-Fi stays up throughout while
the wired interface is configured beside it; the new address is verified from
outside before anything is torn down, and Wi-Fi teardown is explicitly a
separate later change. A botched netplan therefore costs a retry over Wi-Fi
rather than a trip to the rack — which is what substitutes for 'netplan try',
whose interactive rollback needs a TTY that elway cannot provide.
Two preconditions are asserted as steps rather than assumed: the interface must
have carrier (writing a static config for a dead NIC and reporting success is
the failure this avoids), and its MAC must match, since interface names can
renumber across kernels but MACs do not.
Requires nothing from the operator beyond racking the box. The wired NIC has a
distinct MAC from the Wi-Fi one, so the post-move address and switch port are
both discoverable from the UDM rather than needing to be relayed.
NVIDIA GB10, aarch64, 121 GB unified, sm_121. Ships booting to graphical.target
with GDM and GNOME Remote Desktop running.
playbooks/gx10-headless.yaml sets multi-user.target, stops gnome-remote-desktop,
masks the sleep/suspend/hibernate targets, makes logind ignore lid and idle, and
adds sshd keepalives so a stalled link does not kill a long-running job.
Two things learned the hard way and recorded in the playbook:
- gdm is a STATIC unit on Ubuntu, pulled in by display-manager.service and never
'enabled'. A guard of always skips, and a verify
written the same way passes while the desktop is still running. Both now test
is-active. The first run reported six green verifies having not stopped gdm.
- elway's --sudo applies only to ad-hoc --shell/--upload. Playbook steps run as
the connecting user and must carry their own sudo; connect as infra-ops.
The playbook refuses to stop the display manager while a seat session is held,
overridable with --var force_dm_stop=true. Networking is deliberately out of
scope: the box is on a desk on Wi-Fi with a temporary DHCP lease and no ethernet
carrier, and belongs to the rack-install change.
Deploys althing-core 3.1.1 to nh3-extdev. Both markers present on both
boxes; the warning verified behaviourally in four conditions rather than
by grep alone -- mismatched inherited handle warns, matching handle
silent, explicit --handle silent, unlaunched directory silent, and the
warning precedes the output it is about.
The release's own verification line says
`grep -c handles_launched_at dev_launch.py # 2+`. The real count there
is 1, the definition; the other two occurrences are in postbox.py. The
installed tree is byte-identical to the repo at the pushed tag, so the
instruction is wrong rather than the install.
This playbook matches on presence via grep -q, so it passed. Had it
asserted the stated count it would have reported FAILED on a perfect
deploy -- a verification instruction that fails on correct input, which
is the same false-negative this file has now produced three times in
different costumes. Recorded above the variable so the next bump does
not reintroduce a count.
Deploys althing-core 3.1.0 to nh3-extdev and restarts the herald.
Verified by content on both boxes: POST_OFFICE_HINT 0 -> 4 in
post_office_herald.py and resolve_post_office 0 -> 3 in dev_launch.py,
dist-info 3.0.3 -> 3.1.0.
The check took one file:marker pair. 3.1.0 changed two files, so a
single pair would have asserted half a release and passed -- the same
half-passing-silently shape as the version-string check it replaced two
releases ago, one level up. It now takes a space-separated list, reports
each pair individually, and fails if any is missing. Every release's
markers so far are recorded above the variable so the next bump is a
lookup rather than an archaeology exercise.
Also verified the behaviour the release exists for rather than just its
markers. The herald writes its address to $ALTHING_ROOT/post-office and
dev_launch.resolve_post_office reads it when the variable is unset:
env unset -> http://10.100.50.40:8390
env set -> the env value, which wins
env set to blank -> the file, because blank counts as unset
My first attempt tested this through postbox, which still requires the
variable and reported "no post office address is configured" -- correct
behaviour that looked like a failed deploy. dev-launch is the reader,
not postbox.
pypotrace is an sdist that compiles at install time, so every machine
and every CI runner resolving it needs these headers first. That makes
it a recurring per-box action rather than the one-off it arrived as.
Two things learned installing it on nh3-dev are recorded here rather
than left in an althing thread, at forseti's suggestion, because a
thread is not where the next person looks:
Only libagg is a pkg-config consumer. potrace ships no .pc file and is
found via potracelib.h directly, so `pkg-config --exists potrace`
returns false on a correctly configured box. It looks exactly like the
cause and never is.
libagg's pkg-config modversion is 2.7.0 while its Debian package version
is 1:2.6.1-r134. Comparing those two numbers convinces you the wrong
package is installed.
The verify phase asserts the geometry, not the import: a square must
come back as one curve of four CornerSegments. An extension linked
against the wrong thing can import cleanly and return nonsense, so a
successful build is not evidence the module works.
Getting the build probe to run took three passes and the reason is worth
keeping. uv is not on a non-interactive ssh PATH; it is in a different
place on each box; and on nh3-dev it sits inside a 0700 home, so even
the correct absolute path fails `test -x` for the ssh user because the
directory cannot be traversed. The headers are system-wide and root's
business, but the build check is a developer action and has to run as
the user who owns the toolchain.
Deploys althing-core 3.0.3 to nh3-extdev and restarts the herald.
Verified by content on both boxes: PANE_SETTLE_S 0 -> 2 occurrences,
value 0.3, dist-info 3.0.1 -> 3.0.3.
The content check was hardcoded to the 3.0.1 markers, so from the next
release onward it would have kept passing while asserting nothing about
what had just been installed -- a check that verifies the previous
release is indistinguishable from one that works. It now takes the
marker and file as variables, bumped per release, with both releases'
markers recorded so the pattern is obvious rather than folklore.
That is the same defect class as the install step gated on `postbox` not
existing, which this playbook carried until last round: a guard written
correctly for the first run and never re-read on the second.
The post office container was not touched. forseti established by
import graph that althing/post_office/* imports neither changed module
-- the fix is in reach_pane, which is herald code -- and the container
has been up two hours across both herald restarts.
Reinstalls althing-core on nh3-extdev for 3.0.1 (the pane-route fix) and
restarts the herald. Both boxes verified BY CONTENT rather than by
version string -- forseti's own checks, grep for _PANE_ID and _live_pid,
because a dist-info directory records what was installed, not what the
files contain. Both went 0 -> 3 and 0 -> 2.
Two bugs in the playbook this run exposed, both of which only appear on
the second use:
The install step was gated on `postbox` not existing. That guard was
correct for the cutover, when postbox genuinely was absent, and wrong
for every release after it -- postbox exists now, so a version bump
would have silently skipped the install and the playbook would have
reported success having done nothing. `--force` already makes the
reinstall idempotent, so the guard bought nothing and cost correctness.
The post_office variable still pointed at nh3-dev, three hours after the
post office moved to nh3-docker. It failed in the verify rather than at
install time, which reads as a broken deploy rather than as a stale
constant. Worth noting the failure message was the outage semantics
working exactly as designed: "This is an outage, not an answer: do not
treat it as 'no mail'."
Two operator-authorised changes on the same afternoon.
## althing v3 (U9b flag day, one-way, no rollback)
The post office replaced the v2 P2P bus on nh3-dev and nh3-extdev.
One container is the only stateful component; heralds are one per box
and dial out; waiters are one per session. Every v2 command was deleted
rather than deprecated, so a script calling althing-cli now fails loudly
instead of silently talking to nothing.
73 handles seeded from the v2 CLI, which is authoritative over the v2
database's 91 agent rows -- the extra 18 are superseded names, a typo,
an underscore variant, and two machine-qualified handles that v3 makes
a category error. Verified by set difference in both directions rather
than by counting; a peer's "72 rendered" was a line-count artifact.
Deleted 5,043 orphaned wake FIFOs. The reason there were five thousand
is that v2 named them per session with the PID and never reaped them;
v3 names them per handle, so the leak is bounded by construction. That
is a fix in v3, not a cleanup we performed.
nh3-extdev needed its own path: althing lives there as a system wheel
under /opt/uv-tools with entry points in /usr/local/bin, its daemons
were system units rather than user units, and uv is not on the login
user's PATH. Captured as a rerunnable playbook rather than shell
history.
The v2 database is left inert on disk. There is no import path and none
was improvised.
## sec onto GPU0
GPU1 carries the five resident fleet seats and had ~28 GB free against
the ~51 GB this seat reserves, so it could not start there at all. GPU0
has been idle since run 3c was stopped. The compose header, the GPU pin
default and the homepage label all carried the old card number and are
corrected together -- a label that names the wrong GPU is a record that
lies about where the work runs.
Both playbooks carry verify phases that assert effective state. Two of
those verifies failed on green deployments while I was writing them:
one used a Go template that collided with the runner's own {{ }}
substitution, one omitted --handle so it failed on identity rather than
reachability. Both are fixed with the reason recorded inline, because a
verify that reports FAILED on a working system trains you to ignore it.
forseti asked for journald kernel persistence plus sysstat, on the
premise that nh3-dev's three OOM events in 14 days left no evidence.
The premise was wrong. journald has been persistent all along: 15,068
kernel entries in the 82-day previous boot and 351 OOM records across
retained boots, full task tables included. `journalctl -b -1 -k`
returned one entry because it ran as a user in neither adm nor
systemd-journal, and journalctl shows only your own messages in that
case. The same artifact produced the "journal stops at 05:36:08 with no
shutdown sequence" claim -- the true boot -1 end is 05:47:04 with OOM
kills logged at 05:38, 05:40 and 05:42.
So the fix for "no evidence" is a group membership, not a logging
change: usermod -aG adm lkraven, which is the group Debian's journald
ACL names explicitly.
With the journal readable the attribution is already in it. The
versioned Claude Code binary lives at .local/share/claude/versions/,
so OOM victims named 2.1.220 / 2.1.177 / 2.1.168 are CC sessions, as
are those named claude. Every one of the twelve largest resident
processes ever recorded on this box is a CC session, topping out at
18.4 GB. Everything else killed is 30-55 MB collateral, which clears
the althing daemons by measurement rather than by their own sampling.
sysstat and atop are added because the journal records the moment of
the kill, not the ramp, and names the victim rather than the winner.
atop was not requested and is the one that matters: with a dozen panes
open, only a per-process timeseries says which session was growing.
Not done: a cgroup cap on CC sessions. It is the real mitigation and it
would kill long-running sessions mid-work, so it goes to the operator.
Operator call: rather than train beside gen on GPU0, move gen to GPU1 and
stand sec down for the night, so the tune gets a whole 95.60 GiB card and
the fleet's general seat never goes dark beyond its own restart.
Order is load-bearing in both directions and the playbooks enforce it.
gen runs at --gpu-memory-utilization 0.43, which vLLM reads as a fraction
of TOTAL card memory: 42,091 MiB must be FREE at startup or the engine
refuses to boot. GPU1 has 19,446 MiB free while mog-sec is up, so
recreating gen onto GPU1 first would take the main seat down and leave it
down. mog-sec stops first and a hard gate checks the freed memory before
gen is touched. The close playbook mirrors it: gen must vacate GPU1
before mog-sec starts, since mog-sec needs 50,901 MiB of its own.
Close opens with a gate that refuses to run while a process is still
resident on GPU0, so it cannot evict a training run mid-flight.
Override with --var allow_busy_gpu0=true.
Three defects found and fixed while landing this, all worth keeping:
- Verifying GPU residency via `docker inspect --format {{.State.Pid}}`
never matches. vLLM V1 runs EngineCore as a child of the container's
pid 1, and it is the child that holds the memory and that nvidia-smi
reports. Match by cgroup instead.
- A step's `sudo: true` does not extend to its when/creates/changed_when
guards, which run as the login user. The root-only .env made an
unsudo'd grep exit 2, so the GPU-id flip SILENTLY SKIPPED. The
effective-value assert is what caught it.
- That assert originally grepped the config YAML for -\s*'?1'? and failed
against compose's double-quoted `- "1"`. Parse the JSON with jq; an
assert that fails for the wrong reason is worse than no assert.
elway must be invoked as infra-ops@10.250.50.54 rather than the ana-ml2
ssh-target, which resolves to lkraven and has no NOPASSWD sudo.
The board mixed tools with endpoints. A vLLM seat whose href is a /docs page
sat in the same band as ComfyUI; the MQTT broker and the RustDesk relay, which
have no page at all, sat in Apps; and `Service Networking` was thirteen members
spanning three AdGuards, five Dockges, two Traefiks and four headless agents.
Every group is now one of two kinds and they never mix. TOOLS are expanded and
sit at the top of their tab. ENDPOINTS — an API, a broker, a background agent,
an href that is /docs or /ping or nothing — carry `initiallyCollapsed: true`
and sit at the bottom. Collapsed is not hidden: the eyebrow and its rule still
render, so the tab still says the thing exists and one click expands it.
A second rule fell out of the same pass and now shapes the group boundaries: a
group's members should all carry a widget or none should. A stat strip makes a
card ~50px taller, so one widget card in a row of plain ones opens a void under
the plain ones. That is why AdGuard and Traefik get their own groups rather
than sharing one with Dockge, and it is most of why the old Service Networking
band looked broken. AdGuard (ANA) was the last short card in its row and now
carries the same query/blocked/latency strip as its two siblings — one
infra-ops AdGuard login authenticates against all three instances, verified
against each; it lives in that stack's .env on the host and is vaulted.
The sixteen GPU-backed model seats were deliberately NOT relabelled.
`homepage.group` is read at container creation, so clearer names for
`AI - Inference` and friends would have cost a recreate on six vLLM seats, four
eval seats and four TTS engines — multi-minute model reloads on endpoints peers
reach through the gateway. Order plus `initiallyCollapsed` buys the same
separation for nothing, so those names stay as they are on purpose.
28 containers that ARE cheap to bounce were relabelled, across five hosts, via
rerunnable elway playbooks. Their label steps are gated on the old value still
being present, so a second run reports skipped rather than churning. Two verify
steps were wrong on first contact and are fixed with the reason recorded: the
traefik check raced its own recreate, and asserting a model seat is "running"
cannot answer "did I bounce it" when a seat may be legitimately stopped —
container age can, and now does.
The canonical stacks/ tree was synced to the deployed labels afterwards, so
intent and reality agree again on all fourteen tracked stacks.
Also documents the real nature of the post-recreate blank dashboard, which cost
~25 minutes here and an hour on 2026-08-19. `initialSettings":{}` in the served
HTML is the catch branch of the page's data loader, not a warm-up and not a
cache — and the error can vanish entirely, because the logger is assigned inside
the same try and the catch only logs if the logger exists. Ruled out by
measurement this time: all four API routes return 200 with correct content while
the page serves {}, and the previous known-good settings.yaml reproduces it
identically. The README now carries the one-command test and the next lead.
Before/after, all four tabs: http://10.100.10.50:8090/b/homepage-relayout/
Eight stacks on ana-docker kept secret-bearing compose .env files at mode
0644 -- readable by every local account on a box with four interactive
users. Verified as a real exposure, not a theoretical one, by reading one
as `nobody`. Six other stacks already used 0600, so this converges on the
existing house pattern rather than inventing one.
Swept (all verified mode 600, unreadable by nobody, compose config still
rendering as the deploy user, container untouched):
vaultwarden traefik beszel gitea-runner
miniflux news-digest searxng vor
Post-sweep the host has zero secret-bearing .env readable by nobody.
Checked before writing the playbook, and re-checked by the playbook itself
so a bad case cannot be swept in by accident:
- every target is owned by lkraven, the deploy user, so 0600 preserves the
deploy path (asserted by rendering compose config AS lkraven, not root --
checking the mode proves the bits changed, only rendering as the deploy
user proves the next deploy still resolves its variables)
- none is bind-mounted into a container. vaultwarden looked like it was;
that `- .env` is under `env_file:`, not `volumes:`. A genuinely
bind-mounted .env is read by the container UID and 0600 could break it,
so the playbook refuses that case.
Two gotchas recorded in the file:
- elway's variable regex eats any bare identifier in braces, so a docker
--format Go template containing {{end}} or {{println}} fails as an
undefined variable. Grep the raw inspect JSON instead.
- the `ana-docker` ssh alias resolves to lkraven, who needs a sudo
password; run elway against infra-ops@10.250.50.70 for NOPASSWD.
Two defects the worldtree-pinned case exposed in the guard written an hour
ago.
1. The guard compared the container's .Config.Image STRING against the tag
being pinned. That string is only the tag the container was CREATED from,
which can differ from what it actually runs: worldtree-pinned was created
from `:latest` back when that tag pointed at 446e5807, and `:latest` has
since moved to b19afd71d7cc. So the guard refused an instance whose
pinning was correct and necessary. Now it resolves the target tag to an
image ID and compares that against the running image ID -- asserting the
thing actually cared about, that this tag names the bytes now running.
It also fails closed when no such local tag exists.
2. The sed step reported CHANGED unconditionally, so a re-run on an
already-pinned instance claimed work it had not done. Gated behind a
`when:` that skips when the line is already correct; a second run on demo
now reports "2 ok, 0 changed, 2 skipped / overall: OK".
Applied to worldtree-pinned under worldtree-dev authorization. That instance
needed a `docker tag` first -- its image was DANGLING (no repo tags, kept
alive only by the running container), so the fleet's frozen reference was one
`docker rm` from garbage collection. Tagged as :446e5807bf43, then pinned.
All three instances now render a SHA with no floating tag anywhere:
worldtree -> :ae88a057c0ed
worldtree-personal -> :f63529168c13
worldtree-pinned -> :446e5807bf43
Nothing restarted -- pinned still Up 3 months, its start time unchanged.
Both corviduo-dev Worldtree instances carried
`WORLDTREE_IMAGE=.../worldtree:latest` in their .env while running
SHA-tagged images built the same day. The local :latest tag pointed at
b19afd71d7cc, built 2026-06-14 -- 69 days stale. Any `docker compose up`
on either instance, by anyone, silently downgraded that service by 69
days: the same footgun behind the 2026-06-15 outage.
Applied under worldtree-dev authorization (Worldtree #410):
demo -> gitea.phasefinal.com/vh/worldtree:ae88a057c0ed
personal -> gitea.phasefinal.com/vh/worldtree:f63529168c13
Both runs verified compose config resolves every service to the pinned
SHA with no :latest remaining, and that no container restarted. The edit
is inert by design -- it changes what the NEXT recreate pulls.
The playbook guards against pinning a SHA that is not the one actually
running, which would re-arm the same hazard with a different image.
Also records the worldtree-pinned case, deliberately NOT changed here:
that instance runs a DANGLING image (sha256:446e5807, no repo tags),
kept alive only by the running container. It has no tag to pin to, so it
needs `docker tag` first -- and an untagged image referenced only by a
container is one `docker rm` from garbage collection, which for the
fleet's frozen reference instance is worth fixing on its own merits.
Out of the authorized scope (demo + personal); raised with worldtree-dev.
Stopgap: the durable fix is the deploy workflow stamping the deployed SHA
into .env at each deploy, queued repo-side with worldtree-dev.
vh