nh3-pve and esh-pve are the same Minisforum MS-01 (BIOS AHWSA.1.17). With a
card in the x16 slot its root port takes bus 01 and every NIC moves down a
bus (measured on esh-pve), so predictable names change (enp2s0f0np0 ->
enp3s0f0np0 etc.) and vmbr0 would boot with no uplink. systemd .link files
now pin all NICs by MAC, baked into every initramfs and synced to the ESP;
udev confirms the files apply. The AMT-capable I226-LM (enp88s0) leaves
vmbr0's bridge-ports in the file (next boot), so cabling it for AMT cannot
loop the STP-less bridge.
Also: documented the NanoKVM (https://10.100.250.171) as nh3-pve's console
OOB and that AMT is not wired; nh3-dev's Beszel agent no longer binds NAS
shares (it died on the last NH3 cold start); post-boot checklist in
persistent-memory.
- Beszel: NVIDIA agent (stacks/beszel hosts/esh-ml1.yaml), hub system
registered with Status/Disk/CPU/Memory/Temperature alerts; GPU samples
verified (RTX 2000E Ada util, VRAM, power, temp).
- Uptime Kuma: health monitors for the TEI embed (:8001) and rerank (:8013)
services — the sole backends behind the gateway, so the one exception to
"seats are out of Kuma's lane". Reward seat excluded (no consumer).
- Homepage: dockerd on tcp/2375 bound to 10.0.50.80 (playbook step);
esh-ml1-docker added to docker.yaml.
- Dozzle: agent v10.4.1 on esh-ml1. The hub's DOZZLE_REMOTE_AGENT still
named ana-ml2's 10.250.50.54 and irv-ml1's retired 10.100.79.3; repointed
to 10.251.50.54 / 10.6.110.50 (fv-ml1 and irv-ml1 logs were missing).
- Documented nh3-dev's Beszel agent failing on NH3 cold start (bind of an
automounted NAS share); revived by hand, fix still open.
- Audit: Skywork-Reward-V2-Llama-3.1-8B is still #1 of 188 on AllenAI's
RewardBench 2 per-sample results; no Skywork V3; the -40M sibling is
vendor-marked experimental. Our AWQ W4A16 quant: 0.847 vs published bf16
0.860 on a 150-prompt sample (within +/-2.9 pt SE), 96.2% pairwise
agreement. Double BOS from vLLM on pre-templated text costs a further
~2.7 pts; callers must send add_special_tokens=false.
- No working consumer: 0 requests since 2026-09-13; Worldtree Domari points
at a dead IP with a non-vLLM schema (reported to worldtree-dev).
- Move: sha256-identical model copy; vLLM v0.24.0 on esh-ml1 :8003 at 0.55
util (KV 1.30x of a 16k request). Parity vs fv-ml1: 149/150 verdicts,
99.8% pairwise signs, raw |delta| median 0.049.
- Gateway /scalar-judge passthrough -> 10.0.50.80:8003; fv-ml1 vllm-reward
removed (~10.2 GB freed on GPU 1). stacks/vllm now holds only vllm-coder.
Prime, 2026-09-25: TEI serves embedding + reranking for esh-ml1 and the fleet
from now on; fv-ml1 retires both once esh-ml1 is up.
- stacks/embed-rerank: vLLM -> TEI 1.9.4 (89- Ada build), same ports
8001/8013, fail-closed truncation (--auto-truncate false; embed
--max-batch-tokens 32768).
- litellm: qwen3-embedding -> esh-ml1 only (hosted_vllm/, unchanged address);
reranker -> huggingface/ provider at :8013 (hosted_vllm/ 422s on TEI's
`texts` body). DB alias reranker-a3-bge-v2-m3 patched to the same target.
- Verified via the gateway against the retiring fv-ml1 seats: embed cosine
median 0.999927 (n=203); rerank top-1/top-3 29/30.
- stacks/vllm: vllm-embed and vllm-rerank-a3 removed (containers retired on
fv-ml1, GPU 1 freed ~6.1 GB); reward + coder unchanged.
- Bake-off record moved to docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md;
CLAUDE.md gains the TEI convention.
esh-pve: NVIDIA 580.178.04 (open modules, DKMS) installed on the host from
NVIDIA's .run and loaded live, no reboot. nvidia-persistenced unit creates the
device nodes before pve-guests; the T400's vfio-pci ids and `blacklist nvidia`
retired. playbooks/esh-pve-nvidia-host.yaml.
esh-ml1: CT 110, unprivileged Debian 12, 10.0.50.80, GPU nodes via devN,
NVIDIA userspace from the same .run (--no-kernel-modules), docker-ce +
nvidia-container-toolkit (no-cgroups). playbooks/esh-ml1-lxc.yaml. Not in the
vzdump job on purpose. DNS esh-ml1.esh.internal.
stacks/embed-rerank: Qwen3-Embedding-0.6B :8001 + bge-reranker-v2-m3 :8013 on
the same vLLM v0.24.0 digest and flags as fv-ml1. Measured parity: embed
cosine FV-vs-ESH median 0.999908 (min 0.999772), inside both self-noise
floors; rerank max |delta| 0.000145 vs floor 0.000181, identical ranking.
litellm: qwen3-embedding and reranker gain an esh-ml1 deployment at order 2
behind fv-ml1 (order 1). Order fallback proven with throwaway groups:
refused primary +0.15 s, host-down primary ~18.7 s per call, dead-only 500.
Also: repaired the DB-only alias reranker-a3-bge-v2-m3, dead since the
fv-ml1 relocation (still named 10.250.50.54); documented the third
unkillable homepage wedge on esh-docker-vm.
After the 2026-09-24 NH3 outage pfi-gx10 stayed off. It has no BMC, and the
ASUS firmware defaults to staying off after AC loss. With Prime's approval,
bytes 3-4 of the UEFI Setup variable were changed from 0x1d,0x00 to
0x1c,0x01, a community recipe that matches factory power-on units. The
readback shows exactly those two bytes changed. The original 85-byte raw
variable (attr header included) is committed here and also kept on the box.
The README records that an OS shutdown is not a test of this setting. It
stays off by design, and two such "tests" were invalid. The only valid test
is pulling and restoring AC, which Prime will do on 2026-09-25. It also
records the revert procedure.
After the 2026-09-24 NH3 power outage every NFS line from nh3-nas failed at
boot: nh3-docker's /mnt/compose and /mnt/backup, and nh3-dev's /mnt/backup.
The NAS is the slowest box to serve NFS, and the plain fstab mounts stayed
down until remounted by hand. nh3-dev's /mnt/smithy already used
x-systemd.automount, and the same boot shows it self-healing on its next
access. playbooks/nh3-nfs-automount.yaml brings the other lines to that
shape (_netdev,nofail,x-systemd.automount,x-systemd.mount-timeout=30).
Hard-mount semantics are unchanged; only WHEN the mount happens moves.
Applied and read back on nh3-docker and nh3-dev: every nh3-nas line carries
automount, each automount unit is active, and each share is NFS-mounted after
access. A second run on nh3-docker is a no-op. The fstab check judges only
errors the rewrite ADDS, because nh3-dev's installer cdrom line has a
pre-existing verify error. The first run caught that and aborted without
writing.
servers/nh3-pve/README.md now lists each guest's expected state after a
power loss: nh3-laser is on-demand and stays off (Prime), pbs-nh3 now has
onboot=1 (it had none and stayed down), and pfi-gx10 is bare metal that
does not auto-power-on.
Operator directive 2026-09-24: clock times are written in 24-hour local
(US Pacific) format, four digits, no colon. Claude seats get it from the
global CLAUDE.md (dotfiles fe63a67). Codex (~/.codex/AGENTS.md) and Grok
(~/.grok/rules/00-fleettools.md) read this file through symlinks, so the
same rule goes under Non-negotiables here.
Operator ruling 2026-09-24. On ana-docker the stack is `docker compose
down`: the container is removed and port 7878 is closed. Kept for revival:
- the data dir /opt/docker/conf/task-board/data (tasks.db, last written
2026-09-11)
- the task-board:local image
- stacks/task-board/ and the host's compose + .env
The Uptime Kuma monitor (id 3) was deleted before the stop so it could
not page, and its row is removed from monitors.yaml. Homepage drops the
card on its own, since it reads the container's labels.
Hooks: the container log showed no hook POSTs in 30 days. The only
traffic was open browser tabs holding /events, and the plugin was already
uninstalled on nh3-dev. Removed the paragraph that told sessions to call
task_* tools (CLAUDE.md, and the fork-fleet.sh template that seeds new
repos), the vestigial TASK_BOARD_SESSION env in .claude/settings.json, and
the listings in README and FLEETTOOLS.
Global CLAUDE.md now defines how to reach Miranda: every message becomes a
summary on Prime's phone, URGENT means she phones him, and no noise. The
project section had prefixed every message URGENT and said 'when unsure,
send it', which contradicted that. It now points to the global section for
mechanics and keeps only what is infra-ops-specific: the standing order
covers matters already fixed, the consequence test, and the channel's
dependency on svos.service and hermes-gateway (a routine svos restart is
about 100s of deafness).
Operator standing order 2026-09-24: infra-ops informs Miranda of every
high-urgency matter, fixed or needing the operator. Records the send form
(URGENT — subject, what/state/operator-action body), the consequence-based
urgency test, and the channel's dependency on svos.service and
hermes-gateway, with a fallback route for outages that take those down.
Round trip verified: test thread 01M3A0RP4Q8T0KNGH8TMFSNDA6, ack in ~1 min.
The 2026-09-14 normalization (826a63b) moved /opt/docker/compose/<stack>
dirs to root:docker 2775. git refuses a worktree whose top-level dir has a
different owner, so yt-voice-clipper's webhook auto-deploy on irv-ml1
(git fetch/reset as lkraven) failed with 'dubious ownership' from then on.
Found 2026-09-24 when v0.3.13 did not land.
The fix keeps the convention: one safe.directory entry per repo for the
deploy user. Applied on irv-ml1 and the deploy re-run: main 2af9b3a,
/diagnostics reports 0.3.13. A sweep of the other normalized hosts found no
other checkout deployed by a non-root user.
Three of the four failures in the week to 2026-09-23 (09-19, 09-21, 09-23)
had one signature. The Cloudflare zone lookup returned an empty body at its
15s cap, a bare json.load crashed with tracebacks instead of a cause, and the
script carried empty IDs on to a PATCH against zones//dns_records/.
Cloudflare rejected it, so there was no DNS impact, but only by accident.
- Cloudflare calls go through cf(): 3 announced tries, and a call counts
only when the body says success:true.
- success:true is not trusted as shape. pick() validates every body and
prints only the fields asked for, or one line saying why not. It requires
exactly one zone named phasefinal.com and exactly one A record named
headscale.phasefinal.com, each with a non-empty id and content. Two A
records are refused rather than half-updated, and an empty id can no
longer shift the content into the id slot.
- No write without both IDs. The run ends on a confirmation that the record
now reads the new address. The previous final line was an echo whose exit
status was always 0, even when the parse inside it failed.
- Only a global unicast IPv4 is published (python ipaddress is_global).
Loopback, RFC1918, link-local, CGNAT and documentation ranges are retried
and then refused.
- curl -q as the first argument ignores ~/.curlrc, so a verbose config can
never log the bearer token. The vault CLI path is quoted. The empty
data-array expansion is safe under set -u on bash < 4.4.
Tests: services/headscale-ddns/test_headscale_ddns.py, 12 cases with curl,
the vault CLI and sleep stubbed. Each failure case asserts the FATAL line's
stated reason, so a run that died earlier for an unrelated cause cannot pass
it. The documentation-range fixtures (203.0.113.x) were themselves rejected
by the new public-IP guard: a free positive control.
Live: a manual run and a unit run both printed "unchanged 70.230.226.88",
Result=success. Cross-model bug-hunt (heid "Talus", Gróa + seat): all 8
findings folded.
elway's sudo upload did scp-as-user then `sudo mv`, and mv keeps the owner,
so every file it installed "as root" (systemd units, /etc configs, root-run
scripts) ended up owned by the SSH user. A sudoers drop-in installed that
way would be rejected by sudo outright.
elway:
- Sudo uploads now chown to root:root by default. Playbooks can override
with `upload.owner:` and ad-hoc runs with `--owner`. An owner is refused
on a non-sudo upload, and `--owner` outside an ad-hoc `--upload` is an
error rather than silently ignored.
- Ownership and mode are applied to the STAGED file, then a single mv
publishes it, so a failed chown can no longer leave the live path owned
by the SSH user. chown runs before chmod so setuid bits survive. A trap
removes the staged file on every exit path.
- A directory dest is refused before anything moves. mv would otherwise
drop the file inside the directory under a staging name.
- `mode` was spliced unquoted into the remote root shell line. It is now
validated as octal (^[0-7]{3,4}$) and shell-quoted, on both the sudo
and non-sudo paths.
- `mode` and `owner` must be quoted YAML strings. YAML 1.1 turns a bare
0644 into 420, a bare 1000:0 into 60000 and a bare yes into True, and
str() of each is a different value that looks valid. All 85 playbooks
already quote them.
- New preflight(): every step is resolved before any remote action. An
undefined var or a templated owner that resolves badly is now refused
up front, not after earlier steps have already changed the host.
scripts/fleet-ownership-audit.sh (new, read-only) finds files in root's
territory owned by a normal user. Tier A lists /etc, /usr/local, /root and
cron, judging symlinks by their target. Tier X covers files that root-run
units exec from /opt, /srv, /home or /volume1, reading drop-ins and quoted
paths. Tier B only summarises /opt app trees. A host that is unprivileged,
whose find fails, or whose run dies is reported INCOMPLETE (exit 5), never
clean. Completion is marked with a per-run nonce.
Verification: 24 unit tests in scripts/test_elway.py. Live checks on
esh-docker-vm: default lands root:root; the override lands infra-ops:adm at
4755 with setuid intact; a bad group fails with dest untouched and no
staging left; a directory dest is refused; a bad mode is refused before
any copy. Audit positive controls on nh3-dev (a drop-in-only root Exec, a
quoted path containing a space, a symlink to a user-owned target) were all
flagged; esh-docker-vm negative control was 0. Probes removed. Cross-model
bug-hunt (heid, Gróa arm + seat) findings folded.
Operator decision. paperless-ngx's database lives on esh-vm-db and is backed
up at the source by that host's fail-closed pg_dumpall. esh-docker-vm's
second copy had failed auth every night since 2026-04-24 behind a WARN. Its
`> paperless.pg_dump` redirect left a 0-byte file in every snapshot
(confirmed in snapshot 6ec9f74f), which looked like a dump but held nothing.
The block was the only consumer of /etc/restic/dbcreds.env, so the creds
loader is gone and the template dbcreds.env.example is deleted. The host
file was moved (not deleted) to /var/lib/restic/repair-20260923/.
Paperless's media volumes are still captured under /var/lib/docker/volumes.
Also fixes ownership. elway's sudo upload does scp-as-user then `sudo mv`,
so the hook deployed at 08:04 landed infra-ops:infra-ops even though root
executes it. Both esh-docker-vm playbooks now chown it back to root and
verify root:root 700.
Verified: the live hook hash matches canonical (e0d3ddcef1bddf43), and
the manual backup saved snapshot decfae71 with 3 staged dumps and no WARN
lines.
Uptime Kuma moved from esh-docker-vm to ana-docker on 2026-09-22. The
pre-backup hook's fallback lookup, `docker ps | grep -E "uptime.kuma"`,
then matched nothing and exited 1. Under set -euo pipefail that aborted the
hook, and resticprofile treats a failed run-before as fatal, so no snapshot
was taken from 2026-09-22 01:00 until this fix (backup-freshness: 54h stale).
The block is removed rather than guarded because there is nothing on this
host left for it to back up. The header now records the invariant the
"blocks only WARN" promise depends on: every optional-service lookup must sit
inside an `if` test or end in `|| true`. The remaining blocks were checked
and all do.
Deployed with playbooks/esh-docker-vm-restic-drop-kuma-block.yaml (the
pre-fix hook is kept in /var/lib/restic/repair-20260923/). The live hook
hash matches the canonical copy (43e6bea8b8569602). The manual backup saved
snapshot 6ec9f74f, and backup-freshness now reports all backups fresh.
A deliberate `systemctl --user restart hermes-gateway` paged infra-ops as a
FAILED unit (msg 3716) while the unit was already back up. Cause is a Hermes
v0.21.1 race: the planned-stop marker watcher runs the shutdown handler before
systemd's SIGTERM, consumes the marker, and the SIGTERM re-runs the handler,
which then classifies the stop as unexpected and exits 1.
Corrects the README claim that OnFailure never fires on a deliberate restart:
that holds only when the main process exits with a success status. Measured on a
throwaway unit (3/3 paged without SuccessExitStatus, 0/3 with it, and crash
restarts are unaffected), and a survey of every stop on nh3-dev since 09-15
found hermes-gateway to be the only unit that does this.
The host-side fix is drop-in hermes-gateway.service.d/
20-planned-stop-exit1-is-clean.conf (SuccessExitStatus=1). No alarm coverage
is lost: Restart=always ignores the classification, and StartLimitIntervalSec=0
means the unit can never reach `failed` from a start failure.
In-flight rewritten for 2026-09-22 evening. The operator approved both carried
calls ("decide to do both pending"): the NRestarts flap sampler is to be BUILT
(163bb97) and the restic content-assertion ruling is RATIFIED as-is (ba60fda).
Recorded in Recent decisions with tracking pointers, and their modality flipped
in the handoff from do-not-touch to genuine pending work.
safe-rm: installed on nh3-dev, delegated to infra-hermes, now fleet-wide 6/6
with acceptance met. Wiring ruled to /etc/bash.bashrc above the case-$ guard
rather than /etc/environment -- an rc self-guards, smaller blast radius, covers
bash-over-ssh. ⭐ The guarded acceptance test caught infra-hermes's own silent
wiring failure in flight: a probe that cannot destroy what it tests also cannot
lie about it.
⚠ Recorded twice over because it will otherwise be trusted for the wrong class:
safe-rm is a BLACKLIST. It refuses rm -rf /home and deletes an unset-variable
path without complaint. set -u is the actual cover for the habit that prompted
the install.
D-0010/D-0011: genuine, operator-approved, misrouted to this seat by pane_find
matching a ROLLING PANE TITLE. Belayed both, nothing lost. svos-dev owns the fix
and found the sharper half -- pane_send structurally cannot deliver a harness
command, so D-0010 promised an unachievable /clear and its receipt reported
success anyway. {"sent": true} is a claim about transmission, never effect.
Live thread carried: restic/ana/esh-docker-vm has drifted 36h -> 44h against a
48h threshold while every other repo sits at 12h. Drifting, not static.
Corrects my own framing from earlier today. I told the operator D-0010's /clear
"cost nothing by luck of the delivery mechanism". svos-dev verified the code:
bifrost.py:1322 prefixes every relayed string with the card id, so a slash
command never begins the line. pane_send STRUCTURALLY cannot execute a harness
command in a target pane -- a property, not luck.
Which makes the card worse than a misroute. It promised "type /clear into
terminal_2 and clear booth-dev's context"; that effect was never achievable at
all. The executed event then recorded {"sent": true} -- true about the bytes,
false about the effect. Three defects in one approved action, and only the
first was visible.
Recorded because it is the sharpest instance of the day's pattern and the only
one that cost something outside an agent's triage time: it consumed an OPERATOR
APPROVAL. Vuong approved an action believing it would take effect and the
system told him it had.
General form worth carrying beyond SVOS: {"sent": true} is a claim about
transmission, never about effect. Any receipt that cannot distinguish the two
will eventually report success for something that did not happen.
The failed-START notifier built earlier today had its first REAL firing at
15:28: headscale-ddns.service exited 1 after succeeding all afternoon. The
detection worked. The alarm was also useless, and that is the finding.
Both failure paths exited 1 IN SILENCE, so the message said "exit status 1" and
nothing else. An alarm you cannot act on costs the same triage as no alarm at
all -- the notifier did its job and the subject script had no diagnostics for it
to carry.
Cause was transient and harmless: icanhazip.com did not answer inside its 10s
cap, so the IP came back empty and the regex guard refused it. No DNS impact --
the record already held the right address, verified against 1.1.1.1 before
touching anything, and the next timer run succeeded. Arithmetic confirms it:
~17s vault read + 10s curl timeout = 27s against the 28s the failing run took.
Fixed, both verified by making them fail:
- every exit path names its cause; a missing vault key names the key, an
EMPTY token is distinguished from a failed read, and a dead WAN lookup adds
"DNS left unchanged" because that is the fact the reader needs
- the WAN lookup retries 3x with ANNOUNCED attempts -- one third-party blip
should not page a human, and a silent retry would hide a degrading
dependency
⚠ ALSO: this script was not tracked anywhere. A fix to the thing every mesh
client resolves through lived on exactly one disk. Script, unit and timer are
in the repo now.
Measured and recorded: the vault read is 17 of the script's 18 seconds, every
10 minutes. Not a fault, but it bounds any retry budget and it is fleet-wide --
svos-dev's alarm unit carries the same 17-second note.
In-flight rewritten for 2026-09-22. The Booth is booth-dev's now (extracted to
vh/booth, v0.2.0 shipped by them); infra-ops retains only its Uptime Kuma
monitor.
Eight decisions recorded, and three OPEN OPERATOR DECISIONS filed under Recent
decisions with commit tracking pointers rather than under in-flight, per the
deferred-work rule: the NRestarts flap sampler (163bb97), safe-rm, and
confirming the restic content-assertion ruling that reached me relayed by
svos-dev rather than Miranda (ba60fda).
Detail file for the day's throughline: five instrument errors, one shape -- a
tool that enumerates 'things that are fine' has selected against its own
subject. Four were mine, one a peer's, and mine included making theirs, on the
same day we were both cataloguing the habit. Written down because a rule both
parties broke while discussing the rule needs to live where a fresh session
reads it.
Two loose threads carried: talk.service failed while still serving on :8092,
and restic/ana/esh-docker-vm 36h behind the fleet.
Adding nh3-dev to the host list fixed the instance. This fixes the class, on
svos-dev's framing: a hand-maintained list of things to watch, sitting beside a
NAS that already knows which repos exist, means the next repo added is
unwatched BY DEFAULT and nothing says so. The list of what to check can
silently disagree with the set of what exists -- the same shape as every other
instrument fault found this day, only slower-acting.
The check now asks each NAS. A directory is a repository when it has a
snapshots/ child, which cleanly separates real repos from container dirs
(/volume1/Backup/restic/repo/ holds ana|esh|nh3 namespaces and no snapshots of
its own -- verified rather than assumed before building discovery on the
layout).
The hand-written list survives DEMOTED to an EXPECTED set, used only to report
a repo that has VANISHED. Two facts that would otherwise both read as silence
stay distinct:
"a repo exists that nobody watches" -> impossible now, it is discovered
"a repo we expected is gone" -> EXPECTED REPO NOT FOUND
Preventive, not corrective: all 8 repos currently discovered are already in the
expected sets, so this found no live gap. It removes the possibility of the
next one.
Controls run, since a check only ever seen passing is untested: a bogus
expected repo reports EXPECTED REPO NOT FOUND and turns the verdict STALE;
unchanged expectations still report all-fresh; all 8 repos report their age.
Observed while testing, not a fault: restic/ana/esh-docker-vm is 36h old
against 12h for every other repo. Inside the 48h threshold so correctly green,
but it is a day behind the fleet and worth a look.
Operator ruling 2026-09-22 (relayed via svos-dev): have the freshness check
assert snapshot content rather than REST-server reachability, so the green
light is a statement about DATA rather than about a daemon.
⚠ FIRST, A CORRECTION TO MY OWN REPORT. I ran `grep -ic restic` against
backup-freshness-alert.sh -- the WRAPPER -- got 0, and told the operator that
nothing on the fleet verified restic. Wrong. check-backup-freshness.sh has
always checked restic snapshot ages, for seven hosts. I grepped one file of a
two-file tool and generalised, which is the identical error a peer had just
made with dev-backup, made while correcting them.
THE REAL GAP was narrower and still real: the NH3 host list read
`irv-ml1 nh3-docker` and omitted nh3-dev -- the repo holding every Claude Code
session transcript, althing routes, hermes run history and Miranda's
conversation. /volume1/Backup/restic/nh3-dev/ has always existed and always
been written nightly; it was simply never enumerated, so a total failure of the
one repo nobody could reconstruct would have gone unreported indefinitely.
THE CONTENT ASSERTION, built on svos-dev's ladder (list proves the repo
answers; ls proves the index; check proves structure; RESTORE proves the bytes
come back):
- CONJUNCTIVE: latest snapshot newer than MAX_AGE_H AND the probe path
present AND a restore returns non-zero bytes. "A snapshot exists containing
X" is satisfied by a three-month-old one; "the latest is recent" is
satisfied by an empty one. Age alone was the old problem; content alone is
the same problem rotated.
- IT RESTORES. Metadata is what survives the failures worth fearing -- a
pruned or partially-written repo can list a path whose blobs are gone.
~493KB from ~/.local/state/svos, sub-second. Large paths are not restored:
a repo that returns one file will return others.
- IDENTITY: the timer runs as lkraven, which has no NOPASSWD sudo on nh3-dev,
so the probe hops through infra-ops@localhost. /etc/restic is root-only by
design.
⚠ restic ls ALWAYS PRINTS A HEADER LINE, matched or not. A path absent from the
repo returns 1 line; a real one returned 6. Counting with `grep -c .` read the
header as a hit, so the ABSENT case fell through and reported "blobs gone" --
telling an operator the repository was corrupt when the truth was a mistyped
path. Now `grep -c '^/'`.
TESTED BY MAKING IT FAIL, because a check only ever seen passing is untested:
bogus probe path -> "absent from snapshot"; BACKUP_MAX_AGE_HOURS=1 -> "12h old
(>1h)"; healthy -> snapshot id, age, entries, bytes restored.
Two additions, both from a 2026-09-22 exchange with svos-dev.
THE RULE. Coverage is a property of the backup SYSTEM, not of one job's
configured scope. A peer checked dev-backup.sh, found SRC=$HOME/development,
and reported to the operator -- with specifics and unhedged -- that five
home-directory paths including Miranda's entire conversation had never been
backed up anywhere. All five were in that night's restic snapshot. dev-backup
is the hourly job for one directory; resticprofile is the daily job covering
all of /home/lkraven. Checking one job and generalising to the system produced
a confident, false, escalated claim. The runbook now carries the query that
answers the question properly.
THE GAP THAT VERIFYING IT EXPOSED, and it is worse. grep -ic restic against
scripts/backup-freshness-alert.sh returns 0. The checker inspects PBS guest
ages and pings the rest-servers for liveness -- which confirms the server
answers, not that a snapshot was written. If resticprofile stopped entirely the
light would stay green, correctly by its own definition, forever. Restic holds
the whole home directory; PBS holds VM images. The layer with the granular data
is the unwatched one, and the light is not merely blind but actively reassuring
about a system it cannot see.
Recorded as an open gap rather than patched, because fixing it changes what an
existing green light means and people have been reading that light for months.
svos-dev found an interaction between the two detectors by doing the recovery
rather than reasoning about it: clearing a failed unit is REQUIRED before
systemd will start it again, and reset-failed zeroes NRestarts in the same
motion. Verified here -- a start-limited unit refused to start until
reset-failed, which took NRestarts 3 -> 0.
So the standard recovery from a failed-START alarm erases the history a flap
sampler would depend on, routinely rather than as an edge case. A unit that
fails, is cleared, then flaps starts its history from zero, delaying the first
flap alarm by the whole threshold -- exactly the window where a just-fixed unit
is most likely still sick. Mitigation recorded: store a last-seen timestamp so
a counter going BACKWARDS is itself detectable as a reset rather than read as
quiet.
Also recorded as its own shape: 'nothing to count is not the same as not
counting'. The first manual-restart check read NRestarts flat and would have
concluded a sampler goes blind whenever a human touches a unit. Artifact of the
test -- that unit had already hit its start limit and stopped auto-restarting,
so the instrument was reading correctly from a subject that had stopped
producing. Verify, then ask what the verification could not have seen.
OnFailure fires when a unit enters failed. A unit that keeps re-earning its
retries never enters failed, so the alarm never fires for it. svos-dev warned
this was the consequence of the interval divergence; measuring it showed the
gap is most of the box.
MEASURED: a unit shaped like booth/althing-po-herald (Restart=always,
RestartSec=5s, burst 5, interval 10s) ran 9 restarts over 45 seconds cycling
active/activating and NEVER reached failed. No alarm would have fired.
Covered: the 7 timer-driven oneshots (Restart=no, so any failure lands in
failed immediately) and svos.service (burst 3 per 5min -- it genuinely gives
up). NOT covered: ten daemons that flap instead, and hermes-gateway, whose
start limiting is disabled outright.
svos.service's divergent 5min window is deliberate and load-bearing (operator
ruling 2026-09-11, 'fatal both ways'). NOT to be harmonised: that would restore
the flapping the ruling forbids AND silence the alarm on the one daemon it
currently works for. Whether the OTHERS move to 5min is a behaviour change for
ten services and an operator call.
Also recorded, svos-dev's framing of the two installer bugs: a tool that
enumerates 'things that are fine' and acts on them has selected against its own
subject. Neither bug failed randomly -- both were anti-correlated with purpose,
working better the healthier the fleet looked.
Two defects, both found by the tool failing to cover a unit that was already
broken. talk.service has been sitting in `failed` (exit 143) while its
containers keep serving 200 on :8092 -- precisely the "unit died, the thing
keeps serving, nobody knows" case this alarm was built for -- and the installer
had not hooked it.
1. SELECTION: --state=running skipped anything not already healthy. A unit that
is down at install time never got hooked, which inverts the tool's purpose.
Now the union of enabled unit-files and all loaded units.
2. PARSING, and this is the sharper one: systemd decorates a FAILED unit with a
leading "●", so `awk '{print $1}'` returned the bullet instead of the name,
and the sanitiser reduced it to an empty string and dropped the row. The
parser silently lost exactly the rows that matter. --plain suppresses the
decoration. Same shape as every other instrument error today -- it reported
cleanly while looking at the wrong thing.
Selection now 23 units. Deliberately INCLUDES the timer-driven oneshots
(dev-backup, ha-backup, fleet-tls-cert-check, headscale-ddns,
seat-inventory-drift, brokkr-landscape-scan, soong-ci-relay): a backup or a
cert check that fails silently is the same class, and all seven were clean at
install time so they are not a known noise source. EXCLUDES dbus, gpg-agent and
dirmngr as OS plumbing, the notifier's own template and instances, and
svos-failed-alarm -- hooking an alarm with an alarm is a loop with extra steps.
svos-dev challenged a claim this README made -- that a crash-loop yields one
message per episode -- with a measurement: one 2026-09-19 boot-gate refusal on
svos.service produced FIVE transitions into failed, and the operator got five
messages.
Measured here before accepting it, because a peer's number is still a number
someone else took: a unit with Restart=on-failure, burst 3, interval 30s
produced 7 journal failure lines and exactly ONE notifier invocation. So the
multiplier is not universal -- it needs retries spanning start-limit windows or
an external restarter. svos.service carries StartLimitIntervalSec=5min, which
is how it accumulated five.
Both conditions exist on this box, so the guard goes in as cheap insurance
rather than as a fix for something proven here. The README now states both
numbers and which restart policy each of the twelve units carries, since that
is what decides the exposure. Noted with it: hermes-gateway has start limiting
DISABLED, so it retries forever and may never reach failed at all -- worth
knowing before trusting this alarm to cover it.
Design, taken from svos-dev's shape:
- Keyed on a hash of the CAUSE (unit + result + exit status + the shape of
its last error lines), never the unit name alone. A genuinely different
failure inside the window is a new fact and must still page; suppressing by
unit would hide a second, worse failure behind the first.
- Suppression is LOGGED to suppressed.log, never silent. An alarm that
quietly declines to fire is indistinguishable from one that is broken.
⚠ The first test of this appeared to show the cooldown not working, and the
test was wrong rather than the code -- it invoked the script BEFORE failing the
unit, so the two calls legitimately saw different states and computed different
fingerprints. Re-run the way systemd actually invokes it: same cause 3x -> 1
sent, 2 suppressed and logged; a different cause inside the same window -> sent.
svos-dev found the failure neither Beszel nor Uptime Kuma can see, and
measured it: a config change on 09-19 made svos.service refuse to boot, the
RUNNING process predated the change and kept serving, and the service sat one
restart from dark for three days. Every uptime probe was green and correct --
the thing was up. The signal that catches this is failed-START, not down.
A count after that conversation: 13 running user units on nh3-dev, ZERO with
an OnFailure hook. Including althing-po-herald, whose silent failure cuts
infra-ops's own mail delivery -- a blind spot in the notification path every
other alarm on this fleet depends on.
One template, one drop-in, 12 units hooked (dbus excluded as systemd's own
plumbing). Not noise: OnFailure does not fire on a clean restart or a
deliberate stop, and with Restart=on-failure a crash-loop yields one message
per episode rather than one per attempt.
⚠ %i, NEVER %I -- and the acceptance test is the only reason this is right.
%I unescapes the instance name and systemd escaping maps "-" to "/", so the
first run delivered a message for "onfailure/selftest.service", a unit that
does not exist, with a spool path that tried to create directories.
althing-po-herald.service would have arrived as althing/po/herald.service.
It "worked" -- mail was delivered -- which is exactly the kind of success that
is not one. The referring unit passes %n raw, so literal %i is correct.
⚠ The notifier never reports itself, guarded twice on purpose: the template
carries no OnFailure, and the script bails on its own instance name. A
notification loop is the one bug that pages you forever.
Delivery is spool-first. postbox has no outbox, and an alarm for moments
nobody is watching must survive the post office being one of the things that
is down. If the herald itself fails, the message still REACHES the post office
(postbox talks to it directly; the herald only delivers inbound pokes) -- not
pushed, but stored for the next read.
Acceptance-tested twice against a unit that exits 42: once to catch the %I
bug, once to confirm the fix. Test unit removed.
svos-dev's call and the better one: The High Seat is the English name for
Hlidskjalf, the seat Odin watches all the worlds from, and watching every
session at once is what the board does. mdi-monitor-dashboard described the
artifact; the eye describes the job.
Deployed and verified in /api/services, not assumed.
Requested by svos-dev relaying the operator, 2026-09-22. Reversible work, so
the relay is fine to act on without escalating.
Manual services.yaml entry rather than container labels, because SVOS is a
user-level systemd unit (svos.service) on nh3-dev and nh3-dev is NOT one of
the five hosts in docker.yaml -- Homepage has no Docker API to discover it
through. Same reason the Booth, WhereTF, talk and the infra-hermes seat are
listed by hand, and the comment says so at the entry.
⚠ siteMonitor is "/" deliberately. There is no /api/health on this service:
that path 404s, and a monitor pointed at it would report the board
permanently down while it serves perfectly. svos-dev flagged it and it is
verified here -- / returns 200 and serves the SPA (<title>The High Seat</title>).
Group is Apps, which exists in settings.yaml's layout with tab: Main. An
invented group name gets no tab and renders on ALL tabs, which is how
Scriberr's "AI Systems" leaked across the whole dashboard in August.
Icon mdi-monitor-dashboard is my choice -- svos-dev explicitly did not guess
at one and offered to take a different suggestion.
Verified in a browser, not just in the API: the card renders in Apps with a
green site-monitor at 28 ms.
TAILSCALE IPv6. `tailscale status` had been reporting, continuously:
2 add route failures; first was: permission denied
adding address fd7a:115c:a1e0::6/128 from tunnel interface: permission denied
with tailscale0 carrying only 100.64.0.6/32 while headscale had assigned it
an IPv6 address it could not use.
Not a capability problem -- tailscaled runs as root with the full bounding
set. /etc/sysctl.conf:59 sets net.ipv6.conf.default.disable_ipv6=1, and
`default` is inherited by NEWLY CREATED interfaces; tailscale0 is created at
daemon start, inherits it, and the kernel returns EPERM for every attempt.
Fixed with a scoped systemd drop-in rather than flipping the global default.
That line carries no comment, but IPv6-off-by-default on a host with ~26
docker bridges reads as deliberate, and changing it would hand IPv6 to every
future bridge as a side effect of fixing Tailscale.
⚠ It must be ExecStartPost, not /etc/sysctl.d. A sysctl.d entry for a
per-interface key is applied at boot, BEFORE tailscale0 exists, and is
silently ignored -- the setting would look present and do nothing.
Also learned: setting the sysctl on the LIVE interface is not enough.
tailscaled only attempts the address at startup or on a netmap change, so
the verify failed for 60s until the daemon was restarted. Restart is part
of the operation, not an afterthought.
Verified: fd7a:115c:a1e0::6/128 present on tailscale0, health clean, mesh
and services (arbo, ytvc) up.
README. It documented the pre-headscale topology as current -- "Reachable
IP: 10.100.79.3 (WireGuard tunnel endpoint)", "No direct LAN access", and a
refresh caveat telling you to bring WG up. That sends anyone triaging this
host to the wrong layer, which is the exact tax the file exists to prevent.
Now: mesh primary at 100.64.0.6, LAN 10.6.110.50, and wg0 documented as
STILL UP with a live peer -- tailscale uses that address as its direct
endpoint, so it is load-bearing, not vestigial.
Recorded with it, because these cost hours tonight and will cost them again:
- Irvine is a TENANCY behind a Fortinet PFI does not control. Its TLS
inspection breaks Tailscale's relay and control channels (41 cert
warnings/week, 4 control-plane episodes in 14 days). Usually invisible
because direct peer paths carry the data. No fix on our side.
- Diagnose reachability with `tailscale ping`, NOT the status output:
headscale said "online" and status said "active, 19.7 GB" while nothing
on the host answered. Both are last-known state; only a round trip is
liveness.
- The ~26 docker bridges make tailscaled report captive portals.
Two stale claims corrected: the hostname rename it called "pending" is done,
and `ollama` is listed as running on :11434 when it is gone -- verified, no
unit file, nothing listening, no process. It is banned fleet-wide.
vh