Against talk /face's guided pose (first paragraph 246 ms median), SemIf in
parallel adds 32 ms and SemIf-first adds 94 ms (n=72 each, noise floor 16.5 ms).
Removing the pose header saves only ~31 ms, and SemIf shares GPU 1 with the LLM.
Acceptable pose 67% vs 92% on clear-emotion lines, and the mood carried through
mundane follow-ups 7/15 vs 14/15. SemIf gestures far less (13% vs 58%).
README: rotations cost options^2 in suffix tokens, and /decide/shared returns
422 when an object state's last value ends in ) ; or }.
The comment claimed 'gateway-key-gated'; v1.97.0 actually gates auth=true
passthrough routes on per-key metadata allowed_passthrough_routes (OSS path,
not Enterprise), 401 unauthenticated. Grants applied live via /key/update for
all-agents-local and the worldtree gateway key; comment-only change here,
picks up with the next conf deploy.
Harness consumer_fit.py runs a consumer's per-turn decisions over hand-labelled
cases, with rotations, a content-free null control and tagged positive controls.
Cicada: an input-only 'does this earn a visible reaction?' gate scored 30/31
with descriptive options and 19/31 with terse yes/no options. Scoped by
Cicada's 2026-09-20 ruling (affect is emitted once, no mood-ring classifier).
Wyrd: on 3 real seed graphs, the first place-change wording failed its
positive controls (1/6 moves). A location-anchored rewording scored 21/21, and
exit selection scored 18/21. Semif fits the choice, not writing the node.
Order averaging (Prime, after the 739aa03 spike):
- A decision may set orderings: rotations|all (all only for <= 4 options). Every
ordering goes to the engine in one shared batch.
- The reply keeps each native result and adds combined {probabilities (log-mean),
top, agreement, spread}.
- Through the service on SemIf's labelled sets (252 rows): 78.6% -> 88.1%
(group-bootstrap 95% CI +5.1..+14.3). Unanimous agreement is 94.5% accurate.
Fast kernels: flash-linear-attention 0.5.2 and causal-conv1d 1.7.0 are now the
default build. A/B on the empty GPU 3:
- parity with upstream went from 142/144 to 144/144;
- a ~2k-token /decide went from 169 to 92 ms server-side;
- short 3-rotation batches cost ~3-6 ms more.
triton builds a C shim at runtime, so the image carries gcc. Without it the
warm-up failed and startup failed closed.
Heid bug-hunt panel (4/4 arms, thread 01M3H3F4RR7XBP90KQ3A39H4SX), folded:
- Startup validation: VRAM cap 0 no longer means uncapped (C1); limits must be
>= 1 (S1); the token must be visible ASCII (S2); the calibration file must
exist and parse, with T in [0.05, 20] (S8, and C3's NaN leg).
- The body limit is checked before a chunk is kept, and a Unicode-digit
Content-Length no longer crashes (C2, S3).
- Failures while building the response now get the 500 envelope (C3).
- 429 busy past SEMIF_MAX_QUEUE requests in progress (C6).
- The engine releases memory on every non-validation failure, unchained after
gc; an empty OOM message is handled; 'out of memory' RuntimeErrors map to 503
(C4, C5, S9).
- The entry point forces HF_HUB_OFFLINE (S10). README wording fixed (S5, S6).
- New guard tests close the gaps the arms' mutation grids exposed: early stop of
the body read, a shared-route lock, calibration pass-through, the gc cycle,
the exact caps, TorchEngine.load's arch and device checks, and the offline
entry point.
86 tests.
Deployed on fv-ml1 GPU 1: parity 144/144, OOM and burst release verified, shared
capacity 63/51/26/16 rows at ~140/520/1960/3900 prefix tokens.
Latency, measured from nh3-dev (3 runs x 20 per condition; network floor 31 ms):
- /decide short: 71 ms end to end, 38 ms server-side;
- /decide with a ~2,000-token state: 210 / 169 ms;
- shared, 3 rotations: 113 / 79 ms;
- shared, 6 orderings: 137 / 99 ms.
Qwen3.5's fast kernels (causal_conv1d, flash-linear-attention) are not installed,
so transformers falls back to its reference PyTorch paths. That is a speed lever,
and using it needs a parity re-check.
Averaging over option orderings, on SemIf authored144 + perturbations108 (252 rows,
72 groups):
- a single ordering scores 78.6%;
- log-mean over the 3 rotations scores 87.7% (+9.1 pts, group-bootstrap 95% CI +4.7
to +13.8);
- all 6 permutations score 88.1%.
Rotations capture nearly all of the gain. Rows where the rotations agree unanimously
(161) are 94.4% accurate; split rows (91) are 75.8%.
services/semif-serve is a FastAPI wrapper around SemIf's direct and shared torch
scorers (SemIf-OpenJev @ 23cf1f39, MIT). Upstream ships only a batch CLI. The
wrapper loads the pinned Qwen3.5-4B (851bf6e8, BF16) once from the offline HF
cache and returns SemIf's result dicts unchanged, with an optional per-workload
temperature-calibrated view. Contract: semif-serve.contract.md. Built with a
short contract, TDD (39 tests, fake engine and fake torch, no GPU) and a heid
bug-hunt panel (pending).
On the card:
- torch 2.10.0+cu128 with sm_120 kernels, which is SemIf's own stack;
- a hard 12 GiB VRAM cap.
Two defects surfaced only on the card, and each fix is covered by a test:
- 0.1.1: an OOM raised as a chained exception kept the failed request's tensors
alive (11.9 GiB after the 503). It is now raised unchained, after gc.
- 0.1.2: a large request left 12.6 GB reserved on the shared card. After each
call, reserved memory over the baseline + 512 MiB is now released.
Acceptance against SemIf's committed torch predictions (authored144):
- 142/144 same top choice; both misses are exact bf16 ties;
- 144/144 identical prompt hashes;
- deterministic A-vs-A;
- negative control 14/144;
- shared vs direct 72/72.
21 binary criteria over one state take 159 ms. The shared-mode capacity table
under the cap is in stacks/semif/README.md.
The Dockerfile installs dependencies from a manifest with the project version
blanked, so a version bump reuses the ~4 GB torch layer. Verified: 41 s rebuild,
dependency layer CACHED.
DNS: semif.fv.internal. Token: vault semif/api-token.
Prime bootstrapped infra-ops on vm-esh-nas with playbooks/bootstrap-infra-ops-user.yaml.
It got the fleet-pinned uid/gid 850, NOPASSWD sudo with log_output, the docker
group and a 0700 home. That let playbooks/restic-repository-file.yaml migrate
the last restic host: the live profile matched the repo's pre-change sha,
its units no longer carry the URL, its secrets are vaulted, and the live and repo
profiles now match (a5ea75ea). All eight restic hosts are clean.
The staged helper script is gone, both from Prime's home on the host and from the
repo. The docs that described vm-esh-nas as lkraven-only are updated.
docs/pfi/fleet-conventions.md § 3 pins infra-ops to 850, but the bootstrap
playbook let the OS allocate, so every host drifted (1001-2001). New
accounts now get 850 when uid and gid 850 are both free, and fall back to
OS allocation (with a note) when either is taken. The home is set to 0700
per § 1.2. Existing accounts are untouched (the step is gated on id).
Scope note: vm-esh-nas is added to the operator-granted ESH exceptions
(Prime, 2026-09-27).
resticprofile schedule copies env-file values into the generated units, which
are 0644, so RESTIC_REPOSITORY (the rest-server basic-auth password included)
was readable by every local user on every restic host.
New playbooks/restic-repository-file.yaml:
- derives /etc/restic/repository (root 0400) from restic.env;
- uploads the profile switched to repository-file, but only when the live
profile's sha matches the repo copy it was edited from (drift guard);
- checks the repository is reachable through the new profile (cat config);
- regenerates the units and verifies they exist and contain no rest:http.
Applied to ana-docker, fv-ml1 (configs/restic/ana-ml2), esh-docker-vm,
esh-vm-db, irv-ml1, nh3-dev and nh3-docker. An independent check across all
eight restic hosts (these seven plus esh-ml1) found 0 leaking units. nh3-docker's
scheduled unit ran a real backup afterwards (snapshot a29b889d). Each host's
URL and passphrase are vaulted as <host>/etc/restic/{repository,password}.
restic.env is kept (root 0600) because the per-host READMEs and the freshness
probe source it. A rotation must update the vault, restic.env and repository.
vm-esh-nas has no infra-ops account. Its in-place migration script is staged
for Prime to run with sudo, and its repo profile is pre-edited to match.
Also mirrors augaman-dev's 401df2d (compose header only). The config hash on
esh-ml1 is unchanged.
Both hosts had live improvements the repo never recorded:
- esh-docker-vm excludes ESPHome's 539 MB of PlatformIO cache (2026-09-14).
- irv-ml1 runs /etc/restic/arbo-checkpoint.sh before the backup, a non-fatal
SQLite online-backup of arbo's gallery DB. That script is added here too.
The live copies were correct and are the source for the next change. (Also fixes
the fv-ml1 augaman removal time to ~0130 PT.)
Prime removed the second instance after the v0.1.3 bench. esh-ml1 handles a face
in ~48 ms, sits in the house next to the cameras, and holds the verified
backup. fv-ml1's gallery was empty (0 identities). The container, gallery volume,
image, compose dir (with its .env), backup dir and build sources are removed from
fv-ml1. GPU_ID / CARD_SUFFIX stay in the compose for any future second host.
Both hosts are rebuilt from tag v0.1.3 (one ONNX session per detector canvas) and
redeployed. pytest -m gpu tests/vision passes 3/3 on each card.
Server-side for one face, same harness as the v0.1.2 baseline:
- esh-ml1 GPU 144 -> 48 ms
- fv-ml1 GPU 75 -> 27 ms
End to end from nh3-dev: 101.6 and 73.8 ms.
CPU mode got slower on every CPU target: fv-ml1 cpuset 0-5 went 152 -> 205 ms
with a face, and the no-face frame roughly doubled. That is well outside the
run-to-run spread. The suspected cause (not measured) is per-session ORT
thread pools spinning. Reported to augaman-dev. Neither deployment uses CPU
mode.
On esh-ml1 the dependency layer missed the build cache and the rootfs touched
90% until the v0.1.2 image was removed. fv-ml1's build hit the cache, and the
exported requirements are identical, so the stack README now says to check
disk before building on esh-ml1.
Prime asked for augaman on fv-ml1's utility card, beside vllm-coder. Mirror
augaman-dev's f77164f compose, which parameterises the GPU reservation (GPU_ID,
default 0) and the Homepage card name (CARD_SUFFIX). esh-ml1's resolved config is
unchanged: same config hash, no recreate.
On fv-ml1: augaman:0.1.2 built on-box from the tag, GPU_ID=1, healthy on CUDA
at 1264 MiB, and pytest -m gpu tests/vision passes 3/3 on the Blackwell. It has
its own gallery and no gallery backup, so it is fixtures-only. The host's raw
restic copy of /var/lib/docker/volumes is not a consistent SQLite backup.
docs/pfi/augaman-speed-bench/ holds the harness (augaman-dev's recipe plus a
no-face control frame and a face-count check on every response), the raw rows
and the summary. Server-side, one face:
- esh-ml1 GPU 144 ms
- fv-ml1 GPU 75 ms
- fv-ml1 CPU on 6 cores 152 ms
- esh-ml1 CPU 888 ms
It agrees with augaman-dev's independent esh-ml1 measurement once each
harness's floor is subtracted. This is the before for v0.1.3's detector fix.
esh-ml1 is outside vzdump, so augaman's face gallery reaches backup only
through restic. New playbooks/esh-ml1-restic.yaml installs restic 0.14.0 (the
same Debian package as the other ESH hosts) and resticprofile 0.33.1 (pinned,
sha256-checked). It uploads configs/restic/esh-ml1/ and schedules a daily
0100 PT backup plus a Sunday 0500 PT check to rest-server-ana. The CT runs UTC,
so both schedules name the zone explicitly.
pre-backup.sh is fail-closed: it runs augaman's own backup CLI, and any failure,
including a stopped container, aborts the run. Tested with a stub docker that
exits 1: the run returned 1, and neither the snapshot count nor last-success
moved. The restore was verified at identity level against augaman-dev's
public-domain canary (snapshot fd3061a1: the restored copy's digest over
identities and samples matches the live gallery). That meets the operator gate
for real enrollments.
The repository URL is read through repository-file rather than restic.env.
resticprofile schedule copies env-file values into world-readable systemd
units, which publishes the rest-server password on the env-file hosts
(observed on esh-docker-vm). This is recorded in the backups runbook under
Known gaps, and the playbook verifies no generated unit contains the URL.
esh-ml1 is added to the freshness check's expected ana-side repos and to the
runbook tables.
augaman moves to v0.1.2 (dependency layer keyed on the lock without the
project; per-crop embedding). pytest -m gpu tests/vision passes 3/3 on the
card, and the canary survived the container recreate.
Mirror pfi/augaman deploy/compose.yaml as stacks/augaman, with an .env.example and
a README carrying the biometric backup gate. The image is built on esh-ml1 from a
git archive of the release tag, because the box holds no gitea credentials.
Serving on CUDA and visible in nvidia-smi. The gallery backup is not wired yet
(esh-ml1 has no restic), so only public-domain fixtures may be enrolled.
The on-box gpu test fails its batch-vs-single tolerance 3/3; reported to
augaman-dev, who owns the contract.
For ha-dev (operator-approved 2026-09-26). CT 111 on esh-pve at 10.0.90.20:
Matter/Thread IPv6 (Echo ULA + RA route-information) is link-only, so the
server sits on esh-iot and HA reaches it over routed IPv4 ws :5580.
- playbooks/esh-matter-lxc.yaml: kernel RA (accept_ra=1,
rt_info_max_plen=64), forwarding off, Docker ip-forward/iptables off;
nftables admits 5580 from HA 10.0.50.46 only and SSH from mgmt ranges;
the CT is added to esh-pve's vzdump job (fabric credentials).
- stacks/matter-server: ghcr.io/matter-js/matterjs-server:1.4.0 (digest),
host networking, /data on the CT.
- Acceptance: fdad:: SLAAC, ping6 thermostat, 2 Thread RIO routes learned, ws
server_info from inside the HA container; 5580 refused from 10.0.50.45,
nh3-dev and a temporary VLAN 90 netns vantage.
- qwen3-embedding and reranker: second deployment on nh3-ml1 (config);
reranker-a3-bge-v2-m3: second DB deployment via /model/new.
- router_settings.enable_weighted_failover: true. Only the three
multi-deployment groups are affected.
- Measured by stopping nh3-ml1 TEI: without failover 7/40 embeds 500'd;
with it rerank 80/80, embed 38/40 at onset and 60/60 over a 34 s outage;
nh3 rejoins rotation after restart. Split 12/28 embed, 22/18 rerank.
For brokkr's dataset foundry (operator-approved 2026-09-26, relayed).
- stacks/lfm-vl-seat: llama.cpp server-cuda b11176 (digest-pinned), Q5_K_M +
mmproj Q8_0, :8030; gateway alias lfm25-vl-3b (LiteLLM restarted, 36 s).
Positive control exact; null control shows it describes a missing image.
- stacks/vibevoice-asr-seat: audio.cpp v0.8.2-audio8-perf-hotfix (the GGUF's
own runtime, not vibevoice.cpp) on cuda 12.8 runtime + libgomp + libsoxr,
sha256-pinned; :8031 direct. LibriSpeech WER 3/69, RTF 0.07-0.14; ~31 s
cold first request.
Same model revision (df3ce67, blob sha matches fv-ml1), vLLM v0.24.0 and flags;
0.33 of the Ada (~5.4 GB, fv-ml1's absolute budget). Healthy, KV 79,808 tokens.
Parity (teacher-forced true code, 40 FIM prompts, 2,215 tokens, cache-salted):
each host is bit-exact with itself; cross-host |dlogprob| median 0.049, top-1
agreement 0.966 (Blackwell vs Ada + fp8 KV); against ground truth no quality
difference (mean logprob diff +0.008 +/- 0.019 SE). Speed on-box: ~5x slower
(64-tok FIM p50 ~1.0 s vs ~0.2 s; 63 vs 338 tok/s). Gateway coder-fast left on
fv-ml1 pending Prime's call.
- AMT re-IP'd over WS-Man to static 10.100.250.61/24 (gw/DNS .1): AMT keeps
its old DHCP lease across a VLAN move, so it was unreachable until changed
(reached via a temporary on-link /32 on vmbr0 + ssh tunnel, since removed).
- KVM redirection enabled, redirection listener on, IPS_OptInService
OptInRequired=0; all read back. Password vaulted as nh3-pve/amt-admin.
- scripts/amt-wsman.py: stdlib WS-Man get/put/invoke client (digest auth,
AMT 16 legacy-renegotiation TLS), password from $AMT_PW.
- PFI-UDMSE port 6 override: native nh3-mgmt, tagged VLANs blocked (was
forward all / native default). Reservation nh3-pve-amt -> 10.100.250.61.
- nh3-pve: arp_ignore=8 / arp_announce=2 on enp88s0 (now on vmbr0's untagged
L2) so the host never answers ARP for 10.100.250.60 with the AMT port's MAC.
- DNS nh3-pve-amt.nh3.internal -> 10.100.250.61.
- Homepage: NH3-PVE-AMT card under Infra - NH3 (no siteMonitor/ping: AMT drops
ICMP and its legacy-renegotiation TLS fails Homepage's fetch).
- AMT keeps its old 10.100.0.151 lease until rebind/expiry (~1920-2224 PT
2026-09-26); it does not re-DHCP on a VLAN change or link drop (measured).
AMT had no IP because Linux had enp88s0 admin-down (it left vmbr0 on 09-25 and
had no auto stanza); igc powers off the I226-LM PHY when down, so AMT had no
link. With the port up (no IP, unbridged) AMT took a lease within minutes.
- nh3-pve: `auto enp88s0` (manual, unbridged) and sysctl.d disable_ipv6 on it;
backup /root/interfaces.bak-20260925-amt; runtime already matched, no reload.
- UDM: reservation nh3-pve-amt -> 10.100.0.151 (untagged nh3-default; the
cable is on an nh3-sw1 trunk, not the UDM).
- DNS nh3-pve-amt.nh3.internal. AMT 16.1.25 answers TLS :16993 and :664.
nh3-ml1's first Temperature alert (2026-09-25 2105, 88.8 C) was nh3-pve's CPU
package during the nightly vzdump; the GPU sat at 47 C. hwmon is not namespaced,
so an LXC's agent reports its host's coretemp sensors under the LXC's name.
- hosts/{nh3,esh}-ml1.yaml: SENSORS=-coretemp_*,acpitz (blacklist); GPU and
NVMe temperatures verified still reported.
- Hub: Temperature >95 C / 5 m on nh3-pve and esh-pve for the CPUs (i9-13900H
TjMax 100 C; nh3-pve 2-hour averages reach 88 C on busy days).
- nh3-pve README: runs warmer than esh-pve; workload confounds it, check
airflow on the next visit.
NH3 site visit done: Secure Boot off, iGPU restored as boot VGA, AMT port cabled.
- nh3-pve: NVIDIA 580.178.04 (DKMS, open modules) via pve-nvidia-host.yaml.
- nh3-ml1 = CT 109 @ 10.100.50.80 via gpu-lxc.yaml; embed-rerank (TEI 1.9.4)
deployed with HOST_NAME/HOST_IP labels.
- Parity vs esh-ml1 (1,126 texts, 2 runs/host, controls): embed cosine min
0.999993 = own noise floor; overlap@10 1.000 vs MRL-256 positive control
0.684; rerank top-1 1.00, max diff 0.0014 vs floor 0.0020. On-box speed
identical within rep spread.
- gpu-lxc.yaml: first step upgrades lxc-pve to >= 6.0.0-2 (Proxmox fix#7006).
With 6.0.0-1 every docker run in a nesting CT failed on runc 1.5's sysctl
reopen; applied on nh3-pve (one package).
- pve-nvidia-host.yaml: document that the headers meta drags in the newest
kernel (nh3-pve went 6.8.12-11 -> -43 at the next reboot).
- Monitoring: Beszel NVIDIA agent + 5 alerts, Kuma #29/#30, Homepage
nh3-ml1-docker, Dozzle agent (hub 8 clients). DNS nh3-ml1.nh3.internal.
- nh3-pve README: SB/IGFX/driver/kernel state, btmtk oops on -4x kernels,
AMT cabled but unreachable on the network.
Gateway routing to nh3-ml1 is not changed.
- esh-pve-nvidia-host -> pve-nvidia-host: headers/dkms/build-essential step,
nouveau blacklist + guarded unload (refuses if nouveau bound a device)
- esh-ml1-lxc -> gpu-lxc: host vars have no defaults (elway aborts on undefined),
rootfs storage/startup order parameterized, CT kept out of all-guests vzdump jobs
- embed-rerank: Homepage labels take HOST_NAME/HOST_IP, defaults = esh-ml1
nh3-pve and esh-pve are the same Minisforum MS-01 (BIOS AHWSA.1.17). With a
card in the x16 slot its root port takes bus 01 and every NIC moves down a
bus (measured on esh-pve), so predictable names change (enp2s0f0np0 ->
enp3s0f0np0 etc.) and vmbr0 would boot with no uplink. systemd .link files
now pin all NICs by MAC, baked into every initramfs and synced to the ESP;
udev confirms the files apply. The AMT-capable I226-LM (enp88s0) leaves
vmbr0's bridge-ports in the file (next boot), so cabling it for AMT cannot
loop the STP-less bridge.
Also: documented the NanoKVM (https://10.100.250.171) as nh3-pve's console
OOB and that AMT is not wired; nh3-dev's Beszel agent no longer binds NAS
shares (it died on the last NH3 cold start); post-boot checklist in
persistent-memory.
- Beszel: NVIDIA agent (stacks/beszel hosts/esh-ml1.yaml), hub system
registered with Status/Disk/CPU/Memory/Temperature alerts; GPU samples
verified (RTX 2000E Ada util, VRAM, power, temp).
- Uptime Kuma: health monitors for the TEI embed (:8001) and rerank (:8013)
services — the sole backends behind the gateway, so the one exception to
"seats are out of Kuma's lane". Reward seat excluded (no consumer).
- Homepage: dockerd on tcp/2375 bound to 10.0.50.80 (playbook step);
esh-ml1-docker added to docker.yaml.
- Dozzle: agent v10.4.1 on esh-ml1. The hub's DOZZLE_REMOTE_AGENT still
named ana-ml2's 10.250.50.54 and irv-ml1's retired 10.100.79.3; repointed
to 10.251.50.54 / 10.6.110.50 (fv-ml1 and irv-ml1 logs were missing).
- Documented nh3-dev's Beszel agent failing on NH3 cold start (bind of an
automounted NAS share); revived by hand, fix still open.
- Audit: Skywork-Reward-V2-Llama-3.1-8B is still #1 of 188 on AllenAI's
RewardBench 2 per-sample results; no Skywork V3; the -40M sibling is
vendor-marked experimental. Our AWQ W4A16 quant: 0.847 vs published bf16
0.860 on a 150-prompt sample (within +/-2.9 pt SE), 96.2% pairwise
agreement. Double BOS from vLLM on pre-templated text costs a further
~2.7 pts; callers must send add_special_tokens=false.
- No working consumer: 0 requests since 2026-09-13; Worldtree Domari points
at a dead IP with a non-vLLM schema (reported to worldtree-dev).
- Move: sha256-identical model copy; vLLM v0.24.0 on esh-ml1 :8003 at 0.55
util (KV 1.30x of a 16k request). Parity vs fv-ml1: 149/150 verdicts,
99.8% pairwise signs, raw |delta| median 0.049.
- Gateway /scalar-judge passthrough -> 10.0.50.80:8003; fv-ml1 vllm-reward
removed (~10.2 GB freed on GPU 1). stacks/vllm now holds only vllm-coder.
Prime, 2026-09-25: TEI serves embedding + reranking for esh-ml1 and the fleet
from now on; fv-ml1 retires both once esh-ml1 is up.
- stacks/embed-rerank: vLLM -> TEI 1.9.4 (89- Ada build), same ports
8001/8013, fail-closed truncation (--auto-truncate false; embed
--max-batch-tokens 32768).
- litellm: qwen3-embedding -> esh-ml1 only (hosted_vllm/, unchanged address);
reranker -> huggingface/ provider at :8013 (hosted_vllm/ 422s on TEI's
`texts` body). DB alias reranker-a3-bge-v2-m3 patched to the same target.
- Verified via the gateway against the retiring fv-ml1 seats: embed cosine
median 0.999927 (n=203); rerank top-1/top-3 29/30.
- stacks/vllm: vllm-embed and vllm-rerank-a3 removed (containers retired on
fv-ml1, GPU 1 freed ~6.1 GB); reward + coder unchanged.
- Bake-off record moved to docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md;
CLAUDE.md gains the TEI convention.
esh-pve: NVIDIA 580.178.04 (open modules, DKMS) installed on the host from
NVIDIA's .run and loaded live, no reboot. nvidia-persistenced unit creates the
device nodes before pve-guests; the T400's vfio-pci ids and `blacklist nvidia`
retired. playbooks/esh-pve-nvidia-host.yaml.
esh-ml1: CT 110, unprivileged Debian 12, 10.0.50.80, GPU nodes via devN,
NVIDIA userspace from the same .run (--no-kernel-modules), docker-ce +
nvidia-container-toolkit (no-cgroups). playbooks/esh-ml1-lxc.yaml. Not in the
vzdump job on purpose. DNS esh-ml1.esh.internal.
stacks/embed-rerank: Qwen3-Embedding-0.6B :8001 + bge-reranker-v2-m3 :8013 on
the same vLLM v0.24.0 digest and flags as fv-ml1. Measured parity: embed
cosine FV-vs-ESH median 0.999908 (min 0.999772), inside both self-noise
floors; rerank max |delta| 0.000145 vs floor 0.000181, identical ranking.
litellm: qwen3-embedding and reranker gain an esh-ml1 deployment at order 2
behind fv-ml1 (order 1). Order fallback proven with throwaway groups:
refused primary +0.15 s, host-down primary ~18.7 s per call, dead-only 500.
Also: repaired the DB-only alias reranker-a3-bge-v2-m3, dead since the
fv-ml1 relocation (still named 10.250.50.54); documented the third
unkillable homepage wedge on esh-docker-vm.
After the 2026-09-24 NH3 outage pfi-gx10 stayed off. It has no BMC, and the
ASUS firmware defaults to staying off after AC loss. With Prime's approval,
bytes 3-4 of the UEFI Setup variable were changed from 0x1d,0x00 to
0x1c,0x01, a community recipe that matches factory power-on units. The
readback shows exactly those two bytes changed. The original 85-byte raw
variable (attr header included) is committed here and also kept on the box.
The README records that an OS shutdown is not a test of this setting. It
stays off by design, and two such "tests" were invalid. The only valid test
is pulling and restoring AC, which Prime will do on 2026-09-25. It also
records the revert procedure.
After the 2026-09-24 NH3 power outage every NFS line from nh3-nas failed at
boot: nh3-docker's /mnt/compose and /mnt/backup, and nh3-dev's /mnt/backup.
The NAS is the slowest box to serve NFS, and the plain fstab mounts stayed
down until remounted by hand. nh3-dev's /mnt/smithy already used
x-systemd.automount, and the same boot shows it self-healing on its next
access. playbooks/nh3-nfs-automount.yaml brings the other lines to that
shape (_netdev,nofail,x-systemd.automount,x-systemd.mount-timeout=30).
Hard-mount semantics are unchanged; only WHEN the mount happens moves.
Applied and read back on nh3-docker and nh3-dev: every nh3-nas line carries
automount, each automount unit is active, and each share is NFS-mounted after
access. A second run on nh3-docker is a no-op. The fstab check judges only
errors the rewrite ADDS, because nh3-dev's installer cdrom line has a
pre-existing verify error. The first run caught that and aborted without
writing.
servers/nh3-pve/README.md now lists each guest's expected state after a
power loss: nh3-laser is on-demand and stays off (Prime), pbs-nh3 now has
onboot=1 (it had none and stayed down), and pfi-gx10 is bare metal that
does not auto-power-on.
Operator directive 2026-09-24: clock times are written in 24-hour local
(US Pacific) format, four digits, no colon. Claude seats get it from the
global CLAUDE.md (dotfiles fe63a67). Codex (~/.codex/AGENTS.md) and Grok
(~/.grok/rules/00-fleettools.md) read this file through symlinks, so the
same rule goes under Non-negotiables here.
Operator ruling 2026-09-24. On ana-docker the stack is `docker compose
down`: the container is removed and port 7878 is closed. Kept for revival:
- the data dir /opt/docker/conf/task-board/data (tasks.db, last written
2026-09-11)
- the task-board:local image
- stacks/task-board/ and the host's compose + .env
The Uptime Kuma monitor (id 3) was deleted before the stop so it could
not page, and its row is removed from monitors.yaml. Homepage drops the
card on its own, since it reads the container's labels.
Hooks: the container log showed no hook POSTs in 30 days. The only
traffic was open browser tabs holding /events, and the plugin was already
uninstalled on nh3-dev. Removed the paragraph that told sessions to call
task_* tools (CLAUDE.md, and the fork-fleet.sh template that seeds new
repos), the vestigial TASK_BOARD_SESSION env in .claude/settings.json, and
the listings in README and FLEETTOOLS.
Global CLAUDE.md now defines how to reach Miranda: every message becomes a
summary on Prime's phone, URGENT means she phones him, and no noise. The
project section had prefixed every message URGENT and said 'when unsure,
send it', which contradicted that. It now points to the global section for
mechanics and keeps only what is infra-ops-specific: the standing order
covers matters already fixed, the consequence test, and the channel's
dependency on svos.service and hermes-gateway (a routine svos restart is
about 100s of deafness).
Operator standing order 2026-09-24: infra-ops informs Miranda of every
high-urgency matter, fixed or needing the operator. Records the send form
(URGENT — subject, what/state/operator-action body), the consequence-based
urgency test, and the channel's dependency on svos.service and
hermes-gateway, with a fallback route for outages that take those down.
Round trip verified: test thread 01M3A0RP4Q8T0KNGH8TMFSNDA6, ack in ~1 min.
The 2026-09-14 normalization (826a63b) moved /opt/docker/compose/<stack>
dirs to root:docker 2775. git refuses a worktree whose top-level dir has a
different owner, so yt-voice-clipper's webhook auto-deploy on irv-ml1
(git fetch/reset as lkraven) failed with 'dubious ownership' from then on.
Found 2026-09-24 when v0.3.13 did not land.
The fix keeps the convention: one safe.directory entry per repo for the
deploy user. Applied on irv-ml1 and the deploy re-run: main 2af9b3a,
/diagnostics reports 0.3.13. A sweep of the other normalized hosts found no
other checkout deployed by a non-root user.
Three of the four failures in the week to 2026-09-23 (09-19, 09-21, 09-23)
had one signature. The Cloudflare zone lookup returned an empty body at its
15s cap, a bare json.load crashed with tracebacks instead of a cause, and the
script carried empty IDs on to a PATCH against zones//dns_records/.
Cloudflare rejected it, so there was no DNS impact, but only by accident.
- Cloudflare calls go through cf(): 3 announced tries, and a call counts
only when the body says success:true.
- success:true is not trusted as shape. pick() validates every body and
prints only the fields asked for, or one line saying why not. It requires
exactly one zone named phasefinal.com and exactly one A record named
headscale.phasefinal.com, each with a non-empty id and content. Two A
records are refused rather than half-updated, and an empty id can no
longer shift the content into the id slot.
- No write without both IDs. The run ends on a confirmation that the record
now reads the new address. The previous final line was an echo whose exit
status was always 0, even when the parse inside it failed.
- Only a global unicast IPv4 is published (python ipaddress is_global).
Loopback, RFC1918, link-local, CGNAT and documentation ranges are retried
and then refused.
- curl -q as the first argument ignores ~/.curlrc, so a verbose config can
never log the bearer token. The vault CLI path is quoted. The empty
data-array expansion is safe under set -u on bash < 4.4.
Tests: services/headscale-ddns/test_headscale_ddns.py, 12 cases with curl,
the vault CLI and sleep stubbed. Each failure case asserts the FATAL line's
stated reason, so a run that died earlier for an unrelated cause cannot pass
it. The documentation-range fixtures (203.0.113.x) were themselves rejected
by the new public-IP guard: a free positive control.
Live: a manual run and a unit run both printed "unchanged 70.230.226.88",
Result=success. Cross-model bug-hunt (heid "Talus", Gróa + seat): all 8
findings folded.
elway's sudo upload did scp-as-user then `sudo mv`, and mv keeps the owner,
so every file it installed "as root" (systemd units, /etc configs, root-run
scripts) ended up owned by the SSH user. A sudoers drop-in installed that
way would be rejected by sudo outright.
elway:
- Sudo uploads now chown to root:root by default. Playbooks can override
with `upload.owner:` and ad-hoc runs with `--owner`. An owner is refused
on a non-sudo upload, and `--owner` outside an ad-hoc `--upload` is an
error rather than silently ignored.
- Ownership and mode are applied to the STAGED file, then a single mv
publishes it, so a failed chown can no longer leave the live path owned
by the SSH user. chown runs before chmod so setuid bits survive. A trap
removes the staged file on every exit path.
- A directory dest is refused before anything moves. mv would otherwise
drop the file inside the directory under a staging name.
- `mode` was spliced unquoted into the remote root shell line. It is now
validated as octal (^[0-7]{3,4}$) and shell-quoted, on both the sudo
and non-sudo paths.
- `mode` and `owner` must be quoted YAML strings. YAML 1.1 turns a bare
0644 into 420, a bare 1000:0 into 60000 and a bare yes into True, and
str() of each is a different value that looks valid. All 85 playbooks
already quote them.
- New preflight(): every step is resolved before any remote action. An
undefined var or a templated owner that resolves badly is now refused
up front, not after earlier steps have already changed the host.
scripts/fleet-ownership-audit.sh (new, read-only) finds files in root's
territory owned by a normal user. Tier A lists /etc, /usr/local, /root and
cron, judging symlinks by their target. Tier X covers files that root-run
units exec from /opt, /srv, /home or /volume1, reading drop-ins and quoted
paths. Tier B only summarises /opt app trees. A host that is unprivileged,
whose find fails, or whose run dies is reported INCOMPLETE (exit 5), never
clean. Completion is marked with a per-run nonce.
Verification: 24 unit tests in scripts/test_elway.py. Live checks on
esh-docker-vm: default lands root:root; the override lands infra-ops:adm at
4755 with setuid intact; a bad group fails with dest untouched and no
staging left; a directory dest is refused; a bad mode is refused before
any copy. Audit positive controls on nh3-dev (a drop-in-only root Exec, a
quoted path containing a space, a symlink to a user-owned target) were all
flagged; esh-docker-vm negative control was 0. Probes removed. Cross-model
bug-hunt (heid, Gróa arm + seat) findings folded.
Operator decision. paperless-ngx's database lives on esh-vm-db and is backed
up at the source by that host's fail-closed pg_dumpall. esh-docker-vm's
second copy had failed auth every night since 2026-04-24 behind a WARN. Its
`> paperless.pg_dump` redirect left a 0-byte file in every snapshot
(confirmed in snapshot 6ec9f74f), which looked like a dump but held nothing.
The block was the only consumer of /etc/restic/dbcreds.env, so the creds
loader is gone and the template dbcreds.env.example is deleted. The host
file was moved (not deleted) to /var/lib/restic/repair-20260923/.
Paperless's media volumes are still captured under /var/lib/docker/volumes.
Also fixes ownership. elway's sudo upload does scp-as-user then `sudo mv`,
so the hook deployed at 08:04 landed infra-ops:infra-ops even though root
executes it. Both esh-docker-vm playbooks now chown it back to root and
verify root:root 700.
Verified: the live hook hash matches canonical (e0d3ddcef1bddf43), and
the manual backup saved snapshot decfae71 with 3 staged dumps and no WARN
lines.
Uptime Kuma moved from esh-docker-vm to ana-docker on 2026-09-22. The
pre-backup hook's fallback lookup, `docker ps | grep -E "uptime.kuma"`,
then matched nothing and exited 1. Under set -euo pipefail that aborted the
hook, and resticprofile treats a failed run-before as fatal, so no snapshot
was taken from 2026-09-22 01:00 until this fix (backup-freshness: 54h stale).
The block is removed rather than guarded because there is nothing on this
host left for it to back up. The header now records the invariant the
"blocks only WARN" promise depends on: every optional-service lookup must sit
inside an `if` test or end in `|| true`. The remaining blocks were checked
and all do.
Deployed with playbooks/esh-docker-vm-restic-drop-kuma-block.yaml (the
pre-fix hook is kept in /var/lib/restic/repair-20260923/). The live hook
hash matches the canonical copy (43e6bea8b8569602). The manual backup saved
snapshot 6ec9f74f, and backup-freshness now reports all backups fresh.
A deliberate `systemctl --user restart hermes-gateway` paged infra-ops as a
FAILED unit (msg 3716) while the unit was already back up. Cause is a Hermes
v0.21.1 race: the planned-stop marker watcher runs the shutdown handler before
systemd's SIGTERM, consumes the marker, and the SIGTERM re-runs the handler,
which then classifies the stop as unexpected and exits 1.
Corrects the README claim that OnFailure never fires on a deliberate restart:
that holds only when the main process exits with a success status. Measured on a
throwaway unit (3/3 paged without SuccessExitStatus, 0/3 with it, and crash
restarts are unaffected), and a survey of every stop on nh3-dev since 09-15
found hermes-gateway to be the only unit that does this.
The host-side fix is drop-in hermes-gateway.service.d/
20-planned-stop-exit1-is-clean.conf (SuccessExitStatus=1). No alarm coverage
is lost: Restart=always ignores the classification, and StartLimitIntervalSec=0
means the unit can never reach `failed` from a start failure.
In-flight rewritten for 2026-09-22 evening. The operator approved both carried
calls ("decide to do both pending"): the NRestarts flap sampler is to be BUILT
(163bb97) and the restic content-assertion ruling is RATIFIED as-is (ba60fda).
Recorded in Recent decisions with tracking pointers, and their modality flipped
in the handoff from do-not-touch to genuine pending work.
safe-rm: installed on nh3-dev, delegated to infra-hermes, now fleet-wide 6/6
with acceptance met. Wiring ruled to /etc/bash.bashrc above the case-$ guard
rather than /etc/environment -- an rc self-guards, smaller blast radius, covers
bash-over-ssh. ⭐ The guarded acceptance test caught infra-hermes's own silent
wiring failure in flight: a probe that cannot destroy what it tests also cannot
lie about it.
⚠ Recorded twice over because it will otherwise be trusted for the wrong class:
safe-rm is a BLACKLIST. It refuses rm -rf /home and deletes an unset-variable
path without complaint. set -u is the actual cover for the habit that prompted
the install.
D-0010/D-0011: genuine, operator-approved, misrouted to this seat by pane_find
matching a ROLLING PANE TITLE. Belayed both, nothing lost. svos-dev owns the fix
and found the sharper half -- pane_send structurally cannot deliver a harness
command, so D-0010 promised an unachievable /clear and its receipt reported
success anyway. {"sent": true} is a claim about transmission, never effect.
Live thread carried: restic/ana/esh-docker-vm has drifted 36h -> 44h against a
48h threshold while every other repo sits at 12h. Drifting, not static.
Corrects my own framing from earlier today. I told the operator D-0010's /clear
"cost nothing by luck of the delivery mechanism". svos-dev verified the code:
bifrost.py:1322 prefixes every relayed string with the card id, so a slash
command never begins the line. pane_send STRUCTURALLY cannot execute a harness
command in a target pane -- a property, not luck.
Which makes the card worse than a misroute. It promised "type /clear into
terminal_2 and clear booth-dev's context"; that effect was never achievable at
all. The executed event then recorded {"sent": true} -- true about the bytes,
false about the effect. Three defects in one approved action, and only the
first was visible.
Recorded because it is the sharpest instance of the day's pattern and the only
one that cost something outside an agent's triage time: it consumed an OPERATOR
APPROVAL. Vuong approved an action believing it would take effect and the
system told him it had.
General form worth carrying beyond SVOS: {"sent": true} is a claim about
transmission, never about effect. Any receipt that cannot distinguish the two
will eventually report success for something that did not happen.
The failed-START notifier built earlier today had its first REAL firing at
15:28: headscale-ddns.service exited 1 after succeeding all afternoon. The
detection worked. The alarm was also useless, and that is the finding.
Both failure paths exited 1 IN SILENCE, so the message said "exit status 1" and
nothing else. An alarm you cannot act on costs the same triage as no alarm at
all -- the notifier did its job and the subject script had no diagnostics for it
to carry.
Cause was transient and harmless: icanhazip.com did not answer inside its 10s
cap, so the IP came back empty and the regex guard refused it. No DNS impact --
the record already held the right address, verified against 1.1.1.1 before
touching anything, and the next timer run succeeded. Arithmetic confirms it:
~17s vault read + 10s curl timeout = 27s against the 28s the failing run took.
Fixed, both verified by making them fail:
- every exit path names its cause; a missing vault key names the key, an
EMPTY token is distinguished from a failed read, and a dead WAN lookup adds
"DNS left unchanged" because that is the fact the reader needs
- the WAN lookup retries 3x with ANNOUNCED attempts -- one third-party blip
should not page a human, and a silent retry would hide a degrading
dependency
⚠ ALSO: this script was not tracked anywhere. A fix to the thing every mesh
client resolves through lived on exactly one disk. Script, unit and timer are
in the repo now.
Measured and recorded: the vault read is 17 of the script's 18 seconds, every
10 minutes. Not a fault, but it bounds any retry budget and it is fleet-wide --
svos-dev's alarm unit carries the same 17-second note.
In-flight rewritten for 2026-09-22. The Booth is booth-dev's now (extracted to
vh/booth, v0.2.0 shipped by them); infra-ops retains only its Uptime Kuma
monitor.
Eight decisions recorded, and three OPEN OPERATOR DECISIONS filed under Recent
decisions with commit tracking pointers rather than under in-flight, per the
deferred-work rule: the NRestarts flap sampler (163bb97), safe-rm, and
confirming the restic content-assertion ruling that reached me relayed by
svos-dev rather than Miranda (ba60fda).
Detail file for the day's throughline: five instrument errors, one shape -- a
tool that enumerates 'things that are fine' has selected against its own
subject. Four were mine, one a peer's, and mine included making theirs, on the
same day we were both cataloguing the habit. Written down because a rule both
parties broke while discussing the rule needs to live where a fresh session
reads it.
Two loose threads carried: talk.service failed while still serving on :8092,
and restic/ana/esh-docker-vm 36h behind the fleet.
Adding nh3-dev to the host list fixed the instance. This fixes the class, on
svos-dev's framing: a hand-maintained list of things to watch, sitting beside a
NAS that already knows which repos exist, means the next repo added is
unwatched BY DEFAULT and nothing says so. The list of what to check can
silently disagree with the set of what exists -- the same shape as every other
instrument fault found this day, only slower-acting.
The check now asks each NAS. A directory is a repository when it has a
snapshots/ child, which cleanly separates real repos from container dirs
(/volume1/Backup/restic/repo/ holds ana|esh|nh3 namespaces and no snapshots of
its own -- verified rather than assumed before building discovery on the
layout).
The hand-written list survives DEMOTED to an EXPECTED set, used only to report
a repo that has VANISHED. Two facts that would otherwise both read as silence
stay distinct:
"a repo exists that nobody watches" -> impossible now, it is discovered
"a repo we expected is gone" -> EXPECTED REPO NOT FOUND
Preventive, not corrective: all 8 repos currently discovered are already in the
expected sets, so this found no live gap. It removes the possibility of the
next one.
Controls run, since a check only ever seen passing is untested: a bogus
expected repo reports EXPECTED REPO NOT FOUND and turns the verdict STALE;
unchanged expectations still report all-fresh; all 8 repos report their age.
Observed while testing, not a fault: restic/ana/esh-docker-vm is 36h old
against 12h for every other repo. Inside the 48h threshold so correctly green,
but it is a day behind the fleet and worth a look.
Operator ruling 2026-09-22 (relayed via svos-dev): have the freshness check
assert snapshot content rather than REST-server reachability, so the green
light is a statement about DATA rather than about a daemon.
⚠ FIRST, A CORRECTION TO MY OWN REPORT. I ran `grep -ic restic` against
backup-freshness-alert.sh -- the WRAPPER -- got 0, and told the operator that
nothing on the fleet verified restic. Wrong. check-backup-freshness.sh has
always checked restic snapshot ages, for seven hosts. I grepped one file of a
two-file tool and generalised, which is the identical error a peer had just
made with dev-backup, made while correcting them.
THE REAL GAP was narrower and still real: the NH3 host list read
`irv-ml1 nh3-docker` and omitted nh3-dev -- the repo holding every Claude Code
session transcript, althing routes, hermes run history and Miranda's
conversation. /volume1/Backup/restic/nh3-dev/ has always existed and always
been written nightly; it was simply never enumerated, so a total failure of the
one repo nobody could reconstruct would have gone unreported indefinitely.
THE CONTENT ASSERTION, built on svos-dev's ladder (list proves the repo
answers; ls proves the index; check proves structure; RESTORE proves the bytes
come back):
- CONJUNCTIVE: latest snapshot newer than MAX_AGE_H AND the probe path
present AND a restore returns non-zero bytes. "A snapshot exists containing
X" is satisfied by a three-month-old one; "the latest is recent" is
satisfied by an empty one. Age alone was the old problem; content alone is
the same problem rotated.
- IT RESTORES. Metadata is what survives the failures worth fearing -- a
pruned or partially-written repo can list a path whose blobs are gone.
~493KB from ~/.local/state/svos, sub-second. Large paths are not restored:
a repo that returns one file will return others.
- IDENTITY: the timer runs as lkraven, which has no NOPASSWD sudo on nh3-dev,
so the probe hops through infra-ops@localhost. /etc/restic is root-only by
design.
⚠ restic ls ALWAYS PRINTS A HEADER LINE, matched or not. A path absent from the
repo returns 1 line; a real one returned 6. Counting with `grep -c .` read the
header as a hit, so the ABSENT case fell through and reported "blobs gone" --
telling an operator the repository was corrupt when the truth was a mistyped
path. Now `grep -c '^/'`.
TESTED BY MAKING IT FAIL, because a check only ever seen passing is untested:
bogus probe path -> "absent from snapshot"; BACKUP_MAX_AGE_HOURS=1 -> "12h old
(>1h)"; healthy -> snapshot id, age, entries, bytes restored.
Two additions, both from a 2026-09-22 exchange with svos-dev.
THE RULE. Coverage is a property of the backup SYSTEM, not of one job's
configured scope. A peer checked dev-backup.sh, found SRC=$HOME/development,
and reported to the operator -- with specifics and unhedged -- that five
home-directory paths including Miranda's entire conversation had never been
backed up anywhere. All five were in that night's restic snapshot. dev-backup
is the hourly job for one directory; resticprofile is the daily job covering
all of /home/lkraven. Checking one job and generalising to the system produced
a confident, false, escalated claim. The runbook now carries the query that
answers the question properly.
THE GAP THAT VERIFYING IT EXPOSED, and it is worse. grep -ic restic against
scripts/backup-freshness-alert.sh returns 0. The checker inspects PBS guest
ages and pings the rest-servers for liveness -- which confirms the server
answers, not that a snapshot was written. If resticprofile stopped entirely the
light would stay green, correctly by its own definition, forever. Restic holds
the whole home directory; PBS holds VM images. The layer with the granular data
is the unwatched one, and the light is not merely blind but actively reassuring
about a system it cannot see.
Recorded as an open gap rather than patched, because fixing it changes what an
existing green light means and people have been reading that light for months.
svos-dev found an interaction between the two detectors by doing the recovery
rather than reasoning about it: clearing a failed unit is REQUIRED before
systemd will start it again, and reset-failed zeroes NRestarts in the same
motion. Verified here -- a start-limited unit refused to start until
reset-failed, which took NRestarts 3 -> 0.
So the standard recovery from a failed-START alarm erases the history a flap
sampler would depend on, routinely rather than as an edge case. A unit that
fails, is cleared, then flaps starts its history from zero, delaying the first
flap alarm by the whole threshold -- exactly the window where a just-fixed unit
is most likely still sick. Mitigation recorded: store a last-seen timestamp so
a counter going BACKWARDS is itself detectable as a reset rather than read as
quiet.
Also recorded as its own shape: 'nothing to count is not the same as not
counting'. The first manual-restart check read NRestarts flat and would have
concluded a sampler goes blind whenever a human touches a unit. Artifact of the
test -- that unit had already hit its start limit and stopped auto-restarting,
so the instrument was reading correctly from a subject that had stopped
producing. Verify, then ask what the verification could not have seen.
OnFailure fires when a unit enters failed. A unit that keeps re-earning its
retries never enters failed, so the alarm never fires for it. svos-dev warned
this was the consequence of the interval divergence; measuring it showed the
gap is most of the box.
MEASURED: a unit shaped like booth/althing-po-herald (Restart=always,
RestartSec=5s, burst 5, interval 10s) ran 9 restarts over 45 seconds cycling
active/activating and NEVER reached failed. No alarm would have fired.
Covered: the 7 timer-driven oneshots (Restart=no, so any failure lands in
failed immediately) and svos.service (burst 3 per 5min -- it genuinely gives
up). NOT covered: ten daemons that flap instead, and hermes-gateway, whose
start limiting is disabled outright.
svos.service's divergent 5min window is deliberate and load-bearing (operator
ruling 2026-09-11, 'fatal both ways'). NOT to be harmonised: that would restore
the flapping the ruling forbids AND silence the alarm on the one daemon it
currently works for. Whether the OTHERS move to 5min is a behaviour change for
ten services and an operator call.
Also recorded, svos-dev's framing of the two installer bugs: a tool that
enumerates 'things that are fine' and acts on them has selected against its own
subject. Neither bug failed randomly -- both were anti-correlated with purpose,
working better the healthier the fleet looked.
Two defects, both found by the tool failing to cover a unit that was already
broken. talk.service has been sitting in `failed` (exit 143) while its
containers keep serving 200 on :8092 -- precisely the "unit died, the thing
keeps serving, nobody knows" case this alarm was built for -- and the installer
had not hooked it.
1. SELECTION: --state=running skipped anything not already healthy. A unit that
is down at install time never got hooked, which inverts the tool's purpose.
Now the union of enabled unit-files and all loaded units.
2. PARSING, and this is the sharper one: systemd decorates a FAILED unit with a
leading "●", so `awk '{print $1}'` returned the bullet instead of the name,
and the sanitiser reduced it to an empty string and dropped the row. The
parser silently lost exactly the rows that matter. --plain suppresses the
decoration. Same shape as every other instrument error today -- it reported
cleanly while looking at the wrong thing.
Selection now 23 units. Deliberately INCLUDES the timer-driven oneshots
(dev-backup, ha-backup, fleet-tls-cert-check, headscale-ddns,
seat-inventory-drift, brokkr-landscape-scan, soong-ci-relay): a backup or a
cert check that fails silently is the same class, and all seven were clean at
install time so they are not a known noise source. EXCLUDES dbus, gpg-agent and
dirmngr as OS plumbing, the notifier's own template and instances, and
svos-failed-alarm -- hooking an alarm with an alarm is a loop with extra steps.
svos-dev challenged a claim this README made -- that a crash-loop yields one
message per episode -- with a measurement: one 2026-09-19 boot-gate refusal on
svos.service produced FIVE transitions into failed, and the operator got five
messages.
Measured here before accepting it, because a peer's number is still a number
someone else took: a unit with Restart=on-failure, burst 3, interval 30s
produced 7 journal failure lines and exactly ONE notifier invocation. So the
multiplier is not universal -- it needs retries spanning start-limit windows or
an external restarter. svos.service carries StartLimitIntervalSec=5min, which
is how it accumulated five.
Both conditions exist on this box, so the guard goes in as cheap insurance
rather than as a fix for something proven here. The README now states both
numbers and which restart policy each of the twelve units carries, since that
is what decides the exposure. Noted with it: hermes-gateway has start limiting
DISABLED, so it retries forever and may never reach failed at all -- worth
knowing before trusting this alarm to cover it.
Design, taken from svos-dev's shape:
- Keyed on a hash of the CAUSE (unit + result + exit status + the shape of
its last error lines), never the unit name alone. A genuinely different
failure inside the window is a new fact and must still page; suppressing by
unit would hide a second, worse failure behind the first.
- Suppression is LOGGED to suppressed.log, never silent. An alarm that
quietly declines to fire is indistinguishable from one that is broken.
⚠ The first test of this appeared to show the cooldown not working, and the
test was wrong rather than the code -- it invoked the script BEFORE failing the
unit, so the two calls legitimately saw different states and computed different
fingerprints. Re-run the way systemd actually invokes it: same cause 3x -> 1
sent, 2 suppressed and logged; a different cause inside the same window -> sent.
svos-dev found the failure neither Beszel nor Uptime Kuma can see, and
measured it: a config change on 09-19 made svos.service refuse to boot, the
RUNNING process predated the change and kept serving, and the service sat one
restart from dark for three days. Every uptime probe was green and correct --
the thing was up. The signal that catches this is failed-START, not down.
A count after that conversation: 13 running user units on nh3-dev, ZERO with
an OnFailure hook. Including althing-po-herald, whose silent failure cuts
infra-ops's own mail delivery -- a blind spot in the notification path every
other alarm on this fleet depends on.
One template, one drop-in, 12 units hooked (dbus excluded as systemd's own
plumbing). Not noise: OnFailure does not fire on a clean restart or a
deliberate stop, and with Restart=on-failure a crash-loop yields one message
per episode rather than one per attempt.
⚠ %i, NEVER %I -- and the acceptance test is the only reason this is right.
%I unescapes the instance name and systemd escaping maps "-" to "/", so the
first run delivered a message for "onfailure/selftest.service", a unit that
does not exist, with a spool path that tried to create directories.
althing-po-herald.service would have arrived as althing/po/herald.service.
It "worked" -- mail was delivered -- which is exactly the kind of success that
is not one. The referring unit passes %n raw, so literal %i is correct.
⚠ The notifier never reports itself, guarded twice on purpose: the template
carries no OnFailure, and the script bails on its own instance name. A
notification loop is the one bug that pages you forever.
Delivery is spool-first. postbox has no outbox, and an alarm for moments
nobody is watching must survive the post office being one of the things that
is down. If the herald itself fails, the message still REACHES the post office
(postbox talks to it directly; the herald only delivers inbound pokes) -- not
pushed, but stored for the next read.
Acceptance-tested twice against a unit that exits 42: once to catch the %I
bug, once to confirm the fix. Test unit removed.
svos-dev's call and the better one: The High Seat is the English name for
Hlidskjalf, the seat Odin watches all the worlds from, and watching every
session at once is what the board does. mdi-monitor-dashboard described the
artifact; the eye describes the job.
Deployed and verified in /api/services, not assumed.
Requested by svos-dev relaying the operator, 2026-09-22. Reversible work, so
the relay is fine to act on without escalating.
Manual services.yaml entry rather than container labels, because SVOS is a
user-level systemd unit (svos.service) on nh3-dev and nh3-dev is NOT one of
the five hosts in docker.yaml -- Homepage has no Docker API to discover it
through. Same reason the Booth, WhereTF, talk and the infra-hermes seat are
listed by hand, and the comment says so at the entry.
⚠ siteMonitor is "/" deliberately. There is no /api/health on this service:
that path 404s, and a monitor pointed at it would report the board
permanently down while it serves perfectly. svos-dev flagged it and it is
verified here -- / returns 200 and serves the SPA (<title>The High Seat</title>).
Group is Apps, which exists in settings.yaml's layout with tab: Main. An
invented group name gets no tab and renders on ALL tabs, which is how
Scriberr's "AI Systems" leaked across the whole dashboard in August.
Icon mdi-monitor-dashboard is my choice -- svos-dev explicitly did not guess
at one and offered to take a different suggestion.
Verified in a browser, not just in the API: the card renders in Apps with a
green site-monitor at 28 ms.
TAILSCALE IPv6. `tailscale status` had been reporting, continuously:
2 add route failures; first was: permission denied
adding address fd7a:115c:a1e0::6/128 from tunnel interface: permission denied
with tailscale0 carrying only 100.64.0.6/32 while headscale had assigned it
an IPv6 address it could not use.
Not a capability problem -- tailscaled runs as root with the full bounding
set. /etc/sysctl.conf:59 sets net.ipv6.conf.default.disable_ipv6=1, and
`default` is inherited by NEWLY CREATED interfaces; tailscale0 is created at
daemon start, inherits it, and the kernel returns EPERM for every attempt.
Fixed with a scoped systemd drop-in rather than flipping the global default.
That line carries no comment, but IPv6-off-by-default on a host with ~26
docker bridges reads as deliberate, and changing it would hand IPv6 to every
future bridge as a side effect of fixing Tailscale.
⚠ It must be ExecStartPost, not /etc/sysctl.d. A sysctl.d entry for a
per-interface key is applied at boot, BEFORE tailscale0 exists, and is
silently ignored -- the setting would look present and do nothing.
Also learned: setting the sysctl on the LIVE interface is not enough.
tailscaled only attempts the address at startup or on a netmap change, so
the verify failed for 60s until the daemon was restarted. Restart is part
of the operation, not an afterthought.
Verified: fd7a:115c:a1e0::6/128 present on tailscale0, health clean, mesh
and services (arbo, ytvc) up.
README. It documented the pre-headscale topology as current -- "Reachable
IP: 10.100.79.3 (WireGuard tunnel endpoint)", "No direct LAN access", and a
refresh caveat telling you to bring WG up. That sends anyone triaging this
host to the wrong layer, which is the exact tax the file exists to prevent.
Now: mesh primary at 100.64.0.6, LAN 10.6.110.50, and wg0 documented as
STILL UP with a live peer -- tailscale uses that address as its direct
endpoint, so it is load-bearing, not vestigial.
Recorded with it, because these cost hours tonight and will cost them again:
- Irvine is a TENANCY behind a Fortinet PFI does not control. Its TLS
inspection breaks Tailscale's relay and control channels (41 cert
warnings/week, 4 control-plane episodes in 14 days). Usually invisible
because direct peer paths carry the data. No fix on our side.
- Diagnose reachability with `tailscale ping`, NOT the status output:
headscale said "online" and status said "active, 19.7 GB" while nothing
on the host answered. Both are last-known state; only a round trip is
liveness.
- The ~26 docker bridges make tailscaled report captive portals.
Two stale claims corrected: the hostname rename it called "pending" is done,
and `ollama` is listed as running on :11434 when it is gone -- verified, no
unit file, nothing listening, no process. It is banned fleet-wide.
NAMES. Homepage already answers "what is this service called", so the monitor
name is now that name verbatim -- a second naming authority is how drift starts,
and an alert reading "[Uptime Kuma] Beszel hub is DOWN" sends you hunting for a
card that does not exist. Only two rows moved (Beszel hub -> Beszel, Dozzle hub
-> Dozzle); the " hub" suffixes were mine, not the services'.
The remaining mixed case is deliberate and is now documented as such. talk, vor
and task-board are lowercase on Homepage and in their own repos; title-casing
them here would make this board disagree with both. What actually looked messy
was scripts/kuma's own ASCII-ordinal sort, which buried every lowercase name
below every capitalised one. Fixed to case-insensitive.
⚠ RENAME SAFETY, which this pass needed and did not have. The seed keys on NAME,
so editing a name would have read as a brand-new monitor: added fresh, with the
old row orphaned, still checking, still alerting, and holding all the history.
`rename_from:` names the old row for one run. Verified: both renamed monitors
kept their IDs and all 67 heartbeats.
Added with it, an orphan warning for any row on the board the spec no longer
names -- because a forgotten monitor keeps paging. Its first cut diffed against
the PRE-EDIT snapshot and so cried wolf on its own successful renames; it
re-reads the board now. A warning that fires on its own correct work is worse
than no warning.
STATUS PAGE + WIDGET. The Homepage uptimekuma widget reads a PUBLISHED status
page (/api/status-page/<slug>), not the admin API -- which is why the widget
labels were deliberately absent from the rebuild: a dashboard widget pointed at
a 404 is the suspected mechanism behind both of Homepage's unkillable D-state
wedges, so shipping one on purpose would have been daft.
The page now exists at slug `nethealth` (the pre-rebuild slug, so old references
still resolve) and is DECLARED IN monitors.yaml, applied by `kuma seed`. Same
principle as the notification channel: a from-scratch rebuild restores the page,
the channel and the monitors together, and nothing the widget depends on lives
only in Kuma's database. Verified in a browser: "13 SITES UP / 0 SITES DOWN /
100% UPTIME" on the dashboard.
⚠ saveStatusPage calls imgDataUrl.startsWith() unconditionally, so passing null
throws and leaves the page CREATED BUT EMPTY -- which reads as success from
/api/status-page (200, correct title) while the group list is silently blank.
Pass "" instead. Commented at the call site.
Reported by draupnir after the Booth v0.2.0 consumer note, and it is the
higher-impact half of what it found: this file is autoloaded by every agent
family on this box through the shared FLEETTOOLS bootstrap pointer, so it is the
first thing a cold session reads about the Booth. Draupnir was a live specimen —
it read this page earlier in the same session to post a booth, before the
announcement landed, and only luck of scope kept it off the broken surfaces.
Stale and now corrected: `booth asks` presented as the listing verb (it is a
deprecated alias and emits JSON now, not the old table); `<stem>.ask.json` /
`<stem>.answer.json` presented as the storage (one `.marks.json` per booth
replaced them, and the answer sidecar 404s forever, so a remote session polling
it is polling nothing); the `#ask-<stem>` anchor convention.
Not stale, and left alone deliberately: the `data-booth-ask=` placeholder
conventions. The inline placement engine survives v0.2 and is removed in a later
unit, so those still work exactly as documented.
Added: `booth marks` and `booth marks-import`, the marks.json HTTP read path,
the partially-answered-counts-as-open change, and the 409-on-damaged-file
behaviour.
THE BRIDGE. `beszel-althing` hardcoded a "[Beszel] " subject prefix and a
Beszel hub footer from when Beszel was its only caller. Routing Uptime Kuma
through it unchanged would have delivered Kuma outages labelled [Beszel],
pointing the reader at the wrong dashboard -- an alert that lies about its own
source is worse than no alert.
Now a route registry: /beszel and /kuma, each with its own prefix, footer and
payload parser, because the tools do not agree on a shape (Beszel sends
{title, message}; Kuma sends {heartbeat, monitor, msg}). Generalising cost a
dict; a sibling service would have cost a second unit, a second port and a
second thing to notice had died.
Renamed beszel-althing -> althing-alert-bridge with it. A service named after
one consumer that carries two is the invisible coupling that sends a future
session looking in the wrong place.
⚠ /beszel IS FROZEN and this refactor proves it rather than claiming it. The
three original tests were kept BYTE-UNCHANGED -- including the one asserting
the exact postbox argv -- and deliver() still defaults to the Beszel route so
they exercise it. A new test asserts the Kuma footer never leaks into a Beszel
body or vice versa. Verified live after the rename: a real POST to /beszel
landed as "[Beszel] BRIDGE RENAME CHECK" with the correct hub footer, read back
from the thread rather than trusted from the receipt.
Payload shapes are parsed HERE, not via Kuma's custom-webhook-body feature,
because Kuma's notification config lives in its own database -- and that
database was destroyed and rebuilt from scratch hours ago. Anything living only
in a tool's DB is lost on the next rebuild; format knowledge belongs in git,
next to a test.
parse_kuma also handles the monitorless case. testNotification and cert-expiry
alerts carry no monitor and no heartbeat, and the first cut fabricated "unknown
monitor is ?" from them -- caught by sending a real one and reading the subject,
not by the suite. Fixed, pinned, and the earlier test asserting the bad
behaviour was corrected rather than worked around.
KUMA IS NOW WIRED. scripts/kuma gained notification support and the channel is
in monitors.yaml, seeded BEFORE the monitors and with applyExisting, so a
rebuild restores alerting and not just detection. Ground truth from the DB:
13 of 13 monitors carry the channel.
⚠ A THIRD instance of the same class of bug, worth naming: notifications() is
pushed as `notificationList` at LOGIN ONLY -- there is no event to ask with. The
first cut cleared the captured value before waiting, discarding the only copy it
would ever be sent, then blocked for the full timeout and reported an empty
list. That reads exactly like "no channels configured" and is a lie. Same family
as the getMonitorList ack-vs-push trap, different shape.
End-to-end, both shapes, read back from the inbox:
[Uptime Kuma] Homepage is DOWN + target + board link
[Uptime Kuma] althing (infra-ops) Testing (no fabricated subject)
[Beszel] BRIDGE RENAME CHECK + hub footer, unchanged
ALTHING CHAMBER RETIRED (operator). Three of its four containers had never
started -- created 2026-09-19, StartedAt epoch-zero, 0 restarts -- so :7881
refused, and nothing was watching it. Only its valkey was running, on the
project's own network with no external consumer. Stack, compose/build/conf dirs
and the local image removed; the Homepage card went with the label.
Homepage sat dead for three days in September while every monitoring tool
reported correctly. Beszel said its host was up -- it was. Uptime Kuma was not
watching it. The dashboard fell into the seam between two working instruments.
Measured before changing anything:
- Beszel: 18 hosts x {Status, CPU, Memory, Disk, Temperature}. Its alerts
table is (system, name, value, min) -- there is NO url column, so it is
structurally incapable of "this endpoint should return 200". Not a config
gap; the data model.
- Uptime Kuma: 6 rows, 2 of them folders. Four real monitors, all firewalls.
- So the two are NOT redundant. They are disjoint, and the service layer
between them was empty.
REBUILT FROM SCRATCH, operator-authorised ("uptime-kuma was never really
used... you can even dump the existing container and config"). Nothing was
migrated, which also skipped the one-way v1->v2 database migration.
- Pinned to 2.5.5. `:latest` is a documented trap now: upstream keeps it on
the 1.x line, so an August 2026 pull produced an image BUILT 2024-12-20
running 1.23.16. Verified by digest -- latest and 1 share one digest while
2/next carry 2.5.5. Pinned exactly, not floating on 2, for the same reason.
- Moved esh-docker-vm -> ana-docker. House placement rule puts cross-site
services beside the Beszel and Dozzle hubs, and esh-docker-vm has wedged
unkillably twice in four months. A monitor also cannot report the failure
of the host it runs on, so it should not share a failure domain with the
host layer.
- Normalised restart: always -> unless-stopped, which the 2026-08-18 README
flagged as worth doing on the next deliberate touch.
- UPTIME_KUMA_DB_TYPE=sqlite in the compose skips 2.x's interactive database
screen, so the stack comes up ready rather than parked on a form.
scripts/kuma is a first-party Socket.IO client, because both obvious paths are
wrong: there is no REST CRUD API in EITHER major version (server/routers/ holds
exactly two files, /metrics + badges + status pages), and the community wrapper
uptime-kuma-api is abandoned -- last release 2023-09-26, ceiling 1.23.1, no 2.x
support ever.
⚠ getMonitorList's callback returns only {ok:true}; the list arrives as a
SEPARATE pushed monitorList event. Reading the ack yields an empty board that
looks authoritative -- which duplicated all 13 rows on the first re-seed
before the bug was found. The client now waits for the push, and carries a
dedupe verb because of it.
13 monitors seeded from monitors.yaml, keyed on name so a re-run updates rather
than forking the board -- proven by re-running it (0 added, 13 updated), not
assumed. Every URL was probed before being written: all 200. A board that ships
red teaches everyone to ignore it.
Verified: 13 rows, no duplicates, all UP with "200 - OK" read from the database
WITH its WAL (a first read of kuma.db alone showed a stale 26 -- the copy
predated the deletes). Homepage renders exactly one Uptime Kuma card.
⚠ NOT YET WIRED: notification delivery. The board detects but tells nobody,
which is the same gap this work exists to close. The beszel-althing bridge
hardcodes a [Beszel] subject prefix and hub footer, so routing Kuma through it
unchanged would mislabel the alerts. Needs a decision before it is generalised.
vh