althing-seat-daemon took an outside SIGTERM on 2026-09-27 and exited 0, so
Restart=on-failure never revived it; hermes-gateway sat pull-only for five
days with a yellow verdict nobody was watching. Fixed the death mode with a
Restart=always drop-in (also on the jekyll twin, same latent bug) and added
this watchdog for every other death mode: 5-min timer, alert on the second
consecutive non-200 (~10 min sustained), re-alert at most hourly, recovery
mail on green, exit 2 distinguishes a broken alarm wire from a down seat
(backup-freshness precedent). Lifecycle exercised end-to-end against a dead
port before install.
Prime's ruling 2026-10-02. retention.py picks the snapshots to delete:
the newest 48, plus the newest of each of the last 30 days and of each of
the last 12 ISO weeks, counting only days and weeks that have snapshots.
Names that are not exactly YYYY-MM-DD_HHMM are never selected, and the
NAS side refuses any path outside that pattern. The unit fails unless
the number kept equals the number expected. Live run: deleted 1, 0
errors, 48 kept as expected.
Image local/mia:0.1.0 built from stacks/mia: MIA v2 @ bbd8b158 (MIT) with
its pinned submodules, dread-dev's proven Python lock with the torch family
swapped to cu129, and a driver adapted from dread-dev's run_mia.py that
seeds every mesh (fix_random + trimesh's module RNG) and writes
weights_effective into the npz. Weights stay in fv-ml1's shared HF cache
at pinned revisions, mounted read-only.
scripts/mia-run mirrors blender-run: --job DIR is shipped to
fv-ml1:/tank/mia/jobs, one docker run --rm rigs every mesh, out/ comes back.
Acceptance on the four Dread Naught characters: 3.9-4.7 s a mesh (median
of 3) plus 12.7 s model load, peak 3,394 MiB; seeded runs bit-identical
across rotated mesh order; GPU-vs-CPU distances the same size as sampling
noise, with an unseeded GPU run as the positive control.
rsync -a copies a read-only source dir (0555) as read-only, so the hourly
prune's rm -rf could not unlink inside it. From 2026-07-18 every pruned
snapshot was left as a 22-entry husk while the run still logged OK. The
1,740 husks on nh3-nas were removed (0 errors; 48 full snapshots kept).
The prune now runs chmod -R u+w before rm -rf, logs its error count and
the number of snapshots kept, and exits 3 on a failed prune (exit 1 on a
failed rsync) so the systemd unit shows failed instead of passing.
Prime: Scriberr gets the basic fix, v3 stays (no NeMo 3.0.0 surgery). 0002 moves
from proposed/ into the carried set; scriberr-rebuild now applies 0001+0002 by
default (suffix dropout2) and its memory budget becomes a 5,600 MiB regression
guard (Scriberr is on GPU 3). Live on fv-ml1 1602: scripts rewritten from the
patched embed, a 20-min file at 5,502 MiB with retried_gaps reported.
The ~6.5-9 s first-call-per-bucket autotune lived in the container's writable
layer and died on every recreate. 0.1.3 creates /tmp/triton-cache in the image
owned by 10001 so the named volume intern-decision_triton-cache inherits a
writable mount point, and compose mounts it.
Bucket model PROVEN, not inferred: 2,048-token buckets, 16 up to 32,768. After
one warmed call per bucket, 12 random sizes across 8k-32k were all warm (worst
2.09 s); cold entries cost 6.5-9 s. Full cold warm-up 109 s; warm re-run 17 s.
scripts/intern-decision-warmup: one noul call per bucket, MAX_TOKENS from
/health, two-point live calibration of the tokenizer's linear token model (a
single probe overcorrects and the aim oscillates around the bucket edge),
per-bucket wall times, non-zero exit on a missed bucket. Run it after an IMAGE
CHANGE only; the volume carries ordinary recreates (measured: force-recreate,
then a warmed 32k call answered in 2.11 s).
Acceptance on 0.1.3: JevBench 202/231, hard 83/111, 0 diffs / 924; warm 32k GPU
1 peak 15,218 MiB (budget 15,220; a COLD autotune touched 15,224 once, README
caveat); /decide answers. Artifacts in the acceptance dir.
Prime's ask (via the coordinator): investigate the "Parakeet skips
stretches of speech" finding, including other Parakeet weights.
Investigation only; nothing deployed.
Against ground truth (official SCOTUS transcript, Gutenberg #38916) the
drops are real: production v3 loses 140 / 66 clean words per transcript on
the two public files and ~50 on each private one (Whisper-referenced,
Canary-confirmed; adjudicator 129/129 correct on the calibration). Cause:
the v2/v3 0.6B weights collapse deep inside long full-attention windows;
the encoder output is degraded, the audio alone transcribes fine, and
1.1B TDT/RNNT/CTC and CTC-0.6B never do it. Decoding (CUDA graphs, greedy
variants, max_symbols, beam), slice length, local attention, loudness,
resampling and a noise floor do not fix it. Controls: A-vs-A, silence
positive control (>=15 words 36/36), null control, bootstrap floor.
Proposed patch 0002 re-transcribes >=3 s stretches where the audio holds
speech but no word came out (-80 to -90 % lost words on all four
recordings, lower WER, no invented text, +10 MiB) and adds an explicit
PARAKEET_MODEL_PATH with the loaded model recorded in JSON and ModelUsed.
Reviewed at high effort, all findings fixed; built and tested as
scriberr:local-blackwell-a353078-dropout2, not deployed.
scriberr-rebuild: --patches takes DIR[:DIR...]; embeds and seam-checks
both Parakeet scripts (seam-check --standard for the short-audio one).
Scriberr moved to fv-ml1 GPU 3 on 2026-09-30, so the card is no longer idle when the rebuild runs. The memory stage now requires >= 20 GB free instead of an idle card (GPUs 0-2 still fail that) and attributes the peak only to the host PIDs of its own container, captured with docker top alongside the 0.2 s nvidia-smi samples. Verified with five processes on the card: peak 5,496 MiB, identical to the exclusive-card figure.
Carry patches/0001 on our Scriberr build (upstream a353078): adjacent
buffered chunks overlap by 4 s inside --chunk-len and hand over at a word
both chunks transcribed alike, instead of cutting at fixed marks with no
overlap. Pause-aware cutting is included as an opt-in (--pause-search);
it measured neutral once the stitch was right. The Go<->Python CLI and
JSON seam is unchanged.
Bench (4 recordings, 118 min, 3 cut placements each, against a no-cut
whole-file reference; metrics only, private audio stays on fv-ml1):
cuts with an error within +-3 s fall from 52% (93/179) to 22% (41/184)
against a 19% background; floor +-0.08. Positive control: upstream's
cutter +0.33 over background. A-vs-A byte-identical in-process and
across CLI processes. Peak GPU memory unchanged at 5,496 MiB (n=3).
Also found: Parakeet skips runs of >=10 words mid-chunk with any
slicer, upstream's included; not addressed here.
scripts/scriberr-rebuild clones a pinned upstream sha into a new
/opt/docker/src dir, git-apply-checks the patches, builds a distinct
tag, and checks embed, unit tests, the JSON seam (scriberr-seam-check.py)
and the memory budget on idle GPU 3. Deploy stays manual. The upstream
PR is prepared under patches/upstream-pr/ and not opened.
Both verdicts now build the n for the U11b data-deletion gate (three
consecutive PASS batches at off). First real run 20260930T090608Z via the
detached-worktree path: PASS, user median 0.83.
For infra-hermes's daily batch through the U11 off window (operator ruling
2026-09-29 2340; mode revised to off 2026-09-30). Enforces worldtree-dev's
terms: flock plus a refusal while any harness batch runs, code under test =
demo's deployed sha (a detached worktree when the main tree has moved; refuses
on pyproject/uv.lock/packages drift the shared venv cannot honour), no git
commit, no retry. Exit 0 PASS / 1 FAIL / 2 harness error / 3 refused.
Blender is now a mandatory stage in draupnir's pipeline (Prime, 2026-09-28), and draupnir asked
for eight add-ons from extensions.blender.org: SurfacePsycho 0.10.4, CAD Sketcher 0.32.1,
3D-Print Toolbox 1.4.1, STEP Importer 1.2.1, Bool Tool 2.1.0, LoopTools 4.7.7, MeasureIt 1.8.4,
3MF Import/Export 2.7.7.
- stacks/blender/extensions.lock pins each by version and archive sha256.
- scripts/blender-extensions sync builds fv-ml1:/tank/blender-extensions/5.2/system with Blender's
own install-file, pre-warms and byte-compiles it, checks a read-only enable, then swaps it in.
It refuses while the GUI or a blender-run job holds the old directory.
- conf/scripts/startup/fleet_extensions.py enables every package in the System repo: in a timer
in the GUI (after the prefs load), and as --python ahead of the caller's args in
blender-run --extensions (a failed enable exits 1 before the caller's script).
- It also patches SurfacePsycho's sp_overwrite_segment_selection from eval() to literal_eval():
the eval walked past MCP safe mode (control: unpatched ran code, patched refuses).
- blender-run: --extensions (bind mounts via --mount so a missing source fails instead of being
created); USER/LOGNAME set, which CAD Sketcher's getpass needs.
- compose.yaml mounts the repo read-only and the hook into the GUI container. NOT yet deployed.
- scripts/blender-probes/extensions_acceptance.py: one operator run per add-on, safe-mode
compliant. Headless 8/9 online and with --network none; CAD Sketcher sketching is GUI-only.
A Python audit hook saw no network/process events (positive control fired).
draupnir review: the remote job dir was keyed on the basename alone, so two
local dirs with the same name shared one remote dir, and a rerun inherited
stale files. The dir is now <basename>-<8 hex of sha256(abs path)>, and it is
mirrored with rsync --delete, confined to that one directory. There is no rm on
a computed path. Verified: a file deleted locally is gone from the rerun's
remote dir. The /defaults stderr line is documented as harmless noise.
scripts/blender-run launches each call as a docker run --rm of the Blender
image on fv-ml1 GPU 3, capped at 64g / 48 CPUs. It needs no desktop and does
not affect the GUI container's lifecycle. --job DIR stages a local directory
to /tank/blender/jobs/<name>/, runs Blender with that as the cwd, and copies
results back. It always passes --python-exit-code 1, because Blender otherwise
exits 0 when a --python script raises (measured).
Tested headless: Cycles GPU and CPU, EEVEE via EGL, Workbench, an STL
round-trip, and exit codes (3, 7 and 1 pass through). There is no STEP
importer. Written for draupnir's design work, and indexed in FLEETTOOLS with a
detail file.
The MCP server (mcp-for-blender 2.1.1, frozen requirements) runs inside the Blender
container. Its add-on is vendored at upstream 41a18432 (MIT) and started by a
startup hook. scripts/blender-mcp carries the stdio over ssh + docker exec, so the
add-on socket, which runs arbitrary Python with no auth, stays on the container's
localhost with no published port. It also runs there because viewport screenshots
need a filesystem shared by server and Blender. Telemetry is off and safe mode is
on. The hook also defaults Cycles to OptiX on GPU 3, because safe mode forbids
agents from touching preferences.
Verified end to end from nh3-dev: 36 tools; a GPU render of an agent-built scene;
a viewport screenshot; and safe mode refusing 'import os'. Blender left down
(on demand).
esh-ml1 is outside vzdump, so augaman's face gallery reaches backup only
through restic. New playbooks/esh-ml1-restic.yaml installs restic 0.14.0 (the
same Debian package as the other ESH hosts) and resticprofile 0.33.1 (pinned,
sha256-checked). It uploads configs/restic/esh-ml1/ and schedules a daily
0100 PT backup plus a Sunday 0500 PT check to rest-server-ana. The CT runs UTC,
so both schedules name the zone explicitly.
pre-backup.sh is fail-closed: it runs augaman's own backup CLI, and any failure,
including a stopped container, aborts the run. Tested with a stub docker that
exits 1: the run returned 1, and neither the snapshot count nor last-success
moved. The restore was verified at identity level against augaman-dev's
public-domain canary (snapshot fd3061a1: the restored copy's digest over
identities and samples matches the live gallery). That meets the operator gate
for real enrollments.
The repository URL is read through repository-file rather than restic.env.
resticprofile schedule copies env-file values into world-readable systemd
units, which publishes the rest-server password on the env-file hosts
(observed on esh-docker-vm). This is recorded in the backups runbook under
Known gaps, and the playbook verifies no generated unit contains the URL.
esh-ml1 is added to the freshness check's expected ana-side repos and to the
runbook tables.
augaman moves to v0.1.2 (dependency layer keyed on the lock without the
project; per-crop embedding). pytest -m gpu tests/vision passes 3/3 on the
card, and the canary survived the container recreate.
- AMT re-IP'd over WS-Man to static 10.100.250.61/24 (gw/DNS .1): AMT keeps
its old DHCP lease across a VLAN move, so it was unreachable until changed
(reached via a temporary on-link /32 on vmbr0 + ssh tunnel, since removed).
- KVM redirection enabled, redirection listener on, IPS_OptInService
OptInRequired=0; all read back. Password vaulted as nh3-pve/amt-admin.
- scripts/amt-wsman.py: stdlib WS-Man get/put/invoke client (digest auth,
AMT 16 legacy-renegotiation TLS), password from $AMT_PW.
Operator ruling 2026-09-24. On ana-docker the stack is `docker compose
down`: the container is removed and port 7878 is closed. Kept for revival:
- the data dir /opt/docker/conf/task-board/data (tasks.db, last written
2026-09-11)
- the task-board:local image
- stacks/task-board/ and the host's compose + .env
The Uptime Kuma monitor (id 3) was deleted before the stop so it could
not page, and its row is removed from monitors.yaml. Homepage drops the
card on its own, since it reads the container's labels.
Hooks: the container log showed no hook POSTs in 30 days. The only
traffic was open browser tabs holding /events, and the plugin was already
uninstalled on nh3-dev. Removed the paragraph that told sessions to call
task_* tools (CLAUDE.md, and the fork-fleet.sh template that seeds new
repos), the vestigial TASK_BOARD_SESSION env in .claude/settings.json, and
the listings in README and FLEETTOOLS.
elway's sudo upload did scp-as-user then `sudo mv`, and mv keeps the owner,
so every file it installed "as root" (systemd units, /etc configs, root-run
scripts) ended up owned by the SSH user. A sudoers drop-in installed that
way would be rejected by sudo outright.
elway:
- Sudo uploads now chown to root:root by default. Playbooks can override
with `upload.owner:` and ad-hoc runs with `--owner`. An owner is refused
on a non-sudo upload, and `--owner` outside an ad-hoc `--upload` is an
error rather than silently ignored.
- Ownership and mode are applied to the STAGED file, then a single mv
publishes it, so a failed chown can no longer leave the live path owned
by the SSH user. chown runs before chmod so setuid bits survive. A trap
removes the staged file on every exit path.
- A directory dest is refused before anything moves. mv would otherwise
drop the file inside the directory under a staging name.
- `mode` was spliced unquoted into the remote root shell line. It is now
validated as octal (^[0-7]{3,4}$) and shell-quoted, on both the sudo
and non-sudo paths.
- `mode` and `owner` must be quoted YAML strings. YAML 1.1 turns a bare
0644 into 420, a bare 1000:0 into 60000 and a bare yes into True, and
str() of each is a different value that looks valid. All 85 playbooks
already quote them.
- New preflight(): every step is resolved before any remote action. An
undefined var or a templated owner that resolves badly is now refused
up front, not after earlier steps have already changed the host.
scripts/fleet-ownership-audit.sh (new, read-only) finds files in root's
territory owned by a normal user. Tier A lists /etc, /usr/local, /root and
cron, judging symlinks by their target. Tier X covers files that root-run
units exec from /opt, /srv, /home or /volume1, reading drop-ins and quoted
paths. Tier B only summarises /opt app trees. A host that is unprivileged,
whose find fails, or whose run dies is reported INCOMPLETE (exit 5), never
clean. Completion is marked with a per-run nonce.
Verification: 24 unit tests in scripts/test_elway.py. Live checks on
esh-docker-vm: default lands root:root; the override lands infra-ops:adm at
4755 with setuid intact; a bad group fails with dest untouched and no
staging left; a directory dest is refused; a bad mode is refused before
any copy. Audit positive controls on nh3-dev (a drop-in-only root Exec, a
quoted path containing a space, a symlink to a user-owned target) were all
flagged; esh-docker-vm negative control was 0. Probes removed. Cross-model
bug-hunt (heid, Gróa arm + seat) findings folded.
Adding nh3-dev to the host list fixed the instance. This fixes the class, on
svos-dev's framing: a hand-maintained list of things to watch, sitting beside a
NAS that already knows which repos exist, means the next repo added is
unwatched BY DEFAULT and nothing says so. The list of what to check can
silently disagree with the set of what exists -- the same shape as every other
instrument fault found this day, only slower-acting.
The check now asks each NAS. A directory is a repository when it has a
snapshots/ child, which cleanly separates real repos from container dirs
(/volume1/Backup/restic/repo/ holds ana|esh|nh3 namespaces and no snapshots of
its own -- verified rather than assumed before building discovery on the
layout).
The hand-written list survives DEMOTED to an EXPECTED set, used only to report
a repo that has VANISHED. Two facts that would otherwise both read as silence
stay distinct:
"a repo exists that nobody watches" -> impossible now, it is discovered
"a repo we expected is gone" -> EXPECTED REPO NOT FOUND
Preventive, not corrective: all 8 repos currently discovered are already in the
expected sets, so this found no live gap. It removes the possibility of the
next one.
Controls run, since a check only ever seen passing is untested: a bogus
expected repo reports EXPECTED REPO NOT FOUND and turns the verdict STALE;
unchanged expectations still report all-fresh; all 8 repos report their age.
Observed while testing, not a fault: restic/ana/esh-docker-vm is 36h old
against 12h for every other repo. Inside the 48h threshold so correctly green,
but it is a day behind the fleet and worth a look.
Operator ruling 2026-09-22 (relayed via svos-dev): have the freshness check
assert snapshot content rather than REST-server reachability, so the green
light is a statement about DATA rather than about a daemon.
⚠ FIRST, A CORRECTION TO MY OWN REPORT. I ran `grep -ic restic` against
backup-freshness-alert.sh -- the WRAPPER -- got 0, and told the operator that
nothing on the fleet verified restic. Wrong. check-backup-freshness.sh has
always checked restic snapshot ages, for seven hosts. I grepped one file of a
two-file tool and generalised, which is the identical error a peer had just
made with dev-backup, made while correcting them.
THE REAL GAP was narrower and still real: the NH3 host list read
`irv-ml1 nh3-docker` and omitted nh3-dev -- the repo holding every Claude Code
session transcript, althing routes, hermes run history and Miranda's
conversation. /volume1/Backup/restic/nh3-dev/ has always existed and always
been written nightly; it was simply never enumerated, so a total failure of the
one repo nobody could reconstruct would have gone unreported indefinitely.
THE CONTENT ASSERTION, built on svos-dev's ladder (list proves the repo
answers; ls proves the index; check proves structure; RESTORE proves the bytes
come back):
- CONJUNCTIVE: latest snapshot newer than MAX_AGE_H AND the probe path
present AND a restore returns non-zero bytes. "A snapshot exists containing
X" is satisfied by a three-month-old one; "the latest is recent" is
satisfied by an empty one. Age alone was the old problem; content alone is
the same problem rotated.
- IT RESTORES. Metadata is what survives the failures worth fearing -- a
pruned or partially-written repo can list a path whose blobs are gone.
~493KB from ~/.local/state/svos, sub-second. Large paths are not restored:
a repo that returns one file will return others.
- IDENTITY: the timer runs as lkraven, which has no NOPASSWD sudo on nh3-dev,
so the probe hops through infra-ops@localhost. /etc/restic is root-only by
design.
⚠ restic ls ALWAYS PRINTS A HEADER LINE, matched or not. A path absent from the
repo returns 1 line; a real one returned 6. Counting with `grep -c .` read the
header as a hit, so the ABSENT case fell through and reported "blobs gone" --
telling an operator the repository was corrupt when the truth was a mistyped
path. Now `grep -c '^/'`.
TESTED BY MAKING IT FAIL, because a check only ever seen passing is untested:
bogus probe path -> "absent from snapshot"; BACKUP_MAX_AGE_HOURS=1 -> "12h old
(>1h)"; healthy -> snapshot id, age, entries, bytes restored.
NAMES. Homepage already answers "what is this service called", so the monitor
name is now that name verbatim -- a second naming authority is how drift starts,
and an alert reading "[Uptime Kuma] Beszel hub is DOWN" sends you hunting for a
card that does not exist. Only two rows moved (Beszel hub -> Beszel, Dozzle hub
-> Dozzle); the " hub" suffixes were mine, not the services'.
The remaining mixed case is deliberate and is now documented as such. talk, vor
and task-board are lowercase on Homepage and in their own repos; title-casing
them here would make this board disagree with both. What actually looked messy
was scripts/kuma's own ASCII-ordinal sort, which buried every lowercase name
below every capitalised one. Fixed to case-insensitive.
⚠ RENAME SAFETY, which this pass needed and did not have. The seed keys on NAME,
so editing a name would have read as a brand-new monitor: added fresh, with the
old row orphaned, still checking, still alerting, and holding all the history.
`rename_from:` names the old row for one run. Verified: both renamed monitors
kept their IDs and all 67 heartbeats.
Added with it, an orphan warning for any row on the board the spec no longer
names -- because a forgotten monitor keeps paging. Its first cut diffed against
the PRE-EDIT snapshot and so cried wolf on its own successful renames; it
re-reads the board now. A warning that fires on its own correct work is worse
than no warning.
STATUS PAGE + WIDGET. The Homepage uptimekuma widget reads a PUBLISHED status
page (/api/status-page/<slug>), not the admin API -- which is why the widget
labels were deliberately absent from the rebuild: a dashboard widget pointed at
a 404 is the suspected mechanism behind both of Homepage's unkillable D-state
wedges, so shipping one on purpose would have been daft.
The page now exists at slug `nethealth` (the pre-rebuild slug, so old references
still resolve) and is DECLARED IN monitors.yaml, applied by `kuma seed`. Same
principle as the notification channel: a from-scratch rebuild restores the page,
the channel and the monitors together, and nothing the widget depends on lives
only in Kuma's database. Verified in a browser: "13 SITES UP / 0 SITES DOWN /
100% UPTIME" on the dashboard.
⚠ saveStatusPage calls imgDataUrl.startsWith() unconditionally, so passing null
throws and leaves the page CREATED BUT EMPTY -- which reads as success from
/api/status-page (200, correct title) while the group list is silently blank.
Pass "" instead. Commented at the call site.
THE BRIDGE. `beszel-althing` hardcoded a "[Beszel] " subject prefix and a
Beszel hub footer from when Beszel was its only caller. Routing Uptime Kuma
through it unchanged would have delivered Kuma outages labelled [Beszel],
pointing the reader at the wrong dashboard -- an alert that lies about its own
source is worse than no alert.
Now a route registry: /beszel and /kuma, each with its own prefix, footer and
payload parser, because the tools do not agree on a shape (Beszel sends
{title, message}; Kuma sends {heartbeat, monitor, msg}). Generalising cost a
dict; a sibling service would have cost a second unit, a second port and a
second thing to notice had died.
Renamed beszel-althing -> althing-alert-bridge with it. A service named after
one consumer that carries two is the invisible coupling that sends a future
session looking in the wrong place.
⚠ /beszel IS FROZEN and this refactor proves it rather than claiming it. The
three original tests were kept BYTE-UNCHANGED -- including the one asserting
the exact postbox argv -- and deliver() still defaults to the Beszel route so
they exercise it. A new test asserts the Kuma footer never leaks into a Beszel
body or vice versa. Verified live after the rename: a real POST to /beszel
landed as "[Beszel] BRIDGE RENAME CHECK" with the correct hub footer, read back
from the thread rather than trusted from the receipt.
Payload shapes are parsed HERE, not via Kuma's custom-webhook-body feature,
because Kuma's notification config lives in its own database -- and that
database was destroyed and rebuilt from scratch hours ago. Anything living only
in a tool's DB is lost on the next rebuild; format knowledge belongs in git,
next to a test.
parse_kuma also handles the monitorless case. testNotification and cert-expiry
alerts carry no monitor and no heartbeat, and the first cut fabricated "unknown
monitor is ?" from them -- caught by sending a real one and reading the subject,
not by the suite. Fixed, pinned, and the earlier test asserting the bad
behaviour was corrected rather than worked around.
KUMA IS NOW WIRED. scripts/kuma gained notification support and the channel is
in monitors.yaml, seeded BEFORE the monitors and with applyExisting, so a
rebuild restores alerting and not just detection. Ground truth from the DB:
13 of 13 monitors carry the channel.
⚠ A THIRD instance of the same class of bug, worth naming: notifications() is
pushed as `notificationList` at LOGIN ONLY -- there is no event to ask with. The
first cut cleared the captured value before waiting, discarding the only copy it
would ever be sent, then blocked for the full timeout and reported an empty
list. That reads exactly like "no channels configured" and is a lie. Same family
as the getMonitorList ack-vs-push trap, different shape.
End-to-end, both shapes, read back from the inbox:
[Uptime Kuma] Homepage is DOWN + target + board link
[Uptime Kuma] althing (infra-ops) Testing (no fabricated subject)
[Beszel] BRIDGE RENAME CHECK + hub footer, unchanged
ALTHING CHAMBER RETIRED (operator). Three of its four containers had never
started -- created 2026-09-19, StartedAt epoch-zero, 0 restarts -- so :7881
refused, and nothing was watching it. Only its valkey was running, on the
project's own network with no external consumer. Stack, compose/build/conf dirs
and the local image removed; the Homepage card went with the label.
Homepage sat dead for three days in September while every monitoring tool
reported correctly. Beszel said its host was up -- it was. Uptime Kuma was not
watching it. The dashboard fell into the seam between two working instruments.
Measured before changing anything:
- Beszel: 18 hosts x {Status, CPU, Memory, Disk, Temperature}. Its alerts
table is (system, name, value, min) -- there is NO url column, so it is
structurally incapable of "this endpoint should return 200". Not a config
gap; the data model.
- Uptime Kuma: 6 rows, 2 of them folders. Four real monitors, all firewalls.
- So the two are NOT redundant. They are disjoint, and the service layer
between them was empty.
REBUILT FROM SCRATCH, operator-authorised ("uptime-kuma was never really
used... you can even dump the existing container and config"). Nothing was
migrated, which also skipped the one-way v1->v2 database migration.
- Pinned to 2.5.5. `:latest` is a documented trap now: upstream keeps it on
the 1.x line, so an August 2026 pull produced an image BUILT 2024-12-20
running 1.23.16. Verified by digest -- latest and 1 share one digest while
2/next carry 2.5.5. Pinned exactly, not floating on 2, for the same reason.
- Moved esh-docker-vm -> ana-docker. House placement rule puts cross-site
services beside the Beszel and Dozzle hubs, and esh-docker-vm has wedged
unkillably twice in four months. A monitor also cannot report the failure
of the host it runs on, so it should not share a failure domain with the
host layer.
- Normalised restart: always -> unless-stopped, which the 2026-08-18 README
flagged as worth doing on the next deliberate touch.
- UPTIME_KUMA_DB_TYPE=sqlite in the compose skips 2.x's interactive database
screen, so the stack comes up ready rather than parked on a form.
scripts/kuma is a first-party Socket.IO client, because both obvious paths are
wrong: there is no REST CRUD API in EITHER major version (server/routers/ holds
exactly two files, /metrics + badges + status pages), and the community wrapper
uptime-kuma-api is abandoned -- last release 2023-09-26, ceiling 1.23.1, no 2.x
support ever.
⚠ getMonitorList's callback returns only {ok:true}; the list arrives as a
SEPARATE pushed monitorList event. Reading the ack yields an empty board that
looks authoritative -- which duplicated all 13 rows on the first re-seed
before the bug was found. The client now waits for the push, and carries a
dedupe verb because of it.
13 monitors seeded from monitors.yaml, keyed on name so a re-run updates rather
than forking the board -- proven by re-running it (0 added, 13 updated), not
assumed. Every URL was probed before being written: all 200. A board that ships
red teaches everyone to ignore it.
Verified: 13 rows, no duplicates, all UP with "200 - OK" read from the database
WITH its WAL (a first read of kuma.db alone showed a stale 26 -- the copy
predated the deletes). Homepage renders exactly one Uptime Kuma card.
⚠ NOT YET WIRED: notification delivery. The board detects but tells nobody,
which is the same gap this work exists to close. The beszel-althing bridge
hardcodes a [Beszel] subject prefix and hub footer, so routing Kuma through it
unchanged would mislabel the alerts. Needs a decision before it is generalised.
The adapter is STAGED, NOT SHIPPED: adapter_config.json and
adapter_model.safetensors are in /tank/aimodels/voice-adapters/lv-mccarthy-4b-v1
on fv-ml1, verified byte-identical to gx10's checkpoint-300 at the source, after
the local hop and at the destination. The seat CANNOT load it -- nothing in
--lora-modules references it -- and /v1/models was checked after staging and still
serves exactly the three registered voices plus base.
The adapter carries a README recording the full gate result AND its cost, so it
cannot be read as clean by anyone who finds the directory without the gate record.
A copy is committed at scripts/mccarthy-corpus/gate-results/ADAPTER-README.md for
review. It leads with "STAGED, NOT SHIPPED" and says that if lv-mccarthy is absent
from the seat's model list, the decision has not been made.
The compose edit that would actually register it is prepared in the working tree
and deliberately LEFT UNCOMMITTED. stacks/ is canonical intent that deploy-stack.sh
reads; committing it would assert an intent that is precisely the open question.
Ship-script fix: <run-dir> must be an absolute REMOTE path. A leading ~ is expanded
by the local shell, so ~/r49-runs/... became /home/lkraven/... and gx10 answered
"Permission denied" -- a confusing error for a path problem. It now refuses a
non-absolute path outright, with the reason. Verified: the guard fires on a ~ path
and the real staging succeeded on the absolute one.
1,200 generations across five arms. ckpt300 (epoch 0.652) is the best arm in the
run on every axis that resolves:
VOICE +0.177 at 3.2x its pairwise floor (primary), +0.128 at 2.8x with
every punctuation mark stripped. Best point estimate AND best
margin of any arm, spread 0.055/0.038 with no outlier seed.
MEMORISATION 0.12 against real unseen McCarthy's own 0.12 -- identical -- with
a longest match of 10 words against the author's coincidental 12.
All 31 matches read: stock grammar, names are the renamed
inventions, nothing protectable.
DAMAGE ran-on +0.12. Clears the operator's ratified v2 floor of 0.200 by
40%. FAILS AMENDMENT 3's self-imposed 0.100 bar by 0.02.
NOT SHIPPED, and the reason is the bar rather than the adapter. AMENDMENT 3 fixed
ran-on <= 0.100 before either new arm existed, precisely so a marginal number could
not be talked into a ship, and shipping at 0.12 would make that pre-registration
theatre. But the bar's stated rationale was written against ckpt450's pass by 0.01
-- 5% of the threshold -- and ckpt300 clears by 40%. The number excludes a candidate
the reasoning does not. That is an operator call.
ckpt325/350/375 are on disk and one may sit under 0.100. They were deliberately NOT
gated: searching the checkpoint space until something clears is candidate-shopping,
the same family as threshold-shopping approached from the other side.
THREE CLAIMS FROM EARLIER THIS SESSION ARE REFUTED and are corrected in the record:
1. "The damage is flat across epochs and only rotates direction" -- FALSE. ran-on
is non-monotonic (0.38 -> 0.13 -> 0.20 -> 0.28 across epochs 0.49/0.65/0.98/
1.96) with a real minimum near 0.65, and ckpt225 is 48% out-of-band against
ckpt300's 35%.
2. "ckpt300 runs far too short, ckpt225 will clear ran-on by being short" -- FALSE
on both. ckpt225 runs LONG (median 127, 38% over-band) and is the worst arm in
the run. I generalised from SIX generations of one arm, which is the exact n=1
violation the measurement-discipline rule names, committed in the same breath
as a note about being careful.
3. The original "gate an earlier checkpoint, the overshoot may not have arrived
yet" recommendation was RIGHT. Retracting it an hour later on a three-arm read
was the error, not the recommendation.
What is true and unresolved by any checkpoint choice: 35% of ckpt300's generations
miss the 90-140 band against base's 11%, and in-band is 0.65 against 0.89. An
adapter that buys a voice and costs a third of the length compliance is a trade, not
a defect -- but it is the operator's trade to accept.
Raw artifacts for all five arms at scripts/mccarthy-corpus/gate-results/.
Six generations into ckpt300 its outputs are far too SHORT -- 6, 25, 46, 59 words
against a 90-140 ask. At epoch 0.65 the adapter may not have learned the length
target yet.
That exposes a hole in the bar I set an hour ago: ran-on is defined as words > 140,
so a chronically undershooting arm satisfies "ran-on <= 0.100" trivially while
being useless. Same class as the unsatisfiable criterion AMENDMENT 3 exists to
correct, inverted -- too easy rather than impossible.
The bar is NOT being edited again. Three rule-motions in one session is already too
many and a bar patched once per observation is not a bar. Instead the judgment is
stated in the open: a candidate that clears the ran-on term by being short has
cleared it vacuously and will not be shipped, and the full in-band / on-beat /
median / p90 table is reported for every arm so a reader can see which kind of pass
they are looking at.
Gate design for this corpus is evidently not settled. That is an operator call, and
the complete five-arm ladder is being measured so it can be made once rather than
iteratively.
Arithmetic, not a re-reading of results. Criterion 1 needs the candidate's in-band
rate to beat base's 0.89 by more than the 0.200 floor -- so it needs in_band >
1.09, and a rate cannot exceed 1. A PERFECT adapter, every one of 240 generations
inside the 90-140 band, scores +0.11 and fails by 0.09.
So "both candidates failed axis C" carries zero information about either candidate.
That verdict was determined before a single generation existed. It is my error.
The cause is specific: section 6 transcribed score_beats.py's v1 criteria, which the
operator AMENDED on 2026-09-15 -- three weeks before this corpus was built -- for
exactly this reason. Its own docstring: "NOT carried into v2: in-band (unresolvable
-- base maxes it) and on-beat."
The governing rule reverts to the operator's v2: axis C is ran-on only. That is the
rule they ratified and the rule lv-hemingway shipped under. Reverting a
transcription error to its source is not threshold-shopping; the source is older
than the data.
It does not rescue ckpt450 either. Under the corrected axis C its ran-on is +0.19
against a 0.200 floor -- a pass by 0.01, inside the noise the floor exists to
describe, and not a pass worth shipping on an author in copyright with a living
estate.
AMENDMENT 3 therefore gates two more arms on the SAME frozen fixture, seeds and
rule -- ckpt300 (epoch 0.65) and ckpt225 (epoch 0.49), both on disk because
--save-total-limit 60 kept all 56 checkpoints -- since the damage grows
monotonically with epoch and the voice may arrive before it does.
And it sets the ship bar BEFORE those arms exist: voice clearing the pairwise floor
on both reads, memorisation at or near the held-out author's 0.12 with every match
read, and ran-on delta AT MOST 0.100 -- half the measured floor, not merely inside
it, because a rate that clears a noise threshold by 5% of that threshold has not
been shown to clear it. If nothing clears all three, lv-mccarthy is parked pending
a retrain, and that is the report rather than a softer bar.
720 generations, 3 arms x 60 held-out beats x 4 seeds, against the design frozen
in GATE-PREREG.md before any arm existed.
AXIS A VOICE -- PASS, both candidates, both reads. Span 0.661 -> 0.370 = 0.291
achievable; ckpt900 closed 59.1% (+0.172, but only 1.2x its floor), ckpt450 52.2%
(+0.152 at 2.9x). The normalised secondary read HOLDS at +0.124 / +0.114, so about
three quarters of the gain survives stripping every punctuation mark -- the voice
is not the cheap win the register made available.
AXIS B NOT COPIED -- ckpt450 is the cleanest result in the line. 0.12 hit-rate
against the author's own held-out 0.12, and its longest match (11 words) is
SHORTER than the author's coincidental longest (12). All 96 matched runs were
read: stock grammar in the commonest words, the name-shaped hits are the RENAMED
inventions, nothing protectable. The amendment is why this reads as clean -- the
defective base control would have shown 0.12 vs 0.00 as a 12x red flag.
Separately measured: the "his register makes collisions inevitable" story that
was FALSE for Hemingway (0.01) is TRUE for McCarthy (0.12). Neither transfers.
AXIS C NO DAMAGE -- FAIL, both, and it survives reading. 20% (ckpt450) / 28%
(ckpt900) of generations overshoot the 90-140 band against base's 1%; p90 171/190
words, max 297/279. The worst case is degenerate looping, not a long McCarthy
sentence. Base is GOOD on this axis here (0.89 in-band vs Hemingway's 0.05), so
the adapter measurably makes instruction-following worse.
NOT SHIPPED. Section 7 rule 3 makes axis C disqualifying outright.
Recorded honestly: my own prereg's axis C transcribed score_beats.py's v1
criteria, including "in-band up on base", which the operator RETIRED on
2026-09-15 for exactly the reason it fails here -- base maxes it, so it is
unsatisfiable on this corpus regardless of adapter quality. Under the operator's
v2 (ran-on only) ckpt450 passes by 0.01 against a 0.200 floor. That reading was
found AFTER the numbers and was NOT used; lv-bronte's floor defect is the in-line
precedent for finding one and declining to exploit it. The prereg gets fixed
prospectively for the next author, not re-read for this one.
And the finding worth more than the adapter: the two-epoch recipe is now 0 for 3,
and this time the loss curve was CONFIDENTLY wrong. On Bronte and Hemingway the
epoch-1/epoch-2 checkpoints were tied, so preferring the earlier one cost nothing.
Here the curve resolved epoch 2 as better at 4.9x the median neighbour jitter --
and epoch 2 lost every axis that resolves: 4.0x wider seed spread, 1.8x the
author's memorisation rate against 1.0x, more ran-on, worse on-beat. Its only win
is a 0.019 voice point estimate, inside the floor, and its spread is one outlier
seed -- the third occurrence of that shape in the later checkpoint after
lv-bronte's ckpt925 and lv-hemingway's ckpt1750.
Raw artifacts committed at scripts/mccarthy-corpus/gate-results/ so the claims can
be re-read without gx10.
voice_distance.py's quote class shipped this morning as "'‘’“”«»‹›‚„` -- with the
apostrophe characters in it. On a corpus whose defining tic is dont/aint/wont and
whose possessives are everywhere, that made it an apostrophe counter wearing a
quote-mark label.
as implemented TRUE quotes all apostrophes
held-out McCarthy ref 121.1 0.0 121.1
base-unadapted 224.7 19.9 204.8
held-out Hemingway ref 1112.6 694.7 351.7
The corrected column is the one the pre-registration names: 0.0 for McCarthy,
which is exactly what build_corpus_mccarthy.py ASSERTS, and 694.7 for Hemingway,
the documented ~838 scale the 100-per-10k trigger line was anchored to. The
as-implemented column matched neither.
ORDER OF EVENTS, because it is the material fact: the base arm finished first, so
the trigger became evaluable while the adapted arms were still generating. I
evaluated it, saw it FIRE at 224.7, and only then -- reading the reference row of
my own table against a corpus I knew asserts 0.0 -- found the bug. No delta_cb,
memorisation rate or damage number had been read at any point.
Fixing a detector to measure the quantity the frozen rule names is not moving the
rule, but the fix un-fires the trigger and no reader should have to take my word
about my motives. So GATE-PREREG.md AMENDMENT 2 makes the trigger MOOT instead of
adjudicating it: the normalised secondary read is load-bearing UNCONDITIONALLY for
this gate, whichever reading you accept, both columns reported. The fix therefore
has no effect on the verdict.
There is a better reason than the bug anyway: base's true quote density is 19.9
against the reference's 0.0, so it did not fully comply. A small residual cheap win
IS available to the adapter, and the normalised read is what prices it. A threshold
is a blunt instrument for a residual that size.
Apostrophes now get their own column and are never folded into quotes again.
Default path stays byte-identical to the shipped lv-hemingway artifact.
The durable lesson is the one this line keeps relearning in new places: I controlled
strip_punct (2500 -> 0) and the byte-identity of the default path, but never asked
the quote counter for a value whose answer I already knew. The corpus asserts 0.0.
That check cost one line and was available before the gate ever launched.
Shipping a gated adapter was ad-hoc ssh + rsync three times running. This makes
it one reproducible command for the rest of the line (Faulkner, Morrison,
Chandler are next).
The sha verification is the point, not decoration. "Copied the adapter" is a
claim with no honest form but a read-back, and a truncated or half-written
adapter loads without complaint and serves a subtly different voice. The digest
is taken at the source on gx10, after the local hop, and at the destination on
fv-ml1; any mismatch aborts before the seat is ever told the file exists.
It ships only adapter_config.json and adapter_model.safetensors. A checkpoint
dir also holds optimizer state and RNG, which are large, useless to the seat,
and would make the destination digest disagree with any future re-ship.
It deliberately does NOT edit the compose or restart the seat. Registering an
adapter is a reviewed edit to stacks/voices-seat/compose.yaml, because that file
is where the gate verdict and its caveats get written down for whoever reads it
next -- that is a feature of the current process, not friction to automate away.
Verified against the already-shipped lv-hemingway: the live adapter on fv-ml1 is
byte-identical to gx10's checkpoint-850 on both files, which confirms the file
list and the convention this script encodes.
memorization_check.py answers HOW OFTEN an arm collides with the training text.
It does not answer WHAT is being reproduced, and those have different
consequences: a low rate hiding one 30-word run of distinctive imagery is worse
than a high rate of stock dialogue in the commonest words in English.
The lv-hemingway gate shipped at 7x the author's own collision rate precisely
because that reading was done -- every matched run was stock dialogue, max 9
words, no proper noun. But it was done BY HAND and left no instrument, so the
next gate had to repeat it by hand or skip it. This is it.
Prints every maximal matched run sorted by length, with arm/id/seed, and flags
any token capitalised mid-run as a possible proper noun. The flag deliberately
over-reports (sentence-initial I, He, What all trip it) because it is a reading
aid and over-reporting is the safe direction. --train-only matches
memorization_check.py so a collision with held-out text is not reported as
memorisation of training text. A corpus slice is matched against the corpus
every run, and the script REFUSES if that positive control fails -- a matcher
that only ever sees candidate text cannot tell "no matches" from "blind".
Validated against the lv-hemingway record's documented reading, which it
reproduces to the word: longest run 9 words, "swift tristan" flagged as the one
name-shaped hit (it is the RENAMED invented name, not Hemingway's), and the
record's quoted examples -- "came over and sat down at the table", "i don t
think so the girl said" -- both present.
Required by GATE-PREREG.md AMENDMENT 1, which makes reading the matches part of
axis B rather than a follow-up, because McCarthy is in copyright with a living
estate and a match carrying distinctive imagery or a proper noun is
disqualifying in a way a rate number alone is not.
The runbook stopped at D4 pairs, so the two stages that actually produce the
shippable artifact were unrecorded -- the same gap that forced the 2026-09-17
D1->D3 chain to be recovered by rebuilding candidates and matching sha256.
D5 records the 1,380-step run, the loss curve read (median neighbour jitter
0.00393; ckpt900 the minimum, ckpt450 at +4.9x, adapter/ at +18.4x), the fact
that the epoch-3 collapse STEPS rather than drifts, and that adapter/ is the
epoch-3 weights so whatever ships is a checkpoint.
It also records the four provenance fields that look wrong and are not -- all
four reproduce on the yarros and hemingway runs -- and how to bind the run to
its pairs by RECORD COUNT rather than by the relative path it stored.
D6 records the gate invocation, the four deviations from the lv-hemingway gate
and what forced each, and the two instrument questions that had to be settled
before it could run: the punctuation confound on the voice axis (primary /
secondary / pre-registered trigger) and the defective base-unadapted control on
the memorisation axis.
Plus one thing that will otherwise be misread: gen_beats_chat_yarros.py prints
RAN-ON when it finds no paragraph break, a heuristic written for the Yarros
register. McCarthy's register asks for continuous scene prose, so an unbroken
block is the TARGET here and the tag fires on most generations without meaning
damage. The axis-C metric is a different definition and is unaffected.
The lv-hemingway gate established that memorization_check.py's negative control is
defective: base-unadapted writes SUMMARY while the adapted arms write PASTICHE, and
text that does not imitate a register cannot collide with that register's n-grams.
Base's 0.00 partly measures "different register", not "did not memorise". A negative
control that differs from the candidate in a way correlated with the metric is not a
control.
The correct innocent sample is the author himself -- held-out text no arm trained on,
in the same register by construction. That control was computed BY HAND during the
lv-hemingway gate and never committed, so the finding it produced was not
reproducible. This commits it as --heldout-reference, with --train-only to build the
n-gram set from split=="train" records only (the default includes val, so a collision
with held-out text was being counted as memorisation of training text -- an overcount
Bronte's and Hemingway's numbers both carry).
--heldout-reference without --train-only REFUSES, because the held-out text would be
scored against a gram set containing itself.
Instrument controls:
- default path reproduces the shipped lv-hemingway memorization.txt BYTE FOR BYTE
- the new held-out row reproduces the lv-hemingway record's hand-computed numbers
to the digit: 370 samples, hit-rate 0.01, mean-longest 0.1, max 10, at the same
101-word median chunk length -- an independently written instrument reproducing a
documented finding is what makes its zeroes mean absent rather than blind
GATE-PREREG.md AMENDMENT 1 adopts it for lv-mccarthy, appended rather than folded in
so the order of events stays legible. Written while the base arm was still generating
and before any McCarthy number had been read. It moves the reference a candidate is
compared against from a known-broken one to a known-correct one, in the direction
that makes the axis HARDER to pass, on an argument published four days before this
gate existed. Section 7's tiebreak is amended to match, and the amendment records
that an elevated rate is not by itself a no-ship -- rate and exposure are different
questions, every matched run gets READ, and McCarthy being in copyright with a living
estate makes a distinctive match disqualifying in a way a rate number alone is not.
Freezes the lv-mccarthy gate design while no McCarthy generation exists, per the
lv-hemingway precedent: a threshold chosen after seeing the numbers is not a
threshold. Three arms (base-unadapted, ckpt900, ckpt450), 60 beats, 4 seeds.
Settles the voice-axis question D1 deferred. McCarthy's corpus measures 0.0 quote
marks per 10k against Hemingway's 838, so "emit no quotation marks" is the cheapest
way to move a char-bigram Delta without learning a sentence. Three parts:
PRIMARY unchanged -- the mccarthy register names the punctuation and
--system-from drives the base control with the same prompt, so the
cheap win is handed to both sides.
SECONDARY voice_distance.py --secondary-normalised re-runs the whole analysis
with punctuation stripped from the reference and every arm. A
conservative lower bound; reported, never the verdict.
TRIGGER voice_distance.py --punct-report evaluates a pre-registered line --
base-unadapted quote density above 100 per 10k means the control did
not take the win it was handed, and the normalised read is promoted
to load-bearing. 100 is the order-of-magnitude line between this
corpus's 0.0 and Hemingway's 838, chosen now rather than after the
table prints.
ckpt450 is NOT tied with the minimum (+4.9x the 0.00393 median neighbour jitter)
and the pre-registration says so. It is generated to test a stated prior and to
price the memorisation headroom an earlier checkpoint buys on an in-copyright
author, with a decision rule that says exactly what result would let it displace
ckpt900.
Instrument controls, run before this landed:
- the voice_distance.py refactor reproduces the shipped lv-hemingway
voice_distance.txt BYTE FOR BYTE on the default path
- strip_punct drops a quote-bearing sample from 2500.0 to 0.0 marks per 10k
- the secondary read still resolves a gap on the Hemingway arms (+0.410 at
7.3x floor), so a null on McCarthy would be a finding, not a blind detector
Pre-flight re-run at gate time rather than quoted from 2026-09-17:
- leak gate: 0 of 75 renameable, 0 of 37 sub-threshold, 0 separator-split,
four controls green
- beat-contamination audit: 0 of 3942 beats AND 0 of 3942 responses, against
Hemingway's 70 of 7094 -- build_sft_pairs.py --source-entities earned its
mandatory flag
Also records three provenance defects found on first read of the run and their
disposition: the hardcoded "r49-babyyarros-pairs-pilot" run label (cosmetic, same
literal on all three runs), the empty harness_commit (all three runs), and a
pairs_sha256_16 that is not a sha256sum of the file (consistent across runs, so a
cache key rather than a fault). The run is bound to McCarthy's pairs by record
count, not by the label.
The check collapsed two different findings into one verdict. On 2026-09-20 it
printed "RESULT: STALE" while reporting 37 FRESH layers and zero stale ones --
every backup body provably current, the three ❌ rows all yesterday's pre-fix
runs aging out of the 36h window. infra-hermes caught it in triage: a reader,
or a forwarder, could page someone over a state where nothing is stale.
STALE is a claim about backup AGE. A job that ran and errored is a different
claim with different urgency. They now have different words and different exit
codes:
0 all backups fresh
1 STALE -- a body past the threshold, or an endpoint down
3 ERRORED-JOBS -- every body fresh, a vzdump job errored recently
The alert wrapper mirrors the code and matches its own wording to the finding:
🟡 "Backup jobs errored — all bodies fresh" instead of 🔴 "Backup freshness
ALERT", and it now exits with the check's code rather than flattening
everything to 1, so `systemctl status` distinguishes the states too.
This is the same defect class the rest of this script was built to fix, one
level up: not an instrument that fails to look, but one that looks correctly
and then reports the wrong word for what it saw. An alarm that cries outage
over a healthy fleet earns being ignored exactly as fast as one that stays
silent over a broken one.
Verified all three states by forcing each: BACKUP_JOB_WINDOW_HOURS=1 -> exit 0,
default -> exit 3, BACKUP_MAX_AGE_HOURS=1 -> exit 1.
elway passes its `host` argument straight through, and that argument is an ssh
target. Five records of a real jobs.cfg change on esh-pve were written under
host `infra-ops@esh-pve`.
The consequence was not cosmetic. infra-hermes triaged the ESH backup job
errors, queried `--host esh-pve`, got nothing back, and correctly reported the
05:29 exclusion of VM 102 as an unattributed change. The record had existed the
whole time, five lines of it, under a name nobody would think to ask for. A log
you cannot query under the obvious name is not a log.
`normalize_host` strips any `user@` prefix on write AND on query, so records
written before this fix are findable too, and `audit` -- which matches records
by host name -- stops missing them as well.
Verified in both directions: a record written with `infra-ops@esh-pve` and one
written with `esh-pve` now both return for `--host esh-pve`, and the five
historical elway entries surface under that query.
The commit hook was APPENDED to .git/hooks/post-commit so as not to disturb
graphify's block -- and that made attribution a subordinate clause of
graphify's control flow. graphify owns EIGHT `exit 0` paths (rebase / merge /
cherry-pick in progress, GRAPHIFY_SKIP_HOOK, no changed files,
graphify-out-only, no python found), and every one of them returns before
reaching an appended line.
infra-hermes reported commits going unlogged and proposed a lock race: the
ops-log fcntl path held by graphify's detached rebuild long enough for a
best-effort `|| true` invocation to drop. Plausible, and wrong. The cause is
ordering and it is deterministic. eaaacc1 touched only
graphify-out/GRAPH_REPORT.md and hit the graphify-out-only exit; an empty probe
commit hit the no-changed-files exit. Neither could ever have been recorded.
Two data points that look flaky are worth reading as a control-flow question
before a concurrency one.
The block now goes immediately after the shebang. graphify's hook is preserved
byte-for-byte below it, and core.hooksPath stays unused because it would
disable graphify outright.
Verified in both states with real commits, then reset out of history: a
graphify-out-only commit -- the exact case that was silently dropped -- now
records, and a code commit still records AND still triggers graphify's rebuild.
The fallback recorded `lkraven@nh3-dev` when ALTHING_HANDLE was unset, which
reads like an answer while being the absence of one. It now records
`unattributed(lkraven@nh3-dev)` -- handles never contain @ or parentheses, so
the fallback cannot be mistaken for an identity.
Found by the commit hook's first independent test, which was not one I ran:
infra-hermes committed 0fe4da6 and the hook caught it unprompted -- but logged
him as lkraven@nh3-dev, not infra-hermes. His seat exports ALTHING_HANDLE from
althing-infra-hermes-seat-run.sh, while neither hermes-gateway.service nor his
pump unit carries it, so anything he does outside that one script logs with no
handle.
So the hook is real but partially blind: it distinguishes not-infra-ops from
infra-ops, which is enough to have told me e43e262 was not mine, and not
enough to name who made it. The env-var gap is his to close. Making the blind
spot announce itself in the record is mine -- an instrument that cannot answer
should say so rather than return something answer-shaped.
Both harnesses executed top-to-bottom on import, so anything collecting
them (pytest, a glob import) ran live-store side effects. main() guards
plus finally-block cleanup of the _t_* cron output dirs the fabricated
jobs leave behind.
Two agents share ONE checkout of this repo on nh3-dev, and every commit here
is attributed to Vuong Hoang by convention -- so a commit's author line says
nothing about which agent made it. The ops log closed that gap for HOST
changes; it did not cover commits to the shared tree.
Found the hard way today: e43e262 appeared interleaved between two of this
session's commits, in this session's own reflog, and was unattributable from
git, from the ops log AND from the althing bus. It was sitting in the push set
at the time, and forseti had explicitly asked that unrelated management-repo
commits not be pushed -- so the one thing needed to honour that request was
the one thing nothing recorded.
The hook APPENDS to .git/hooks/post-commit rather than replacing it, because
graphify already owns that file and core.hooksPath would disable it.
Best-effort by construction: a failure here must never fail a commit.
deploy-stack.sh claims and releases around its own work. When the agent
already held a longer claim for a multi-step operation, the deploy refreshed
it, then released it on exit -- silently dropping the protection partway
through the very operation it was guarding.
Caught live: a 45-minute claim on nh3-docker/althing-post-office, taken to
cover a build-push-deploy-verify rollout, was gone by the time the rollout
finished. Nothing refused anything, and nothing said so.
`ops-log claim` now exits 10 when the claim was already the caller's, and
leaves the holder file untouched. deploy-stack.sh treats 10 as "not mine to
release". Untouched matters as much as the exit code: a refresh would
overwrite the reason and TTL the original claimant chose, so a deliberate
45-minute "3.7.0 rollout in progress" would degrade into "deploy-stack.sh
<host> <stack>" and whoever got refused would read the wrong story.
Verified in three states: no pre-existing claim -> deploy claims and releases;
caller's own wider claim -> deploy refreshes nothing and leaves it standing
with its original reason; another agent's claim -> still refused with exit 3.
Disposable harnesses from the wake-RFI thread (01M2WT3F): exit1-behavior-test.py
covers the monitor exit-code/gate path including a sustained-failure/no-backoff
assertion; hash_gate_repro.py covers the detection-time hash persistence
crash-before-read boundary with consumption/unchanged/changed controls.
Both fail nonzero on mismatch. Committing so seat artifact provenance
rides git rather than file mtimes.
Conditions ran unprivileged no matter what the step declared, and that fails
in the dangerous direction. A root-requiring `when:` -- `pvesh get ...` exits
255 for a non-root user -- returns non-zero, elway reports the step `skipped`,
and a playbook that never performed its change reports overall OK. "Skipped"
is indistinguishable from working idempotency, so the run looks correct.
Found the hard way on esh-pve: three consecutive runs of an exclusion playbook
reported success while changing nothing. Only the verify phase caught it, by
continuing to report the thing the steps claimed to have handled -- which is
exactly why verify runs unconditionally.
`creates:` had the same fault from the other side: a path under /root is
unreadable to the login user, so `test -e` said absent and the step re-ran
every time. It now correctly reports the file as already present.
Sudo-less steps are unaffected: their conditions still evaluate as the login
user, which is what they mean. Only a step that declares sudo: true gets
privileged condition evaluation, so no existing playbook changes meaning
unless it was already silently broken.
Snapshot age is structurally blind to a backup job that executes every night
and fails every night. Nothing new is written, so the group simply ages, and
the fault only surfaces once it crosses the 48h threshold -- days after the
first failure, with the evidence sitting in a task log nobody reads.
Two live cases, both found today and both invisible for a week by this exact
mechanism:
* esh-nas-pve CT 107 (vm-jellyfin): a backup run died around 09-06 and left
a stale `backup` lock, so every nightly since failed instantly with "CT is
locked (backup)". Age named it on ~09-12. Task status would have named it
on 09-07.
* esh-pve VM 102 (esh-vm-workstation): failing nightly since ~09-06 with
"timeout waiting on systemd". Same six-day gap.
PVE already records every task result in /var/log/pve/tasks/index. This reads
it on all four non-tenant PVE nodes and reports any vzdump in the last 36h
whose status is not OK, as its own section that sets the exit code.
It found a third case on its first run: esh-nas-pve's job had been reporting
`job errors` nightly while every guest on that node read 0-1h fresh, so no
age-based check could ever have flagged it.
Window is BACKUP_JOB_WINDOW_HOURS (default 36 -- longer than a daily cycle so
one missed run does not hide a failure). A node whose task log cannot be read
is reported, never assumed healthy.
ana-scale (CT 114) is a subnet-router LXC, excluded from vzdump on 2026-09-07
after a backup lock on its ESH counterpart blackholed that entire site. The
freshness check knew nothing about that and reported it 🔴 STALE every single
morning, which is how an alarm teaches you to ignore it.
Such guests now get their own section: printed every run, never hidden, and
not counted as a fault.
The subtlety is in how coverage is computed, and the obvious implementation is
wrong twice over:
* Reading one job's `exclude` list gets ana CT 109 (ana-nas) exactly
backwards. It IS excluded from the 03:00 all-guests job AND it has its own
dedicated 22:00 job. Suppressing on the exclude list would have stopped
alarming on a guest that is genuinely backed up -- trading a noisy alarm
for a blind one.
* ESH's job uses an explicit `vmid 100..107` INCLUDE list, so esh-scale 108
is excluded by OMISSION and appears in no exclude list at all.
So coverage is a union across every enabled job on the cluster, and a guest is
"intentionally not backed up" only when none of them covers it.
If coverage cannot be read, nothing is suppressed and the gap is reported: an
unreachable PVE node means we do not know, and a backup alarm must fail loud.
The SureFire namespace is never consulted (tenant property), so its guests can
never be suppressed either.
Verified against the live fleet on all four paths: CT 114 suppressed; CT 109
NOT suppressed despite being in an exclude list; esh-vm-workstation 102, which
a job really does cover and which really is failing, still reports STALE; and
with a PVE node made unreachable, 114 returns to STALE with the gap named.
vh
Vuong Hoang