eea9eb2ae5dfd31f31f69fe82fbc8df8fcb85feb
100
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 vh
|
eea9eb2ae5 |
docs(fleet-conventions): git checkouts under root:docker need safe.directory for the deploy user
The 2026-09-14 normalization (
|
||
|
vh
|
fedd4b6d95 |
fix(headscale-ddns): retry Cloudflare, validate every response, never write blind
Three of the four failures in the week to 2026-09-23 (09-19, 09-21, 09-23) had one signature. The Cloudflare zone lookup returned an empty body at its 15s cap, a bare json.load crashed with tracebacks instead of a cause, and the script carried empty IDs on to a PATCH against zones//dns_records/. Cloudflare rejected it, so there was no DNS impact, but only by accident. - Cloudflare calls go through cf(): 3 announced tries, and a call counts only when the body says success:true. - success:true is not trusted as shape. pick() validates every body and prints only the fields asked for, or one line saying why not. It requires exactly one zone named phasefinal.com and exactly one A record named headscale.phasefinal.com, each with a non-empty id and content. Two A records are refused rather than half-updated, and an empty id can no longer shift the content into the id slot. - No write without both IDs. The run ends on a confirmation that the record now reads the new address. The previous final line was an echo whose exit status was always 0, even when the parse inside it failed. - Only a global unicast IPv4 is published (python ipaddress is_global). Loopback, RFC1918, link-local, CGNAT and documentation ranges are retried and then refused. - curl -q as the first argument ignores ~/.curlrc, so a verbose config can never log the bearer token. The vault CLI path is quoted. The empty data-array expansion is safe under set -u on bash < 4.4. Tests: services/headscale-ddns/test_headscale_ddns.py, 12 cases with curl, the vault CLI and sleep stubbed. Each failure case asserts the FATAL line's stated reason, so a run that died earlier for an unrelated cause cannot pass it. The documentation-range fixtures (203.0.113.x) were themselves rejected by the new public-IP guard: a free positive control. Live: a manual run and a unit run both printed "unchanged 70.230.226.88", Result=success. Cross-model bug-hunt (heid "Talus", Gróa + seat): all 8 findings folded. |
||
|
vh
|
466f7aa4e6 |
fix(elway): sudo uploads land root:root, validated and staged; add fleet ownership audit
elway's sudo upload did scp-as-user then `sudo mv`, and mv keeps the owner,
so every file it installed "as root" (systemd units, /etc configs, root-run
scripts) ended up owned by the SSH user. A sudoers drop-in installed that
way would be rejected by sudo outright.
elway:
- Sudo uploads now chown to root:root by default. Playbooks can override
with `upload.owner:` and ad-hoc runs with `--owner`. An owner is refused
on a non-sudo upload, and `--owner` outside an ad-hoc `--upload` is an
error rather than silently ignored.
- Ownership and mode are applied to the STAGED file, then a single mv
publishes it, so a failed chown can no longer leave the live path owned
by the SSH user. chown runs before chmod so setuid bits survive. A trap
removes the staged file on every exit path.
- A directory dest is refused before anything moves. mv would otherwise
drop the file inside the directory under a staging name.
- `mode` was spliced unquoted into the remote root shell line. It is now
validated as octal (^[0-7]{3,4}$) and shell-quoted, on both the sudo
and non-sudo paths.
- `mode` and `owner` must be quoted YAML strings. YAML 1.1 turns a bare
0644 into 420, a bare 1000:0 into 60000 and a bare yes into True, and
str() of each is a different value that looks valid. All 85 playbooks
already quote them.
- New preflight(): every step is resolved before any remote action. An
undefined var or a templated owner that resolves badly is now refused
up front, not after earlier steps have already changed the host.
scripts/fleet-ownership-audit.sh (new, read-only) finds files in root's
territory owned by a normal user. Tier A lists /etc, /usr/local, /root and
cron, judging symlinks by their target. Tier X covers files that root-run
units exec from /opt, /srv, /home or /volume1, reading drop-ins and quoted
paths. Tier B only summarises /opt app trees. A host that is unprivileged,
whose find fails, or whose run dies is reported INCOMPLETE (exit 5), never
clean. Completion is marked with a per-run nonce.
Verification: 24 unit tests in scripts/test_elway.py. Live checks on
esh-docker-vm: default lands root:root; the override lands infra-ops:adm at
4755 with setuid intact; a bad group fails with dest untouched and no
staging left; a directory dest is refused; a bad mode is refused before
any copy. Audit positive controls on nh3-dev (a drop-in-only root Exec, a
quoted path containing a space, a symlink to a user-owned target) were all
flagged; esh-docker-vm negative control was 0. Probes removed. Cross-model
bug-hunt (heid, Gróa arm + seat) findings folded.
|
||
|
vh
|
6e8da46a28 |
fix(restic/esh-docker-vm): stop dumping paperless's Postgres from this host
Operator decision. paperless-ngx's database lives on esh-vm-db and is backed up at the source by that host's fail-closed pg_dumpall. esh-docker-vm's second copy had failed auth every night since 2026-04-24 behind a WARN. Its `> paperless.pg_dump` redirect left a 0-byte file in every snapshot (confirmed in snapshot 6ec9f74f), which looked like a dump but held nothing. The block was the only consumer of /etc/restic/dbcreds.env, so the creds loader is gone and the template dbcreds.env.example is deleted. The host file was moved (not deleted) to /var/lib/restic/repair-20260923/. Paperless's media volumes are still captured under /var/lib/docker/volumes. Also fixes ownership. elway's sudo upload does scp-as-user then `sudo mv`, so the hook deployed at 08:04 landed infra-ops:infra-ops even though root executes it. Both esh-docker-vm playbooks now chown it back to root and verify root:root 700. Verified: the live hook hash matches canonical (e0d3ddcef1bddf43), and the manual backup saved snapshot decfae71 with 3 staged dumps and no WARN lines. |
||
|
vh
|
25e41d2ab5 |
fix(restic/esh-docker-vm): drop the uptime-kuma hook block that aborted every backup
Uptime Kuma moved from esh-docker-vm to ana-docker on 2026-09-22. The pre-backup hook's fallback lookup, `docker ps | grep -E "uptime.kuma"`, then matched nothing and exited 1. Under set -euo pipefail that aborted the hook, and resticprofile treats a failed run-before as fatal, so no snapshot was taken from 2026-09-22 01:00 until this fix (backup-freshness: 54h stale). The block is removed rather than guarded because there is nothing on this host left for it to back up. The header now records the invariant the "blocks only WARN" promise depends on: every optional-service lookup must sit inside an `if` test or end in `|| true`. The remaining blocks were checked and all do. Deployed with playbooks/esh-docker-vm-restic-drop-kuma-block.yaml (the pre-fix hook is kept in /var/lib/restic/repair-20260923/). The live hook hash matches the canonical copy (43e6bea8b8569602). The manual backup saved snapshot 6ec9f74f, and backup-freshness now reports all backups fresh. |
||
|
vh
|
c2b0a05754 |
docs(notify-failure): a stop that exits non-zero pages as a failure
A deliberate `systemctl --user restart hermes-gateway` paged infra-ops as a FAILED unit (msg 3716) while the unit was already back up. Cause is a Hermes v0.21.1 race: the planned-stop marker watcher runs the shutdown handler before systemd's SIGTERM, consumes the marker, and the SIGTERM re-runs the handler, which then classifies the stop as unexpected and exits 1. Corrects the README claim that OnFailure never fires on a deliberate restart: that holds only when the main process exits with a success status. Measured on a throwaway unit (3/3 paged without SuccessExitStatus, 0/3 with it, and crash restarts are unaffected), and a survey of every stop on nh3-dev since 09-15 found hermes-gateway to be the only unit that does this. The host-side fix is drop-in hermes-gateway.service.d/ 20-planned-stop-exit1-is-clean.conf (SuccessExitStatus=1). No alarm coverage is lost: Restart=always ignores the classification, and StartLimitIntervalSec=0 means the unit can never reach `failed` from a start failure. |
||
|
vh
|
6d616391fd |
memory: snapshot — both carried decisions approved, safe-rm fleet-wide, the misroute closed
In-flight rewritten for 2026-09-22 evening. The operator approved both carried
calls ("decide to do both pending"): the NRestarts flap sampler is to be BUILT
(
|
||
|
vh
|
3eab4da379 |
memory: the sixth instrument error, and the one that consumed an operator approval
Corrects my own framing from earlier today. I told the operator D-0010's /clear
"cost nothing by luck of the delivery mechanism". svos-dev verified the code:
bifrost.py:1322 prefixes every relayed string with the card id, so a slash
command never begins the line. pane_send STRUCTURALLY cannot execute a harness
command in a target pane -- a property, not luck.
Which makes the card worse than a misroute. It promised "type /clear into
terminal_2 and clear booth-dev's context"; that effect was never achievable at
all. The executed event then recorded {"sent": true} -- true about the bytes,
false about the effect. Three defects in one approved action, and only the
first was visible.
Recorded because it is the sharpest instance of the day's pattern and the only
one that cost something outside an agent's triage time: it consumed an OPERATOR
APPROVAL. Vuong approved an action believing it would take effect and the
system told him it had.
General form worth carrying beyond SVOS: {"sent": true} is a claim about
transmission, never about effect. Any receipt that cannot distinguish the two
will eventually report success for something that did not happen.
|
||
|
vh
|
30517fd603 |
fix(headscale-ddns): say why it failed, retry the WAN lookup, and track it at all
The failed-START notifier built earlier today had its first REAL firing at
15:28: headscale-ddns.service exited 1 after succeeding all afternoon. The
detection worked. The alarm was also useless, and that is the finding.
Both failure paths exited 1 IN SILENCE, so the message said "exit status 1" and
nothing else. An alarm you cannot act on costs the same triage as no alarm at
all -- the notifier did its job and the subject script had no diagnostics for it
to carry.
Cause was transient and harmless: icanhazip.com did not answer inside its 10s
cap, so the IP came back empty and the regex guard refused it. No DNS impact --
the record already held the right address, verified against 1.1.1.1 before
touching anything, and the next timer run succeeded. Arithmetic confirms it:
~17s vault read + 10s curl timeout = 27s against the 28s the failing run took.
Fixed, both verified by making them fail:
- every exit path names its cause; a missing vault key names the key, an
EMPTY token is distinguished from a failed read, and a dead WAN lookup adds
"DNS left unchanged" because that is the fact the reader needs
- the WAN lookup retries 3x with ANNOUNCED attempts -- one third-party blip
should not page a human, and a silent retry would hide a degrading
dependency
⚠ ALSO: this script was not tracked anywhere. A fix to the thing every mesh
client resolves through lived on exactly one disk. Script, unit and timer are
in the repo now.
Measured and recorded: the vault read is 17 of the script's 18 seconds, every
10 minutes. Not a fault, but it bounds any retry budget and it is fleet-wide --
svos-dev's alarm unit carries the same 17-second note.
|
||
|
vh
|
a2ade84470 |
memory: snapshot — Booth handed to booth-dev, Kuma rebuilt, alerts generalized, five instrument errors
In-flight rewritten for 2026-09-22. The Booth is booth-dev's now (extracted to vh/booth, v0.2.0 shipped by them); infra-ops retains only its Uptime Kuma monitor. Eight decisions recorded, and three OPEN OPERATOR DECISIONS filed under Recent decisions with commit tracking pointers rather than under in-flight, per the deferred-work rule: the NRestarts flap sampler ( |
||
|
vh
|
4979869731 |
feat(backups): discover restic repos instead of enumerating them
Adding nh3-dev to the host list fixed the instance. This fixes the class, on
svos-dev's framing: a hand-maintained list of things to watch, sitting beside a
NAS that already knows which repos exist, means the next repo added is
unwatched BY DEFAULT and nothing says so. The list of what to check can
silently disagree with the set of what exists -- the same shape as every other
instrument fault found this day, only slower-acting.
The check now asks each NAS. A directory is a repository when it has a
snapshots/ child, which cleanly separates real repos from container dirs
(/volume1/Backup/restic/repo/ holds ana|esh|nh3 namespaces and no snapshots of
its own -- verified rather than assumed before building discovery on the
layout).
The hand-written list survives DEMOTED to an EXPECTED set, used only to report
a repo that has VANISHED. Two facts that would otherwise both read as silence
stay distinct:
"a repo exists that nobody watches" -> impossible now, it is discovered
"a repo we expected is gone" -> EXPECTED REPO NOT FOUND
Preventive, not corrective: all 8 repos currently discovered are already in the
expected sets, so this found no live gap. It removes the possibility of the
next one.
Controls run, since a check only ever seen passing is untested: a bogus
expected repo reports EXPECTED REPO NOT FOUND and turns the verdict STALE;
unchanged expectations still report all-fresh; all 8 repos report their age.
Observed while testing, not a fault: restic/ana/esh-docker-vm is 36h old
against 12h for every other repo. Inside the 48h threshold so correctly green,
but it is a day behind the fleet and worth a look.
|
||
|
vh
|
ba60fda16c |
feat(backups): assert restic CONTENT, and add the host that was never checked
Operator ruling 2026-09-22 (relayed via svos-dev): have the freshness check
assert snapshot content rather than REST-server reachability, so the green
light is a statement about DATA rather than about a daemon.
⚠ FIRST, A CORRECTION TO MY OWN REPORT. I ran `grep -ic restic` against
backup-freshness-alert.sh -- the WRAPPER -- got 0, and told the operator that
nothing on the fleet verified restic. Wrong. check-backup-freshness.sh has
always checked restic snapshot ages, for seven hosts. I grepped one file of a
two-file tool and generalised, which is the identical error a peer had just
made with dev-backup, made while correcting them.
THE REAL GAP was narrower and still real: the NH3 host list read
`irv-ml1 nh3-docker` and omitted nh3-dev -- the repo holding every Claude Code
session transcript, althing routes, hermes run history and Miranda's
conversation. /volume1/Backup/restic/nh3-dev/ has always existed and always
been written nightly; it was simply never enumerated, so a total failure of the
one repo nobody could reconstruct would have gone unreported indefinitely.
THE CONTENT ASSERTION, built on svos-dev's ladder (list proves the repo
answers; ls proves the index; check proves structure; RESTORE proves the bytes
come back):
- CONJUNCTIVE: latest snapshot newer than MAX_AGE_H AND the probe path
present AND a restore returns non-zero bytes. "A snapshot exists containing
X" is satisfied by a three-month-old one; "the latest is recent" is
satisfied by an empty one. Age alone was the old problem; content alone is
the same problem rotated.
- IT RESTORES. Metadata is what survives the failures worth fearing -- a
pruned or partially-written repo can list a path whose blobs are gone.
~493KB from ~/.local/state/svos, sub-second. Large paths are not restored:
a repo that returns one file will return others.
- IDENTITY: the timer runs as lkraven, which has no NOPASSWD sudo on nh3-dev,
so the probe hops through infra-ops@localhost. /etc/restic is root-only by
design.
⚠ restic ls ALWAYS PRINTS A HEADER LINE, matched or not. A path absent from the
repo returns 1 line; a real one returned 6. Counting with `grep -c .` read the
header as a hit, so the ABSENT case fell through and reported "blobs gone" --
telling an operator the repository was corrupt when the truth was a mistyped
path. Now `grep -c '^/'`.
TESTED BY MAKING IT FAIL, because a check only ever seen passing is untested:
bogus probe path -> "absent from snapshot"; BACKUP_MAX_AGE_HOURS=1 -> "12h old
(>1h)"; healthy -> snapshot id, age, entries, bytes restored.
|
||
|
vh
|
a4cf2ba0dc |
docs(backups): ask the repo not the job, and record that nothing watches restic
Two additions, both from a 2026-09-22 exchange with svos-dev. THE RULE. Coverage is a property of the backup SYSTEM, not of one job's configured scope. A peer checked dev-backup.sh, found SRC=$HOME/development, and reported to the operator -- with specifics and unhedged -- that five home-directory paths including Miranda's entire conversation had never been backed up anywhere. All five were in that night's restic snapshot. dev-backup is the hourly job for one directory; resticprofile is the daily job covering all of /home/lkraven. Checking one job and generalising to the system produced a confident, false, escalated claim. The runbook now carries the query that answers the question properly. THE GAP THAT VERIFYING IT EXPOSED, and it is worse. grep -ic restic against scripts/backup-freshness-alert.sh returns 0. The checker inspects PBS guest ages and pings the rest-servers for liveness -- which confirms the server answers, not that a snapshot was written. If resticprofile stopped entirely the light would stay green, correctly by its own definition, forever. Restic holds the whole home directory; PBS holds VM images. The layer with the granular data is the unwatched one, and the light is not merely blind but actively reassuring about a system it cannot see. Recorded as an open gap rather than patched, because fixing it changes what an existing green light means and people have been reading that light for months. |
||
|
vh
|
00a03921ff |
docs(alerts): reset-failed sits on the remediation path, which a flap sampler must design for
svos-dev found an interaction between the two detectors by doing the recovery rather than reasoning about it: clearing a failed unit is REQUIRED before systemd will start it again, and reset-failed zeroes NRestarts in the same motion. Verified here -- a start-limited unit refused to start until reset-failed, which took NRestarts 3 -> 0. So the standard recovery from a failed-START alarm erases the history a flap sampler would depend on, routinely rather than as an edge case. A unit that fails, is cleared, then flaps starts its history from zero, delaying the first flap alarm by the whole threshold -- exactly the window where a just-fixed unit is most likely still sick. Mitigation recorded: store a last-seen timestamp so a counter going BACKWARDS is itself detectable as a reset rather than read as quiet. Also recorded as its own shape: 'nothing to count is not the same as not counting'. The first manual-restart check read NRestarts flat and would have concluded a sampler goes blind whenever a human touches a unit. Artifact of the test -- that unit had already hit its start limit and stopped auto-restarting, so the instrument was reading correctly from a subject that had stopped producing. Verify, then ask what the verification could not have seen. |
||
|
vh
|
163bb97dea |
docs(alerts): the alarm covers far less than I claimed — measured
OnFailure fires when a unit enters failed. A unit that keeps re-earning its retries never enters failed, so the alarm never fires for it. svos-dev warned this was the consequence of the interval divergence; measuring it showed the gap is most of the box. MEASURED: a unit shaped like booth/althing-po-herald (Restart=always, RestartSec=5s, burst 5, interval 10s) ran 9 restarts over 45 seconds cycling active/activating and NEVER reached failed. No alarm would have fired. Covered: the 7 timer-driven oneshots (Restart=no, so any failure lands in failed immediately) and svos.service (burst 3 per 5min -- it genuinely gives up). NOT covered: ten daemons that flap instead, and hermes-gateway, whose start limiting is disabled outright. svos.service's divergent 5min window is deliberate and load-bearing (operator ruling 2026-09-11, 'fatal both ways'). NOT to be harmonised: that would restore the flapping the ruling forbids AND silence the alarm on the one daemon it currently works for. Whether the OTHERS move to 5min is a behaviour change for ten services and an operator call. Also recorded, svos-dev's framing of the two installer bugs: a tool that enumerates 'things that are fine' and acts on them has selected against its own subject. Neither bug failed randomly -- both were anti-correlated with purpose, working better the healthier the fleet looked. |
||
|
vh
|
ef70b2ffa1 |
fix(alerts): the installer skipped failed units — the exact ones it exists for
Two defects, both found by the tool failing to cover a unit that was already
broken. talk.service has been sitting in `failed` (exit 143) while its
containers keep serving 200 on :8092 -- precisely the "unit died, the thing
keeps serving, nobody knows" case this alarm was built for -- and the installer
had not hooked it.
1. SELECTION: --state=running skipped anything not already healthy. A unit that
is down at install time never got hooked, which inverts the tool's purpose.
Now the union of enabled unit-files and all loaded units.
2. PARSING, and this is the sharper one: systemd decorates a FAILED unit with a
leading "●", so `awk '{print $1}'` returned the bullet instead of the name,
and the sanitiser reduced it to an empty string and dropped the row. The
parser silently lost exactly the rows that matter. --plain suppresses the
decoration. Same shape as every other instrument error today -- it reported
cleanly while looking at the wrong thing.
Selection now 23 units. Deliberately INCLUDES the timer-driven oneshots
(dev-backup, ha-backup, fleet-tls-cert-check, headscale-ddns,
seat-inventory-drift, brokkr-landscape-scan, soong-ci-relay): a backup or a
cert check that fails silently is the same class, and all seven were clean at
install time so they are not a known noise source. EXCLUDES dbus, gpg-agent and
dirmngr as OS plumbing, the notifier's own template and instances, and
svos-failed-alarm -- hooking an alarm with an alarm is a loop with extra steps.
|
||
|
vh
|
02a6fc019d |
fix(alerts): suppress duplicate failure alarms, keyed on the cause
svos-dev challenged a claim this README made -- that a crash-loop yields one
message per episode -- with a measurement: one 2026-09-19 boot-gate refusal on
svos.service produced FIVE transitions into failed, and the operator got five
messages.
Measured here before accepting it, because a peer's number is still a number
someone else took: a unit with Restart=on-failure, burst 3, interval 30s
produced 7 journal failure lines and exactly ONE notifier invocation. So the
multiplier is not universal -- it needs retries spanning start-limit windows or
an external restarter. svos.service carries StartLimitIntervalSec=5min, which
is how it accumulated five.
Both conditions exist on this box, so the guard goes in as cheap insurance
rather than as a fix for something proven here. The README now states both
numbers and which restart policy each of the twelve units carries, since that
is what decides the exposure. Noted with it: hermes-gateway has start limiting
DISABLED, so it retries forever and may never reach failed at all -- worth
knowing before trusting this alarm to cover it.
Design, taken from svos-dev's shape:
- Keyed on a hash of the CAUSE (unit + result + exit status + the shape of
its last error lines), never the unit name alone. A genuinely different
failure inside the window is a new fact and must still page; suppressing by
unit would hide a second, worse failure behind the first.
- Suppression is LOGGED to suppressed.log, never silent. An alarm that
quietly declines to fire is indistinguishable from one that is broken.
⚠ The first test of this appeared to show the cooldown not working, and the
test was wrong rather than the code -- it invoked the script BEFORE failing the
unit, so the two calls legitimately saw different states and computed different
fingerprints. Re-run the way systemd actually invokes it: same cause 3x -> 1
sent, 2 suppressed and logged; a different cause inside the same window -> sent.
|
||
|
vh
|
1be39d1ea2 |
feat(alerts): failed-START alarms for every fleet user unit on nh3-dev
svos-dev found the failure neither Beszel nor Uptime Kuma can see, and measured it: a config change on 09-19 made svos.service refuse to boot, the RUNNING process predated the change and kept serving, and the service sat one restart from dark for three days. Every uptime probe was green and correct -- the thing was up. The signal that catches this is failed-START, not down. A count after that conversation: 13 running user units on nh3-dev, ZERO with an OnFailure hook. Including althing-po-herald, whose silent failure cuts infra-ops's own mail delivery -- a blind spot in the notification path every other alarm on this fleet depends on. One template, one drop-in, 12 units hooked (dbus excluded as systemd's own plumbing). Not noise: OnFailure does not fire on a clean restart or a deliberate stop, and with Restart=on-failure a crash-loop yields one message per episode rather than one per attempt. ⚠ %i, NEVER %I -- and the acceptance test is the only reason this is right. %I unescapes the instance name and systemd escaping maps "-" to "/", so the first run delivered a message for "onfailure/selftest.service", a unit that does not exist, with a spool path that tried to create directories. althing-po-herald.service would have arrived as althing/po/herald.service. It "worked" -- mail was delivered -- which is exactly the kind of success that is not one. The referring unit passes %n raw, so literal %i is correct. ⚠ The notifier never reports itself, guarded twice on purpose: the template carries no OnFailure, and the script bails on its own instance name. A notification loop is the one bug that pages you forever. Delivery is spool-first. postbox has no outbox, and an alarm for moments nobody is watching must survive the post office being one of the things that is down. If the herald itself fails, the message still REACHES the post office (postbox talks to it directly; the herald only delivers inbound pokes) -- not pushed, but stored for the next read. Acceptance-tested twice against a unit that exits 42: once to catch the %I bug, once to confirm the fix. Test unit removed. |
||
|
vh
|
5178fdea3c |
fix(homepage): The High Seat icon -> mdi-eye-outline
svos-dev's call and the better one: The High Seat is the English name for Hlidskjalf, the seat Odin watches all the worlds from, and watching every session at once is what the board does. mdi-monitor-dashboard described the artifact; the eye describes the job. Deployed and verified in /api/services, not assumed. |
||
|
vh
|
0ef25dfdb6 |
feat(homepage): add The High Seat (SVOS board, nh3-dev:8770)
Requested by svos-dev relaying the operator, 2026-09-22. Reversible work, so the relay is fine to act on without escalating. Manual services.yaml entry rather than container labels, because SVOS is a user-level systemd unit (svos.service) on nh3-dev and nh3-dev is NOT one of the five hosts in docker.yaml -- Homepage has no Docker API to discover it through. Same reason the Booth, WhereTF, talk and the infra-hermes seat are listed by hand, and the comment says so at the entry. ⚠ siteMonitor is "/" deliberately. There is no /api/health on this service: that path 404s, and a monitor pointed at it would report the board permanently down while it serves perfectly. svos-dev flagged it and it is verified here -- / returns 200 and serves the SPA (<title>The High Seat</title>). Group is Apps, which exists in settings.yaml's layout with tab: Main. An invented group name gets no tab and renders on ALL tabs, which is how Scriberr's "AI Systems" leaked across the whole dashboard in August. Icon mdi-monitor-dashboard is my choice -- svos-dev explicitly did not guess at one and offered to take a different suggestion. Verified in a browser, not just in the API: the card renders in Apps with a green site-monitor at 28 ms. |
||
|
vh
|
38bb20ceda |
fix(irv-ml1): tailscaled could never add its IPv6 mesh address, and the README described a topology two cutovers old
TAILSCALE IPv6. `tailscale status` had been reporting, continuously:
2 add route failures; first was: permission denied
adding address fd7a:115c:a1e0::6/128 from tunnel interface: permission denied
with tailscale0 carrying only 100.64.0.6/32 while headscale had assigned it
an IPv6 address it could not use.
Not a capability problem -- tailscaled runs as root with the full bounding
set. /etc/sysctl.conf:59 sets net.ipv6.conf.default.disable_ipv6=1, and
`default` is inherited by NEWLY CREATED interfaces; tailscale0 is created at
daemon start, inherits it, and the kernel returns EPERM for every attempt.
Fixed with a scoped systemd drop-in rather than flipping the global default.
That line carries no comment, but IPv6-off-by-default on a host with ~26
docker bridges reads as deliberate, and changing it would hand IPv6 to every
future bridge as a side effect of fixing Tailscale.
⚠ It must be ExecStartPost, not /etc/sysctl.d. A sysctl.d entry for a
per-interface key is applied at boot, BEFORE tailscale0 exists, and is
silently ignored -- the setting would look present and do nothing.
Also learned: setting the sysctl on the LIVE interface is not enough.
tailscaled only attempts the address at startup or on a netmap change, so
the verify failed for 60s until the daemon was restarted. Restart is part
of the operation, not an afterthought.
Verified: fd7a:115c:a1e0::6/128 present on tailscale0, health clean, mesh
and services (arbo, ytvc) up.
README. It documented the pre-headscale topology as current -- "Reachable
IP: 10.100.79.3 (WireGuard tunnel endpoint)", "No direct LAN access", and a
refresh caveat telling you to bring WG up. That sends anyone triaging this
host to the wrong layer, which is the exact tax the file exists to prevent.
Now: mesh primary at 100.64.0.6, LAN 10.6.110.50, and wg0 documented as
STILL UP with a live peer -- tailscale uses that address as its direct
endpoint, so it is load-bearing, not vestigial.
Recorded with it, because these cost hours tonight and will cost them again:
- Irvine is a TENANCY behind a Fortinet PFI does not control. Its TLS
inspection breaks Tailscale's relay and control channels (41 cert
warnings/week, 4 control-plane episodes in 14 days). Usually invisible
because direct peer paths carry the data. No fix on our side.
- Diagnose reachability with `tailscale ping`, NOT the status output:
headscale said "online" and status said "active, 19.7 GB" while nothing
on the host answered. Both are last-known state; only a round trip is
liveness.
- The ~26 docker bridges make tailscaled report captive portals.
Two stale claims corrected: the hostname rename it called "pending" is done,
and `ollama` is listed as running on :11434 when it is gone -- verified, no
unit file, nothing listening, no process. It is banned fleet-wide.
|
||
|
vh
|
94899d6fa3 |
feat(uptimekuma): normalize names off Homepage, publish the status page, restore the widget
NAMES. Homepage already answers "what is this service called", so the monitor name is now that name verbatim -- a second naming authority is how drift starts, and an alert reading "[Uptime Kuma] Beszel hub is DOWN" sends you hunting for a card that does not exist. Only two rows moved (Beszel hub -> Beszel, Dozzle hub -> Dozzle); the " hub" suffixes were mine, not the services'. The remaining mixed case is deliberate and is now documented as such. talk, vor and task-board are lowercase on Homepage and in their own repos; title-casing them here would make this board disagree with both. What actually looked messy was scripts/kuma's own ASCII-ordinal sort, which buried every lowercase name below every capitalised one. Fixed to case-insensitive. ⚠ RENAME SAFETY, which this pass needed and did not have. The seed keys on NAME, so editing a name would have read as a brand-new monitor: added fresh, with the old row orphaned, still checking, still alerting, and holding all the history. `rename_from:` names the old row for one run. Verified: both renamed monitors kept their IDs and all 67 heartbeats. Added with it, an orphan warning for any row on the board the spec no longer names -- because a forgotten monitor keeps paging. Its first cut diffed against the PRE-EDIT snapshot and so cried wolf on its own successful renames; it re-reads the board now. A warning that fires on its own correct work is worse than no warning. STATUS PAGE + WIDGET. The Homepage uptimekuma widget reads a PUBLISHED status page (/api/status-page/<slug>), not the admin API -- which is why the widget labels were deliberately absent from the rebuild: a dashboard widget pointed at a 404 is the suspected mechanism behind both of Homepage's unkillable D-state wedges, so shipping one on purpose would have been daft. The page now exists at slug `nethealth` (the pre-rebuild slug, so old references still resolve) and is DECLARED IN monitors.yaml, applied by `kuma seed`. Same principle as the notification channel: a from-scratch rebuild restores the page, the channel and the monitors together, and nothing the widget depends on lives only in Kuma's database. Verified in a browser: "13 SITES UP / 0 SITES DOWN / 100% UPTIME" on the dashboard. ⚠ saveStatusPage calls imgDataUrl.startsWith() unconditionally, so passing null throws and leaves the page CREATED BUT EMPTY -- which reads as success from /api/status-page (200, correct title) while the group list is silently blank. Pass "" instead. Commented at the call site. |
||
|
vh
|
95ab344990 |
docs(booth): the fleet index still documented the retired ask sidecars
Reported by draupnir after the Booth v0.2.0 consumer note, and it is the higher-impact half of what it found: this file is autoloaded by every agent family on this box through the shared FLEETTOOLS bootstrap pointer, so it is the first thing a cold session reads about the Booth. Draupnir was a live specimen — it read this page earlier in the same session to post a booth, before the announcement landed, and only luck of scope kept it off the broken surfaces. Stale and now corrected: `booth asks` presented as the listing verb (it is a deprecated alias and emits JSON now, not the old table); `<stem>.ask.json` / `<stem>.answer.json` presented as the storage (one `.marks.json` per booth replaced them, and the answer sidecar 404s forever, so a remote session polling it is polling nothing); the `#ask-<stem>` anchor convention. Not stale, and left alone deliberately: the `data-booth-ask=` placeholder conventions. The inline placement engine survives v0.2 and is removed in a later unit, so those still work exactly as documented. Added: `booth marks` and `booth marks-import`, the marks.json HTTP read path, the partially-answered-counts-as-open change, and the 409-on-damaged-file behaviour. |
||
|
vh
|
6f0a9b9fae |
feat(alerts): generalize the althing bridge, wire Kuma to it, retire chamber
THE BRIDGE. `beszel-althing` hardcoded a "[Beszel] " subject prefix and a
Beszel hub footer from when Beszel was its only caller. Routing Uptime Kuma
through it unchanged would have delivered Kuma outages labelled [Beszel],
pointing the reader at the wrong dashboard -- an alert that lies about its own
source is worse than no alert.
Now a route registry: /beszel and /kuma, each with its own prefix, footer and
payload parser, because the tools do not agree on a shape (Beszel sends
{title, message}; Kuma sends {heartbeat, monitor, msg}). Generalising cost a
dict; a sibling service would have cost a second unit, a second port and a
second thing to notice had died.
Renamed beszel-althing -> althing-alert-bridge with it. A service named after
one consumer that carries two is the invisible coupling that sends a future
session looking in the wrong place.
⚠ /beszel IS FROZEN and this refactor proves it rather than claiming it. The
three original tests were kept BYTE-UNCHANGED -- including the one asserting
the exact postbox argv -- and deliver() still defaults to the Beszel route so
they exercise it. A new test asserts the Kuma footer never leaks into a Beszel
body or vice versa. Verified live after the rename: a real POST to /beszel
landed as "[Beszel] BRIDGE RENAME CHECK" with the correct hub footer, read back
from the thread rather than trusted from the receipt.
Payload shapes are parsed HERE, not via Kuma's custom-webhook-body feature,
because Kuma's notification config lives in its own database -- and that
database was destroyed and rebuilt from scratch hours ago. Anything living only
in a tool's DB is lost on the next rebuild; format knowledge belongs in git,
next to a test.
parse_kuma also handles the monitorless case. testNotification and cert-expiry
alerts carry no monitor and no heartbeat, and the first cut fabricated "unknown
monitor is ?" from them -- caught by sending a real one and reading the subject,
not by the suite. Fixed, pinned, and the earlier test asserting the bad
behaviour was corrected rather than worked around.
KUMA IS NOW WIRED. scripts/kuma gained notification support and the channel is
in monitors.yaml, seeded BEFORE the monitors and with applyExisting, so a
rebuild restores alerting and not just detection. Ground truth from the DB:
13 of 13 monitors carry the channel.
⚠ A THIRD instance of the same class of bug, worth naming: notifications() is
pushed as `notificationList` at LOGIN ONLY -- there is no event to ask with. The
first cut cleared the captured value before waiting, discarding the only copy it
would ever be sent, then blocked for the full timeout and reported an empty
list. That reads exactly like "no channels configured" and is a lie. Same family
as the getMonitorList ack-vs-push trap, different shape.
End-to-end, both shapes, read back from the inbox:
[Uptime Kuma] Homepage is DOWN + target + board link
[Uptime Kuma] althing (infra-ops) Testing (no fabricated subject)
[Beszel] BRIDGE RENAME CHECK + hub footer, unchanged
ALTHING CHAMBER RETIRED (operator). Three of its four containers had never
started -- created 2026-09-19, StartedAt epoch-zero, 0 restarts -- so :7881
refused, and nothing was watching it. Only its valkey was running, on the
project's own network with no external consumer. Stack, compose/build/conf dirs
and the local image removed; the Homepage card went with the label.
|
||
|
vh
|
3a85a6bce1 |
feat(uptimekuma): rebuild on 2.5.5 as the fleet's service layer, with a client
Homepage sat dead for three days in September while every monitoring tool
reported correctly. Beszel said its host was up -- it was. Uptime Kuma was not
watching it. The dashboard fell into the seam between two working instruments.
Measured before changing anything:
- Beszel: 18 hosts x {Status, CPU, Memory, Disk, Temperature}. Its alerts
table is (system, name, value, min) -- there is NO url column, so it is
structurally incapable of "this endpoint should return 200". Not a config
gap; the data model.
- Uptime Kuma: 6 rows, 2 of them folders. Four real monitors, all firewalls.
- So the two are NOT redundant. They are disjoint, and the service layer
between them was empty.
REBUILT FROM SCRATCH, operator-authorised ("uptime-kuma was never really
used... you can even dump the existing container and config"). Nothing was
migrated, which also skipped the one-way v1->v2 database migration.
- Pinned to 2.5.5. `:latest` is a documented trap now: upstream keeps it on
the 1.x line, so an August 2026 pull produced an image BUILT 2024-12-20
running 1.23.16. Verified by digest -- latest and 1 share one digest while
2/next carry 2.5.5. Pinned exactly, not floating on 2, for the same reason.
- Moved esh-docker-vm -> ana-docker. House placement rule puts cross-site
services beside the Beszel and Dozzle hubs, and esh-docker-vm has wedged
unkillably twice in four months. A monitor also cannot report the failure
of the host it runs on, so it should not share a failure domain with the
host layer.
- Normalised restart: always -> unless-stopped, which the 2026-08-18 README
flagged as worth doing on the next deliberate touch.
- UPTIME_KUMA_DB_TYPE=sqlite in the compose skips 2.x's interactive database
screen, so the stack comes up ready rather than parked on a form.
scripts/kuma is a first-party Socket.IO client, because both obvious paths are
wrong: there is no REST CRUD API in EITHER major version (server/routers/ holds
exactly two files, /metrics + badges + status pages), and the community wrapper
uptime-kuma-api is abandoned -- last release 2023-09-26, ceiling 1.23.1, no 2.x
support ever.
⚠ getMonitorList's callback returns only {ok:true}; the list arrives as a
SEPARATE pushed monitorList event. Reading the ack yields an empty board that
looks authoritative -- which duplicated all 13 rows on the first re-seed
before the bug was found. The client now waits for the push, and carries a
dedupe verb because of it.
13 monitors seeded from monitors.yaml, keyed on name so a re-run updates rather
than forking the board -- proven by re-running it (0 added, 13 updated), not
assumed. Every URL was probed before being written: all 200. A board that ships
red teaches everyone to ignore it.
Verified: 13 rows, no duplicates, all UP with "200 - OK" read from the database
WITH its WAL (a first read of kuma.db alone showed a stale 26 -- the copy
predated the deletes). Homepage renders exactly one Uptime Kuma card.
⚠ NOT YET WIRED: notification delivery. The board detects but tells nobody,
which is the same gap this work exists to close. The beszel-althing bridge
hardcodes a [Beszel] subject prefix and hub footer, so routing Kuma through it
unchanged would mislabel the alerts. Needs a decision before it is generalised.
|
||
|
vh
|
14bd95d76d |
chore: extract the Booth to its own repo (vh/booth)
The Booth is now one of the most-used fleet tools -- 17 agent handles post to it daily -- and it is taking an information-architecture rework plus a cross-agent SVOS design retrofit from design-dev. That work wants its own ROADMAP, contracts and blast radius, not the fleet-infrastructure repo's. All 29 commits moved with it via `git subtree split`; the history carries real lessons (two shipped-dead controls, the verbatim-injection traps) that a squashed import would have thrown away. Live service repointed and verified: the user unit and the ~/.local/bin/booth symlink now resolve into ~/development/booth, healthz answers, all 24 booths intact. services/booth/ keeps a pointer README, same shape as the chatterbox-fast and tts-stack extractions. |
||
|
vh
|
e52def115c |
memory: snapshot — the ops log, and a day spent on instruments that report without looking
Archived 15 entries (Recent decisions 14, Tried and abandoned 1) oldest-first to archival-memory.md; 5 held back on the open-deferred-work guard and 164 on the 14-day guard, so the index stays over the soft cap at 477 lines. An over-cap file that keeps live decisions beats a scannable one that lost a belayed item. Four new detail files cover the day: the ops log and its four self-inflicted failure modes, the Booth's two dead controls and the four-iteration layout probe, the Gitea org grant plus the dead claude-bot token that had been misreporting permissions, and the disk triage that rescued a LoRA adapter from a directory this box sweeps at three days. lv-mccarthy's run outcome remains unverified after two days and is the first line of the in-flight section and step 1 of the handoff. |
||
|
vh
|
2e08edcaab |
fix(draupnir-slicer): warn that every invocation writes result.json to CWD
brokkr-smithy-dev exercised the slicer on real geometry and found OrcaSlicer writes result.json into the working directory on --info. I verified it here and it is worse: --help does it too, which is as read-only as an invocation gets. An empty directory gained a result.json from --help alone. Encoded in the wrapper and the Dockerfile rather than left to be rediscovered, and the wrapper is now committed beside the image that backs it instead of living only in a home directory on one host. ⚠ The reason they found it and I did not is worth keeping in the header: I verified --help and called the slicer done. Usage text proves the binary starts, not that it ingests our geometry or that it behaves while doing so. They ran both controls through to STL and got manifold=yes, 520 facets, return_code 0 on each -- which is the verification I should have run before reporting the provisioning complete. |
||
|
vh
|
e574b91ff3 |
feat(irv-ml1): containerise OrcaSlicer for Draupnir's sliceability check
Not tidiness — necessity, established by ldd rather than assumed. Every OrcaSlicer release ships only an Ubuntu 24.04 AppImage, which needs GLIBC_2.38 and GLIBCXX_3.4.32. irv-ml1 is Debian 12 on glibc 2.36. That gap is not installable without moving the host to Debian 13, which is not a thing to do to a box running twelve GPU and audio services in order to slice a plate. Reaching back to an Orca built for Ubuntu 22.04 would run, and would mean pinning permanently to a stale build -- the exact trap brokkr-smithy-dev rejected when they chose Orca over an old PrusaSlicer tag. The container keeps the slicer current AND leaves the host alone, which is the same argument that made FreeCAD an AppImage, taken one step further because the host could not satisfy this one at all. The AppImage is extracted at BUILD time; --appimage-extract-and-run re-extracts to /tmp on every invocation, which is wasted seconds and wasted disk per slice. ~/bin/orca-slice wraps it so the dispatcher calls a binary and the artifact root is mounted at the same path inside and out. ⚠ Recorded honestly: I installed libwebkit2gtk-4.1-0 and 8 dependencies on the host chasing errors serially before enumerating with ldd, and only then found the glibc wall behind them. Those nine packages are unnecessary. Left in place rather than autoremoved on a box running production services; flagged for the operator. |
||
|
vh
|
34c4179d55 |
feat(irv-ml1): provision the Draupnir geometry engine, acceptance-tested
Operator-authorized. build123d 0.12.0 + OCP, numpy 2.4.6, trimesh 5.1.0 in a 3.11 venv, FreeCAD 1.0.0 headless as an AppImage, artifact root on smithy. The acceptance test is the point and it PASSES: brokkr-smithy-dev shipped a contrastive control pair whose every property is construction-known, so the install is checked against arithmetic rather than against a render. Both parts build, both bounding boxes are exactly 40x40x5, and both volumes match to float tolerance. These scripts had never been executed by anyone -- build123d is deliberately absent from the authoring box -- so this was their first run. FreeCAD is an AppImage BY CHOICE: the distro package drags a GUI/Qt dependency tree onto a host running twelve GPU and audio services, where an AppImage is self-contained, pinned and removable with one rm. Slicer DEFERRED rather than guessed. PrusaSlicer's latest release ships no Linux AppImage at all, and a guessed release filename had already produced a 4 KB error page in this same session. It is the one item they said could lag. Two things worth keeping in the header: irv-ml1's ssh config pins the gitea host to gitea_brokkr_smithy with IdentitiesOnly, whose comment is irv-ml1-r14-deploy and NOT the arbo key that sorts first in ~/.ssh/*.pub -- I registered the wrong one first, and the failure names the right one. And `ssh irv-ml1` is lkraven with no NOPASSWD sudo while `ssh infra-ops@irv-ml1` has it, so only the artifact-root step needs the other identity. The deploy key was registered with claude-bot's own token rather than the operator's site-admin token -- the org grant from earlier today doing real work on its first use. |
||
|
vh
|
07c9cb293b |
fix(booth): release was painting over the delete ×, which was unclickable
Operator: "release button covers delete button". Measured before touching
anything: release 58x24 at (323,266), × 30x30 at (349,268) — 30x22 px of
overlap on a 30px button, and `elementFromPoint` at the ×'s centre returned the
release form. The × I added yesterday was 100% unclickable from the moment it
shipped.
Cause: both were `position:absolute` on the same corner, each with its own
guessed offset, and `release` is the later sibling so it won. Replaced with one
flex row positioned once — release left, × right at the card corner where the
ephemeral lane's × already lives, so muscle memory transfers and neither can
drift back on top of the other when a label changes width.
Verified by measurement, not inspection: overlap 0 px, and clicks at each
control's centre now land on that control. The ephemeral lane's × and ★ were
re-checked and are unaffected.
ADDS scripts/layout-probe.py, because markup inspection STRUCTURALLY cannot see
this and I have now shipped two dead controls in two days by reading templates
instead of rendering them. It asks a real browser what a click at each
control's centre would hit.
It took four iterations to become trustworthy and the failures are the point:
1. `top.contains(el)` counted an ANCESTOR overlay as a hit -- the exact case
it exists to catch. Version one reported OK for a real overlay.
2. elementFromPoint is viewport-relative, so everything below the fold read
as occluded and buried the real findings.
3. getBoundingClientRect() on a WRAPPED INLINE element is the union of its
line boxes, whose centre lands in the gutter between them -- three zip
links reported occluded by their own parent. Real geometry, wrong question.
Only the fourth version fires on a genuine overlay while staying silent on the
clean page. Both controls were run; a probe never seen to fail proves nothing.
173 tests pass.
|
||
|
vh
|
751eecb771 |
fix(booth): the reveal button was inert; add kept-lane wipe and in-booth keep
Three operator reports, one of them a real defect I had claimed was working. THE REVEAL BUTTON DID NOTHING, for a day. Its handler sat after the content block's closing tag, and a child template's out-of-block content is silently DISCARDED by Jinja. The button rendered. The handler never reached the browser. Two commits and a README paragraph said click-to-reveal worked, and the suite passed the entire time because nothing asserted against the SERVED page -- the template really did contain the code. Two guards, both confirmed to FAIL when the defect is reintroduced rather than merely added and assumed protective: * test_reveal_handler_actually_reaches_the_served_page greps the response * test_no_orphaned_markup_after_the_content_block guards the structure While moving it, caught a second instance of the same class: the explanatory comment I wrote for the fix contained a literal Jinja endblock tag, which Jinja would have parsed as a real tag and used to close the block early. KEPT-LANE ×. Wiping a kept booth required release-then-find-it-in-the-other- lane. That protected nothing and cost a hunt -- the board you just released is loose in a feed that turns over, and you have to go find it to finish a job you had already decided on. Protection now lives in the confirmation, which names the booth and says KEPT, instead of in the number of lanes you must traverse. Release stays as the reversible option. IN-BOOTH KEEP. `☆ keep` / `★ kept — release` beside "Wipe now", so promoting does not mean navigating back to the index. The booth page did not previously know its own kept state; it does now. Both post a `next` field to stay put -- and `next` is a form field, so it is attacker-controlled: only same-site absolute paths are honoured, with `//host`, schemes and backslashes refused, tested. 173 tests pass. |
||
|
vh
|
8227a81272 | memory: ravenpen.com registered — registrar, expiry, and the scope boundary I can't cross | ||
|
vh
|
838ea60fc0 | homepage: surface the infra-hermes bus seat status page (nh3-dev:8766) | ||
|
vh
|
7fe4102458 |
fix(backups): stop saying STALE over a fleet whose every backup is fresh
The check collapsed two different findings into one verdict. On 2026-09-20 it printed "RESULT: STALE" while reporting 37 FRESH layers and zero stale ones -- every backup body provably current, the three ❌ rows all yesterday's pre-fix runs aging out of the 36h window. infra-hermes caught it in triage: a reader, or a forwarder, could page someone over a state where nothing is stale. STALE is a claim about backup AGE. A job that ran and errored is a different claim with different urgency. They now have different words and different exit codes: 0 all backups fresh 1 STALE -- a body past the threshold, or an endpoint down 3 ERRORED-JOBS -- every body fresh, a vzdump job errored recently The alert wrapper mirrors the code and matches its own wording to the finding: 🟡 "Backup jobs errored — all bodies fresh" instead of 🔴 "Backup freshness ALERT", and it now exits with the check's code rather than flattening everything to 1, so `systemctl status` distinguishes the states too. This is the same defect class the rest of this script was built to fix, one level up: not an instrument that fails to look, but one that looks correctly and then reports the wrong word for what it saw. An alarm that cries outage over a healthy fleet earns being ignored exactly as fast as one that stays silent over a broken one. Verified all three states by forcing each: BACKUP_JOB_WINDOW_HOURS=1 -> exit 0, default -> exit 3, BACKUP_MAX_AGE_HOURS=1 -> exit 1. |
||
|
vh
|
271cb11b70 |
fix(booth): put the blur toggle on every item kind, and make it look like a control
The operator asked "no UI option to blur/unblur?" and was right twice over. MISSING ENTIRELY ON TWO OF THREE BRANCHES. booth.html renders docs, media and everything-else through separate <figure> blocks. The toggle went into the media branch only, so inline docs -- the branch that puts readable text straight on the page, the one that needs blur most -- had no control at all, and `other` files only got a caption row if they happened to carry a caption. This is the SECOND time this feature shipped having patched some branches and not others; the blurred class itself had the same gap one commit ago. So the toggle is now a single Jinja macro called from all three sites, which makes "patched two of three" impossible rather than merely unlikely, and test_every_item_kind_gets_exactly_one_blur_toggle counts toggles against figures across mixed kinds so a fourth branch cannot quietly skip it. INVISIBLE WHERE IT DID RENDER. v1 was a bare `◌` at 0.78rem in --muted with no border, no label and no hover affordance. It now reads `◌ blur` / `◉ blurred` with a border, matching the other per-item controls. A control nobody can find is a control that is not there. Docs get it in the doc bar beside ⤢ ⬇ ✕, with stopPropagation so submitting it does not collapse the <details> it lives inside. Verified live on all three kinds: 3 figures, 3 toggles, and the POST round trip blurs and un-blurs. 167 tests pass. |
||
|
vh
|
2e7fd7157e |
feat(booth): close the keep round trip, and add cosmetic per-item blur
Two operator requests.
KEEP, BOTH DIRECTIONS. The kept lane could already release a booth back to
ephemeral, but an ephemeral booth could only be promoted from a shell -- so the
round trip was closed only if you had ssh. The /keep route and the `booth keep`
verb both already existed; only the button was missing. Adds ★ to the ephemeral
card, mirroring × on the other shoulder.
BLUR. Per-item cosmetic censoring: `booth blur <name> <file>...`, a ◌/◉ toggle
in each caption row, and 👁 click-to-reveal. State is `.blurred` in the booth
dir, one booth-relative path per line -- the same filesystem-is-the-state idiom
as .pins and .forever. An empty set deletes the marker rather than leaving a
zero-byte file, so `ls -a` tells the truth.
⚠ BLUR IS NOT ACCESS CONTROL, and the code, the docs and a test all say so on
purpose. A blurred item is still served at its own URL, still in the zip, still
on disk. The Booth has no auth by design. test_blur_is_cosmetic_the_file_is_
still_served asserts the 200 deliberately: if someone later "hardens" this into
a 403 that test fails, and it should, because half-implemented access control is
more dangerous than none.
Reveal is per-viewer and never persisted; a reload re-hides. With JS off an item
stays blurred, which is the safe direction to fail in.
Two things the first pass got wrong, both caught by checking rather than
assuming:
* The cover thumb. index.html has IDENTICAL markup in the kept and ephemeral
lanes, so a single-occurrence replace patched only the kept one and the
ephemeral front page happily displayed the thing someone had hidden. The
test that caught it was itself wrong first -- it matched the bare string
"blurred-thumb", which is in base.html's stylesheet on every page and so
passed in both states. It now asserts the attribute.
* Inline docs render through their own <figure> branch and were left
unblurred -- the branch that puts readable text straight on the page, so it
needed blur more than images do. The suite passed; a live curl caught it.
165 tests pass (154 pre-existing, unchanged).
|
||
|
vh
|
8305145ce1 |
docs(fv-ml1): record the 20 A circuit, its real ceiling, and what it forbids
Operator confirmed 2026-09-19 that fv-ml1 and the R420 running OPNsense are the only loads on a dedicated 20 A circuit. The governing number is 1920 W, not 2400: a GPU inference host running for hours is a continuous load, so NEC's 80% rule applies. Worst case lands at ~1625 W with the current caps -- about 85% of budget. Measured via the BMC rather than assumed: 390 W instantaneous, 461 W max over a 2423 s sample, with GPUs at idle, giving a ~313 W non-GPU baseline. Compare the GPU caps against the 300 W stock TGP, NOT the 325 W firmware ceiling. The operator corrected this: 275 W across four cards saves 100 W, not the 200 W you get by measuring against a number nobody would ever run at. Stock 300 W would put the circuit near 90%, which is not illegal but leaves nothing for a heavier R420, PSU efficiency, or a warm day. Keep the caps. The coupling matters more than the trip. OPNsense IS the Fountain Valley edge and shares the breaker with the thing most likely to trip it, so an overload takes the router with it and removes the remote path needed to diagnose or power-cycle anything. fv-ml1's four PSUs do not help -- PSU redundancy protects against a PSU dying, not against the circuit going away. Three things are explicitly NOT measured and the file says so: fv-ml1 under real 4-GPU load, whether the BMC reports AC input or DC output, and the R420's actual draw. Treat 1625 W as a floor. Also corrects the hardware section, which claimed 2x GPUs. nvidia-smi reports four. |
||
|
vh
|
cebffae56d |
docs(ops-log): record how this instrument has failed, not just the fixes
Four attribution failures shipped on the day it was built, and every one of them RECORDED SOMETHING -- just not something findable or attributable. The individual fixes are in the commit log; the PATTERN was not written down anywhere, so a future session would meet each one as an isolated bug rather than as the fifth instance of a known shape. infra-hermes asked for this after the fourth one: 'worth an audit line item so the pattern is visible, not just the fixes.' He was right. The operative warning is at the bottom: absence of an entry is not evidence of absence of the event until you have checked this tool could have recorded it. |
||
|
vh
|
f3b68e23a4 |
fix(ops-log): record the host's NAME, not the ssh target
elway passes its `host` argument straight through, and that argument is an ssh target. Five records of a real jobs.cfg change on esh-pve were written under host `infra-ops@esh-pve`. The consequence was not cosmetic. infra-hermes triaged the ESH backup job errors, queried `--host esh-pve`, got nothing back, and correctly reported the 05:29 exclusion of VM 102 as an unattributed change. The record had existed the whole time, five lines of it, under a name nobody would think to ask for. A log you cannot query under the obvious name is not a log. `normalize_host` strips any `user@` prefix on write AND on query, so records written before this fix are findable too, and `audit` -- which matches records by host name -- stops missing them as well. Verified in both directions: a record written with `infra-ops@esh-pve` and one written with `esh-pve` now both return for `--host esh-pve`, and the five historical elway entries surface under that query. |
||
|
vh
|
2e823538f7 | memory: snapshot — the 3.7.0 rollout, and three tries to attribute a commit | ||
|
vh
|
9141a41f6c |
fix(ops-log): attribution must run FIRST, not after another hook's exits
The commit hook was APPENDED to .git/hooks/post-commit so as not to disturb
graphify's block -- and that made attribution a subordinate clause of
graphify's control flow. graphify owns EIGHT `exit 0` paths (rebase / merge /
cherry-pick in progress, GRAPHIFY_SKIP_HOOK, no changed files,
graphify-out-only, no python found), and every one of them returns before
reaching an appended line.
infra-hermes reported commits going unlogged and proposed a lock race: the
ops-log fcntl path held by graphify's detached rebuild long enough for a
best-effort `|| true` invocation to drop. Plausible, and wrong. The cause is
ordering and it is deterministic.
|
||
|
vh
|
eaaacc1c84 | Refresh graphify report | ||
|
vh
|
4e778aeddc |
fix(ops-log): say 'unattributed' when there is no handle, not a unix login
The fallback recorded `lkraven@nh3-dev` when ALTHING_HANDLE was unset, which reads like an answer while being the absence of one. It now records `unattributed(lkraven@nh3-dev)` -- handles never contain @ or parentheses, so the fallback cannot be mistaken for an identity. Found by the commit hook's first independent test, which was not one I ran: infra-hermes committed |
||
|
vh
|
0fe4da64c4 |
Guard seat test scripts behind __main__; clean _t_* cron output residue
Both harnesses executed top-to-bottom on import, so anything collecting them (pytest, a glob import) ran live-store side effects. main() guards plus finally-block cleanup of the _t_* cron output dirs the fabricated jobs leave behind. |
||
|
vh
|
0dc8e9096e |
feat(ops-log): record who committed, by althing handle
Two agents share ONE checkout of this repo on nh3-dev, and every commit here
is attributed to Vuong Hoang by convention -- so a commit's author line says
nothing about which agent made it. The ops log closed that gap for HOST
changes; it did not cover commits to the shared tree.
Found the hard way today:
|
||
|
vh
|
3e7d3a3e3d |
fix(ops-log): a sub-tool must not drop the claim wrapping its caller
deploy-stack.sh claims and releases around its own work. When the agent already held a longer claim for a multi-step operation, the deploy refreshed it, then released it on exit -- silently dropping the protection partway through the very operation it was guarding. Caught live: a 45-minute claim on nh3-docker/althing-post-office, taken to cover a build-push-deploy-verify rollout, was gone by the time the rollout finished. Nothing refused anything, and nothing said so. `ops-log claim` now exits 10 when the claim was already the caller's, and leaves the holder file untouched. deploy-stack.sh treats 10 as "not mine to release". Untouched matters as much as the exit code: a refresh would overwrite the reason and TTL the original claimant chose, so a deliberate 45-minute "3.7.0 rollout in progress" would degrade into "deploy-stack.sh <host> <stack>" and whoever got refused would read the wrong story. Verified in three states: no pre-existing claim -> deploy claims and releases; caller's own wider claim -> deploy refreshes nothing and leaves it standing with its original reason; another agent's claim -> still refused with exit 3. |
||
|
vh
|
920f37c6c2 |
chore(althing): pin the post office to 3.7.0
Canonical pin bump for the 3.7.0 rollout forseti approved and released (althing 6db955f, tag v3.7.0). Image built from that tree and pushed as sha256:df0709b3f532557aebb47d6ece486ae32e851776f732862094608d55eefcd4e7; previous pin was 3.6.3@sha256:978f8553 and remains the rollback target. Content-verified rather than tag-verified: `postbox --version` inside the built image reports 3.7.0 before it was pushed, and again inside the running container after. A tag is a claim. Also adds playbooks/nh3-extdev-althing-upgrade.yaml -- a version-parameterised upgrade for the box a git pull cannot move. The existing nh3-extdev-althing-v3.yaml is the historical v2->v3 CUTOVER and is left alone rather than mutated per release; it still carries v2-retirement steps that document what happened. Every check in the new playbook asserts what the installed entry point reports, including one that the herald restarted within the last 5 minutes rather than surviving from before -- "active" does not mean "running the code you just installed". |
||
|
vh
|
e43e2626f1 |
Add althing seat monitor-behavior test artifacts
Disposable harnesses from the wake-RFI thread (01M2WT3F): exit1-behavior-test.py covers the monitor exit-code/gate path including a sustained-failure/no-backoff assertion; hash_gate_repro.py covers the detection-time hash persistence crash-before-read boundary with consumption/unchanged/changed controls. Both fail nonzero on mismatch. Committing so seat artifact provenance rides git rather than file mtimes. |
||
|
vh
|
5fee7868b3 |
fix(playbooks): don't let the preserve step depend on its own guard
/etc/restic is 0700 root, so before elway
|
||
|
vh
|
519fca4669 | memory: snapshot — a dead alarm, two blind instruments, and an elway bug that reported success | ||
|
vh
|
e50e57b5f2 |
chore(esh): exclude VM 102 from the nightly backup job
esh-vm-workstation is a Windows 11 Parsec/RDP sandbox -- no password, no state anyone needs recovered (operator ruling, 2026-09-19). Its nightly vzdump has failed since ~2026-09-06 with "timeout waiting on systemd", and the ESH job has reported `job errors` every night as a result, so a guest nobody needs backed up was manufacturing a real job failure daily. The job uses an explicit `vmid` INCLUDE list, so exclusion means removal from that list -- there is no `exclude` directive here, which is the same distinction that makes esh-scale CT 108 invisible in any exclude list. The reason now lives in the job's own `comment`, and check-backup-freshness.sh reports 102 under "NOT BACKED UP BY POLICY" rather than going quiet about it. This is a policy decision, not a fix: the systemd-scope failure on stopped guests is NOT diagnosed, merely no longer reached on this node. A stopped guest that DOES matter would hit the same wall. The playbook carries two hard-won warnings in its header. Read the job through the API, never `grep -A<n>` on jobs.cfg -- `vmid` sorts last in the block, so adding the comment line pushed it outside the window and every condition silently inverted. And use YAML block scalars for the JSON-extraction expressions; double-quoted ones did not survive YAML unescaping plus elway's bash -c quoting. Verified: vmid 100,101,103,104,105,106,107; the other seven guests still covered; job still enabled; a rerun skips the change rather than repeating it. |
||
|
vh
|
754db4bc0b |
fix(elway): evaluate when:/creates:/changed_when: with the step's own sudo
Conditions ran unprivileged no matter what the step declared, and that fails in the dangerous direction. A root-requiring `when:` -- `pvesh get ...` exits 255 for a non-root user -- returns non-zero, elway reports the step `skipped`, and a playbook that never performed its change reports overall OK. "Skipped" is indistinguishable from working idempotency, so the run looks correct. Found the hard way on esh-pve: three consecutive runs of an exclusion playbook reported success while changing nothing. Only the verify phase caught it, by continuing to report the thing the steps claimed to have handled -- which is exactly why verify runs unconditionally. `creates:` had the same fault from the other side: a path under /root is unreadable to the login user, so `test -e` said absent and the step re-ran every time. It now correctly reports the file as already present. Sudo-less steps are unaffected: their conditions still evaluate as the login user, which is what they mean. Only a step that declares sudo: true gets privileged condition evaluation, so no existing playbook changes meaning unless it was already silently broken. |
||
|
vh
|
ba26852ec6 |
feat(backups): catch a job that runs and errors, not just one that goes stale
Snapshot age is structurally blind to a backup job that executes every night
and fails every night. Nothing new is written, so the group simply ages, and
the fault only surfaces once it crosses the 48h threshold -- days after the
first failure, with the evidence sitting in a task log nobody reads.
Two live cases, both found today and both invisible for a week by this exact
mechanism:
* esh-nas-pve CT 107 (vm-jellyfin): a backup run died around 09-06 and left
a stale `backup` lock, so every nightly since failed instantly with "CT is
locked (backup)". Age named it on ~09-12. Task status would have named it
on 09-07.
* esh-pve VM 102 (esh-vm-workstation): failing nightly since ~09-06 with
"timeout waiting on systemd". Same six-day gap.
PVE already records every task result in /var/log/pve/tasks/index. This reads
it on all four non-tenant PVE nodes and reports any vzdump in the last 36h
whose status is not OK, as its own section that sets the exit code.
It found a third case on its first run: esh-nas-pve's job had been reporting
`job errors` nightly while every guest on that node read 0-1h fresh, so no
age-based check could ever have flagged it.
Window is BACKUP_JOB_WINDOW_HOURS (default 36 -- longer than a daily cycle so
one missed run does not hide a failure). A node whose task log cannot be read
is reported, never assumed healthy.
|
||
|
vh
|
5be25be081 |
fix(backups): stop paging on guests that are deliberately not backed up
ana-scale (CT 114) is a subnet-router LXC, excluded from vzdump on 2026-09-07
after a backup lock on its ESH counterpart blackholed that entire site. The
freshness check knew nothing about that and reported it 🔴 STALE every single
morning, which is how an alarm teaches you to ignore it.
Such guests now get their own section: printed every run, never hidden, and
not counted as a fault.
The subtlety is in how coverage is computed, and the obvious implementation is
wrong twice over:
* Reading one job's `exclude` list gets ana CT 109 (ana-nas) exactly
backwards. It IS excluded from the 03:00 all-guests job AND it has its own
dedicated 22:00 job. Suppressing on the exclude list would have stopped
alarming on a guest that is genuinely backed up -- trading a noisy alarm
for a blind one.
* ESH's job uses an explicit `vmid 100..107` INCLUDE list, so esh-scale 108
is excluded by OMISSION and appears in no exclude list at all.
So coverage is a union across every enabled job on the cluster, and a guest is
"intentionally not backed up" only when none of them covers it.
If coverage cannot be read, nothing is suppressed and the gap is reported: an
unreachable PVE node means we do not know, and a backup alarm must fail loud.
The SureFire namespace is never consulted (tenant property), so its guests can
never be suppressed either.
Verified against the live fleet on all four paths: CT 114 suppressed; CT 109
NOT suppressed despite being in an exclude list; esh-vm-workstation 102, which
a job really does cover and which really is failing, still reports STALE; and
with a PVE node made unreachable, 114 returns to STALE with the gap named.
|
||
|
vh
|
e979ccb337 |
fix(backups): the freshness alarm had no wire — reconnect it and make it testable
The daily backup-freshness check has been unable to raise an alert since the
2026-08-28 althing v3 cutover. It called althing-cli, which v3 DELETED rather
than deprecated. The check itself never stopped working: it detected three
stale backups every morning and told nobody, and the only trace was a WARN
line inside a unit that was already reporting `failed` for the stale backups
themselves. Three weeks, silent.
Four changes, because swapping the binary alone would have left it dead:
* althing-cli -> postbox.
* Add ALTHING_POST_OFFICE to the systemd user unit AND to the installer that
writes it. postbox has no default address by design and a user unit
inherits nothing from the interactive shell, so the binary swap on its own
would have failed with a different message. Fixing only the live unit
would have been undone by the next installer run; the two are now verified
to agree.
* Recipient infra-ops -> infra-hermes. This runs AS infra-ops, so the old
address mailed the alarm to itself — the mirror trap named in CLAUDE.md.
Day-to-day checks are infra-hermes's half of the split; he escalates.
* Split the exit codes. 1 now means "backups stale, someone was told";
2 means "the alert path itself failed". A broken alarm is a worse fault
than the thing it watches and must not be indistinguishable from it.
Adds --test-alert: a positive control that sends a real message through the
real path on demand. The wire was cut for three weeks precisely because
nothing ever exercised it in the healthy state, and an alarm whose success
path is never run is not known to work.
Verified: positive control delivered; missing-address and unreachable-post-
office both correctly exit 2; a real run through systemd delivered the alert
and exited 1.
|
||
|
vh
|
ffe7b24935 |
feat(ops-log): attribute host changes across two agents sharing one identity
infra-ops and infra-hermes act as the same OS identity and dockerd does not
log exec per caller, so host-side changes carry no fingerprint. Git cannot
close the gap either: every commit here is attributed to Vuong Hoang by
convention, which is correct for authorship and useless for attribution.
On 2026-09-18 a second session edited the searxng stack mid-deploy, crash-
looping fleet search for ~4 minutes, and the author was unidentifiable.
scripts/ops-log records one line per host-changing action and holds a
lightweight claim so two agents do not deploy the same stack at once.
Four design questions, settled:
* Central on nh3-dev, not per-host and not the post office. Both agents
run as the same unix user there, so one file is shared with zero
provisioning. Per-host needs a writable path on ~25 heterogeneous boxes
and stores "we changed host Y" on host Y. journald looked free but shows
an unprivileged reader only their own _UID, which would have split the
log silently between the infra-ops and lkraven halves of the fleet.
* The claim is advisory and enforced in the tooling. deploy-stack.sh
refuses a foreign claim across the diff, the prompt and the apply -- the
whole review window, which is where the collision happened. Acquire is
mkdir, so it is atomic rather than probably-fine. Stale claims auto-break
and the break is recorded.
* Writers are automatic. deploy-stack.sh and elway record themselves; a log
that depends on remembering is the same class of instrument as a health
check that passes in both states.
* There is a detector. `ops-log audit` asks each host what changed on disk
and compares it to the newest log line for that stack, covering the
manual ssh-and-edit path the automatic writers structurally cannot.
ops-log being absent or broken never blocks a deploy; only a live foreign
claim does. `ops-log baseline` marks the 136 stacks that predate the
instrument so the detector starts from today rather than reporting the whole
fleet forever and training us to ignore it.
An unreachable host reports INCOMPLETE and exit 5, never clean.
|
||
|
vh
|
4d826e17e3 |
memory: infra-hermes is infra-ops' assistant, and the ops log is assigned
Operator ruling 2026-09-19, recorded in three places because each serves a different reader. CLAUDE.md gets the structural facts so a fresh session has them without reading anything else; persistent-memory gets the dated decision and the assigned work; auto-memory gets the durable working relationship. The division: infra-ops keeps improving infrastructure tooling plus the hard calls, infra-hermes takes day-to-day checks, triage and routine operations, either may perform infra ops, and infra-ops may task him downward while he escalates upward. He is explicitly NOT Miranda. The global CLAUDE.md names Miranda as the sole trusted relay of operator authority and that exception does not extend to him, so a directive he relays is information rather than authorization — reversible relayed work executes, irreversible or fleet-affecting goes to the operator. He has acknowledged it in those terms. ⚠ The two handles differ by one character in the middle of a word and the fleet's OS identity is infra-ops, so a misaddressed page still mails the sender themselves. That trap is now documented alongside the existing mirror warning rather than replacing it. Building the ops log is assigned and not started. Two agents now share one fingerprint-less OS identity: ssh infra-ops@<host> is either of us and dockerd exec is not logged per-caller. The precipitating incident is on the record — 2026-09-18, a second session edited the searxng stack mid-deploy, crash-looped fleet search for ~4 minutes, and the author was unidentifiable because every commit is attributed to Vuong Hoang by convention. The parked attribution-gap memory is unparked and points here. The open design questions are noted as mine to settle, the load-bearing one being whether deploy-stack.sh and elway write to the log automatically. A log that depends on remembering is the same class of instrument as a health check that passes in both states, and this repo spent yesterday learning what those cost. |
||
|
vh
|
148a5a34da |
memory: snapshot — three silent fleet faults found and fixed in one afternoon
An infrastructure day with no training work, and the through-line is that every fault was invisible to monitoring. NH3↔Anaheim had been crossing a throttled DERP relay rather than a direct path for long enough to carry 78 GB; `.internal` DNS was failing roughly one lookup in ten from two independent causes; SearXNG had exactly one working general web engine. Nothing alarmed on any of it. All three surfaced because tts-dev had a 1545 ms voice-loop budget and chose to measure rather than adapt around the problem. Also landed: althing v3.6.3, which makes hyphenated search work for the first time on a fleet whose hostnames are nearly all hyphenated; FleetTools, a capability index autoloaded by Claude, Codex and Grok from one symlinked file; Miranda's Hermes plugin moved from a copy to a repo symlink; Worldtree's env.sh secrets vaulted. Six detail files. The in-flight section is rewritten and shrinks 142 lines to 64 — it opens on the one thing this session did NOT verify, lv-mccarthy's run outcome, which was left untouched and must not be assumed good. Two foot-guns recorded, both mine: the ESH egress experiment reverted on a diagnosis the rollback itself falsified, and `!ENV` in searxng settings, which has no constructor in that build and crash-looped the container ten times. No archival this run. 165 of 169 dated entries are under the 14-day guard and the remaining four all carry open deferred pointers, so the index stays over the soft cap at 480 lines — an over-cap file that keeps live decisions beats a scannable one that lost one. |
||
|
vh
|
5a9fad8240 |
docs(mesh): record the ana-gw port-forward that ended the NH3↔ANA relay
Every NH3→Anaheim flow had been crossing Tailscale's LA DERP relay rather than a direct path, for long enough to have carried 78 GB tx on the NH3 side alone. DERP is a throttled fallback, so this imposed both a fixed round-trip penalty and a bandwidth ceiling on LiteLLM, Beszel, task-board, vor and the Henge alike. It surfaced as a voice-loop latency report from tts-dev, not as a network alarm, because nothing monitors whether a mesh path is direct. ana-scale advertised 38.120.12.42:41641 while the Anaheim NAT mapped it to :60798 with no port-mapping protocol available, so inbound hole-punching always failed. Four additive objects on ana-gw give it a stable inbound endpoint. tailscale ping nh3-scale->ana-scale 373-522 ms via DERP -> 6 ms direct STT via the ANA gateway, 96 kB clip 1.399-1.449 s -> 0.237-0.270 s Beszel HTTP nh3-dev->ana-docker 0.94-1.29 s -> 0.014-0.016 s Documents the house template that matters for this box: it runs central-nat, so a policy dstaddr is the real internal address and not the VIP. Also records that the pre-change config was captured with `show` to a local file rather than a tftp job, since this edge has no out-of-band access and a backup is mandatory before touching it. irv-ml1 remains relayed and is called out as outstanding. |
||
|
vh
|
5b20b02fb9 |
chore(searxng): adopt the concurrent v4 work, with its dead mechanism marked
Picks up uncommitted searxng changes left by another session and makes them truthful rather than committing them as written. The stack itself verifies clean: canonical and live are byte-identical for both compose.yaml and searxng-settings.yml, the container is running with zero restarts, and live queries return 51-54 results from 5-6 engines with braveapi contributing 20 each time. compose.yaml gains SEARXNG_BRAVE_API_KEY, which NOTHING READS. It was added on the belief that settings.yml could pull it via `!ENV SEARXNG_BRAVE_API_KEY`; this build has no !ENV YAML constructor, so that attempt made the file unparseable and crash-looped the container ten times with fleet search down. The comment claiming the variable is "consumed by settings.yml" is replaced with what is actually true. The variable is kept, unused, in case upstream ever gains env interpolation — a comment that lies is worse than a variable that does nothing. The Tier A playbook is marked superseded FOR THE SETTINGS FILE ONLY, and scoped deliberately: its v4 design uploads a settings file carrying the !ENV tag, which would re-break the container, so settings deployment goes through scripts/deploy-stack.sh like every other stack. Its .env merge and up-d-not-restart steps remain useful, as do its two warnings recording real bugs it hit — a wholesale .env overwrite that clobbered SEARXNG_SECRET, and a sed that inserted literal backslash-n into compose.yaml. An unscoped "superseded" banner would have buried those; that failure mode cost an outage earlier today. Also folds in the regenerated graphify report. |
||
|
vh
|
d812bfe96d |
feat(homepage): update the talk tile to the inverted mark
talk shipped a reworked mark at v18 on operator ruling — the 1024x1024 cerulean field rect is gone, the bubble now carries #03adfb where it used to carry #2e2d30, and the three waveform bars are holes rather than filled shapes. Path data is byte-identical to the original trace; only the two fills moved. Fetched from the app and from the booth and confirmed the two sources are byte-identical before taking either. tts-dev flagged a real risk with the change: with the field gone the tile background shows THROUGH the waveform holes, so a tile close to #03adfb would swallow the bars. Checked rather than assumed. Homepage's card surface is --sea-20, oklch(0.31 0.022 262) = #2a313c, a dark desaturated navy; the bubble against it is 5.22:1, well clear of the 3:1 bar for non-text graphics. The page ground behind it is 6.73:1. Safe on this tile specifically — the earlier "reads well against the tile background" judgement was about a solid square and did not carry over on its own. Also refines the Next.js note in CLAUDE.md, which was over-broad. A NEW file in the images mount 404s until restart, but REPLACING an existing file's bytes serves immediately with no restart — measured here, the served hash matched the new file straight after rsync. It is the route table that freezes at container start, not the file contents. The previous wording would have had people bouncing Homepage for every icon tweak. |
||
|
vh
|
9219942037 |
feat(searxng): enable the keyed braveapi engine
Brave Search API key wired literally into the settings file and committed. Operator decision, 2026-09-18, made explicitly: this is a free-tier key on a rate-limited service of marginal value — "if the service is useless, so is the key" — so it does not justify the machinery that keeping it out of git would cost. The key remains in the vault at nh3-docker/searxng-brave-api-key as well. This is a scoped judgement about one low-value credential and not a change to the no-secrets-in-git rule for anything else. ⚠ It cannot be un-committed. Rotation means issuing a new key at Brave and replacing the line; never a history rewrite, since the repo is shared and other sessions commit to it. There is no supported alternative in this build. An earlier attempt used `api_key: !ENV SEARXNG_BRAVE_API_KEY`, which crash-looped the container ten times with search down fleet-wide: the settings loader has no !ENV YAML constructor, reads only SEARXNG_SETTINGS_PATH from the environment, and the entrypoint substitutes only `ultrasecretkey` at template-creation time. The variable reaches the container and is never read. Literal or nothing. Key verified against Brave's API directly before wiring, and verified in place after: three consecutive queries returned 55-63 results from six engines with braveapi contributing 20 each time, while google cse and marginalia remain quota-suspended. General web engines are now seven, up from one this morning. |
||
|
vh
|
274d3e2443 |
fix(searxng): six general web engines by default, not one
Root cause of the silent-empty-results failure peedlar-dev reported. Of 55 general-category engines, only seven were enabled-by-default, and six of those are dictionary, translation, currency or encyclopedia engines that return nothing for an ordinary web query. `google cse` was the instance's ONLY general web engine, so a single quota exhaustion produced HTTP 200 with an empty results array and no error, for every consumer on the fleet. The distinction that matters: `inactive: false` only makes an engine selectable, `disabled: false` puts it in the default set. The other 48 were selectable-but-off, which an API client has no way to change. Enables five keyless engines, each bang-probed first and returning real results with no API key: duckduckgo web 10, bing 10, yep 20, yahoo 7, wiby 12. General web engines go 1 -> 6. Deliberately excluded: mojeek, qwant, startpage and the brave scraper, all of which CAPTCHA or rate-limit this egress, and seznam, which times out. Verified under the live failure condition rather than a simulated one. google cse is still quota-suspended right now, and three consecutive queries returned 38-41 results from 4-5 engines each. The single point of failure is gone while the failing engine is still failing. Also adopts the concurrent v4 settings work from the other session — marginalia on its public key, and the captcha'd-scraper removals — plus the fix for the crash-loop that work introduced: this build has no !ENV YAML constructor, so `api_key: !ENV SEARXNG_BRAVE_API_KEY` made the file unparseable and the container restarted ten times with search down fleet-wide. That block stays commented; the vaulted Brave key is valid but has no supported path into the settings file, which is a separate decision. |
||
|
vh
|
9a428fded9 |
fix(searxng): update to 2026.9.18 — all four engines restored
searxng had been answering from google cse alone for at least a day, with brave and startpage suspended and duckduckgo returning CAPTCHA. Updating the image from 2026.9.3+a1144dda3 to 2026.9.18+c0042add3 restored all four engines immediately, and they held across 11 consecutive queries run after the change specifically to rule out a freshly-reset circuit breaker flattering the first measurement. before searxng/searxng@sha256:3602e6ddbeba037f5d800d1ed9d296a8b93c9f5b3cf9d05fa179d0e766dd59a1 after searxng/searxng@sha256:e0027a772aeeea55bf642256aae6fb3344ffa5f25ca665898c2ea821101334c4 The image stays on :latest rather than being digest-pinned. For this stack that is deliberate and now demonstrated: upstream ships engine-handler fixes as providers change their bot gating, so being current is the mitigation, and a pin would have frozen the breakage in place. The post office is pinned for the opposite reason — it is the fleet message bus and must not move under us. README corrected. It had carried two successive wrong diagnoses, both blaming egress, and now records the real cause plus the two measurements that falsified them: reverting to direct NH3 egress reproduced the failure exactly, and a live !ddg probe on a freshly restarted container also CAPTCHA'd, ruling out a stale suspension timer. Both wrong claims asserted causation from correlation without a baseline. The health-script blind spot is unchanged and still called out: scripts/searxng-health.sh reports the same passing result whether four engines answer or one. |
||
|
vh
|
ca5f0a91c0 | searxng: sync settings with live (captcha-era engine set) | ||
|
vh
|
f8ec4c3182 |
searxng: remove captcha'd scraped engines, keep API-backed set
Measured 3/3 probes: duckduckgo/startpage CAPTCHA, brave rate-ban, wikidata 403 from this egress. Mojeek tried and also 403'd. Notes on keyed-engine path to restore breadth recorded in the settings header. |
||
|
vh
|
1a35181b67 |
revert(searxng): return search egress to direct NH3
Reverts the outgoing.proxies block added in |
||
|
vh
|
6ddb453b20 |
chore(graphify): refresh the knowledge-graph report
Regenerated by the commit hook. Corpus has grown from 382 files / ~576k words at the 2026-09-01 snapshot to 623 files / ~838k words, and the graph from 3906 nodes / 4144 edges to 5468 / 5971. Deterministic tree-sitter extraction only — zero token cost, 98% EXTRACTED. |
||
|
vh
|
156e12619d |
feat(searxng): route search egress through the esh-scale SOCKS5 proxy
Committing work deployed on 2026-09-17 that had been left uncommitted, so canonical intent stops disagreeing with the running host. The deployed /opt/docker/conf/searxng/searxng-settings.yml is byte-identical to the canonical file here, verified before this commit. Search requests and their DNS now exit via socks5h://10.0.50.65:1080 on esh-scale (CT 108), an application-level proxy rather than a host-wide exit node; no route or firewall changes. microsocks runs as nobody under searxng-egress.service, binds only 10.0.50.65:1080, and bypasses SOCKS auth for source 10.100.50.40 alone — every other source must supply a password regenerated at each start and never distributed. Verified active and enabled. There is deliberately no direct-NH3 fallback: an ESH outage must fail the search rather than silently revert egress. ⚠ THE CHANGE HAS NOT ACHIEVED ITS PURPOSE AS DEPLOYED. Two independent live queries, 2026-09-18, both report brave "Suspended: too many requests", duckduckgo "CAPTCHA" and startpage "Suspended: CAPTCHA", leaving google cse as the only answering engine. Moving egress off NH3's residential address is what this change did, and CAPTCHA avoidance was the stated reason searxng sits at NH3 at all. The README anticipated the risk in its Dependency note; it has materialised. Rollback procedure is in the README and the pre-change config is kept on the host as searxng-settings.yml.pre-esh-20260917. Measured egress also drifted from the value recorded at cutover: the README notes 154.50.58.126, the proxy now exits 128.177.138.182. Expected — the README pins no public IP and calls out WAN failover — but recorded here so the number in the doc is not mistaken for current. Also retargets seat-inventory.py's default host from the mesh address 100.64.0.7 to fv-ml1's LAN address 10.251.50.54, routed by the site gateway. |
||
|
vh
|
670ac9e8a0 |
deploy(althing): pin the post office to 3.6.3
Canonical intent still named the 3.6.2 digest while nh3-docker was running 3.6.3, so the next scripts/deploy-stack.sh run against this stack would have silently rolled the fleet message bus back and taken the hyphenated-search fix with it. Caught by forseti during independent post-deploy verification. 3.6.3 is the literal-search fallback: a query containing a hyphen was parsed by FTS5 as a column filter, raised OperationalError, and search() returned [] — indistinguishable from "no results" — so every hyphenated term on this fleet silently matched nothing. nh3-docker, irv-ml1, esh-docker-vm, tts-dev and every other hyphenated name were unsearchable. Deployed digest verified against the running container before this pin: sha256:978f85533674ee248d6c6f29c54ffab0bc2cb16332c18c9fb8bfda1d566e2de4, built from git archive of tag v3.6.3 (5de41b7). The image line is the only difference between canonical and live; the two files are now identical, so a managed deploy is a no-op rather than a regression. |
||
|
vh
|
21d24c50b4 |
docs(fleettools): autoload for Codex and Grok, and a vaulted gateway key
Codex reads a global AGENTS.md from CODEX_HOME; Grok always scans ~/.grok/rules/ and loads every *.md in it regardless of name. Both were empty, so AGENT-BOOTSTRAP.md is symlinked into each rather than copied — one file, three agent families, no drift surface. The bootstrap is a pointer, not a second index: it names ~/FLEETTOOLS.md, gives the three live-inventory endpoints, and inlines only the rules that must hold even if the agent never opens anything else — attribution to Vuong Hoang, no committed secrets, the operator owns architectural calls, n=1 is not a measurement, and absence of a signal is not a safe reading of it. The shared all-agents LiteLLM key was single-copy in ~/.claude/CLAUDE.md and is now also in the vault at litellm/all-agents-shared-key, per the standing directive that durable credentials never live in one place. It stays inline in CLAUDE.md too, since every session needs it and a vault round-trip measured over two minutes. Namespace is service-scoped rather than host-prefixed because the key is fleet-wide, matching the existing att/fortigate/headscale/unifi/worldtree entries. |
||
|
vh
|
53c3e8000e |
docs: add FleetTools — an agent-family-agnostic index of fleet capability
Every agent on nh3-dev — Claude, Codex, Grok, Aider — needs the same answers: what runs here, how do I call it, what will bite me. Until now that lived in ~/.claude/CLAUDE.md, which only Claude sessions load, and it was interleaved with operator preferences that other families have no use for. Two-tier by design, matching the persistent-memory split: FLEETTOOLS.md is a 135-line index an agent reads whole, and each entry links to a detail file it opens only when it actually needs that tool. Reading the index costs about a fifth of reading the tree. Detail paths are absolute so they resolve from any working directory, since a non-Claude agent will cat the path rather than follow a markdown link. ~/FLEETTOOLS.md symlinks to the index for discovery. Rule zero is that live inventories get queried, not transcribed: Homepage /api/services, asset-engine /api/v1/services, LiteLLM /v1/models, and every FastAPI seat's /openapi.json. A copied service table would be stale within a month and this repo already has a standing rule against second copies that drift. Contents verified against the running fleet rather than copied from existing docs: binaries resolved on PATH, seven endpoints probed live, the LiteLLM roster counted at 40 models where the old note said ~30. No credentials are included; the vault and its CLI are pointed at instead. |
||
|
vh
|
d6a9d70b9e |
feat(homepage): add talk tile with its commissioned mark
talk has served on nh3-dev since 2026-09-08 with no dashboard presence. nh3-dev is not a Docker-stack host and is absent from docker.yaml, so label auto-discovery cannot reach it — this is a manual services.yaml entry in Apps, beside the Booth and WhereTF which are there for the same reason. siteMonitor points straight at the app: talk.nh3.phasefinal.com:8092 now presents the Let's Encrypt *.nh3.phasefinal.com wildcard (valid to 2026-12-06), so no cert or port special-casing is needed. Icon is copied into the images mount rather than hot-linked from the booth, which is scratch space. Document the two traps that cost time here: Homepage v2 serves nothing but custom.css/custom.js out of the config dir, and Next.js fixes its public/ route manifest at container start, so a newly added image 404s until the container is restarted. |
||
|
vh
|
d94b5a1934 |
memory: snapshot — lv-mccarthy training launched on gx10, and the next voice seat is measured rather than chosen
In-flight rewritten to the live training run (~150/1380, ETA ~00:45 PT) with the --save-total-limit finding that would otherwise have deleted the epoch-1/epoch-2 checkpoints both prior gates were decided on. Two decisions added: the next-seat ranking (Faulkner, Morrison, Chandler -- and the finding that the corpus size ranking inverts the voice ranking, with King and Christie as the two biggest non-candidates), and the romantasy register measured on the gate's own char-bigram instrument (Yarros is the cluster outlier we already shipped; Maas is the centroid and so the worst pick; Kenyon at 27 val units if the lane gets a seat). Auto-archival: 4 entries moved to archival-memory.md; 4 held back by the open-deferred guard. |
||
|
vh
|
36f2e4dbbc | memory: ravenpen.com surfaced not executed, and the althing follow-up hamr-dev is waiting on | ||
|
vh
|
d8f4844a43 | memory: planned 2026-09-18 dragonfireacoustics move to Namecheap/Cloudflare — DNS-first ordering and the wildcard-masking trap | ||
|
vh
|
8d2b5f6b2e | memory: dragonfireacoustics zone facts — wildcard at a dead IP, Google MX that a transfer would drop, no SPF/DMARC | ||
|
vh
|
33d5d32fa9 | memory: correct the dragonfirepro read — the customer LOST the domain, and dragonfireacoustics expires in six weeks unlocked at eNom | ||
|
vh
|
14b78b4586 | memory: dragonfireacoustics.com is a dead vhost on pfi-ana-webhost — sole tenant, expired cert, EOL OS, publicly exposed | ||
|
vh
|
90e71ca75e | memory: headscale split-DNS for nh3.phasefinal.com so mesh clients resolve the internal-only wildcard | ||
|
vh
|
9a1c028b9f | memory: ESH back on the Cityside static (confirmed four ways); lv-hemingway left as-is per operator | ||
|
vh
|
3ac13c5351 | memory: lv-mccarthy D4 pairs built — 3,673 train / 269 val, the largest val fixture in the line | ||
|
vh
|
370b16ca15 | memory: re-derive the shipped-corpora split-leak numbers with the committed gate | ||
|
vh
|
707fae2b2c |
perf(leak_gate): one alternation pass for the split scan — lv-hemingway went from timing out at 5 min to 35 s
Per-surface scanning is O(surfaces x copies x corpus). lv-mccarthy (108 surfaces, 36 copies) finished in 8 s; lv-hemingway (881 surfaces, 10 copies) was still running at 5 minutes and had to be killed. A gate too slow to run is not a gate. Same trick scan() already uses: build one alternation, map the matched string back to its surface by stripping separators. Regression: identical verdict and identical per-surface hit counts on the pre-fix lv-mccarthy tree (5 surfaces, 78 hits) and on the fixed one (0). Re-derived on the two shipped corpora with the committed instrument rather than a scratch probe: lv-hemingway GATE FAILED Pasionaria, Primitivo, Chicote -- 6 hits each, all 6 copies lv-bronte GATE PASSED 0 |
||
|
vh
|
328e9b1e56 | memory: snapshot — the mccarthy leak gate passed with five protagonist names in every copy, and the chain it happened on was unrecorded | ||
|
vh
|
c55966433f |
fix(lv-mccarthy): the leak gate passed with five protagonist names still in every copy
`leak_gate.py` scans `\b(Surface)\b`. Any character inserted inside a name defeats
that pattern outright, so a mangled occurrence is unrenameable by rename.py AND
unreportable by the gate. lv-mccarthy's 2026-09-17 tree passed at "0 of 75
renameable and 0 of 37 sub-threshold" while carrying 13 occurrences of Bell,
Chigurh, Moss, Toadvine and Glanton in all six copies:
B ell C higurh M oss T oadvine a small-caps drop cap kept as its own token
Toad-vine Glan-ton a print line-break hyphen kept by the extractor
Every visible occurrence HAD been renamed, which is what made the residue invisible
to a spot-read. Fixed at three levels, all three of which must stay:
build_corpus_mccarthy.py rules 4 and 5 repair the source text — 32 split initials
with a lowercase remainder, 5 hyphen-split names, each with an expected count so a
master change fails the build. Rule 4's letter class is consonants only: `I` opens
1,966 paragraphs, `A` 143 and `Y` 32 (Spanish `y`); folding any would corrupt 2,141
lines to fix 32.
leak_gate.py gains a separator-tolerant pass with its own positive and negative
controls, and it FAILS the gate. Validated against the pre-fix tree: reports all
five surfaces, exits 1. Its fragment filter is what makes it usable — a naive scan
returns 18 false positives on Hemingway (`God damn`, `I run`) against 3 real ones;
requiring one fragment to be a non-word of the corpus cleared all 18 and kept all 3.
The whole D1→D3 chain is reproduced byte-identically before and after, so the fix
is the only delta: 6 works, the entity map, the final map and all 36 copy files.
Cross-checked on the shipped corpora: lv-bronte is clean of this class, lv-hemingway
carries 3 (`Primi tivo`, `Pasionar ia`, `Chi cote`) and is live on fv-ml1.
Also in build_sft_pairs.py, both needed before lv-mccarthy's pairs:
DEFECT 4, hard-wrap reflow. Measured on the SHIPPED lv-bronte adapter, which emits
mid-sentence line breaks at 12.46 per 1k chars against 0.00 for its own base control
and 0.00 for every Hemingway arm. McCarthy is the mixed case — The Road is wrapped,
the other five works are not — so the corpus teaches the break as a coin flip. The
obvious fix (join every interior newline) corrupts 46 two-speaker exchanges whose
blank line was lost, and unmarked dialogue is the one thing this adapter exists to
learn; the rule splits on sentence-final punctuation instead and takes the cheaper
error. Self-targeting and off by default, so every shipped pair set is unchanged.
A `mccarthy` register, which names the punctuation deliberately: the eval drives the
base control arm with this same prompt, so tics left out of it are a surface trick
only the adapter can perform, and delta_cb is a character-bigram measure.
drop_leading_heading now also consumes Blood Meridian's dash-separated chapter
arguments — 131 paragraphs, 0 in every other work of all three corpora.
And a RUNBOOK, because the D1→D3 session recorded nothing and the chain had to be
recovered by rebuilding candidates and matching sha256 against the artifacts on disk.
|
||
|
vh
|
4dce0d0a43 | memory: snapshot — lv-mccarthy through D3 on gx10, SFT pairs next (a mccarthy register must be written first) | ||
|
vh
|
5ddb0472e4 |
lv-mccarthy D3 on gx10: leak gate PASSED, and the val split is now bigger than Hemingway's
~/lv-mccarthy on pfi-gx10: corpus-clean, corpus-renamed (6 copies, 1,002 records), scripts.
leak gate 0 of 75 renameable and 0 of 37 sub-threshold survive in any copy
positive control 108/108 surfaces found in the unrenamed source
negative control nonce absent from both trees
THREE McCARTHY-SPECIFIC DECISIONS, each forced by a measurement.
1. --scope corpus, NOT the default per-work map. The Border Trilogy shares characters
across books -- 9 surfaces appear in more than one work, including Parham (The
Crossing + Cities of the Plain), Grady and Cole (All the Pretty Horses + Cities of
the Plain), Socorro and Héctor. A per-work map would give John Grady a different
invented name in each novel, turning one character into two.
2. A NEW `mccarthy` rename preset rather than reusing `hemingway`. Both are
Spanish-inflected, but Hemingway's romance pool carries it_IT and fr_FR for his
Italian and French casts, and McCarthy writes neither language -- drawing from it
would drop Italian and French surnames into a Texas-Mexico border novel. en_GB goes
for the same reason. en_US + es_MX/es_ES at an even share.
3. --min-cap 5 to MATCH the entity map's floor. The first gate run FAILED with 45
survivors, and the diagnosis is the Brontë lesson exactly: entities.py admits
cap >= 5 while rename.py only renamed cap >= 8, so every entity between 5 and 7 sat
in the map, was never renamed, and was counted as a leak. Hemingway never hit it
because its map had sub_threshold_total 0.
⭐ --holdout-chapter NOW TAKES A LIST, and this is the change with the most downstream
effect. The val split is one chapter index per work, so its SIZE is set by how many
WORKS a corpus has, not how many words:
Hemingway 10 works -> 9 val units -> 36,563 words/copy -> gate DECISIVE
Brontë 4 works -> 4 val units -> 17,043 words/copy -> gate MARGINAL
McCarthy 6 works -> 6 val units -> ~18,000 would have been Brontë's end of that
Holding out chapters 7 AND 17 gives 11 units and 40,653 words per copy -- larger than
Hemingway's, at a cost of 7% of the corpus -- on a corpus 40% smaller than his. No
amount of corpus size fixes a val split that scales with work count.
THE HUMAN GENDER PASS IS NOW AN AUDITABLE FILE, not a hand edit. The honorific/window
resolver scored 21 correct / 3 held / 1 WRONG against a 26-name control; the base-rate
proximity resolver built for Hemingway scored 18/6/1 and its own guard correctly
REFUSED to write. So the incumbent stands and four entries are fixed by hand in
gender_overrides_mccarthy.json, each carrying its evidence.
⚠ All four are female and all four look male-dominated in raw pronoun counts, because
this corpus runs 29,144 male pronouns to 5,036 female -- a base rate of 85.3% male.
Carla Jean Moss at 31m/21f would be 44m/8f at that base rate, so 21 female against an
expected 8 is decisive. Same arithmetic that recovered Pilar and Brett on Hemingway.
Alfonsa was in my control set and is correctly absent from the map at 4 occurrences,
below the min-count floor -- an error in the control, not the pipeline.
apply_gender_overrides.py refuses two ways: a name absent from the map is an error
rather than a silent no-op, and overruling a gender the detector already holds needs
an explicit "correcting": true so it cannot look like filling a held entity in a diff.
|
||
|
vh
|
5aa10bf138 |
lv-mccarthy D2: entity map + stoplist, both audits green — and audit_stoplist was scanning its own rationale
Entity map at ~/mccarthy-corpus/entities.json. 123 surfaces after a 107-surface stoplist.
entities.py 27/27 controls -- 19 positive (Glanton, Toadvine, Rawlins, Blevins,
Alejandra, Chigurh, Moss, Bell, Boyd, Holden, Tobin, Magdalena,
Eduardo, Parham, Socorro, Webster, Redbo, Niño, Franklin) and 8 negative
audit_stoplist PASS -- no stoplisted surface is ever addressed as a person
audit_entity_map PASS -- positive `boy` 0.89, negative band tops out at Riddle 0.17,
all 5 remaining flags on the read-and-cleared list
⚠⚠ A DEFECT IN audit_stoplist.py ITSELF, latent for every corpus before this one. It built
its surface set from every list value in the stoplist JSON -- including `_why`, which by
convention is a LIST OF PROSE LINES. Every sentence of the rationale went into the matcher,
and the empty separator line matched the honorific pattern 139 times, printing a flag with no
surface name at the top of the report, above the one real catch. It now skips `_`-prefixed
metadata keys and empty strings.
THE ONE REAL CATCH WAS A CONTRADICTION INSIDE MY OWN FILE. `Franklin` sat in the geography
list because it is the old name for El Paso, while the same file's context note recorded
'I'm here to see Mr Franklin' -- a lawyer in All the Pretty Horses. The honorific audit found
the contradiction between the two halves of the file. Franklin is now renameable.
A SECOND SELF-INFLICTED ONE: the fragments list was a speculative A-Z, which stoplisted `I`
and `A` -- ordinary English words -- and `Sir I dont think I can do that` duly tripped the
honorific audit. It is now the four letters actually MEASURED as entities (E, H, T, K).
Stoplist what the entity map produced, not the alphabet.
Everything ambiguous was read in context before placement, and the reasoning is in the file:
Socorro is the ranch COOK in Cities of the Plain, not the New Mexico town -- renameable
Webster, Jackson, Harlan, Lamar are Glanton's men and lawmen, not places -- renameable
Niño, Keno, Redbo are HORSES, the author's inventions -- renameable, the `Inglés` precedent
Mangas, Travis, Venada, Moderno are genuinely dual-use -- renamed, the safe direction
Santa, Varas, Griffin, Eagle, Avenue, Calle, Terrell are real geography -- stoplisted
Yaqui and Gilenos are real peoples; Ford and Hashknives are a brand and a real outfit
Ed (Ed Tom Bell) and JC are short but are names, read and kept renameable
Sensitivity floor, stated because it is part of the result: the top 170 of 199 surfaces were
classified. The bottom 29 were not individually read, so a rare real-world referent may be
renamed -- the safe direction, an accepted cost, not an oversight.
|
||
|
vh
|
0fa68cb465 |
lv-mccarthy D1 fix: three small-caps defects the entity map caught, and one I nearly added
D2's entity map returned `E`, `H`, `T` and `K` as renameable entities with 17-33 capitalised
occurrences each. A bare initial is never a name -- that is the `G` class from the Hemingway
build, where `G` was about to be renamed to a surname 248 times. Reading them in context
showed the McCarthy editions set section openings in small caps and the extractor mangled
them three different ways, none of which the D1 build repaired:
1. SPLIT INITIAL `T HE HOUSE was built` -> `The house was built` 32 cases
Hemingway's restore_smallcaps only fires on TWO or more split initials in a line, so it
is structurally blind to these single ones.
2. UNMARKED RUN `THEY STOOD in the doorway` -> `They stood in the ...` 88 cases
Concentrated in Cities of the Plain (49) and The Crossing (37).
3. LOST INITIAL `HE CANDLEFLAME` -> `THE CANDLEFLAME` 1 case
Rule 1 requires a FOLLOWING all-caps word, because `A TV was playing` and `A Mexican was
changing` are an article plus a capitalised word, not a drop cap. All four such probes
verified untouched. Rule 2's `[a-z]` lookahead is what makes it safe: lowercasing every
all-caps run at a block start would eat a genuine shout or a sign, and requiring the run to
be followed immediately by a lowercase word means it is a sentence continuing. All 23
distinct first words of the 88 were checked and are real words -- HE, WHEN, THE, THEY,
QUINQUAGESIMA -- except one, which was case 3.
⚠⚠ AND A SECOND LOST-INITIAL ENTRY WAS NEARLY SHIPPED THAT WOULD HAVE CORRUPTED THE TEXT.
`HEY RODE` -> `THEY RODE` looked right from a survey of the BUILT corpus. The raw master has
`THEY RODE` intact, twice: `HEY RODE` was matching as a SUBSTRING, and the unanchored replace
produced `TTHEY RODE`, which rule 2 then lowercased to `Tthey rode`. Two things caught it --
the count assertion (expected 1, replaced 2) and then reading the master. Rule 3 is now a
block-anchored regex rather than a string replace, so a substring cannot fire it.
⚠ My first corruption check also missed it, searching for `TTHEY` when the pipeline had
already lowercased it to `Tthey`. Check the shape the pipeline actually emits, not the shape
you imagined it would.
Totals move 584,756 -> 584,716 words, 167 units unchanged. Both guards still pass: quote
marks 0.0/10k, author's own name 26 -> 0. Entity map positive control is 14/14 on real
McCarthy characters (Glanton, Toadvine, Rawlins, Blevins, Alejandra, Chigurh, Moss, Bell,
Boyd, Holden, Tobin, Magdalena, Eduardo, Parham); `T` and `E` no longer appear as entities.
|
||
|
vh
|
82aa0b6d76 |
lv-krakauer: PARKED — research is not a voice (operator, henge id 82)
Operator ruling: "he's a great writer because of his research, not because he has a strong identifiable voice." That reason is about the AUTHOR rather than the data, and it is the better of the two on the table -- the other being the unmeasurable fraction of quoted material. It also names a selection criterion this line did not have: ask whether there IS a voice worth adapting before investigating whether a clean corpus can be built. That question was never asked here. I surveyed the holdings, built the corpus, measured all fifteen containment pairs, and fixed three stripping defects the name guard caught -- all real work, none of it touching the thing that decided it. A voice adapter is worth its corpus-plus-training-plus-gate only when the target has a prose signature a reader could pick out blind. McCarthy: 0.0 quote marks per 10k against Hemingway's 838. Hemingway: spare declaratives, heavy unattributed dialogue. Brontë: periodic sentences built on semicolons and dashes. If that sentence is hard to write, the author is a park. Nothing is deleted. The corpus (126 units, 422,880 words) and the builder stay committed and re-runnable; the park entry records what exists, what was never started (D2), and what would unpark it -- a re-extraction preserving indentation and italics, which would fix the quoted- material problem but not the operator's objection. The builder's own docstring now carries a stop notice so a future session finds the reason at the artifact rather than only in memory. |
||
|
vh
|
a5ddfed81e | memory: snapshot — McCarthy and Krakauer D1 built, Krakauer's quotation scope is an open operator call | ||
|
vh
|
4be063071a |
lv-krakauer D1: 126 units, 422,880 words — and an unmeasured fraction is not his prose
The first non-fiction corpus in this line. Builds clean and should not be trained on until
an operator scope call is made; the reason is in the module docstring and the manifest.
into-the-wild 25u 67,606w caps-title [smallcaps 21][back -1,015][epi -52]
missoula 32u 115,841w chapter-word [smallcaps 8][front -858][back -2,874]
under-the-banner-of-heaven 33u 118,171w caps-title
where-men-win-glory 36u 121,262w chapter-word [smallcaps 3][front -1,548][back -6,093]
⚠⚠ THE UNRESOLVED PROBLEM IS QUOTATION, AND IT IS NOT MEASURED BECAUSE IT CANNOT BE.
Krakauer quotes constantly and at length -- McCandless's journals and letters, Tillman's
diaries, court transcripts, depositions, Mormon historical documents, and whole paragraphs
of Jack London and Wallace Stegner at the chapter heads. In print those are indented or
italic; the extraction lost both, so inside the master they are ordinary paragraphs and no
signal this builder can read separates them from his own sentences.
Only 52 words were removable -- chapter-head epigraphs whose all-caps attribution line
survived. That is 0.01% and it is NOT the answer: the method would report 0.0% for a book
made entirely of undated block quotes. The stated floor rather than the number is what a
reader needs. This is the same error as excluding The Torrents of Spring from Hemingway --
another author's style under the target's name -- distributed rather than concentrated, and
the fraction is unknown. Scope is the operator's call, exactly as fiction-only was.
THREE DEFECTS THE NAME GUARD CAUGHT, none of which the build would have reported otherwise:
1. Back matter searched only the LAST unit. Where Men Win Glory's ACKNOWLEDGMENTS sits at
94.8% and the splitter made 41 units, so the apparatus landed in unit 37 with NOTES and
BIBLIOGRAPHY after it -- all past a strip that only looked at unit 41. Into the Wild
kept its acknowledgments AND a full-page advertisement for another of his books. Now
windowed to the last 25% and cut before the split.
2. Relying on the splitter to drop front matter did not work. Units begin at the first
heading mark, and in two works the ebook's table of contents sits above the author's
note -- giving the splitter a `Chapter Thirty-Two` to start on, so unit 1 swallowed the
apparatus and its signed `Jon Krakauer , February 2015`. Now cut at that signature,
windowed to the first 10%.
3. Zero was the wrong bar. 21 survivors became 2, and both were read: `Lewis Krakauer
loved his five children deeply` is Krakauer writing about his own father in the two
autobiographical chapters of Into the Wild, and the other is a reader's letter he
quotes calling him a kook. Hemingway's own name in his corpus was always publisher
apparatus, so 0 was right there; this author writes about himself. The allowance is
pinned at 2 and every survivor is printed with context, so a master change or a strip
that stops working fails loudly instead of widening in silence.
Both strips are windowed in OPPOSITE directions from McCarthy's, which is the point worth
carrying: McCarthy's apparatus is at the end and the earliest marker wins; Krakauer's is at
both ends and the same marker words appear in his front matter at 0.0-0.6% of the file.
|
||
|
vh
|
f3bf3ca89c |
lv-mccarthy D1: 167 units, 584,756 words, and a style that looks exactly like damage
Six complete novels from the licensed Kvasir masters. Same record schema as the Brontë,
Yarros and Hemingway builders, so entities.py, rename.py, leak_gate.py and the trainers run
unchanged. Splits via the new shared split_units module.
all-the-pretty-horses 33u 99,242w paragraph-blocks [back -1,768w] [drop cap restored]
blood-meridian 23u 116,651w roman-numeral [back -354w]
cities-of-the-plain 30u 90,166w paragraph-blocks
no-country-for-old-men 13u 69,841w roman-numeral [back -463w]
the-crossing 49u 149,985w paragraph-blocks [back -30w]
the-road 19u 58,871w paragraph-blocks
THE THING THIS BUILDER PROTECTS IS A VOICE THAT READS AS A DEFECT. McCarthy uses no
quotation marks and drops the apostrophe from most contractions -- dont, aint, wont, didnt.
Measured over the built corpus: 0.0 quote marks per 10k words against Hemingway's 838, and
123 apostrophes against his 241. repair_typography.py normalises "toward what the text
does" and would put the quotes back, deleting the single most identifiable thing about the
author before training starts. This builder runs NO typography normalisation and then
ASSERTS the quote density, so a future well-meaning change fails the build instead of
quietly undoing it.
⚠ That same property will make the voice gate easy to pass for the wrong reason.
voice_distance.py is Burrows's Delta over character bigrams; an adapter that learns only
"emit no quotation marks" moves delta_cb a long way without having learned a sentence. A
punctuation-normalised secondary read needs pre-registering before this one is gated.
Exclusions, measured rather than assumed:
- two truncated catalogue rows dropped for their complete mobi siblings (Blood Meridian
epub 1,167w, The Crossing epub 222w -- both real prose, both `accepted`)
- nothing else. All 15 cross-work 8-gram containment pairs measured on the Hemingway
precedent; worst is 0.10%. Six independent works, no subsumption.
Back matter rides inside the last unit in four of six works and the marker differs every
time -- THE END, a dumped Table of Contents, a Reader's Guide, an About-the-Author, press
blurbs, a CIP page. It carried the author's own name 26 times across the raw masters. Both
guards report and gate: name 26 -> 0, quotes 0.0/10k.
⚠⚠ The back-matter strip runs BEFORE the split here, inverting the Hemingway order. Blood
Meridian and The Crossing end with a dumped table of contents made of bare roman numerals on
their own lines -- the exact shape of a chapter marker. Splitting first feeds the TOC to the
splitter as two dozen extra chapters; only the 150-word floor accidentally saves it today.
One lost drop cap is patched by name, not by heuristic: the All the Pretty Horses epub opens
`HE CANDLEFLAME` because the decorative T was an image the extractor dropped. A general
restore-the-missing-initial rule would have to guess the letter, so this is asserted against
the known string and fails loudly if the master ever changes.
The alphabet is re-derived, not inherited: 1,411 non-ASCII letters across 14 forms
(á é í ñ ó ú ü). The Border Trilogy is half set in Mexico, so the Yarros ASCII-only
conclusion does not transfer -- same finding as Hemingway, same reason.
|
||
|
vh
|
705fa3a65b |
split_units: choose a unit mode by SIZE, not by count, and fall back to paragraph blocks
McCarthy and Krakauer both need this before a corpus can be built, so it is a shared module
rather than a third copy of the Hemingway splitter.
THE INHERITED RULE IS "MOST UNITS ABOVE A FLOOR" AND IT BREAKS ON PART MARKERS. Measured:
Cities of the Plain 4 roman marks -> 4 units, median 22,312w <- the book's PARTS
The Crossing 4 roman marks -> 4 units, median 37,310w <- same
"Most units" scores 4 over the 1 that finding-nothing gives, so it wins, and the existing
guard only fires at exactly one unit. A 37,000-word "chapter" sails through and every
downstream tool accepts it. Size is now the eligibility test: a mode qualifies only if its
median unit is inside [600, 12000] AND no single unit holds half the work.
TWO THINGS A CONTROL RUN CAUGHT, BOTH NOW FIXED IN THE RULE. The first version scored
eligible modes by "median closest to target". Run over Hemingway, whose markers are known
good, it chose caps-title over the book's own chapters on True at First Light:
bare-numeral 20 units median 5,337w max 11,155 <- the real chapters
caps-title 6 units median 777w max 113,886 <- median looked BETTER
caps-title matched five stray all-caps lines, so five tiny units sat beside one holding 97%
of the book. A median cannot see that distribution; a max bound can. And caps-title is the
weakest of the four signals, which is why the tiebreak among eligible modes is now PRIORITY
(contents > chapter-word > roman > bare-numeral > caps-title), not size.
CONTROLS, both green after the fix:
positive Hemingway's ten works, markers known good -> 8/10 reproduce the shipped mode and
unit count exactly. The two differences are explained, neither is a mode error:
short-stories used `contents`, which the harness does not supply, and The Old Man
and the Sea was deliberately kept whole as CONTINUOUS.
negative 40,000 words with no blank lines -> 1 unit. It refuses to fabricate divisions
out of unstructured text rather than returning a plausible section count.
Result on the two new authors: McCarthy 167 units / 587,233 words, Krakauer 135 / 431,938,
both median ~3,200-3,500w against Hemingway's 3,128.
⚠ CORRECTION TO AN EARLIER SURVEY. I reported that all four Krakauer works carry zero
chapter markers. That was wrong and it was my regex, not the books: the survey pattern
required "Chapter" followed by a numeral, and Krakauer writes "CHAPTER ONE". Missoula and
Where Men Win Glory split on chapter-word (33 and 41 units); Into the Wild and Under the
Banner of Heaven on caps-title (28 and 33). Only McCarthy's All the Pretty Horses, Cities of
the Plain, The Crossing and The Road actually need the fallback.
The Hemingway builder is deliberately NOT repointed at this module. Its corpus is shipped and
its provenance sha is pinned by a live adapter; the one behavioural difference (The Old Man
and the Sea would section into 9 rather than stay whole) is an improvement nobody asked for
on a corpus nobody should churn.
|
||
|
vh
|
9f35c8d659 |
booth: four arms, one beat, one author-neutral prompt
Six beats through voices-base, lv-bronte, lv-yarros and lv-hemingway, all served from the same process on fv-ml1 :8027 so only the adapter varies. Operator-requested side-by-side. http://10.100.10.50:8090/b/lv-voices-four-arms/ (24h TTL; also on the link board) THE PROMPT NAMES NO AUTHOR, deliberately. Each adapter trained under a prompt naming its own, so driving all four with any one of those hands that arm a hint the others do not get and the page would be measuring the prompt rather than the voice. The shared task skeleton is kept and the author clause removed. One asymmetry is disclosed on the page: Brontë and Hemingway trained on "a SHORT PASSAGE ... may run to several paragraphs" while Yarros trained on "ONE paragraph", so the neutral prompt sits slightly off-distribution for all three rather than for one. THE CONTROL GETS A 4x LARGER TOKEN BUDGET, and publishing it any other way would have been dishonest. Measured at the gate's 320-token budget: voices-base median 26 prose words, 181-257 words of <think> planning first, and 5 of 12 cells never reach the prose at all the adapters 0 of 12 failures each, empty think block in 12 of 12, median 97-105 words The adapters learned to skip the reasoning phase; the carrier has not. Showing the starved control would conflate voice with budget discipline, so the control runs at 1200 tokens and finishes every time, median 121 words. Both numbers are on the page. Two seeds per cell behind a toggle, because one sample of a sampled process is an anecdote, and a blind-mode toggle that hides which column is which. Sampler matches the gate harness (temperature 0.9, top_p 0.95, "BEAT: " prefix). Checked before publishing rather than after: all 36 adapter generations scored for verbatim 8-gram reuse, each arm against ITS OWN corpus. Brontë 0, Yarros 0, Hemingway 2 of 12 with a longest run of 8 words, that run being "i don t know i don t know". Layout verified by rendering it, not by reading the CSS: four equal 374px columns at 1600px wide, no horizontal overflow, 24 cards, 48 panes. ⚠ nh3-dev's shared /opt/ms-playwright tops out at chromium-1234, so playwright must be pinned to 1.61.0; a bare `npm i playwright` pulls 1.63 and asks for a browser build that is not there. |
||
|
vh
|
300ecc1276 |
voices-seat: ship lv-hemingway (ckpt850), and replace the memorisation control that passed it
Live on vllm-voices (fv-ml1 GPU0 :8027) beside voices-base, lv-yarros and lv-bronte.
Healthy 190 s after recreate, four models served, GPU0 96,092 -> 96,090 MiB. The adapter
was verified byte-identical to checkpoint-850 by sha256 across both transfer hops, and the
seat was verified by generating, not by reading its config: base emits 170 words of <think>
planning and never writes the passage, lv-hemingway writes the scene.
Gate design was pre-registered before any generation existed (
|
||
|
vh
|
5e6611466c |
audit_pairs_sourcenames: --filter-out, so the detector is also the fix
An already-built pair set cannot be repaired by build_sft_pairs.py --source-entities;
that flag only works at generation time. Hemingway's and Yarros's sets both predate it.
The contamination is in the BEAT, so dropping the row removes it outright. Measured on
the Hemingway train pairs: 7,094 -> 7,024, 70 dropped, 0.99% of the training data. That
is cheaper and cleaner than regenerating 70 beats against a second generator session,
which would leave the set mixed-provenance for the sake of 1% more data.
Verified by read-back rather than by the write succeeding: re-auditing the filtered file
reports 0 of 7,024 on both columns, controls green, GATE PASS.
Two refusals rather than a best-effort write:
- a contaminated RESPONSE column aborts. That is a different fault -- pairs built
against an unrenamed corpus -- and dropping rows would hide it instead of fixing it.
- more than one --pairs input aborts, because the output is a single file and would
silently merge train and val into one.
Also cross-validated the detector against the lv-bronte pair sets on real data, where the
answer is already on the record:
pairs-full + pairs-val (post-fix) 0 of 3,858 matches the recorded "0 leaks across
3,858 pairs" exactly
pairs-full.CONTAMINATED 15 of 792 = 1.89%, Rochester x6, Jane, Brocklehurst
x2, Beck, Fairfax, Burns, Helen, Eyre -- against a
record of "13 of the first 714 beats (1.8%)" with
the same names
An independently written instrument reproducing a documented finding at the right
magnitude, on the right names, is the control that says its zeroes mean absent and not
blind.
|
||
|
vh
|
2e9b118e70 |
lv-bronte: the voice axis passes under the corrected floor rule — amended, not rewritten
lv-bronte shipped 2026-09-17 with a FAILED voice axis written into its compose comment,
its NFS README and its gate record. That verdict no longer stands, and this records the
correction in all three places without deleting what they said.
The floor rule is now pairwise (commit
|