3e7d3a3e3ddd3fbbf71a6f73d06607f428880cd1
100
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 vh
|
3e7d3a3e3d |
fix(ops-log): a sub-tool must not drop the claim wrapping its caller
deploy-stack.sh claims and releases around its own work. When the agent already held a longer claim for a multi-step operation, the deploy refreshed it, then released it on exit -- silently dropping the protection partway through the very operation it was guarding. Caught live: a 45-minute claim on nh3-docker/althing-post-office, taken to cover a build-push-deploy-verify rollout, was gone by the time the rollout finished. Nothing refused anything, and nothing said so. `ops-log claim` now exits 10 when the claim was already the caller's, and leaves the holder file untouched. deploy-stack.sh treats 10 as "not mine to release". Untouched matters as much as the exit code: a refresh would overwrite the reason and TTL the original claimant chose, so a deliberate 45-minute "3.7.0 rollout in progress" would degrade into "deploy-stack.sh <host> <stack>" and whoever got refused would read the wrong story. Verified in three states: no pre-existing claim -> deploy claims and releases; caller's own wider claim -> deploy refreshes nothing and leaves it standing with its original reason; another agent's claim -> still refused with exit 3. |
||
|
vh
|
920f37c6c2 |
chore(althing): pin the post office to 3.7.0
Canonical pin bump for the 3.7.0 rollout forseti approved and released (althing 6db955f, tag v3.7.0). Image built from that tree and pushed as sha256:df0709b3f532557aebb47d6ece486ae32e851776f732862094608d55eefcd4e7; previous pin was 3.6.3@sha256:978f8553 and remains the rollback target. Content-verified rather than tag-verified: `postbox --version` inside the built image reports 3.7.0 before it was pushed, and again inside the running container after. A tag is a claim. Also adds playbooks/nh3-extdev-althing-upgrade.yaml -- a version-parameterised upgrade for the box a git pull cannot move. The existing nh3-extdev-althing-v3.yaml is the historical v2->v3 CUTOVER and is left alone rather than mutated per release; it still carries v2-retirement steps that document what happened. Every check in the new playbook asserts what the installed entry point reports, including one that the herald restarted within the last 5 minutes rather than surviving from before -- "active" does not mean "running the code you just installed". |
||
|
vh
|
e43e2626f1 |
Add althing seat monitor-behavior test artifacts
Disposable harnesses from the wake-RFI thread (01M2WT3F): exit1-behavior-test.py covers the monitor exit-code/gate path including a sustained-failure/no-backoff assertion; hash_gate_repro.py covers the detection-time hash persistence crash-before-read boundary with consumption/unchanged/changed controls. Both fail nonzero on mismatch. Committing so seat artifact provenance rides git rather than file mtimes. |
||
|
vh
|
5fee7868b3 |
fix(playbooks): don't let the preserve step depend on its own guard
/etc/restic is 0700 root, so before elway
|
||
|
vh
|
519fca4669 | memory: snapshot — a dead alarm, two blind instruments, and an elway bug that reported success | ||
|
vh
|
e50e57b5f2 |
chore(esh): exclude VM 102 from the nightly backup job
esh-vm-workstation is a Windows 11 Parsec/RDP sandbox -- no password, no state anyone needs recovered (operator ruling, 2026-09-19). Its nightly vzdump has failed since ~2026-09-06 with "timeout waiting on systemd", and the ESH job has reported `job errors` every night as a result, so a guest nobody needs backed up was manufacturing a real job failure daily. The job uses an explicit `vmid` INCLUDE list, so exclusion means removal from that list -- there is no `exclude` directive here, which is the same distinction that makes esh-scale CT 108 invisible in any exclude list. The reason now lives in the job's own `comment`, and check-backup-freshness.sh reports 102 under "NOT BACKED UP BY POLICY" rather than going quiet about it. This is a policy decision, not a fix: the systemd-scope failure on stopped guests is NOT diagnosed, merely no longer reached on this node. A stopped guest that DOES matter would hit the same wall. The playbook carries two hard-won warnings in its header. Read the job through the API, never `grep -A<n>` on jobs.cfg -- `vmid` sorts last in the block, so adding the comment line pushed it outside the window and every condition silently inverted. And use YAML block scalars for the JSON-extraction expressions; double-quoted ones did not survive YAML unescaping plus elway's bash -c quoting. Verified: vmid 100,101,103,104,105,106,107; the other seven guests still covered; job still enabled; a rerun skips the change rather than repeating it. |
||
|
vh
|
754db4bc0b |
fix(elway): evaluate when:/creates:/changed_when: with the step's own sudo
Conditions ran unprivileged no matter what the step declared, and that fails in the dangerous direction. A root-requiring `when:` -- `pvesh get ...` exits 255 for a non-root user -- returns non-zero, elway reports the step `skipped`, and a playbook that never performed its change reports overall OK. "Skipped" is indistinguishable from working idempotency, so the run looks correct. Found the hard way on esh-pve: three consecutive runs of an exclusion playbook reported success while changing nothing. Only the verify phase caught it, by continuing to report the thing the steps claimed to have handled -- which is exactly why verify runs unconditionally. `creates:` had the same fault from the other side: a path under /root is unreadable to the login user, so `test -e` said absent and the step re-ran every time. It now correctly reports the file as already present. Sudo-less steps are unaffected: their conditions still evaluate as the login user, which is what they mean. Only a step that declares sudo: true gets privileged condition evaluation, so no existing playbook changes meaning unless it was already silently broken. |
||
|
vh
|
ba26852ec6 |
feat(backups): catch a job that runs and errors, not just one that goes stale
Snapshot age is structurally blind to a backup job that executes every night
and fails every night. Nothing new is written, so the group simply ages, and
the fault only surfaces once it crosses the 48h threshold -- days after the
first failure, with the evidence sitting in a task log nobody reads.
Two live cases, both found today and both invisible for a week by this exact
mechanism:
* esh-nas-pve CT 107 (vm-jellyfin): a backup run died around 09-06 and left
a stale `backup` lock, so every nightly since failed instantly with "CT is
locked (backup)". Age named it on ~09-12. Task status would have named it
on 09-07.
* esh-pve VM 102 (esh-vm-workstation): failing nightly since ~09-06 with
"timeout waiting on systemd". Same six-day gap.
PVE already records every task result in /var/log/pve/tasks/index. This reads
it on all four non-tenant PVE nodes and reports any vzdump in the last 36h
whose status is not OK, as its own section that sets the exit code.
It found a third case on its first run: esh-nas-pve's job had been reporting
`job errors` nightly while every guest on that node read 0-1h fresh, so no
age-based check could ever have flagged it.
Window is BACKUP_JOB_WINDOW_HOURS (default 36 -- longer than a daily cycle so
one missed run does not hide a failure). A node whose task log cannot be read
is reported, never assumed healthy.
|
||
|
vh
|
5be25be081 |
fix(backups): stop paging on guests that are deliberately not backed up
ana-scale (CT 114) is a subnet-router LXC, excluded from vzdump on 2026-09-07
after a backup lock on its ESH counterpart blackholed that entire site. The
freshness check knew nothing about that and reported it 🔴 STALE every single
morning, which is how an alarm teaches you to ignore it.
Such guests now get their own section: printed every run, never hidden, and
not counted as a fault.
The subtlety is in how coverage is computed, and the obvious implementation is
wrong twice over:
* Reading one job's `exclude` list gets ana CT 109 (ana-nas) exactly
backwards. It IS excluded from the 03:00 all-guests job AND it has its own
dedicated 22:00 job. Suppressing on the exclude list would have stopped
alarming on a guest that is genuinely backed up -- trading a noisy alarm
for a blind one.
* ESH's job uses an explicit `vmid 100..107` INCLUDE list, so esh-scale 108
is excluded by OMISSION and appears in no exclude list at all.
So coverage is a union across every enabled job on the cluster, and a guest is
"intentionally not backed up" only when none of them covers it.
If coverage cannot be read, nothing is suppressed and the gap is reported: an
unreachable PVE node means we do not know, and a backup alarm must fail loud.
The SureFire namespace is never consulted (tenant property), so its guests can
never be suppressed either.
Verified against the live fleet on all four paths: CT 114 suppressed; CT 109
NOT suppressed despite being in an exclude list; esh-vm-workstation 102, which
a job really does cover and which really is failing, still reports STALE; and
with a PVE node made unreachable, 114 returns to STALE with the gap named.
|
||
|
vh
|
e979ccb337 |
fix(backups): the freshness alarm had no wire — reconnect it and make it testable
The daily backup-freshness check has been unable to raise an alert since the
2026-08-28 althing v3 cutover. It called althing-cli, which v3 DELETED rather
than deprecated. The check itself never stopped working: it detected three
stale backups every morning and told nobody, and the only trace was a WARN
line inside a unit that was already reporting `failed` for the stale backups
themselves. Three weeks, silent.
Four changes, because swapping the binary alone would have left it dead:
* althing-cli -> postbox.
* Add ALTHING_POST_OFFICE to the systemd user unit AND to the installer that
writes it. postbox has no default address by design and a user unit
inherits nothing from the interactive shell, so the binary swap on its own
would have failed with a different message. Fixing only the live unit
would have been undone by the next installer run; the two are now verified
to agree.
* Recipient infra-ops -> infra-hermes. This runs AS infra-ops, so the old
address mailed the alarm to itself — the mirror trap named in CLAUDE.md.
Day-to-day checks are infra-hermes's half of the split; he escalates.
* Split the exit codes. 1 now means "backups stale, someone was told";
2 means "the alert path itself failed". A broken alarm is a worse fault
than the thing it watches and must not be indistinguishable from it.
Adds --test-alert: a positive control that sends a real message through the
real path on demand. The wire was cut for three weeks precisely because
nothing ever exercised it in the healthy state, and an alarm whose success
path is never run is not known to work.
Verified: positive control delivered; missing-address and unreachable-post-
office both correctly exit 2; a real run through systemd delivered the alert
and exited 1.
|
||
|
vh
|
ffe7b24935 |
feat(ops-log): attribute host changes across two agents sharing one identity
infra-ops and infra-hermes act as the same OS identity and dockerd does not
log exec per caller, so host-side changes carry no fingerprint. Git cannot
close the gap either: every commit here is attributed to Vuong Hoang by
convention, which is correct for authorship and useless for attribution.
On 2026-09-18 a second session edited the searxng stack mid-deploy, crash-
looping fleet search for ~4 minutes, and the author was unidentifiable.
scripts/ops-log records one line per host-changing action and holds a
lightweight claim so two agents do not deploy the same stack at once.
Four design questions, settled:
* Central on nh3-dev, not per-host and not the post office. Both agents
run as the same unix user there, so one file is shared with zero
provisioning. Per-host needs a writable path on ~25 heterogeneous boxes
and stores "we changed host Y" on host Y. journald looked free but shows
an unprivileged reader only their own _UID, which would have split the
log silently between the infra-ops and lkraven halves of the fleet.
* The claim is advisory and enforced in the tooling. deploy-stack.sh
refuses a foreign claim across the diff, the prompt and the apply -- the
whole review window, which is where the collision happened. Acquire is
mkdir, so it is atomic rather than probably-fine. Stale claims auto-break
and the break is recorded.
* Writers are automatic. deploy-stack.sh and elway record themselves; a log
that depends on remembering is the same class of instrument as a health
check that passes in both states.
* There is a detector. `ops-log audit` asks each host what changed on disk
and compares it to the newest log line for that stack, covering the
manual ssh-and-edit path the automatic writers structurally cannot.
ops-log being absent or broken never blocks a deploy; only a live foreign
claim does. `ops-log baseline` marks the 136 stacks that predate the
instrument so the detector starts from today rather than reporting the whole
fleet forever and training us to ignore it.
An unreachable host reports INCOMPLETE and exit 5, never clean.
|
||
|
vh
|
4d826e17e3 |
memory: infra-hermes is infra-ops' assistant, and the ops log is assigned
Operator ruling 2026-09-19, recorded in three places because each serves a different reader. CLAUDE.md gets the structural facts so a fresh session has them without reading anything else; persistent-memory gets the dated decision and the assigned work; auto-memory gets the durable working relationship. The division: infra-ops keeps improving infrastructure tooling plus the hard calls, infra-hermes takes day-to-day checks, triage and routine operations, either may perform infra ops, and infra-ops may task him downward while he escalates upward. He is explicitly NOT Miranda. The global CLAUDE.md names Miranda as the sole trusted relay of operator authority and that exception does not extend to him, so a directive he relays is information rather than authorization — reversible relayed work executes, irreversible or fleet-affecting goes to the operator. He has acknowledged it in those terms. ⚠ The two handles differ by one character in the middle of a word and the fleet's OS identity is infra-ops, so a misaddressed page still mails the sender themselves. That trap is now documented alongside the existing mirror warning rather than replacing it. Building the ops log is assigned and not started. Two agents now share one fingerprint-less OS identity: ssh infra-ops@<host> is either of us and dockerd exec is not logged per-caller. The precipitating incident is on the record — 2026-09-18, a second session edited the searxng stack mid-deploy, crash-looped fleet search for ~4 minutes, and the author was unidentifiable because every commit is attributed to Vuong Hoang by convention. The parked attribution-gap memory is unparked and points here. The open design questions are noted as mine to settle, the load-bearing one being whether deploy-stack.sh and elway write to the log automatically. A log that depends on remembering is the same class of instrument as a health check that passes in both states, and this repo spent yesterday learning what those cost. |
||
|
vh
|
148a5a34da |
memory: snapshot — three silent fleet faults found and fixed in one afternoon
An infrastructure day with no training work, and the through-line is that every fault was invisible to monitoring. NH3↔Anaheim had been crossing a throttled DERP relay rather than a direct path for long enough to carry 78 GB; `.internal` DNS was failing roughly one lookup in ten from two independent causes; SearXNG had exactly one working general web engine. Nothing alarmed on any of it. All three surfaced because tts-dev had a 1545 ms voice-loop budget and chose to measure rather than adapt around the problem. Also landed: althing v3.6.3, which makes hyphenated search work for the first time on a fleet whose hostnames are nearly all hyphenated; FleetTools, a capability index autoloaded by Claude, Codex and Grok from one symlinked file; Miranda's Hermes plugin moved from a copy to a repo symlink; Worldtree's env.sh secrets vaulted. Six detail files. The in-flight section is rewritten and shrinks 142 lines to 64 — it opens on the one thing this session did NOT verify, lv-mccarthy's run outcome, which was left untouched and must not be assumed good. Two foot-guns recorded, both mine: the ESH egress experiment reverted on a diagnosis the rollback itself falsified, and `!ENV` in searxng settings, which has no constructor in that build and crash-looped the container ten times. No archival this run. 165 of 169 dated entries are under the 14-day guard and the remaining four all carry open deferred pointers, so the index stays over the soft cap at 480 lines — an over-cap file that keeps live decisions beats a scannable one that lost one. |
||
|
vh
|
5a9fad8240 |
docs(mesh): record the ana-gw port-forward that ended the NH3↔ANA relay
Every NH3→Anaheim flow had been crossing Tailscale's LA DERP relay rather than a direct path, for long enough to have carried 78 GB tx on the NH3 side alone. DERP is a throttled fallback, so this imposed both a fixed round-trip penalty and a bandwidth ceiling on LiteLLM, Beszel, task-board, vor and the Henge alike. It surfaced as a voice-loop latency report from tts-dev, not as a network alarm, because nothing monitors whether a mesh path is direct. ana-scale advertised 38.120.12.42:41641 while the Anaheim NAT mapped it to :60798 with no port-mapping protocol available, so inbound hole-punching always failed. Four additive objects on ana-gw give it a stable inbound endpoint. tailscale ping nh3-scale->ana-scale 373-522 ms via DERP -> 6 ms direct STT via the ANA gateway, 96 kB clip 1.399-1.449 s -> 0.237-0.270 s Beszel HTTP nh3-dev->ana-docker 0.94-1.29 s -> 0.014-0.016 s Documents the house template that matters for this box: it runs central-nat, so a policy dstaddr is the real internal address and not the VIP. Also records that the pre-change config was captured with `show` to a local file rather than a tftp job, since this edge has no out-of-band access and a backup is mandatory before touching it. irv-ml1 remains relayed and is called out as outstanding. |
||
|
vh
|
5b20b02fb9 |
chore(searxng): adopt the concurrent v4 work, with its dead mechanism marked
Picks up uncommitted searxng changes left by another session and makes them truthful rather than committing them as written. The stack itself verifies clean: canonical and live are byte-identical for both compose.yaml and searxng-settings.yml, the container is running with zero restarts, and live queries return 51-54 results from 5-6 engines with braveapi contributing 20 each time. compose.yaml gains SEARXNG_BRAVE_API_KEY, which NOTHING READS. It was added on the belief that settings.yml could pull it via `!ENV SEARXNG_BRAVE_API_KEY`; this build has no !ENV YAML constructor, so that attempt made the file unparseable and crash-looped the container ten times with fleet search down. The comment claiming the variable is "consumed by settings.yml" is replaced with what is actually true. The variable is kept, unused, in case upstream ever gains env interpolation — a comment that lies is worse than a variable that does nothing. The Tier A playbook is marked superseded FOR THE SETTINGS FILE ONLY, and scoped deliberately: its v4 design uploads a settings file carrying the !ENV tag, which would re-break the container, so settings deployment goes through scripts/deploy-stack.sh like every other stack. Its .env merge and up-d-not-restart steps remain useful, as do its two warnings recording real bugs it hit — a wholesale .env overwrite that clobbered SEARXNG_SECRET, and a sed that inserted literal backslash-n into compose.yaml. An unscoped "superseded" banner would have buried those; that failure mode cost an outage earlier today. Also folds in the regenerated graphify report. |
||
|
vh
|
d812bfe96d |
feat(homepage): update the talk tile to the inverted mark
talk shipped a reworked mark at v18 on operator ruling — the 1024x1024 cerulean field rect is gone, the bubble now carries #03adfb where it used to carry #2e2d30, and the three waveform bars are holes rather than filled shapes. Path data is byte-identical to the original trace; only the two fills moved. Fetched from the app and from the booth and confirmed the two sources are byte-identical before taking either. tts-dev flagged a real risk with the change: with the field gone the tile background shows THROUGH the waveform holes, so a tile close to #03adfb would swallow the bars. Checked rather than assumed. Homepage's card surface is --sea-20, oklch(0.31 0.022 262) = #2a313c, a dark desaturated navy; the bubble against it is 5.22:1, well clear of the 3:1 bar for non-text graphics. The page ground behind it is 6.73:1. Safe on this tile specifically — the earlier "reads well against the tile background" judgement was about a solid square and did not carry over on its own. Also refines the Next.js note in CLAUDE.md, which was over-broad. A NEW file in the images mount 404s until restart, but REPLACING an existing file's bytes serves immediately with no restart — measured here, the served hash matched the new file straight after rsync. It is the route table that freezes at container start, not the file contents. The previous wording would have had people bouncing Homepage for every icon tweak. |
||
|
vh
|
9219942037 |
feat(searxng): enable the keyed braveapi engine
Brave Search API key wired literally into the settings file and committed. Operator decision, 2026-09-18, made explicitly: this is a free-tier key on a rate-limited service of marginal value — "if the service is useless, so is the key" — so it does not justify the machinery that keeping it out of git would cost. The key remains in the vault at nh3-docker/searxng-brave-api-key as well. This is a scoped judgement about one low-value credential and not a change to the no-secrets-in-git rule for anything else. ⚠ It cannot be un-committed. Rotation means issuing a new key at Brave and replacing the line; never a history rewrite, since the repo is shared and other sessions commit to it. There is no supported alternative in this build. An earlier attempt used `api_key: !ENV SEARXNG_BRAVE_API_KEY`, which crash-looped the container ten times with search down fleet-wide: the settings loader has no !ENV YAML constructor, reads only SEARXNG_SETTINGS_PATH from the environment, and the entrypoint substitutes only `ultrasecretkey` at template-creation time. The variable reaches the container and is never read. Literal or nothing. Key verified against Brave's API directly before wiring, and verified in place after: three consecutive queries returned 55-63 results from six engines with braveapi contributing 20 each time, while google cse and marginalia remain quota-suspended. General web engines are now seven, up from one this morning. |
||
|
vh
|
274d3e2443 |
fix(searxng): six general web engines by default, not one
Root cause of the silent-empty-results failure peedlar-dev reported. Of 55 general-category engines, only seven were enabled-by-default, and six of those are dictionary, translation, currency or encyclopedia engines that return nothing for an ordinary web query. `google cse` was the instance's ONLY general web engine, so a single quota exhaustion produced HTTP 200 with an empty results array and no error, for every consumer on the fleet. The distinction that matters: `inactive: false` only makes an engine selectable, `disabled: false` puts it in the default set. The other 48 were selectable-but-off, which an API client has no way to change. Enables five keyless engines, each bang-probed first and returning real results with no API key: duckduckgo web 10, bing 10, yep 20, yahoo 7, wiby 12. General web engines go 1 -> 6. Deliberately excluded: mojeek, qwant, startpage and the brave scraper, all of which CAPTCHA or rate-limit this egress, and seznam, which times out. Verified under the live failure condition rather than a simulated one. google cse is still quota-suspended right now, and three consecutive queries returned 38-41 results from 4-5 engines each. The single point of failure is gone while the failing engine is still failing. Also adopts the concurrent v4 settings work from the other session — marginalia on its public key, and the captcha'd-scraper removals — plus the fix for the crash-loop that work introduced: this build has no !ENV YAML constructor, so `api_key: !ENV SEARXNG_BRAVE_API_KEY` made the file unparseable and the container restarted ten times with search down fleet-wide. That block stays commented; the vaulted Brave key is valid but has no supported path into the settings file, which is a separate decision. |
||
|
vh
|
9a428fded9 |
fix(searxng): update to 2026.9.18 — all four engines restored
searxng had been answering from google cse alone for at least a day, with brave and startpage suspended and duckduckgo returning CAPTCHA. Updating the image from 2026.9.3+a1144dda3 to 2026.9.18+c0042add3 restored all four engines immediately, and they held across 11 consecutive queries run after the change specifically to rule out a freshly-reset circuit breaker flattering the first measurement. before searxng/searxng@sha256:3602e6ddbeba037f5d800d1ed9d296a8b93c9f5b3cf9d05fa179d0e766dd59a1 after searxng/searxng@sha256:e0027a772aeeea55bf642256aae6fb3344ffa5f25ca665898c2ea821101334c4 The image stays on :latest rather than being digest-pinned. For this stack that is deliberate and now demonstrated: upstream ships engine-handler fixes as providers change their bot gating, so being current is the mitigation, and a pin would have frozen the breakage in place. The post office is pinned for the opposite reason — it is the fleet message bus and must not move under us. README corrected. It had carried two successive wrong diagnoses, both blaming egress, and now records the real cause plus the two measurements that falsified them: reverting to direct NH3 egress reproduced the failure exactly, and a live !ddg probe on a freshly restarted container also CAPTCHA'd, ruling out a stale suspension timer. Both wrong claims asserted causation from correlation without a baseline. The health-script blind spot is unchanged and still called out: scripts/searxng-health.sh reports the same passing result whether four engines answer or one. |
||
|
vh
|
ca5f0a91c0 | searxng: sync settings with live (captcha-era engine set) | ||
|
vh
|
f8ec4c3182 |
searxng: remove captcha'd scraped engines, keep API-backed set
Measured 3/3 probes: duckduckgo/startpage CAPTCHA, brave rate-ban, wikidata 403 from this egress. Mojeek tried and also 403'd. Notes on keyed-engine path to restore breadth recorded in the settings header. |
||
|
vh
|
1a35181b67 |
revert(searxng): return search egress to direct NH3
Reverts the outgoing.proxies block added in |
||
|
vh
|
6ddb453b20 |
chore(graphify): refresh the knowledge-graph report
Regenerated by the commit hook. Corpus has grown from 382 files / ~576k words at the 2026-09-01 snapshot to 623 files / ~838k words, and the graph from 3906 nodes / 4144 edges to 5468 / 5971. Deterministic tree-sitter extraction only — zero token cost, 98% EXTRACTED. |
||
|
vh
|
156e12619d |
feat(searxng): route search egress through the esh-scale SOCKS5 proxy
Committing work deployed on 2026-09-17 that had been left uncommitted, so canonical intent stops disagreeing with the running host. The deployed /opt/docker/conf/searxng/searxng-settings.yml is byte-identical to the canonical file here, verified before this commit. Search requests and their DNS now exit via socks5h://10.0.50.65:1080 on esh-scale (CT 108), an application-level proxy rather than a host-wide exit node; no route or firewall changes. microsocks runs as nobody under searxng-egress.service, binds only 10.0.50.65:1080, and bypasses SOCKS auth for source 10.100.50.40 alone — every other source must supply a password regenerated at each start and never distributed. Verified active and enabled. There is deliberately no direct-NH3 fallback: an ESH outage must fail the search rather than silently revert egress. ⚠ THE CHANGE HAS NOT ACHIEVED ITS PURPOSE AS DEPLOYED. Two independent live queries, 2026-09-18, both report brave "Suspended: too many requests", duckduckgo "CAPTCHA" and startpage "Suspended: CAPTCHA", leaving google cse as the only answering engine. Moving egress off NH3's residential address is what this change did, and CAPTCHA avoidance was the stated reason searxng sits at NH3 at all. The README anticipated the risk in its Dependency note; it has materialised. Rollback procedure is in the README and the pre-change config is kept on the host as searxng-settings.yml.pre-esh-20260917. Measured egress also drifted from the value recorded at cutover: the README notes 154.50.58.126, the proxy now exits 128.177.138.182. Expected — the README pins no public IP and calls out WAN failover — but recorded here so the number in the doc is not mistaken for current. Also retargets seat-inventory.py's default host from the mesh address 100.64.0.7 to fv-ml1's LAN address 10.251.50.54, routed by the site gateway. |
||
|
vh
|
670ac9e8a0 |
deploy(althing): pin the post office to 3.6.3
Canonical intent still named the 3.6.2 digest while nh3-docker was running 3.6.3, so the next scripts/deploy-stack.sh run against this stack would have silently rolled the fleet message bus back and taken the hyphenated-search fix with it. Caught by forseti during independent post-deploy verification. 3.6.3 is the literal-search fallback: a query containing a hyphen was parsed by FTS5 as a column filter, raised OperationalError, and search() returned [] — indistinguishable from "no results" — so every hyphenated term on this fleet silently matched nothing. nh3-docker, irv-ml1, esh-docker-vm, tts-dev and every other hyphenated name were unsearchable. Deployed digest verified against the running container before this pin: sha256:978f85533674ee248d6c6f29c54ffab0bc2cb16332c18c9fb8bfda1d566e2de4, built from git archive of tag v3.6.3 (5de41b7). The image line is the only difference between canonical and live; the two files are now identical, so a managed deploy is a no-op rather than a regression. |
||
|
vh
|
21d24c50b4 |
docs(fleettools): autoload for Codex and Grok, and a vaulted gateway key
Codex reads a global AGENTS.md from CODEX_HOME; Grok always scans ~/.grok/rules/ and loads every *.md in it regardless of name. Both were empty, so AGENT-BOOTSTRAP.md is symlinked into each rather than copied — one file, three agent families, no drift surface. The bootstrap is a pointer, not a second index: it names ~/FLEETTOOLS.md, gives the three live-inventory endpoints, and inlines only the rules that must hold even if the agent never opens anything else — attribution to Vuong Hoang, no committed secrets, the operator owns architectural calls, n=1 is not a measurement, and absence of a signal is not a safe reading of it. The shared all-agents LiteLLM key was single-copy in ~/.claude/CLAUDE.md and is now also in the vault at litellm/all-agents-shared-key, per the standing directive that durable credentials never live in one place. It stays inline in CLAUDE.md too, since every session needs it and a vault round-trip measured over two minutes. Namespace is service-scoped rather than host-prefixed because the key is fleet-wide, matching the existing att/fortigate/headscale/unifi/worldtree entries. |
||
|
vh
|
53c3e8000e |
docs: add FleetTools — an agent-family-agnostic index of fleet capability
Every agent on nh3-dev — Claude, Codex, Grok, Aider — needs the same answers: what runs here, how do I call it, what will bite me. Until now that lived in ~/.claude/CLAUDE.md, which only Claude sessions load, and it was interleaved with operator preferences that other families have no use for. Two-tier by design, matching the persistent-memory split: FLEETTOOLS.md is a 135-line index an agent reads whole, and each entry links to a detail file it opens only when it actually needs that tool. Reading the index costs about a fifth of reading the tree. Detail paths are absolute so they resolve from any working directory, since a non-Claude agent will cat the path rather than follow a markdown link. ~/FLEETTOOLS.md symlinks to the index for discovery. Rule zero is that live inventories get queried, not transcribed: Homepage /api/services, asset-engine /api/v1/services, LiteLLM /v1/models, and every FastAPI seat's /openapi.json. A copied service table would be stale within a month and this repo already has a standing rule against second copies that drift. Contents verified against the running fleet rather than copied from existing docs: binaries resolved on PATH, seven endpoints probed live, the LiteLLM roster counted at 40 models where the old note said ~30. No credentials are included; the vault and its CLI are pointed at instead. |
||
|
vh
|
d6a9d70b9e |
feat(homepage): add talk tile with its commissioned mark
talk has served on nh3-dev since 2026-09-08 with no dashboard presence. nh3-dev is not a Docker-stack host and is absent from docker.yaml, so label auto-discovery cannot reach it — this is a manual services.yaml entry in Apps, beside the Booth and WhereTF which are there for the same reason. siteMonitor points straight at the app: talk.nh3.phasefinal.com:8092 now presents the Let's Encrypt *.nh3.phasefinal.com wildcard (valid to 2026-12-06), so no cert or port special-casing is needed. Icon is copied into the images mount rather than hot-linked from the booth, which is scratch space. Document the two traps that cost time here: Homepage v2 serves nothing but custom.css/custom.js out of the config dir, and Next.js fixes its public/ route manifest at container start, so a newly added image 404s until the container is restarted. |
||
|
vh
|
d94b5a1934 |
memory: snapshot — lv-mccarthy training launched on gx10, and the next voice seat is measured rather than chosen
In-flight rewritten to the live training run (~150/1380, ETA ~00:45 PT) with the --save-total-limit finding that would otherwise have deleted the epoch-1/epoch-2 checkpoints both prior gates were decided on. Two decisions added: the next-seat ranking (Faulkner, Morrison, Chandler -- and the finding that the corpus size ranking inverts the voice ranking, with King and Christie as the two biggest non-candidates), and the romantasy register measured on the gate's own char-bigram instrument (Yarros is the cluster outlier we already shipped; Maas is the centroid and so the worst pick; Kenyon at 27 val units if the lane gets a seat). Auto-archival: 4 entries moved to archival-memory.md; 4 held back by the open-deferred guard. |
||
|
vh
|
36f2e4dbbc | memory: ravenpen.com surfaced not executed, and the althing follow-up hamr-dev is waiting on | ||
|
vh
|
d8f4844a43 | memory: planned 2026-09-18 dragonfireacoustics move to Namecheap/Cloudflare — DNS-first ordering and the wildcard-masking trap | ||
|
vh
|
8d2b5f6b2e | memory: dragonfireacoustics zone facts — wildcard at a dead IP, Google MX that a transfer would drop, no SPF/DMARC | ||
|
vh
|
33d5d32fa9 | memory: correct the dragonfirepro read — the customer LOST the domain, and dragonfireacoustics expires in six weeks unlocked at eNom | ||
|
vh
|
14b78b4586 | memory: dragonfireacoustics.com is a dead vhost on pfi-ana-webhost — sole tenant, expired cert, EOL OS, publicly exposed | ||
|
vh
|
90e71ca75e | memory: headscale split-DNS for nh3.phasefinal.com so mesh clients resolve the internal-only wildcard | ||
|
vh
|
9a1c028b9f | memory: ESH back on the Cityside static (confirmed four ways); lv-hemingway left as-is per operator | ||
|
vh
|
3ac13c5351 | memory: lv-mccarthy D4 pairs built — 3,673 train / 269 val, the largest val fixture in the line | ||
|
vh
|
370b16ca15 | memory: re-derive the shipped-corpora split-leak numbers with the committed gate | ||
|
vh
|
707fae2b2c |
perf(leak_gate): one alternation pass for the split scan — lv-hemingway went from timing out at 5 min to 35 s
Per-surface scanning is O(surfaces x copies x corpus). lv-mccarthy (108 surfaces, 36 copies) finished in 8 s; lv-hemingway (881 surfaces, 10 copies) was still running at 5 minutes and had to be killed. A gate too slow to run is not a gate. Same trick scan() already uses: build one alternation, map the matched string back to its surface by stripping separators. Regression: identical verdict and identical per-surface hit counts on the pre-fix lv-mccarthy tree (5 surfaces, 78 hits) and on the fixed one (0). Re-derived on the two shipped corpora with the committed instrument rather than a scratch probe: lv-hemingway GATE FAILED Pasionaria, Primitivo, Chicote -- 6 hits each, all 6 copies lv-bronte GATE PASSED 0 |
||
|
vh
|
328e9b1e56 | memory: snapshot — the mccarthy leak gate passed with five protagonist names in every copy, and the chain it happened on was unrecorded | ||
|
vh
|
c55966433f |
fix(lv-mccarthy): the leak gate passed with five protagonist names still in every copy
`leak_gate.py` scans `\b(Surface)\b`. Any character inserted inside a name defeats
that pattern outright, so a mangled occurrence is unrenameable by rename.py AND
unreportable by the gate. lv-mccarthy's 2026-09-17 tree passed at "0 of 75
renameable and 0 of 37 sub-threshold" while carrying 13 occurrences of Bell,
Chigurh, Moss, Toadvine and Glanton in all six copies:
B ell C higurh M oss T oadvine a small-caps drop cap kept as its own token
Toad-vine Glan-ton a print line-break hyphen kept by the extractor
Every visible occurrence HAD been renamed, which is what made the residue invisible
to a spot-read. Fixed at three levels, all three of which must stay:
build_corpus_mccarthy.py rules 4 and 5 repair the source text — 32 split initials
with a lowercase remainder, 5 hyphen-split names, each with an expected count so a
master change fails the build. Rule 4's letter class is consonants only: `I` opens
1,966 paragraphs, `A` 143 and `Y` 32 (Spanish `y`); folding any would corrupt 2,141
lines to fix 32.
leak_gate.py gains a separator-tolerant pass with its own positive and negative
controls, and it FAILS the gate. Validated against the pre-fix tree: reports all
five surfaces, exits 1. Its fragment filter is what makes it usable — a naive scan
returns 18 false positives on Hemingway (`God damn`, `I run`) against 3 real ones;
requiring one fragment to be a non-word of the corpus cleared all 18 and kept all 3.
The whole D1→D3 chain is reproduced byte-identically before and after, so the fix
is the only delta: 6 works, the entity map, the final map and all 36 copy files.
Cross-checked on the shipped corpora: lv-bronte is clean of this class, lv-hemingway
carries 3 (`Primi tivo`, `Pasionar ia`, `Chi cote`) and is live on fv-ml1.
Also in build_sft_pairs.py, both needed before lv-mccarthy's pairs:
DEFECT 4, hard-wrap reflow. Measured on the SHIPPED lv-bronte adapter, which emits
mid-sentence line breaks at 12.46 per 1k chars against 0.00 for its own base control
and 0.00 for every Hemingway arm. McCarthy is the mixed case — The Road is wrapped,
the other five works are not — so the corpus teaches the break as a coin flip. The
obvious fix (join every interior newline) corrupts 46 two-speaker exchanges whose
blank line was lost, and unmarked dialogue is the one thing this adapter exists to
learn; the rule splits on sentence-final punctuation instead and takes the cheaper
error. Self-targeting and off by default, so every shipped pair set is unchanged.
A `mccarthy` register, which names the punctuation deliberately: the eval drives the
base control arm with this same prompt, so tics left out of it are a surface trick
only the adapter can perform, and delta_cb is a character-bigram measure.
drop_leading_heading now also consumes Blood Meridian's dash-separated chapter
arguments — 131 paragraphs, 0 in every other work of all three corpora.
And a RUNBOOK, because the D1→D3 session recorded nothing and the chain had to be
recovered by rebuilding candidates and matching sha256 against the artifacts on disk.
|
||
|
vh
|
4dce0d0a43 | memory: snapshot — lv-mccarthy through D3 on gx10, SFT pairs next (a mccarthy register must be written first) | ||
|
vh
|
5ddb0472e4 |
lv-mccarthy D3 on gx10: leak gate PASSED, and the val split is now bigger than Hemingway's
~/lv-mccarthy on pfi-gx10: corpus-clean, corpus-renamed (6 copies, 1,002 records), scripts.
leak gate 0 of 75 renameable and 0 of 37 sub-threshold survive in any copy
positive control 108/108 surfaces found in the unrenamed source
negative control nonce absent from both trees
THREE McCARTHY-SPECIFIC DECISIONS, each forced by a measurement.
1. --scope corpus, NOT the default per-work map. The Border Trilogy shares characters
across books -- 9 surfaces appear in more than one work, including Parham (The
Crossing + Cities of the Plain), Grady and Cole (All the Pretty Horses + Cities of
the Plain), Socorro and Héctor. A per-work map would give John Grady a different
invented name in each novel, turning one character into two.
2. A NEW `mccarthy` rename preset rather than reusing `hemingway`. Both are
Spanish-inflected, but Hemingway's romance pool carries it_IT and fr_FR for his
Italian and French casts, and McCarthy writes neither language -- drawing from it
would drop Italian and French surnames into a Texas-Mexico border novel. en_GB goes
for the same reason. en_US + es_MX/es_ES at an even share.
3. --min-cap 5 to MATCH the entity map's floor. The first gate run FAILED with 45
survivors, and the diagnosis is the Brontë lesson exactly: entities.py admits
cap >= 5 while rename.py only renamed cap >= 8, so every entity between 5 and 7 sat
in the map, was never renamed, and was counted as a leak. Hemingway never hit it
because its map had sub_threshold_total 0.
⭐ --holdout-chapter NOW TAKES A LIST, and this is the change with the most downstream
effect. The val split is one chapter index per work, so its SIZE is set by how many
WORKS a corpus has, not how many words:
Hemingway 10 works -> 9 val units -> 36,563 words/copy -> gate DECISIVE
Brontë 4 works -> 4 val units -> 17,043 words/copy -> gate MARGINAL
McCarthy 6 works -> 6 val units -> ~18,000 would have been Brontë's end of that
Holding out chapters 7 AND 17 gives 11 units and 40,653 words per copy -- larger than
Hemingway's, at a cost of 7% of the corpus -- on a corpus 40% smaller than his. No
amount of corpus size fixes a val split that scales with work count.
THE HUMAN GENDER PASS IS NOW AN AUDITABLE FILE, not a hand edit. The honorific/window
resolver scored 21 correct / 3 held / 1 WRONG against a 26-name control; the base-rate
proximity resolver built for Hemingway scored 18/6/1 and its own guard correctly
REFUSED to write. So the incumbent stands and four entries are fixed by hand in
gender_overrides_mccarthy.json, each carrying its evidence.
⚠ All four are female and all four look male-dominated in raw pronoun counts, because
this corpus runs 29,144 male pronouns to 5,036 female -- a base rate of 85.3% male.
Carla Jean Moss at 31m/21f would be 44m/8f at that base rate, so 21 female against an
expected 8 is decisive. Same arithmetic that recovered Pilar and Brett on Hemingway.
Alfonsa was in my control set and is correctly absent from the map at 4 occurrences,
below the min-count floor -- an error in the control, not the pipeline.
apply_gender_overrides.py refuses two ways: a name absent from the map is an error
rather than a silent no-op, and overruling a gender the detector already holds needs
an explicit "correcting": true so it cannot look like filling a held entity in a diff.
|
||
|
vh
|
5aa10bf138 |
lv-mccarthy D2: entity map + stoplist, both audits green — and audit_stoplist was scanning its own rationale
Entity map at ~/mccarthy-corpus/entities.json. 123 surfaces after a 107-surface stoplist.
entities.py 27/27 controls -- 19 positive (Glanton, Toadvine, Rawlins, Blevins,
Alejandra, Chigurh, Moss, Bell, Boyd, Holden, Tobin, Magdalena,
Eduardo, Parham, Socorro, Webster, Redbo, Niño, Franklin) and 8 negative
audit_stoplist PASS -- no stoplisted surface is ever addressed as a person
audit_entity_map PASS -- positive `boy` 0.89, negative band tops out at Riddle 0.17,
all 5 remaining flags on the read-and-cleared list
⚠⚠ A DEFECT IN audit_stoplist.py ITSELF, latent for every corpus before this one. It built
its surface set from every list value in the stoplist JSON -- including `_why`, which by
convention is a LIST OF PROSE LINES. Every sentence of the rationale went into the matcher,
and the empty separator line matched the honorific pattern 139 times, printing a flag with no
surface name at the top of the report, above the one real catch. It now skips `_`-prefixed
metadata keys and empty strings.
THE ONE REAL CATCH WAS A CONTRADICTION INSIDE MY OWN FILE. `Franklin` sat in the geography
list because it is the old name for El Paso, while the same file's context note recorded
'I'm here to see Mr Franklin' -- a lawyer in All the Pretty Horses. The honorific audit found
the contradiction between the two halves of the file. Franklin is now renameable.
A SECOND SELF-INFLICTED ONE: the fragments list was a speculative A-Z, which stoplisted `I`
and `A` -- ordinary English words -- and `Sir I dont think I can do that` duly tripped the
honorific audit. It is now the four letters actually MEASURED as entities (E, H, T, K).
Stoplist what the entity map produced, not the alphabet.
Everything ambiguous was read in context before placement, and the reasoning is in the file:
Socorro is the ranch COOK in Cities of the Plain, not the New Mexico town -- renameable
Webster, Jackson, Harlan, Lamar are Glanton's men and lawmen, not places -- renameable
Niño, Keno, Redbo are HORSES, the author's inventions -- renameable, the `Inglés` precedent
Mangas, Travis, Venada, Moderno are genuinely dual-use -- renamed, the safe direction
Santa, Varas, Griffin, Eagle, Avenue, Calle, Terrell are real geography -- stoplisted
Yaqui and Gilenos are real peoples; Ford and Hashknives are a brand and a real outfit
Ed (Ed Tom Bell) and JC are short but are names, read and kept renameable
Sensitivity floor, stated because it is part of the result: the top 170 of 199 surfaces were
classified. The bottom 29 were not individually read, so a rare real-world referent may be
renamed -- the safe direction, an accepted cost, not an oversight.
|
||
|
vh
|
0fa68cb465 |
lv-mccarthy D1 fix: three small-caps defects the entity map caught, and one I nearly added
D2's entity map returned `E`, `H`, `T` and `K` as renameable entities with 17-33 capitalised
occurrences each. A bare initial is never a name -- that is the `G` class from the Hemingway
build, where `G` was about to be renamed to a surname 248 times. Reading them in context
showed the McCarthy editions set section openings in small caps and the extractor mangled
them three different ways, none of which the D1 build repaired:
1. SPLIT INITIAL `T HE HOUSE was built` -> `The house was built` 32 cases
Hemingway's restore_smallcaps only fires on TWO or more split initials in a line, so it
is structurally blind to these single ones.
2. UNMARKED RUN `THEY STOOD in the doorway` -> `They stood in the ...` 88 cases
Concentrated in Cities of the Plain (49) and The Crossing (37).
3. LOST INITIAL `HE CANDLEFLAME` -> `THE CANDLEFLAME` 1 case
Rule 1 requires a FOLLOWING all-caps word, because `A TV was playing` and `A Mexican was
changing` are an article plus a capitalised word, not a drop cap. All four such probes
verified untouched. Rule 2's `[a-z]` lookahead is what makes it safe: lowercasing every
all-caps run at a block start would eat a genuine shout or a sign, and requiring the run to
be followed immediately by a lowercase word means it is a sentence continuing. All 23
distinct first words of the 88 were checked and are real words -- HE, WHEN, THE, THEY,
QUINQUAGESIMA -- except one, which was case 3.
⚠⚠ AND A SECOND LOST-INITIAL ENTRY WAS NEARLY SHIPPED THAT WOULD HAVE CORRUPTED THE TEXT.
`HEY RODE` -> `THEY RODE` looked right from a survey of the BUILT corpus. The raw master has
`THEY RODE` intact, twice: `HEY RODE` was matching as a SUBSTRING, and the unanchored replace
produced `TTHEY RODE`, which rule 2 then lowercased to `Tthey rode`. Two things caught it --
the count assertion (expected 1, replaced 2) and then reading the master. Rule 3 is now a
block-anchored regex rather than a string replace, so a substring cannot fire it.
⚠ My first corruption check also missed it, searching for `TTHEY` when the pipeline had
already lowercased it to `Tthey`. Check the shape the pipeline actually emits, not the shape
you imagined it would.
Totals move 584,756 -> 584,716 words, 167 units unchanged. Both guards still pass: quote
marks 0.0/10k, author's own name 26 -> 0. Entity map positive control is 14/14 on real
McCarthy characters (Glanton, Toadvine, Rawlins, Blevins, Alejandra, Chigurh, Moss, Bell,
Boyd, Holden, Tobin, Magdalena, Eduardo, Parham); `T` and `E` no longer appear as entities.
|
||
|
vh
|
82aa0b6d76 |
lv-krakauer: PARKED — research is not a voice (operator, henge id 82)
Operator ruling: "he's a great writer because of his research, not because he has a strong identifiable voice." That reason is about the AUTHOR rather than the data, and it is the better of the two on the table -- the other being the unmeasurable fraction of quoted material. It also names a selection criterion this line did not have: ask whether there IS a voice worth adapting before investigating whether a clean corpus can be built. That question was never asked here. I surveyed the holdings, built the corpus, measured all fifteen containment pairs, and fixed three stripping defects the name guard caught -- all real work, none of it touching the thing that decided it. A voice adapter is worth its corpus-plus-training-plus-gate only when the target has a prose signature a reader could pick out blind. McCarthy: 0.0 quote marks per 10k against Hemingway's 838. Hemingway: spare declaratives, heavy unattributed dialogue. Brontë: periodic sentences built on semicolons and dashes. If that sentence is hard to write, the author is a park. Nothing is deleted. The corpus (126 units, 422,880 words) and the builder stay committed and re-runnable; the park entry records what exists, what was never started (D2), and what would unpark it -- a re-extraction preserving indentation and italics, which would fix the quoted- material problem but not the operator's objection. The builder's own docstring now carries a stop notice so a future session finds the reason at the artifact rather than only in memory. |
||
|
vh
|
a5ddfed81e | memory: snapshot — McCarthy and Krakauer D1 built, Krakauer's quotation scope is an open operator call | ||
|
vh
|
4be063071a |
lv-krakauer D1: 126 units, 422,880 words — and an unmeasured fraction is not his prose
The first non-fiction corpus in this line. Builds clean and should not be trained on until
an operator scope call is made; the reason is in the module docstring and the manifest.
into-the-wild 25u 67,606w caps-title [smallcaps 21][back -1,015][epi -52]
missoula 32u 115,841w chapter-word [smallcaps 8][front -858][back -2,874]
under-the-banner-of-heaven 33u 118,171w caps-title
where-men-win-glory 36u 121,262w chapter-word [smallcaps 3][front -1,548][back -6,093]
⚠⚠ THE UNRESOLVED PROBLEM IS QUOTATION, AND IT IS NOT MEASURED BECAUSE IT CANNOT BE.
Krakauer quotes constantly and at length -- McCandless's journals and letters, Tillman's
diaries, court transcripts, depositions, Mormon historical documents, and whole paragraphs
of Jack London and Wallace Stegner at the chapter heads. In print those are indented or
italic; the extraction lost both, so inside the master they are ordinary paragraphs and no
signal this builder can read separates them from his own sentences.
Only 52 words were removable -- chapter-head epigraphs whose all-caps attribution line
survived. That is 0.01% and it is NOT the answer: the method would report 0.0% for a book
made entirely of undated block quotes. The stated floor rather than the number is what a
reader needs. This is the same error as excluding The Torrents of Spring from Hemingway --
another author's style under the target's name -- distributed rather than concentrated, and
the fraction is unknown. Scope is the operator's call, exactly as fiction-only was.
THREE DEFECTS THE NAME GUARD CAUGHT, none of which the build would have reported otherwise:
1. Back matter searched only the LAST unit. Where Men Win Glory's ACKNOWLEDGMENTS sits at
94.8% and the splitter made 41 units, so the apparatus landed in unit 37 with NOTES and
BIBLIOGRAPHY after it -- all past a strip that only looked at unit 41. Into the Wild
kept its acknowledgments AND a full-page advertisement for another of his books. Now
windowed to the last 25% and cut before the split.
2. Relying on the splitter to drop front matter did not work. Units begin at the first
heading mark, and in two works the ebook's table of contents sits above the author's
note -- giving the splitter a `Chapter Thirty-Two` to start on, so unit 1 swallowed the
apparatus and its signed `Jon Krakauer , February 2015`. Now cut at that signature,
windowed to the first 10%.
3. Zero was the wrong bar. 21 survivors became 2, and both were read: `Lewis Krakauer
loved his five children deeply` is Krakauer writing about his own father in the two
autobiographical chapters of Into the Wild, and the other is a reader's letter he
quotes calling him a kook. Hemingway's own name in his corpus was always publisher
apparatus, so 0 was right there; this author writes about himself. The allowance is
pinned at 2 and every survivor is printed with context, so a master change or a strip
that stops working fails loudly instead of widening in silence.
Both strips are windowed in OPPOSITE directions from McCarthy's, which is the point worth
carrying: McCarthy's apparatus is at the end and the earliest marker wins; Krakauer's is at
both ends and the same marker words appear in his front matter at 0.0-0.6% of the file.
|
||
|
vh
|
f3bf3ca89c |
lv-mccarthy D1: 167 units, 584,756 words, and a style that looks exactly like damage
Six complete novels from the licensed Kvasir masters. Same record schema as the Brontë,
Yarros and Hemingway builders, so entities.py, rename.py, leak_gate.py and the trainers run
unchanged. Splits via the new shared split_units module.
all-the-pretty-horses 33u 99,242w paragraph-blocks [back -1,768w] [drop cap restored]
blood-meridian 23u 116,651w roman-numeral [back -354w]
cities-of-the-plain 30u 90,166w paragraph-blocks
no-country-for-old-men 13u 69,841w roman-numeral [back -463w]
the-crossing 49u 149,985w paragraph-blocks [back -30w]
the-road 19u 58,871w paragraph-blocks
THE THING THIS BUILDER PROTECTS IS A VOICE THAT READS AS A DEFECT. McCarthy uses no
quotation marks and drops the apostrophe from most contractions -- dont, aint, wont, didnt.
Measured over the built corpus: 0.0 quote marks per 10k words against Hemingway's 838, and
123 apostrophes against his 241. repair_typography.py normalises "toward what the text
does" and would put the quotes back, deleting the single most identifiable thing about the
author before training starts. This builder runs NO typography normalisation and then
ASSERTS the quote density, so a future well-meaning change fails the build instead of
quietly undoing it.
⚠ That same property will make the voice gate easy to pass for the wrong reason.
voice_distance.py is Burrows's Delta over character bigrams; an adapter that learns only
"emit no quotation marks" moves delta_cb a long way without having learned a sentence. A
punctuation-normalised secondary read needs pre-registering before this one is gated.
Exclusions, measured rather than assumed:
- two truncated catalogue rows dropped for their complete mobi siblings (Blood Meridian
epub 1,167w, The Crossing epub 222w -- both real prose, both `accepted`)
- nothing else. All 15 cross-work 8-gram containment pairs measured on the Hemingway
precedent; worst is 0.10%. Six independent works, no subsumption.
Back matter rides inside the last unit in four of six works and the marker differs every
time -- THE END, a dumped Table of Contents, a Reader's Guide, an About-the-Author, press
blurbs, a CIP page. It carried the author's own name 26 times across the raw masters. Both
guards report and gate: name 26 -> 0, quotes 0.0/10k.
⚠⚠ The back-matter strip runs BEFORE the split here, inverting the Hemingway order. Blood
Meridian and The Crossing end with a dumped table of contents made of bare roman numerals on
their own lines -- the exact shape of a chapter marker. Splitting first feeds the TOC to the
splitter as two dozen extra chapters; only the 150-word floor accidentally saves it today.
One lost drop cap is patched by name, not by heuristic: the All the Pretty Horses epub opens
`HE CANDLEFLAME` because the decorative T was an image the extractor dropped. A general
restore-the-missing-initial rule would have to guess the letter, so this is asserted against
the known string and fails loudly if the master ever changes.
The alphabet is re-derived, not inherited: 1,411 non-ASCII letters across 14 forms
(á é í ñ ó ú ü). The Border Trilogy is half set in Mexico, so the Yarros ASCII-only
conclusion does not transfer -- same finding as Hemingway, same reason.
|
||
|
vh
|
705fa3a65b |
split_units: choose a unit mode by SIZE, not by count, and fall back to paragraph blocks
McCarthy and Krakauer both need this before a corpus can be built, so it is a shared module
rather than a third copy of the Hemingway splitter.
THE INHERITED RULE IS "MOST UNITS ABOVE A FLOOR" AND IT BREAKS ON PART MARKERS. Measured:
Cities of the Plain 4 roman marks -> 4 units, median 22,312w <- the book's PARTS
The Crossing 4 roman marks -> 4 units, median 37,310w <- same
"Most units" scores 4 over the 1 that finding-nothing gives, so it wins, and the existing
guard only fires at exactly one unit. A 37,000-word "chapter" sails through and every
downstream tool accepts it. Size is now the eligibility test: a mode qualifies only if its
median unit is inside [600, 12000] AND no single unit holds half the work.
TWO THINGS A CONTROL RUN CAUGHT, BOTH NOW FIXED IN THE RULE. The first version scored
eligible modes by "median closest to target". Run over Hemingway, whose markers are known
good, it chose caps-title over the book's own chapters on True at First Light:
bare-numeral 20 units median 5,337w max 11,155 <- the real chapters
caps-title 6 units median 777w max 113,886 <- median looked BETTER
caps-title matched five stray all-caps lines, so five tiny units sat beside one holding 97%
of the book. A median cannot see that distribution; a max bound can. And caps-title is the
weakest of the four signals, which is why the tiebreak among eligible modes is now PRIORITY
(contents > chapter-word > roman > bare-numeral > caps-title), not size.
CONTROLS, both green after the fix:
positive Hemingway's ten works, markers known good -> 8/10 reproduce the shipped mode and
unit count exactly. The two differences are explained, neither is a mode error:
short-stories used `contents`, which the harness does not supply, and The Old Man
and the Sea was deliberately kept whole as CONTINUOUS.
negative 40,000 words with no blank lines -> 1 unit. It refuses to fabricate divisions
out of unstructured text rather than returning a plausible section count.
Result on the two new authors: McCarthy 167 units / 587,233 words, Krakauer 135 / 431,938,
both median ~3,200-3,500w against Hemingway's 3,128.
⚠ CORRECTION TO AN EARLIER SURVEY. I reported that all four Krakauer works carry zero
chapter markers. That was wrong and it was my regex, not the books: the survey pattern
required "Chapter" followed by a numeral, and Krakauer writes "CHAPTER ONE". Missoula and
Where Men Win Glory split on chapter-word (33 and 41 units); Into the Wild and Under the
Banner of Heaven on caps-title (28 and 33). Only McCarthy's All the Pretty Horses, Cities of
the Plain, The Crossing and The Road actually need the fallback.
The Hemingway builder is deliberately NOT repointed at this module. Its corpus is shipped and
its provenance sha is pinned by a live adapter; the one behavioural difference (The Old Man
and the Sea would section into 9 rather than stay whole) is an improvement nobody asked for
on a corpus nobody should churn.
|
||
|
vh
|
9f35c8d659 |
booth: four arms, one beat, one author-neutral prompt
Six beats through voices-base, lv-bronte, lv-yarros and lv-hemingway, all served from the same process on fv-ml1 :8027 so only the adapter varies. Operator-requested side-by-side. http://10.100.10.50:8090/b/lv-voices-four-arms/ (24h TTL; also on the link board) THE PROMPT NAMES NO AUTHOR, deliberately. Each adapter trained under a prompt naming its own, so driving all four with any one of those hands that arm a hint the others do not get and the page would be measuring the prompt rather than the voice. The shared task skeleton is kept and the author clause removed. One asymmetry is disclosed on the page: Brontë and Hemingway trained on "a SHORT PASSAGE ... may run to several paragraphs" while Yarros trained on "ONE paragraph", so the neutral prompt sits slightly off-distribution for all three rather than for one. THE CONTROL GETS A 4x LARGER TOKEN BUDGET, and publishing it any other way would have been dishonest. Measured at the gate's 320-token budget: voices-base median 26 prose words, 181-257 words of <think> planning first, and 5 of 12 cells never reach the prose at all the adapters 0 of 12 failures each, empty think block in 12 of 12, median 97-105 words The adapters learned to skip the reasoning phase; the carrier has not. Showing the starved control would conflate voice with budget discipline, so the control runs at 1200 tokens and finishes every time, median 121 words. Both numbers are on the page. Two seeds per cell behind a toggle, because one sample of a sampled process is an anecdote, and a blind-mode toggle that hides which column is which. Sampler matches the gate harness (temperature 0.9, top_p 0.95, "BEAT: " prefix). Checked before publishing rather than after: all 36 adapter generations scored for verbatim 8-gram reuse, each arm against ITS OWN corpus. Brontë 0, Yarros 0, Hemingway 2 of 12 with a longest run of 8 words, that run being "i don t know i don t know". Layout verified by rendering it, not by reading the CSS: four equal 374px columns at 1600px wide, no horizontal overflow, 24 cards, 48 panes. ⚠ nh3-dev's shared /opt/ms-playwright tops out at chromium-1234, so playwright must be pinned to 1.61.0; a bare `npm i playwright` pulls 1.63 and asks for a browser build that is not there. |
||
|
vh
|
300ecc1276 |
voices-seat: ship lv-hemingway (ckpt850), and replace the memorisation control that passed it
Live on vllm-voices (fv-ml1 GPU0 :8027) beside voices-base, lv-yarros and lv-bronte.
Healthy 190 s after recreate, four models served, GPU0 96,092 -> 96,090 MiB. The adapter
was verified byte-identical to checkpoint-850 by sha256 across both transfer hops, and the
seat was verified by generating, not by reading its config: base emits 170 words of <think>
planning and never writes the passage, lv-hemingway writes the scene.
Gate design was pre-registered before any generation existed (
|
||
|
vh
|
5e6611466c |
audit_pairs_sourcenames: --filter-out, so the detector is also the fix
An already-built pair set cannot be repaired by build_sft_pairs.py --source-entities;
that flag only works at generation time. Hemingway's and Yarros's sets both predate it.
The contamination is in the BEAT, so dropping the row removes it outright. Measured on
the Hemingway train pairs: 7,094 -> 7,024, 70 dropped, 0.99% of the training data. That
is cheaper and cleaner than regenerating 70 beats against a second generator session,
which would leave the set mixed-provenance for the sake of 1% more data.
Verified by read-back rather than by the write succeeding: re-auditing the filtered file
reports 0 of 7,024 on both columns, controls green, GATE PASS.
Two refusals rather than a best-effort write:
- a contaminated RESPONSE column aborts. That is a different fault -- pairs built
against an unrenamed corpus -- and dropping rows would hide it instead of fixing it.
- more than one --pairs input aborts, because the output is a single file and would
silently merge train and val into one.
Also cross-validated the detector against the lv-bronte pair sets on real data, where the
answer is already on the record:
pairs-full + pairs-val (post-fix) 0 of 3,858 matches the recorded "0 leaks across
3,858 pairs" exactly
pairs-full.CONTAMINATED 15 of 792 = 1.89%, Rochester x6, Jane, Brocklehurst
x2, Beck, Fairfax, Burns, Helen, Eyre -- against a
record of "13 of the first 714 beats (1.8%)" with
the same names
An independently written instrument reproducing a documented finding at the right
magnitude, on the right names, is the control that says its zeroes mean absent and not
blind.
|
||
|
vh
|
2e9b118e70 |
lv-bronte: the voice axis passes under the corrected floor rule — amended, not rewritten
lv-bronte shipped 2026-09-17 with a FAILED voice axis written into its compose comment,
its NFS README and its gate record. That verdict no longer stands, and this records the
correction in all three places without deleting what they said.
The floor rule is now pairwise (commit
|
||
|
vh
|
dcc1abc7ea |
orientation: override the gitea NAME on the host, not each repo's remote
nh3-dev was reaching gitea over the public route from every repo on the box. brokkr-smithy-dev flagged it while pushing a new repo: brokkr-smithy, sleipnir, Galdrabok and kvasir all carried git@gitea.phasefinal.com remotes, and brokkr-smithy is pushed several times a week, so the fail2ban trigger this doc already warned about was live and recurring rather than dormant. Measured before changing anything, because the plausible explanation was a split-horizon rewrite making the public name internally correct: getent hosts gitea.phasefinal.com -> 38.120.12.44 (public, ana-srv1) grep -i gitea ~/.ssh/config -> nothing ssh -G git@gitea.phasefinal.com -> hostname gitea.phasefinal.com, port 22 No rewrite, no alias, no per-repo exception. A `Host gitea.phasefinal.com` block pointing at 10.250.50.70:222 now covers every repo on the box at once, which beats rewriting N remotes: it also catches repos nobody audited and fresh clones that copy the public URL out of a README, and nothing has to be remembered next time. Verified as a route change and not just a config edit: both paths already authenticated as `vh` with the same key, and `git ls-remote origin HEAD` succeeds over the alias in brokkr-smithy and in this repo. Backup at ~/.ssh/config.bak-20260917-020929. The alias is per-host; the doc now says to check `ssh -G` rather than assume another host inherits it. |
||
|
vh
|
051b99e063 |
audit_entity_map: the rename can damage the prose and no gate will ever say so
audit_stoplist.py finds surfaces wrongly held OUT of the entity map -- a stoplisted character is an undetectable leak. This is the mirror: surfaces wrongly held IN it. leak_gate.py only ever asks whether the author's names are GONE, never whether non-names were spared, so renaming `the Chinese` into an invented surname passes it perfectly. Found sideways on Hemingway. The pairs audit reported beats naming African, Chinese, X-ray, Republican and Cezanne as leaks -- correctly, those surfaces really were removed from the corpus. Reading why turned up the larger defect: they should never have been renameable in the first place. Measured on the Hemingway map, both controls green: positive `other` 764/1356 article-preceded = 0.56 negative 100 honorific-confirmed people, highest Inglés at 0.26, bulk 0.00-0.06 FLAGGED 130 of 946 surfaces, 1,616 instances = 0.162% of corpus words The signal is an article in front of the surface: you write `the Frenchman` and `a Martini`, never `the Rinaldi`. It is a heuristic and every hit is reported FOR READING, never auto-removed -- `the Widow` and `the Informer` are genuine Hemingway epithet-names that SHOULD be renamed, and the band's own top entry makes the point, since Inglés at 0.26 is an in-world nickname deliberately kept renameable and sits just under the bar. Initials are excluded from the negative-control band rather than admitted to it. `Mr. P.` is an initial, not a person, so letting it in lets a map defect poison the control that validates the detector -- on Hemingway `P` (0.32, every occurrence `the P. O. U. M.`) was the one surface failing a band whose next highest was 0.26. Initials take no article and are invisible to the scan anyway, so every surface of two characters or fewer is now listed unconditionally. Sixteen of them are in this map, C at 274 occurrences; the same class as the `G` that was caught by hand about to be renamed to a surname 248 times. The unresolved count that drives the exit code is computed over every flagged surface, not the --show slice. Tying a gate's verdict to a display flag is the same defect as a log filter that turns a real event into a clean zero. Also corrects a wrong claim in audit_pairs_sourcenames.py's docstring: the Hemingway rename did not HOLD 591 surfaces. Paris, Madrid and Spain survive because the stoplist keeps them out of the entity map before it is built, so the map is exactly the removed set -- 941 surfaces, 941 removed, 0 kept. Measured per run rather than assumed, because a pipeline that carried kept surfaces into the map would report every `Paris` as a leak. |
||
|
vh
|
0bb4938518 |
lv-hemingway: pre-register the v2 gate, and fix the floor rule that decided lv-bronte
The gate design is written before any generation exists, because lv-bronte's
verdict turned on a choice that was only visible after the numbers printed.
THE FLOOR RULE IS NOW PAIRWISE. lv-bronte computed the noise floor as the largest
within-arm seed spread across ALL arms present. Its ckpt475 shipped at +0.193
against a 0.251 floor set entirely by ckpt925 -- a third arm nobody was shipping,
on one outlier seed. Scored against the arm it was actually compared to, the floor
is 0.092 and the same gap clears at 2.1x. A candidate's verdict must not depend on
which other arms happened to be generated. voice_distance.py now prints both floors
and flags any disagreement, so the lv-bronte record stays comparable.
audit_pairs_sourcenames.py closes the blind spot leak_gate.py has by construction:
it reads the corpus and the renamed copies, never the generated beats, so it cannot
see a beat-writing model restoring the author's real character names. Run over the
Hemingway pairs, which predate build_sft_pairs.py --source-entities:
val 0 of 200 -- the eval fixture is clean, the gate is unconfounded
train 70 of 7,094 (0.96%) -- Santiago x16, Catherine x7, Rinaldi x3, Brett,
Harry, Jake, Pablo, Nick, Maria ...
responses 0 of 7,294 -- the lv-bronte beat-only signature exactly
A matched surface is only counted when the rename actually removed it, verified
against the renamed copies, so a beat naming a held real-world place is not a leak.
Controls run every time: 941/941 surfaces found in the unrenamed source, nonce
absent from both trees, and 6 planted canonical names detected 6/6.
voice_distance.py --author is now REQUIRED. It was hardcoded "Yarros" and printed
"reference: held-out Yarros" over Brontë's numbers into a committed artifact. A
default would have moved the silent-wrong-label failure rather than removed it. The
stale "one seed-pair per arm / corroborates Base < Instruct" footer is replaced with
what the run actually carries.
Gate design: three arms (base-unadapted, ckpt1750, ckpt850), 60 beats, 4 seeds.
ckpt850 is present because the loss curve cannot separate it from ckpt1750 -- +0.0040
against a 0.0044 median neighbour jitter, with three checkpoints inside one jitter of
the minimum. adapter/ is excluded: +0.0762 is 17.4x the jitter and is resolved without
a gate.
|
||
|
vh
|
c445ce9e93 |
memory: snapshot — lv-bronte shipped with a failed voice axis, next goal is landing lv-hemingway
In-flight rewritten for the next goal. lv-hemingway is TRAINED and nothing else has been done to it: ship candidate is checkpoint-1750 (ep 1.97, eval 2.2783), the end-of-run adapter is 0.0763 worse, and the v2 gate has not been run. Every instrument it needs was parameterised during the lv-bronte run tonight and the in-flight section names all four with their traps. New detail files: 2026-09-17-lv-bronte-gate.md shipped, voice axis failed, why anyway 2026-09-17-beat-contamination-leak.md the leak the corpus gate cannot see 2026-09-17-esh-fiber-outages.md two Cityside failures, rotation fragility Also commits the memorization_check.py parameterisation, which was left uncommitted: its hardcoded Yarros defaults would have compared a Hemingway arm against the Yarros corpus and reported a meaningless clean zero. Auto-archival: index was 415 lines pre-run, over the 300 cap. Only five entries cleared the 14-day age guard, and three of those carry open deferred pointers (fused MoE park 47, nconnect=8, AI-tab belayed) and are referenced by in-flight. A fourth — every CI job on pfi-fleet runs as root on ana-docker — is a live security property rather than settled history, so it is held back deliberately. One entry archived. The file stays over cap, which is the guard working: an over-cap file that keeps live decisions beats a scannable one that lost them. |
||
|
vh
|
61840f3131 |
voices-seat: ship lv-bronte (ckpt475) with its failed voice axis on the record
lv-bronte is live on vllm-voices (fv-ml1 GPU0 :8027) alongside voices-base and lv-yarros. The seat lists all three; container healthy; GPU0 96092 -> 96090 MiB, so the adapter cost nothing measurable. IT DID NOT PASS ITS VOICE GATE, and the artifact says so in three places — this commit, a comment in the compose file, and a README beside the adapter on NFS — because an adapter found without its provenance will otherwise be read as a pass. VOICE FAIL +0.193 delta_cb vs base, against a 0.251 measured noise floor NOT COPIED PASS 8-gram hit-rate 0.00, longest 0 - identical to the control NO DAMAGE PASS ran-on +0.15 against a 0.400 floor Shipped on three grounds, none of them that the number was nearly good enough: it is additive (a named LoRA nobody reaches without asking for it), reversible (one compose line; hot-unload measures 0.003 s), and clean on the axis that carries actual risk - verbatim regurgitation of the source, on a public-domain corpus, measured against a positive control that saturates at 160. The voice result is UNDERPOWERED rather than absent: it closed 48% of the span from base to the same-author target and beat the control on every individual seed. The cause is structural - 81 val pairs against Hemingway's 200, from a 678k-word corpus against 994k - and neither more beats nor more seeds fixes it, because the floor is a range statistic and ranges widen with n. ckpt475 over ckpt925: indistinguishable on voice (0.017 apart), but ckpt925 has a verbatim 8-gram hit where this has none, and is 2.7x less stable seed-to-seed (0.251 vs 0.092) with a degeneracy probe showing no collapse to explain it. |
||
|
vh
|
9b360e477d |
memory: lv-bronte gated — voice axis fails, ship decision open
Records the full v2 gate result and three findings that outlive the ship call: 1. The effect is UNDERPOWERED, not absent. Both candidates closed 48-52% of the achievable span to held-out Bronte and beat base on every individual seed, but the gaps sit under the measured floor. Sensitivity floor stated so the negative is falsifiable: cannot resolve better than ~0.251 delta_cb at 30 beats x 4 seeds. Cause is structural — 81 val pairs against Hemingway's 200, from a 678k-word corpus against 994k — and neither more beats nor more seeds fixes it. 2. A DEFECT IN THE v2 RULE. The floor is the largest within-arm spread across ALL arms, so adding a third noisier arm raised the bar that failed the clean one. Run as a two-arm gate the floor would have been 0.092 and the candidate would have cleared at 2.1x. Deliberately NOT exploited — choosing the floor that passes your preferred answer is the failure pre-registration exists to prevent — but the rule should state whether the floor spans the compared pair or every arm present. As written, a verdict depends on which other arms you happened to run. 3. The two-epochs-on-a-three-epoch-schedule recipe did NOT transfer. Bronte's two minima are 0.0022 apart against a 0.0046 jitter; epoch 2 buys nothing over epoch 1. The epoch-3 collapse (+0.075, ~16x jitter) is the only robust part. The outlier seed was diagnosed rather than waved away: a repeat-5gram degeneracy probe is uniform at 0.0078-0.0102 across every seed and both arms, so it is genuine delta_cb variance and the floor stands. |
||
|
vh
|
8bb7686a16 |
audit_stoplist: a stoplist entry is an assertion the leak gate cannot check
Stoplisting a surface removes it from the entity map, so rename never touches it
and the gate never scans for it. That is exactly what a stoplist is FOR when the
surface is a real-world referent — and exactly how a wrongly stoplisted CHARACTER
becomes an undetectable leak. The gate reports 0 of N surviving and is telling the
truth about the set it was given.
Found by luck on lv-bronte: a generated beat said "Mrs. Leaven", and Leaven had
been filed under scripture as the bread noun. Reading it back: "Robert Leaven,
the coachman" — Bessie's married surname in Jane Eyre.
Running the audit instead of trusting that luck caught two more:
Pierrot "Madame Pierrot: she comes from Lisle, in France" — a teacher in
The Professor, filed as the commedia dell'arte figure
Samuel "Mr. Samuel Wynne" — filed as scripture
and correctly CLEARED two:
Wellington "that Baal of a Lord Wellington" — the real Duke
Moses "the Rev. Moses Barraclough" — the documented dual-use
Signal is an honorific in front of the surface: real-world referents are not
addressed as Mr/Mrs/Miss/Madame/Lord. It is a heuristic and not a proof, which is
why every hit is REPORTED FOR READING and never auto-removed — Wellington and
Moses both trip it and both are correct. Exit 1 on anything not on --allow, so it
can gate a pipeline.
Blast radius of the three errors was 16 of 3781 train pairs and 3 of 80 val —
small, but they are the author's characters in training data, which is the one
thing this pipeline exists to prevent. Corpus rebuilt rather than dropping the
affected pairs: a corpus on disk that disagrees with its committed config is how
superseded claims get made. Gate re-passes at 0 of 368 (three more surfaces than
before, exactly the restored characters), both controls green.
|
||
|
vh
|
e9e8c40b83 |
eval harness: sample the beat fixture from held-out val, and bind the eval prompt to the trained one
Two harness defects that would each make a voice number uninterpretable. build_beat_fixture.py — the fixture is now SAMPLED from the val split rather than hand-written. The original BabyYarros fixture was five hand-written beats about a stray dog and a kitten: wrong genre, so 'He licked her clean' came back as explicit sex from a romantasy adapter, and n=5 had a noise floor of 0.800 that manufactured a +0.45 result which collapsed to +0.08 at n=120. Sampling from val makes it in-genre and held out by construction, spread across works so a naive head(30) is not one novel. Refuses outright if the pairs carry any split but val, because a fixture drawn from training data makes every downstream number a memorisation measurement wearing a voice label. gen_beats_chat_yarros.py --system-from — the SYS constant in this harness is Yarros's. Driving a Bronte or Hemingway adapter with it measures the arm under a system prompt it was never trained on and confounds the carrier change with a prompt change. Rather than duplicate the register table and rely on whoever runs it to pick the matching one, read the prompt out of the pair build's own provenance, which is the artefact that records what the adapter actually saw. |
||
|
vh
|
7964d077de |
bronte-corpus: runbook — the five deviations and what the controls caught
Records the reproducible chain and, more usefully, why it diverges from the Yarros/Hemingway pipeline in five places, each forced by a measurement rather than a preference. Includes the control post-mortem, which is worth keeping because in three of four cases the CONTROL was wrong and the detector was right — the opposite of the reflex. Adele vs Adele-with-a-grave, Hollow at a 0.235 lowercase ratio, and Grace at 0.224 were all correct refusals. Blanche, at 0.0526 against a 0.05 bar, was the one real detector miss. |
||
|
vh
|
533cc0ce81 |
build_sft_pairs: reject beats that name characters the rename removed
A leak the corpus gate structurally cannot see, found on lv-bronte. The rename strips the author's names from the prose and leak_gate.py proves they are gone — 0 of 365 surviving on Brontë, both controls green. But the beat is written by an LLM that READ THE PASSAGE, and if it recognises the book it supplies the canonical names out of its own training. The beat is the INSTRUCTION half of the pair, so training on it re-teaches exactly the inventions the rename pipeline exists to remove, and the gate never looks at it: the gate reads the corpus and the renamed copies, never the generated beats. MEASURED on the first 714 Brontë pairs, before the filter existed: 13 beats (1.8%) named source characters — Rochester x6, Jane x3, Brocklehurst x2, Beck, Fairfax, Helen, Burns, Eyre, Reed, Rivers 0 of 714 RESPONSES did. The rename was perfect; the instruction side was not. One beat read "Saoirse confirms Rochester's flaws, then agrees in English to marry him" — a renamed name and a canonical one in the same sentence, which is the mechanism in miniature. Exposure scales with how well the generator knows the book, so it is WORST for public-domain classics and mildest for recent work. That is exactly why the Yarros and Hemingway runs came up clean and Brontë did not — their clean runs are NOT evidence this cannot happen to them, and both should be rebuilt with --source-entities if they are ever regenerated. Adds a `sourcename` reject to vet() plus --source-entities, which takes the UNRENAMED entity map and refuses any beat naming a surface from it. Firing at roughly 3% of attempts on Brontë. Also adds a `bronte` register. Brontë is the far end of the same axis from Hemingway and the register has to say so, or the beat-writer produces modern summary prose the passages never match. |
||
|
vh
|
fc834a8a23 |
bronte-corpus: gate lv-bronte for real — 0 of 365 with both controls green
The Brontë corpus's "0 of 203" was a HAND COUNT made before leak_gate.py
existed. On Yarros the automated instrument read 212 surviving where a hand
count said 86, so the hand count was never evidence. This runs the real gate,
and getting it to pass required fixing four defects the hand count could not
have seen.
CORPUS DEFECTS (repair_corpus_bronte.py, both measured):
- 1,922 words of publisher back matter inside Shirley's last unit — a
T. Nelson & Sons catalogue advertising Ainsworth, Marryat, Verne, Kingsley
and Dickens, plus a Gutenberg transcriber's punctuation list. Not Brontë,
and the source of the entity CHARLES. Same structural cause as the
Hemingway run: a splitter cuts on headings, nothing follows the final one.
- 1,368 Gutenberg italic spans. Two harms: they teach the adapter to emit
underscores, and the underscore is a word character, so the gate's
word-boundary scan cannot match inside an italicised name. An entity in
italics is invisible to the gate — the same never-renamed-AND-never-
reported shape as Yarros's possessive-only Afendra.
DETECTOR GAPS (phrase_map_bronte.json):
- Blanche is 19 capitalised against ONE lowercase — ratio 0.0526, over the
0.05 bar by a single token, so a named character with 19 mentions is
dropped by a hair.
- Grace (0.224) and Hollow (0.235) are refused correctly — both are common
nouns — but Grace Poole and Hollow's Mill are Brontë's. Sampling all 21
bare capitalised Grace found 20 are the character in direct address and
exactly one is the theological noun.
- Five compounds whose every component is non-renameable survive verbatim:
Moor House, Marsh End, Vale Hall, Bigben Close, Royd Lane. The other 77
audited phrases do not, because each has a renameable component.
GENDER (pin_known_gender.py): the inherited resolver put Jane MALE across 336
occurrences. Hemingway's base-rate resolver is strictly better here (1 wrong vs
4) but still fails on Jane, and the failure is structural, not tuning — Brontë's
three narrators are first-person, so their names appear almost only in dialogue
surrounded by other characters' pronouns. Ground truth is pinned separately from
the resolver's evaluation so the two are never conflated.
Also: min-cap lowered 8 to 3, which pulled Bertha, Ferndean, Rochesters and
Creemsvort in from below the old floor; corpus-scope rename so a name below
threshold in one novel is not printed verbatim there while renamed in another.
Gate: 0 of 365 surviving, positive control 365/365, negative control clean,
phrase audit 0 of 82. Floor stated: 3 capitals per work, 5 recurrences.
|
||
|
vh
|
a5745dcf72 |
playbooks: generic per-service stack image update (pull + recreate + verify)
Adds playbooks/update-stack-image.yaml — pull the newest image for one
compose stack service and recreate it, with a verify phase that asserts
the container's image id equals what the tag now resolves to rather than
trusting a 'Up' line from docker ps.
Scoped to a single service on purpose: the recreate is 'up -d <service>',
never a bare 'up -d', which would recreate every service in the project.
Go template format strings are written bare; elway's {{ identifier }}
substitution leaves them alone, but {{end}} / {{else}} would match and
die as undefined variables, so the health read uses {{json .State.Health}}
instead of an if/else.
First use: drawio on esh-docker-vm, 28.1.2 -> 31.4.6.
|
||
|
vh
|
1ffb6d7af8 |
memory: report the /snapshot handoff defect to galdrabok, with the mechanism
Sent with both specimens. Adds the root cause, which is in SYSTEM_PROMPT rather than the model. Next steps is the only one of the three generated sections with no empty case. Watch out for is told to omit itself when there are no gotchas and Resume here is told what to say when nothing is in flight, but Next steps is told only that it is a numbered, ordered, concrete list. With nothing in flight the sole action-shaped nouns in the input are the deferred items, and the nothing-in-flight rule points the model straight at them by asking it to name the most recent open pointer. Nothing in the prompt protects modality. Invent nothing and trace every claim to the input are both satisfied - the items really are in the input - while their deferred-ness is exactly what gets dropped. The verbatim-identifier rule already establishes that some attributes of the input must survive restructuring untouched; modality is one of them and only identifiers are guarded. Proposed two prompt changes to galdrabok: an empty-case escape for Next steps, and a rule making deferred, parked, belayed and deliberately-not-done items constraints belonging in Watch out for rather than steps. Offered as a caller's diagnosis since the skill is theirs. Noted that galdrabok-dev is pull mode, so there is no herald poke and they will see it on their next check. |
||
|
vh
|
9d36c74572 |
memory: snapshot refresh — breeze settled, util does not predict residency, handoff defect
Incremental over
|
||
|
vh
|
7a33bd9f09 |
memory: breeze stays put; TTS-stack move to fv-ml1 parked at id 75
Operator ruling: leave breeze-tts on irv-ml1 and park moving it, bragi and tts-gateway to fv-ml1 until the embedder, reranker and reward seats are evacuated. Parked as move-the-tts-stack-breeze-tts-bragi-tts-gateway (id 75) with the trigger, the footprints and the migration gotchas, so it resurfaces with everything needed rather than as a bare line. Two things worth having recorded against the trigger. All three services move as a set because only breeze is GPU-resident at ~10.3 GiB and growing, while bragi and tts-gateway are CPU-only proxies - co-location with the gateway is the entire reason not to move breeze alone, since that is what puts a cross-site hop on every TTS call. And the trigger as stated names gpu0, but vllm-embed, vllm-rerank-a3 and vllm-reward are all pinned to GPU 1. GPU 1 is the constrained card at 0.975 committed with 4,336 MiB free, while GPU 0 has 11,982 MiB free and carries the live chat path, so evacuating those three relieves GPU 1 rather than GPU 0. Recorded as a confirm-before-executing rather than silently corrected, since it changes where the TTS stack would land. Also notes that bragi and tts-gateway reach each other by name only through extra_hosts pins, because containers on irv-ml1 cannot resolve nh3.internal - those pins travel with them and need re-pointing at the new host. |
||
|
vh
|
b0e7b408d9 |
memory: breeze-tts sizing and the fv-ml1 GPU 0 placement recommendation
The operator asked this mid-sweep and the answer never reached durable memory - caught only because he asked again after the snapshot. Recommendation is not to move it. Re-measured rather than reciting the earlier figure, which was right when taken and is now wrong: breeze holds 10,316 MiB after 53 minutes of uptime against 9,218 MiB shortly after warm-up. The footprint grows with use, consistent with PyTorch's caching allocator not returning memory - probably caching rather than a leak, but resident either way and counting against any neighbour. Two points is a trend, not a curve; whether it plateaus is unmeasured and stated as such. That changes the placement answer. fv-ml1 GPU 0 has 11,982 MiB free, so the margin is 1.7 GB and shrinking rather than the 2.8 GB the earlier number implied, on the card carrying the live chat serving path. The stronger objection is topology rather than VRAM: tts-gateway runs on irv-ml1 and reaches breeze on the same box, so moving breeze alone puts a cross-site hop on every TTS call against a 478 ms to-first-sample budget. Moving it properly means moving the gateway too. It is also not constrained where it sits - the 3090 still has 10 GB free. Also records the trap that nearly produced a wrong number: breeze reports nothing at idle when queried on the wrong GPU, because BREEZE_GPU_DEVICES=0 is the 3090 rather than the A6000. An idle query of the A6000 shows it absent entirely. |
||
|
vh
|
653f7fb939 |
memory: snapshot — Parakeet STT, svos_miranda live, talk v10, address sweep, secrets fix
Closes both of the previous session's named jobs and six unplanned pieces of work. Nothing is in flight and nothing is blocked. Parakeet STT live on fv-ml1 GPU 0 behind LiteLLM ext-stt and whisper-1; GPU 3 is now a documented reserve after the operator caught an 800 MiB seat parked on the one pristine 96 GB card. svos_miranda enabled and Miranda serving, with agent.disabled_toolsets deleted and staying out by operator ruling. talk v10 deployed as the STT seat's first consumer. The irv-ml1 dead-address sweep is complete at 0 of 112 Homepage cards, having turned up four live breakages on other hosts. The secrets-broker concurrency bug is fixed, and ~/.local/bin/secret is a symlink rather than a stale copy. Auto-archival fired at 433 lines but moved only one entry: three of the four candidates old enough to qualify carry open deferred-work pointers - a park id, an althing thread, and an explicit 'untracked by operator choice' - and the guard held them. The index stays over cap at 389 lines, which is the correct trade: nearly every entry is genuinely under fourteen days old. The generated handoff needed correcting in-session before it shipped. The model had turned three operator-deferred items into a to-do list and invited the next session to commit files that predate this one. Both would have read as instructions to a fresh context, which is the durable-false-warning failure this session spent the day documenting. |
||
|
vh
|
f4320ff57e |
docs(memory): the talk-deploy permission problem never existed
Vuong asked me to find and fix the harness issue blocking tts-dev from deploying talk. There was no harness issue, and no issue of any kind. /opt/docker/compose on nh3-dev is root:docker 2775, agent sessions run as lkraven, and lkraven is in the docker group. A mkdir settles it in one second and nobody ran one for nine days. There is also no tts-dev OS account, so the group request had no referent. It held together because a stale persistent-memory row supplied a plausible mechanism and the operator's routing instruction - 'give it to infra' - was read as corroboration of a capability limit. Those are different claims and only one was ever stated: a routing preference explains where work went, never whether it could have gone elsewhere. A contradicting ls -la was on screen in the same session and was dropped. I then repeated the claim to the operator as fact in a deploy report, which put a second name behind it. Then I did the same thing one layer up. Finding no OS problem and no deny rule, I inferred an auto-mode classifier refusal because the shape fit, and committed a settings.json into tts-dev's repo on that inference. Their mkdir showed the path writes with no refusal at all, so the hypothesis was wrong and the commit is reverted. I had spent the night writing up this failure class and still built a fix for a layer nobody had shown me failing. That commit also claimed a doc correction it did not contain: the edit and the commit were chained in one invocation, the edit's anchor assertion failed because the target text had already been fixed, and the commit ran regardless. Amended before reverting. Never chain an edit and its commit in one invocation. The rule worth keeping is that 'I can't do X' from any source is a hypothesis until someone runs the command and pastes the error, and that 'there is no error text, because there was no error' is a possible answer. |
||
|
vh
|
af8d6df387 |
docs(memory): name the fleet's characteristic failure mode
svos-dev observed that three instances of the same shape turned up between two agents in one night and that it is starting to look like a characteristic failure rather than a coincidence. Collecting all nine from today, because the class is more useful than any instance. The shape is a check that reads the input to a transformation and gets reported as if it read the output - or more generally, the instrument answering instead of the system, in a form shaped exactly like a real answer. What makes it expensive is not that things break but that the broken state is indistinguishable from a legitimate one, so it passes review and is found later by accident. Every one of the nine passed a check. The tell is stated so it can be recognised prospectively: whenever 'broken' and 'legitimately empty, absent or off' produce the same output, the cheap check cannot tell them apart by construction. Remedies that actually worked today: measure the output rather than the input; positive controls, since a method that has only ever passed cannot tell you it is not blind; true-negative controls, because two apparent failures in the secrets-broker test were names I had invented and would have been read as a partial fix; refuse to emit the ambiguous value, which was the real fix rather than the lock; and do not declare victory on a plausible fix, which is the only reason the session-establishment root cause was found at all. |
||
|
vh
|
0193b31aad |
fix(secrets-broker): bw is not concurrency-safe — serialise, and never return an empty secret with exit 0
Reported by svos-dev after parallelising four vault reads in SVOS's systemd wrapper. Reproduced here and it is worse than reported: four concurrent secret get calls for distinct items returned empty strings with exit code 0, zero of four succeeding against their one of four. No error, no timeout, no diagnostic. The shape is the problem, not the race. A caller treating an empty optional secret as 'not configured' degrades silently and never learns otherwise - it cost SVOS the ability to page the operator while the process logged a clean startup line. Root cause is session establishment, not item reads. Every invocation runs bw unlock, and concurrent unlocks against the shared appdata dir invalidate each other. The damage then surfaces downstream as an empty listing or an empty item body, which is why a per-call lock is useless: by the time the read runs the session it holds is already dead. So the lock wraps the whole command instead. Three changes. The command-level lock makes concurrent callers queue. cmd_get now refuses an empty value rather than printing it, since a stored secret is never legitimately zero-length. And find() no longer coerces empty stdout to '[]' - that turned a broken read into a confident 'no such secret', the same silent-wrong-answer shape one layer up. Verified: four parallel reads of four real items now return all four correctly, serialised at the honest ~17s each. A name that genuinely does not exist still fails loudly, so the guard did not simply mute the negative case. Also replaces the copy at ~/.local/bin/secret with a symlink to this file. It was a plain copy in sync by luck, and every edit here silently left the live tool behind. |
||
|
vh
|
66c860d6c1 |
fix(sweep): retire the dead 10.100.79.3 address across the fleet
Operator-directed. The wg0 lifeline retired at the 2026-09-06 headscale cutover is on no interface anywhere, so anything pointing at it gets no route at all. Homepage went from 9 dead cards to 0 of 112. The load-bearing part is that there is no single right target: it depends on who resolves it. The operator's browser and the Homepage and open-webui containers on esh-docker-vm all resolve nh3.internal, so those get the name and survive the next renumber. Containers on irv-ml1 and ana-docker cannot resolve it at all, so those get the IP. litellm on ana-docker looked like a counterexample and is not: it resolves the name only through its own extra_hosts entry, while asset-engine on the same host fails on it. Test from the container you are about to change, never from a neighbour. Before committing to the name I confirmed the Homepage container actually fetches ytvc's healthz through it in production rather than assuming resolution implies reach. On irv-ml1, 24 files swept and 14 comment-only hits left as port-allocation history. Seven running containers recreated so the labels took. Seven dormant ones carried stale labels because editing a compose file does not touch an existing container object - fixed with compose create --force-recreate, which rebuilds the container without starting it, the right tool for a deliberately dormant stack. The sweep's real find was off irv-ml1 entirely: four live values on two other hosts, silently dead for nine days and alerting nobody. Open WebUI's read-aloud TTS, asset-engine's inference host, and two skaldsong TTS URLs. Both running services were recreated and verified reaching their targets afterwards rather than merely carrying the new string. One self-inflicted outage worth recording: I recreated breeze-tts for a cosmetic label change and took ext-tts down for its ~90s CUDA-graph warm-up, returning 500. I caught it only because I had taken a baseline before touching it. A label-only edit still costs a full model reload on a GPU container. |
||
|
vh
|
8bc46e5132 |
docs(memory): record the gap tts-dev found in my served-page gate
They adopted the gate as tts-stack tools/gate_served_page.py and extended it in a place that matters: my version would have passed a broken page. A worklet lives inside a template literal, so a syntax error in it is invisible to a parse of the enclosing script - it is just a string until addModule compiles it at runtime, where it fails as a rejected promise and the page quietly falls back to buffered playback or records nothing. Silent degradation, which is harder to notice than a dead page rather than easier. They parse the worklet separately, and they positive-controlled the whole thing against two deliberately broken pages rather than assuming a gate that has only ever passed is not blind. The second control - valid enclosing script, broken worklet - is the one my version fails. The lesson on my own work is the useful part: I built a gate for the failure I had just been shown and stopped at its boundary. The class is 'code that is a string at parse time and code at run time'; an inline script is one instance and a template-literal worklet is another. I checked the instance, not the class. Also promotes the underlying rule to the index, since it was named twice tonight from two unrelated directions: a check that reads an artifact as stored cannot see a transformation that happens between storage and execution. |
||
|
vh
|
e113660b08 |
docs(memory): talk v10 deployed with a served-artifact gate
Operator-instructed via tts-dev. First consumer of the ext-stt seat stood up the same night - talk can now listen as well as speak. Relayed authorization was fine to act on because the work is reversible: one-line tag rollback, v1..v9 retained, compose and .env backed up. Checked the escape hatch existed rather than believing the message that described it. Gated properly: build, throwaway on a non-live port, four acceptance checks, tear down, then cut over in a separate invocation. Re-ran all four against production afterwards, because a gate that only ever ran against the throwaway proves the image rather than the deployment, and confirmed the two new env vars inside the running container rather than in the file. Added a fifth gate worth keeping. tts-dev's worst bug this cycle was a JS escape inside a Python string arriving transformed, closing the string and killing the entire inline script while the page still rendered and both import and node --check passed - because the file still held the backslash. So: fetch the page over HTTP, extract the inline script from the response body, and node --check that. Same instrument pointed at the other side of the transformation, and over the wire it also catches anything that mangles the body after TLS and ASGI. That is the second instance tonight of one rule: a check that reads the artifact as stored cannot see a transformation between storage and execution. provider=cuda in a log is the same error - an echo of configured intent read as a measurement of running reality. Also notes an open question for the operator: talk deploys route through infra-ops only because tts-dev's identity is not in nh3-dev's docker group. The durable fix is a group membership, not a standing relay. |
||
|
vh
|
01f0014489 |
docs(memory): SVOS/Miranda fully live; bank two restart patterns from svos-dev
svos-dev restarted :8770 at 02:17 and both roster lines printed clean. Confirmed from this side rather than taken on their word: :8770 answers 200 on the new pid, an unauthenticated Bifrost dispatch gets 401, and Hermes reports 29 toolsets with svos_miranda the sole enabled=True row. Two patterns from their restart that generalise past this service. A dry-run boot against the still-held port: start the new process while the old one still owns the socket, and it proves every check above the bind before dying on EADDRINUSE. Zero downtime, no commitment, and it turns a one-way restart into a rehearsed one. Worth doing for any service whose startup validates before binding. And a trap: SIGTERM released the port but left the process alive for 35 seconds, needing SIGKILL. The port was free that entire time, so a script waiting on port availability would have started the replacement alongside a still-running old process. Kill by PID and wait on the PID, never on the port - a freed port is not evidence of a dead process, the same way an unreachable post office is an outage rather than an empty inbox. |
||
|
vh
|
406769e64b |
docs(memory): bank the Parakeet bench result and tts-dev's storage-vs-execution lesson
The IRV seat was retired on tts-dev's numbers: it lost to the FV seat at both clip lengths and to whisper-large-v3 at 6.24s. Their length sweep fits ~58ms fixed + 56ms per audio-second with an asymptote of ~17.8x realtime, which independently reproduces our 17x on a different clip and harness, and the gateway hop measured below their harness resolution so ext-stt is the right consumer path. Two caveats recorded against our own numbers: their between-run variance is 20% because GPU 0 carries the live chat path, and our 0.50s median came off an idle GPU 3 - a best case, not a comparable. Their RTFx retraction is the durable part: published RTFx is batched throughput on datacenter hardware rather than single-stream latency, and the two differ by ~200x. Also banks the shape their acceptance gate caught, because it generalises past their repo. A JS escape inside a Python string arrives transformed, closing the string and killing the whole inline script, while the page still renders and both import and node --check pass - the file still holds the backslash. That is the same failure as reading provider=cuda out of a log: a check that reads the artifact as stored cannot see a transformation that happens between storage and execution. Both check the input to a transformation and get reported as if they checked its output. |
||
|
vh
|
868b56642c |
docs(memory): svos-dev fixed the roster check; disabled_toolsets deleted from config
svos-dev landed c9d2a96 - build_miranda_roster now returns an empty disabled list unconditionally and the startup line no longer names the key. The 28-name list is removed from ~/.hermes/config.yaml rather than left commented, since a paste-ready array behind a hash is what a future session uncomments; a short warning stands in its place. Their mechanism is better than mine and replaces it in the record. _get_platform_tools resolves platform_toolsets first and applies global suppression last, so subtracting 28 names from a one-element platform set is a no-op by resolution order - not merely 'adds no safety on top'. That holds for any future platform; the measurement only established the single case. And the endpoint already carried the answer. _handle_toolsets computes each row's enabled as membership in the per-platform set, so verified live: 29 rows with svos_miranda the only one reporting enabled=True. A check reading that field rather than counting rows was correct all along, against a config that never needed the key. |
||
|
vh
|
379fc27e7d |
feat(hermes): enable svos_miranda live; retire irv parakeet and voice-studio
Four operator rulings executed. svos_miranda is live in Hermes. Gateway restarted 02:10 (PID 3107822 -> 3901622, confirmed by observing the change). /v1/toolsets now reports 29 rows including svos_miranda, and an api_server session resolves to exactly the 8 plugin tools with the write-klass five absent. agent.disabled_toolsets stays off permanently: 'i dont want the tools disabled everywhere'. That key is a global end-of-pipeline subtraction rather than an api_server-scoped one - measured, a default session goes 46 tools to 20 - and it is unnecessary anyway, since platform_toolsets.api_server alone produces the exact 8-tool surface. The operator's own session was verified intact at 46 tools after the restart, which was the point of the ruling. The consequence is now SVOS's to absorb: it must stop verifying against the global roster before it restarts, because that roster is 29 by design and will not shrink. Two workable options went to svos-dev - verify the api_server surface instead, or relax the check to 'svos_miranda present and write-klass absent'. The second also survives any unrelated plugin landing on this host, which matters because 'stt' already appears in that endpoint's rows while resolving it logs 'Unknown toolset'. irv parakeet retired: it lost tts-dev's bench to the FV seat at both clip lengths and to whisper-large-v3 at 6.24s. Checked for consumers first - no gateway alias pointed at it, and every other reference on that host was a comment in a port-allocation register. Retirement banner on its README names the replacement. voice-studio stopped: it existed for the dots mint loop and Breeze obsoleted dots on 2026-09-06, so it was retired rather than repaired. |
||
|
vh
|
1f98a1be32 |
docs(memory): voice-studio is retired not broken; agent.disabled_toolsets is global
Two corrections and one finding from the same night. voice-studio: operator ruled the stack out of service. It existed for the dots mint/audition loop and dots was decommissioned 2026-09-06 when Breeze took the fleet seat, so its reason to exist went with it - which is also why nine days of breakage alerted nobody. No v11 rebuild. The gate one-liner was applied minutes before the retraction landed and was left in place rather than reverted, since the value it replaced was a dead address and reverting is another recreate of a stack that is going away. Container not stopped: it was already running, and 'down for now' arrived as a relayed paraphrase rather than an instruction. The two host-level facts survive the stack. Containers on irv-ml1 cannot resolve nh3.internal at all, so on that host the DNS name is the WRONG fix for a dead-IP bug - it swaps a dead address for an unresolvable one. Confirm resolution from inside the container before recommending a name. And a stale link can have more than one drift behind it: voice-studio had three stacked, two of them invisible from the host compose file. Hermes: svos_miranda is installed and enabled in config but the gateway was NOT restarted, so it is not live. agent.disabled_toolsets as specified by svos-dev is not scoped to api_server - it is a strict end-of-pipeline subtraction applied to every session on every platform. Measured: a default session goes 46 tools to 20, losing memory, file, terminal, web, browser and more. It is also unnecessary: platform_toolsets.api_server alone resolves an api_server session to exactly the 8 svos_miranda tools. The line buys only SVOS's startup check, which reads a global endpoint to verify a per-platform property. Left commented out with the measurement inline so an incidental restart cannot gut the assistant. |
||
|
vh
|
2fccaf7128 |
docs(memory): record the Parakeet bench result and a 96-place stale address on irv-ml1
tts-dev benched both endpoints against a Whisper baseline. FV wins at both clip lengths (155/391 ms vs IRV 354/1010 vs whisper-large-v3 457/690) — IRV is slower than the incumbent at 6.24 s, so the duplicate seat is now retirable on evidence rather than on tidiness. Their length sweep fits ~58 ms fixed + 56 ms per audio-second, asymptote ~17.8x realtime, independently reproducing our 17x on a different clip and a different harness. The gateway hop measured below their harness resolution, so ext-stt is the right consumer path. Two caveats recorded against our own numbers: their between-run variance is ±20% because GPU 0 carries the live chat path, and our 0.50 s median was taken on an idle GPU 3 — marked as a best case, not a comparable. Also records tts-dev's retraction, which is the durable lesson: published RTFx is batched throughput on datacenter hardware, not single-stream latency, and the two differ by ~200x. Their plan had projected 60-120 ms from it. Separately, chasing the one stale Homepage href they flagged turned up 96 occurrences of the retired wg0 lifeline 10.100.79.3 under /opt/docker on irv-ml1. Most are cosmetic, but voice-studio is genuinely broken: it is configured to reach studio-gate at that address, both are running, they sit on separate docker networks, and the address is on no interface on the host. Failing since the 2026-09-06 cutover with nothing alerting. ext-tts verified unaffected. Not fixed here — eight containers to recreate, three load-bearing, and the voice-studio repair touches app.py rather than config. Surfaced with evidence. The pattern is the third of its shape: a retired address needs a repo-wide grep by ADDRESS rather than by hostname, and container labels live in no file the sweep reads until the container is recreated. |
||
|
vh
|
caa04801f3 |
fix(parakeet): move the seat from the empty GPU 3 to GPU 0
Placed on GPU 3 first because it was the empty card. That was the wrong read:
the seat is ~800 MiB, under 1% of a 96 GB card, so the question was never "where
does it fit" but "whose headroom is cheapest to spend".
vLLM sizes its KV cache as a fraction of TOTAL VRAM, not free VRAM. A resident
tenant on an otherwise-clean card therefore does not cost its own megabytes — it
costs the profiling margin of whatever full-size seat lands there later, and
flash-next needs 93 GiB of 96. A 96 GB card at 2 MiB can still take that; the
same card at 922 MiB is one where the next big seat needs its utilization
hand-trimmed, which this repo's flash-next history shows is both thin and silent
when it goes wrong.
Committed utilization per card is the number that governs, not free bytes:
GPU 0 0.40 + 0.48 = 0.88 ~13 GB spare <- moved here
GPU 1 0.52+0.24+0.10+0.055+0.03+0.03 = 0.975 ~4.3 GB
GPU 2 0.96 ~1.8 GB
GPU 3 - kept empty as reserve
GPU 3 is back to 2 MiB / 97,247 MiB free and is now documented as a deliberate
reserve rather than a spare.
Post-move n=5 on the same clip: 0.68 / 0.54 / 0.54 / 0.52 / 0.53 s, median 0.54 s
against 0.50 s on GPU 3. The spreads overlap at this sample size and no difference
is claimed; the GPU 3 figure was taken on an idle card and is now noted as a best
case, since the seat shares GPU 0 with the hot serving path. Silence control and
the gateway round-trip both re-verified after the move.
Also records both Parakeet endpoints (FV v3 on Blackwell, IRV v2 on a 3090) and
the four confounds that make them not an A/B pair, sent to tts-dev for the bench.
|
||
|
vh
|
b9b14b5baf |
feat(parakeet): stand up Parakeet STT on fv-ml1 GPU 3 + LiteLLM ext-stt/whisper-1
Retargets the existing sherpa-onnx stack from irv-ml1 to fv-ml1's utility card and puts it behind the gateway. GPU 3 was the only card with room: 0/1/2 carry the vLLM seats at 84-95.5 GB of 96. Changes: - compose: pin GPU via `device_ids: ["3"]` (the dead on-host stub used `count: all`, which would have handed a 0.6B ASR seat all four cards); join traefik-net; port 8300; homepage href to the live FV address. - .env.example: default to the v3 int8 model (25 European languages, 464 MiB) rather than English-only v2; models to /tank/parakeet/models. - app.py: warm the recognizer at startup before uvicorn accepts traffic. The warmup is not an optimisation. ONNX Runtime's CUDA EP compiles and autotunes lazily on the FIRST DECODE, and on sm_120 that measured 45.7s cold (reproduced at 45.1s on a second container) against ~0.50s warm. A 45s first request is indistinguishable from a hang and LiteLLM's default timeout abandons it long before it returns. Decoding 1s of silence at load moves the cost inside the healthcheck's 300s start_period; first real request after restart is now 0.65s. Verification, because "provider=cuda" in the log is only an echo of the env var: ORT falls back to CPU silently and still returns correct text, so the service being up and the transcript being right establishes nothing. The discriminator is a process on GPU 3 (922 MiB), confirmed. Controls both directions — a known TTS sentence transcribes near-exactly (positive), 3s of digital silence returns empty (null). Warm throughput 0.50s median on an 8.52s clip, n=5, spread 0.47-0.65s, single-stream, one clip: a smoke measurement with its harness stated, not a benchmark. Gateway aliases `ext-stt` (engine-neutral, mirrors ext-tts) and `whisper-1` (OpenAI-compatible drop-in) registered via POST /model/new, i.e. LiteLLM's Postgres store where the ext-tts family already lives — no gateway restart, and config.yaml is consequently not a complete picture of what the gateway serves. Both verified end to end. The aliases use a raw IP deliberately: ana-docker resolves no .internal names at all (resolv.conf points at 1.1.1.1), and LiteLLM only reaches irv-ml1 through a hand-pinned extra_hosts entry. A second hosts entry would mean recreating the container and bouncing the gateway for every consumer. Also records the svos_miranda plugin validation pass and its structural findings, and notes that the irv-ml1 parakeet is still running — there are two now, and retiring the old one is the operator's call. |
||
|
vh
|
c6b6435c52 |
memory: snapshot — mesh retirement complete, next session points at svos-dev + STT
Captures the close-out of the fleet networking session: mesh membership retired for both fv-ml1 and nh3-dev, leaving six nodes that each have a job, with fv-ml1 carrying a break-glass rejoin instead of standing membership and exactly one live reusable pre-auth key left fleet-wide. In-flight rewritten to lead with the two jobs the operator named for the next session -- drain the svos-dev message that has been unread since 00:52, then stand up an STT service from nothing -- so a fresh context opens on the work rather than on the history. |
||
|
vh
|
a65cdf65d9 |
feat(mesh): retire nh3-dev from the mesh and revert the masquerade it required
nh3-dev sits on the NH3 LAN and reaches every site through its own default gateway; RouteAll was already false, so it never used the tunnel for routing. Membership bought a 100.64.0.4 address nothing referenced -- grep across the repo and ~/development found only docs and memory hits. It also cost something concrete. A host running Tailscale installs -A ts-input -s 100.64.0.0/10 ! -i tailscale0 -j DROP, and because the fleet's subnet routers preserve source rather than masquerading RFC1918, a mesh client's packet reached nh3-dev's ens18 still sourced 100.64.x and was dropped silently. That is why nh3-dev.nh3.internal failed from the mesh while every NH3 host that does not run Tailscale worked, and it needed a -d 10.100.10.50/32 -j MASQUERADE exception on nh3-scale to paper over. Retiring the membership removed the anti-spoof rule, so the exception went with it -- mesh-exit-masq.sh is back to the two rules it had before yesterday. Verified after: nh3-dev reachable at 10.100.10.50 from ESH, Anaheim, FV, Irvine and NH3, and reaching all four sites plus the internet itself. fv-ml1 unaffected. The mesh is now six nodes and every one has a job: three site routers, vb-gateway, irv-ml1 (Irvine's own router, no separate scale node), and the operator's MacBook Air. Nothing is enrolled just in case. |
||
|
vh
|
959a743256 |
feat(fv): invert the watchdog to break-glass; fv-ml1 off the mesh
Operator's design, and a better one. Once the FV SNAT rules landed, fv-ml1's mesh membership was redundant for routing and its only remaining value was as a second way in. Keeping it enrolled bought a standing second door; joining on demand buys the same recovery path without one. normal tailscaled stopped + disabled; fleet reached via the gateway SNAT fault nh3-dev / nh3-docker unreachable while the WAN is up action start tailscaled + tailscale up -> reachable at its 100.64.x address fv-ml1 is now off the mesh and its node record deleted. Verified it still reaches NH3, ESH, Anaheim, Irvine and the internet on the SNAT path alone, then the break-glass fired on cue (counted 1..4, joined at 5 as 100.64.0.10), answered ping and ssh from nh3-dev, and was closed again cleanly. No auto-leave, deliberately: once open the door stays open until a human runs systemctl disable --now tailscaled. A watchdog that re-closes on recovery flaps, and a flapping recovery path is down exactly when someone finally looks. It also skips entirely when already on the mesh, which is what makes it idempotent after firing. The question exposed a hole worth more than the redesign. The stored rejoin key was one of the 2026-09-12 FV cutover keys, expiring 2026-09-19 -- a break-glass credential that dies in four days and fails silently at the only moment it matters. Replaced with a dedicated 1-year reusable key (headscale ID 8, expires 2027-09-15), vaulted as fv-ml1/headscale-breakglass-key, root:600 on the host. That also closes the standing self-join risk rather than trading it: the two stale reusable keys (IDs 5, 6) are expired, so the mesh now has exactly one live reusable key -- purpose-built, on a host we control -- instead of two orphans nobody owned. Rejoin uses --accept-routes=false and the reason is in the script: on 2026-09-14 tailscale up --accept-routes on this box accepted its OWN subnet from the gateway and black-holed it. That happened with a human watching; here it runs unattended, during an incident, on a box already in trouble. |
||
|
vh
|
838132cd6b |
memory: snapshot — FV cross-site routing fixed, fleet conventions pinned
Session captured: the FV outbound-NAT root cause and its diagnostic signature, the fv-ml1 dead man's switch, fleet identity/group/path conventions and the root:docker normalization, nh3-dev's ts-input reachability fix, ESPHome modernisation and the kb KB-search tool, and the Hermes bearer rotation release. Six new detail files. Tried-and-abandoned gains three: probing OPNsense endpoints by POSTing at them (which rebooted the FV firewall), advertising a /32 from nh3-dev, and the nh3-scale remote-site masquerade rules that fired but were not the fix. Housekeeping: 8 Recent-decisions entries archived to archival-memory.md, and 21 oversized inline entries split into detail files per the two-tier rule -- they had been sitting fully inline in the index, which is what the split exists to prevent. Two pointers to a detail file archived this run were repointed at archival-memory.md. The index is 389 lines, still over the ~300 soft cap. The archival guards stop it there: only 4 further entries are old enough to move and every one carries an open deferred-work pointer. An over-cap file that keeps live decisions beats a scannable one that lost a deferred call. |
||
|
vh
|
0ab9da5b89 |
fix(fv): broaden the Tailscale SNAT rules from fv-ml1/32 to the FV LAN /24
All four outbound-NAT rules on the FV gateway now match source 10.251.50.0/24 instead of fv-ml1's single address, so a second host at FV works on arrival rather than reproducing a failure whose symptoms point at routing rather than NAT. Anaheim got a /24 rule of its own through the API. The 2026-09-13 ANA rule was written with write_config and is invisible to source_nat/search_rule, so leaving it as the only ANA coverage would have kept one destination on a different code path from the other three. The legacy /32 rule is now redundant but harmless -- it NATs identically and first-match wins -- and is noted in the runbook for deletion from the UI, since it is the one rule the API cannot see. Descriptions rewritten to name the real scope. Three of them said "fv-ml1 to X" while covering the whole subnet, and a description that understates a rule's reach is the same trap as the Anaheim-only scope that caused this. Verified after: fv-ml1 reaches NH3, nh3-dev, ESH, Anaheim, Irvine, the mesh and the internet; nh3-dev, esh-docker-vm and ana-docker all reach FV and each other; the FV BMC remains reachable inbound. Pre-change config backup taken. Also records that 10.251.250.0/24 (BMC/management) is deliberately NOT covered -- inbound reachability is what out-of-band recovery needs, but a management host originating traffic to another site would hit this same wall. |
||
|
vh
|
fa04f450fb |
fix(fv): extend the Tailscale SNAT rule to NH3, ESH and Irvine
FV could not reach any site but Anaheim. The cause was a single outbound-NAT rule on the FV gateway, added 2026-09-13 and scoped to Anaheim only -- docs/runbooks/fv-to-ana-nat.md says so in as many words: "Other remote sites remain outside this fix's scope." Three mirrors added, same interface and source, only the destination differing: 10.100.0.0/16, 10.0.0.0/16 and 10.6.110.0/24. After: fv-ml1 reaches NH3, ESH, Anaheim, Irvine, the mesh and the internet. Regression sweep clean across nh3-dev, nh3-docker and esh-docker-vm. The runbook now records what the failure looks like, because it presents as a routing or Tailscale fault and is neither. fv-ml1 reached mesh addresses perfectly and LAN addresses not at all; the FV firewall log showed the outbound passing with src=10.251.50.54 and no reply returning; temporary counting rules proved nh3-scale received 5 packets and sent 4 replies; both peers' AllowedIPs were correct. The discriminator that settles it is that every other site pair works -- nh3-docker to esh/ana/FV and esh-docker-vm to FV all succeed -- so a general subnet-to-subnet limitation is ruled out and only outbound SNAT is left. Also reverts the remote-site MASQUERADE rules added to nh3-scale earlier on the asymmetric-return theory. They fired but were not the fix, so they are removed rather than left to accumulate as NAT that achieves nothing. Applied via source_nat/add_rule + apply with a pre-change config backup taken first. Source scope is still fv-ml1's /32, so a second FV host will hit this again -- flagged in the runbook. |
||
|
vh
|
8c8559b8ec |
feat(fv): mesh dead-man's switch on fv-ml1; partial progress on FV cross-site routing
WATCHDOG (done, proven). fv-mesh-watchdog probes two independent anchors every
minute and, after 5 consecutive failures, puts Tailscale back to known-good:
accept-routes off, re-up against headscale with a stored key. It touches
nothing else — a watchdog with a wide remit is a second way to lose the box.
Two anchors that cannot share a failure mode: a plain-internet one and a
mesh-only one. If BOTH fail the site uplink is down, Tailscale cannot fix that,
and it deliberately does nothing — thrashing tailscaled during an ISP outage
turns a wait into an incident. Disable file at /etc/fv-watchdog.disable for
planned work.
Proven by positive control, not assumed: counter incremented 1..4 without
acting, fired the restore at 5 (tailscale up ran, tailscaled restarted), and
reset to 0 once the real anchor returned. fv-ml1 stayed reachable throughout.
This exists because a on fv-ml1 black-holed it
from its own LAN earlier the same day: it accepted 10.251.0.0/16 from the
gateway — its OWN subnet — and routed the local network through the tunnel.
FV CROSS-SITE ROUTING (partial). Two changes landed, the path is still broken:
1. acceptSubnetRoutes 0 -> 1 on the FV gateway's tailscale plugin, via
settings/set + service/reconfigure (the documented apply, not a reboot).
The GATEWAY now has 10.0/16, 10.100/16 and 10.250/16 in its routing table
and reaches NH3 and ESH itself. It could not before.
2. Remote-site MASQUERADE rules on nh3-scale. The existing jump matched only
-s 100.64.0.0/10, so traffic from another site's LAN never entered
MESH-EXIT and kept its original source; an NH3 host then replied via its
own LAN router instead of back through nh3-scale, making the path
asymmetric. The rule is confirmed firing (counter increments on FV
traffic) but does not complete the path.
Still failing: fv-ml1 -> NH3/ESH LAN addresses. Mesh addresses work perfectly
from fv-ml1 (100.64.0.1, 100.64.0.4), Anaheim works over the metro link, and
the FV firewall log shows the outbound passing on tailscale0 with
src=10.251.50.54 and no reply ever returning. The remaining gap is forwarded
FV-LAN traffic specifically, not the gateway's own.
Full regression sweep clean: nh3-dev, ana-docker and esh-docker-vm all reach
all four sites plus the internet.
|
||
|
vh
|
80d982d1d8 |
feat(backup): stage the FV firewall config in ana-docker's nightly restic run
The FV edge firewall was not backed up anywhere. Its config now lands in
/var/lib/restic/stage/fv-gateway-config.xml via ana-docker's pre-backup hook,
so the existing 01:00 restic snapshot captures it. ana-docker is one of the
three egress addresses the firewall's WAN allowlist permits, which is why the
pull lives there rather than with the FV hardware — a site that has lost power
cannot back itself up, and FV lost power two days ago.
Non-fatal by design: an unreachable firewall must not abort the nightly
database dumps. But a bad pull must not be promoted either. The summary loop
only rejects EMPTY staged files, and this endpoint answers an auth failure
with a perfectly non-empty HTML error page — which would have been backed up
as a firewall config that is the right size and restores nothing. The block
checks the body really contains <opnsense> and writes nothing otherwise.
Three tests cover it, including the HTML-error-page case. The first draft of
those tests was worthless: _fv returned a Path out of a TemporaryDirectory
context, so the tree was deleted before the assertions ran and every
exists()-is-False check passed regardless of what the script did. Only the
positive test failed, which is the sole reason the broken negatives were
caught. They now snapshot inside the tempdir's lifetime, and the docstring
says why.
Also records two OPNsense API lessons in docs/pfi/opnsense-api-reference.md:
endpoints are actions and must never be probed for existence by POSTing at
them — that is how /api/core/system/reboot took the FV site dark for 3.5
minutes while looking for an apply call this same file already documented —
and the apply step is service/reconfigure, which auth/user notably lacks, so
an API-only key edit persists in config.xml and does nothing until the OS user
sync runs at boot.
Credentials in /etc/restic/fv-gateway.env (root:600), template committed,
values vaulted as fv-gateway/opnsense-api-{key,secret}. Pre-change config
snapshot vaulted as fv-gateway/config-backup-20260914.
|
||
|
vh
|
9dbd829b9d |
fix(mesh): make nh3-dev reachable at its LAN address from the mesh
One rule on nh3-scale (CT 107): -d 10.100.10.50/32 -j MASQUERADE, above the RFC1918 RETURNs in /usr/local/sbin/mesh-exit-masq.sh, so it survives a reboot rather than living only in the running ruleset. Cause. A host that runs Tailscale installs -A ts-input -s 100.64.0.0/10 ! -i tailscale0 -j DROP. The fleet's subnet routers run NoSNAT: true with RFC1918 explicitly exempted from masquerade — deliberate source preservation, and a departure from Tailscale's own --snat-subnet-routes=true default — so a mesh client's packet reached nh3-dev's ens18 still sourced 100.64.x and died at the anti-spoof rule. Every NH3 host that does not run Tailscale was unaffected, which is why this read as a DNS or routing fault rather than a policy one. Masquerading just this destination makes it behave like every other host and leaves source preservation absolute elsewhere. Verified before and after against 13 targets from nh3-dev and 9 from the MacBook Air, and again after restarting the service so the chain was rebuilt from the script rather than from the manual insert. nh3-dev.nh3.internal now resolves and connects from the mesh, ssh and the Booth port included, with no script changes anywhere. Records the failed approach prominently, because it is the attractive one: advertising 10.100.10.50/32 from nh3-dev itself black-holed it from ESH, Anaheim, FV and Irvine. ip rule there puts lookup 52 at priority 5270 ahead of main at 32766, and becoming a subnet router let table 52 capture cross-site traffic the node has no accepted route for. Its own LAN and the internet kept working throughout, so a single-host check confirms a break it cannot see. |
||
|
vh
|
e64193171b |
docs(nh3-dev): Hermes bearer rotation hold released
svos-dev split their Bifrost wall's HS256 signing key off the Hermes Bearer (svos main 7165272), so nh3-dev/hermes/api-server-key is free to rotate again. The previous note said do-not-rotate and would have made a future session refuse a legitimate rotation on stale grounds. Not rotating now: the key was minted today, is vaulted, and has never been exposed — rotation is a hygiene action with a trigger, and none applies. What changed is the capability, which is what the record needs to reflect. Also records two things for when the svos_miranda plugin arrives: it will reference the dispatch key rather than the Bearer (expected, not a defect), and its tools array is legitimately seven or eight entries because repo_read is conditional on a config block SVOS owns. A third number is a real fault. |
||
|
vh
|
c659fa5020 |
fix(esphome): correct the record — mDNS advertisement was never removed
ha-dev caught a false claim I committed in
|
||
|
vh
|
687c6999f3 |
fix(esphome): actually disable remote-build — two switches, only one closes the port
ha-dev found the WS API and tested the read half; this runs the write. But the
command they identified is the wrong half, which is worth recording because the
naming actively misleads.
remote_build/set_offloader_settings {remote_builds_enabled: false}
the OUTBOUND half — this dashboard sending builds to peers.
Persists, reads back false, and leaves the receiver listening.
remote_build/set_settings {enabled: false}
the receiver-side master switch, per ReceiverController.set_settings's
own docstring. Tears the listener down live, no restart needed.
Set both. Verified across a restart: 6055 absent, zero peer-link bind lines,
zero mDNS advertisements, both switches read back false. Persisted at
_remote_build.enabled in /config/.device-builder.json — which did not exist
until the flag was first changed, so 'no on-disk representation' was true only
of the default state.
ESPHOME_REMOTE_BUILD_HOST=127.0.0.1 is KEPT as a backstop rather than removed.
The off state now lives in one JSON file whose in-code default is enabled:True
(controllers/remote_build/_state.py) and whose module's stores soft-recover to
an empty model on a malformed blob rather than erroring — so a lost or corrupt
settings file silently re-enables remote-build. With the env var set, that
regression binds loopback instead of 0.0.0.0.
Also finishes deploy-stack.sh properly. This was patched three times in one
session because -a is -rlptgoD and a non-root identity cannot apply owner,
group, permissions OR times to a root-owned directory; each patch fixed one
letter and the next deploy failed on the next one, every time exiting 23 AFTER
a successful transfer. The rule is now written into the script: the deploy
syncs content, the conventions own metadata. --no-o --no-g --no-perms
--omit-dir-times. Verified: clean run, destination keeps 2775 root:docker with
setgid intact.
|
||
|
vh
|
8073a6aed9 |
fix(esphome): bind the remote-build peer-link to loopback; finish the rsync fix
ha-dev asked for the Device Builder 1.0.0 remote-build receiver to be turned off: one instance, builds run locally, so the feature has no role, and it was binding 0.0.0.0:6055 with mDNS advertisement on a privileged host-network container that writes firmware to devices. Reading the source first changed the framing. controllers/remote_build/ _state.py declares 'remote_builds_enabled: bool = True', so nobody enabled it — it arrived on by default with the rewrite. And the flag has no on-disk representation until it is changed: neither .device-builder.json nor .device-builder-preferences.json carries it, and the only writer is the app's own command API behind the UI. Setting it from a playbook would mean inventing a schema for a model I have not read. So this binds ESPHOME_REMOTE_BUILD_HOST=127.0.0.1 — a documented env var, no entrypoint override — which removes the LAN reachability now and is verifiable (ss reports 127.0.0.1:6055, was 0.0.0.0:6055). It is explicitly NOT the off switch ha-dev asked for and the compose comment says so; the Settings toggle is one UI click and the line can go once someone flips it. Also completes yesterday's deploy-stack.sh fix, which was half a fix. --no-o --no-g stopped rsync chgrp-ing a root:docker destination as a non-root identity, but the very next deploy failed the same way one layer along — 'failed to set times on ...' — because a non-root identity cannot utime() a root-owned directory either. Same exit 23 after a successful transfer. Added --omit-dir-times. Fixing only the group half looked fixed until the next run, which is the whole reason this is worth a line in the script's comment. |
||
|
vh
|
d1769ed114 |
feat(esphome): pin 2026.8.2, relocate config into backup coverage, rotate creds
ha-dev requested all three on esh-docker-vm (operator-authorized); the stack had no canonical copy, so it is added to stacks/ rather than edited in place. Pinned ghcr.io/esphome/esphome:2026.8.2 — it was bare, which is exactly how it sat on 2025.8.2 for a year: docker pulled latest once at container creation (2026-04-20, from a layer cached 2025-08-29) and never re-pulled. Every current Everything Presence sensor failed config validation on that build. Verified after: esphome version reports 2026.8.2 and the vendor's own Pro package now validates clean (exit 0, 'Configuration is valid!'), which is the item that unblocks the six waiting sensors. Relocated /path/to/esphome/config (the upstream template placeholder, taken literally by docker) to /opt/docker/conf/esphome, matching the mosquitto pattern. Copied and checksum-verified all 5763 files before removing the original, with a tarball kept at /root/pre-change-archive/. Credentials moved off test/ChangeMe to the vaulted 32-char secret (esh-docker-vm/esphome-dashboard), passed via a host-only .env so nothing plaintext enters git. Three things the job surfaced that were not in the request: The directory is 538 MB, not the 3 KB reported — .esphome/platformio is 508 MB of PlatformIO toolchain and .esphome/build another 31 MB, both regenerable. Relocating as-asked would have inflated restic's /opt/docker source ~45x against its own ~12 MB budget, so both subtrees are excluded in /etc/restic/profiles.yaml. The 3 KB of actual config is now covered, which was the point. 2026.8.2 logs a DEPRECATION for the bare USERNAME/PASSWORD env names and says they will stop working in a future release — a silent auth loss on some later bump, on a privileged host-network container that can flash any ESP device on the LAN. Switched to ESPHOME_USERNAME/ESPHOME_PASSWORD; the warning is gone. Device Builder 1.0.0 opens a NEW listener on 0.0.0.0:6055 (remote-build peer-link) that 2025.8.2 did not have. Also fixes deploy-stack.sh: plain 'rsync -a' makes rsync chgrp the destination as the deploy identity, which since the 2026-09-14 root:docker normalisation is not root. It failed with 'Operation not permitted' and exit 23 AFTER transferring content — a loud error on a deploy that had succeeded. --no-o --no-g lets the setgid bit assign the group instead. |
||
|
vh
|
68fa80f44d |
feat(scripts): add kb — direct search over the personal Worldtree KB
The Worldtree HTTP API cannot answer a question about the operator's notes.
/search there searches conversation MESSAGES, so a note that plainly exists
comes back as a clean empty result with no error attached. On 2026-09-14 a
search for 'shrimp' returned 0 hits; searching for 'the' and 'a' also returned
0, which is the only reason the empty result was read as an empty ACCOUNT
rather than an empty KB. kb reads the markdown tree directly instead:
deterministic, ~0.9s for 7,634 files, no tokens.
Two measurements shaped the design rather than being assumed:
7,492 of 7,634 notes are INGESTED library material (4,155 fiction chapters,
3,287 book sections, 50 academic papers) and only ~142 are hand-written.
A flat relevance list buries the wanted note under a hundred chapters of
Austen, so NOTES and LIBRARY are ranked and reported separately.
Only 137 notes carry a frontmatter summary: key. Ingested notes use a
'## Summary' body heading instead and some have neither, so the description
falls back through all three shapes.
Two bugs caught by controls before shipping, both of which produced confident
wrong output rather than an error:
Deriving the word list from argv meant a quoted
NOTES — 40 matches, showing 12
Sous Vide Shrimp
ATLAS/Cooking/Sous Vide/Sous Vide Shrimp.md
Thawed shrimp should be sous vide at 135°F (57°C) for 30-40 minutes.
Beef Stew
ATLAS/Cooking/Sous Vide/Beef Stew.md
This note outlines sous vide cooking temperatures and times for stew meat
Pulled Pork
ATLAS/Cooking/Sous Vide/Pulled Pork.md
This note explains how to cook pulled pork sous vide: set the precision
Brisket Sous Vide
ATLAS/Cooking/Sous Vide/Brisket Sous Vide.md
Here''s a concise summary:
Ribs Sous Vide
ATLAS/Cooking/Sous Vide/Ribs Sous Vide.md
Here''s a concise summary:
Derusting Solution
ATLAS/Chemistry/Derusting Solution.md
This note details how to create an enhanced rust removal soak by adding specific
CNC with Raspberry Pi, USBIP & Camera
clippings/CNC with Raspberry Pi, USBIP & Camera.md
Here''s a concise summary of the note:
SF - Victor
ATLAS/Buy List/SF - Victor.md
This order confirmation details 7 separate shipments totaling $2,533.45,
Espresso Martini
ATLAS/Cooking/Espresso Martini.md
This note provides a recipe for a cocktail combining vodka, coffee liqueur,
Brazilian Cheese Bread - Pão de Queijo
ATLAS/Cooking/Brazilian Cheese Bread - Pão de Queijo.md
This note provides a recipe for Brazilian cheese bread (#brazilian #food
Congee Chao
ATLAS/Cooking/Congee Chao.md
This note provides the basic ratio (1 part rice to 7 parts water) for making
White Bread
ATLAS/Cooking/Baking/White Bread.md
Here''s a concise summary:
LIBRARY (ingested books, fiction, papers) — 635 matches, showing 12
Pride and Prejudice — CHAPTER XXI.
fiction/rex390-pnp/ch23.md
Following Mr. Collins’s proposal, Elizabeth encounters Wickham and learns that Jane has received a letter from Caroline Bingley announcing the party's immediate departure for London. While Jane interprets this move as definitive proof of Bingley’s indifference and permanent absence, Elizabeth remain
Pride and Prejudice — CHAPTER XXIV.
fiction/rex390-pnp/ch26.md
Following Bingley’s letter confirming his settlement in London and growing intimacy with Miss Darcy, Elizabeth doubts the sincerity of his attachment to Jane, while Jane remains optimistic that external influences rather than design are responsible for their separation. The sisters debate these diff
Pride and Prejudice — “On the Stairs.” CHAPTERXXVII.
fiction/rex390-pnp/ch29.md
Elizabeth reunites with Jane in London, where Mrs. Gardiner reveals that Jane suffers from periodic dejection despite her cheerful exterior, and the women debate whether Mr. Wickham’s pursuit of Miss King is motivated by mercenary or prudent reasons. Elizabeth then accepts an invitation from her aun
Pride and Prejudice — CHAPTER XXXII.
fiction/rex390-pnp/ch34.md
Mr. Darcy’s frequent visits to Hunsford Parsonage spark speculation among the locals, particularly Mrs. Collins, who suspects he is in love with Elizabeth despite her own dismissal of the idea. Their initial interactions reveal a clash of perspectives on social convenience and local attachment, whil
Pride and Prejudice — Chapter XLVI.
fiction/rex390-pnp/ch48.md
Following Lydia’s elopement with Wickham, Elizabeth Bennet informs Mr. Darcy of the scandal, reflecting that her earlier failure to reveal Wickham’s true character may have prevented the crisis and doubting their intent to marry due to their lack of funds. While Darcy offers sympathetic silence befo
Pride and Prejudice — CHAPTER XIII
fiction/rex390-pnp/ch15.md
Mr. Bennet announces that Mr. Collins, the heir to Longbourn, will visit on November 18th, prompting mixed reactions from his family regarding the entail and Collins’s pompous letter. Upon arrival, the tall and stately visitor formally compliments Mrs. Bennet’s daughters and praises the estate, thou
Pride and Prejudice — Covering a screen. CHAPTER VIII.
fiction/rex390-pnp/ch10.md
In Chapter VIII, Elizabeth endures the superficial sympathy and class-based mockery of the Bingley sisters while they criticize her muddy appearance and "low connections," even as Darcy defends her eyes and acknowledges her sisterly affection. The chapter highlights a clash of values when Darcy argu
Pride and Prejudice — “Conjecturing as to the date.” CHAPTER XLIII.
fiction/rex390-pnp/ch45.md
Elizabeth’s visit to Pemberley fundamentally shifts her perception of Mr. Darcy, as the estate’s elegance and Mrs. Reynolds’ glowing testimony reveal his true character as a kind master and brother. This admiration deepens into gratitude upon seeing his portrait, softening her view of his past pride
Pride and Prejudice — CHAPTER LVI.
fiction/rex390-pnp/ch58.md
Lady Catherine de Bourgh arrives at Longbourn to confront Elizabeth Bennet, demanding she promise never to accept Mr. Darcy’s hand based on claims of superior lineage and the scandal surrounding the Bennet family. She argues that Elizabeth’s inferior birth and lack of fortune constitute a disgracefu
Pride and Prejudice — PRIDE. and PREJUDICE
fiction/rex390-pnp/ch02.md
Jane Austen’s *Pride and Prejudice* is presented as her most perfect work, distinguished by its structural regularity where every incident drives the plot toward a denouement strictly connected to earlier events. The novel’s supreme merit lies in its masterpieces of humor and character creation, whi
Pride and Prejudice — A note for Miss Bennet. CHAPTER VII.
fiction/rex390-pnp/ch09.md
Mr. Bennet’s estate entailed on a distant relation leaves his daughters with limited financial security, yet the family’s attention is dominated by the arrival of the militia in Meryton rather than Mr. Bingley’s fortune. Mrs. Bennet successfully engineers Jane’s stay at Netherfield by sending her ou
Pride and Prejudice — CHAPTER XVI.
fiction/rex390-pnp/ch18.md
In Chapter XVI, Mr. Collins and the Bennet cousins visit Meryton, where Mr. Wickham captivates the room and initiates a conversation with Elizabeth regarding Mr. Darcy’s character. Wickham claims that Darcy unjustly withheld a valuable living promised by his father, attributing this act to jealousy
arrived as ONE element and became a single three-word pattern. The phrase
never appears in a note titled 'Sous Vide Shrimp', so the tool reported
'no match' for a note it had just found for the bare word 'shrimp'. The
needle is now split on whitespace.
Resolving the payload from dirname $0 broke the moment it was symlinked onto
PATH. Now readlink -f.
cat refuses any path resolving outside the KB root — the remote half runs as
root because the volume is root-owned.
|