memorization_check.py answers HOW OFTEN an arm collides with the training text.
It does not answer WHAT is being reproduced, and those have different
consequences: a low rate hiding one 30-word run of distinctive imagery is worse
than a high rate of stock dialogue in the commonest words in English.
The lv-hemingway gate shipped at 7x the author's own collision rate precisely
because that reading was done -- every matched run was stock dialogue, max 9
words, no proper noun. But it was done BY HAND and left no instrument, so the
next gate had to repeat it by hand or skip it. This is it.
Prints every maximal matched run sorted by length, with arm/id/seed, and flags
any token capitalised mid-run as a possible proper noun. The flag deliberately
over-reports (sentence-initial I, He, What all trip it) because it is a reading
aid and over-reporting is the safe direction. --train-only matches
memorization_check.py so a collision with held-out text is not reported as
memorisation of training text. A corpus slice is matched against the corpus
every run, and the script REFUSES if that positive control fails -- a matcher
that only ever sees candidate text cannot tell "no matches" from "blind".
Validated against the lv-hemingway record's documented reading, which it
reproduces to the word: longest run 9 words, "swift tristan" flagged as the one
name-shaped hit (it is the RENAMED invented name, not Hemingway's), and the
record's quoted examples -- "came over and sat down at the table", "i don t
think so the girl said" -- both present.
Required by GATE-PREREG.md AMENDMENT 1, which makes reading the matches part of
axis B rather than a follow-up, because McCarthy is in copyright with a living
estate and a match carrying distinctive imagery or a proper noun is
disqualifying in a way a rate number alone is not.
The runbook stopped at D4 pairs, so the two stages that actually produce the
shippable artifact were unrecorded -- the same gap that forced the 2026-09-17
D1->D3 chain to be recovered by rebuilding candidates and matching sha256.
D5 records the 1,380-step run, the loss curve read (median neighbour jitter
0.00393; ckpt900 the minimum, ckpt450 at +4.9x, adapter/ at +18.4x), the fact
that the epoch-3 collapse STEPS rather than drifts, and that adapter/ is the
epoch-3 weights so whatever ships is a checkpoint.
It also records the four provenance fields that look wrong and are not -- all
four reproduce on the yarros and hemingway runs -- and how to bind the run to
its pairs by RECORD COUNT rather than by the relative path it stored.
D6 records the gate invocation, the four deviations from the lv-hemingway gate
and what forced each, and the two instrument questions that had to be settled
before it could run: the punctuation confound on the voice axis (primary /
secondary / pre-registered trigger) and the defective base-unadapted control on
the memorisation axis.
Plus one thing that will otherwise be misread: gen_beats_chat_yarros.py prints
RAN-ON when it finds no paragraph break, a heuristic written for the Yarros
register. McCarthy's register asks for continuous scene prose, so an unbroken
block is the TARGET here and the tag fires on most generations without meaning
damage. The axis-C metric is a different definition and is unaffected.
The lv-hemingway gate established that memorization_check.py's negative control is
defective: base-unadapted writes SUMMARY while the adapted arms write PASTICHE, and
text that does not imitate a register cannot collide with that register's n-grams.
Base's 0.00 partly measures "different register", not "did not memorise". A negative
control that differs from the candidate in a way correlated with the metric is not a
control.
The correct innocent sample is the author himself -- held-out text no arm trained on,
in the same register by construction. That control was computed BY HAND during the
lv-hemingway gate and never committed, so the finding it produced was not
reproducible. This commits it as --heldout-reference, with --train-only to build the
n-gram set from split=="train" records only (the default includes val, so a collision
with held-out text was being counted as memorisation of training text -- an overcount
Bronte's and Hemingway's numbers both carry).
--heldout-reference without --train-only REFUSES, because the held-out text would be
scored against a gram set containing itself.
Instrument controls:
- default path reproduces the shipped lv-hemingway memorization.txt BYTE FOR BYTE
- the new held-out row reproduces the lv-hemingway record's hand-computed numbers
to the digit: 370 samples, hit-rate 0.01, mean-longest 0.1, max 10, at the same
101-word median chunk length -- an independently written instrument reproducing a
documented finding is what makes its zeroes mean absent rather than blind
GATE-PREREG.md AMENDMENT 1 adopts it for lv-mccarthy, appended rather than folded in
so the order of events stays legible. Written while the base arm was still generating
and before any McCarthy number had been read. It moves the reference a candidate is
compared against from a known-broken one to a known-correct one, in the direction
that makes the axis HARDER to pass, on an argument published four days before this
gate existed. Section 7's tiebreak is amended to match, and the amendment records
that an elevated rate is not by itself a no-ship -- rate and exposure are different
questions, every matched run gets READ, and McCarthy being in copyright with a living
estate makes a distinctive match disqualifying in a way a rate number alone is not.
Freezes the lv-mccarthy gate design while no McCarthy generation exists, per the
lv-hemingway precedent: a threshold chosen after seeing the numbers is not a
threshold. Three arms (base-unadapted, ckpt900, ckpt450), 60 beats, 4 seeds.
Settles the voice-axis question D1 deferred. McCarthy's corpus measures 0.0 quote
marks per 10k against Hemingway's 838, so "emit no quotation marks" is the cheapest
way to move a char-bigram Delta without learning a sentence. Three parts:
PRIMARY unchanged -- the mccarthy register names the punctuation and
--system-from drives the base control with the same prompt, so the
cheap win is handed to both sides.
SECONDARY voice_distance.py --secondary-normalised re-runs the whole analysis
with punctuation stripped from the reference and every arm. A
conservative lower bound; reported, never the verdict.
TRIGGER voice_distance.py --punct-report evaluates a pre-registered line --
base-unadapted quote density above 100 per 10k means the control did
not take the win it was handed, and the normalised read is promoted
to load-bearing. 100 is the order-of-magnitude line between this
corpus's 0.0 and Hemingway's 838, chosen now rather than after the
table prints.
ckpt450 is NOT tied with the minimum (+4.9x the 0.00393 median neighbour jitter)
and the pre-registration says so. It is generated to test a stated prior and to
price the memorisation headroom an earlier checkpoint buys on an in-copyright
author, with a decision rule that says exactly what result would let it displace
ckpt900.
Instrument controls, run before this landed:
- the voice_distance.py refactor reproduces the shipped lv-hemingway
voice_distance.txt BYTE FOR BYTE on the default path
- strip_punct drops a quote-bearing sample from 2500.0 to 0.0 marks per 10k
- the secondary read still resolves a gap on the Hemingway arms (+0.410 at
7.3x floor), so a null on McCarthy would be a finding, not a blind detector
Pre-flight re-run at gate time rather than quoted from 2026-09-17:
- leak gate: 0 of 75 renameable, 0 of 37 sub-threshold, 0 separator-split,
four controls green
- beat-contamination audit: 0 of 3942 beats AND 0 of 3942 responses, against
Hemingway's 70 of 7094 -- build_sft_pairs.py --source-entities earned its
mandatory flag
Also records three provenance defects found on first read of the run and their
disposition: the hardcoded "r49-babyyarros-pairs-pilot" run label (cosmetic, same
literal on all three runs), the empty harness_commit (all three runs), and a
pairs_sha256_16 that is not a sha256sum of the file (consistent across runs, so a
cache key rather than a fault). The run is bound to McCarthy's pairs by record
count, not by the label.
Archived 15 entries (Recent decisions 14, Tried and abandoned 1) oldest-first
to archival-memory.md; 5 held back on the open-deferred-work guard and 164 on
the 14-day guard, so the index stays over the soft cap at 477 lines. An
over-cap file that keeps live decisions beats a scannable one that lost a
belayed item.
Four new detail files cover the day: the ops log and its four self-inflicted
failure modes, the Booth's two dead controls and the four-iteration layout
probe, the Gitea org grant plus the dead claude-bot token that had been
misreporting permissions, and the disk triage that rescued a LoRA adapter from
a directory this box sweeps at three days.
lv-mccarthy's run outcome remains unverified after two days and is the first
line of the in-flight section and step 1 of the handoff.
brokkr-smithy-dev exercised the slicer on real geometry and found OrcaSlicer
writes result.json into the working directory on --info. I verified it here and
it is worse: --help does it too, which is as read-only as an invocation gets.
An empty directory gained a result.json from --help alone.
Encoded in the wrapper and the Dockerfile rather than left to be rediscovered,
and the wrapper is now committed beside the image that backs it instead of
living only in a home directory on one host.
⚠ The reason they found it and I did not is worth keeping in the header: I
verified --help and called the slicer done. Usage text proves the binary
starts, not that it ingests our geometry or that it behaves while doing so.
They ran both controls through to STL and got manifold=yes, 520 facets,
return_code 0 on each -- which is the verification I should have run before
reporting the provisioning complete.
Not tidiness — necessity, established by ldd rather than assumed. Every
OrcaSlicer release ships only an Ubuntu 24.04 AppImage, which needs GLIBC_2.38
and GLIBCXX_3.4.32. irv-ml1 is Debian 12 on glibc 2.36. That gap is not
installable without moving the host to Debian 13, which is not a thing to do to
a box running twelve GPU and audio services in order to slice a plate.
Reaching back to an Orca built for Ubuntu 22.04 would run, and would mean
pinning permanently to a stale build -- the exact trap brokkr-smithy-dev
rejected when they chose Orca over an old PrusaSlicer tag. The container keeps
the slicer current AND leaves the host alone, which is the same argument that
made FreeCAD an AppImage, taken one step further because the host could not
satisfy this one at all.
The AppImage is extracted at BUILD time; --appimage-extract-and-run re-extracts
to /tmp on every invocation, which is wasted seconds and wasted disk per slice.
~/bin/orca-slice wraps it so the dispatcher calls a binary and the artifact root
is mounted at the same path inside and out.
⚠ Recorded honestly: I installed libwebkit2gtk-4.1-0 and 8 dependencies on the
host chasing errors serially before enumerating with ldd, and only then found
the glibc wall behind them. Those nine packages are unnecessary. Left in place
rather than autoremoved on a box running production services; flagged for the
operator.
Operator-authorized. build123d 0.12.0 + OCP, numpy 2.4.6, trimesh 5.1.0 in a
3.11 venv, FreeCAD 1.0.0 headless as an AppImage, artifact root on smithy.
The acceptance test is the point and it PASSES: brokkr-smithy-dev shipped a
contrastive control pair whose every property is construction-known, so the
install is checked against arithmetic rather than against a render. Both parts
build, both bounding boxes are exactly 40x40x5, and both volumes match to float
tolerance. These scripts had never been executed by anyone -- build123d is
deliberately absent from the authoring box -- so this was their first run.
FreeCAD is an AppImage BY CHOICE: the distro package drags a GUI/Qt dependency
tree onto a host running twelve GPU and audio services, where an AppImage is
self-contained, pinned and removable with one rm.
Slicer DEFERRED rather than guessed. PrusaSlicer's latest release ships no
Linux AppImage at all, and a guessed release filename had already produced a
4 KB error page in this same session. It is the one item they said could lag.
Two things worth keeping in the header: irv-ml1's ssh config pins the gitea
host to gitea_brokkr_smithy with IdentitiesOnly, whose comment is
irv-ml1-r14-deploy and NOT the arbo key that sorts first in ~/.ssh/*.pub -- I
registered the wrong one first, and the failure names the right one. And
`ssh irv-ml1` is lkraven with no NOPASSWD sudo while `ssh infra-ops@irv-ml1`
has it, so only the artifact-root step needs the other identity.
The deploy key was registered with claude-bot's own token rather than the
operator's site-admin token -- the org grant from earlier today doing real work
on its first use.
Operator: "release button covers delete button". Measured before touching
anything: release 58x24 at (323,266), × 30x30 at (349,268) — 30x22 px of
overlap on a 30px button, and `elementFromPoint` at the ×'s centre returned the
release form. The × I added yesterday was 100% unclickable from the moment it
shipped.
Cause: both were `position:absolute` on the same corner, each with its own
guessed offset, and `release` is the later sibling so it won. Replaced with one
flex row positioned once — release left, × right at the card corner where the
ephemeral lane's × already lives, so muscle memory transfers and neither can
drift back on top of the other when a label changes width.
Verified by measurement, not inspection: overlap 0 px, and clicks at each
control's centre now land on that control. The ephemeral lane's × and ★ were
re-checked and are unaffected.
ADDS scripts/layout-probe.py, because markup inspection STRUCTURALLY cannot see
this and I have now shipped two dead controls in two days by reading templates
instead of rendering them. It asks a real browser what a click at each
control's centre would hit.
It took four iterations to become trustworthy and the failures are the point:
1. `top.contains(el)` counted an ANCESTOR overlay as a hit -- the exact case
it exists to catch. Version one reported OK for a real overlay.
2. elementFromPoint is viewport-relative, so everything below the fold read
as occluded and buried the real findings.
3. getBoundingClientRect() on a WRAPPED INLINE element is the union of its
line boxes, whose centre lands in the gutter between them -- three zip
links reported occluded by their own parent. Real geometry, wrong question.
Only the fourth version fires on a genuine overlay while staying silent on the
clean page. Both controls were run; a probe never seen to fail proves nothing.
173 tests pass.
Three operator reports, one of them a real defect I had claimed was working.
THE REVEAL BUTTON DID NOTHING, for a day. Its handler sat after the content
block's closing tag, and a child template's out-of-block content is silently
DISCARDED by Jinja. The button rendered. The handler never reached the browser.
Two commits and a README paragraph said click-to-reveal worked, and the suite
passed the entire time because nothing asserted against the SERVED page -- the
template really did contain the code.
Two guards, both confirmed to FAIL when the defect is reintroduced rather than
merely added and assumed protective:
* test_reveal_handler_actually_reaches_the_served_page greps the response
* test_no_orphaned_markup_after_the_content_block guards the structure
While moving it, caught a second instance of the same class: the explanatory
comment I wrote for the fix contained a literal Jinja endblock tag, which Jinja
would have parsed as a real tag and used to close the block early.
KEPT-LANE ×. Wiping a kept booth required release-then-find-it-in-the-other-
lane. That protected nothing and cost a hunt -- the board you just released is
loose in a feed that turns over, and you have to go find it to finish a job you
had already decided on. Protection now lives in the confirmation, which names
the booth and says KEPT, instead of in the number of lanes you must traverse.
Release stays as the reversible option.
IN-BOOTH KEEP. `☆ keep` / `★ kept — release` beside "Wipe now", so promoting
does not mean navigating back to the index. The booth page did not previously
know its own kept state; it does now. Both post a `next` field to stay put --
and `next` is a form field, so it is attacker-controlled: only same-site
absolute paths are honoured, with `//host`, schemes and backslashes refused,
tested.
173 tests pass.
The check collapsed two different findings into one verdict. On 2026-09-20 it
printed "RESULT: STALE" while reporting 37 FRESH layers and zero stale ones --
every backup body provably current, the three ❌ rows all yesterday's pre-fix
runs aging out of the 36h window. infra-hermes caught it in triage: a reader,
or a forwarder, could page someone over a state where nothing is stale.
STALE is a claim about backup AGE. A job that ran and errored is a different
claim with different urgency. They now have different words and different exit
codes:
0 all backups fresh
1 STALE -- a body past the threshold, or an endpoint down
3 ERRORED-JOBS -- every body fresh, a vzdump job errored recently
The alert wrapper mirrors the code and matches its own wording to the finding:
🟡 "Backup jobs errored — all bodies fresh" instead of 🔴 "Backup freshness
ALERT", and it now exits with the check's code rather than flattening
everything to 1, so `systemctl status` distinguishes the states too.
This is the same defect class the rest of this script was built to fix, one
level up: not an instrument that fails to look, but one that looks correctly
and then reports the wrong word for what it saw. An alarm that cries outage
over a healthy fleet earns being ignored exactly as fast as one that stays
silent over a broken one.
Verified all three states by forcing each: BACKUP_JOB_WINDOW_HOURS=1 -> exit 0,
default -> exit 3, BACKUP_MAX_AGE_HOURS=1 -> exit 1.
The operator asked "no UI option to blur/unblur?" and was right twice over.
MISSING ENTIRELY ON TWO OF THREE BRANCHES. booth.html renders docs, media and
everything-else through separate <figure> blocks. The toggle went into the
media branch only, so inline docs -- the branch that puts readable text straight
on the page, the one that needs blur most -- had no control at all, and `other`
files only got a caption row if they happened to carry a caption.
This is the SECOND time this feature shipped having patched some branches and
not others; the blurred class itself had the same gap one commit ago. So the
toggle is now a single Jinja macro called from all three sites, which makes
"patched two of three" impossible rather than merely unlikely, and
test_every_item_kind_gets_exactly_one_blur_toggle counts toggles against figures
across mixed kinds so a fourth branch cannot quietly skip it.
INVISIBLE WHERE IT DID RENDER. v1 was a bare `◌` at 0.78rem in --muted with no
border, no label and no hover affordance. It now reads `◌ blur` / `◉ blurred`
with a border, matching the other per-item controls. A control nobody can find
is a control that is not there.
Docs get it in the doc bar beside ⤢ ⬇ ✕, with stopPropagation so submitting it
does not collapse the <details> it lives inside.
Verified live on all three kinds: 3 figures, 3 toggles, and the POST round trip
blurs and un-blurs. 167 tests pass.
Two operator requests.
KEEP, BOTH DIRECTIONS. The kept lane could already release a booth back to
ephemeral, but an ephemeral booth could only be promoted from a shell -- so the
round trip was closed only if you had ssh. The /keep route and the `booth keep`
verb both already existed; only the button was missing. Adds ★ to the ephemeral
card, mirroring × on the other shoulder.
BLUR. Per-item cosmetic censoring: `booth blur <name> <file>...`, a ◌/◉ toggle
in each caption row, and 👁 click-to-reveal. State is `.blurred` in the booth
dir, one booth-relative path per line -- the same filesystem-is-the-state idiom
as .pins and .forever. An empty set deletes the marker rather than leaving a
zero-byte file, so `ls -a` tells the truth.
⚠ BLUR IS NOT ACCESS CONTROL, and the code, the docs and a test all say so on
purpose. A blurred item is still served at its own URL, still in the zip, still
on disk. The Booth has no auth by design. test_blur_is_cosmetic_the_file_is_
still_served asserts the 200 deliberately: if someone later "hardens" this into
a 403 that test fails, and it should, because half-implemented access control is
more dangerous than none.
Reveal is per-viewer and never persisted; a reload re-hides. With JS off an item
stays blurred, which is the safe direction to fail in.
Two things the first pass got wrong, both caught by checking rather than
assuming:
* The cover thumb. index.html has IDENTICAL markup in the kept and ephemeral
lanes, so a single-occurrence replace patched only the kept one and the
ephemeral front page happily displayed the thing someone had hidden. The
test that caught it was itself wrong first -- it matched the bare string
"blurred-thumb", which is in base.html's stylesheet on every page and so
passed in both states. It now asserts the attribute.
* Inline docs render through their own <figure> branch and were left
unblurred -- the branch that puts readable text straight on the page, so it
needed blur more than images do. The suite passed; a live curl caught it.
165 tests pass (154 pre-existing, unchanged).
Operator confirmed 2026-09-19 that fv-ml1 and the R420 running OPNsense are the
only loads on a dedicated 20 A circuit.
The governing number is 1920 W, not 2400: a GPU inference host running for hours
is a continuous load, so NEC's 80% rule applies. Worst case lands at ~1625 W
with the current caps -- about 85% of budget.
Measured via the BMC rather than assumed: 390 W instantaneous, 461 W max over a
2423 s sample, with GPUs at idle, giving a ~313 W non-GPU baseline.
Compare the GPU caps against the 300 W stock TGP, NOT the 325 W firmware
ceiling. The operator corrected this: 275 W across four cards saves 100 W, not
the 200 W you get by measuring against a number nobody would ever run at. Stock
300 W would put the circuit near 90%, which is not illegal but leaves nothing
for a heavier R420, PSU efficiency, or a warm day. Keep the caps.
The coupling matters more than the trip. OPNsense IS the Fountain Valley edge
and shares the breaker with the thing most likely to trip it, so an overload
takes the router with it and removes the remote path needed to diagnose or
power-cycle anything. fv-ml1's four PSUs do not help -- PSU redundancy protects
against a PSU dying, not against the circuit going away.
Three things are explicitly NOT measured and the file says so: fv-ml1 under real
4-GPU load, whether the BMC reports AC input or DC output, and the R420's actual
draw. Treat 1625 W as a floor.
Also corrects the hardware section, which claimed 2x GPUs. nvidia-smi reports
four.
Four attribution failures shipped on the day it was built, and every one of
them RECORDED SOMETHING -- just not something findable or attributable. The
individual fixes are in the commit log; the PATTERN was not written down
anywhere, so a future session would meet each one as an isolated bug rather
than as the fifth instance of a known shape.
infra-hermes asked for this after the fourth one: 'worth an audit line item so
the pattern is visible, not just the fixes.' He was right.
The operative warning is at the bottom: absence of an entry is not evidence of
absence of the event until you have checked this tool could have recorded it.
elway passes its `host` argument straight through, and that argument is an ssh
target. Five records of a real jobs.cfg change on esh-pve were written under
host `infra-ops@esh-pve`.
The consequence was not cosmetic. infra-hermes triaged the ESH backup job
errors, queried `--host esh-pve`, got nothing back, and correctly reported the
05:29 exclusion of VM 102 as an unattributed change. The record had existed the
whole time, five lines of it, under a name nobody would think to ask for. A log
you cannot query under the obvious name is not a log.
`normalize_host` strips any `user@` prefix on write AND on query, so records
written before this fix are findable too, and `audit` -- which matches records
by host name -- stops missing them as well.
Verified in both directions: a record written with `infra-ops@esh-pve` and one
written with `esh-pve` now both return for `--host esh-pve`, and the five
historical elway entries surface under that query.
The commit hook was APPENDED to .git/hooks/post-commit so as not to disturb
graphify's block -- and that made attribution a subordinate clause of
graphify's control flow. graphify owns EIGHT `exit 0` paths (rebase / merge /
cherry-pick in progress, GRAPHIFY_SKIP_HOOK, no changed files,
graphify-out-only, no python found), and every one of them returns before
reaching an appended line.
infra-hermes reported commits going unlogged and proposed a lock race: the
ops-log fcntl path held by graphify's detached rebuild long enough for a
best-effort `|| true` invocation to drop. Plausible, and wrong. The cause is
ordering and it is deterministic. eaaacc1 touched only
graphify-out/GRAPH_REPORT.md and hit the graphify-out-only exit; an empty probe
commit hit the no-changed-files exit. Neither could ever have been recorded.
Two data points that look flaky are worth reading as a control-flow question
before a concurrency one.
The block now goes immediately after the shebang. graphify's hook is preserved
byte-for-byte below it, and core.hooksPath stays unused because it would
disable graphify outright.
Verified in both states with real commits, then reset out of history: a
graphify-out-only commit -- the exact case that was silently dropped -- now
records, and a code commit still records AND still triggers graphify's rebuild.
The fallback recorded `lkraven@nh3-dev` when ALTHING_HANDLE was unset, which
reads like an answer while being the absence of one. It now records
`unattributed(lkraven@nh3-dev)` -- handles never contain @ or parentheses, so
the fallback cannot be mistaken for an identity.
Found by the commit hook's first independent test, which was not one I ran:
infra-hermes committed 0fe4da6 and the hook caught it unprompted -- but logged
him as lkraven@nh3-dev, not infra-hermes. His seat exports ALTHING_HANDLE from
althing-infra-hermes-seat-run.sh, while neither hermes-gateway.service nor his
pump unit carries it, so anything he does outside that one script logs with no
handle.
So the hook is real but partially blind: it distinguishes not-infra-ops from
infra-ops, which is enough to have told me e43e262 was not mine, and not
enough to name who made it. The env-var gap is his to close. Making the blind
spot announce itself in the record is mine -- an instrument that cannot answer
should say so rather than return something answer-shaped.
Both harnesses executed top-to-bottom on import, so anything collecting
them (pytest, a glob import) ran live-store side effects. main() guards
plus finally-block cleanup of the _t_* cron output dirs the fabricated
jobs leave behind.
Two agents share ONE checkout of this repo on nh3-dev, and every commit here
is attributed to Vuong Hoang by convention -- so a commit's author line says
nothing about which agent made it. The ops log closed that gap for HOST
changes; it did not cover commits to the shared tree.
Found the hard way today: e43e262 appeared interleaved between two of this
session's commits, in this session's own reflog, and was unattributable from
git, from the ops log AND from the althing bus. It was sitting in the push set
at the time, and forseti had explicitly asked that unrelated management-repo
commits not be pushed -- so the one thing needed to honour that request was
the one thing nothing recorded.
The hook APPENDS to .git/hooks/post-commit rather than replacing it, because
graphify already owns that file and core.hooksPath would disable it.
Best-effort by construction: a failure here must never fail a commit.
deploy-stack.sh claims and releases around its own work. When the agent
already held a longer claim for a multi-step operation, the deploy refreshed
it, then released it on exit -- silently dropping the protection partway
through the very operation it was guarding.
Caught live: a 45-minute claim on nh3-docker/althing-post-office, taken to
cover a build-push-deploy-verify rollout, was gone by the time the rollout
finished. Nothing refused anything, and nothing said so.
`ops-log claim` now exits 10 when the claim was already the caller's, and
leaves the holder file untouched. deploy-stack.sh treats 10 as "not mine to
release". Untouched matters as much as the exit code: a refresh would
overwrite the reason and TTL the original claimant chose, so a deliberate
45-minute "3.7.0 rollout in progress" would degrade into "deploy-stack.sh
<host> <stack>" and whoever got refused would read the wrong story.
Verified in three states: no pre-existing claim -> deploy claims and releases;
caller's own wider claim -> deploy refreshes nothing and leaves it standing
with its original reason; another agent's claim -> still refused with exit 3.
Canonical pin bump for the 3.7.0 rollout forseti approved and released
(althing 6db955f, tag v3.7.0). Image built from that tree and pushed as
sha256:df0709b3f532557aebb47d6ece486ae32e851776f732862094608d55eefcd4e7;
previous pin was 3.6.3@sha256:978f8553 and remains the rollback target.
Content-verified rather than tag-verified: `postbox --version` inside the
built image reports 3.7.0 before it was pushed, and again inside the running
container after. A tag is a claim.
Also adds playbooks/nh3-extdev-althing-upgrade.yaml -- a version-parameterised
upgrade for the box a git pull cannot move. The existing
nh3-extdev-althing-v3.yaml is the historical v2->v3 CUTOVER and is left alone
rather than mutated per release; it still carries v2-retirement steps that
document what happened.
Every check in the new playbook asserts what the installed entry point
reports, including one that the herald restarted within the last 5 minutes
rather than surviving from before -- "active" does not mean "running the code
you just installed".
Disposable harnesses from the wake-RFI thread (01M2WT3F): exit1-behavior-test.py
covers the monitor exit-code/gate path including a sustained-failure/no-backoff
assertion; hash_gate_repro.py covers the detection-time hash persistence
crash-before-read boundary with consumption/unchanged/changed controls.
Both fail nonzero on mismatch. Committing so seat artifact provenance
rides git rather than file mtimes.
/etc/restic is 0700 root, so before elway 754db4b this step's creates: guard
evaluated unprivileged, always reported the file absent, and the step re-ran
on every invocation. A plain cp -p would then have overwritten the preserved
ORIGINAL with the current post-repair file -- destroying the one thing the
step exists to keep.
Verified it never fired: the preserved copy still holds the original
(md5 7b42324c, mtime 2026-04-21) and differs from the live hook (c79bef9a,
2026-09-15). The guard is fixed now, but a destructive step should not depend
on its guard being right.
esh-vm-workstation is a Windows 11 Parsec/RDP sandbox -- no password, no state
anyone needs recovered (operator ruling, 2026-09-19). Its nightly vzdump has
failed since ~2026-09-06 with "timeout waiting on systemd", and the ESH job has
reported `job errors` every night as a result, so a guest nobody needs backed
up was manufacturing a real job failure daily.
The job uses an explicit `vmid` INCLUDE list, so exclusion means removal from
that list -- there is no `exclude` directive here, which is the same
distinction that makes esh-scale CT 108 invisible in any exclude list. The
reason now lives in the job's own `comment`, and check-backup-freshness.sh
reports 102 under "NOT BACKED UP BY POLICY" rather than going quiet about it.
This is a policy decision, not a fix: the systemd-scope failure on stopped
guests is NOT diagnosed, merely no longer reached on this node. A stopped
guest that DOES matter would hit the same wall.
The playbook carries two hard-won warnings in its header. Read the job through
the API, never `grep -A<n>` on jobs.cfg -- `vmid` sorts last in the block, so
adding the comment line pushed it outside the window and every condition
silently inverted. And use YAML block scalars for the JSON-extraction
expressions; double-quoted ones did not survive YAML unescaping plus elway's
bash -c quoting.
Verified: vmid 100,101,103,104,105,106,107; the other seven guests still
covered; job still enabled; a rerun skips the change rather than repeating it.
Conditions ran unprivileged no matter what the step declared, and that fails
in the dangerous direction. A root-requiring `when:` -- `pvesh get ...` exits
255 for a non-root user -- returns non-zero, elway reports the step `skipped`,
and a playbook that never performed its change reports overall OK. "Skipped"
is indistinguishable from working idempotency, so the run looks correct.
Found the hard way on esh-pve: three consecutive runs of an exclusion playbook
reported success while changing nothing. Only the verify phase caught it, by
continuing to report the thing the steps claimed to have handled -- which is
exactly why verify runs unconditionally.
`creates:` had the same fault from the other side: a path under /root is
unreadable to the login user, so `test -e` said absent and the step re-ran
every time. It now correctly reports the file as already present.
Sudo-less steps are unaffected: their conditions still evaluate as the login
user, which is what they mean. Only a step that declares sudo: true gets
privileged condition evaluation, so no existing playbook changes meaning
unless it was already silently broken.
Snapshot age is structurally blind to a backup job that executes every night
and fails every night. Nothing new is written, so the group simply ages, and
the fault only surfaces once it crosses the 48h threshold -- days after the
first failure, with the evidence sitting in a task log nobody reads.
Two live cases, both found today and both invisible for a week by this exact
mechanism:
* esh-nas-pve CT 107 (vm-jellyfin): a backup run died around 09-06 and left
a stale `backup` lock, so every nightly since failed instantly with "CT is
locked (backup)". Age named it on ~09-12. Task status would have named it
on 09-07.
* esh-pve VM 102 (esh-vm-workstation): failing nightly since ~09-06 with
"timeout waiting on systemd". Same six-day gap.
PVE already records every task result in /var/log/pve/tasks/index. This reads
it on all four non-tenant PVE nodes and reports any vzdump in the last 36h
whose status is not OK, as its own section that sets the exit code.
It found a third case on its first run: esh-nas-pve's job had been reporting
`job errors` nightly while every guest on that node read 0-1h fresh, so no
age-based check could ever have flagged it.
Window is BACKUP_JOB_WINDOW_HOURS (default 36 -- longer than a daily cycle so
one missed run does not hide a failure). A node whose task log cannot be read
is reported, never assumed healthy.
ana-scale (CT 114) is a subnet-router LXC, excluded from vzdump on 2026-09-07
after a backup lock on its ESH counterpart blackholed that entire site. The
freshness check knew nothing about that and reported it 🔴 STALE every single
morning, which is how an alarm teaches you to ignore it.
Such guests now get their own section: printed every run, never hidden, and
not counted as a fault.
The subtlety is in how coverage is computed, and the obvious implementation is
wrong twice over:
* Reading one job's `exclude` list gets ana CT 109 (ana-nas) exactly
backwards. It IS excluded from the 03:00 all-guests job AND it has its own
dedicated 22:00 job. Suppressing on the exclude list would have stopped
alarming on a guest that is genuinely backed up -- trading a noisy alarm
for a blind one.
* ESH's job uses an explicit `vmid 100..107` INCLUDE list, so esh-scale 108
is excluded by OMISSION and appears in no exclude list at all.
So coverage is a union across every enabled job on the cluster, and a guest is
"intentionally not backed up" only when none of them covers it.
If coverage cannot be read, nothing is suppressed and the gap is reported: an
unreachable PVE node means we do not know, and a backup alarm must fail loud.
The SureFire namespace is never consulted (tenant property), so its guests can
never be suppressed either.
Verified against the live fleet on all four paths: CT 114 suppressed; CT 109
NOT suppressed despite being in an exclude list; esh-vm-workstation 102, which
a job really does cover and which really is failing, still reports STALE; and
with a PVE node made unreachable, 114 returns to STALE with the gap named.
The daily backup-freshness check has been unable to raise an alert since the
2026-08-28 althing v3 cutover. It called althing-cli, which v3 DELETED rather
than deprecated. The check itself never stopped working: it detected three
stale backups every morning and told nobody, and the only trace was a WARN
line inside a unit that was already reporting `failed` for the stale backups
themselves. Three weeks, silent.
Four changes, because swapping the binary alone would have left it dead:
* althing-cli -> postbox.
* Add ALTHING_POST_OFFICE to the systemd user unit AND to the installer that
writes it. postbox has no default address by design and a user unit
inherits nothing from the interactive shell, so the binary swap on its own
would have failed with a different message. Fixing only the live unit
would have been undone by the next installer run; the two are now verified
to agree.
* Recipient infra-ops -> infra-hermes. This runs AS infra-ops, so the old
address mailed the alarm to itself — the mirror trap named in CLAUDE.md.
Day-to-day checks are infra-hermes's half of the split; he escalates.
* Split the exit codes. 1 now means "backups stale, someone was told";
2 means "the alert path itself failed". A broken alarm is a worse fault
than the thing it watches and must not be indistinguishable from it.
Adds --test-alert: a positive control that sends a real message through the
real path on demand. The wire was cut for three weeks precisely because
nothing ever exercised it in the healthy state, and an alarm whose success
path is never run is not known to work.
Verified: positive control delivered; missing-address and unreachable-post-
office both correctly exit 2; a real run through systemd delivered the alert
and exited 1.
infra-ops and infra-hermes act as the same OS identity and dockerd does not
log exec per caller, so host-side changes carry no fingerprint. Git cannot
close the gap either: every commit here is attributed to Vuong Hoang by
convention, which is correct for authorship and useless for attribution.
On 2026-09-18 a second session edited the searxng stack mid-deploy, crash-
looping fleet search for ~4 minutes, and the author was unidentifiable.
scripts/ops-log records one line per host-changing action and holds a
lightweight claim so two agents do not deploy the same stack at once.
Four design questions, settled:
* Central on nh3-dev, not per-host and not the post office. Both agents
run as the same unix user there, so one file is shared with zero
provisioning. Per-host needs a writable path on ~25 heterogeneous boxes
and stores "we changed host Y" on host Y. journald looked free but shows
an unprivileged reader only their own _UID, which would have split the
log silently between the infra-ops and lkraven halves of the fleet.
* The claim is advisory and enforced in the tooling. deploy-stack.sh
refuses a foreign claim across the diff, the prompt and the apply -- the
whole review window, which is where the collision happened. Acquire is
mkdir, so it is atomic rather than probably-fine. Stale claims auto-break
and the break is recorded.
* Writers are automatic. deploy-stack.sh and elway record themselves; a log
that depends on remembering is the same class of instrument as a health
check that passes in both states.
* There is a detector. `ops-log audit` asks each host what changed on disk
and compares it to the newest log line for that stack, covering the
manual ssh-and-edit path the automatic writers structurally cannot.
ops-log being absent or broken never blocks a deploy; only a live foreign
claim does. `ops-log baseline` marks the 136 stacks that predate the
instrument so the detector starts from today rather than reporting the whole
fleet forever and training us to ignore it.
An unreachable host reports INCOMPLETE and exit 5, never clean.
Operator ruling 2026-09-19, recorded in three places because each serves a
different reader. CLAUDE.md gets the structural facts so a fresh session has
them without reading anything else; persistent-memory gets the dated decision
and the assigned work; auto-memory gets the durable working relationship.
The division: infra-ops keeps improving infrastructure tooling plus the hard
calls, infra-hermes takes day-to-day checks, triage and routine operations,
either may perform infra ops, and infra-ops may task him downward while he
escalates upward.
He is explicitly NOT Miranda. The global CLAUDE.md names Miranda as the sole
trusted relay of operator authority and that exception does not extend to him,
so a directive he relays is information rather than authorization — reversible
relayed work executes, irreversible or fleet-affecting goes to the operator.
He has acknowledged it in those terms.
⚠ The two handles differ by one character in the middle of a word and the
fleet's OS identity is infra-ops, so a misaddressed page still mails the sender
themselves. That trap is now documented alongside the existing mirror warning
rather than replacing it.
Building the ops log is assigned and not started. Two agents now share one
fingerprint-less OS identity: ssh infra-ops@<host> is either of us and dockerd
exec is not logged per-caller. The precipitating incident is on the record —
2026-09-18, a second session edited the searxng stack mid-deploy, crash-looped
fleet search for ~4 minutes, and the author was unidentifiable because every
commit is attributed to Vuong Hoang by convention. The parked attribution-gap
memory is unparked and points here.
The open design questions are noted as mine to settle, the load-bearing one
being whether deploy-stack.sh and elway write to the log automatically. A log
that depends on remembering is the same class of instrument as a health check
that passes in both states, and this repo spent yesterday learning what those
cost.
An infrastructure day with no training work, and the through-line is that
every fault was invisible to monitoring. NH3↔Anaheim had been crossing a
throttled DERP relay rather than a direct path for long enough to carry 78 GB;
`.internal` DNS was failing roughly one lookup in ten from two independent
causes; SearXNG had exactly one working general web engine. Nothing alarmed on
any of it. All three surfaced because tts-dev had a 1545 ms voice-loop budget
and chose to measure rather than adapt around the problem.
Also landed: althing v3.6.3, which makes hyphenated search work for the first
time on a fleet whose hostnames are nearly all hyphenated; FleetTools, a
capability index autoloaded by Claude, Codex and Grok from one symlinked file;
Miranda's Hermes plugin moved from a copy to a repo symlink; Worldtree's
env.sh secrets vaulted.
Six detail files. The in-flight section is rewritten and shrinks 142 lines to
64 — it opens on the one thing this session did NOT verify, lv-mccarthy's run
outcome, which was left untouched and must not be assumed good.
Two foot-guns recorded, both mine: the ESH egress experiment reverted on a
diagnosis the rollback itself falsified, and `!ENV` in searxng settings, which
has no constructor in that build and crash-looped the container ten times.
No archival this run. 165 of 169 dated entries are under the 14-day guard and
the remaining four all carry open deferred pointers, so the index stays over
the soft cap at 480 lines — an over-cap file that keeps live decisions beats a
scannable one that lost one.
Every NH3→Anaheim flow had been crossing Tailscale's LA DERP relay rather than
a direct path, for long enough to have carried 78 GB tx on the NH3 side alone.
DERP is a throttled fallback, so this imposed both a fixed round-trip penalty
and a bandwidth ceiling on LiteLLM, Beszel, task-board, vor and the Henge
alike. It surfaced as a voice-loop latency report from tts-dev, not as a
network alarm, because nothing monitors whether a mesh path is direct.
ana-scale advertised 38.120.12.42:41641 while the Anaheim NAT mapped it to
:60798 with no port-mapping protocol available, so inbound hole-punching always
failed. Four additive objects on ana-gw give it a stable inbound endpoint.
tailscale ping nh3-scale->ana-scale 373-522 ms via DERP -> 6 ms direct
STT via the ANA gateway, 96 kB clip 1.399-1.449 s -> 0.237-0.270 s
Beszel HTTP nh3-dev->ana-docker 0.94-1.29 s -> 0.014-0.016 s
Documents the house template that matters for this box: it runs central-nat,
so a policy dstaddr is the real internal address and not the VIP. Also records
that the pre-change config was captured with `show` to a local file rather
than a tftp job, since this edge has no out-of-band access and a backup is
mandatory before touching it.
irv-ml1 remains relayed and is called out as outstanding.
Picks up uncommitted searxng changes left by another session and makes them
truthful rather than committing them as written. The stack itself verifies
clean: canonical and live are byte-identical for both compose.yaml and
searxng-settings.yml, the container is running with zero restarts, and live
queries return 51-54 results from 5-6 engines with braveapi contributing 20
each time.
compose.yaml gains SEARXNG_BRAVE_API_KEY, which NOTHING READS. It was added on
the belief that settings.yml could pull it via `!ENV SEARXNG_BRAVE_API_KEY`;
this build has no !ENV YAML constructor, so that attempt made the file
unparseable and crash-looped the container ten times with fleet search down.
The comment claiming the variable is "consumed by settings.yml" is replaced
with what is actually true. The variable is kept, unused, in case upstream ever
gains env interpolation — a comment that lies is worse than a variable that
does nothing.
The Tier A playbook is marked superseded FOR THE SETTINGS FILE ONLY, and
scoped deliberately: its v4 design uploads a settings file carrying the !ENV
tag, which would re-break the container, so settings deployment goes through
scripts/deploy-stack.sh like every other stack. Its .env merge and
up-d-not-restart steps remain useful, as do its two warnings recording real
bugs it hit — a wholesale .env overwrite that clobbered SEARXNG_SECRET, and a
sed that inserted literal backslash-n into compose.yaml. An unscoped
"superseded" banner would have buried those; that failure mode cost an outage
earlier today.
Also folds in the regenerated graphify report.
talk shipped a reworked mark at v18 on operator ruling — the 1024x1024
cerulean field rect is gone, the bubble now carries #03adfb where it used to
carry #2e2d30, and the three waveform bars are holes rather than filled
shapes. Path data is byte-identical to the original trace; only the two fills
moved. Fetched from the app and from the booth and confirmed the two sources
are byte-identical before taking either.
tts-dev flagged a real risk with the change: with the field gone the tile
background shows THROUGH the waveform holes, so a tile close to #03adfb would
swallow the bars. Checked rather than assumed. Homepage's card surface is
--sea-20, oklch(0.31 0.022 262) = #2a313c, a dark desaturated navy; the bubble
against it is 5.22:1, well clear of the 3:1 bar for non-text graphics. The
page ground behind it is 6.73:1. Safe on this tile specifically — the earlier
"reads well against the tile background" judgement was about a solid square
and did not carry over on its own.
Also refines the Next.js note in CLAUDE.md, which was over-broad. A NEW file in
the images mount 404s until restart, but REPLACING an existing file's bytes
serves immediately with no restart — measured here, the served hash matched the
new file straight after rsync. It is the route table that freezes at container
start, not the file contents. The previous wording would have had people
bouncing Homepage for every icon tweak.
Brave Search API key wired literally into the settings file and committed.
Operator decision, 2026-09-18, made explicitly: this is a free-tier key on a
rate-limited service of marginal value — "if the service is useless, so is the
key" — so it does not justify the machinery that keeping it out of git would
cost. The key remains in the vault at nh3-docker/searxng-brave-api-key as
well. This is a scoped judgement about one low-value credential and not a
change to the no-secrets-in-git rule for anything else.
⚠ It cannot be un-committed. Rotation means issuing a new key at Brave and
replacing the line; never a history rewrite, since the repo is shared and
other sessions commit to it.
There is no supported alternative in this build. An earlier attempt used
`api_key: !ENV SEARXNG_BRAVE_API_KEY`, which crash-looped the container ten
times with search down fleet-wide: the settings loader has no !ENV YAML
constructor, reads only SEARXNG_SETTINGS_PATH from the environment, and the
entrypoint substitutes only `ultrasecretkey` at template-creation time. The
variable reaches the container and is never read. Literal or nothing.
Key verified against Brave's API directly before wiring, and verified in place
after: three consecutive queries returned 55-63 results from six engines with
braveapi contributing 20 each time, while google cse and marginalia remain
quota-suspended. General web engines are now seven, up from one this morning.
Root cause of the silent-empty-results failure peedlar-dev reported. Of 55
general-category engines, only seven were enabled-by-default, and six of those
are dictionary, translation, currency or encyclopedia engines that return
nothing for an ordinary web query. `google cse` was the instance's ONLY general
web engine, so a single quota exhaustion produced HTTP 200 with an empty
results array and no error, for every consumer on the fleet.
The distinction that matters: `inactive: false` only makes an engine
selectable, `disabled: false` puts it in the default set. The other 48 were
selectable-but-off, which an API client has no way to change.
Enables five keyless engines, each bang-probed first and returning real results
with no API key: duckduckgo web 10, bing 10, yep 20, yahoo 7, wiby 12. General
web engines go 1 -> 6. Deliberately excluded: mojeek, qwant, startpage and the
brave scraper, all of which CAPTCHA or rate-limit this egress, and seznam,
which times out.
Verified under the live failure condition rather than a simulated one. google
cse is still quota-suspended right now, and three consecutive queries returned
38-41 results from 4-5 engines each. The single point of failure is gone while
the failing engine is still failing.
Also adopts the concurrent v4 settings work from the other session — marginalia
on its public key, and the captcha'd-scraper removals — plus the fix for the
crash-loop that work introduced: this build has no !ENV YAML constructor, so
`api_key: !ENV SEARXNG_BRAVE_API_KEY` made the file unparseable and the
container restarted ten times with search down fleet-wide. That block stays
commented; the vaulted Brave key is valid but has no supported path into the
settings file, which is a separate decision.
searxng had been answering from google cse alone for at least a day, with
brave and startpage suspended and duckduckgo returning CAPTCHA. Updating the
image from 2026.9.3+a1144dda3 to 2026.9.18+c0042add3 restored all four
engines immediately, and they held across 11 consecutive queries run after the
change specifically to rule out a freshly-reset circuit breaker flattering the
first measurement.
before searxng/searxng@sha256:3602e6ddbeba037f5d800d1ed9d296a8b93c9f5b3cf9d05fa179d0e766dd59a1
after searxng/searxng@sha256:e0027a772aeeea55bf642256aae6fb3344ffa5f25ca665898c2ea821101334c4
The image stays on :latest rather than being digest-pinned. For this stack that
is deliberate and now demonstrated: upstream ships engine-handler fixes as
providers change their bot gating, so being current is the mitigation, and a
pin would have frozen the breakage in place. The post office is pinned for the
opposite reason — it is the fleet message bus and must not move under us.
README corrected. It had carried two successive wrong diagnoses, both blaming
egress, and now records the real cause plus the two measurements that
falsified them: reverting to direct NH3 egress reproduced the failure exactly,
and a live !ddg probe on a freshly restarted container also CAPTCHA'd, ruling
out a stale suspension timer. Both wrong claims asserted causation from
correlation without a baseline.
The health-script blind spot is unchanged and still called out:
scripts/searxng-health.sh reports the same passing result whether four engines
answer or one.
Measured 3/3 probes: duckduckgo/startpage CAPTCHA, brave rate-ban, wikidata
403 from this egress. Mojeek tried and also 403'd. Notes on keyed-engine
path to restore breadth recorded in the settings header.
Reverts the outgoing.proxies block added in 156e126. Canonical restored from
that commit's parent and verified byte-identical to the host's
searxng-settings.yml.pre-esh-20260917 backup, then deployed via
scripts/deploy-stack.sh so canonical and host converge rather than drift. The
esh-scale searxng-egress.service is stopped and disabled; tailscaled on that
container was not touched.
⚠ THE ROLLBACK DID NOT RESTORE THE ENGINES, WHICH FALSIFIES THE REASON GIVEN
FOR IT. 156e126 recorded that moving egress to ESH had cost three of four
engines. Measured after this revert, with egress confirmed back on
70.230.226.88 and the same instrument used for the before-measurement, the
result is identical: brave and startpage suspended, duckduckgo CAPTCHA, google
cse the only engine answering. Per-engine bang probes confirm duckduckgo is
CAPTCHA-ing the residential address live, so this is not a stale suspension
timer.
The engine failures therefore have some other cause and predate or are
independent of the ESH move. The claim in 156e126 asserted causation from a
correlation without measuring the pre-change state; the only evidence for
"residential egress avoids CAPTCHAs" was a comment dated 2026-09-03, which is
no longer true of this address.
The revert still stands on its own merits: ESH egress bought no measurable
improvement while adding a hard dependency on ESH WAN and mesh availability
for all fleet search, so the simpler configuration is the better one. It is
simply not the fix for the engines.
README rewritten to match: direct NH3 is documented as current, the ESH
attempt is kept as history with its measured outcome, and the health script's
blind spot is called out — scripts/searxng-health.sh prints a passing result
while three engines are blocked, because it gates on "any results returned"
and treats failed engines as informational. That script needs to fail on
blocked engines before any future egress change, or the next regression is
equally invisible.
Regenerated by the commit hook. Corpus has grown from 382 files / ~576k words
at the 2026-09-01 snapshot to 623 files / ~838k words, and the graph from 3906
nodes / 4144 edges to 5468 / 5971. Deterministic tree-sitter extraction only —
zero token cost, 98% EXTRACTED.
Committing work deployed on 2026-09-17 that had been left uncommitted, so
canonical intent stops disagreeing with the running host. The deployed
/opt/docker/conf/searxng/searxng-settings.yml is byte-identical to the
canonical file here, verified before this commit.
Search requests and their DNS now exit via socks5h://10.0.50.65:1080 on
esh-scale (CT 108), an application-level proxy rather than a host-wide exit
node; no route or firewall changes. microsocks runs as nobody under
searxng-egress.service, binds only 10.0.50.65:1080, and bypasses SOCKS auth
for source 10.100.50.40 alone — every other source must supply a password
regenerated at each start and never distributed. Verified active and enabled.
There is deliberately no direct-NH3 fallback: an ESH outage must fail the
search rather than silently revert egress.
⚠ THE CHANGE HAS NOT ACHIEVED ITS PURPOSE AS DEPLOYED. Two independent live
queries, 2026-09-18, both report brave "Suspended: too many requests",
duckduckgo "CAPTCHA" and startpage "Suspended: CAPTCHA", leaving google cse
as the only answering engine. Moving egress off NH3's residential address is
what this change did, and CAPTCHA avoidance was the stated reason searxng sits
at NH3 at all. The README anticipated the risk in its Dependency note; it has
materialised. Rollback procedure is in the README and the pre-change config is
kept on the host as searxng-settings.yml.pre-esh-20260917.
Measured egress also drifted from the value recorded at cutover: the README
notes 154.50.58.126, the proxy now exits 128.177.138.182. Expected — the README
pins no public IP and calls out WAN failover — but recorded here so the number
in the doc is not mistaken for current.
Also retargets seat-inventory.py's default host from the mesh address
100.64.0.7 to fv-ml1's LAN address 10.251.50.54, routed by the site gateway.
Canonical intent still named the 3.6.2 digest while nh3-docker was running
3.6.3, so the next scripts/deploy-stack.sh run against this stack would have
silently rolled the fleet message bus back and taken the hyphenated-search
fix with it. Caught by forseti during independent post-deploy verification.
3.6.3 is the literal-search fallback: a query containing a hyphen was parsed
by FTS5 as a column filter, raised OperationalError, and search() returned []
— indistinguishable from "no results" — so every hyphenated term on this
fleet silently matched nothing. nh3-docker, irv-ml1, esh-docker-vm, tts-dev
and every other hyphenated name were unsearchable.
Deployed digest verified against the running container before this pin:
sha256:978f85533674ee248d6c6f29c54ffab0bc2cb16332c18c9fb8bfda1d566e2de4,
built from git archive of tag v3.6.3 (5de41b7). The image line is the only
difference between canonical and live; the two files are now identical, so a
managed deploy is a no-op rather than a regression.
Codex reads a global AGENTS.md from CODEX_HOME; Grok always scans
~/.grok/rules/ and loads every *.md in it regardless of name. Both were
empty, so AGENT-BOOTSTRAP.md is symlinked into each rather than copied
— one file, three agent families, no drift surface.
The bootstrap is a pointer, not a second index: it names
~/FLEETTOOLS.md, gives the three live-inventory endpoints, and inlines
only the rules that must hold even if the agent never opens anything
else — attribution to Vuong Hoang, no committed secrets, the operator
owns architectural calls, n=1 is not a measurement, and absence of a
signal is not a safe reading of it.
The shared all-agents LiteLLM key was single-copy in ~/.claude/CLAUDE.md
and is now also in the vault at litellm/all-agents-shared-key, per the
standing directive that durable credentials never live in one place.
It stays inline in CLAUDE.md too, since every session needs it and a
vault round-trip measured over two minutes. Namespace is service-scoped
rather than host-prefixed because the key is fleet-wide, matching the
existing att/fortigate/headscale/unifi/worldtree entries.
Vuong Hoang
vh