3eab4da3794fb04c2d7f407381d855369ffdc1f2
114
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 vh
|
30517fd603 |
fix(headscale-ddns): say why it failed, retry the WAN lookup, and track it at all
The failed-START notifier built earlier today had its first REAL firing at
15:28: headscale-ddns.service exited 1 after succeeding all afternoon. The
detection worked. The alarm was also useless, and that is the finding.
Both failure paths exited 1 IN SILENCE, so the message said "exit status 1" and
nothing else. An alarm you cannot act on costs the same triage as no alarm at
all -- the notifier did its job and the subject script had no diagnostics for it
to carry.
Cause was transient and harmless: icanhazip.com did not answer inside its 10s
cap, so the IP came back empty and the regex guard refused it. No DNS impact --
the record already held the right address, verified against 1.1.1.1 before
touching anything, and the next timer run succeeded. Arithmetic confirms it:
~17s vault read + 10s curl timeout = 27s against the 28s the failing run took.
Fixed, both verified by making them fail:
- every exit path names its cause; a missing vault key names the key, an
EMPTY token is distinguished from a failed read, and a dead WAN lookup adds
"DNS left unchanged" because that is the fact the reader needs
- the WAN lookup retries 3x with ANNOUNCED attempts -- one third-party blip
should not page a human, and a silent retry would hide a degrading
dependency
⚠ ALSO: this script was not tracked anywhere. A fix to the thing every mesh
client resolves through lived on exactly one disk. Script, unit and timer are
in the repo now.
Measured and recorded: the vault read is 17 of the script's 18 seconds, every
10 minutes. Not a fault, but it bounds any retry budget and it is fleet-wide --
svos-dev's alarm unit carries the same 17-second note.
|
||
|
vh
|
00a03921ff |
docs(alerts): reset-failed sits on the remediation path, which a flap sampler must design for
svos-dev found an interaction between the two detectors by doing the recovery rather than reasoning about it: clearing a failed unit is REQUIRED before systemd will start it again, and reset-failed zeroes NRestarts in the same motion. Verified here -- a start-limited unit refused to start until reset-failed, which took NRestarts 3 -> 0. So the standard recovery from a failed-START alarm erases the history a flap sampler would depend on, routinely rather than as an edge case. A unit that fails, is cleared, then flaps starts its history from zero, delaying the first flap alarm by the whole threshold -- exactly the window where a just-fixed unit is most likely still sick. Mitigation recorded: store a last-seen timestamp so a counter going BACKWARDS is itself detectable as a reset rather than read as quiet. Also recorded as its own shape: 'nothing to count is not the same as not counting'. The first manual-restart check read NRestarts flat and would have concluded a sampler goes blind whenever a human touches a unit. Artifact of the test -- that unit had already hit its start limit and stopped auto-restarting, so the instrument was reading correctly from a subject that had stopped producing. Verify, then ask what the verification could not have seen. |
||
|
vh
|
163bb97dea |
docs(alerts): the alarm covers far less than I claimed — measured
OnFailure fires when a unit enters failed. A unit that keeps re-earning its retries never enters failed, so the alarm never fires for it. svos-dev warned this was the consequence of the interval divergence; measuring it showed the gap is most of the box. MEASURED: a unit shaped like booth/althing-po-herald (Restart=always, RestartSec=5s, burst 5, interval 10s) ran 9 restarts over 45 seconds cycling active/activating and NEVER reached failed. No alarm would have fired. Covered: the 7 timer-driven oneshots (Restart=no, so any failure lands in failed immediately) and svos.service (burst 3 per 5min -- it genuinely gives up). NOT covered: ten daemons that flap instead, and hermes-gateway, whose start limiting is disabled outright. svos.service's divergent 5min window is deliberate and load-bearing (operator ruling 2026-09-11, 'fatal both ways'). NOT to be harmonised: that would restore the flapping the ruling forbids AND silence the alarm on the one daemon it currently works for. Whether the OTHERS move to 5min is a behaviour change for ten services and an operator call. Also recorded, svos-dev's framing of the two installer bugs: a tool that enumerates 'things that are fine' and acts on them has selected against its own subject. Neither bug failed randomly -- both were anti-correlated with purpose, working better the healthier the fleet looked. |
||
|
vh
|
ef70b2ffa1 |
fix(alerts): the installer skipped failed units — the exact ones it exists for
Two defects, both found by the tool failing to cover a unit that was already
broken. talk.service has been sitting in `failed` (exit 143) while its
containers keep serving 200 on :8092 -- precisely the "unit died, the thing
keeps serving, nobody knows" case this alarm was built for -- and the installer
had not hooked it.
1. SELECTION: --state=running skipped anything not already healthy. A unit that
is down at install time never got hooked, which inverts the tool's purpose.
Now the union of enabled unit-files and all loaded units.
2. PARSING, and this is the sharper one: systemd decorates a FAILED unit with a
leading "●", so `awk '{print $1}'` returned the bullet instead of the name,
and the sanitiser reduced it to an empty string and dropped the row. The
parser silently lost exactly the rows that matter. --plain suppresses the
decoration. Same shape as every other instrument error today -- it reported
cleanly while looking at the wrong thing.
Selection now 23 units. Deliberately INCLUDES the timer-driven oneshots
(dev-backup, ha-backup, fleet-tls-cert-check, headscale-ddns,
seat-inventory-drift, brokkr-landscape-scan, soong-ci-relay): a backup or a
cert check that fails silently is the same class, and all seven were clean at
install time so they are not a known noise source. EXCLUDES dbus, gpg-agent and
dirmngr as OS plumbing, the notifier's own template and instances, and
svos-failed-alarm -- hooking an alarm with an alarm is a loop with extra steps.
|
||
|
vh
|
02a6fc019d |
fix(alerts): suppress duplicate failure alarms, keyed on the cause
svos-dev challenged a claim this README made -- that a crash-loop yields one
message per episode -- with a measurement: one 2026-09-19 boot-gate refusal on
svos.service produced FIVE transitions into failed, and the operator got five
messages.
Measured here before accepting it, because a peer's number is still a number
someone else took: a unit with Restart=on-failure, burst 3, interval 30s
produced 7 journal failure lines and exactly ONE notifier invocation. So the
multiplier is not universal -- it needs retries spanning start-limit windows or
an external restarter. svos.service carries StartLimitIntervalSec=5min, which
is how it accumulated five.
Both conditions exist on this box, so the guard goes in as cheap insurance
rather than as a fix for something proven here. The README now states both
numbers and which restart policy each of the twelve units carries, since that
is what decides the exposure. Noted with it: hermes-gateway has start limiting
DISABLED, so it retries forever and may never reach failed at all -- worth
knowing before trusting this alarm to cover it.
Design, taken from svos-dev's shape:
- Keyed on a hash of the CAUSE (unit + result + exit status + the shape of
its last error lines), never the unit name alone. A genuinely different
failure inside the window is a new fact and must still page; suppressing by
unit would hide a second, worse failure behind the first.
- Suppression is LOGGED to suppressed.log, never silent. An alarm that
quietly declines to fire is indistinguishable from one that is broken.
⚠ The first test of this appeared to show the cooldown not working, and the
test was wrong rather than the code -- it invoked the script BEFORE failing the
unit, so the two calls legitimately saw different states and computed different
fingerprints. Re-run the way systemd actually invokes it: same cause 3x -> 1
sent, 2 suppressed and logged; a different cause inside the same window -> sent.
|
||
|
vh
|
1be39d1ea2 |
feat(alerts): failed-START alarms for every fleet user unit on nh3-dev
svos-dev found the failure neither Beszel nor Uptime Kuma can see, and measured it: a config change on 09-19 made svos.service refuse to boot, the RUNNING process predated the change and kept serving, and the service sat one restart from dark for three days. Every uptime probe was green and correct -- the thing was up. The signal that catches this is failed-START, not down. A count after that conversation: 13 running user units on nh3-dev, ZERO with an OnFailure hook. Including althing-po-herald, whose silent failure cuts infra-ops's own mail delivery -- a blind spot in the notification path every other alarm on this fleet depends on. One template, one drop-in, 12 units hooked (dbus excluded as systemd's own plumbing). Not noise: OnFailure does not fire on a clean restart or a deliberate stop, and with Restart=on-failure a crash-loop yields one message per episode rather than one per attempt. ⚠ %i, NEVER %I -- and the acceptance test is the only reason this is right. %I unescapes the instance name and systemd escaping maps "-" to "/", so the first run delivered a message for "onfailure/selftest.service", a unit that does not exist, with a spool path that tried to create directories. althing-po-herald.service would have arrived as althing/po/herald.service. It "worked" -- mail was delivered -- which is exactly the kind of success that is not one. The referring unit passes %n raw, so literal %i is correct. ⚠ The notifier never reports itself, guarded twice on purpose: the template carries no OnFailure, and the script bails on its own instance name. A notification loop is the one bug that pages you forever. Delivery is spool-first. postbox has no outbox, and an alarm for moments nobody is watching must survive the post office being one of the things that is down. If the herald itself fails, the message still REACHES the post office (postbox talks to it directly; the herald only delivers inbound pokes) -- not pushed, but stored for the next read. Acceptance-tested twice against a unit that exits 42: once to catch the %I bug, once to confirm the fix. Test unit removed. |
||
|
vh
|
38bb20ceda |
fix(irv-ml1): tailscaled could never add its IPv6 mesh address, and the README described a topology two cutovers old
TAILSCALE IPv6. `tailscale status` had been reporting, continuously:
2 add route failures; first was: permission denied
adding address fd7a:115c:a1e0::6/128 from tunnel interface: permission denied
with tailscale0 carrying only 100.64.0.6/32 while headscale had assigned it
an IPv6 address it could not use.
Not a capability problem -- tailscaled runs as root with the full bounding
set. /etc/sysctl.conf:59 sets net.ipv6.conf.default.disable_ipv6=1, and
`default` is inherited by NEWLY CREATED interfaces; tailscale0 is created at
daemon start, inherits it, and the kernel returns EPERM for every attempt.
Fixed with a scoped systemd drop-in rather than flipping the global default.
That line carries no comment, but IPv6-off-by-default on a host with ~26
docker bridges reads as deliberate, and changing it would hand IPv6 to every
future bridge as a side effect of fixing Tailscale.
⚠ It must be ExecStartPost, not /etc/sysctl.d. A sysctl.d entry for a
per-interface key is applied at boot, BEFORE tailscale0 exists, and is
silently ignored -- the setting would look present and do nothing.
Also learned: setting the sysctl on the LIVE interface is not enough.
tailscaled only attempts the address at startup or on a netmap change, so
the verify failed for 60s until the daemon was restarted. Restart is part
of the operation, not an afterthought.
Verified: fd7a:115c:a1e0::6/128 present on tailscale0, health clean, mesh
and services (arbo, ytvc) up.
README. It documented the pre-headscale topology as current -- "Reachable
IP: 10.100.79.3 (WireGuard tunnel endpoint)", "No direct LAN access", and a
refresh caveat telling you to bring WG up. That sends anyone triaging this
host to the wrong layer, which is the exact tax the file exists to prevent.
Now: mesh primary at 100.64.0.6, LAN 10.6.110.50, and wg0 documented as
STILL UP with a live peer -- tailscale uses that address as its direct
endpoint, so it is load-bearing, not vestigial.
Recorded with it, because these cost hours tonight and will cost them again:
- Irvine is a TENANCY behind a Fortinet PFI does not control. Its TLS
inspection breaks Tailscale's relay and control channels (41 cert
warnings/week, 4 control-plane episodes in 14 days). Usually invisible
because direct peer paths carry the data. No fix on our side.
- Diagnose reachability with `tailscale ping`, NOT the status output:
headscale said "online" and status said "active, 19.7 GB" while nothing
on the host answered. Both are last-known state; only a round trip is
liveness.
- The ~26 docker bridges make tailscaled report captive portals.
Two stale claims corrected: the hostname rename it called "pending" is done,
and `ollama` is listed as running on :11434 when it is gone -- verified, no
unit file, nothing listening, no process. It is banned fleet-wide.
|
||
|
vh
|
6f0a9b9fae |
feat(alerts): generalize the althing bridge, wire Kuma to it, retire chamber
THE BRIDGE. `beszel-althing` hardcoded a "[Beszel] " subject prefix and a
Beszel hub footer from when Beszel was its only caller. Routing Uptime Kuma
through it unchanged would have delivered Kuma outages labelled [Beszel],
pointing the reader at the wrong dashboard -- an alert that lies about its own
source is worse than no alert.
Now a route registry: /beszel and /kuma, each with its own prefix, footer and
payload parser, because the tools do not agree on a shape (Beszel sends
{title, message}; Kuma sends {heartbeat, monitor, msg}). Generalising cost a
dict; a sibling service would have cost a second unit, a second port and a
second thing to notice had died.
Renamed beszel-althing -> althing-alert-bridge with it. A service named after
one consumer that carries two is the invisible coupling that sends a future
session looking in the wrong place.
⚠ /beszel IS FROZEN and this refactor proves it rather than claiming it. The
three original tests were kept BYTE-UNCHANGED -- including the one asserting
the exact postbox argv -- and deliver() still defaults to the Beszel route so
they exercise it. A new test asserts the Kuma footer never leaks into a Beszel
body or vice versa. Verified live after the rename: a real POST to /beszel
landed as "[Beszel] BRIDGE RENAME CHECK" with the correct hub footer, read back
from the thread rather than trusted from the receipt.
Payload shapes are parsed HERE, not via Kuma's custom-webhook-body feature,
because Kuma's notification config lives in its own database -- and that
database was destroyed and rebuilt from scratch hours ago. Anything living only
in a tool's DB is lost on the next rebuild; format knowledge belongs in git,
next to a test.
parse_kuma also handles the monitorless case. testNotification and cert-expiry
alerts carry no monitor and no heartbeat, and the first cut fabricated "unknown
monitor is ?" from them -- caught by sending a real one and reading the subject,
not by the suite. Fixed, pinned, and the earlier test asserting the bad
behaviour was corrected rather than worked around.
KUMA IS NOW WIRED. scripts/kuma gained notification support and the channel is
in monitors.yaml, seeded BEFORE the monitors and with applyExisting, so a
rebuild restores alerting and not just detection. Ground truth from the DB:
13 of 13 monitors carry the channel.
⚠ A THIRD instance of the same class of bug, worth naming: notifications() is
pushed as `notificationList` at LOGIN ONLY -- there is no event to ask with. The
first cut cleared the captured value before waiting, discarding the only copy it
would ever be sent, then blocked for the full timeout and reported an empty
list. That reads exactly like "no channels configured" and is a lie. Same family
as the getMonitorList ack-vs-push trap, different shape.
End-to-end, both shapes, read back from the inbox:
[Uptime Kuma] Homepage is DOWN + target + board link
[Uptime Kuma] althing (infra-ops) Testing (no fabricated subject)
[Beszel] BRIDGE RENAME CHECK + hub footer, unchanged
ALTHING CHAMBER RETIRED (operator). Three of its four containers had never
started -- created 2026-09-19, StartedAt epoch-zero, 0 restarts -- so :7881
refused, and nothing was watching it. Only its valkey was running, on the
project's own network with no external consumer. Stack, compose/build/conf dirs
and the local image removed; the Homepage card went with the label.
|
||
|
vh
|
14bd95d76d |
chore: extract the Booth to its own repo (vh/booth)
The Booth is now one of the most-used fleet tools -- 17 agent handles post to it daily -- and it is taking an information-architecture rework plus a cross-agent SVOS design retrofit from design-dev. That work wants its own ROADMAP, contracts and blast radius, not the fleet-infrastructure repo's. All 29 commits moved with it via `git subtree split`; the history carries real lessons (two shipped-dead controls, the verbatim-injection traps) that a squashed import would have thrown away. Live service repointed and verified: the user unit and the ~/.local/bin/booth symlink now resolve into ~/development/booth, healthz answers, all 24 booths intact. services/booth/ keeps a pointer README, same shape as the chatterbox-fast and tts-stack extractions. |
||
|
vh
|
2e08edcaab |
fix(draupnir-slicer): warn that every invocation writes result.json to CWD
brokkr-smithy-dev exercised the slicer on real geometry and found OrcaSlicer writes result.json into the working directory on --info. I verified it here and it is worse: --help does it too, which is as read-only as an invocation gets. An empty directory gained a result.json from --help alone. Encoded in the wrapper and the Dockerfile rather than left to be rediscovered, and the wrapper is now committed beside the image that backs it instead of living only in a home directory on one host. ⚠ The reason they found it and I did not is worth keeping in the header: I verified --help and called the slicer done. Usage text proves the binary starts, not that it ingests our geometry or that it behaves while doing so. They ran both controls through to STL and got manifold=yes, 520 facets, return_code 0 on each -- which is the verification I should have run before reporting the provisioning complete. |
||
|
vh
|
e574b91ff3 |
feat(irv-ml1): containerise OrcaSlicer for Draupnir's sliceability check
Not tidiness — necessity, established by ldd rather than assumed. Every OrcaSlicer release ships only an Ubuntu 24.04 AppImage, which needs GLIBC_2.38 and GLIBCXX_3.4.32. irv-ml1 is Debian 12 on glibc 2.36. That gap is not installable without moving the host to Debian 13, which is not a thing to do to a box running twelve GPU and audio services in order to slice a plate. Reaching back to an Orca built for Ubuntu 22.04 would run, and would mean pinning permanently to a stale build -- the exact trap brokkr-smithy-dev rejected when they chose Orca over an old PrusaSlicer tag. The container keeps the slicer current AND leaves the host alone, which is the same argument that made FreeCAD an AppImage, taken one step further because the host could not satisfy this one at all. The AppImage is extracted at BUILD time; --appimage-extract-and-run re-extracts to /tmp on every invocation, which is wasted seconds and wasted disk per slice. ~/bin/orca-slice wraps it so the dispatcher calls a binary and the artifact root is mounted at the same path inside and out. ⚠ Recorded honestly: I installed libwebkit2gtk-4.1-0 and 8 dependencies on the host chasing errors serially before enumerating with ldd, and only then found the glibc wall behind them. Those nine packages are unnecessary. Left in place rather than autoremoved on a box running production services; flagged for the operator. |
||
|
vh
|
07c9cb293b |
fix(booth): release was painting over the delete ×, which was unclickable
Operator: "release button covers delete button". Measured before touching
anything: release 58x24 at (323,266), × 30x30 at (349,268) — 30x22 px of
overlap on a 30px button, and `elementFromPoint` at the ×'s centre returned the
release form. The × I added yesterday was 100% unclickable from the moment it
shipped.
Cause: both were `position:absolute` on the same corner, each with its own
guessed offset, and `release` is the later sibling so it won. Replaced with one
flex row positioned once — release left, × right at the card corner where the
ephemeral lane's × already lives, so muscle memory transfers and neither can
drift back on top of the other when a label changes width.
Verified by measurement, not inspection: overlap 0 px, and clicks at each
control's centre now land on that control. The ephemeral lane's × and ★ were
re-checked and are unaffected.
ADDS scripts/layout-probe.py, because markup inspection STRUCTURALLY cannot see
this and I have now shipped two dead controls in two days by reading templates
instead of rendering them. It asks a real browser what a click at each
control's centre would hit.
It took four iterations to become trustworthy and the failures are the point:
1. `top.contains(el)` counted an ANCESTOR overlay as a hit -- the exact case
it exists to catch. Version one reported OK for a real overlay.
2. elementFromPoint is viewport-relative, so everything below the fold read
as occluded and buried the real findings.
3. getBoundingClientRect() on a WRAPPED INLINE element is the union of its
line boxes, whose centre lands in the gutter between them -- three zip
links reported occluded by their own parent. Real geometry, wrong question.
Only the fourth version fires on a genuine overlay while staying silent on the
clean page. Both controls were run; a probe never seen to fail proves nothing.
173 tests pass.
|
||
|
vh
|
751eecb771 |
fix(booth): the reveal button was inert; add kept-lane wipe and in-booth keep
Three operator reports, one of them a real defect I had claimed was working. THE REVEAL BUTTON DID NOTHING, for a day. Its handler sat after the content block's closing tag, and a child template's out-of-block content is silently DISCARDED by Jinja. The button rendered. The handler never reached the browser. Two commits and a README paragraph said click-to-reveal worked, and the suite passed the entire time because nothing asserted against the SERVED page -- the template really did contain the code. Two guards, both confirmed to FAIL when the defect is reintroduced rather than merely added and assumed protective: * test_reveal_handler_actually_reaches_the_served_page greps the response * test_no_orphaned_markup_after_the_content_block guards the structure While moving it, caught a second instance of the same class: the explanatory comment I wrote for the fix contained a literal Jinja endblock tag, which Jinja would have parsed as a real tag and used to close the block early. KEPT-LANE ×. Wiping a kept booth required release-then-find-it-in-the-other- lane. That protected nothing and cost a hunt -- the board you just released is loose in a feed that turns over, and you have to go find it to finish a job you had already decided on. Protection now lives in the confirmation, which names the booth and says KEPT, instead of in the number of lanes you must traverse. Release stays as the reversible option. IN-BOOTH KEEP. `☆ keep` / `★ kept — release` beside "Wipe now", so promoting does not mean navigating back to the index. The booth page did not previously know its own kept state; it does now. Both post a `next` field to stay put -- and `next` is a form field, so it is attacker-controlled: only same-site absolute paths are honoured, with `//host`, schemes and backslashes refused, tested. 173 tests pass. |
||
|
vh
|
271cb11b70 |
fix(booth): put the blur toggle on every item kind, and make it look like a control
The operator asked "no UI option to blur/unblur?" and was right twice over. MISSING ENTIRELY ON TWO OF THREE BRANCHES. booth.html renders docs, media and everything-else through separate <figure> blocks. The toggle went into the media branch only, so inline docs -- the branch that puts readable text straight on the page, the one that needs blur most -- had no control at all, and `other` files only got a caption row if they happened to carry a caption. This is the SECOND time this feature shipped having patched some branches and not others; the blurred class itself had the same gap one commit ago. So the toggle is now a single Jinja macro called from all three sites, which makes "patched two of three" impossible rather than merely unlikely, and test_every_item_kind_gets_exactly_one_blur_toggle counts toggles against figures across mixed kinds so a fourth branch cannot quietly skip it. INVISIBLE WHERE IT DID RENDER. v1 was a bare `◌` at 0.78rem in --muted with no border, no label and no hover affordance. It now reads `◌ blur` / `◉ blurred` with a border, matching the other per-item controls. A control nobody can find is a control that is not there. Docs get it in the doc bar beside ⤢ ⬇ ✕, with stopPropagation so submitting it does not collapse the <details> it lives inside. Verified live on all three kinds: 3 figures, 3 toggles, and the POST round trip blurs and un-blurs. 167 tests pass. |
||
|
vh
|
2e7fd7157e |
feat(booth): close the keep round trip, and add cosmetic per-item blur
Two operator requests.
KEEP, BOTH DIRECTIONS. The kept lane could already release a booth back to
ephemeral, but an ephemeral booth could only be promoted from a shell -- so the
round trip was closed only if you had ssh. The /keep route and the `booth keep`
verb both already existed; only the button was missing. Adds ★ to the ephemeral
card, mirroring × on the other shoulder.
BLUR. Per-item cosmetic censoring: `booth blur <name> <file>...`, a ◌/◉ toggle
in each caption row, and 👁 click-to-reveal. State is `.blurred` in the booth
dir, one booth-relative path per line -- the same filesystem-is-the-state idiom
as .pins and .forever. An empty set deletes the marker rather than leaving a
zero-byte file, so `ls -a` tells the truth.
⚠ BLUR IS NOT ACCESS CONTROL, and the code, the docs and a test all say so on
purpose. A blurred item is still served at its own URL, still in the zip, still
on disk. The Booth has no auth by design. test_blur_is_cosmetic_the_file_is_
still_served asserts the 200 deliberately: if someone later "hardens" this into
a 403 that test fails, and it should, because half-implemented access control is
more dangerous than none.
Reveal is per-viewer and never persisted; a reload re-hides. With JS off an item
stays blurred, which is the safe direction to fail in.
Two things the first pass got wrong, both caught by checking rather than
assuming:
* The cover thumb. index.html has IDENTICAL markup in the kept and ephemeral
lanes, so a single-occurrence replace patched only the kept one and the
ephemeral front page happily displayed the thing someone had hidden. The
test that caught it was itself wrong first -- it matched the bare string
"blurred-thumb", which is in base.html's stylesheet on every page and so
passed in both states. It now asserts the attribute.
* Inline docs render through their own <figure> branch and were left
unblurred -- the branch that puts readable text straight on the page, so it
needed blur more than images do. The suite passed; a live curl caught it.
165 tests pass (154 pre-existing, unchanged).
|
||
 Vuong Hoang
|
cf9d167453 | grok-token-broker: shelved — operator kept the jail, so the renewal feature has no consumer | ||
|
Vuong Hoang
|
b907a0e46a | grok-token-broker: the probe's blast radius is BOTH Groa transports, not one | ||
|
Vuong Hoang
|
ebc4dac6d8 | grok-token-broker: hold a refreshable session credential behind a rotation-safety gate | ||
|
vh
|
0193b31aad |
fix(secrets-broker): bw is not concurrency-safe — serialise, and never return an empty secret with exit 0
Reported by svos-dev after parallelising four vault reads in SVOS's systemd wrapper. Reproduced here and it is worse than reported: four concurrent secret get calls for distinct items returned empty strings with exit code 0, zero of four succeeding against their one of four. No error, no timeout, no diagnostic. The shape is the problem, not the race. A caller treating an empty optional secret as 'not configured' degrades silently and never learns otherwise - it cost SVOS the ability to page the operator while the process logged a clean startup line. Root cause is session establishment, not item reads. Every invocation runs bw unlock, and concurrent unlocks against the shared appdata dir invalidate each other. The damage then surfaces downstream as an empty listing or an empty item body, which is why a per-call lock is useless: by the time the read runs the session it holds is already dead. So the lock wraps the whole command instead. Three changes. The command-level lock makes concurrent callers queue. cmd_get now refuses an empty value rather than printing it, since a stored secret is never legitimately zero-length. And find() no longer coerces empty stdout to '[]' - that turned a broken read into a confident 'no such secret', the same silent-wrong-answer shape one layer up. Verified: four parallel reads of four real items now return all four correctly, serialised at the honest ~17s each. A name that genuinely does not exist still fails loudly, so the guard did not simply mute the negative case. Also replaces the copy at ~/.local/bin/secret with a symlink to this file. It was a plain copy in sync by luck, and every edit here silently left the live tool behind. |
||
|
vh
|
a91b841d86 |
feat(fv-ml1): generate the seat inventory from the live box instead of maintaining it by hand
The seat documentation must stay current, and a hand-written document cannot. The LiteLLM config described char-rp as a 31B model on a host and GPU it had not been on since 2026-08-24 -- three weeks of silent drift in a file that read as authoritative, and the reason a seat spent that period serving a model nobody intended. Anything typed here drifts the same way; anything read off the running containers cannot. scripts/seat-inventory.py derives the whole document from the host: - placement and VRAM from nvidia-smi compute-apps, mapped to containers through /proc/<pid>/cgroup -- nvidia-smi reports the vLLM engine child while docker reports the container pid, so matching them directly silently yields nothing - weights and KV tokens parsed from each engine's own startup log, not derived arithmetically, with concurrency computed as KV tokens over context - architecture, layer and expert counts, and the exact quantization group scheme (W4A4 vs W4A16 distinguished) from each model's config.json - speculative-decoding method and k from the container argv, which is how the three incompatible methods on this box became visible - lineage from the .PROVENANCE.txt SIBLING files -- they sit beside the model directory, not inside it, which is why an earlier pass wrongly reported two fully-documented seats as having no provenance - gateway aliases resolved from the LiteLLM config on ana-docker --check compares the committed document against the live box and exits non-zero when they diverge, ignoring only the generation timestamp. Suitable for CI or a scheduled drift alarm; read-only throughout, safe against production. Also commits the KV_CACHE_BYTES override added to the MTP campaign runner, which asserts the flag exists in the derived argv and aborts rather than running a campaign that silently ignored it. |
||
|
vh
|
f9a8f176ca |
fix(mtp-bench): give each arm its own container name instead of tuning teardown waits
Container teardown latency here is unpredictable, not merely slow. Measured 2026-09-13 on the same container in the same session: once ~55s, once 0s. Removing a container holding ~92 GB of GPU memory plus the offloaded PLE mapping leaves an Exited record owning its name for that whole window, so any fixed wait or retry budget is a guess that will eventually be wrong -- a 12x5s retry lost arm k1 by roughly two attempts. Arms now use $NAME_BASE-$arm, set in boot() after stop_bench has torn down the previous arm. Names are never reused, so the collision is impossible by construction and teardown latency stops being load-bearing. cleanup() reaps every fn-mtp-bench-* container at exit. stop_bench keeps the GPU-memory wait -- the next container genuinely needs the device, and unlike the container record nvidia-smi clears promptly and reports truthfully. It no longer waits on the container listing at all. Three earlier hypotheses were wrong and are recorded so they are not retried: the name is not held by a phantom (the container is present and Exited), the removal does not fail (rm -f succeeds; it is asynchronous), and GPU memory release does not track name release (memory frees first, by a wide margin). |
||
|
vh
|
aa5ee7e0a6 |
fix(mtp-bench): docker ps -a is the wrong probe for name release; wait on GPU memory and retry the conflict
The previous fix polled `docker ps -a` until the bench container's name disappeared. That probe is useless here and the fix was a no-op: measured 2026-09-13, the container stops being listed while the daemon still holds the name reservation, so the poll returns false early and `docker run` hits a Conflict naming a container ID that `docker inspect` already reports as 'no such object'. Arm k1 was lost twice this way. Two changes, each aimed at something actually observed: - stop_bench waits on GPU memory falling below 1000 MiB rather than on the container listing. That is the resource the next container genuinely needs, and nvidia-smi reports it truthfully. - boot() retries `docker run` while stderr matches 'already in use', up to 12 attempts at 5s. The daemon's name lag is transient, and forfeiting an arm over it is silent data loss -- run_arm turns a failed boot into a skipped arm that still lets the campaign report success. Both probes verified against real artifacts before deploy: grep -c 'already in use' on the captured k1.runerr returns 1, and the nvidia-smi query returns a bare integer that compares correctly under [ -gt ]. The earlier fix passed bash -n and was still inert, so syntax is not the check that matters here. |
||
|
vh
|
67a7dc539e |
fix(mtp-bench): stop_bench raced docker name release, silently dropping arms
`docker rm -f` returns once removal is INITIATED, not once complete. The
bench container holds ~92 GB of GPU memory plus the offloaded PLE mapping, so
the name stays reserved for several seconds while the device is released. The
next `docker run --name` then dies with a name Conflict.
That failure was near-invisible: run_arm treats a failed boot as
`arm SKIPPED`, the campaign continues, and it still prints CAMPAIGN DONE.
Observed 2026-09-13 -- arm k1 lost the race after off_A's loaded container,
while k2 won it only because k1 had never started a container to tear down.
Every arm that follows one which actually ran is exposed, so k3 and off_B were
both on track to vanish from a run that would have reported success.
stop_bench now polls until the name is released (120s ceiling, warns and
continues). Added a completeness gate: the campaign asserts every expected
res-<arm>-rep<n>.json exists and reports CAMPAIGN INCOMPLETE naming each
missing result rather than DONE, so a gutted run cannot look like a clean one.
Verified the docker ps format string empirically -- an earlier draft nested
quotes so the template rendered as '{{.Names}}' with literal quotes, which
grep -qx could never match, making the wait a no-op that passed bash -n.
|
||
|
vh
|
0a9cf58e19 |
fix(mtp-bench): peak-power summary mis-keyed GPU 0 by timestamp
power.log lines begin '<HH:MM:SS> 0, <W> W, ...', so splitting the first pipe-field on ', ' yields a[1] = '20:18:00 0' rather than '0'. GPU 0 was therefore keyed by sample timestamp, emitting one bogus row per sample and no recoverable peak, while GPUs 1-3 aggregated correctly. Take the GPU index as the last whitespace-separated token of a[1]. Verified against a two-sample fixture: the old parser emits a spurious row per timestamp, the patched one reports 0/1/2/3 once each at the correct peaks. The end-of-campaign summary is the GPU-side input to the fv-ml1 circuit budget, so a silently wrong GPU 0 row is a measurement fault, not cosmetic. The in-flight campaign still runs the old on-host copy (editing a running bash script corrupts execution by byte offset); its summary will be recomputed from the raw power.log. |
||
|
vh
|
524da702ef |
flash-next-mtp-bench: run on GPU 3 / :8023 without touching the production seat
Operator-directed so the campaign could run in parallel with live gen-large testing. Strips the production-restore tail entirely -- the only container the driver can create or remove is fn-mtp-bench, and the sole remaining compose call is a read-only config --format json to derive argv. Adds per-card power+memory sampling every 10 s to power.log, because two cards under load was the risk being accepted and a record beats an argument. That power log is now the only surviving measurement of what the load drew before Fountain Valley went dark; it lives on /tank rather than in the container. |
||
|
vh
|
7e62a07341 |
flash-next-seat: full 262K context, KV pinned at a measured 14 GiB, gen-large on the gateway
Operator-directed: raise context to the model's native maximum and take as much KV as the card safely allows, and expose the seat through LiteLLM as `gen-large`. max_model_len 131,072 -> 262,144 KV cache 8.76 -> 14.00 GiB (332,721 -> 560,654 tokens) concurrency 2.54x@128K -> 2.14x@262K ⚠ 16.00 GiB WAS TRIED FIRST AND IS TOO AGGRESSIVE. A 155,497-token non-repeating prefill drove GPU 2 to 97,074 of 97,887 MiB and the caching allocator logged "OOM on device 0 while trying to allocate 488636416 bytes (free: 422117376)" -- 466 MiB wanted against 403 MiB free. The request completed, so nothing failed visibly; that is one step before the shape that crashed stacks/mog-sec twice on 2026-09-10 (~1.04 GiB wanted, ~600 MB free). Backed off to 14.00 GiB, which re-probes clean: zero allocator warnings, a 155,557-token prefill in 14.2 s, and 2,085 MiB still free at peak. The reason the first estimate was wrong is worth keeping, because it is not obvious and it inverts the usual advice: --kv-cache-memory makes vLLM SKIP MEMORY PROFILING ENTIRELY and ignore --gpu-memory-utilization. The profiler was the thing accounting for deep-prefill activation, so pinning bytes switched off the protection that the pin was supposed to formalise. vLLM's own "--kv-cache-memory=18745235968 (17.46 GiB) to fully utilize gpu memory" line is computed from a profile measured at max-num-batched-tokens depth and sits 3.5 GiB above what a 150K-token request survives; open #54764 compounds it, since PLE short-conv prefill pads every request in a batch to the batch-MAX query length. max-num-batched-tokens stays at 8192 -- it is what bounds the activation peak, and doubling max_model_len left the profiled peak unchanged at 1.65 GiB precisely because the peak tracks chunk size, not context length. Gateway: `gen-large` added to the LiteLLM model_list, pointing at fv-ml1:8022. One alias on purpose -- a single alias cannot trip the shared-config enable_thinking mutation footgun, which needs two over the same (model, api_base). Sampling is the checkpoint's own declared set (temp 1.0 / top_p 0.95 / top_k 20); presence_penalty, min_p and repetition_penalty are left unset because the checkpoint declares no canonical value for them. Verified registered for both the infra-ops admin key and the shared all-agents key, since a new model behind a scoped allowlist 403s silently. Also adds services/flash-next-mtp-bench/ -- the MTP measurement campaign and its rationale. MTP stays off, but on "not yet measured here" rather than on vLLM's 4xH100 recipe number, which is a cross-harness comparison and not evidence about a TP=1 Blackwell seat. |
||
|
vh
|
b8435ceb6f |
Set mog-sec's context to a measured ceiling so it refuses instead of dying
Operator: "loading up the context killed sec again." That reproducer is what finally made the failure legible, and it showed the previous four fixes had all been aimed at the wrong quantity. What the KV pool can hold and what the card can process at depth are different numbers. Cutting context 420k to 384k to 320k, pinning the KV in bytes, and dropping the prefill chunk from 16384 to 4096 all sized the pool. The crashes were governed by the transient needed to process a prefill chunk against a quarter million tokens of resident KV, which scales with depth and not with pool size. Each change helped and none fixed it. Bisected against the real reproducer, with a non-repeating prompt because prefix caching would let a repeated one hash to cached blocks and never prefill deep: 113,247 prompt tokens SURVIVED (27 s) 200,088 prompt tokens SURVIVED (174 s) ~285,000 prompt tokens ENGINE DIED, HTTP 500, container restarted The sustainable ceiling therefore sits between 200k and 285k with gen idle, and gen shares the card with its load uncontrolled, so 163,840 takes about 20% margin under the proven-good depth rather than sitting at the measured edge. The ceiling's purpose is the refusal. Verified after the change: a 149,073-token request serves in 41 s, and requests at both 200k and the ~285k depth that killed the engine now return a clean 400 naming the limit in under a second with the seat untouched. A seat that refuses what it cannot serve is strictly better than one that dies trying. Concurrency went 1.03x to 2.09x. The compose header's "served at native 262K" was never actually deliverable on a shared card; it had simply not been exercised at depth until today. The probe is committed rather than described, so the ceiling can be re-measured when the card's tenancy changes. |
||
|
vh
|
9a916a759f |
Swap the MeroMero A4B onto the erp-seat seat as char-rp-fast, retire the Pfish-6 alias
Operator: "replace that a4b moe over pfish-6 -- remove the pfish-6 alias and create an alias for char-rp-fast." G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16 is live on ana-ml2 :8021 under its own served name, behind the new gateway alias char-rp-fast. Pfish-6 is gone from the gateway and now returns an explicit 400 rather than a substitution; 0 of 17 LiteLLM keys scoped it, so nothing was orphaned. The compose project name stays erp-seat because asset-engine derives seat liveness from it. The first quant of that A4B served NaN and passed its healthcheck doing it. It was built with the dense v2-31B recipe, whose ignore list has no router regex, so all 30 MoE routers were quantized to 4 bits -- and a 4-bit router changes which experts run rather than degrading them. Quant rc=0, healthcheck green, correct KV pool, correct served name, and every completion returned finish_reason=length with the full token count and content: null. The model was emitting a full budget of tokens that decoded to the empty string. Raw /v1/completions was empty too, ruling out the chat template and the reasoning parser. The signal that named it was logprobs: vLLM refused to serialize the response, "Out of range float values are not JSON compliant: nan". The lesson is about the control rather than the router. That tree had already been structurally diffed and passed -- against a verified-good DENSE quant of the same Gemma-4 family. A dense model has no routers, so the single thing that was wrong was the single thing the control could not distinguish. Diffing instead against Pfish-6, a known-good quant of the same 26B-A4B MoE, gave it in one line: 222 ignore entries against 252, the 30 missing being layers.N.router.proj. A positive control is only worth what it can distinguish, and "same family" is not "same architecture class". Re-quantized with the MoE recipe, whose dry-run asserts 11,520 expert Linears and refuses a router in the quantize set before any GPU time. The live seat then passed prose with no channel-prefix leak, a solid-colour image read correctly, an auto tool call parsed, finite logprobs, and KV 534,649 tokens / 2.04x carried over from Pfish-6 unchanged. The broken tree is parked on ana-ml2 as ...-NVFP4A16.BROKEN-routers-quantized-20260910. Section 4.4's temp port was not reachable: 15.9 GiB of weights plus KV plus multimodal encoder-cache profiling does not fit in the ~19 GiB free beside GPU1's six other tenants -- 0.20 utilization refused admission, 0.185 OOM'd in encoder profiling. The substitute was reversibility and ordering: named .env backup, prove the seat on its real port while no alias points at it, move the alias last. That is why a NaN-serving seat never reached a consumer. The seat was down about 16 minutes across two attempts; no consumer saw a broken alias. Playbook gains the router-quant failure signature and the control-class rule in 3.15, and a logprobs check in 4.4. seat_verify.py carries that check as check 6. Quality is NOT established: no RP eval, no long-context check, no A/B against Pfish-6 or char-rp. Samplers are the author's card values, untuned here. |
||
|
vh
|
1a5bc2ddf1 |
Land the MeroMero v2-31B NVFP4A16 quant and record the pinned-transformers trap
The v2 dense quant had failed four times. Attempt 5 lands it at 19 G. The blocker was not what it looked like. `AmbiguousGlobalPerLayerAttributeError` on `head_dim` read as a malformed upload -- DogOnKeyboard's config carries a `per_layer_config` key zerofata's canonical one lacks -- and the standing fix was to force `allow_global_per_layer_attribute_access=True`. Both halves were wrong. `pip install llmcompressor==0.13.0` downgrades transformers 5.16.1 -> 5.14.1. The config was serialized by 5.16.1, which materializes `per_layer_config` from `global_head_dim` + `layer_types`; 5.14.1 carries the heterogeneity guard but not the gemma4 resolver. Under the image's own transformers the same config loads fine. `:latest` was also re-pulled during attempt 4 and no earlier run, so the toolchain moved mid-diagnosis. Two things separated "malformed upload" from "moved toolchain": reproducing the real failing call (a bare AutoConfig load does not reproduce it; the trigger is reached through AutoTokenizer) and keeping zerofata's canonical tree, quantized cleanly on 2026-08-21, as a positive control. The fix drops `per_layer_config` rather than forcing global access. It is exactly redundant -- keys are precisely the ten full_attention layer indices, sole value (512, 4), verbatim the global fields -- and forcing instead would make `config.head_dim` answer 256 to the callers building the 512-wide layers. patch_perlayer.py re-proves that redundancy at apply time and refuses if it ever stops holding. Verified on the tensor table rather than the exit code: the output is identical family-for-family and count-for-count to the August canonical quant, with 356 BF16 vision-tower tensors preserved and input_activations=None. A GPU-free load leaves 0 tensors on meta and generates coherent prose. The section 4.4 serve test has NOT run -- GPU1 has 19.9 GB free against 19.5 GB of weights, so it needs a live seat displaced. Also fixes the A4B output, which had a truncation cap baked into its tokenizer (max_length 8192) from being quantized with the calibration corpus. Playbook gains section 3.17 for the pinned-transformers class and sharpens 3.16 to say drop the dataset outright for any A16 scheme. |
||
|
vh
|
eb75713c1b | Wire Beszel fleet filesystems, GPU telemetry, dashboard and alerts | ||
|
vh
|
6972e7ef7f |
feat(erp-seat): run 7 quantized to NVFP4A16 and serving as the trial seat on ana-ml2
- services/erp-seat-quant/run_quant_erp_v7.sh: v6 runner retargeted; dry-run gate passed identically (11,725 targets, 11,520 experts = 30x128x3, routers+vision BF16) - 49 GiB bf16 relayed gx10 -> ana-ml2 (no key path either way; nh3-dev relays), checksums verified against source; quant 49 -> 16 GiB, all post-steps clean - stacks/erp-seat: .env-driven swap to erp-tune-v7-nvfp4a16, served under its TRUE name; homepage labels + README updated, v6 rollback path recorded - stacks/litellm: trial -> erp-tune-v7-nvfp4a16 (config-file alias; /model/update refuses a config model, so this is an edit + restart) |
||
|
vh
|
c335c38c19 |
fix(booth): a partial ask answer is recorded, not refused
Operator: the form failed when a question was left blank. Refusing the whole submission over one blank threw away the picks that were made, and the HTML `required` on the radios blocked it in the browser before the server saw it. - answered questions recorded; blank ones land in `unanswered`; `complete` says whether the set is finished; a blank question carrying a note keeps the note - `required` dropped from both templates so the browser cannot block a partial - refused only when there is no pick anywhere AND no notes (a 400 — that would flip an open ask to answered with no decision recorded); a choice outside the option list is still an error - new ◐ partial state with an n/N count; skipped questions render as skipped - README + global CLAUDE.md tell reading sessions to check `complete` - 154 tests; v0.1.15 |
||
|
vh
|
784c555dbd | docs(booth): record WHEN inline ask placement earns its markup — the artifact-in-working-memory test, not just the placement rule (framing from tts-dev) | ||
|
vh
|
6e0b85ba27 | docs: refresh what today's work made stale — booth asks (inline placement promoted to its own section), ana-ml2 nvme7 settled by the scrub result, nh3-dev booth entry + the CLI-on-PATH fix, run-07 runbook outcome + serving state | ||
|
vh
|
78c3a7c170 |
feat(booth): asks render INLINE in a verbatim report, placed by the author
Operator verdict on the separate /asks page: the question belongs with the artifact it is about. A four-voice audition wants each voice's radio group under that voice's audio, and one submit for the lot. - booth/inline.py: data-booth-ask="stem" | "stem:key" | data-booth-ask-submit, plus <!-- booth:ask ... --> comments; unknown stem left alone, not blanked - _ask_inline.html: self-contained fragments (own scoped styles, no JS), per-question groups bound to one form via the HTML5 form= attribute so a scattered multi-question ask still POSTs once - unplaced questions and a missing submit block are appended, so a partially marked-up page can never produce an unsubmittable 400 - chip becomes a jump link to the first open ask; /asks page kept as a fallback - 6 tests (one caught the partial-placement drop); v0.1.14 |
||
|
vh
|
a56743ade3 |
fix(booth): asks were invisible in a booth serving its own index.html
A custom index.html is returned verbatim, so booth.html's asks panel never rendered there — a valid ask (emmie-anchor/anchor.ask.json) was listed by the CLI and shown nowhere, with nothing to say so. - panel extracted to _asks.html; new GET /b/<name>/asks standalone page - verbatim pages get an amber '? N open asks' chip beside the back chip - POST /answer honours back=asks so answering returns to that page - single-question asks now keep an optional 'title' (was silently dropped) - README + routes table; 8 regression tests; v0.1.12 |
||
|
vh
|
2a186e4762 | fix(booth): CLI resolves its source through the ~/.local/bin symlink (readlink -f), so ask/asks/answer/unlink work from any cwd | ||
|
vh
|
c85a700141 |
feat(booth): multi-question asks — a questions list renders one form with a radio group per question and lands as one answer sidecar keyed by question
- asks.py: single {prompt, options} and multi {title, questions:[{key, prompt, options, notes?}]} both normalise to questions[]; per-question notes; every question required on submit
- /answer reads choice.<key> / notes.<key> / notes for multi; single shape unchanged
- booth asks prints per-question picks; README + CLI header; install step symlinks the CLI to ~/.local/bin; v0.1.10; 135 tests
|
||
|
vh
|
3fe01225a9 |
feat(booth): asks — a multiple-choice question a session poses in a booth, answered by the operator as a radio form + notes, written back as an answer sidecar
- booth/asks.py (stdlib): <stem>.ask.json question / <stem>.answer.json answer; normalise+validate, atomic write, list with answer folded in, broken asks surfaced not hidden - POST /b/<name>/answer: validates choice against the ask (400), unknown stem 404, re-answer overwrites - booth.html asks panel above the gallery; amber open / green answered; JS-off form POST; index card + booth header badge for open asks - CLI: booth ask / asks / answer [--wait [SECS]]; remote sessions poll <stem>.answer.json over HTTP - ask/answer files excluded from gallery items and item counts; 23 tests; v0.1.9 |
||
|
vh
|
911ff20356 |
feat(erp-seat): NVFP4A16 quant pipeline for the Gemma-4 26B-A4B MoE ERP tune + ana-ml2 GPU1 serve stack
- services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py: linearize_moe first (playbook §3.15), asserts the expert Linear count, routers/vision/audio/norms/lm_head ignored, W4A16 for RP long-session fidelity, post-steps restore processor configs + template and reset the tokenizer truncation cap (§3.14); --dry-run proves targets before GPU time - services/erp-seat-quant/run_quant_erp_v6.sh: detached container on GPU1 (vllm-llmcompressor) - stacks/erp-seat: serve recipe copied from gemma4-charrp, true served name only, port 8021 |
||
|
vh
|
76fdf45925 |
feat(booth): pin/favorite, multi-select delete, newest-first link board
The standing link board grew from a flat oldest-first list with a per-row × into a manageable board: newest links lead, favorites stay on top, and several dead links can go in one pass. - Ordering: order_for_display() renders pinned rows first, then newest-first within each group (the board is an append log, so newest = most recently posted — the row you usually came to grab). - Pin/favorite: a per-row ★ toggles pinned state via POST /b/<name>/pin. State lives in a .pins sidecar dotfile (one content id per line), NOT inline in links.md — so links.md stays a pure atomic-append log (many sessions post concurrently) and a row's content id never changes just because it was pinned. remove_link_entry drops a removed row's pin; orphaned pins are inert (renderer only stars a live id). - Multi-select delete: checkboxes feed POST /b/<name>/unlink-many (repeated 'sel' content ids), with a select-all box and a live count. The per-row × stays for single removal. - One <form> with formaction buttons, so checkboxes, ×, ★, and bulk delete coexist without nested forms AND all work with JS off; JS only adds select-all and the live count. Per-row × confirm reads desc/url from data-* attrs, so an arbitrary posted description can't break into the JS. - Every action is keyed by content id, never row position — same race-safety the existing × has, extended to the bulk path. - Fixed pre-existing undefined --fg/--bg CSS refs in the board styles. Tests: +19 (pins round-trip, ordering, orphan-inert, remove-unpins, /pin and /unlink-many endpoints, board render + order). Full suite 102 passing. Deployed to nh3-dev booth.service; verified live (newest-first, pin round-trip, bulk delete) against the real 31-row board with no data loss. |
||
|
vh
|
0f748ea54e |
feat(searxng): move to nh3-docker, update, and expose as an MCP tool
The ana-docker instance was returning zero results for every query while reporting healthy — 4.5 months stale (2026.4.17 against a current 2026.9.3), its engine scrapers rotted against sites that had changed. /healthz proves the web app answers and says nothing about whether search works, so seven days of green sat on top of a search box that found nothing. Moved to nh3-docker rather than updated in place, because the colo egress is the other half of the problem: 38.120.12.42 is a datacenter address that DuckDuckGo and Startpage CAPTCHA, while nh3-docker egresses residentially at 70.230.226.88. Same reasoning as the fleet's residential proxy for yt-dlp, applied at the source instead of around it. Config corrected along the way: base_url said searxng.pfi.local, a name retired on 2026-08-19, while the environment said something else — the env won so nothing broke and the file quietly lied. The karmasearch.videos removal key never matched, because the engine's real name has a space. scripts/searxng-health.sh asserts results > 0 across three unrelated queries. That is the check that would have caught this, and the only kind that can: the mechanism was healthy throughout. services/searxng-mcp exposes it as `web_search` at user scope, so every Claude Code session has it. Zero results raise rather than returning an empty list — an empty list is indistinguishable from a broken aggregator, which is precisely how this hid. Old instance stopped and removed; DNS alias repointed to searxng.nh3.internal. |
||
|
vh
|
0ad332bb4a |
feat(booth): per-row link removal + render the link board as real UI
The standing link board is the one MULTI-WRITER booth -- every agent session appends operator-facing URLs to it. "Delete the folder" was the only granularity available, so removing one dead link meant hand-editing markdown. It is 32 rows and only grows. booth links row number, entry id, raw row booth unlink 3 by row number booth unlink 8b40e0a5 by entry id (what the UI's x posts) POST /b/<name>/unlink form field `entry` = content id ROWS ARE ADDRESSED BY CONTENT ID, NEVER BY POSITION. The board is append-only and multi-writer: another session can post between listing it and clicking x, and an index would then delete a neighbour. An id either matches the row you saw or matches nothing. A row number typed at the CLI is resolved to its id BEFORE anything is deleted. Appends and prunes now take the same flock on .links.lock, so a post cannot be lost inside a prune's read-modify-write. UI: a booth carrying links.md renders as rows -- description, URL, provenance, copy button, per-row x -- instead of a markdown blob. links.md is filtered out of the gallery so it does not appear twice; the header counts LINKS not files; the empty-state and the one-click "Wipe now" both stand down for a board (same rule as the kept lane: nothing durable is one click from gone). booth/links.py extracted, STDLIB ONLY. The CLI needs this logic and must not require the service venv -- importing app.py drags in FastAPI, so deleting a line from a text file would have needed a web framework installed. THREE BUGS FOUND BY TESTING, all in the shell wrapper while the module was correct throughout -- module-only tests would have caught none of them: - `[ "$n" -eq 0 ] && echo ...` as the LAST statement made `booth links` exit 1 whenever the board had rows. `unlink`'s index lookup calls it inside $( ) under `set -e`, so a successful listing killed the caller and the removal silently did nothing while reporting success. - ids are 8 hex chars and roughly one in forty is ALL DIGITS; those were read as row numbers, resolved to nothing, and removed nothing. Now disambiguated by the id's actual shape, not by "is it numeric". - filtering links.md out of the gallery left `items` empty, so a full board rendered "This booth is empty" and an empty <div class="gallery"> under 32 visible rows. 87 tests (was 76): parser tolerance of hand-written prose, content-id stability across concurrent appends, removal precision, UI branch behaviour for board/normal/empty booths, and subprocess CLI tests pinning the two shell bugs. Deployed to nh3-dev and verified against the live 32-row board read-only; board file byte-identical afterwards. |
||
|
vh
|
4be880f36c |
feat(booth): kept boards can be deleted from the UI; document the TTL-reset trap
Kept boards had no delete path in the UI at all. The kept lane deliberately omits the wipe control -- a one-click wipe next to the durable stuff is a footgun -- but "deliberate" had been implemented as "impossible": the only routes out were ssh or a hand-written API call. Now it is two deliberate acts. A `release` control on kept cards drops the sentinel, the board moves to the ephemeral lane, and the existing x wipes it from there. Release is reversible -- POST /b/<name>/keep pins it again. POST /b/<name>/unkeep release the pin POST /b/<name>/keep pin it (round-trip, so release is not a one-way door) FOUND WHILE TESTING, and it invalidates the previously-documented workaround: removing the sentinel BUMPS the booth directory's mtime, and booth age is the newest mtime in the tree -- so a released board's clock RESETS from 10,000s to 0s and it survives another full TTL. The old comment said "remove the sentinel first (it rejoins the sweep)", which is true but means the board lives another 24h, not that it gets reaped. Unkeep-and-wait is a delay, not a delete. test_releasing_a_board_RESETS_its_ttl_clock pins that behaviour deliberately so nobody re-derives the workaround. Release is what unlocks the x; the x is what deletes. CLI: `booth rm` already worked on kept boards but said nothing about it. It now announces "(was KEPT -- durable board)" so wiping something durable can never look identical to wiping run output. Not a block -- a CLI user naming a booth is being explicit. 5 new tests (67 pass). Verified live on nh3-dev: release renders on all four kept boards, the ephemeral lane keeps its x, and the links board is untouched with its sentinel intact. |
||
|
vh
|
0755ba7d00 |
fix(quant): stop baking the calibration truncation cap into the shipped tokenizer
load_calib tokenizes with tok(..., truncation=True, max_length=seqlen). For a
fast tokenizer that mutates the Rust backend's truncation state in place, and
the subsequent tok.save_pretrained() persisted it, so every mixed-NVFP4 build
shipped a tokenizer.json carrying
"truncation": {"direction": "Right", "max_length": 2048, ...}
against a source whose value is null. Every prompt was clamped at the
calibration length, permanently.
It hid because older transformers does not enforce the text-vs-ids count
check. On a newer one the seat dies at startup with a message that names
images and never mentions tokenizers:
ValueError: Mismatch in `image` token count between text and `input_ids`.
Got ids=[2047] and text=[16384].
The cap also silently limited image resolution well before it killed
anything -- at 2048 the largest servable image is about 1448x1448, since
(edge/patch)^2 / merge^2 image tokens have to fit under it.
Fix saves a pristine tokenizer re-read from the source rather than the
mutated calibration object, and then asserts truncation is null so the
defect fails the build instead of shipping again.
Playbook gains section 3.14 with the symptom, the cause, the audit one-liner
and a table of which builds were affected, plus a fourth mandatory post-step.
The transferable lesson is called out: this is the third case of an artifact
carrying config authored against an older transformers that a newer one
begins enforcing, so an image bump is a config-compatibility event rather
than just a version change.
|
||
|
vh
|
36c173c6a1 |
feat(mog-sec): quant + serve M.O.G.-SEC pen-test seat; PPL on gen; retire fable
Autonomous overnight run under the operator's full-autonomy grant. End state:
fleet up, gen seat untouched, a new verified pen-test seat serving where fable was.
PPL on the orcarouter gen seat (fable downed to free GPU1 for a nospec probe,
probe torn down after): mean 7.07 / median 5.76, within noise of heresy 6.910 /
5.625 and identical to our recipe's usual 7.059. The gen-seat search is settled.
M.O.G.-SEC: chose Blackfrost-Research/M.O.G.-SEC-27B-1M-CTX-BF16 (rev deede677)
over the pre-made ModelOpt NVFP4, which was disqualified on W4A4 4-bit activations
(the AEON degradation mode, catastrophic on a 1M-context model), zero MTP tensors,
and ModelOpt format. Pulled, format-screened (P(<think>) 1.11e-05, clean), quanted
in-house to mixed NVFP4+FP8 (23.4 GB, MTP + vision preserved), and served in the
retired fable slot.
stacks/mog-sec ana-ml2 GPU1 :8019, KV 418,218 tok / 1.60x @ 262K
aliases mog-sec (non-thinking), mog-sec-reasoning (thinking)
gates surface 6/6, MTP 55.3%, format 0/15 leak, vision 7/3/1,
capability 4/4 (delivers offensive-security content)
Served at native 262K, NOT the card's 1M -- the 1M needs YaRN (absent from the
weights' config) plus the SGLang/DFlash2 path the repo ships a deployment kit for,
neither of which is our vLLM surface. A real 1M seat is a separate SGLang project.
Retired char-rp-reasoning + char-rp-fable (zero traffic, pointed at the downed
fable :8019; now 404 cleanly, not repointed -- a security model is not an RP model).
char-rp (meromero) untouched. Vision preprocessor built from the model's own
image_processor block, same trick as the MeroMero seat.
GPU0 seats (gen, meromero) were untouched and healthy throughout. The quant ran in
GPU1 free space with no production seat stopped except fable, which was replaced.
|
||
|
vh
|
e4576f0989 |
test(gen-seat): PPL on orcarouter — mean 7.07 / median 5.76, within noise of heresy
Measured against a spec-decode-free probe on GPU1 (fable downed to free the VRAM, probe torn down after). eval_quality.py aborts PPL under --speculative-config, so a nospec probe is the only way to read it. orcarouter mean 7.0655 median 5.758 heresy mean 6.910 median 5.625 (+2.2% mean) our recipe's usual mixed-quant PPL: 7.059 -- orcarouter is identical to it So orcarouter matches heresy on fidelity and wins on every other axis: MTP acceptance 58.4% vs 47.2%, zero think-leak, vision 7/8. The gen-seat search that ran through Cold-Fusion, heresy, and preetpatel is settled on orcarouter. |
||
|
vh
|
ce09ac4fa6 |
test(gen-seat): add a real vision battery — orcarouter scores 7/8
surface_test.py's vision check is one image and one word. It proves the tower loads; it does not prove the tower works. This battery uses generated images with known ground truth so every answer is objectively gradeable. Against orcarouter NVFP4-mixed on the `gen` alias: T1 OCR, 5 lines incl. one at 18px PASS all 5 exact T2 counting + attribute binding PASS 7 circles / 3 triangles / 1 square T3 bar chart, 6 values + max/min PASS 6/6 exact T4b occlusion, star behind rectangle PASS T4c aspect ratio of a 160x140 rectangle FAIL called it taller than wide T5 two images, which has text PASS T6 four images, the seat's cap PASS all four named T7 five images, one over the cap PASS rejected with HTTP 400 No <think> leak on any vision call. The single miss is fine-grained relative-dimension estimation on a near-square shape, and it reproduced across two runs (the longer T4 called the same rectangle "equal width and height"). Counting, OCR, chart values and occlusion ordering are all solid, so this is a precise-geometry weakness, not a broken tower. Recorded so nobody builds a feature on this model judging relative sizes. T7 earns its place separately: it confirms the per-prompt image cap fails loudly with a 400 rather than silently dropping the extra image. |
||
|
vh
|
f85d102813 |
test(gen-seat): orcarouter passes every gate — in-band MTP head delivers +11 points
Gates run against the live seat while the operator tested in parallel. <think> leak (n=30, 4 prompt types + multi-turn) 0/30, 0 empty MTP acceptance 58.4% @ 117.11 tok/s median surface 6/6 abliteration survival 4/4 compliance deterministic quality gens coherent and correct PPL still blocked For scale on the leak gate, the abandoned h300 build scored 8/30 on this exact instrument, and its abliteration-survival samples had 2 of 4 open with "<think>Ok, let's figure this out:". Orcarouter has none. The headline is MTP acceptance. 58.4% against heresy's byte-identical base head at 47.2% is +11 points, and it sits level with our own in-band L35 at 59.1%. That is the additive in-band-vs-graft delta the entire Cold-Fusion experiment was built to measure and never cleanly delivered -- orcarouter handed it over for free because the author had already done the Robinson edit on the head. Surface 6/6 covers plain chat, vision, tool calling, the thinking split, a 36,042-token long-context retrieval, and streaming. PPL remains blocked on a spec-decode-free probe seat: it needs ~22 GB and GPU1 has ~16 GB free. Comparison target is heresy at 6.910. |
||
|
vh
|
c8f128bdff |
feat(gen-seat): quant orcarouter — its MTP head is already Robinson-abliterated in-band
Pulled orcarouter/Qwen3.8-27B-Uncensored at rev 9878936b (55.5 GB, gated, our token has access) and built /tank/aimodels/qwen38-27b-orcarouter-nvfp4-mixed (23.4 GB, mixed NVFP4+FP8). Verified, not yet cut over. The operator asked whether we could apply the Robinson path to the MTP head. We cannot, because the author already did. compare_mtp_head.py against the verbatim base graft: 13 of 15 tensors byte-identical, exactly 2 differ -- mtp.layers.0.self_attn.o_proj.weight and mtp.layers.0.mlp.down_proj.weight, which are precisely the two residual writers our own abliterate.py targets (EXPECT_MTP_WRITERS = 2). Reverse-engineered the edit from the weights alone (mtp_delta.py, added here): sigma2/sigma1 = 0.0164 on BOTH tensors rank-1, a single-direction projection |cos| between the two recovered dirs = 1.0000 ONE shared direction ||delta||/||W|| = 1.42% and 1.41% a gentle, consistent projection sink energy dim 3994 = 0.0000% sink-clean; Heretic's was 6.18% That is the Robinson in-band MTP abliteration, already applied, with a direction that passes our sink screen outright. Nothing to do but preserve it, and the quant carries it byte-identically. This is the configuration the entire Cold-Fusion experiment was designed to test and never cleanly delivered. The new format screen paid for itself on its first real use: think_prior.py on the bf16 BEFORE any GPU time gave P(<think>) = 1.23e-06 at rank 52, against Cold-Fusion stock 0.1850 and h300 0.2216. Roughly 150,000x cleaner. Two durable findings about the pipeline itself: The quant needs ~17 GB, not a whole card. It ran entirely in GPU1's spare 16 GB with ZERO production seats stopped -- the h300 run's "stop BOTH GPU0 seats" was never necessary, it simply had a free card by coincidence. The first attempt OOM'd by 2.37 GiB at layer 64 of 65 with 3.57 GiB reserved-but-unallocated, which is fragmentation, and PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True closed it. post_quant.py now builds a missing output index from the safetensors headers. A sub-23 GB quant saves one bare shard with no index, and post_quant needs one; this has broken three separate rounds and been hand-fixed every time. The header is read by struct-unpacking the u64 length and parsing the JSON -- never safe_open, which mmaps the whole 22 GB shard and ENOMEMs on ZFS. Artifact verified: mixed-precision, 1968 tensors, 15 mtp, 333 visual, re:^mtp.* present in the ignore list (llm-compressor pruned it as always), preproc restored. Imatrix deferred per operator; the log confirms the usual uniform-MSE fallback, so this build stays apples-to-apples with heresy's PPL 6.910. |