Prime's call via brokkr: h2o-lightning-4b replaces the intern-decision seat once
albok-dev re-validates Nemi's sensitivity judge on it. services/h2o-decision-front
builds on infra-hermes's admission proxy (30f36f2): 32,000-token refuse-not-trim,
the seat's bearer token, and intern-decision-serve's /decide/shared mapping onto
/v1/systemone so the judge runs unchanged. stacks/h2o-eval serves it on GPU 3
:8034 exactly as benched (vLLM 0.30.0 + unmodified vendor shim; weights hashed
against HF b43f759b). INC keeps serving production on GPU 1.
albok-service leaves nh3-docker before unit 4 puts the personal vault and
licensed books into restricted wings. On nh3-docker, portainer, portainer_agent,
dockge (socket rw) and dockerd on plaintext tcp/2375 were each root over
/srv/albok. The new host runs only albok-service: no socket containers and no
TCP Docker API, so its Homepage tile is a static services.yaml entry.
- nh3-pve-2: vmbr0 made VLAN-aware (applied by reboot, no guests running);
pbs-ana storage in the new namespace nh3-pve-2, plus a 21:30 vzdump job
(pbs-nh3's recursive sync mirrors it)
- VM 110 nh3-albok, 10.100.50.44 on VLAN 50, Debian 13 cloud-init, 4 vCPU
x86-64-v3 / 12 GiB / 100 GB; built to fleet conventions (vh 1000,
infra-ops 850, docker 851)
- compose: mem_limit 4g -> 8g, homepage labels removed; host port 8392 kept
- cutover by tar stream, verified by digest (contents plus owner, group, mode,
mtime and type identical); albok.nh3.internal now points at nh3-albok
- old /srv/albok stays on nh3-docker (container removed, compose renamed)
until acceptance and the operator's approval to delete it
- sudo-io archive hosts and ops-log STACK_HOSTS include nh3-albok
Vendor-path benches on fv-ml1 GPU 3 at the 14.4 GiB / 32k cap. Neither
candidate replaces Intern-Decision-4B: H2O reproduces its vendor figure
(205x3) and has the best calibration, but rests 1 GiB over the cap and
is 2.2x slower at the 21-criteria shape; Clef loses on accuracy (191-192),
hard tier (72-73/111), and ships a silent 16,384-token input slice.
Adds the namedef cell and JB_AUTH wiring to the 09-30 harness client.
A/B of the live STT seat (fv-ml1 GPU 0, sherpa-onnx int8 v3) against
nvidia/parakeet-unified-en-0.6b, measured on GPU 3 with the seat's own image,
k2-fsa's published unified int8 export, fp32/fp16 exports made with k2-fsa's
recipe, v2 int8, and NeMo 3.0.0 (fp32, bf16 autocast, bf16 weights).
- Seat int8 graph runs on one CPU thread (cpu/wall 1.00, GPU 2-9%).
- unified-en under NeMo: -121/-234/-530 ms vs the seat at 1-3/3-8/8-20 s
(paired, n=120/bin; floor <=6 ms; +50 ms positive control reads +52-54).
- unified-en WER lower in every runtime: -0.7 pp clean, -1.5 pp other,
-3.2 to -4.4 pp AMI (paired CIs exclude 0).
- Seat defects found: hard 400 s input ceiling (HTTP 500), truncation after
a quiet 1.5 s pause, and severe long-window dropouts (int8 v3 only).
- B-bf16w needs +0.8 to +1.5 GB over the seat's 1,690 MiB on GPU 0.
Raw requests, hypotheses, manifests and the full harness under
services/parakeet-ab-2026-09-30/. No deploy; live seat untouched apart
from 240 light test requests.
The ~6.5-9 s first-call-per-bucket autotune lived in the container's writable
layer and died on every recreate. 0.1.3 creates /tmp/triton-cache in the image
owned by 10001 so the named volume intern-decision_triton-cache inherits a
writable mount point, and compose mounts it.
Bucket model PROVEN, not inferred: 2,048-token buckets, 16 up to 32,768. After
one warmed call per bucket, 12 random sizes across 8k-32k were all warm (worst
2.09 s); cold entries cost 6.5-9 s. Full cold warm-up 109 s; warm re-run 17 s.
scripts/intern-decision-warmup: one noul call per bucket, MAX_TOKENS from
/health, two-point live calibration of the tokenizer's linear token model (a
single probe overcorrects and the aim oscillates around the bucket edge),
per-bucket wall times, non-zero exit on a missed bucket. Run it after an IMAGE
CHANGE only; the volume carries ordinary recreates (measured: force-recreate,
then a warmed 32k call answered in 2.11 s).
Acceptance on 0.1.3: JevBench 202/231, hard 83/111, 0 diffs / 924; warm 32k GPU
1 peak 15,218 MiB (budget 15,220; a COLD autotune touched 15,224 once, README
caveat); /decide answers. Artifacts in the acceptance dir.
Audit finding on 0.1.1: a Jev client that always sends an images array with no
images was rejected for nothing. Only a non-empty value is 422 now. Also aligns
the contract's response example with the wire (model is the name@revision
string, not an object).
Re-accepted live: images []/null -> 200, ["a.png"] -> 422; JevBench all 202/231,
hard 83/111, 0 changed rows across the bench's r1..r4 (924).
Straight passthrough to the checkpoint's own DecisionEngine.predict — never the
semif mapping, whose different prompt would change the answers. Reuses the one
inference thread, bearer auth, MAX_QUEUE, VRAM cap and error envelope; no new
concurrency. 1..16 questions in ONE call (never chunked: Jev questions share a
prompt); images 422; over MAX_TOKENS 422 before the forward. /health advertises
the surface. Response 'model' is a string name@revision (JevBench's runner
hashes it; a dict broke its manifest step).
Acceptance on the live service (see acceptance/systemone-2026-09-30/): JevBench
v1.2.16 typesafe adapter over the 231 public items scores all 202/231, hard
83/111, with 0 changed answers across all 924 rows of the bench's own r1..r4;
controls 401/422x3 (token boundary proven at 7168 pass / 7169 refuse); GPU 1
per-process peak 9,866 MiB under the largest accepted request (budget 9,876);
/decide/shared positive control unchanged. 120 tests green.
intern-decision deployed 0941 PT (cap 9.0 GiB, MAX_TOKENS 7168). Live acceptance: positive
control 240/259 and Wyrd 79/84, bit-identical to the bench (0/560 rows, Δp 0); negative control
10/122/14; largest accepted requests 200 with no 503; per-process 8,812 MiB at rest and 9,866 peak;
GPU 1 Free 15,442 before and 6,581 after (lowest 5,569 under load). Latency from nh3-dev:
21 criteria 114 ms, 16 over ~3,900 tokens 238 ms. semif README banner now REPLACED with the
rollback; fv-ml1 GPU 1 note updated.
Both are required in compose because they are coupled: MAX_TOKENS is checked before the forward
pass, so an oversized call is a clear 422 instead of reaching the cap as a 503. Pre-deploy floor
is nvidia-smi Free >= 15,400 MiB on GPU 1 (card peak 9,876 + scriberr 5,496).
stacks/intern-decision: compose (GPU 1, :8033, hard VRAM cap as the single .env knob,
healthcheck, Homepage group 'AI - Eval & Retrieval'), .env.example and README.
dns: intern-decision.fv.internal -> fv-ml1 (synced to ana/esh/nh3).
acceptance on fv-ml1 GPU 3, 3 fresh processes: bit-identical to the Jev bench's native rows
(pooled 240/259, Wyrd 79/84, 0/560 flips, Δp 0), negative control 10/122/14, 0 flips across
restarts; largest accepted request 200 at a 10,134 MiB card peak under a 9.25 GiB cap; 503 and
recovery proven at a tight cap. GPU 1 deploy held: nvidia-smi Free on GPU 1 is 15,442 MiB.
- a failure while building the response (a non-finite number included) is a 500 inside the
envelope, never a 422 or a render crash outside it
- an engine ValueError keeps its message but is released and raised unchained, like an OOM
- the prompt is built (and the model's own validation run) before the forward
- the row cap is counted before any ordering is built
- /health reads a device name cached at load, so it makes no driver call off the inference thread
torch keeps CUDA state per host thread (cuBLAS handles and workspaces), partly outside the
per-process VRAM cap. Scoring on anyio's threadpool let 40 threads each create it: measured on
fv-ml1 GPU 3, +252 MiB outside the cap and +326 MiB inside, which pushed the process past the
10,300 MiB GPU 1 budget. Load, warm-up and every call now run on the same single thread.
Its dataclasses use postponed annotations and look their module up in sys.modules while the
class is built; importing by file path without registering it failed startup (closed).
Contract, service and tests (fake engine, no GPU). Scores through the checkpoint's own
inference.py (DecisionEngine.predict, sha256-pinned); maps semif decisions onto Jev choice
questions, packs /decide/shared into calls of at most 16, runs orderings in waves, and keeps
semif's error mapping, admission, body limit and hard VRAM cap. Deltas from semif-serve are
listed in the contract.
Operator ask relayed by brokkr-smithy-dev. Positive control (SemIf 187/231,
hard 0.613) reproduced exactly; negative control and a 4-restart noise floor
(0 flips) measured. On our replaced-baseline sets no candidate beats SemIf-with-
rotations beyond the ~4-pt floor; Intern-Decision-4B native matches it at one
ordering, is better on Wyrd, fits 9.7/10.3 GB and is 1.5-2.3x faster. JevBench
rank does not transfer. Raw per-item data kept out of git.
SemIf's shared scorer trims one token at the state boundary. When an object
state's last value ends in ')', ';' or '}', the JSON that follows re-merges two
tokens back, so score_shared refused the request with 422. The engine now wraps
semif_phase1.shared._state_prefix to keep only the tokens the full prompts
share. Each row scores the same token sequence; only the prefill/suffix split
moves.
Startup proves the fix is in effect, not just installed (heid bug hunt SKAL,
folded). It checks that the hook is callable and is what score_shared resolves,
that an ordinary state keeps upstream's whole prefix, and that a merge-prone
state scores through the shared path.
Real tokenizer: 154 states, 23 refused before and 0 after, with no ordinary or
authored144 prefix changed. Acceptance: 144/144 parity. Shared vs direct 71/72;
the miss is a bf16 tie that flipped across a plain restart (see README).
Against talk /face's guided pose (first paragraph 246 ms median), SemIf in
parallel adds 32 ms and SemIf-first adds 94 ms (n=72 each, noise floor 16.5 ms).
Removing the pose header saves only ~31 ms, and SemIf shares GPU 1 with the LLM.
Acceptable pose 67% vs 92% on clear-emotion lines, and the mood carried through
mundane follow-ups 7/15 vs 14/15. SemIf gestures far less (13% vs 58%).
README: rotations cost options^2 in suffix tokens, and /decide/shared returns
422 when an object state's last value ends in ) ; or }.
Harness consumer_fit.py runs a consumer's per-turn decisions over hand-labelled
cases, with rotations, a content-free null control and tagged positive controls.
Cicada: an input-only 'does this earn a visible reaction?' gate scored 30/31
with descriptive options and 19/31 with terse yes/no options. Scoped by
Cicada's 2026-09-20 ruling (affect is emitted once, no mood-ring classifier).
Wyrd: on 3 real seed graphs, the first place-change wording failed its
positive controls (1/6 moves). A location-anchored rewording scored 21/21, and
exit selection scored 18/21. Semif fits the choice, not writing the node.
Order averaging (Prime, after the 739aa03 spike):
- A decision may set orderings: rotations|all (all only for <= 4 options). Every
ordering goes to the engine in one shared batch.
- The reply keeps each native result and adds combined {probabilities (log-mean),
top, agreement, spread}.
- Through the service on SemIf's labelled sets (252 rows): 78.6% -> 88.1%
(group-bootstrap 95% CI +5.1..+14.3). Unanimous agreement is 94.5% accurate.
Fast kernels: flash-linear-attention 0.5.2 and causal-conv1d 1.7.0 are now the
default build. A/B on the empty GPU 3:
- parity with upstream went from 142/144 to 144/144;
- a ~2k-token /decide went from 169 to 92 ms server-side;
- short 3-rotation batches cost ~3-6 ms more.
triton builds a C shim at runtime, so the image carries gcc. Without it the
warm-up failed and startup failed closed.
Heid bug-hunt panel (4/4 arms, thread 01M3H3F4RR7XBP90KQ3A39H4SX), folded:
- Startup validation: VRAM cap 0 no longer means uncapped (C1); limits must be
>= 1 (S1); the token must be visible ASCII (S2); the calibration file must
exist and parse, with T in [0.05, 20] (S8, and C3's NaN leg).
- The body limit is checked before a chunk is kept, and a Unicode-digit
Content-Length no longer crashes (C2, S3).
- Failures while building the response now get the 500 envelope (C3).
- 429 busy past SEMIF_MAX_QUEUE requests in progress (C6).
- The engine releases memory on every non-validation failure, unchained after
gc; an empty OOM message is handled; 'out of memory' RuntimeErrors map to 503
(C4, C5, S9).
- The entry point forces HF_HUB_OFFLINE (S10). README wording fixed (S5, S6).
- New guard tests close the gaps the arms' mutation grids exposed: early stop of
the body read, a shared-route lock, calibration pass-through, the gc cycle,
the exact caps, TorchEngine.load's arch and device checks, and the offline
entry point.
86 tests.
Deployed on fv-ml1 GPU 1: parity 144/144, OOM and burst release verified, shared
capacity 63/51/26/16 rows at ~140/520/1960/3900 prefix tokens.
Latency, measured from nh3-dev (3 runs x 20 per condition; network floor 31 ms):
- /decide short: 71 ms end to end, 38 ms server-side;
- /decide with a ~2,000-token state: 210 / 169 ms;
- shared, 3 rotations: 113 / 79 ms;
- shared, 6 orderings: 137 / 99 ms.
Qwen3.5's fast kernels (causal_conv1d, flash-linear-attention) are not installed,
so transformers falls back to its reference PyTorch paths. That is a speed lever,
and using it needs a parity re-check.
Averaging over option orderings, on SemIf authored144 + perturbations108 (252 rows,
72 groups):
- a single ordering scores 78.6%;
- log-mean over the 3 rotations scores 87.7% (+9.1 pts, group-bootstrap 95% CI +4.7
to +13.8);
- all 6 permutations score 88.1%.
Rotations capture nearly all of the gain. Rows where the rotations agree unanimously
(161) are 94.4% accurate; split rows (91) are 75.8%.
services/semif-serve is a FastAPI wrapper around SemIf's direct and shared torch
scorers (SemIf-OpenJev @ 23cf1f39, MIT). Upstream ships only a batch CLI. The
wrapper loads the pinned Qwen3.5-4B (851bf6e8, BF16) once from the offline HF
cache and returns SemIf's result dicts unchanged, with an optional per-workload
temperature-calibrated view. Contract: semif-serve.contract.md. Built with a
short contract, TDD (39 tests, fake engine and fake torch, no GPU) and a heid
bug-hunt panel (pending).
On the card:
- torch 2.10.0+cu128 with sm_120 kernels, which is SemIf's own stack;
- a hard 12 GiB VRAM cap.
Two defects surfaced only on the card, and each fix is covered by a test:
- 0.1.1: an OOM raised as a chained exception kept the failed request's tensors
alive (11.9 GiB after the 503). It is now raised unchained, after gc.
- 0.1.2: a large request left 12.6 GB reserved on the shared card. After each
call, reserved memory over the baseline + 512 MiB is now released.
Acceptance against SemIf's committed torch predictions (authored144):
- 142/144 same top choice; both misses are exact bf16 ties;
- 144/144 identical prompt hashes;
- deterministic A-vs-A;
- negative control 14/144;
- shared vs direct 72/72.
21 binary criteria over one state take 159 ms. The shared-mode capacity table
under the cap is in stacks/semif/README.md.
The Dockerfile installs dependencies from a manifest with the project version
blanked, so a version bump reuses the ~4 GB torch layer. Verified: 41 s rebuild,
dependency layer CACHED.
DNS: semif.fv.internal. Token: vault semif/api-token.
Three of the four failures in the week to 2026-09-23 (09-19, 09-21, 09-23)
had one signature. The Cloudflare zone lookup returned an empty body at its
15s cap, a bare json.load crashed with tracebacks instead of a cause, and the
script carried empty IDs on to a PATCH against zones//dns_records/.
Cloudflare rejected it, so there was no DNS impact, but only by accident.
- Cloudflare calls go through cf(): 3 announced tries, and a call counts
only when the body says success:true.
- success:true is not trusted as shape. pick() validates every body and
prints only the fields asked for, or one line saying why not. It requires
exactly one zone named phasefinal.com and exactly one A record named
headscale.phasefinal.com, each with a non-empty id and content. Two A
records are refused rather than half-updated, and an empty id can no
longer shift the content into the id slot.
- No write without both IDs. The run ends on a confirmation that the record
now reads the new address. The previous final line was an echo whose exit
status was always 0, even when the parse inside it failed.
- Only a global unicast IPv4 is published (python ipaddress is_global).
Loopback, RFC1918, link-local, CGNAT and documentation ranges are retried
and then refused.
- curl -q as the first argument ignores ~/.curlrc, so a verbose config can
never log the bearer token. The vault CLI path is quoted. The empty
data-array expansion is safe under set -u on bash < 4.4.
Tests: services/headscale-ddns/test_headscale_ddns.py, 12 cases with curl,
the vault CLI and sleep stubbed. Each failure case asserts the FATAL line's
stated reason, so a run that died earlier for an unrelated cause cannot pass
it. The documentation-range fixtures (203.0.113.x) were themselves rejected
by the new public-IP guard: a free positive control.
Live: a manual run and a unit run both printed "unchanged 70.230.226.88",
Result=success. Cross-model bug-hunt (heid "Talus", Gróa + seat): all 8
findings folded.
A deliberate `systemctl --user restart hermes-gateway` paged infra-ops as a
FAILED unit (msg 3716) while the unit was already back up. Cause is a Hermes
v0.21.1 race: the planned-stop marker watcher runs the shutdown handler before
systemd's SIGTERM, consumes the marker, and the SIGTERM re-runs the handler,
which then classifies the stop as unexpected and exits 1.
Corrects the README claim that OnFailure never fires on a deliberate restart:
that holds only when the main process exits with a success status. Measured on a
throwaway unit (3/3 paged without SuccessExitStatus, 0/3 with it, and crash
restarts are unaffected), and a survey of every stop on nh3-dev since 09-15
found hermes-gateway to be the only unit that does this.
The host-side fix is drop-in hermes-gateway.service.d/
20-planned-stop-exit1-is-clean.conf (SuccessExitStatus=1). No alarm coverage
is lost: Restart=always ignores the classification, and StartLimitIntervalSec=0
means the unit can never reach `failed` from a start failure.
The failed-START notifier built earlier today had its first REAL firing at
15:28: headscale-ddns.service exited 1 after succeeding all afternoon. The
detection worked. The alarm was also useless, and that is the finding.
Both failure paths exited 1 IN SILENCE, so the message said "exit status 1" and
nothing else. An alarm you cannot act on costs the same triage as no alarm at
all -- the notifier did its job and the subject script had no diagnostics for it
to carry.
Cause was transient and harmless: icanhazip.com did not answer inside its 10s
cap, so the IP came back empty and the regex guard refused it. No DNS impact --
the record already held the right address, verified against 1.1.1.1 before
touching anything, and the next timer run succeeded. Arithmetic confirms it:
~17s vault read + 10s curl timeout = 27s against the 28s the failing run took.
Fixed, both verified by making them fail:
- every exit path names its cause; a missing vault key names the key, an
EMPTY token is distinguished from a failed read, and a dead WAN lookup adds
"DNS left unchanged" because that is the fact the reader needs
- the WAN lookup retries 3x with ANNOUNCED attempts -- one third-party blip
should not page a human, and a silent retry would hide a degrading
dependency
⚠ ALSO: this script was not tracked anywhere. A fix to the thing every mesh
client resolves through lived on exactly one disk. Script, unit and timer are
in the repo now.
Measured and recorded: the vault read is 17 of the script's 18 seconds, every
10 minutes. Not a fault, but it bounds any retry budget and it is fleet-wide --
svos-dev's alarm unit carries the same 17-second note.
svos-dev found an interaction between the two detectors by doing the recovery
rather than reasoning about it: clearing a failed unit is REQUIRED before
systemd will start it again, and reset-failed zeroes NRestarts in the same
motion. Verified here -- a start-limited unit refused to start until
reset-failed, which took NRestarts 3 -> 0.
So the standard recovery from a failed-START alarm erases the history a flap
sampler would depend on, routinely rather than as an edge case. A unit that
fails, is cleared, then flaps starts its history from zero, delaying the first
flap alarm by the whole threshold -- exactly the window where a just-fixed unit
is most likely still sick. Mitigation recorded: store a last-seen timestamp so
a counter going BACKWARDS is itself detectable as a reset rather than read as
quiet.
Also recorded as its own shape: 'nothing to count is not the same as not
counting'. The first manual-restart check read NRestarts flat and would have
concluded a sampler goes blind whenever a human touches a unit. Artifact of the
test -- that unit had already hit its start limit and stopped auto-restarting,
so the instrument was reading correctly from a subject that had stopped
producing. Verify, then ask what the verification could not have seen.
OnFailure fires when a unit enters failed. A unit that keeps re-earning its
retries never enters failed, so the alarm never fires for it. svos-dev warned
this was the consequence of the interval divergence; measuring it showed the
gap is most of the box.
MEASURED: a unit shaped like booth/althing-po-herald (Restart=always,
RestartSec=5s, burst 5, interval 10s) ran 9 restarts over 45 seconds cycling
active/activating and NEVER reached failed. No alarm would have fired.
Covered: the 7 timer-driven oneshots (Restart=no, so any failure lands in
failed immediately) and svos.service (burst 3 per 5min -- it genuinely gives
up). NOT covered: ten daemons that flap instead, and hermes-gateway, whose
start limiting is disabled outright.
svos.service's divergent 5min window is deliberate and load-bearing (operator
ruling 2026-09-11, 'fatal both ways'). NOT to be harmonised: that would restore
the flapping the ruling forbids AND silence the alarm on the one daemon it
currently works for. Whether the OTHERS move to 5min is a behaviour change for
ten services and an operator call.
Also recorded, svos-dev's framing of the two installer bugs: a tool that
enumerates 'things that are fine' and acts on them has selected against its own
subject. Neither bug failed randomly -- both were anti-correlated with purpose,
working better the healthier the fleet looked.
Two defects, both found by the tool failing to cover a unit that was already
broken. talk.service has been sitting in `failed` (exit 143) while its
containers keep serving 200 on :8092 -- precisely the "unit died, the thing
keeps serving, nobody knows" case this alarm was built for -- and the installer
had not hooked it.
1. SELECTION: --state=running skipped anything not already healthy. A unit that
is down at install time never got hooked, which inverts the tool's purpose.
Now the union of enabled unit-files and all loaded units.
2. PARSING, and this is the sharper one: systemd decorates a FAILED unit with a
leading "●", so `awk '{print $1}'` returned the bullet instead of the name,
and the sanitiser reduced it to an empty string and dropped the row. The
parser silently lost exactly the rows that matter. --plain suppresses the
decoration. Same shape as every other instrument error today -- it reported
cleanly while looking at the wrong thing.
Selection now 23 units. Deliberately INCLUDES the timer-driven oneshots
(dev-backup, ha-backup, fleet-tls-cert-check, headscale-ddns,
seat-inventory-drift, brokkr-landscape-scan, soong-ci-relay): a backup or a
cert check that fails silently is the same class, and all seven were clean at
install time so they are not a known noise source. EXCLUDES dbus, gpg-agent and
dirmngr as OS plumbing, the notifier's own template and instances, and
svos-failed-alarm -- hooking an alarm with an alarm is a loop with extra steps.
svos-dev challenged a claim this README made -- that a crash-loop yields one
message per episode -- with a measurement: one 2026-09-19 boot-gate refusal on
svos.service produced FIVE transitions into failed, and the operator got five
messages.
Measured here before accepting it, because a peer's number is still a number
someone else took: a unit with Restart=on-failure, burst 3, interval 30s
produced 7 journal failure lines and exactly ONE notifier invocation. So the
multiplier is not universal -- it needs retries spanning start-limit windows or
an external restarter. svos.service carries StartLimitIntervalSec=5min, which
is how it accumulated five.
Both conditions exist on this box, so the guard goes in as cheap insurance
rather than as a fix for something proven here. The README now states both
numbers and which restart policy each of the twelve units carries, since that
is what decides the exposure. Noted with it: hermes-gateway has start limiting
DISABLED, so it retries forever and may never reach failed at all -- worth
knowing before trusting this alarm to cover it.
Design, taken from svos-dev's shape:
- Keyed on a hash of the CAUSE (unit + result + exit status + the shape of
its last error lines), never the unit name alone. A genuinely different
failure inside the window is a new fact and must still page; suppressing by
unit would hide a second, worse failure behind the first.
- Suppression is LOGGED to suppressed.log, never silent. An alarm that
quietly declines to fire is indistinguishable from one that is broken.
⚠ The first test of this appeared to show the cooldown not working, and the
test was wrong rather than the code -- it invoked the script BEFORE failing the
unit, so the two calls legitimately saw different states and computed different
fingerprints. Re-run the way systemd actually invokes it: same cause 3x -> 1
sent, 2 suppressed and logged; a different cause inside the same window -> sent.
svos-dev found the failure neither Beszel nor Uptime Kuma can see, and
measured it: a config change on 09-19 made svos.service refuse to boot, the
RUNNING process predated the change and kept serving, and the service sat one
restart from dark for three days. Every uptime probe was green and correct --
the thing was up. The signal that catches this is failed-START, not down.
A count after that conversation: 13 running user units on nh3-dev, ZERO with
an OnFailure hook. Including althing-po-herald, whose silent failure cuts
infra-ops's own mail delivery -- a blind spot in the notification path every
other alarm on this fleet depends on.
One template, one drop-in, 12 units hooked (dbus excluded as systemd's own
plumbing). Not noise: OnFailure does not fire on a clean restart or a
deliberate stop, and with Restart=on-failure a crash-loop yields one message
per episode rather than one per attempt.
⚠ %i, NEVER %I -- and the acceptance test is the only reason this is right.
%I unescapes the instance name and systemd escaping maps "-" to "/", so the
first run delivered a message for "onfailure/selftest.service", a unit that
does not exist, with a spool path that tried to create directories.
althing-po-herald.service would have arrived as althing/po/herald.service.
It "worked" -- mail was delivered -- which is exactly the kind of success that
is not one. The referring unit passes %n raw, so literal %i is correct.
⚠ The notifier never reports itself, guarded twice on purpose: the template
carries no OnFailure, and the script bails on its own instance name. A
notification loop is the one bug that pages you forever.
Delivery is spool-first. postbox has no outbox, and an alarm for moments
nobody is watching must survive the post office being one of the things that
is down. If the herald itself fails, the message still REACHES the post office
(postbox talks to it directly; the herald only delivers inbound pokes) -- not
pushed, but stored for the next read.
Acceptance-tested twice against a unit that exits 42: once to catch the %I
bug, once to confirm the fix. Test unit removed.
TAILSCALE IPv6. `tailscale status` had been reporting, continuously:
2 add route failures; first was: permission denied
adding address fd7a:115c:a1e0::6/128 from tunnel interface: permission denied
with tailscale0 carrying only 100.64.0.6/32 while headscale had assigned it
an IPv6 address it could not use.
Not a capability problem -- tailscaled runs as root with the full bounding
set. /etc/sysctl.conf:59 sets net.ipv6.conf.default.disable_ipv6=1, and
`default` is inherited by NEWLY CREATED interfaces; tailscale0 is created at
daemon start, inherits it, and the kernel returns EPERM for every attempt.
Fixed with a scoped systemd drop-in rather than flipping the global default.
That line carries no comment, but IPv6-off-by-default on a host with ~26
docker bridges reads as deliberate, and changing it would hand IPv6 to every
future bridge as a side effect of fixing Tailscale.
⚠ It must be ExecStartPost, not /etc/sysctl.d. A sysctl.d entry for a
per-interface key is applied at boot, BEFORE tailscale0 exists, and is
silently ignored -- the setting would look present and do nothing.
Also learned: setting the sysctl on the LIVE interface is not enough.
tailscaled only attempts the address at startup or on a netmap change, so
the verify failed for 60s until the daemon was restarted. Restart is part
of the operation, not an afterthought.
Verified: fd7a:115c:a1e0::6/128 present on tailscale0, health clean, mesh
and services (arbo, ytvc) up.
README. It documented the pre-headscale topology as current -- "Reachable
IP: 10.100.79.3 (WireGuard tunnel endpoint)", "No direct LAN access", and a
refresh caveat telling you to bring WG up. That sends anyone triaging this
host to the wrong layer, which is the exact tax the file exists to prevent.
Now: mesh primary at 100.64.0.6, LAN 10.6.110.50, and wg0 documented as
STILL UP with a live peer -- tailscale uses that address as its direct
endpoint, so it is load-bearing, not vestigial.
Recorded with it, because these cost hours tonight and will cost them again:
- Irvine is a TENANCY behind a Fortinet PFI does not control. Its TLS
inspection breaks Tailscale's relay and control channels (41 cert
warnings/week, 4 control-plane episodes in 14 days). Usually invisible
because direct peer paths carry the data. No fix on our side.
- Diagnose reachability with `tailscale ping`, NOT the status output:
headscale said "online" and status said "active, 19.7 GB" while nothing
on the host answered. Both are last-known state; only a round trip is
liveness.
- The ~26 docker bridges make tailscaled report captive portals.
Two stale claims corrected: the hostname rename it called "pending" is done,
and `ollama` is listed as running on :11434 when it is gone -- verified, no
unit file, nothing listening, no process. It is banned fleet-wide.
THE BRIDGE. `beszel-althing` hardcoded a "[Beszel] " subject prefix and a
Beszel hub footer from when Beszel was its only caller. Routing Uptime Kuma
through it unchanged would have delivered Kuma outages labelled [Beszel],
pointing the reader at the wrong dashboard -- an alert that lies about its own
source is worse than no alert.
Now a route registry: /beszel and /kuma, each with its own prefix, footer and
payload parser, because the tools do not agree on a shape (Beszel sends
{title, message}; Kuma sends {heartbeat, monitor, msg}). Generalising cost a
dict; a sibling service would have cost a second unit, a second port and a
second thing to notice had died.
Renamed beszel-althing -> althing-alert-bridge with it. A service named after
one consumer that carries two is the invisible coupling that sends a future
session looking in the wrong place.
⚠ /beszel IS FROZEN and this refactor proves it rather than claiming it. The
three original tests were kept BYTE-UNCHANGED -- including the one asserting
the exact postbox argv -- and deliver() still defaults to the Beszel route so
they exercise it. A new test asserts the Kuma footer never leaks into a Beszel
body or vice versa. Verified live after the rename: a real POST to /beszel
landed as "[Beszel] BRIDGE RENAME CHECK" with the correct hub footer, read back
from the thread rather than trusted from the receipt.
Payload shapes are parsed HERE, not via Kuma's custom-webhook-body feature,
because Kuma's notification config lives in its own database -- and that
database was destroyed and rebuilt from scratch hours ago. Anything living only
in a tool's DB is lost on the next rebuild; format knowledge belongs in git,
next to a test.
parse_kuma also handles the monitorless case. testNotification and cert-expiry
alerts carry no monitor and no heartbeat, and the first cut fabricated "unknown
monitor is ?" from them -- caught by sending a real one and reading the subject,
not by the suite. Fixed, pinned, and the earlier test asserting the bad
behaviour was corrected rather than worked around.
KUMA IS NOW WIRED. scripts/kuma gained notification support and the channel is
in monitors.yaml, seeded BEFORE the monitors and with applyExisting, so a
rebuild restores alerting and not just detection. Ground truth from the DB:
13 of 13 monitors carry the channel.
⚠ A THIRD instance of the same class of bug, worth naming: notifications() is
pushed as `notificationList` at LOGIN ONLY -- there is no event to ask with. The
first cut cleared the captured value before waiting, discarding the only copy it
would ever be sent, then blocked for the full timeout and reported an empty
list. That reads exactly like "no channels configured" and is a lie. Same family
as the getMonitorList ack-vs-push trap, different shape.
End-to-end, both shapes, read back from the inbox:
[Uptime Kuma] Homepage is DOWN + target + board link
[Uptime Kuma] althing (infra-ops) Testing (no fabricated subject)
[Beszel] BRIDGE RENAME CHECK + hub footer, unchanged
ALTHING CHAMBER RETIRED (operator). Three of its four containers had never
started -- created 2026-09-19, StartedAt epoch-zero, 0 restarts -- so :7881
refused, and nothing was watching it. Only its valkey was running, on the
project's own network with no external consumer. Stack, compose/build/conf dirs
and the local image removed; the Homepage card went with the label.
The Booth is now one of the most-used fleet tools -- 17 agent handles post to
it daily -- and it is taking an information-architecture rework plus a
cross-agent SVOS design retrofit from design-dev. That work wants its own
ROADMAP, contracts and blast radius, not the fleet-infrastructure repo's.
All 29 commits moved with it via `git subtree split`; the history carries real
lessons (two shipped-dead controls, the verbatim-injection traps) that a
squashed import would have thrown away.
Live service repointed and verified: the user unit and the ~/.local/bin/booth
symlink now resolve into ~/development/booth, healthz answers, all 24 booths
intact. services/booth/ keeps a pointer README, same shape as the
chatterbox-fast and tts-stack extractions.
brokkr-smithy-dev exercised the slicer on real geometry and found OrcaSlicer
writes result.json into the working directory on --info. I verified it here and
it is worse: --help does it too, which is as read-only as an invocation gets.
An empty directory gained a result.json from --help alone.
Encoded in the wrapper and the Dockerfile rather than left to be rediscovered,
and the wrapper is now committed beside the image that backs it instead of
living only in a home directory on one host.
⚠ The reason they found it and I did not is worth keeping in the header: I
verified --help and called the slicer done. Usage text proves the binary
starts, not that it ingests our geometry or that it behaves while doing so.
They ran both controls through to STL and got manifold=yes, 520 facets,
return_code 0 on each -- which is the verification I should have run before
reporting the provisioning complete.
Not tidiness — necessity, established by ldd rather than assumed. Every
OrcaSlicer release ships only an Ubuntu 24.04 AppImage, which needs GLIBC_2.38
and GLIBCXX_3.4.32. irv-ml1 is Debian 12 on glibc 2.36. That gap is not
installable without moving the host to Debian 13, which is not a thing to do to
a box running twelve GPU and audio services in order to slice a plate.
Reaching back to an Orca built for Ubuntu 22.04 would run, and would mean
pinning permanently to a stale build -- the exact trap brokkr-smithy-dev
rejected when they chose Orca over an old PrusaSlicer tag. The container keeps
the slicer current AND leaves the host alone, which is the same argument that
made FreeCAD an AppImage, taken one step further because the host could not
satisfy this one at all.
The AppImage is extracted at BUILD time; --appimage-extract-and-run re-extracts
to /tmp on every invocation, which is wasted seconds and wasted disk per slice.
~/bin/orca-slice wraps it so the dispatcher calls a binary and the artifact root
is mounted at the same path inside and out.
⚠ Recorded honestly: I installed libwebkit2gtk-4.1-0 and 8 dependencies on the
host chasing errors serially before enumerating with ldd, and only then found
the glibc wall behind them. Those nine packages are unnecessary. Left in place
rather than autoremoved on a box running production services; flagged for the
operator.
Operator: "release button covers delete button". Measured before touching
anything: release 58x24 at (323,266), × 30x30 at (349,268) — 30x22 px of
overlap on a 30px button, and `elementFromPoint` at the ×'s centre returned the
release form. The × I added yesterday was 100% unclickable from the moment it
shipped.
Cause: both were `position:absolute` on the same corner, each with its own
guessed offset, and `release` is the later sibling so it won. Replaced with one
flex row positioned once — release left, × right at the card corner where the
ephemeral lane's × already lives, so muscle memory transfers and neither can
drift back on top of the other when a label changes width.
Verified by measurement, not inspection: overlap 0 px, and clicks at each
control's centre now land on that control. The ephemeral lane's × and ★ were
re-checked and are unaffected.
ADDS scripts/layout-probe.py, because markup inspection STRUCTURALLY cannot see
this and I have now shipped two dead controls in two days by reading templates
instead of rendering them. It asks a real browser what a click at each
control's centre would hit.
It took four iterations to become trustworthy and the failures are the point:
1. `top.contains(el)` counted an ANCESTOR overlay as a hit -- the exact case
it exists to catch. Version one reported OK for a real overlay.
2. elementFromPoint is viewport-relative, so everything below the fold read
as occluded and buried the real findings.
3. getBoundingClientRect() on a WRAPPED INLINE element is the union of its
line boxes, whose centre lands in the gutter between them -- three zip
links reported occluded by their own parent. Real geometry, wrong question.
Only the fourth version fires on a genuine overlay while staying silent on the
clean page. Both controls were run; a probe never seen to fail proves nothing.
173 tests pass.
Three operator reports, one of them a real defect I had claimed was working.
THE REVEAL BUTTON DID NOTHING, for a day. Its handler sat after the content
block's closing tag, and a child template's out-of-block content is silently
DISCARDED by Jinja. The button rendered. The handler never reached the browser.
Two commits and a README paragraph said click-to-reveal worked, and the suite
passed the entire time because nothing asserted against the SERVED page -- the
template really did contain the code.
Two guards, both confirmed to FAIL when the defect is reintroduced rather than
merely added and assumed protective:
* test_reveal_handler_actually_reaches_the_served_page greps the response
* test_no_orphaned_markup_after_the_content_block guards the structure
While moving it, caught a second instance of the same class: the explanatory
comment I wrote for the fix contained a literal Jinja endblock tag, which Jinja
would have parsed as a real tag and used to close the block early.
KEPT-LANE ×. Wiping a kept booth required release-then-find-it-in-the-other-
lane. That protected nothing and cost a hunt -- the board you just released is
loose in a feed that turns over, and you have to go find it to finish a job you
had already decided on. Protection now lives in the confirmation, which names
the booth and says KEPT, instead of in the number of lanes you must traverse.
Release stays as the reversible option.
IN-BOOTH KEEP. `☆ keep` / `★ kept — release` beside "Wipe now", so promoting
does not mean navigating back to the index. The booth page did not previously
know its own kept state; it does now. Both post a `next` field to stay put --
and `next` is a form field, so it is attacker-controlled: only same-site
absolute paths are honoured, with `//host`, schemes and backslashes refused,
tested.
173 tests pass.
The operator asked "no UI option to blur/unblur?" and was right twice over.
MISSING ENTIRELY ON TWO OF THREE BRANCHES. booth.html renders docs, media and
everything-else through separate <figure> blocks. The toggle went into the
media branch only, so inline docs -- the branch that puts readable text straight
on the page, the one that needs blur most -- had no control at all, and `other`
files only got a caption row if they happened to carry a caption.
This is the SECOND time this feature shipped having patched some branches and
not others; the blurred class itself had the same gap one commit ago. So the
toggle is now a single Jinja macro called from all three sites, which makes
"patched two of three" impossible rather than merely unlikely, and
test_every_item_kind_gets_exactly_one_blur_toggle counts toggles against figures
across mixed kinds so a fourth branch cannot quietly skip it.
INVISIBLE WHERE IT DID RENDER. v1 was a bare `◌` at 0.78rem in --muted with no
border, no label and no hover affordance. It now reads `◌ blur` / `◉ blurred`
with a border, matching the other per-item controls. A control nobody can find
is a control that is not there.
Docs get it in the doc bar beside ⤢ ⬇ ✕, with stopPropagation so submitting it
does not collapse the <details> it lives inside.
Verified live on all three kinds: 3 figures, 3 toggles, and the POST round trip
blurs and un-blurs. 167 tests pass.
Two operator requests.
KEEP, BOTH DIRECTIONS. The kept lane could already release a booth back to
ephemeral, but an ephemeral booth could only be promoted from a shell -- so the
round trip was closed only if you had ssh. The /keep route and the `booth keep`
verb both already existed; only the button was missing. Adds ★ to the ephemeral
card, mirroring × on the other shoulder.
BLUR. Per-item cosmetic censoring: `booth blur <name> <file>...`, a ◌/◉ toggle
in each caption row, and 👁 click-to-reveal. State is `.blurred` in the booth
dir, one booth-relative path per line -- the same filesystem-is-the-state idiom
as .pins and .forever. An empty set deletes the marker rather than leaving a
zero-byte file, so `ls -a` tells the truth.
⚠ BLUR IS NOT ACCESS CONTROL, and the code, the docs and a test all say so on
purpose. A blurred item is still served at its own URL, still in the zip, still
on disk. The Booth has no auth by design. test_blur_is_cosmetic_the_file_is_
still_served asserts the 200 deliberately: if someone later "hardens" this into
a 403 that test fails, and it should, because half-implemented access control is
more dangerous than none.
Reveal is per-viewer and never persisted; a reload re-hides. With JS off an item
stays blurred, which is the safe direction to fail in.
Two things the first pass got wrong, both caught by checking rather than
assuming:
* The cover thumb. index.html has IDENTICAL markup in the kept and ephemeral
lanes, so a single-occurrence replace patched only the kept one and the
ephemeral front page happily displayed the thing someone had hidden. The
test that caught it was itself wrong first -- it matched the bare string
"blurred-thumb", which is in base.html's stylesheet on every page and so
passed in both states. It now asserts the attribute.
* Inline docs render through their own <figure> branch and were left
unblurred -- the branch that puts readable text straight on the page, so it
needed blur more than images do. The suite passed; a live curl caught it.
165 tests pass (154 pre-existing, unchanged).
vh
Vuong Hoang