The MCP server (mcp-for-blender 2.1.1, frozen requirements) runs inside the Blender
container. Its add-on is vendored at upstream 41a18432 (MIT) and started by a
startup hook. scripts/blender-mcp carries the stdio over ssh + docker exec, so the
add-on socket, which runs arbitrary Python with no auth, stays on the container's
localhost with no published port. It also runs there because viewport screenshots
need a filesystem shared by server and Blender. Telemetry is off and safe mode is
on. The hook also defaults Cycles to OptiX on GPU 3, because safe mode forbids
agents from touching preferences.
Verified end to end from nh3-dev: 36 tools; a GPU render of an agent-built scene;
a viewport screenshot; and safe mode refusing 'import os'. Blender left down
(on demand).
Uses linuxserver/blender (Selkies Wayland desktop, NVENC), digest-pinned. The
container sees only GPU 3, has restart "no", and runs only while in use,
because GPU 3 is the reserve card for a full-size vLLM seat. Web desktop on
:3001 with basic auth (vault fv-ml1/blender-web-password). /work is on /tank
and is not backed up; /config lives under /opt/docker (restic).
Acceptance: Cycles finds the card on OptiX and CUDA (sm_120 kernels ship in the
build). The heavy self-test renders in 3.54 s on OptiX vs 22.71 s on CPU, a
functional check with n=1. Web auth answers 401 without credentials and with a
wrong password, and 200 with the right one.
The shim holds 10.0.50.47/24, which gives two equal connected 10.0.50.0/24 routes,
and ens18's wins. Host-to-HA traffic therefore left via the macvlan parent and was
dropped. HA lost MQTT to the broker on this host on 2026-08-19, 2026-09-21 and
2026-09-25 (the last lasted two days). This adds an ifupdown if-up.d hook that
routes 10.0.50.46/32 via macvlan-shim; /etc/network/interfaces is not edited.
Verified: the route resolves via the shim, the host pings HA, HA reaches :1883,
and HA reconnected to the broker. Diagnosis by ha-dev.
Also corrects the zigbee2mqtt acceptance note, which had wrongly reported HA as
connected.
Requested by ha-dev; approved by Prime in this session. Radio: SLZB-MR1U chip 0
(EFR32MG21, EmberZNet 8.0.2) at tcp://10.0.90.10:6638, adapter ember. A fresh
network was formed on channel 25, PAN 0xCFF4. The broker is reached as mosquitto
user zigbee2mqtt, with HA discovery on homeassistant/. The frontend on :8099 is
token-protected.
State and the network key stay host-only in /opt/docker/data/zigbee2mqtt
(root 0700, restic). The repo carries only compose, .env.example and the README.
configuration.yaml refers to the secrets as !secret.yaml, and those references
survived Z2M's v4->v5 settings migration.
SemIf's shared scorer trims one token at the state boundary. When an object
state's last value ends in ')', ';' or '}', the JSON that follows re-merges two
tokens back, so score_shared refused the request with 422. The engine now wraps
semif_phase1.shared._state_prefix to keep only the tokens the full prompts
share. Each row scores the same token sequence; only the prefill/suffix split
moves.
Startup proves the fix is in effect, not just installed (heid bug hunt SKAL,
folded). It checks that the hook is callable and is what score_shared resolves,
that an ordinary state keeps upstream's whole prefix, and that a merge-prone
state scores through the shared path.
Real tokenizer: 154 states, 23 refused before and 0 after, with no ordinary or
authored144 prefix changed. Acceptance: 144/144 parity. Shared vs direct 71/72;
the miss is a bf16 tie that flipped across a plain restart (see README).
Against talk /face's guided pose (first paragraph 246 ms median), SemIf in
parallel adds 32 ms and SemIf-first adds 94 ms (n=72 each, noise floor 16.5 ms).
Removing the pose header saves only ~31 ms, and SemIf shares GPU 1 with the LLM.
Acceptable pose 67% vs 92% on clear-emotion lines, and the mood carried through
mundane follow-ups 7/15 vs 14/15. SemIf gestures far less (13% vs 58%).
README: rotations cost options^2 in suffix tokens, and /decide/shared returns
422 when an object state's last value ends in ) ; or }.
The comment claimed 'gateway-key-gated'; v1.97.0 actually gates auth=true
passthrough routes on per-key metadata allowed_passthrough_routes (OSS path,
not Enterprise), 401 unauthenticated. Grants applied live via /key/update for
all-agents-local and the worldtree gateway key; comment-only change here,
picks up with the next conf deploy.
Order averaging (Prime, after the 739aa03 spike):
- A decision may set orderings: rotations|all (all only for <= 4 options). Every
ordering goes to the engine in one shared batch.
- The reply keeps each native result and adds combined {probabilities (log-mean),
top, agreement, spread}.
- Through the service on SemIf's labelled sets (252 rows): 78.6% -> 88.1%
(group-bootstrap 95% CI +5.1..+14.3). Unanimous agreement is 94.5% accurate.
Fast kernels: flash-linear-attention 0.5.2 and causal-conv1d 1.7.0 are now the
default build. A/B on the empty GPU 3:
- parity with upstream went from 142/144 to 144/144;
- a ~2k-token /decide went from 169 to 92 ms server-side;
- short 3-rotation batches cost ~3-6 ms more.
triton builds a C shim at runtime, so the image carries gcc. Without it the
warm-up failed and startup failed closed.
Heid bug-hunt panel (4/4 arms, thread 01M3H3F4RR7XBP90KQ3A39H4SX), folded:
- Startup validation: VRAM cap 0 no longer means uncapped (C1); limits must be
>= 1 (S1); the token must be visible ASCII (S2); the calibration file must
exist and parse, with T in [0.05, 20] (S8, and C3's NaN leg).
- The body limit is checked before a chunk is kept, and a Unicode-digit
Content-Length no longer crashes (C2, S3).
- Failures while building the response now get the 500 envelope (C3).
- 429 busy past SEMIF_MAX_QUEUE requests in progress (C6).
- The engine releases memory on every non-validation failure, unchained after
gc; an empty OOM message is handled; 'out of memory' RuntimeErrors map to 503
(C4, C5, S9).
- The entry point forces HF_HUB_OFFLINE (S10). README wording fixed (S5, S6).
- New guard tests close the gaps the arms' mutation grids exposed: early stop of
the body read, a shared-route lock, calibration pass-through, the gc cycle,
the exact caps, TorchEngine.load's arch and device checks, and the offline
entry point.
86 tests.
Deployed on fv-ml1 GPU 1: parity 144/144, OOM and burst release verified, shared
capacity 63/51/26/16 rows at ~140/520/1960/3900 prefix tokens.
services/semif-serve is a FastAPI wrapper around SemIf's direct and shared torch
scorers (SemIf-OpenJev @ 23cf1f39, MIT). Upstream ships only a batch CLI. The
wrapper loads the pinned Qwen3.5-4B (851bf6e8, BF16) once from the offline HF
cache and returns SemIf's result dicts unchanged, with an optional per-workload
temperature-calibrated view. Contract: semif-serve.contract.md. Built with a
short contract, TDD (39 tests, fake engine and fake torch, no GPU) and a heid
bug-hunt panel (pending).
On the card:
- torch 2.10.0+cu128 with sm_120 kernels, which is SemIf's own stack;
- a hard 12 GiB VRAM cap.
Two defects surfaced only on the card, and each fix is covered by a test:
- 0.1.1: an OOM raised as a chained exception kept the failed request's tensors
alive (11.9 GiB after the 503). It is now raised unchained, after gc.
- 0.1.2: a large request left 12.6 GB reserved on the shared card. After each
call, reserved memory over the baseline + 512 MiB is now released.
Acceptance against SemIf's committed torch predictions (authored144):
- 142/144 same top choice; both misses are exact bf16 ties;
- 144/144 identical prompt hashes;
- deterministic A-vs-A;
- negative control 14/144;
- shared vs direct 72/72.
21 binary criteria over one state take 159 ms. The shared-mode capacity table
under the cap is in stacks/semif/README.md.
The Dockerfile installs dependencies from a manifest with the project version
blanked, so a version bump reuses the ~4 GB torch layer. Verified: 41 s rebuild,
dependency layer CACHED.
DNS: semif.fv.internal. Token: vault semif/api-token.
Prime bootstrapped infra-ops on vm-esh-nas with playbooks/bootstrap-infra-ops-user.yaml.
It got the fleet-pinned uid/gid 850, NOPASSWD sudo with log_output, the docker
group and a 0700 home. That let playbooks/restic-repository-file.yaml migrate
the last restic host: the live profile matched the repo's pre-change sha,
its units no longer carry the URL, its secrets are vaulted, and the live and repo
profiles now match (a5ea75ea). All eight restic hosts are clean.
The staged helper script is gone, both from Prime's home on the host and from the
repo. The docs that described vm-esh-nas as lkraven-only are updated.
resticprofile schedule copies env-file values into the generated units, which
are 0644, so RESTIC_REPOSITORY (the rest-server basic-auth password included)
was readable by every local user on every restic host.
New playbooks/restic-repository-file.yaml:
- derives /etc/restic/repository (root 0400) from restic.env;
- uploads the profile switched to repository-file, but only when the live
profile's sha matches the repo copy it was edited from (drift guard);
- checks the repository is reachable through the new profile (cat config);
- regenerates the units and verifies they exist and contain no rest:http.
Applied to ana-docker, fv-ml1 (configs/restic/ana-ml2), esh-docker-vm,
esh-vm-db, irv-ml1, nh3-dev and nh3-docker. An independent check across all
eight restic hosts (these seven plus esh-ml1) found 0 leaking units. nh3-docker's
scheduled unit ran a real backup afterwards (snapshot a29b889d). Each host's
URL and passphrase are vaulted as <host>/etc/restic/{repository,password}.
restic.env is kept (root 0600) because the per-host READMEs and the freshness
probe source it. A rotation must update the vault, restic.env and repository.
vm-esh-nas has no infra-ops account. Its in-place migration script is staged
for Prime to run with sudo, and its repo profile is pre-edited to match.
Also mirrors augaman-dev's 401df2d (compose header only). The config hash on
esh-ml1 is unchanged.
Both hosts had live improvements the repo never recorded:
- esh-docker-vm excludes ESPHome's 539 MB of PlatformIO cache (2026-09-14).
- irv-ml1 runs /etc/restic/arbo-checkpoint.sh before the backup, a non-fatal
SQLite online-backup of arbo's gallery DB. That script is added here too.
The live copies were correct and are the source for the next change. (Also fixes
the fv-ml1 augaman removal time to ~0130 PT.)
Prime removed the second instance after the v0.1.3 bench. esh-ml1 handles a face
in ~48 ms, sits in the house next to the cameras, and holds the verified
backup. fv-ml1's gallery was empty (0 identities). The container, gallery volume,
image, compose dir (with its .env), backup dir and build sources are removed from
fv-ml1. GPU_ID / CARD_SUFFIX stay in the compose for any future second host.
Both hosts are rebuilt from tag v0.1.3 (one ONNX session per detector canvas) and
redeployed. pytest -m gpu tests/vision passes 3/3 on each card.
Server-side for one face, same harness as the v0.1.2 baseline:
- esh-ml1 GPU 144 -> 48 ms
- fv-ml1 GPU 75 -> 27 ms
End to end from nh3-dev: 101.6 and 73.8 ms.
CPU mode got slower on every CPU target: fv-ml1 cpuset 0-5 went 152 -> 205 ms
with a face, and the no-face frame roughly doubled. That is well outside the
run-to-run spread. The suspected cause (not measured) is per-session ORT
thread pools spinning. Reported to augaman-dev. Neither deployment uses CPU
mode.
On esh-ml1 the dependency layer missed the build cache and the rootfs touched
90% until the v0.1.2 image was removed. fv-ml1's build hit the cache, and the
exported requirements are identical, so the stack README now says to check
disk before building on esh-ml1.
Prime asked for augaman on fv-ml1's utility card, beside vllm-coder. Mirror
augaman-dev's f77164f compose, which parameterises the GPU reservation (GPU_ID,
default 0) and the Homepage card name (CARD_SUFFIX). esh-ml1's resolved config is
unchanged: same config hash, no recreate.
On fv-ml1: augaman:0.1.2 built on-box from the tag, GPU_ID=1, healthy on CUDA
at 1264 MiB, and pytest -m gpu tests/vision passes 3/3 on the Blackwell. It has
its own gallery and no gallery backup, so it is fixtures-only. The host's raw
restic copy of /var/lib/docker/volumes is not a consistent SQLite backup.
docs/pfi/augaman-speed-bench/ holds the harness (augaman-dev's recipe plus a
no-face control frame and a face-count check on every response), the raw rows
and the summary. Server-side, one face:
- esh-ml1 GPU 144 ms
- fv-ml1 GPU 75 ms
- fv-ml1 CPU on 6 cores 152 ms
- esh-ml1 CPU 888 ms
It agrees with augaman-dev's independent esh-ml1 measurement once each
harness's floor is subtracted. This is the before for v0.1.3's detector fix.
esh-ml1 is outside vzdump, so augaman's face gallery reaches backup only
through restic. New playbooks/esh-ml1-restic.yaml installs restic 0.14.0 (the
same Debian package as the other ESH hosts) and resticprofile 0.33.1 (pinned,
sha256-checked). It uploads configs/restic/esh-ml1/ and schedules a daily
0100 PT backup plus a Sunday 0500 PT check to rest-server-ana. The CT runs UTC,
so both schedules name the zone explicitly.
pre-backup.sh is fail-closed: it runs augaman's own backup CLI, and any failure,
including a stopped container, aborts the run. Tested with a stub docker that
exits 1: the run returned 1, and neither the snapshot count nor last-success
moved. The restore was verified at identity level against augaman-dev's
public-domain canary (snapshot fd3061a1: the restored copy's digest over
identities and samples matches the live gallery). That meets the operator gate
for real enrollments.
The repository URL is read through repository-file rather than restic.env.
resticprofile schedule copies env-file values into world-readable systemd
units, which publishes the rest-server password on the env-file hosts
(observed on esh-docker-vm). This is recorded in the backups runbook under
Known gaps, and the playbook verifies no generated unit contains the URL.
esh-ml1 is added to the freshness check's expected ana-side repos and to the
runbook tables.
augaman moves to v0.1.2 (dependency layer keyed on the lock without the
project; per-crop embedding). pytest -m gpu tests/vision passes 3/3 on the
card, and the canary survived the container recreate.
Mirror pfi/augaman deploy/compose.yaml as stacks/augaman, with an .env.example and
a README carrying the biometric backup gate. The image is built on esh-ml1 from a
git archive of the release tag, because the box holds no gitea credentials.
Serving on CUDA and visible in nvidia-smi. The gallery backup is not wired yet
(esh-ml1 has no restic), so only public-domain fixtures may be enrolled.
The on-box gpu test fails its batch-vs-single tolerance 3/3; reported to
augaman-dev, who owns the contract.
For ha-dev (operator-approved 2026-09-26). CT 111 on esh-pve at 10.0.90.20:
Matter/Thread IPv6 (Echo ULA + RA route-information) is link-only, so the
server sits on esh-iot and HA reaches it over routed IPv4 ws :5580.
- playbooks/esh-matter-lxc.yaml: kernel RA (accept_ra=1,
rt_info_max_plen=64), forwarding off, Docker ip-forward/iptables off;
nftables admits 5580 from HA 10.0.50.46 only and SSH from mgmt ranges;
the CT is added to esh-pve's vzdump job (fabric credentials).
- stacks/matter-server: ghcr.io/matter-js/matterjs-server:1.4.0 (digest),
host networking, /data on the CT.
- Acceptance: fdad:: SLAAC, ping6 thermostat, 2 Thread RIO routes learned, ws
server_info from inside the HA container; 5580 refused from 10.0.50.45,
nh3-dev and a temporary VLAN 90 netns vantage.
- qwen3-embedding and reranker: second deployment on nh3-ml1 (config);
reranker-a3-bge-v2-m3: second DB deployment via /model/new.
- router_settings.enable_weighted_failover: true. Only the three
multi-deployment groups are affected.
- Measured by stopping nh3-ml1 TEI: without failover 7/40 embeds 500'd;
with it rerank 80/80, embed 38/40 at onset and 60/60 over a 34 s outage;
nh3 rejoins rotation after restart. Split 12/28 embed, 22/18 rerank.
For brokkr's dataset foundry (operator-approved 2026-09-26, relayed).
- stacks/lfm-vl-seat: llama.cpp server-cuda b11176 (digest-pinned), Q5_K_M +
mmproj Q8_0, :8030; gateway alias lfm25-vl-3b (LiteLLM restarted, 36 s).
Positive control exact; null control shows it describes a missing image.
- stacks/vibevoice-asr-seat: audio.cpp v0.8.2-audio8-perf-hotfix (the GGUF's
own runtime, not vibevoice.cpp) on cuda 12.8 runtime + libgomp + libsoxr,
sha256-pinned; :8031 direct. LibriSpeech WER 3/69, RTF 0.07-0.14; ~31 s
cold first request.
Same model revision (df3ce67, blob sha matches fv-ml1), vLLM v0.24.0 and flags;
0.33 of the Ada (~5.4 GB, fv-ml1's absolute budget). Healthy, KV 79,808 tokens.
Parity (teacher-forced true code, 40 FIM prompts, 2,215 tokens, cache-salted):
each host is bit-exact with itself; cross-host |dlogprob| median 0.049, top-1
agreement 0.966 (Blackwell vs Ada + fp8 KV); against ground truth no quality
difference (mean logprob diff +0.008 +/- 0.019 SE). Speed on-box: ~5x slower
(64-tok FIM p50 ~1.0 s vs ~0.2 s; 63 vs 338 tok/s). Gateway coder-fast left on
fv-ml1 pending Prime's call.
- PFI-UDMSE port 6 override: native nh3-mgmt, tagged VLANs blocked (was
forward all / native default). Reservation nh3-pve-amt -> 10.100.250.61.
- nh3-pve: arp_ignore=8 / arp_announce=2 on enp88s0 (now on vmbr0's untagged
L2) so the host never answers ARP for 10.100.250.60 with the AMT port's MAC.
- DNS nh3-pve-amt.nh3.internal -> 10.100.250.61.
- Homepage: NH3-PVE-AMT card under Infra - NH3 (no siteMonitor/ping: AMT drops
ICMP and its legacy-renegotiation TLS fails Homepage's fetch).
- AMT keeps its old 10.100.0.151 lease until rebind/expiry (~1920-2224 PT
2026-09-26); it does not re-DHCP on a VLAN change or link drop (measured).
nh3-ml1's first Temperature alert (2026-09-25 2105, 88.8 C) was nh3-pve's CPU
package during the nightly vzdump; the GPU sat at 47 C. hwmon is not namespaced,
so an LXC's agent reports its host's coretemp sensors under the LXC's name.
- hosts/{nh3,esh}-ml1.yaml: SENSORS=-coretemp_*,acpitz (blacklist); GPU and
NVMe temperatures verified still reported.
- Hub: Temperature >95 C / 5 m on nh3-pve and esh-pve for the CPUs (i9-13900H
TjMax 100 C; nh3-pve 2-hour averages reach 88 C on busy days).
- nh3-pve README: runs warmer than esh-pve; workload confounds it, check
airflow on the next visit.
NH3 site visit done: Secure Boot off, iGPU restored as boot VGA, AMT port cabled.
- nh3-pve: NVIDIA 580.178.04 (DKMS, open modules) via pve-nvidia-host.yaml.
- nh3-ml1 = CT 109 @ 10.100.50.80 via gpu-lxc.yaml; embed-rerank (TEI 1.9.4)
deployed with HOST_NAME/HOST_IP labels.
- Parity vs esh-ml1 (1,126 texts, 2 runs/host, controls): embed cosine min
0.999993 = own noise floor; overlap@10 1.000 vs MRL-256 positive control
0.684; rerank top-1 1.00, max diff 0.0014 vs floor 0.0020. On-box speed
identical within rep spread.
- gpu-lxc.yaml: first step upgrades lxc-pve to >= 6.0.0-2 (Proxmox fix#7006).
With 6.0.0-1 every docker run in a nesting CT failed on runc 1.5's sysctl
reopen; applied on nh3-pve (one package).
- pve-nvidia-host.yaml: document that the headers meta drags in the newest
kernel (nh3-pve went 6.8.12-11 -> -43 at the next reboot).
- Monitoring: Beszel NVIDIA agent + 5 alerts, Kuma #29/#30, Homepage
nh3-ml1-docker, Dozzle agent (hub 8 clients). DNS nh3-ml1.nh3.internal.
- nh3-pve README: SB/IGFX/driver/kernel state, btmtk oops on -4x kernels,
AMT cabled but unreachable on the network.
Gateway routing to nh3-ml1 is not changed.
- esh-pve-nvidia-host -> pve-nvidia-host: headers/dkms/build-essential step,
nouveau blacklist + guarded unload (refuses if nouveau bound a device)
- esh-ml1-lxc -> gpu-lxc: host vars have no defaults (elway aborts on undefined),
rootfs storage/startup order parameterized, CT kept out of all-guests vzdump jobs
- embed-rerank: Homepage labels take HOST_NAME/HOST_IP, defaults = esh-ml1
nh3-pve and esh-pve are the same Minisforum MS-01 (BIOS AHWSA.1.17). With a
card in the x16 slot its root port takes bus 01 and every NIC moves down a
bus (measured on esh-pve), so predictable names change (enp2s0f0np0 ->
enp3s0f0np0 etc.) and vmbr0 would boot with no uplink. systemd .link files
now pin all NICs by MAC, baked into every initramfs and synced to the ESP;
udev confirms the files apply. The AMT-capable I226-LM (enp88s0) leaves
vmbr0's bridge-ports in the file (next boot), so cabling it for AMT cannot
loop the STP-less bridge.
Also: documented the NanoKVM (https://10.100.250.171) as nh3-pve's console
OOB and that AMT is not wired; nh3-dev's Beszel agent no longer binds NAS
shares (it died on the last NH3 cold start); post-boot checklist in
persistent-memory.
- Beszel: NVIDIA agent (stacks/beszel hosts/esh-ml1.yaml), hub system
registered with Status/Disk/CPU/Memory/Temperature alerts; GPU samples
verified (RTX 2000E Ada util, VRAM, power, temp).
- Uptime Kuma: health monitors for the TEI embed (:8001) and rerank (:8013)
services — the sole backends behind the gateway, so the one exception to
"seats are out of Kuma's lane". Reward seat excluded (no consumer).
- Homepage: dockerd on tcp/2375 bound to 10.0.50.80 (playbook step);
esh-ml1-docker added to docker.yaml.
- Dozzle: agent v10.4.1 on esh-ml1. The hub's DOZZLE_REMOTE_AGENT still
named ana-ml2's 10.250.50.54 and irv-ml1's retired 10.100.79.3; repointed
to 10.251.50.54 / 10.6.110.50 (fv-ml1 and irv-ml1 logs were missing).
- Documented nh3-dev's Beszel agent failing on NH3 cold start (bind of an
automounted NAS share); revived by hand, fix still open.
- Audit: Skywork-Reward-V2-Llama-3.1-8B is still #1 of 188 on AllenAI's
RewardBench 2 per-sample results; no Skywork V3; the -40M sibling is
vendor-marked experimental. Our AWQ W4A16 quant: 0.847 vs published bf16
0.860 on a 150-prompt sample (within +/-2.9 pt SE), 96.2% pairwise
agreement. Double BOS from vLLM on pre-templated text costs a further
~2.7 pts; callers must send add_special_tokens=false.
- No working consumer: 0 requests since 2026-09-13; Worldtree Domari points
at a dead IP with a non-vLLM schema (reported to worldtree-dev).
- Move: sha256-identical model copy; vLLM v0.24.0 on esh-ml1 :8003 at 0.55
util (KV 1.30x of a 16k request). Parity vs fv-ml1: 149/150 verdicts,
99.8% pairwise signs, raw |delta| median 0.049.
- Gateway /scalar-judge passthrough -> 10.0.50.80:8003; fv-ml1 vllm-reward
removed (~10.2 GB freed on GPU 1). stacks/vllm now holds only vllm-coder.
Prime, 2026-09-25: TEI serves embedding + reranking for esh-ml1 and the fleet
from now on; fv-ml1 retires both once esh-ml1 is up.
- stacks/embed-rerank: vLLM -> TEI 1.9.4 (89- Ada build), same ports
8001/8013, fail-closed truncation (--auto-truncate false; embed
--max-batch-tokens 32768).
- litellm: qwen3-embedding -> esh-ml1 only (hosted_vllm/, unchanged address);
reranker -> huggingface/ provider at :8013 (hosted_vllm/ 422s on TEI's
`texts` body). DB alias reranker-a3-bge-v2-m3 patched to the same target.
- Verified via the gateway against the retiring fv-ml1 seats: embed cosine
median 0.999927 (n=203); rerank top-1/top-3 29/30.
- stacks/vllm: vllm-embed and vllm-rerank-a3 removed (containers retired on
fv-ml1, GPU 1 freed ~6.1 GB); reward + coder unchanged.
- Bake-off record moved to docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md;
CLAUDE.md gains the TEI convention.
esh-pve: NVIDIA 580.178.04 (open modules, DKMS) installed on the host from
NVIDIA's .run and loaded live, no reboot. nvidia-persistenced unit creates the
device nodes before pve-guests; the T400's vfio-pci ids and `blacklist nvidia`
retired. playbooks/esh-pve-nvidia-host.yaml.
esh-ml1: CT 110, unprivileged Debian 12, 10.0.50.80, GPU nodes via devN,
NVIDIA userspace from the same .run (--no-kernel-modules), docker-ce +
nvidia-container-toolkit (no-cgroups). playbooks/esh-ml1-lxc.yaml. Not in the
vzdump job on purpose. DNS esh-ml1.esh.internal.
stacks/embed-rerank: Qwen3-Embedding-0.6B :8001 + bge-reranker-v2-m3 :8013 on
the same vLLM v0.24.0 digest and flags as fv-ml1. Measured parity: embed
cosine FV-vs-ESH median 0.999908 (min 0.999772), inside both self-noise
floors; rerank max |delta| 0.000145 vs floor 0.000181, identical ranking.
litellm: qwen3-embedding and reranker gain an esh-ml1 deployment at order 2
behind fv-ml1 (order 1). Order fallback proven with throwaway groups:
refused primary +0.15 s, host-down primary ~18.7 s per call, dead-only 500.
Also: repaired the DB-only alias reranker-a3-bge-v2-m3, dead since the
fv-ml1 relocation (still named 10.250.50.54); documented the third
unkillable homepage wedge on esh-docker-vm.
Operator ruling 2026-09-24. On ana-docker the stack is `docker compose
down`: the container is removed and port 7878 is closed. Kept for revival:
- the data dir /opt/docker/conf/task-board/data (tasks.db, last written
2026-09-11)
- the task-board:local image
- stacks/task-board/ and the host's compose + .env
The Uptime Kuma monitor (id 3) was deleted before the stop so it could
not page, and its row is removed from monitors.yaml. Homepage drops the
card on its own, since it reads the container's labels.
Hooks: the container log showed no hook POSTs in 30 days. The only
traffic was open browser tabs holding /events, and the plugin was already
uninstalled on nh3-dev. Removed the paragraph that told sessions to call
task_* tools (CLAUDE.md, and the fork-fleet.sh template that seeds new
repos), the vestigial TASK_BOARD_SESSION env in .claude/settings.json, and
the listings in README and FLEETTOOLS.
svos-dev's call and the better one: The High Seat is the English name for
Hlidskjalf, the seat Odin watches all the worlds from, and watching every
session at once is what the board does. mdi-monitor-dashboard described the
artifact; the eye describes the job.
Deployed and verified in /api/services, not assumed.
Requested by svos-dev relaying the operator, 2026-09-22. Reversible work, so
the relay is fine to act on without escalating.
Manual services.yaml entry rather than container labels, because SVOS is a
user-level systemd unit (svos.service) on nh3-dev and nh3-dev is NOT one of
the five hosts in docker.yaml -- Homepage has no Docker API to discover it
through. Same reason the Booth, WhereTF, talk and the infra-hermes seat are
listed by hand, and the comment says so at the entry.
⚠ siteMonitor is "/" deliberately. There is no /api/health on this service:
that path 404s, and a monitor pointed at it would report the board
permanently down while it serves perfectly. svos-dev flagged it and it is
verified here -- / returns 200 and serves the SPA (<title>The High Seat</title>).
Group is Apps, which exists in settings.yaml's layout with tab: Main. An
invented group name gets no tab and renders on ALL tabs, which is how
Scriberr's "AI Systems" leaked across the whole dashboard in August.
Icon mdi-monitor-dashboard is my choice -- svos-dev explicitly did not guess
at one and offered to take a different suggestion.
Verified in a browser, not just in the API: the card renders in Apps with a
green site-monitor at 28 ms.
NAMES. Homepage already answers "what is this service called", so the monitor
name is now that name verbatim -- a second naming authority is how drift starts,
and an alert reading "[Uptime Kuma] Beszel hub is DOWN" sends you hunting for a
card that does not exist. Only two rows moved (Beszel hub -> Beszel, Dozzle hub
-> Dozzle); the " hub" suffixes were mine, not the services'.
The remaining mixed case is deliberate and is now documented as such. talk, vor
and task-board are lowercase on Homepage and in their own repos; title-casing
them here would make this board disagree with both. What actually looked messy
was scripts/kuma's own ASCII-ordinal sort, which buried every lowercase name
below every capitalised one. Fixed to case-insensitive.
⚠ RENAME SAFETY, which this pass needed and did not have. The seed keys on NAME,
so editing a name would have read as a brand-new monitor: added fresh, with the
old row orphaned, still checking, still alerting, and holding all the history.
`rename_from:` names the old row for one run. Verified: both renamed monitors
kept their IDs and all 67 heartbeats.
Added with it, an orphan warning for any row on the board the spec no longer
names -- because a forgotten monitor keeps paging. Its first cut diffed against
the PRE-EDIT snapshot and so cried wolf on its own successful renames; it
re-reads the board now. A warning that fires on its own correct work is worse
than no warning.
STATUS PAGE + WIDGET. The Homepage uptimekuma widget reads a PUBLISHED status
page (/api/status-page/<slug>), not the admin API -- which is why the widget
labels were deliberately absent from the rebuild: a dashboard widget pointed at
a 404 is the suspected mechanism behind both of Homepage's unkillable D-state
wedges, so shipping one on purpose would have been daft.
The page now exists at slug `nethealth` (the pre-rebuild slug, so old references
still resolve) and is DECLARED IN monitors.yaml, applied by `kuma seed`. Same
principle as the notification channel: a from-scratch rebuild restores the page,
the channel and the monitors together, and nothing the widget depends on lives
only in Kuma's database. Verified in a browser: "13 SITES UP / 0 SITES DOWN /
100% UPTIME" on the dashboard.
⚠ saveStatusPage calls imgDataUrl.startsWith() unconditionally, so passing null
throws and leaves the page CREATED BUT EMPTY -- which reads as success from
/api/status-page (200, correct title) while the group list is silently blank.
Pass "" instead. Commented at the call site.
THE BRIDGE. `beszel-althing` hardcoded a "[Beszel] " subject prefix and a
Beszel hub footer from when Beszel was its only caller. Routing Uptime Kuma
through it unchanged would have delivered Kuma outages labelled [Beszel],
pointing the reader at the wrong dashboard -- an alert that lies about its own
source is worse than no alert.
Now a route registry: /beszel and /kuma, each with its own prefix, footer and
payload parser, because the tools do not agree on a shape (Beszel sends
{title, message}; Kuma sends {heartbeat, monitor, msg}). Generalising cost a
dict; a sibling service would have cost a second unit, a second port and a
second thing to notice had died.
Renamed beszel-althing -> althing-alert-bridge with it. A service named after
one consumer that carries two is the invisible coupling that sends a future
session looking in the wrong place.
⚠ /beszel IS FROZEN and this refactor proves it rather than claiming it. The
three original tests were kept BYTE-UNCHANGED -- including the one asserting
the exact postbox argv -- and deliver() still defaults to the Beszel route so
they exercise it. A new test asserts the Kuma footer never leaks into a Beszel
body or vice versa. Verified live after the rename: a real POST to /beszel
landed as "[Beszel] BRIDGE RENAME CHECK" with the correct hub footer, read back
from the thread rather than trusted from the receipt.
Payload shapes are parsed HERE, not via Kuma's custom-webhook-body feature,
because Kuma's notification config lives in its own database -- and that
database was destroyed and rebuilt from scratch hours ago. Anything living only
in a tool's DB is lost on the next rebuild; format knowledge belongs in git,
next to a test.
parse_kuma also handles the monitorless case. testNotification and cert-expiry
alerts carry no monitor and no heartbeat, and the first cut fabricated "unknown
monitor is ?" from them -- caught by sending a real one and reading the subject,
not by the suite. Fixed, pinned, and the earlier test asserting the bad
behaviour was corrected rather than worked around.
KUMA IS NOW WIRED. scripts/kuma gained notification support and the channel is
in monitors.yaml, seeded BEFORE the monitors and with applyExisting, so a
rebuild restores alerting and not just detection. Ground truth from the DB:
13 of 13 monitors carry the channel.
⚠ A THIRD instance of the same class of bug, worth naming: notifications() is
pushed as `notificationList` at LOGIN ONLY -- there is no event to ask with. The
first cut cleared the captured value before waiting, discarding the only copy it
would ever be sent, then blocked for the full timeout and reported an empty
list. That reads exactly like "no channels configured" and is a lie. Same family
as the getMonitorList ack-vs-push trap, different shape.
End-to-end, both shapes, read back from the inbox:
[Uptime Kuma] Homepage is DOWN + target + board link
[Uptime Kuma] althing (infra-ops) Testing (no fabricated subject)
[Beszel] BRIDGE RENAME CHECK + hub footer, unchanged
ALTHING CHAMBER RETIRED (operator). Three of its four containers had never
started -- created 2026-09-19, StartedAt epoch-zero, 0 restarts -- so :7881
refused, and nothing was watching it. Only its valkey was running, on the
project's own network with no external consumer. Stack, compose/build/conf dirs
and the local image removed; the Homepage card went with the label.
Homepage sat dead for three days in September while every monitoring tool
reported correctly. Beszel said its host was up -- it was. Uptime Kuma was not
watching it. The dashboard fell into the seam between two working instruments.
Measured before changing anything:
- Beszel: 18 hosts x {Status, CPU, Memory, Disk, Temperature}. Its alerts
table is (system, name, value, min) -- there is NO url column, so it is
structurally incapable of "this endpoint should return 200". Not a config
gap; the data model.
- Uptime Kuma: 6 rows, 2 of them folders. Four real monitors, all firewalls.
- So the two are NOT redundant. They are disjoint, and the service layer
between them was empty.
REBUILT FROM SCRATCH, operator-authorised ("uptime-kuma was never really
used... you can even dump the existing container and config"). Nothing was
migrated, which also skipped the one-way v1->v2 database migration.
- Pinned to 2.5.5. `:latest` is a documented trap now: upstream keeps it on
the 1.x line, so an August 2026 pull produced an image BUILT 2024-12-20
running 1.23.16. Verified by digest -- latest and 1 share one digest while
2/next carry 2.5.5. Pinned exactly, not floating on 2, for the same reason.
- Moved esh-docker-vm -> ana-docker. House placement rule puts cross-site
services beside the Beszel and Dozzle hubs, and esh-docker-vm has wedged
unkillably twice in four months. A monitor also cannot report the failure
of the host it runs on, so it should not share a failure domain with the
host layer.
- Normalised restart: always -> unless-stopped, which the 2026-08-18 README
flagged as worth doing on the next deliberate touch.
- UPTIME_KUMA_DB_TYPE=sqlite in the compose skips 2.x's interactive database
screen, so the stack comes up ready rather than parked on a form.
scripts/kuma is a first-party Socket.IO client, because both obvious paths are
wrong: there is no REST CRUD API in EITHER major version (server/routers/ holds
exactly two files, /metrics + badges + status pages), and the community wrapper
uptime-kuma-api is abandoned -- last release 2023-09-26, ceiling 1.23.1, no 2.x
support ever.
⚠ getMonitorList's callback returns only {ok:true}; the list arrives as a
SEPARATE pushed monitorList event. Reading the ack yields an empty board that
looks authoritative -- which duplicated all 13 rows on the first re-seed
before the bug was found. The client now waits for the push, and carries a
dedupe verb because of it.
13 monitors seeded from monitors.yaml, keyed on name so a re-run updates rather
than forking the board -- proven by re-running it (0 added, 13 updated), not
assumed. Every URL was probed before being written: all 200. A board that ships
red teaches everyone to ignore it.
Verified: 13 rows, no duplicates, all UP with "200 - OK" read from the database
WITH its WAL (a first read of kuma.db alone showed a stale 26 -- the copy
predated the deletes). Homepage renders exactly one Uptime Kuma card.
⚠ NOT YET WIRED: notification delivery. The board detects but tells nobody,
which is the same gap this work exists to close. The beszel-althing bridge
hardcodes a [Beszel] subject prefix and hub footer, so routing Kuma through it
unchanged would mislabel the alerts. Needs a decision before it is generalised.
Found while smoke-testing the lv-mccarthy ship. Measured live:
default -> '<think>\n\n</think>\n\nThere were no horses in the road...'
enable_thinking=false -> 'The sun was hot on the dry riverbed and the stones were red...'
This is the Qwen3-4B-Instruct CHAT TEMPLATE, not an adapter property, so it applies
to lv-yarros, lv-bronte and lv-hemingway equally and has done since this seat went
up on 2026-09-16.
No gate number is affected: gen_beats_chat_yarros.py sets enable_thinking when the
template supports it, so every arm in every r49 gate was generated without the tags.
But a caller that does not pass chat_template_kwargs gets 17 junk characters at the
head of every passage -- and any word-count or in-band check run over that string is
counting the tags as prose. Skaldsong should be checked.
Shipped on the operator's standing authorisation, "ship it if the gate passes".
The gate design of record for this line -- the operator's own v2 rule, ratified
2026-09-15 -- passes on all three axes:
VOICE +0.177 delta_cb at 3.2x the pairwise floor, the BEST arm of five,
and +0.128 at 2.8x on the pre-registered punctuation-stripped
secondary read, so ~3/4 of the gain is words rather than marks.
Closed 60.8% of the reachable span on an axis deliberately made
harder than Bronte's or Hemingway's.
NOT COPIED 0.12 hit-rate against HELD-OUT McCARTHY'S OWN 0.12 -- 1.0x the
author -- with a longest verbatim run of 10 words against the 12
genuinely unseen McCarthy shares with the train split by
coincidence. All 31 matches READ: stock grammar, name-shaped hits
are the renamed inventions, nothing protectable. This matters:
McCarthy is in copyright with a living estate.
NO DAMAGE ran-on +0.12 against the 0.200 floor, 40% headroom, and the
measured minimum across the five-arm ladder.
ckpt300 over ckpt900, the eval-loss minimum: the two-epoch recipe is now 0 for 3,
and this time the curve was CONFIDENT and wrong -- ckpt900 is the minimum and
ckpt300 sits +15.8x the median neighbour jitter above it, yet ckpt300 wins every
axis that resolves (3.2x vs 1.2x voice margin, 2.7x tighter seed spread,
memorisation 1.0x the author vs 1.8x, ran-on 0.13 vs 0.28). On this schedule read
the axes, not the curve.
THE COST IS REAL AND IS WRITTEN INTO THE COMPOSE, THE ADAPTER README AND THE
MEMORY RECORD RATHER THAN GLOSSED. In-band is 0.65 against base's 0.89 and on-beat
0.42 against 0.71 -- 35% of generations miss the requested 90-140 word band against
base's 11%. Axis C is ran-on only and is structurally blind to this; that blindness
was identified and written down before these numbers existed. No checkpoint choice
fixes it: every adapted arm is 33-48% out-of-band and ran-on is non-monotonic in
epoch. If a caller depends on a word budget, expect one generation in three to miss
it, and the fix is a retrain targeting length rather than a different checkpoint.
Rollback is one line and 0.003 s: drop the --lora-modules entry, or POST
/v1/unload_lora_adapter. The other three voices are untouched.
Canonical pin bump for the 3.7.0 rollout forseti approved and released
(althing 6db955f, tag v3.7.0). Image built from that tree and pushed as
sha256:df0709b3f532557aebb47d6ece486ae32e851776f732862094608d55eefcd4e7;
previous pin was 3.6.3@sha256:978f8553 and remains the rollback target.
Content-verified rather than tag-verified: `postbox --version` inside the
built image reports 3.7.0 before it was pushed, and again inside the running
container after. A tag is a claim.
Also adds playbooks/nh3-extdev-althing-upgrade.yaml -- a version-parameterised
upgrade for the box a git pull cannot move. The existing
nh3-extdev-althing-v3.yaml is the historical v2->v3 CUTOVER and is left alone
rather than mutated per release; it still carries v2-retirement steps that
document what happened.
Every check in the new playbook asserts what the installed entry point
reports, including one that the herald restarted within the last 5 minutes
rather than surviving from before -- "active" does not mean "running the code
you just installed".
Picks up uncommitted searxng changes left by another session and makes them
truthful rather than committing them as written. The stack itself verifies
clean: canonical and live are byte-identical for both compose.yaml and
searxng-settings.yml, the container is running with zero restarts, and live
queries return 51-54 results from 5-6 engines with braveapi contributing 20
each time.
compose.yaml gains SEARXNG_BRAVE_API_KEY, which NOTHING READS. It was added on
the belief that settings.yml could pull it via `!ENV SEARXNG_BRAVE_API_KEY`;
this build has no !ENV YAML constructor, so that attempt made the file
unparseable and crash-looped the container ten times with fleet search down.
The comment claiming the variable is "consumed by settings.yml" is replaced
with what is actually true. The variable is kept, unused, in case upstream ever
gains env interpolation — a comment that lies is worse than a variable that
does nothing.
The Tier A playbook is marked superseded FOR THE SETTINGS FILE ONLY, and
scoped deliberately: its v4 design uploads a settings file carrying the !ENV
tag, which would re-break the container, so settings deployment goes through
scripts/deploy-stack.sh like every other stack. Its .env merge and
up-d-not-restart steps remain useful, as do its two warnings recording real
bugs it hit — a wholesale .env overwrite that clobbered SEARXNG_SECRET, and a
sed that inserted literal backslash-n into compose.yaml. An unscoped
"superseded" banner would have buried those; that failure mode cost an outage
earlier today.
Also folds in the regenerated graphify report.
vh
Vuong Hoang