0f032fac2c65a2b55514cb0b8370dfdb7140c03d
426
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 vh
|
6d901c0e8d |
fix(beszel): GPU-LXC temperature alerts watch the GPU, not the hypervisor CPU
nh3-ml1's first Temperature alert (2026-09-25 2105, 88.8 C) was nh3-pve's CPU
package during the nightly vzdump; the GPU sat at 47 C. hwmon is not namespaced,
so an LXC's agent reports its host's coretemp sensors under the LXC's name.
- hosts/{nh3,esh}-ml1.yaml: SENSORS=-coretemp_*,acpitz (blacklist); GPU and
NVMe temperatures verified still reported.
- Hub: Temperature >95 C / 5 m on nh3-pve and esh-pve for the CPUs (i9-13900H
TjMax 100 C; nh3-pve 2-hour averages reach 88 C on busy days).
- nh3-pve README: runs warmer than esh-pve; workload confounds it, check
airflow on the next visit.
|
||
|
vh
|
5960526c3f |
feat(nh3-ml1): second TEI embed/rerank backend live on nh3-pve; parity-verified vs esh-ml1
NH3 site visit done: Secure Boot off, iGPU restored as boot VGA, AMT port cabled. - nh3-pve: NVIDIA 580.178.04 (DKMS, open modules) via pve-nvidia-host.yaml. - nh3-ml1 = CT 109 @ 10.100.50.80 via gpu-lxc.yaml; embed-rerank (TEI 1.9.4) deployed with HOST_NAME/HOST_IP labels. - Parity vs esh-ml1 (1,126 texts, 2 runs/host, controls): embed cosine min 0.999993 = own noise floor; overlap@10 1.000 vs MRL-256 positive control 0.684; rerank top-1 1.00, max diff 0.0014 vs floor 0.0020. On-box speed identical within rep spread. - gpu-lxc.yaml: first step upgrades lxc-pve to >= 6.0.0-2 (Proxmox fix #7006). With 6.0.0-1 every docker run in a nesting CT failed on runc 1.5's sysctl reopen; applied on nh3-pve (one package). - pve-nvidia-host.yaml: document that the headers meta drags in the newest kernel (nh3-pve went 6.8.12-11 -> -43 at the next reboot). - Monitoring: Beszel NVIDIA agent + 5 alerts, Kuma #29/#30, Homepage nh3-ml1-docker, Dozzle agent (hub 8 clients). DNS nh3-ml1.nh3.internal. - nh3-pve README: SB/IGFX/driver/kernel state, btmtk oops on -4x kernels, AMT cabled but unreachable on the network. Gateway routing to nh3-ml1 is not changed. |
||
|
vh
|
bc278d4ba8 |
refactor(playbooks): host-generic GPU host + GPU LXC playbooks for nh3-ml1
- esh-pve-nvidia-host -> pve-nvidia-host: headers/dkms/build-essential step, nouveau blacklist + guarded unload (refuses if nouveau bound a device) - esh-ml1-lxc -> gpu-lxc: host vars have no defaults (elway aborts on undefined), rootfs storage/startup order parameterized, CT kept out of all-guests vzdump jobs - embed-rerank: Homepage labels take HOST_NAME/HOST_IP, defaults = esh-ml1 |
||
|
vh
|
2118449881 |
feat(nh3-pve): prepare for GPU install — pin NIC names by MAC, pull AMT port from vmbr0
nh3-pve and esh-pve are the same Minisforum MS-01 (BIOS AHWSA.1.17). With a card in the x16 slot its root port takes bus 01 and every NIC moves down a bus (measured on esh-pve), so predictable names change (enp2s0f0np0 -> enp3s0f0np0 etc.) and vmbr0 would boot with no uplink. systemd .link files now pin all NICs by MAC, baked into every initramfs and synced to the ESP; udev confirms the files apply. The AMT-capable I226-LM (enp88s0) leaves vmbr0's bridge-ports in the file (next boot), so cabling it for AMT cannot loop the STP-less bridge. Also: documented the NanoKVM (https://10.100.250.171) as nh3-pve's console OOB and that AMT is not wired; nh3-dev's Beszel agent no longer binds NAS shares (it died on the last NH3 cold start); post-boot checklist in persistent-memory. |
||
|
vh
|
422a27cc1d | chore(homepage): commit the live 'hermes-gateway seat' card rename (operator handle split, 2026-09-24) | ||
|
vh
|
65dc586497 |
feat(esh-ml1): wire telemetry and monitoring; fix Dozzle's stale agent list
- Beszel: NVIDIA agent (stacks/beszel hosts/esh-ml1.yaml), hub system registered with Status/Disk/CPU/Memory/Temperature alerts; GPU samples verified (RTX 2000E Ada util, VRAM, power, temp). - Uptime Kuma: health monitors for the TEI embed (:8001) and rerank (:8013) services — the sole backends behind the gateway, so the one exception to "seats are out of Kuma's lane". Reward seat excluded (no consumer). - Homepage: dockerd on tcp/2375 bound to 10.0.50.80 (playbook step); esh-ml1-docker added to docker.yaml. - Dozzle: agent v10.4.1 on esh-ml1. The hub's DOZZLE_REMOTE_AGENT still named ana-ml2's 10.250.50.54 and irv-ml1's retired 10.100.79.3; repointed to 10.251.50.54 / 10.6.110.50 (fv-ml1 and irv-ml1 logs were missing). - Documented nh3-dev's Beszel agent failing on NH3 cold start (bind of an automounted NAS share); revived by hand, fix still open. |
||
|
vh
|
52612cbe96 |
feat(reward-seat): move Skywork reward seat from fv-ml1 to esh-ml1; audit finds nothing superseding it
- Audit: Skywork-Reward-V2-Llama-3.1-8B is still #1 of 188 on AllenAI's RewardBench 2 per-sample results; no Skywork V3; the -40M sibling is vendor-marked experimental. Our AWQ W4A16 quant: 0.847 vs published bf16 0.860 on a 150-prompt sample (within +/-2.9 pt SE), 96.2% pairwise agreement. Double BOS from vLLM on pre-templated text costs a further ~2.7 pts; callers must send add_special_tokens=false. - No working consumer: 0 requests since 2026-09-13; Worldtree Domari points at a dead IP with a non-vLLM schema (reported to worldtree-dev). - Move: sha256-identical model copy; vLLM v0.24.0 on esh-ml1 :8003 at 0.55 util (KV 1.30x of a 16k request). Parity vs fv-ml1: 149/150 verdicts, 99.8% pairwise signs, raw |delta| median 0.049. - Gateway /scalar-judge passthrough -> 10.0.50.80:8003; fv-ml1 vllm-reward removed (~10.2 GB freed on GPU 1). stacks/vllm now holds only vllm-coder. |
||
|
vh
|
7bdac80878 |
feat(embed-rerank): TEI is the fleet embed/rerank engine; esh-ml1 sole backend; retire fv-ml1 seats
Prime, 2026-09-25: TEI serves embedding + reranking for esh-ml1 and the fleet from now on; fv-ml1 retires both once esh-ml1 is up. - stacks/embed-rerank: vLLM -> TEI 1.9.4 (89- Ada build), same ports 8001/8013, fail-closed truncation (--auto-truncate false; embed --max-batch-tokens 32768). - litellm: qwen3-embedding -> esh-ml1 only (hosted_vllm/, unchanged address); reranker -> huggingface/ provider at :8013 (hosted_vllm/ 422s on TEI's `texts` body). DB alias reranker-a3-bge-v2-m3 patched to the same target. - Verified via the gateway against the retiring fv-ml1 seats: embed cosine median 0.999927 (n=203); rerank top-1/top-3 29/30. - stacks/vllm: vllm-embed and vllm-rerank-a3 removed (containers retired on fv-ml1, GPU 1 freed ~6.1 GB); reward + coder unchanged. - Bake-off record moved to docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md; CLAUDE.md gains the TEI convention. |
||
|
vh
|
49b4bf0177 | fix(tei-bakeoff): fail-closed truncation; record Dvalin review and nevermore threshold check | ||
|
vh
|
582b150131 | feat(tei-bakeoff): TEI 1.9.4 vs vLLM on esh-ml1 — parity holds, not faster, much lighter | ||
|
vh
|
5402568b76 |
feat(esh-ml1): RTX 2000E Ada on esh-pve serves embed + rerank as a LiteLLM failover
esh-pve: NVIDIA 580.178.04 (open modules, DKMS) installed on the host from NVIDIA's .run and loaded live, no reboot. nvidia-persistenced unit creates the device nodes before pve-guests; the T400's vfio-pci ids and `blacklist nvidia` retired. playbooks/esh-pve-nvidia-host.yaml. esh-ml1: CT 110, unprivileged Debian 12, 10.0.50.80, GPU nodes via devN, NVIDIA userspace from the same .run (--no-kernel-modules), docker-ce + nvidia-container-toolkit (no-cgroups). playbooks/esh-ml1-lxc.yaml. Not in the vzdump job on purpose. DNS esh-ml1.esh.internal. stacks/embed-rerank: Qwen3-Embedding-0.6B :8001 + bge-reranker-v2-m3 :8013 on the same vLLM v0.24.0 digest and flags as fv-ml1. Measured parity: embed cosine FV-vs-ESH median 0.999908 (min 0.999772), inside both self-noise floors; rerank max |delta| 0.000145 vs floor 0.000181, identical ranking. litellm: qwen3-embedding and reranker gain an esh-ml1 deployment at order 2 behind fv-ml1 (order 1). Order fallback proven with throwaway groups: refused primary +0.15 s, host-down primary ~18.7 s per call, dead-only 500. Also: repaired the DB-only alias reranker-a3-bge-v2-m3, dead since the fv-ml1 relocation (still named 10.250.50.54); documented the third unkillable homepage wedge on esh-docker-vm. |
||
|
vh
|
e6da607767 |
chore(task-board): mothball it; superseded by the High Seat and ledger
Operator ruling 2026-09-24. On ana-docker the stack is `docker compose down`: the container is removed and port 7878 is closed. Kept for revival: - the data dir /opt/docker/conf/task-board/data (tasks.db, last written 2026-09-11) - the task-board:local image - stacks/task-board/ and the host's compose + .env The Uptime Kuma monitor (id 3) was deleted before the stop so it could not page, and its row is removed from monitors.yaml. Homepage drops the card on its own, since it reads the container's labels. Hooks: the container log showed no hook POSTs in 30 days. The only traffic was open browser tabs holding /events, and the plugin was already uninstalled on nh3-dev. Removed the paragraph that told sessions to call task_* tools (CLAUDE.md, and the fork-fleet.sh template that seeds new repos), the vestigial TASK_BOARD_SESSION env in .claude/settings.json, and the listings in README and FLEETTOOLS. |
||
|
vh
|
5178fdea3c |
fix(homepage): The High Seat icon -> mdi-eye-outline
svos-dev's call and the better one: The High Seat is the English name for Hlidskjalf, the seat Odin watches all the worlds from, and watching every session at once is what the board does. mdi-monitor-dashboard described the artifact; the eye describes the job. Deployed and verified in /api/services, not assumed. |
||
|
vh
|
0ef25dfdb6 |
feat(homepage): add The High Seat (SVOS board, nh3-dev:8770)
Requested by svos-dev relaying the operator, 2026-09-22. Reversible work, so the relay is fine to act on without escalating. Manual services.yaml entry rather than container labels, because SVOS is a user-level systemd unit (svos.service) on nh3-dev and nh3-dev is NOT one of the five hosts in docker.yaml -- Homepage has no Docker API to discover it through. Same reason the Booth, WhereTF, talk and the infra-hermes seat are listed by hand, and the comment says so at the entry. ⚠ siteMonitor is "/" deliberately. There is no /api/health on this service: that path 404s, and a monitor pointed at it would report the board permanently down while it serves perfectly. svos-dev flagged it and it is verified here -- / returns 200 and serves the SPA (<title>The High Seat</title>). Group is Apps, which exists in settings.yaml's layout with tab: Main. An invented group name gets no tab and renders on ALL tabs, which is how Scriberr's "AI Systems" leaked across the whole dashboard in August. Icon mdi-monitor-dashboard is my choice -- svos-dev explicitly did not guess at one and offered to take a different suggestion. Verified in a browser, not just in the API: the card renders in Apps with a green site-monitor at 28 ms. |
||
|
vh
|
94899d6fa3 |
feat(uptimekuma): normalize names off Homepage, publish the status page, restore the widget
NAMES. Homepage already answers "what is this service called", so the monitor name is now that name verbatim -- a second naming authority is how drift starts, and an alert reading "[Uptime Kuma] Beszel hub is DOWN" sends you hunting for a card that does not exist. Only two rows moved (Beszel hub -> Beszel, Dozzle hub -> Dozzle); the " hub" suffixes were mine, not the services'. The remaining mixed case is deliberate and is now documented as such. talk, vor and task-board are lowercase on Homepage and in their own repos; title-casing them here would make this board disagree with both. What actually looked messy was scripts/kuma's own ASCII-ordinal sort, which buried every lowercase name below every capitalised one. Fixed to case-insensitive. ⚠ RENAME SAFETY, which this pass needed and did not have. The seed keys on NAME, so editing a name would have read as a brand-new monitor: added fresh, with the old row orphaned, still checking, still alerting, and holding all the history. `rename_from:` names the old row for one run. Verified: both renamed monitors kept their IDs and all 67 heartbeats. Added with it, an orphan warning for any row on the board the spec no longer names -- because a forgotten monitor keeps paging. Its first cut diffed against the PRE-EDIT snapshot and so cried wolf on its own successful renames; it re-reads the board now. A warning that fires on its own correct work is worse than no warning. STATUS PAGE + WIDGET. The Homepage uptimekuma widget reads a PUBLISHED status page (/api/status-page/<slug>), not the admin API -- which is why the widget labels were deliberately absent from the rebuild: a dashboard widget pointed at a 404 is the suspected mechanism behind both of Homepage's unkillable D-state wedges, so shipping one on purpose would have been daft. The page now exists at slug `nethealth` (the pre-rebuild slug, so old references still resolve) and is DECLARED IN monitors.yaml, applied by `kuma seed`. Same principle as the notification channel: a from-scratch rebuild restores the page, the channel and the monitors together, and nothing the widget depends on lives only in Kuma's database. Verified in a browser: "13 SITES UP / 0 SITES DOWN / 100% UPTIME" on the dashboard. ⚠ saveStatusPage calls imgDataUrl.startsWith() unconditionally, so passing null throws and leaves the page CREATED BUT EMPTY -- which reads as success from /api/status-page (200, correct title) while the group list is silently blank. Pass "" instead. Commented at the call site. |
||
|
vh
|
6f0a9b9fae |
feat(alerts): generalize the althing bridge, wire Kuma to it, retire chamber
THE BRIDGE. `beszel-althing` hardcoded a "[Beszel] " subject prefix and a
Beszel hub footer from when Beszel was its only caller. Routing Uptime Kuma
through it unchanged would have delivered Kuma outages labelled [Beszel],
pointing the reader at the wrong dashboard -- an alert that lies about its own
source is worse than no alert.
Now a route registry: /beszel and /kuma, each with its own prefix, footer and
payload parser, because the tools do not agree on a shape (Beszel sends
{title, message}; Kuma sends {heartbeat, monitor, msg}). Generalising cost a
dict; a sibling service would have cost a second unit, a second port and a
second thing to notice had died.
Renamed beszel-althing -> althing-alert-bridge with it. A service named after
one consumer that carries two is the invisible coupling that sends a future
session looking in the wrong place.
⚠ /beszel IS FROZEN and this refactor proves it rather than claiming it. The
three original tests were kept BYTE-UNCHANGED -- including the one asserting
the exact postbox argv -- and deliver() still defaults to the Beszel route so
they exercise it. A new test asserts the Kuma footer never leaks into a Beszel
body or vice versa. Verified live after the rename: a real POST to /beszel
landed as "[Beszel] BRIDGE RENAME CHECK" with the correct hub footer, read back
from the thread rather than trusted from the receipt.
Payload shapes are parsed HERE, not via Kuma's custom-webhook-body feature,
because Kuma's notification config lives in its own database -- and that
database was destroyed and rebuilt from scratch hours ago. Anything living only
in a tool's DB is lost on the next rebuild; format knowledge belongs in git,
next to a test.
parse_kuma also handles the monitorless case. testNotification and cert-expiry
alerts carry no monitor and no heartbeat, and the first cut fabricated "unknown
monitor is ?" from them -- caught by sending a real one and reading the subject,
not by the suite. Fixed, pinned, and the earlier test asserting the bad
behaviour was corrected rather than worked around.
KUMA IS NOW WIRED. scripts/kuma gained notification support and the channel is
in monitors.yaml, seeded BEFORE the monitors and with applyExisting, so a
rebuild restores alerting and not just detection. Ground truth from the DB:
13 of 13 monitors carry the channel.
⚠ A THIRD instance of the same class of bug, worth naming: notifications() is
pushed as `notificationList` at LOGIN ONLY -- there is no event to ask with. The
first cut cleared the captured value before waiting, discarding the only copy it
would ever be sent, then blocked for the full timeout and reported an empty
list. That reads exactly like "no channels configured" and is a lie. Same family
as the getMonitorList ack-vs-push trap, different shape.
End-to-end, both shapes, read back from the inbox:
[Uptime Kuma] Homepage is DOWN + target + board link
[Uptime Kuma] althing (infra-ops) Testing (no fabricated subject)
[Beszel] BRIDGE RENAME CHECK + hub footer, unchanged
ALTHING CHAMBER RETIRED (operator). Three of its four containers had never
started -- created 2026-09-19, StartedAt epoch-zero, 0 restarts -- so :7881
refused, and nothing was watching it. Only its valkey was running, on the
project's own network with no external consumer. Stack, compose/build/conf dirs
and the local image removed; the Homepage card went with the label.
|
||
|
vh
|
3a85a6bce1 |
feat(uptimekuma): rebuild on 2.5.5 as the fleet's service layer, with a client
Homepage sat dead for three days in September while every monitoring tool
reported correctly. Beszel said its host was up -- it was. Uptime Kuma was not
watching it. The dashboard fell into the seam between two working instruments.
Measured before changing anything:
- Beszel: 18 hosts x {Status, CPU, Memory, Disk, Temperature}. Its alerts
table is (system, name, value, min) -- there is NO url column, so it is
structurally incapable of "this endpoint should return 200". Not a config
gap; the data model.
- Uptime Kuma: 6 rows, 2 of them folders. Four real monitors, all firewalls.
- So the two are NOT redundant. They are disjoint, and the service layer
between them was empty.
REBUILT FROM SCRATCH, operator-authorised ("uptime-kuma was never really
used... you can even dump the existing container and config"). Nothing was
migrated, which also skipped the one-way v1->v2 database migration.
- Pinned to 2.5.5. `:latest` is a documented trap now: upstream keeps it on
the 1.x line, so an August 2026 pull produced an image BUILT 2024-12-20
running 1.23.16. Verified by digest -- latest and 1 share one digest while
2/next carry 2.5.5. Pinned exactly, not floating on 2, for the same reason.
- Moved esh-docker-vm -> ana-docker. House placement rule puts cross-site
services beside the Beszel and Dozzle hubs, and esh-docker-vm has wedged
unkillably twice in four months. A monitor also cannot report the failure
of the host it runs on, so it should not share a failure domain with the
host layer.
- Normalised restart: always -> unless-stopped, which the 2026-08-18 README
flagged as worth doing on the next deliberate touch.
- UPTIME_KUMA_DB_TYPE=sqlite in the compose skips 2.x's interactive database
screen, so the stack comes up ready rather than parked on a form.
scripts/kuma is a first-party Socket.IO client, because both obvious paths are
wrong: there is no REST CRUD API in EITHER major version (server/routers/ holds
exactly two files, /metrics + badges + status pages), and the community wrapper
uptime-kuma-api is abandoned -- last release 2023-09-26, ceiling 1.23.1, no 2.x
support ever.
⚠ getMonitorList's callback returns only {ok:true}; the list arrives as a
SEPARATE pushed monitorList event. Reading the ack yields an empty board that
looks authoritative -- which duplicated all 13 rows on the first re-seed
before the bug was found. The client now waits for the push, and carries a
dedupe verb because of it.
13 monitors seeded from monitors.yaml, keyed on name so a re-run updates rather
than forking the board -- proven by re-running it (0 added, 13 updated), not
assumed. Every URL was probed before being written: all 200. A board that ships
red teaches everyone to ignore it.
Verified: 13 rows, no duplicates, all UP with "200 - OK" read from the database
WITH its WAL (a first read of kuma.db alone showed a stale 26 -- the copy
predated the deletes). Homepage renders exactly one Uptime Kuma card.
⚠ NOT YET WIRED: notification delivery. The board detects but tells nobody,
which is the same gap this work exists to close. The beszel-althing bridge
hardcodes a [Beszel] subject prefix and hub footer, so routing Kuma through it
unchanged would mislabel the alerts. Needs a decision before it is generalised.
|
||
 Vuong Hoang
|
6692701571 |
docs(voices-seat): every voice prefixes an empty think block unless the caller disables it
Found while smoke-testing the lv-mccarthy ship. Measured live: default -> '<think>\n\n</think>\n\nThere were no horses in the road...' enable_thinking=false -> 'The sun was hot on the dry riverbed and the stones were red...' This is the Qwen3-4B-Instruct CHAT TEMPLATE, not an adapter property, so it applies to lv-yarros, lv-bronte and lv-hemingway equally and has done since this seat went up on 2026-09-16. No gate number is affected: gen_beats_chat_yarros.py sets enable_thinking when the template supports it, so every arm in every r49 gate was generated without the tags. But a caller that does not pass chat_template_kwargs gets 17 junk characters at the head of every passage -- and any word-count or in-band check run over that string is counting the tags as prose. Skaldsong should be checked. |
||
|
Vuong Hoang
|
17db37e1cc |
feat(voices-seat): ship lv-mccarthy (checkpoint-300) as the fourth author voice
Shipped on the operator's standing authorisation, "ship it if the gate passes".
The gate design of record for this line -- the operator's own v2 rule, ratified
2026-09-15 -- passes on all three axes:
VOICE +0.177 delta_cb at 3.2x the pairwise floor, the BEST arm of five,
and +0.128 at 2.8x on the pre-registered punctuation-stripped
secondary read, so ~3/4 of the gain is words rather than marks.
Closed 60.8% of the reachable span on an axis deliberately made
harder than Bronte's or Hemingway's.
NOT COPIED 0.12 hit-rate against HELD-OUT McCARTHY'S OWN 0.12 -- 1.0x the
author -- with a longest verbatim run of 10 words against the 12
genuinely unseen McCarthy shares with the train split by
coincidence. All 31 matches READ: stock grammar, name-shaped hits
are the renamed inventions, nothing protectable. This matters:
McCarthy is in copyright with a living estate.
NO DAMAGE ran-on +0.12 against the 0.200 floor, 40% headroom, and the
measured minimum across the five-arm ladder.
ckpt300 over ckpt900, the eval-loss minimum: the two-epoch recipe is now 0 for 3,
and this time the curve was CONFIDENT and wrong -- ckpt900 is the minimum and
ckpt300 sits +15.8x the median neighbour jitter above it, yet ckpt300 wins every
axis that resolves (3.2x vs 1.2x voice margin, 2.7x tighter seed spread,
memorisation 1.0x the author vs 1.8x, ran-on 0.13 vs 0.28). On this schedule read
the axes, not the curve.
THE COST IS REAL AND IS WRITTEN INTO THE COMPOSE, THE ADAPTER README AND THE
MEMORY RECORD RATHER THAN GLOSSED. In-band is 0.65 against base's 0.89 and on-beat
0.42 against 0.71 -- 35% of generations miss the requested 90-140 word band against
base's 11%. Axis C is ran-on only and is structurally blind to this; that blindness
was identified and written down before these numbers existed. No checkpoint choice
fixes it: every adapted arm is 33-48% out-of-band and ran-on is non-monotonic in
epoch. If a caller depends on a word budget, expect one generation in three to miss
it, and the fix is a retrain targeting length rather than a different checkpoint.
Rollback is one line and 0.003 s: drop the --lora-modules entry, or POST
/v1/unload_lora_adapter. The other three voices are untouched.
|
||
|
vh
|
838ea60fc0 | homepage: surface the infra-hermes bus seat status page (nh3-dev:8766) | ||
|
vh
|
920f37c6c2 |
chore(althing): pin the post office to 3.7.0
Canonical pin bump for the 3.7.0 rollout forseti approved and released (althing 6db955f, tag v3.7.0). Image built from that tree and pushed as sha256:df0709b3f532557aebb47d6ece486ae32e851776f732862094608d55eefcd4e7; previous pin was 3.6.3@sha256:978f8553 and remains the rollback target. Content-verified rather than tag-verified: `postbox --version` inside the built image reports 3.7.0 before it was pushed, and again inside the running container after. A tag is a claim. Also adds playbooks/nh3-extdev-althing-upgrade.yaml -- a version-parameterised upgrade for the box a git pull cannot move. The existing nh3-extdev-althing-v3.yaml is the historical v2->v3 CUTOVER and is left alone rather than mutated per release; it still carries v2-retirement steps that document what happened. Every check in the new playbook asserts what the installed entry point reports, including one that the herald restarted within the last 5 minutes rather than surviving from before -- "active" does not mean "running the code you just installed". |
||
|
vh
|
5b20b02fb9 |
chore(searxng): adopt the concurrent v4 work, with its dead mechanism marked
Picks up uncommitted searxng changes left by another session and makes them truthful rather than committing them as written. The stack itself verifies clean: canonical and live are byte-identical for both compose.yaml and searxng-settings.yml, the container is running with zero restarts, and live queries return 51-54 results from 5-6 engines with braveapi contributing 20 each time. compose.yaml gains SEARXNG_BRAVE_API_KEY, which NOTHING READS. It was added on the belief that settings.yml could pull it via `!ENV SEARXNG_BRAVE_API_KEY`; this build has no !ENV YAML constructor, so that attempt made the file unparseable and crash-looped the container ten times with fleet search down. The comment claiming the variable is "consumed by settings.yml" is replaced with what is actually true. The variable is kept, unused, in case upstream ever gains env interpolation — a comment that lies is worse than a variable that does nothing. The Tier A playbook is marked superseded FOR THE SETTINGS FILE ONLY, and scoped deliberately: its v4 design uploads a settings file carrying the !ENV tag, which would re-break the container, so settings deployment goes through scripts/deploy-stack.sh like every other stack. Its .env merge and up-d-not-restart steps remain useful, as do its two warnings recording real bugs it hit — a wholesale .env overwrite that clobbered SEARXNG_SECRET, and a sed that inserted literal backslash-n into compose.yaml. An unscoped "superseded" banner would have buried those; that failure mode cost an outage earlier today. Also folds in the regenerated graphify report. |
||
|
vh
|
d812bfe96d |
feat(homepage): update the talk tile to the inverted mark
talk shipped a reworked mark at v18 on operator ruling — the 1024x1024 cerulean field rect is gone, the bubble now carries #03adfb where it used to carry #2e2d30, and the three waveform bars are holes rather than filled shapes. Path data is byte-identical to the original trace; only the two fills moved. Fetched from the app and from the booth and confirmed the two sources are byte-identical before taking either. tts-dev flagged a real risk with the change: with the field gone the tile background shows THROUGH the waveform holes, so a tile close to #03adfb would swallow the bars. Checked rather than assumed. Homepage's card surface is --sea-20, oklch(0.31 0.022 262) = #2a313c, a dark desaturated navy; the bubble against it is 5.22:1, well clear of the 3:1 bar for non-text graphics. The page ground behind it is 6.73:1. Safe on this tile specifically — the earlier "reads well against the tile background" judgement was about a solid square and did not carry over on its own. Also refines the Next.js note in CLAUDE.md, which was over-broad. A NEW file in the images mount 404s until restart, but REPLACING an existing file's bytes serves immediately with no restart — measured here, the served hash matched the new file straight after rsync. It is the route table that freezes at container start, not the file contents. The previous wording would have had people bouncing Homepage for every icon tweak. |
||
|
vh
|
9219942037 |
feat(searxng): enable the keyed braveapi engine
Brave Search API key wired literally into the settings file and committed. Operator decision, 2026-09-18, made explicitly: this is a free-tier key on a rate-limited service of marginal value — "if the service is useless, so is the key" — so it does not justify the machinery that keeping it out of git would cost. The key remains in the vault at nh3-docker/searxng-brave-api-key as well. This is a scoped judgement about one low-value credential and not a change to the no-secrets-in-git rule for anything else. ⚠ It cannot be un-committed. Rotation means issuing a new key at Brave and replacing the line; never a history rewrite, since the repo is shared and other sessions commit to it. There is no supported alternative in this build. An earlier attempt used `api_key: !ENV SEARXNG_BRAVE_API_KEY`, which crash-looped the container ten times with search down fleet-wide: the settings loader has no !ENV YAML constructor, reads only SEARXNG_SETTINGS_PATH from the environment, and the entrypoint substitutes only `ultrasecretkey` at template-creation time. The variable reaches the container and is never read. Literal or nothing. Key verified against Brave's API directly before wiring, and verified in place after: three consecutive queries returned 55-63 results from six engines with braveapi contributing 20 each time, while google cse and marginalia remain quota-suspended. General web engines are now seven, up from one this morning. |
||
|
vh
|
274d3e2443 |
fix(searxng): six general web engines by default, not one
Root cause of the silent-empty-results failure peedlar-dev reported. Of 55 general-category engines, only seven were enabled-by-default, and six of those are dictionary, translation, currency or encyclopedia engines that return nothing for an ordinary web query. `google cse` was the instance's ONLY general web engine, so a single quota exhaustion produced HTTP 200 with an empty results array and no error, for every consumer on the fleet. The distinction that matters: `inactive: false` only makes an engine selectable, `disabled: false` puts it in the default set. The other 48 were selectable-but-off, which an API client has no way to change. Enables five keyless engines, each bang-probed first and returning real results with no API key: duckduckgo web 10, bing 10, yep 20, yahoo 7, wiby 12. General web engines go 1 -> 6. Deliberately excluded: mojeek, qwant, startpage and the brave scraper, all of which CAPTCHA or rate-limit this egress, and seznam, which times out. Verified under the live failure condition rather than a simulated one. google cse is still quota-suspended right now, and three consecutive queries returned 38-41 results from 4-5 engines each. The single point of failure is gone while the failing engine is still failing. Also adopts the concurrent v4 settings work from the other session — marginalia on its public key, and the captcha'd-scraper removals — plus the fix for the crash-loop that work introduced: this build has no !ENV YAML constructor, so `api_key: !ENV SEARXNG_BRAVE_API_KEY` made the file unparseable and the container restarted ten times with search down fleet-wide. That block stays commented; the vaulted Brave key is valid but has no supported path into the settings file, which is a separate decision. |
||
|
vh
|
9a428fded9 |
fix(searxng): update to 2026.9.18 — all four engines restored
searxng had been answering from google cse alone for at least a day, with brave and startpage suspended and duckduckgo returning CAPTCHA. Updating the image from 2026.9.3+a1144dda3 to 2026.9.18+c0042add3 restored all four engines immediately, and they held across 11 consecutive queries run after the change specifically to rule out a freshly-reset circuit breaker flattering the first measurement. before searxng/searxng@sha256:3602e6ddbeba037f5d800d1ed9d296a8b93c9f5b3cf9d05fa179d0e766dd59a1 after searxng/searxng@sha256:e0027a772aeeea55bf642256aae6fb3344ffa5f25ca665898c2ea821101334c4 The image stays on :latest rather than being digest-pinned. For this stack that is deliberate and now demonstrated: upstream ships engine-handler fixes as providers change their bot gating, so being current is the mitigation, and a pin would have frozen the breakage in place. The post office is pinned for the opposite reason — it is the fleet message bus and must not move under us. README corrected. It had carried two successive wrong diagnoses, both blaming egress, and now records the real cause plus the two measurements that falsified them: reverting to direct NH3 egress reproduced the failure exactly, and a live !ddg probe on a freshly restarted container also CAPTCHA'd, ruling out a stale suspension timer. Both wrong claims asserted causation from correlation without a baseline. The health-script blind spot is unchanged and still called out: scripts/searxng-health.sh reports the same passing result whether four engines answer or one. |
||
|
vh
|
ca5f0a91c0 | searxng: sync settings with live (captcha-era engine set) | ||
|
vh
|
1a35181b67 |
revert(searxng): return search egress to direct NH3
Reverts the outgoing.proxies block added in |
||
|
vh
|
156e12619d |
feat(searxng): route search egress through the esh-scale SOCKS5 proxy
Committing work deployed on 2026-09-17 that had been left uncommitted, so canonical intent stops disagreeing with the running host. The deployed /opt/docker/conf/searxng/searxng-settings.yml is byte-identical to the canonical file here, verified before this commit. Search requests and their DNS now exit via socks5h://10.0.50.65:1080 on esh-scale (CT 108), an application-level proxy rather than a host-wide exit node; no route or firewall changes. microsocks runs as nobody under searxng-egress.service, binds only 10.0.50.65:1080, and bypasses SOCKS auth for source 10.100.50.40 alone — every other source must supply a password regenerated at each start and never distributed. Verified active and enabled. There is deliberately no direct-NH3 fallback: an ESH outage must fail the search rather than silently revert egress. ⚠ THE CHANGE HAS NOT ACHIEVED ITS PURPOSE AS DEPLOYED. Two independent live queries, 2026-09-18, both report brave "Suspended: too many requests", duckduckgo "CAPTCHA" and startpage "Suspended: CAPTCHA", leaving google cse as the only answering engine. Moving egress off NH3's residential address is what this change did, and CAPTCHA avoidance was the stated reason searxng sits at NH3 at all. The README anticipated the risk in its Dependency note; it has materialised. Rollback procedure is in the README and the pre-change config is kept on the host as searxng-settings.yml.pre-esh-20260917. Measured egress also drifted from the value recorded at cutover: the README notes 154.50.58.126, the proxy now exits 128.177.138.182. Expected — the README pins no public IP and calls out WAN failover — but recorded here so the number in the doc is not mistaken for current. Also retargets seat-inventory.py's default host from the mesh address 100.64.0.7 to fv-ml1's LAN address 10.251.50.54, routed by the site gateway. |
||
|
vh
|
670ac9e8a0 |
deploy(althing): pin the post office to 3.6.3
Canonical intent still named the 3.6.2 digest while nh3-docker was running 3.6.3, so the next scripts/deploy-stack.sh run against this stack would have silently rolled the fleet message bus back and taken the hyphenated-search fix with it. Caught by forseti during independent post-deploy verification. 3.6.3 is the literal-search fallback: a query containing a hyphen was parsed by FTS5 as a column filter, raised OperationalError, and search() returned [] — indistinguishable from "no results" — so every hyphenated term on this fleet silently matched nothing. nh3-docker, irv-ml1, esh-docker-vm, tts-dev and every other hyphenated name were unsearchable. Deployed digest verified against the running container before this pin: sha256:978f85533674ee248d6c6f29c54ffab0bc2cb16332c18c9fb8bfda1d566e2de4, built from git archive of tag v3.6.3 (5de41b7). The image line is the only difference between canonical and live; the two files are now identical, so a managed deploy is a no-op rather than a regression. |
||
|
vh
|
d6a9d70b9e |
feat(homepage): add talk tile with its commissioned mark
talk has served on nh3-dev since 2026-09-08 with no dashboard presence. nh3-dev is not a Docker-stack host and is absent from docker.yaml, so label auto-discovery cannot reach it — this is a manual services.yaml entry in Apps, beside the Booth and WhereTF which are there for the same reason. siteMonitor points straight at the app: talk.nh3.phasefinal.com:8092 now presents the Let's Encrypt *.nh3.phasefinal.com wildcard (valid to 2026-12-06), so no cert or port special-casing is needed. Icon is copied into the images mount rather than hot-linked from the booth, which is scratch space. Document the two traps that cost time here: Homepage v2 serves nothing but custom.css/custom.js out of the config dir, and Next.js fixes its public/ route manifest at container start, so a newly added image 404s until the container is restarted. |
||
|
vh
|
300ecc1276 |
voices-seat: ship lv-hemingway (ckpt850), and replace the memorisation control that passed it
Live on vllm-voices (fv-ml1 GPU0 :8027) beside voices-base, lv-yarros and lv-bronte.
Healthy 190 s after recreate, four models served, GPU0 96,092 -> 96,090 MiB. The adapter
was verified byte-identical to checkpoint-850 by sha256 across both transfer hops, and the
seat was verified by generating, not by reading its config: base emits 170 words of <think>
planning and never writes the passage, lv-hemingway writes the scene.
Gate design was pre-registered before any generation existed (
|
||
|
vh
|
2e9b118e70 |
lv-bronte: the voice axis passes under the corrected floor rule — amended, not rewritten
lv-bronte shipped 2026-09-17 with a FAILED voice axis written into its compose comment,
its NFS README and its gate record. That verdict no longer stands, and this records the
correction in all three places without deleting what they said.
The floor rule is now pairwise (commit
|
||
|
vh
|
61840f3131 |
voices-seat: ship lv-bronte (ckpt475) with its failed voice axis on the record
lv-bronte is live on vllm-voices (fv-ml1 GPU0 :8027) alongside voices-base and lv-yarros. The seat lists all three; container healthy; GPU0 96092 -> 96090 MiB, so the adapter cost nothing measurable. IT DID NOT PASS ITS VOICE GATE, and the artifact says so in three places — this commit, a comment in the compose file, and a README beside the adapter on NFS — because an adapter found without its provenance will otherwise be read as a pass. VOICE FAIL +0.193 delta_cb vs base, against a 0.251 measured noise floor NOT COPIED PASS 8-gram hit-rate 0.00, longest 0 - identical to the control NO DAMAGE PASS ran-on +0.15 against a 0.400 floor Shipped on three grounds, none of them that the number was nearly good enough: it is additive (a named LoRA nobody reaches without asking for it), reversible (one compose line; hot-unload measures 0.003 s), and clean on the axis that carries actual risk - verbatim regurgitation of the source, on a public-domain corpus, measured against a positive control that saturates at 160. The voice result is UNDERPOWERED rather than absent: it closed 48% of the span from base to the same-author target and beat the control on every individual seed. The cause is structural - 81 val pairs against Hemingway's 200, from a 678k-word corpus against 994k - and neither more beats nor more seeds fixes it, because the floor is a range statistic and ranges widen with n. ckpt475 over ckpt925: indistinguishable on voice (0.017 apart), but ckpt925 has a verbatim 8-gram hit where this has none, and is 2.7x less stable seed-to-seed (0.251 vs 0.092) with a degeneracy probe showing no collapse to explain it. |
||
|
Vuong Hoang
|
d17bd3df86 | voices-seat: one carrier, lv-<author> LoRA adapters, with the measured cost and placement limits | ||
|
vh
|
66c860d6c1 |
fix(sweep): retire the dead 10.100.79.3 address across the fleet
Operator-directed. The wg0 lifeline retired at the 2026-09-06 headscale cutover is on no interface anywhere, so anything pointing at it gets no route at all. Homepage went from 9 dead cards to 0 of 112. The load-bearing part is that there is no single right target: it depends on who resolves it. The operator's browser and the Homepage and open-webui containers on esh-docker-vm all resolve nh3.internal, so those get the name and survive the next renumber. Containers on irv-ml1 and ana-docker cannot resolve it at all, so those get the IP. litellm on ana-docker looked like a counterexample and is not: it resolves the name only through its own extra_hosts entry, while asset-engine on the same host fails on it. Test from the container you are about to change, never from a neighbour. Before committing to the name I confirmed the Homepage container actually fetches ytvc's healthz through it in production rather than assuming resolution implies reach. On irv-ml1, 24 files swept and 14 comment-only hits left as port-allocation history. Seven running containers recreated so the labels took. Seven dormant ones carried stale labels because editing a compose file does not touch an existing container object - fixed with compose create --force-recreate, which rebuilds the container without starting it, the right tool for a deliberately dormant stack. The sweep's real find was off irv-ml1 entirely: four live values on two other hosts, silently dead for nine days and alerting nobody. Open WebUI's read-aloud TTS, asset-engine's inference host, and two skaldsong TTS URLs. Both running services were recreated and verified reaching their targets afterwards rather than merely carrying the new string. One self-inflicted outage worth recording: I recreated breeze-tts for a cosmetic label change and took ext-tts down for its ~90s CUDA-graph warm-up, returning 500. I caught it only because I had taken a baseline before touching it. A label-only edit still costs a full model reload on a GPU container. |
||
|
vh
|
caa04801f3 |
fix(parakeet): move the seat from the empty GPU 3 to GPU 0
Placed on GPU 3 first because it was the empty card. That was the wrong read:
the seat is ~800 MiB, under 1% of a 96 GB card, so the question was never "where
does it fit" but "whose headroom is cheapest to spend".
vLLM sizes its KV cache as a fraction of TOTAL VRAM, not free VRAM. A resident
tenant on an otherwise-clean card therefore does not cost its own megabytes — it
costs the profiling margin of whatever full-size seat lands there later, and
flash-next needs 93 GiB of 96. A 96 GB card at 2 MiB can still take that; the
same card at 922 MiB is one where the next big seat needs its utilization
hand-trimmed, which this repo's flash-next history shows is both thin and silent
when it goes wrong.
Committed utilization per card is the number that governs, not free bytes:
GPU 0 0.40 + 0.48 = 0.88 ~13 GB spare <- moved here
GPU 1 0.52+0.24+0.10+0.055+0.03+0.03 = 0.975 ~4.3 GB
GPU 2 0.96 ~1.8 GB
GPU 3 - kept empty as reserve
GPU 3 is back to 2 MiB / 97,247 MiB free and is now documented as a deliberate
reserve rather than a spare.
Post-move n=5 on the same clip: 0.68 / 0.54 / 0.54 / 0.52 / 0.53 s, median 0.54 s
against 0.50 s on GPU 3. The spreads overlap at this sample size and no difference
is claimed; the GPU 3 figure was taken on an idle card and is now noted as a best
case, since the seat shares GPU 0 with the hot serving path. Silence control and
the gateway round-trip both re-verified after the move.
Also records both Parakeet endpoints (FV v3 on Blackwell, IRV v2 on a 3090) and
the four confounds that make them not an A/B pair, sent to tts-dev for the bench.
|
||
|
vh
|
b9b14b5baf |
feat(parakeet): stand up Parakeet STT on fv-ml1 GPU 3 + LiteLLM ext-stt/whisper-1
Retargets the existing sherpa-onnx stack from irv-ml1 to fv-ml1's utility card and puts it behind the gateway. GPU 3 was the only card with room: 0/1/2 carry the vLLM seats at 84-95.5 GB of 96. Changes: - compose: pin GPU via `device_ids: ["3"]` (the dead on-host stub used `count: all`, which would have handed a 0.6B ASR seat all four cards); join traefik-net; port 8300; homepage href to the live FV address. - .env.example: default to the v3 int8 model (25 European languages, 464 MiB) rather than English-only v2; models to /tank/parakeet/models. - app.py: warm the recognizer at startup before uvicorn accepts traffic. The warmup is not an optimisation. ONNX Runtime's CUDA EP compiles and autotunes lazily on the FIRST DECODE, and on sm_120 that measured 45.7s cold (reproduced at 45.1s on a second container) against ~0.50s warm. A 45s first request is indistinguishable from a hang and LiteLLM's default timeout abandons it long before it returns. Decoding 1s of silence at load moves the cost inside the healthcheck's 300s start_period; first real request after restart is now 0.65s. Verification, because "provider=cuda" in the log is only an echo of the env var: ORT falls back to CPU silently and still returns correct text, so the service being up and the transcript being right establishes nothing. The discriminator is a process on GPU 3 (922 MiB), confirmed. Controls both directions — a known TTS sentence transcribes near-exactly (positive), 3s of digital silence returns empty (null). Warm throughput 0.50s median on an 8.52s clip, n=5, spread 0.47-0.65s, single-stream, one clip: a smoke measurement with its harness stated, not a benchmark. Gateway aliases `ext-stt` (engine-neutral, mirrors ext-tts) and `whisper-1` (OpenAI-compatible drop-in) registered via POST /model/new, i.e. LiteLLM's Postgres store where the ext-tts family already lives — no gateway restart, and config.yaml is consequently not a complete picture of what the gateway serves. Both verified end to end. The aliases use a raw IP deliberately: ana-docker resolves no .internal names at all (resolv.conf points at 1.1.1.1), and LiteLLM only reaches irv-ml1 through a hand-pinned extra_hosts entry. A second hosts entry would mean recreating the container and bouncing the gateway for every consumer. Also records the svos_miranda plugin validation pass and its structural findings, and notes that the irv-ml1 parakeet is still running — there are two now, and retiring the old one is the operator's call. |
||
|
vh
|
c659fa5020 |
fix(esphome): correct the record — mDNS advertisement was never removed
ha-dev caught a false claim I committed in
|
||
|
vh
|
687c6999f3 |
fix(esphome): actually disable remote-build — two switches, only one closes the port
ha-dev found the WS API and tested the read half; this runs the write. But the
command they identified is the wrong half, which is worth recording because the
naming actively misleads.
remote_build/set_offloader_settings {remote_builds_enabled: false}
the OUTBOUND half — this dashboard sending builds to peers.
Persists, reads back false, and leaves the receiver listening.
remote_build/set_settings {enabled: false}
the receiver-side master switch, per ReceiverController.set_settings's
own docstring. Tears the listener down live, no restart needed.
Set both. Verified across a restart: 6055 absent, zero peer-link bind lines,
zero mDNS advertisements, both switches read back false. Persisted at
_remote_build.enabled in /config/.device-builder.json — which did not exist
until the flag was first changed, so 'no on-disk representation' was true only
of the default state.
ESPHOME_REMOTE_BUILD_HOST=127.0.0.1 is KEPT as a backstop rather than removed.
The off state now lives in one JSON file whose in-code default is enabled:True
(controllers/remote_build/_state.py) and whose module's stores soft-recover to
an empty model on a malformed blob rather than erroring — so a lost or corrupt
settings file silently re-enables remote-build. With the env var set, that
regression binds loopback instead of 0.0.0.0.
Also finishes deploy-stack.sh properly. This was patched three times in one
session because -a is -rlptgoD and a non-root identity cannot apply owner,
group, permissions OR times to a root-owned directory; each patch fixed one
letter and the next deploy failed on the next one, every time exiting 23 AFTER
a successful transfer. The rule is now written into the script: the deploy
syncs content, the conventions own metadata. --no-o --no-g --no-perms
--omit-dir-times. Verified: clean run, destination keeps 2775 root:docker with
setgid intact.
|
||
|
vh
|
8073a6aed9 |
fix(esphome): bind the remote-build peer-link to loopback; finish the rsync fix
ha-dev asked for the Device Builder 1.0.0 remote-build receiver to be turned off: one instance, builds run locally, so the feature has no role, and it was binding 0.0.0.0:6055 with mDNS advertisement on a privileged host-network container that writes firmware to devices. Reading the source first changed the framing. controllers/remote_build/ _state.py declares 'remote_builds_enabled: bool = True', so nobody enabled it — it arrived on by default with the rewrite. And the flag has no on-disk representation until it is changed: neither .device-builder.json nor .device-builder-preferences.json carries it, and the only writer is the app's own command API behind the UI. Setting it from a playbook would mean inventing a schema for a model I have not read. So this binds ESPHOME_REMOTE_BUILD_HOST=127.0.0.1 — a documented env var, no entrypoint override — which removes the LAN reachability now and is verifiable (ss reports 127.0.0.1:6055, was 0.0.0.0:6055). It is explicitly NOT the off switch ha-dev asked for and the compose comment says so; the Settings toggle is one UI click and the line can go once someone flips it. Also completes yesterday's deploy-stack.sh fix, which was half a fix. --no-o --no-g stopped rsync chgrp-ing a root:docker destination as a non-root identity, but the very next deploy failed the same way one layer along — 'failed to set times on ...' — because a non-root identity cannot utime() a root-owned directory either. Same exit 23 after a successful transfer. Added --omit-dir-times. Fixing only the group half looked fixed until the next run, which is the whole reason this is worth a line in the script's comment. |
||
|
vh
|
d1769ed114 |
feat(esphome): pin 2026.8.2, relocate config into backup coverage, rotate creds
ha-dev requested all three on esh-docker-vm (operator-authorized); the stack had no canonical copy, so it is added to stacks/ rather than edited in place. Pinned ghcr.io/esphome/esphome:2026.8.2 — it was bare, which is exactly how it sat on 2025.8.2 for a year: docker pulled latest once at container creation (2026-04-20, from a layer cached 2025-08-29) and never re-pulled. Every current Everything Presence sensor failed config validation on that build. Verified after: esphome version reports 2026.8.2 and the vendor's own Pro package now validates clean (exit 0, 'Configuration is valid!'), which is the item that unblocks the six waiting sensors. Relocated /path/to/esphome/config (the upstream template placeholder, taken literally by docker) to /opt/docker/conf/esphome, matching the mosquitto pattern. Copied and checksum-verified all 5763 files before removing the original, with a tarball kept at /root/pre-change-archive/. Credentials moved off test/ChangeMe to the vaulted 32-char secret (esh-docker-vm/esphome-dashboard), passed via a host-only .env so nothing plaintext enters git. Three things the job surfaced that were not in the request: The directory is 538 MB, not the 3 KB reported — .esphome/platformio is 508 MB of PlatformIO toolchain and .esphome/build another 31 MB, both regenerable. Relocating as-asked would have inflated restic's /opt/docker source ~45x against its own ~12 MB budget, so both subtrees are excluded in /etc/restic/profiles.yaml. The 3 KB of actual config is now covered, which was the point. 2026.8.2 logs a DEPRECATION for the bare USERNAME/PASSWORD env names and says they will stop working in a future release — a silent auth loss on some later bump, on a privileged host-network container that can flash any ESP device on the LAN. Switched to ESPHOME_USERNAME/ESPHOME_PASSWORD; the warning is gone. Device Builder 1.0.0 opens a NEW listener on 0.0.0.0:6055 (remote-build peer-link) that 2025.8.2 did not have. Also fixes deploy-stack.sh: plain 'rsync -a' makes rsync chgrp the destination as the deploy identity, which since the 2026-09-14 root:docker normalisation is not root. It failed with 'Operation not permitted' and exit 23 AFTER transferring content — a loud error on a deploy that had succeeded. --no-o --no-g lets the setgid bit assign the group instead. |
||
|
vh
|
05f3d3b2a3 |
feat(fv-ml1): add gen-small A3B seat; sec/gen/char to native 262K; coder reclaim; gateway
Full seat rebalance across GPU0/GPU1 (flash on GPU2 and empty GPU3 untouched), operator-directed. Every target seat now serves native 262,144 context with concurrency in the requested 1.2-2.5x band, verified from live boot logs: cyberprev (sec) 262144 @ 1.37x depth-probed CLEAN to 259,722 tokens flash-next (gen) 262144 @ 1.31x (untouched, already in band) gen-small (NEW) 262144 @ 2.56x MTP k=3 measured 69.6% accept / 3.09 len char-rp 262144 @ 1.22x (was 1.02x; util 0.40->0.52) char-rp-fast 262144 @ 2.04x (util cap 0.30->0.24, pinned KV unchanged) - gen-small: NEW seat, Qwen3.6-35B-A3B (3B active MoE), llmfan46 Heretic (MPOA) NVFP4 experts-only, already on disk at qwen36-35b-a3b-heretic-nvfp4. There is no general Qwen3.8 A3B (3.8 MoEs are Flash-Next and the 2.4T), so this is the 3.6 fallback the operator specified. GPU0, :8026, MTP k=3, coherent and MTP-verified before wiring. gen-small / gen-small-reasoning gateway aliases. - coder: 8192 @ 13.75x -> 16384 @ 4.70x (util 0.06->0.055). Context doubled, waste cut. Not the exact 2-3x target: the 1.5B weight+overhead floor (~4.2 GiB) sits just under the util knob's resolution, so hitting <=3x reliably needs a --kv-cache-memory byte pin (compose change) rather than the util fraction. - cyberprev raised 163840 -> 262144: depth-probed with non-repeating prompts to 259,722 tokens, clean (no OOM, memory flat). Unlike mog-sec (same base arch, capped at 163840 for depth crashes), this checkpoint holds native depth. - Gateway (operator calls): summarizer + classifier -> gen-small; new classifier-large -> gen-large (flash) for the accuracy tier; summarizer-large stays on flash. All verified end-to-end. - GPU1 hit its ceiling raising char-rp; resolved by trimming char-rp-fast's reservation cap (its KV is pinned, so concurrency held at 2.04x) rather than moving a utility seat -- the shared GPU_ID on reward/embed/rerank made a single-seat move messier than the in-GPU rebalance. Seat inventory regenerated from the live containers. |
||
|
vh
|
207798de66 |
feat(litellm): promote cyberprev into the sec / sec-reasoning seats; retire mog-sec
Operator instruction: take down the existing sec seat (mog-sec) and promote hotdogs (cyberprev) into the sec and sec-reasoning gateway seats. - mog-sec container (vllm-mog-sec, :8019, fv-ml1 GPU0) taken down; ~48 GB freed on GPU0 (cyberprev, already co-resident there, is now the sole GPU0 chat seat). - Gateway sec -> hosted_vllm/cyberprev-27b @ :8025; sec-reasoning -> hosted_vllm/cyberprev-27b-thinking @ :8025. sec/sec-reasoning are ROLE aliases, so this is a promotion, not silent substitution (samplers were already identical between the sec blocks and cyberprev, so only model+api_base changed). - Removed the standalone cyberprev-27b / cyberprev-reasoning gateway aliases added in the prior commit -- now redundant with sec/sec-reasoning, and the fleet convention is a role alias on the gateway with the model's served-name only at the vLLM layer (as mog-sec had). cyberprev's vLLM served-names are unchanged. - Verified e2e through the gateway: sec answers (nmap -sV version detection), sec-reasoning answers with a thinking split (127 reasoning tokens); retired mog-sec-27b now 400s. Note: mog-sec was the fleet's only offense+defense/blue-team seat; the sec role is now offense-only (cyberprev tool-calling). Operator-directed after reviewing the capability comparison. mog-sec stack files retained for a future restore. |
||
|
vh
|
3906c6842c |
docs: correct sec-seat lineage — M.O.G.-SEC/mog-sec is an offense+defense SFT finetune, not a persona-on-stock
The sentinel-r3 header and two memory notes described mog-sec (Blackfrost
M.O.G.-SEC / Qwentium) as 'a persona system prompt on stock weights'. Its card
is explicit that it is NOT: base_model_relation: finetune on Qwen/Qwen3.8-27B,
a refusal-free offense+defense cybersecurity SFT with YaRN 1M context ('not a
system-prompt sticker on a stock Qwen'). So all three sec-seat candidates are
Qwen3.8-27B SFT finetunes and differ in training focus, not in kind:
mog-sec = broad offense+defense SFT; sentinel-r3 = pentest agent-trajectory SFT;
cyberprev = cyber tool-calling LoRA SFT on an abliterated base.
|
||
|
vh
|
196416f3da |
docs(cyberprev-seat): correct model lineage — it is a cyber SFT finetune, not an abliteration
Earlier commits and the compose header called cyberprev "an abliteration of Qwen3.8-27B (refusal-direction weight edit, no finetune)". That was wrong. Per the hotdogs model card (base_model_relation: finetune), cyberprev is an offensive-security agentic tool-calling LoRA, merged at scale 1.0, trained on hotdogs/cyber-sft-agent-qwen38 (8,400 rows across 22 pentest tools: nmap, sqlmap, metasploit, hydra, crackmapexec, ...). The abliteration is INHERITED from its base (hotdogs/Qwen3.8-27B-abliterated); the new capability in this repo is the cyber SFT itself. So the sec-seat lineage is: mog-sec = persona on stock weights; sentinel-r3 = pentest-trajectory SFT on stock weights; cyberprev = cyber tool-calling SFT on an abliterated base. All three are SFT-or-prompt variants; the earlier framing of cyberprev as "not a finetune" was backwards. Corrects the compose header lineage block, the homepage dashboard description, and the gateway alias comment. No functional change; the running seat and its weights are unchanged (label refreshes on next container recreate). |
||
|
vh
|
b94d173887 |
fix(cyberprev-seat): serve coherent v2 quant from name-repaired source; wire gateway
The first cyberprev quant served pure gibberish despite passing every structural check. Root cause: the hotdogs/Qwen3.8-27B-abliterated-cyber-preview checkpoint ships broken -- 850 of 1199 tensors (the whole transformer body) carry a TRIPLED `language_model.` prefix, confirmed present in the upstream HF index (an unsloth export bug, not our download). transformers silently loaded the unmapped tensors as random-init, the pipeline quantized noise, and structural verification passed because it inspects the output config, never that input names mapped. A plain greedy coherence smoke caught it; nothing structural could. Fix: rewrite tensor names against a known-good reference of the same architecture (qwen38-27b-uncensored-bf16), verified the repaired name-set == reference name-set exactly (1199 tensors, 0 collisions), and re-quantized from the corrected source (cyberprev-bf16-fixed). v2 is coherence-verified (spec off AND live with dflash): Paris/Berlin/Rome, correct arithmetic precedence, primes, clean cyber-domain chat with a proper thinking split. dflash k=7 measured at 2.77 mean acceptance length. Seat serves cyberprev-27b / cyberprev-27b-thinking on :8025/GPU0 with dflash k=7, displacing the retired sentinel-r3. Gateway: removed the retired sentinel-r3 / sentinel-r3-reasoning aliases and added cyberprev-27b / cyberprev-reasoning -- NOT repointed, since serving cyberprev weights under sentinel's name is silent substitution. Verified end-to-end through the gateway; sentinel-r3 now 400s. Playbook gains a superseded-claims row: structural verification does not imply a good quant; a coherence smoke is the load-bearing gate, and third-party sources must have their tensor names diffed against a reference before quantizing. Broken v1 quant dir removed. |
||
|
vh
|
1418edb03e |
feat(cyberprev-seat): mixed-NVFP4 quant of the abliterated cyber-preview, displacing sentinel-r3
Third sec-seat candidate: hotdogs/Qwen3.8-27B-abliterated-cyber-preview, an abliteration (refusal-direction weight edit) of Qwen3.8-27B aimed at the cyber-offense refusal surface -- distinct from mog-sec (persona on stock weights) and sentinel-r3 (SFT finetune). Operator instruction: quant it, take sentinel down, serve it with mtp or dflash. Quantized to the house mixed recipe via services/gen-seat-mixed-quant/ (NVFP4 W4A4 on MLP layers 0-55 + FP8 W8A8 on attn/linear_attn/lm_head/MLP 56-63, FP8 KV). The prior attempt (2026-09-11/14) died with "Cannot determine num_attention_heads" because it ran from a bare .venv whose newer compressed-tensors reads that field at top level; quant_mixed_nvfp4.py already promotes text_config attention fields for exactly this reason, and the run through the canonical vLLM-image + llmcompressor 0.13.0 / compressed-tensors 0.18.0 path (versions recorded from the container) completed clean. post_quant.py did its job: grafted the 15 MTP tensors verbatim (BF16), and re-injected re:^mtp.* into the ignore list after llm-compressor pruned it for matching no module at quant time -- without which vLLM loads the grafted head uninitialised and speculative decoding runs at 0% acceptance. It also caught a missing preprocessor_config.json (absent from the abliterated source AND its hotdogs upstream); restored from Qwen/Qwen3.8-27B, verified byte-identical to the working sentinel-r3 build, so the vision tower preprocesses. Verified from the tensors, not the config: NVFP4 covers MLP 0-55, FP8 covers 56-63, no overlap, 168 weight_packed tensors (56x3), 15 BF16 MTP, 333 BF16 vision. 51.0 GiB bf16 -> 21.0 GiB. Served under its own name (cyberprev-27b / -thinking), NOT sentinel-r3 -- serving different weights under a retired name is silent substitution. Takes over :8025 and GPU 0 (co-resident with mog-sec). dflash k=7 is the default, carried from the sentinel measurement, but is re-measured against MTP on this ABLITERATED body before cutover, since abliteration is exactly what can desync an MTP head. Drafter (qwen38-27b-dflash2-drafter) shared with mog-sec. Context capped at 163840 not native 262K, inherited from mog-sec's hard-won depth lesson; raise only after a non-repeating deep-prefill probe on this checkpoint. |
||
|
vh
|
4954cf6756 |
test(flash-next-seat): depth-probe orca to 258K clean; record a blind instrument
Bisected context depth on the orcarouter checkpoint with non-repeating prompts (unique random hex per probe, so prefix caching cannot short-circuit the prefill). Six depths from 31,978 to 258,517 tokens, all served. The load-bearing evidence is the engine allocator log: zero OOM, CUBLAS, or illegal-memory entries across the run. That is the same detector that caught the dealignai near-miss at 155K on the previous checkpoint, where it did fire. The probe also ran under real concurrent operator load, making it a stricter test than a solo run rather than a weaker one. Positive control passed: a mis-sized first attempt produced a ~265K-token prompt and got a clean 400 naming the limit instead of killing the engine, so the probe could detect the failure mode it was looking for. Calibration for re-runs: random hex words tokenize at 7.9 tokens/word here. vLLM #54919 (long prefill starving decode for 3-7 minutes) did not reproduce: 258K prefilled in 28.9 s, roughly 8,900 tok/s, scaling near-linearly from 32K. Records that the probe's memory-headroom half was BLIND and must not be reused. It reported an identical 95,460 MiB used / 2,427 MiB free on every row across an 8x range of depths, which is the tell. Two causes: --kv-cache-memory pins the pool and the engine logs "skipped memory profiling", so GPU usage is flat with respect to depth; and the actual risk is a transient activation spike during prefill, which before/after nvidia-smi bracketing structurally cannot observe. Peak-activation headroom therefore remains unmeasured; the pass/fail result rests on the allocator log alone. Also qualifies the earlier 167.5 tok/s decode figure as a possibly-contended lower bound, and records the operator's independent 140 tok/s average measured in real use while this probe was loading the same card. |
||
|
vh
|
1b5d6ba23a |
docs(flash-next-seat): dealignai weights deleted — record that no local rollback exists
Operator instruction: delete the displaced dealignai checkpoint. 125 GiB reclaimed from /tank (59% -> 57% used). Verified before removing: not mounted by any running or exited container, no symlinks, no inodes shared with the converted orcarouter directory. Every "rollback is two .env keys" statement across the stack README, the .env.example, persistent-memory and its detail file was true when written and is false now -- the .env backup still names paths that no longer exist. Corrected in place rather than left as false reassurance, since a stale rollback instruction is discovered precisely when it is needed. Reverting this seat now costs a 126 GiB re-download. The quality A/B against dealignai is likewise no longer runnable locally: its reference arm is gone. The pristine 170 GiB orcarouter download is retained deliberately -- it is what makes the PLE bf16->FP8 conversion reproducible without re-fetching -- and that is now recorded so a future session does not reclaim it as an obvious duplicate. Also notes that ~75 GiB of non-PLE shards are duplicated between the pristine and converted orca directories (the convert's hardlinks hit EXDEV across two container bind mounts); both now sit directly on /tank, so relinking would reclaim it if /tank ever tightens. |