From 6f0a9b9faede6b57612c77761f0babfed7cb5f9f Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 21 Sep 2026 23:11:54 -0700 Subject: [PATCH] feat(alerts): generalize the althing bridge, wire Kuma to it, retire chamber MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit THE BRIDGE. `beszel-althing` hardcoded a "[Beszel] " subject prefix and a Beszel hub footer from when Beszel was its only caller. Routing Uptime Kuma through it unchanged would have delivered Kuma outages labelled [Beszel], pointing the reader at the wrong dashboard -- an alert that lies about its own source is worse than no alert. Now a route registry: /beszel and /kuma, each with its own prefix, footer and payload parser, because the tools do not agree on a shape (Beszel sends {title, message}; Kuma sends {heartbeat, monitor, msg}). Generalising cost a dict; a sibling service would have cost a second unit, a second port and a second thing to notice had died. Renamed beszel-althing -> althing-alert-bridge with it. A service named after one consumer that carries two is the invisible coupling that sends a future session looking in the wrong place. ⚠ /beszel IS FROZEN and this refactor proves it rather than claiming it. The three original tests were kept BYTE-UNCHANGED -- including the one asserting the exact postbox argv -- and deliver() still defaults to the Beszel route so they exercise it. A new test asserts the Kuma footer never leaks into a Beszel body or vice versa. Verified live after the rename: a real POST to /beszel landed as "[Beszel] BRIDGE RENAME CHECK" with the correct hub footer, read back from the thread rather than trusted from the receipt. Payload shapes are parsed HERE, not via Kuma's custom-webhook-body feature, because Kuma's notification config lives in its own database -- and that database was destroyed and rebuilt from scratch hours ago. Anything living only in a tool's DB is lost on the next rebuild; format knowledge belongs in git, next to a test. parse_kuma also handles the monitorless case. testNotification and cert-expiry alerts carry no monitor and no heartbeat, and the first cut fabricated "unknown monitor is ?" from them -- caught by sending a real one and reading the subject, not by the suite. Fixed, pinned, and the earlier test asserting the bad behaviour was corrected rather than worked around. KUMA IS NOW WIRED. scripts/kuma gained notification support and the channel is in monitors.yaml, seeded BEFORE the monitors and with applyExisting, so a rebuild restores alerting and not just detection. Ground truth from the DB: 13 of 13 monitors carry the channel. ⚠ A THIRD instance of the same class of bug, worth naming: notifications() is pushed as `notificationList` at LOGIN ONLY -- there is no event to ask with. The first cut cleared the captured value before waiting, discarding the only copy it would ever be sent, then blocked for the full timeout and reported an empty list. That reads exactly like "no channels configured" and is a lie. Same family as the getMonitorList ack-vs-push trap, different shape. End-to-end, both shapes, read back from the inbox: [Uptime Kuma] Homepage is DOWN + target + board link [Uptime Kuma] althing (infra-ops) Testing (no fabricated subject) [Beszel] BRIDGE RENAME CHECK + hub footer, unchanged ALTHING CHAMBER RETIRED (operator). Three of its four containers had never started -- created 2026-09-19, StartedAt epoch-zero, 0 restarts -- so :7881 refused, and nothing was watching it. Only its valkey was running, on the project's own network with no external consumer. Stack, compose/build/conf dirs and the local image removed; the Homepage card went with the label. --- graphify-out/GRAPH_REPORT.md | 580 ++++++++++-------- playbooks/althing-alert-bridge.yaml | 33 + playbooks/beszel-althing.yaml | 20 - playbooks/deploy-althing-chamber.yaml | 147 ----- scripts/kuma | 69 +++ services/althing-alert-bridge/README.md | 71 +++ .../althing-alert-bridge.service | 27 + services/althing-alert-bridge/bridge.py | 162 +++++ services/althing-alert-bridge/test_bridge.py | 109 ++++ services/beszel-althing/README.md | 36 -- .../beszel-althing/beszel-althing.service | 24 - services/beszel-althing/bridge.py | 63 -- services/beszel-althing/test_bridge.py | 30 - stacks/althing-chamber/.env.example | 27 - stacks/althing-chamber/README.md | 110 ---- stacks/althing-chamber/compose.yaml | 124 ---- .../gitea-workflow-deploy.yaml.example | 91 --- stacks/beszel/README.md | 2 +- stacks/uptimekuma/monitors.yaml | 20 +- 19 files changed, 818 insertions(+), 927 deletions(-) create mode 100644 playbooks/althing-alert-bridge.yaml delete mode 100644 playbooks/beszel-althing.yaml delete mode 100644 playbooks/deploy-althing-chamber.yaml create mode 100644 services/althing-alert-bridge/README.md create mode 100644 services/althing-alert-bridge/althing-alert-bridge.service create mode 100644 services/althing-alert-bridge/bridge.py create mode 100644 services/althing-alert-bridge/test_bridge.py delete mode 100644 services/beszel-althing/README.md delete mode 100644 services/beszel-althing/beszel-althing.service delete mode 100644 services/beszel-althing/bridge.py delete mode 100644 services/beszel-althing/test_bridge.py delete mode 100644 stacks/althing-chamber/.env.example delete mode 100644 stacks/althing-chamber/README.md delete mode 100644 stacks/althing-chamber/compose.yaml delete mode 100644 stacks/althing-chamber/gitea-workflow-deploy.yaml.example diff --git a/graphify-out/GRAPH_REPORT.md b/graphify-out/GRAPH_REPORT.md index 8fcbe7a..f8df98e 100644 --- a/graphify-out/GRAPH_REPORT.md +++ b/graphify-out/GRAPH_REPORT.md @@ -1,16 +1,16 @@ -# Graph Report - eshpfi-management (2026-09-19) +# Graph Report - eshpfi-management (2026-09-21) ## Corpus Check -- 635 files · ~854,877 words +- 630 files · ~853,619 words - Verdict: corpus is large enough that graph structure adds value. ## Summary -- 5578 nodes · 6128 edges · 640 communities (484 shown, 156 thin omitted) -- Extraction: 98% EXTRACTED · 2% INFERRED · 0% AMBIGUOUS · INFERRED: 105 edges (avg confidence: 0.81) +- 5385 nodes · 5633 edges · 660 communities (499 shown, 161 thin omitted) +- Extraction: 99% EXTRACTED · 1% INFERRED · 0% AMBIGUOUS · INFERRED: 49 edges (avg confidence: 0.74) - Token cost: 0 input · 0 output ## Graph Freshness -- Built from commit: `4e778aed` +- Built from commit: `3a85a6bc` - Run `git rev-parse HEAD` and compare to check if the graph is stale. - Run `graphify update .` after code changes (no API cost). @@ -29,7 +29,7 @@ - CLAUDE.md - PFI-ANA Docker Stack - restic / esh-vm-db -- properties +- togglable - Deploy (one-time setup) - Deploy (one-time setup) - Deploy (one-time setup) @@ -62,7 +62,7 @@ - skaldsong - ana-nas → nh3-nas restic mirror - asset-engine -- _body +- fixture - CatalogModel - FieldOption - Chatterbox Turbo @@ -93,7 +93,7 @@ - pfi-pteradactyl - 6. Quick Reference Cards - secret -- test_booth.py +- uptimekuma — the fleet's SERVICE-layer monitor (ana-docker) - pfi-tacticalrmm - Qwen3-TTS - configs/restic @@ -123,7 +123,7 @@ - task-board - vor - Zonos -- test_asks.py +- parametrize - Fleet bootstrap — read this first - CatalogSectionGroup - 3. Per-model breakdowns @@ -136,8 +136,8 @@ - Phase 0 — Prerequisites (one-time, before Phase 1) - ace-step - `[2026-09-01]` pfi-gx10 (ASUS Ascent GX10) onboarded headless — and it is the intended new home for run 3c -- _touch -- togglable +- `[2026-09-01]` A named failure class: a correct check aimed at the wrong object +- score_beats.py - settings.local.json - Observability for the vLLM stack (ana-ml2) - server_inspect.sh script @@ -154,7 +154,7 @@ - `[2026-09-15]` FV cross-site routing fixed — one NAT rule scoped to Anaheim only - hash_gate_repro.py - ops-log -- lv-mccarthy — corpus → gate → pairs +- lv-mccarthy — corpus → leak gate → pairs → train → v2 gate - 7. Critical Warnings by Model - think-leak — reproducers for the h300 unterminated-`` defect - cold_start_s @@ -199,7 +199,7 @@ - OmniVoice - entrypoint.sh script - DeliveryTests -- [2026-08-15] Uncensored gen seat: Qwen3.8-27B-Uncensored deployed; the definitive MTP-graft fix +- `[2026-09-04]` Forcing 10G on the ESH-Media DAC FAILED progressively — and I called the plateau too early - mOrpheus — uncensored Orpheus TTS (irv-ml1) - check-backup-freshness.sh - Abliteration recipe — Qwen3.8-27B (MTP-aware, vision-preserving) @@ -236,7 +236,7 @@ - `[2026-08-20]` The Heretic-300 epic — Cold-Fusion abliteration, end to end - eval_w4a16.json - ppl_mixed.json -- _board +- Path - ~26k - ESH-Media 10G uplink — the DAC that negotiated 1G, and the fix - _Finder @@ -268,7 +268,7 @@ - mtp_n5.json - probe_nvfp4.json - `[2026-08-27]` The dose-response says benefit and damage are ONE direction in weight space -- build_gallery +- Refusal retention — the axis the gate did not have, and the axis I measured wrong - gen-seat mixed-precision quant — NVFP4 W4A4 MLP + FP8 W8A8 attention - probe.py - bench_aeon.json @@ -292,7 +292,7 @@ - Dark-Scarlett regeneration spec — the 8 dropped axes - muninn-gate - Refusal probe — char-rp-reasoning seat -- booth/__init__.py +- gpu.py - esh-vm-docker-watchdog.sh - zed-fim-proxy - mimir-inbox @@ -303,8 +303,8 @@ - HookTests - esh-vm-db/pre-backup.sh - pull-verify-jenerallee78.sh -- uptimekuma — service uptime monitor (esh-docker-vm) -- The Booth +- mime_from_field +- The Booth — moved to its own repository - `[2026-08-20]` Cold-Fusion abliteration LANDED — layer 35, and the three false diagnoses corrected - ~26k - mtp_coldfusion_L35.json @@ -321,13 +321,13 @@ - A VM wedges on IO and the disk is innocent - index-tts/app.py - update.sh script -- 2026-09-06-headscale-mesh-phase1.md +- worker/app.py - DFlash2 speculative decoding — measured on our own stack (2026-08-22) - `[2026-08-26]` Run 3 corpus built, then held — and the megamix containment defect it exposed - Ops lessons playbook — the transferable ones - launch-run-06.sh - refusal_probe.py -- worker/app.py +- invocation.py - Beszel fleet monitoring - Author-voice LoRA regime on pfi-gx10 — training-side prep - eval_orcarouter_ppl.json @@ -367,7 +367,7 @@ - ERP/RP tune run-01 COMPLETE — 7.36h, gate passed on the axis it was built for - judge-bench — evaluate a candidate LLM-as-judge seat - NVFP4A16 serving pipeline — built, validated, and the MoE landmine it found -- booth +- CatalogField - `[2026-08-27]` Anaheim tripped a power breaker — and four guests including the NAS never came back - [2026-08-23] hrafn adopted; its CI deploy reported green while deploying nothing - [2026-08-23] Worldtree b187 shipped; all three instances de-armed from a 69-day-stale `:latest`; Matrix homeserver re-plumbed @@ -384,7 +384,7 @@ - Run 7 retired, Pfish-6 declared, and the detector bug that nearly buried it (2026-09-09) - `[2026-09-01]` The Ada inference server is a Dell R750xa bought used — and the reseller stripped four things Dell shipped - entities.py -- `[2026-09-01]` althing 3.2.0 deployed on nh3-dev — pane routes, and `uv tool install .` lies +- `[2026-09-04]` esh-nas SMB account for the AudioGridder box — and the NAS is effectively open to the whole LAN - recover_eval_series.py - `[2026-08-28]` althing v3.0.0 flag day (U9b) — the post office replaced the P2P bus, one-way - counted_classifier.py @@ -405,7 +405,7 @@ - quant_with_gen_down.sh - `[2026-08-19]` Homepage cleaned up, then themed with Australis Skyfall + an Arbo-generated background - `[2026-09-15]` Fleet identity/group/path conventions pinned + docker trees normalized -- services.schema.json +- license - `[2026-08-28]` A stale `ALTHING_HANDLE` silently reads ANOTHER agent's inbox and reports it empty - 2026-09-08-run6-gate-csam-adjudication.md - `[2026-08-20]` Cold-Fusion abliteration — Robinson recipe captured, and the transformers/DeltaNet bf16-NaN fight @@ -415,55 +415,52 @@ - voice_distance.py - Provisioning a Mac for the fleet - `[2026-09-03]` Run 3c STAGED on pfi-gx10 — verified end to end, deliberately NOT launched -- CatalogResponse -- status +- properties +- stream_chunks - 2026-09-09-ana-ml2-pool-actions-deferred.md - MeroMero NVFP4A16 quants (Gemma-4) — instruments and the failure chain - searxng_mcp/server.py - provision-mac-dsh.sh - Path - build.py -- accepted_types +- status - OPNsense API — vendored reference + field-shape notes - `[2026-08-19]` esh-pve hard-froze for 4.5h — and took the whole house's DNS with it - erp-seat/README.md - `[2026-08-19]` Four unmanaged stacks found on live hosts — and two of them were quietly broken -- `[2026-09-04]` ERP run 3c trained on pfi-gx10 and GATED — the 20x LR cut erased the diversity gain and did NOT remove the safety regression +- 2026-09-21-booth-two-dead-controls.md - searxng-health.sh - searxng-mcp -- category +- max - pfi-gx10 — ERP-seat SFT run 3c, staged and ready to launch -- 2026-09-05-floor-claim-n2-retraction.md -- step +- name - `[2026-09-15]` ESPHome modernised for ha-dev; `kb` search tool for the personal Worldtree KB - pfi-gx10 — ERP-seat SFT run 7 (opening-split slot) - `[2026-09-10]` MeroMero — both quants landed; the v2 dense took five attempts - launch-run-05.sh - 2026-09-06-pfi-pve-naspool-raidz2-rebuild.md - Current state / in-flight -- `[2026-09-04]` `gen` moved to ana-ml2 GPU0 — and I sized it against the wrong number, twice in one hour +- 2026-09-21-disk-triage-and-the-rescued-adapter.md - launch-run-03c.sh -- `[2026-09-04]` Forcing 10G on the ESH-Media DAC FAILED progressively — and I called the plateau too early -- content_type -- `[2026-09-04]` esh-nas SMB account for the AudioGridder box — and the NAS is effectively open to the whole LAN +- 2026-09-21-gitea-orgs-and-the-dead-token.md +- max_length - remote-ssh - verify_corpus.py - [2026-09-18] althing search returned ZERO for every hyphenated query — silently, for everyone -- booth - tok_repro.py -- description -- 2026-09-17-esh-fiber-outages.md +- Path +- 2026-09-08-esh-static-wan-followups-and-ytvc.md - 4. Pipeline shape - pov_gender.py - `[2026-09-10]` althing 3.6.2 rolled — post office + both heralds, and it was TWO nodes not seven - FV to Anaheim: scoped Tailscale NAT repair - setup-nh3-scale-socks-egress.sh -- max_length +- [2026-08-24] ana-gw public admin surface closed to zero, ACME listener included - 2026-09-10-beszel-fleet-wiring.md - bench_lora_step.py - probe_carrier.py - run_quant_erp_v7.sh -- invocation.py +- lv-mccarthy-4b-v1 - tokenize_name_pool.py - Path - generate_arms.py @@ -472,7 +469,7 @@ - beszel-althing/README.md - 2026-09-12-esh-vm-db-restic-repair.md - build_booth_yarros.py -- method +- `[2026-09-21]` lv-mccarthy: gate run, NOT SHIPPED — the voice works, the memorisation is the cleanest in the line, and the length discipline is gone - chain-instruct-probe.sh - Parakeet STT on fv-ml1 GPU 3 (2026-09-15) - BackupHookTests @@ -491,7 +488,7 @@ - 2026-09-10-ana-docker-disk-repair.md - char-rp-fast — swapping the MeroMero A4B onto the erp-seat seat (2026-09-10) - seat_verify.py -- Path +- content_type - build_booth.py - deep_ctx_probe.py - Plex hardware transcoding on the Arc A580 (esh-pve-nas, LXC 105) @@ -529,7 +526,7 @@ - heretic2-charrp-reasoning — NVFP4 + native MTP reasoning seat (fv-ml1) - build_sft_pairs.py - FleetTools — what this system can do, and where the details live -- BaseModel +- rename.py - gen_beats_chat_yarros.py - Arbo — catalog-driven image generation engine - 2026-09-08-yt-voice-clipper-back-up.md @@ -552,7 +549,7 @@ - irv-ml1 dead-address sweep — EXECUTED 2026-09-15 (operator-directed) - fv-mesh-watchdog.sh - grok-token-broker -- optional +- recover_direction - kb - 2026-09-10-an-author-voice-adapter-transfers-subject.md - 2026-09-10-babybronte-h02-adapter-the-voice-transferred.md @@ -581,28 +578,27 @@ - Fleet orientation — start here - verify_quant.py - [2026-08-24] ESH DNS fixed at the IPv6 layer, and the naming scheme went live -- booth/app.py - gen_booth_four_arms.py - memorization_check.py -- source_jsonpath -- score_beats.py +- `[2026-09-17]` lv-hemingway: SHIPPED on ckpt850 — the line's first clean voice pass, and one axis that needs reading +- description - Direct inference seats + the Asset Engine control plane -- properties +- lv-mccarthy v2 gate — pre-registration - LiteLLM gateway — the fleet's OpenAI-compatible front door - Fleet SSH — the infra-ops identity and host map - build_beat_fixture.py - The Booth — ephemeral media board + operator decisions - eshpfi-management - source_url -- $ref +- section_groups - pin_known_gender.py - Speech — TTS and ASR endpoints -- type +- `[2026-09-17]` lv-mccarthy D1→D3 — built, gated, and every stage caught a defect in the stage before it - repair_corpus_bronte.py - 2026-09-16-grok-broker-shelved.md - 2026-09-17-lv-bronte-gate.md -- endpoint -- estimated_latency +- properties +- method - Path - memorization_selfsim.py - eval-hemingway.sh @@ -610,52 +606,76 @@ - voice-prep.py - althing — inter-agent message bus - Known gotchas (save yourself time) -- `[2026-09-01]` A named failure class: a correct check aimed at the wrong object -- Refusal retention — the axis the gate did not have, and the axis I measured wrong +- show_memorisation_matches.py +- services.schema.json - apply_gender_overrides.py - min - elway — SSH playbook runner (CHANGING things) - graphify — repo knowledge graph -- host +- `[2026-09-17]` Romantasy measured as a register — it is real, we already took its best voice, and the obvious next pick is its worst - Path - Observability — where to look when something is wrong - henge.md - playwright.md -- license -- version +- `[2026-09-17]` lv-bronte: corpus gated for real, adapter trained, SHIPPED with a FAILED voice axis on the record +- eval-mccarthy.sh - install.sh - post-commit-ops-log +- mccarthy-corpus/voice-prep.py +- `[2026-09-17]` McCarthy and Krakauer D1 — both corpora built, and one of them should not be trained yet +- source_jsonpath - Backup architecture (two layers) - license_warning +- `[2026-09-17]` The leak gate passed with five protagonist names still in every copy +- `[2026-09-04]` ERP run 3c trained on pfi-gx10 and GATED — the 20x LR cut erased the diversity gain and did NOT remove the safety regression +- Design decisions, and why +- category +- $ref +- AMENDMENT-3 ROUND: five arms, and ckpt300 wins every axis — held at the gate by 0.02 +- options +- BaseModel +- streamable +- required +- orca-slice +- SHIPPED 2026-09-21 18:01 PT — `lv-mccarthy` = checkpoint-300 on `voices-seat` (fv-ml1 GPU0 :8027) +- estimated_latency +- host +- ship-voice-adapter.sh +- `[2026-09-17]` Which voices earn a training seat next — measured against the catalogue, not chosen by taste +- lifecycle +- adapter/server.py +- record_to_ops_log +- _Tee +- sanitize_tts_text ## God Nodes (most connected - your core abstractions) -1. ``[2026-09-01]` althing 3.2.0 deployed on nh3-dev — pane routes, and `uv tool install .` lies` - 36 edges -2. `_touch()` - 30 edges -3. `[2026-08-15] Uncensored gen seat: Qwen3.8-27B-Uncensored deployed; the definitive MTP-graft fix` - 27 edges -4. `_board()` - 25 edges -5. `parse_link_entries()` - 24 edges -6. `AskError` - 21 edges -7. `Recent decisions (archived 2026-08-03 batch)` - 21 edges -8. `build_command()` - 21 edges -9. `VM 102 — Matrix Synapse Deployment` - 21 edges -10. `_ask()` - 19 edges +1. ``[2026-09-04]` esh-nas SMB account for the AudioGridder box — and the NAS is effectively open to the whole LAN` - 37 edges +2. ``[2026-09-04]` Forcing 10G on the ESH-Media DAC FAILED progressively — and I called the plateau too early` - 29 edges +3. `Recent decisions (archived 2026-08-03 batch)` - 21 edges +4. `build_command()` - 21 edges +5. `VM 102 — Matrix Synapse Deployment` - 21 edges +6. `JobManager` - 18 edges +7. `3. The recurring landmines` - 18 edges +8. `DFlash2 speculative decoding — measured on our own stack (2026-08-22)` - 18 edges +9. ``[2026-08-28]` althing v3.0.0 flag day (U9b) — the post office replaced the P2P bus, one-way` - 18 edges +10. `Headscale overlay mesh — plan (2026-09-06)` - 17 edges ## Surprising Connections (you probably didn't know these) - `_read_cli_credential()` --calls--> `walk()` [INFERRED] services/grok-token-broker/broker.py → scripts/kb-search.py - `convert()` --references--> `Any` [EXTRACTED] tools/mistral-small4-nvfp4/convert_hf_to_native.py → stacks/omnivoice/app.py -- `list_booths()` --calls--> `list_asks()` [INFERRED] - services/booth/booth/app.py → services/booth/booth/asks.py -- `test_gallery_hides_ask_and_answer_files()` --calls--> `build_gallery()` [INFERRED] - services/booth/tests/test_asks.py → services/booth/booth/app.py -- `test_list_booths_counts_open_asks()` --calls--> `list_booths()` [INFERRED] - services/booth/tests/test_asks.py → services/booth/booth/app.py +- `_speaker_embedding()` --references--> `Tensor` [EXTRACTED] + stacks/zonos/adapter/server.py → services/coldfusion-abliteration/kl_divergence.py +- `read_tensor()` --references--> `Tensor` [EXTRACTED] + tools/mistral-small4-nvfp4/convert_hf_to_native.py → services/coldfusion-abliteration/kl_divergence.py +- `split_experts()` --references--> `Tensor` [EXTRACTED] + tools/mistral-small4-nvfp4/convert_hf_to_native.py → services/coldfusion-abliteration/kl_divergence.py ## Import Cycles - None detected. -## Communities (640 total, 156 thin omitted) +## Communities (660 total, 161 thin omitted) ### Community 0 - "quant_modelopt.py" Cohesion: 0.53 @@ -713,9 +733,9 @@ Nodes (18): 1. Dockge — Docker Compose Management UI, 2. llama-swap — Multi- Cohesion: 0.11 Nodes (17): 1. Create rest-server-ana htpasswd entry, 2. Install secrets on esh-vm-db, 3. Initialize the repo, 4. Install prerequisites (restic, resticprofile, mongosh client), 5. Deploy profile + hook, 6. Schedule + verify, Deploy (one-time), Full host config (+9 more) -### Community 14 - "properties" -Cohesion: 0.09 -Nodes (23): properties, default, title, anyOf, default, title, title, type (+15 more) +### Community 14 - "togglable" +Cohesion: 0.40 +Nodes (5): togglable, default, description, title, type ### Community 15 - "Deploy (one-time setup)" Cohesion: 0.12 @@ -845,13 +865,9 @@ Nodes (8): ana-nas → nh3-nas restic mirror, Deploy / redeploy, Files, Operate, Cohesion: 0.22 Nodes (8): asset-engine, Automated (Gitea Actions, push-to-main), Catalog drift, Deploy, Manual (elway from a workstation), Network model, Outputs directory growth, Path layout (on ana-docker) -### Community 47 - "_body" -Cohesion: 0.18 -Nodes (11): _body(), Rendered body only — the stylesheet mentions class names too., Filtering links.md out of the gallery leaves items empty — the empty state must…, Same rule as the kept lane: no single click destroys a durable board., test_a_genuinely_empty_booth_still_says_so(), test_board_booth_counts_links_not_files(), test_board_booth_does_not_claim_to_be_empty(), test_board_booth_has_no_one_click_wipe() (+3 more) - ### Community 48 - "CatalogModel" -Cohesion: 0.12 -Nodes (17): additionalProperties, properties, required, title, type, CatalogModel, title, type (+9 more) +Cohesion: 0.14 +Nodes (14): additionalProperties, properties, required, title, type, CatalogModel, anyOf, default (+6 more) ### Community 49 - "FieldOption" Cohesion: 0.13 @@ -926,8 +942,8 @@ Cohesion: 0.25 Nodes (7): backrest, Backup scope, when wired up, Deploy, First-time configuration (in the UI), Role in the fleet, Scaling knobs, Why not `offen/docker-volume-backup`? ### Community 67 - "omnivoice/app.py" -Cohesion: 0.06 -Nodes (46): Any, ChunkConfig, ChunkResult, ClockFn, GenerateFn, _base_gen_kwargs(), _chunk_config(), GenParams (+38 more) +Cohesion: 0.09 +Nodes (26): Any, ChunkConfig, ChunkResult, _base_gen_kwargs(), _chunk_config(), GenParams, _log_chunk(), _pcm16() (+18 more) ### Community 68 - "calibre-web-automated" Cohesion: 0.25 @@ -969,9 +985,9 @@ Nodes (8): 6. Quick Reference Cards, Min-P by Model, Presence Penalty by Model, Cohesion: 0.26 Nodes (23): Exception, bw(), BwError, cmd_backfill(), cmd_get(), cmd_list(), cmd_put(), cmd_rm() (+15 more) -### Community 78 - "test_booth.py" -Cohesion: 0.05 -Nodes (33): generate_pickup_id(), (rendered, is_html). Markdown → HTML (fenced code, tables, sane lists); plain…, Inject a floating 'all booths' back-chip — and the Booth favicon, if the page…, Reduce a client-supplied filename to a safe basename (no path, no hidden)., A human-readable id like '4-wombat' or 'star-84'. `exists(name)->bool` gates…, render_doc(), safe_upload_name(), wrap_verbatim_html() (+25 more) +### Community 78 - "uptimekuma — the fleet's SERVICE-layer monitor (ana-docker)" +Cohesion: 0.29 +Nodes (6): Data, Rebuilt from scratch, 2026-09-21, Scripting it, The Homepage widget is deliberately absent, uptimekuma — the fleet's SERVICE-layer monitor (ana-docker), Why it exists — the lane split ### Community 79 - "pfi-tacticalrmm" Cohesion: 0.25 @@ -1006,8 +1022,8 @@ Cohesion: 0.25 Nodes (7): API, Deploy, Gotchas, Switching to the 7B variant, VibeVoice, Voices, Why this stack exists ### Community 87 - "properties" -Cohesion: 0.14 -Nodes (14): properties, anyOf, default, $ref, lifecycle, model, reproducibility, response (+6 more) +Cohesion: 0.15 +Nodes (13): properties, title, type, $ref, endpoint, model, reproducibility, response (+5 more) ### Community 88 - "Dia / Dia2" Cohesion: 0.29 @@ -1062,8 +1078,8 @@ Cohesion: 0.22 Nodes (8): Address structure, Applying it to a host, Deployed, ESH IPv6 naming scheme, Getting into a host with no direct root, ⚠ Proxmox bridges need `accept_ra=2` or SLAAC never runs, The names, What can and cannot carry a name ### Community 101 - "fv-ml1" -Cohesion: 0.22 -Nodes (8): fv-ml1, GPU allocation policy, Hardware, Key paths, Network, NVMe drive inventory (`tank`, raidz2-0) — read 2026-09-09 via `nvme-cli`, Refresh state, Running stacks +Cohesion: 0.12 +Nodes (15): Derived worst case, fv-ml1, GPU allocation policy, Hardware, Key paths, Measured (BMC, 2026-09-19, GPUs at idle), Network, NVMe drive inventory (`tank`, raidz2-0) — read 2026-09-09 via `nvme-cli` (+7 more) ### Community 102 - "NH3 egress proxy (nh3-dev)" Cohesion: 0.29 @@ -1089,17 +1105,13 @@ Nodes (6): Automated (Gitea Actions, push-to-main), Centralized vs local, Deploy Cohesion: 0.29 Nodes (6): Adapter request (example), Deploy, Notes, Two surfaces: Gradio eval (8199) + REST adapter (8203), Why this stack exists, Zonos -### Community 108 - "test_asks.py" -Cohesion: 0.07 -Nodes (65): parametrize, ask_stem(), AskError, _blank(), _bool(), _clean_notes(), list_asks(), load_ask() (+57 more) - ### Community 109 - "Fleet bootstrap — read this first" Cohesion: 0.40 Nodes (4): Deeper context, Fleet bootstrap — read this first, Non-negotiables, Three endpoints that answer "what exists right now" ### Community 110 - "CatalogSectionGroup" -Cohesion: 0.18 -Nodes (11): additionalProperties, description, properties, required, title, type, CatalogSectionGroup, anyOf (+3 more) +Cohesion: 0.14 +Nodes (14): additionalProperties, description, properties, required, title, type, CatalogSectionGroup, anyOf (+6 more) ### Community 111 - "3. Per-model breakdowns" Cohesion: 0.06 @@ -1141,13 +1153,13 @@ Nodes (4): ace-step, API surface, Deploy, Tunables Cohesion: 0.06 Nodes (31): `[2026-09-01 23:09]` Liveness check — alive, idle, and STILL BARE (nothing has changed), `[2026-09-01 23:2x]` OUTFITTED (userspace) — CUDA WORKS on sm_121, full training stack imports, `[2026-09-01 23:5x]` Run-3c throughput probe — IN FLIGHT, and the ana-ml2 baseline is now on record, `[2026-09-01]` pfi-gx10 (ASUS Ascent GX10) onboarded headless — and it is the intended new home for run 3c, `[2026-09-02]` GX10 BASELINE + TOOLING VERIFIED — ~79.35 s/it on the run-3c shape, Byproduct: IF run 3c ever moves here, the GX10 is the right home — and infra-ops' reversal was WRONG, ⚠ DOCKER'S BUILDKIT DB WAS ALREADY CORRUPT — the restart surfaced it, did not cause it, Exact run-3c geometry (from run-03c.json + a real adapter_config.json) (+23 more) -### Community 121 - "_touch" -Cohesion: 0.06 -Nodes (42): booth_age_seconds(), is_expired(), is_kept(), _newest_mtime(), Newest mtime among a folder and everything under it., Pure age question. Deliberately does NOT consider the keep sentinel. Expiry…, True if this booth carries the keep sentinel and must never be swept., Wipe every direct-child booth older than the TTL. Returns names wiped. Only… (+34 more) +### Community 121 - "`[2026-09-01]` A named failure class: a correct check aimed at the wrong object" +Cohesion: 0.50 +Nodes (4): `[2026-09-01]` A named failure class: a correct check aimed at the wrong object, Disposition, Related lessons banked the same day, The instances -### Community 122 - "togglable" -Cohesion: 0.40 -Nodes (5): togglable, default, description, title, type +### Community 122 - "score_beats.py" +Cohesion: 0.39 +Nodes (7): load(), main(), per_seed(), Score beat arms against a decision rule fixed BEFORE any arm's output was read.…, Return (text, words, ran_on) for the chosen measurement surface. The harness…, summarise(), view() ### Community 123 - "settings.local.json" Cohesion: 0.40 @@ -1202,12 +1214,12 @@ Cohesion: 0.08 Nodes (20): `[2026-09-15]` FV cross-site routing fixed — one NAT rule scoped to Anaheim only, Fix, Root cause, ⭐ The diagnostic signature, so the next person skips the evening, Wrong turns worth not repeating, `[2026-09-15]` Break-glass mesh path on fv-ml1 (inverted from a restore-watchdog), Design choices that matter, ⭐ INVERTED the same night, on the operator's suggestion (+12 more) ### Community 138 - "ops-log" -Cohesion: 0.14 -Nodes (33): agent_id(), all_audit_hosts(), append(), build_parser(), claim_key(), claim_state(), cmd_audit(), cmd_baseline() (+25 more) +Cohesion: 0.13 +Nodes (35): agent_id(), all_audit_hosts(), append(), build_parser(), claim_key(), claim_state(), cmd_audit(), cmd_baseline() (+27 more) -### Community 139 - "lv-mccarthy — corpus → gate → pairs" -Cohesion: 0.33 -Nodes (5): Corpus properties worth knowing before you change anything, lv-mccarthy — corpus → gate → pairs, The chain, The defect the gate could not see, and now can, The deviations, and what forced each +### Community 139 - "lv-mccarthy — corpus → leak gate → pairs → train → v2 gate" +Cohesion: 0.17 +Nodes (11): A log tag that reads backwards on this corpus, Corpus properties worth knowing before you change anything, D5 — train the adapter, D6 — the gate, lv-mccarthy — corpus → leak gate → pairs → train → v2 gate, Provenance fields that look wrong and are not, The chain, The defect the gate could not see, and now can (+3 more) ### Community 140 - "7. Critical Warnings by Model" Cohesion: 0.40 @@ -1234,8 +1246,8 @@ Cohesion: 0.20 Nodes (9): Deploy, First run takes a while, Gotchas, Optional: summarisation via the LiteLLM gateway, Rebuilding, scriberr — self-hosted transcription + diarization (fv-ml1, GPU1), Storage — deliberately on /tank, The image is built locally, and that is not incidental (+1 more) ### Community 146 - "elway" -Cohesion: 0.08 -Nodes (39): Namespace, _as_str(), build_adhoc_playbook(), ensure_sudo(), evaluate_changed(), evaluate_skip(), load_playbook(), _load_step() (+31 more) +Cohesion: 0.11 +Nodes (32): Namespace, _as_str(), build_adhoc_playbook(), ensure_sudo(), evaluate_changed(), evaluate_skip(), load_playbook(), _load_step() (+24 more) ### Community 147 - "generate_incumbent_arm.py" Cohesion: 0.60 @@ -1314,8 +1326,8 @@ Cohesion: 0.25 Nodes (7): Fleet backup architecture & freshness runbook, Known failure mode: rest-server-ana crash-loop (the 2026-05-06 → 2026-06-20 outage), Known gaps / TODO, Layer 1 — PBS (whole-VM/CT images), Layer 2 — restic (granular file + DB), The 2-minute freshness check, TL;DR — coverage matrix ### Community 179 - "convert_hf_to_native.py" -Cohesion: 0.10 -Nodes (29): _get_model(), OpenAI-ish /v1/audio/speech adapter in front of the Zonos Python SDK. Why this, _speaker_embedding(), speech(), SpeechRequest, ArgumentParser, _is_mtp(), main() (+21 more) +Cohesion: 0.15 +Nodes (22): ArgumentParser, _is_mtp(), main(), _tensor_nbytes(), build_parser(), convert(), ConvertStats, iter_native_reference_names() (+14 more) ### Community 180 - "StripEmptyTools" Cohesion: 0.29 @@ -1329,9 +1341,9 @@ Nodes (6): Gotchas (each one cost a failed run), Mistral Small 4 → NVFP4 (visi Cohesion: 0.17 Nodes (11): asset-engine, Deploy, Footprint, How it's served, OmniVoice, Placement, Streaming — earlier first-audio (with a diffusion floor), Text sanitization (+3 more) -### Community 186 - "[2026-08-15] Uncensored gen seat: Qwen3.8-27B-Uncensored deployed; the definitive MTP-graft fix" +### Community 186 - "`[2026-09-04]` Forcing 10G on the ESH-Media DAC FAILED progressively — and I called the plateau too early" Cohesion: 0.08 -Nodes (24): [2026-08-15] Uncensored gen seat: Qwen3.8-27B-Uncensored deployed; the definitive MTP-graft fix, #384 reindex + #381 restart + verify, AS-BUILT (2026-07-18, same-day execution), AS-BUILT (2026-07-18, same-day execution), donut voice (65-frost → Zonos gateway), Lessons (also in Tried-and-abandoned), mimir-inbox deployed (#377), muninn-gate → muninn-dispatch 0.1.5 (+16 more) +Nodes (26): `[2026-09-04]` Forcing 10G on the ESH-Media DAC FAILED progressively — and I called the plateau too early, #384 reindex + #381 restart + verify, AS-BUILT (2026-07-18, same-day execution), AS-BUILT (2026-07-18, same-day execution), donut voice (65-frost → Zonos gateway), ⚠ IT DEGRADED, AND MY "PLATEAU" READ WAS WRONG, Lessons (also in Tried-and-abandoned), Method notes worth keeping (+18 more) ### Community 187 - "mOrpheus — uncensored Orpheus TTS (irv-ml1)" Cohesion: 0.33 @@ -1449,10 +1461,6 @@ Nodes (29): base, compliance_rate, len, refused, text, gens, A train leaves at 1 Cohesion: 0.07 Nodes (29): base, compliance_rate, len, refused, text, gens, A train leaves at 14:20 and arrives at 17:05. How long is the journey? Show your reasoning briefly., Explain, in three sentences, why a heat pump can deliver more heat energy than the electrical energy it consumes. (+21 more) -### Community 227 - "_board" -Cohesion: 0.09 -Nodes (45): link_entry_id(), order_for_display(), parse_link_entries(), Path, Standing link board: parse and prune the multi-writer link log. STDLIB ONLY, ON…, Atomic replace of the pins file. Caller must hold the board lock., Pinned entry ids for a board. Missing file → empty set. Lock-free: a set read…, Flip one row's pinned state. Returns the NEW state (True = now pinned). Held… (+37 more) - ### Community 228 - "~26k" Cohesion: 0.20 Nodes (9): ~26k, approx_prompt_tokens, prefill_tok_s, ttft_median_s, ~6.5k, approx_prompt_tokens, prefill_tok_s, ttft_median_s (+1 more) @@ -1561,9 +1569,9 @@ Nodes (9): accs, model, mtp_accept_median, rates, tag, tok_s_max, tok_s_mean, to Cohesion: 0.20 Nodes (8): `[2026-08-27]` The dose-response says benefit and damage are ONE direction in weight space, ⚠ INSTRUMENT FLOOR — temperature 0 is NOT deterministic under concurrency, The conclusion, which is stronger than an exchange rate, The T4 mechanism is NOT register capture, `[2026-08-27]` The corpus mix was specified in a unit the optimiser never sees, The fix — a RENAME, not a reordering, The mechanic, Why it survived review -### Community 261 - "build_gallery" -Cohesion: 0.10 -Nodes (21): booth_image_names(), build_gallery(), classify(), doc_kind(), list_booths(), markdown' | 'text' | None — a booth file viewable as a readable page., Image files in a booth, in gallery (sorted-rel) order — for viewer prev/next., image | video | audio | other, by extension. (+13 more) +### Community 261 - "Refusal retention — the axis the gate did not have, and the axis I measured wrong" +Cohesion: 0.50 +Nodes (4): ⚠⚠ But it is the WRONG AXIS — brokkr's catch, and it is the better one, Refusal retention — the axis the gate did not have, and the axis I measured wrong, The measurement — controlled, single instrument, both arms, Why it exists ### Community 262 - "gen-seat mixed-precision quant — NVFP4 W4A4 MLP + FP8 W8A8 attention" Cohesion: 0.20 @@ -1653,6 +1661,10 @@ Nodes (6): Build + deploy, Config (single-writer, infra-ops), Key wiring (why it Cohesion: 0.18 Nodes (10): By category, Headline — overall refusal rate, `heresy`, Per-prompt detail, Refusal curve by intensity, Refusal probe — char-rp-reasoning seat, Refusal receipts (first 3 per model), Response length & latency (prose-volume proxy) (+2 more) +### Community 288 - "gpu.py" +Cohesion: 0.42 +Nodes (8): _bus_id_by_index(), _cmdline(), gpu_status(), _query_compute_apps(), _query_devices(), GPU status for arbo's device-aware scheduler (GET /gpu-status). arbo steers a l, Return `{devices: [...], tts_on_3090: bool}`. Degrades to an error field on nvid, _run() + ### Community 290 - "zed-fim-proxy" Cohesion: 0.50 Nodes (3): Deploy, Security model (stdlib proxy in `conf/proxy.py`), zed-fim-proxy @@ -1681,13 +1693,13 @@ Nodes (12): Containers, Not on this hypervisor, PFI-ANA Proxmox guest inventory Cohesion: 0.70 Nodes (4): error(), log(), pre-backup.sh script, warn() -### Community 299 - "uptimekuma — service uptime monitor (esh-docker-vm)" -Cohesion: 0.29 -Nodes (6): Data, Deploy, Deviations from house convention (deliberate, not oversights), The container name, uptimekuma — service uptime monitor (esh-docker-vm), Why it is in this repo +### Community 299 - "mime_from_field" +Cohesion: 0.50 +Nodes (4): anyOf, default, title, mime_from_field -### Community 300 - "The Booth" -Cohesion: 0.12 -Nodes (15): Asks — let the operator pick one of N, and read the pick back, Deleting a kept board, How a session posts, Install / update, Kept boards — the one exception to the 24h rule, Notes / non-goals, Ops, Routes (+7 more) +### Community 300 - "The Booth — moved to its own repository" +Cohesion: 0.40 +Nodes (4): Data, Deployed service, The Booth — moved to its own repository, What stays here ### Community 301 - "`[2026-08-20]` Cold-Fusion abliteration LANDED — layer 35, and the three false diagnoses corrected" Cohesion: 0.22 @@ -1745,6 +1757,10 @@ Nodes (6): A VM wedges on IO and the disk is innocent, Fix, Fleecing was enabled Cohesion: 0.22 Nodes (8): _chunk_to_pcm_bytes(), IndexTTS-2 — minimal FastAPI wrapper. Upstream (https://github.com/index-tts/in, 44-byte RIFF/WAVE/PCM header with placeholder data length so the payload can, Normalize whatever IndexTTS-2 yields (torch tensor, numpy array, int16 or fl, _resolve(), SpeechRequest, synthesize(), _wav_header() +### Community 321 - "worker/app.py" +Cohesion: 0.14 +Nodes (10): RuntimeError, get_gpu_status(), post_train(), FastAPI surface for the LoRA training worker (§4.1 API — arbo is the client). E, TrainRequest, Static configuration for the LoRA training worker. Everything load-bearing is a, LoRA training worker — a host service (runs as llmuser on irv-ml1) that runs sd-, Busy (+2 more) + ### Community 322 - "DFlash2 speculative decoding — measured on our own stack (2026-08-22)" Cohesion: 0.10 Nodes (21): Also identified, ⚠ Blast radius — the gating constraint, invisible from the host itself, ⚠️ CONFOUNDED — and the "before" state is itself unreliable, DFlash2 speculative decoding — measured on our own stack (2026-08-22), 🔶 HYPOTHESIS — why our acceptance trails the published numbers, Live state — PROMOTED to the compose stack 2026-08-22, ✅ MEASURED — how DFlash2 runs (answers "can one drafter serve both seats?"), ✅ MEASURED — it works on our stack, which the card does not claim (+13 more) @@ -1761,9 +1777,9 @@ Nodes (16): 10. IPv6 collapses two independent exposure controls into one, and i Cohesion: 0.70 Nodes (4): ask(), classify(), load_prompts(), main() -### Community 327 - "worker/app.py" -Cohesion: 0.12 -Nodes (18): RuntimeError, get_gpu_status(), post_train(), FastAPI surface for the LoRA training worker (§4.1 API — arbo is the client). E, TrainRequest, Static configuration for the LoRA training worker. Everything load-bearing is a, _bus_id_by_index(), _cmdline() (+10 more) +### Community 327 - "invocation.py" +Cohesion: 0.19 +Nodes (14): Path, test_published_relative_path(), test_published_relative_path_explicit_train_id_wins(), ValueError, InvalidTrainRequest, published_relative_path(), Fixed-invocation command builder — the enforcement point for INV-T7. arbo hands, The ComfyUI-relative loras path for a succeeded LoRA (Phase 2 publish step). (+6 more) ### Community 329 - "Beszel fleet monitoring" Cohesion: 0.33 @@ -1798,8 +1814,8 @@ Cohesion: 0.06 Nodes (35): additionalProperties, properties, required, title, type, additionalProperties, properties, required (+27 more) ### Community 338 - "archival-memory.md" -Cohesion: 0.10 -Nodes (17): [2026-08-23] Every secret-bearing `.env` on ana-docker tightened to 0600, [2026-08-24] ana-gw public admin surface closed to zero, ACME listener included, [2026-08-24] Scriberr transcription deployed on ana-ml2, GPU1, `[2026-09-01]` irv-ml1 GPU resident map — and dots-tts is holding 2.4x its recorded VRAM, `[2026-09-02]` althing 3.3.0 deployed — the `cc` channel, and a plugin-cache false green., `[2026-09-02]` althing deploy is SIX surfaces, and #6 is outside the althing repo: `~/.claude/settings.json` `crossSessionInb, `[2026-09-03]` nh3-dev wedged for ~40 min and it was the BACKUP, not the disk — a stalled cross-site vzdump holding every gue, Archival memory — eshpfi-management (+9 more) +Cohesion: 0.07 +Nodes (24): [2026-08-15] Uncensored gen seat: Qwen3.8-27B-Uncensored deployed; the definitive MTP-graft fix, [2026-08-23] Every secret-bearing `.env` on ana-docker tightened to 0600, [2026-08-24] Scriberr transcription deployed on ana-ml2, GPU1, `[2026-09-01]` althing 3.2.0 deployed on nh3-dev — pane routes, and `uv tool install .` lies, `[2026-09-01]` irv-ml1 GPU resident map — and dots-tts is holding 2.4x its recorded VRAM, `[2026-09-02]` althing 3.3.0 deployed — the `cc` channel, and a plugin-cache false green., `[2026-09-02]` althing deploy is SIX surfaces, and #6 is outside the althing repo: `~/.claude/settings.json` `crossSessionInb, `[2026-09-03]` nh3-dev wedged for ~40 min and it was the BACKUP, not the disk — a stalled cross-site vzdump holding every gue (+16 more) ### Community 339 - "`[2026-08-24]` char-rp seat: OOM root-cause, Gemma-4 MoE swap, and the abliterated trainee base" Cohesion: 0.22 @@ -1909,6 +1925,10 @@ Nodes (7): Findings that generalise, judge-bench — evaluate a candidate LLM-as Cohesion: 0.33 Nodes (6): Four silent defects the dry run found, ⚠ MERGED WEIGHTS ARE MANDATORY — and not for the reason we assumed, NVFP4A16 serving pipeline — built, validated, and the MoE landmine it found, Scheme: NVFP4A16, deviating from the playbook default, on measured grounds, ⚠⚠ THE LANDMINE: a `targets=["Linear"]` recipe misses EVERY MoE expert, Verified on the emitted artifact +### Community 373 - "CatalogField" +Cohesion: 0.33 +Nodes (6): additionalProperties, description, required, title, type, CatalogField + ### Community 374 - "`[2026-08-27]` Anaheim tripped a power breaker — and four guests including the NAS never came back" Cohesion: 0.06 Nodes (26): [2026-08-23] `/mnt/smithy` mounted on ana-ml2 — read-only and SOFT, deliberately not matching nh3-dev, The performance reality, measured on the same file through the same mount, Why not in fstab, Why soft, not hard, `[2026-08-27]` Anaheim tripped a power breaker — and four guests including the NAS never came back, Blast radius beyond us, ⚠ `/mnt/smithy` is manual by design — it will be missing after EVERY reboot, Power capacity is now the open item (+18 more) @@ -1973,9 +1993,9 @@ Nodes (7): `[2026-09-01]` The Ada inference server is a Dell R750xa bought used Cohesion: 0.14 Nodes (20): detect(), honorific_hits(), link_identities(), load(), main(), mid_sentence(), Path, ratio_rejects() (+12 more) -### Community 390 - "`[2026-09-01]` althing 3.2.0 deployed on nh3-dev — pane routes, and `uv tool install .` lies" -Cohesion: 0.06 -Nodes (36): `[2026-09-01]` althing 3.2.0 deployed on nh3-dev — pane routes, and `uv tool install .` lies, `[2026-09-02]` THE LESSON forseti AND I BOTH WANT KEPT, ✅ 3.2.1 OUTCOME — all four answered; delivery verified on FOUR TUI families, 🔴 3.2.1 SAME NIGHT (22:40) — 3.2.0 WROTE INTO PANES PEOPLE WERE TYPING IN, ✅ 3.2.2 DEPLOYED 22:51 (`c4ce947`) — four of five open items closed, ✅ 3.2.3 DEPLOYED — `althing-listen` now REFUSES on a pane seat (exit 4), ⚠ 3.2.4: A SEAT COULD HAVE BEEN SILENTLY PULL-ONLY SINCE 3.1.2 — and one may be, ✅ 3.2.4 + THE FOURTH DEPLOY SURFACE CLOSED (plugin cache 0.0.1 -> 0.1.1) (+28 more) +### Community 390 - "`[2026-09-04]` esh-nas SMB account for the AudioGridder box — and the NAS is effectively open to the whole LAN" +Cohesion: 0.05 +Nodes (37): `[2026-09-02]` THE LESSON forseti AND I BOTH WANT KEPT, `[2026-09-04]` esh-nas SMB account for the AudioGridder box — and the NAS is effectively open to the whole LAN, ✅ 3.2.1 OUTCOME — all four answered; delivery verified on FOUR TUI families, 🔴 3.2.1 SAME NIGHT (22:40) — 3.2.0 WROTE INTO PANES PEOPLE WERE TYPING IN, ✅ 3.2.2 DEPLOYED 22:51 (`c4ce947`) — four of five open items closed, ✅ 3.2.3 DEPLOYED — `althing-listen` now REFUSES on a pane seat (exit 4), ⚠ 3.2.4: A SEAT COULD HAVE BEEN SILENTLY PULL-ONLY SINCE 3.1.2 — and one may be, ✅ 3.2.4 + THE FOURTH DEPLOY SURFACE CLOSED (plugin cache 0.0.1 -> 0.1.1) (+29 more) ### Community 392 - "`[2026-08-28]` althing v3.0.0 flag day (U9b) — the post office replaced the P2P bus, one-way" Cohesion: 0.08 @@ -2025,9 +2045,9 @@ Nodes (8): `[2026-08-19]` Homepage cleaned up, then themed with Australis Skyfal Cohesion: 0.33 Nodes (5): `[2026-09-15]` Fleet identity/group/path conventions pinned + docker trees normalized, Accounts, deploy-stack.sh, fixed three times before the rule was written, Deploy trees → `root:docker 2775` setgid, all 5 hosts, Pinned allocation map -### Community 413 - "services.schema.json" -Cohesion: 0.25 -Nodes (7): additionalProperties, description, $id, required, $schema, title, type +### Community 413 - "license" +Cohesion: 0.67 +Nodes (3): title, type, license ### Community 414 - "`[2026-08-28]` A stale `ALTHING_HANDLE` silently reads ANOTHER agent's inbox and reports it empty" Cohesion: 0.40 @@ -2050,8 +2070,8 @@ Cohesion: 0.20 Nodes (9): Current egress: direct from NH3, ⚠ /healthz cannot tell you whether search works, Historical placement: NH3 rather than the colo, searxng, Secret, The 2026-09-18 engine outage — cause was the build, not egress, The ESH egress experiment (2026-09-17 → reverted 2026-09-18), The MCP tool (+1 more) ### Community 420 - "voice_distance.py" -Cohesion: 0.52 -Nodes (6): Counter, bigrams(), delta(), main(), profile(), Did the adapter move the voice TOWARD the held-out author? A seat-free relative… +Cohesion: 0.22 +Nodes (15): Counter, Pattern, analyse(), bigrams(), delta(), density(), identity(), load_arms() (+7 more) ### Community 421 - "Provisioning a Mac for the fleet" Cohesion: 0.33 @@ -2061,13 +2081,13 @@ Nodes (5): Harness specifics, Operator steps (stage 1), Provisioning a Mac for t Cohesion: 0.22 Nodes (8): `[2026-09-03]` Run 3c STAGED on pfi-gx10 — verified end to end, deliberately NOT launched, ⚠ I TRIPPED THE pkill SELF-MATCH AGAIN, ~20 MINUTES AFTER READING THE MEMORY ABOUT IT, NOT verified — the honest gap, The config is provably the same run, The launcher's other guards, each bought with a past failure, The verification that actually mattered — and it was NOT free reasoning, What is on the box, Why the slow box is still the right box (unchanged, restated because it is the whole case) -### Community 423 - "CatalogResponse" -Cohesion: 0.33 -Nodes (6): additionalProperties, description, required, title, type, CatalogResponse +### Community 423 - "properties" +Cohesion: 0.10 +Nodes (20): anyOf, default, title, properties, default, title, default, title (+12 more) -### Community 424 - "status" -Cohesion: 0.40 -Nodes (5): status, default, enum, title, type +### Community 424 - "stream_chunks" +Cohesion: 0.19 +Nodes (17): ClockFn, GenerateFn, ChunkConfig, _ema(), _est_gen_time(), plan_chunk(), protect_first_audio(), Split into sentence units, preserving punctuation. Whitespace-collapsed. (+9 more) ### Community 426 - "MeroMero NVFP4A16 quants (Gemma-4) — instruments and the failure chain" Cohesion: 0.29 @@ -2085,9 +2105,9 @@ Nodes (4): remote(), say(), provision-mac-dsh.sh script, step() Cohesion: 0.48 Nodes (6): expand_light(), font_faces(), main(), Concatenate the vendored token files, refusing to build if one moved., sha256(), vendored_css() -### Community 431 - "accepted_types" -Cohesion: 0.50 -Nodes (4): anyOf, default, title, accepted_types +### Community 431 - "status" +Cohesion: 0.40 +Nodes (5): status, default, enum, title, type ### Community 432 - "OPNsense API — vendored reference + field-shape notes" Cohesion: 0.18 @@ -2101,25 +2121,17 @@ Nodes (5): `[2026-08-19]` esh-pve hard-froze for 4.5h — and took the whole hou Cohesion: 0.33 Nodes (6): `[2026-08-19]` Four unmanaged stacks found on live hosts — and two of them were quietly broken, heretic2-charrp-reasoning — tracked, with its shim, SeaFile — not broken, never restarted, SearXNG — the healthcheck was eating itself, Superseded in-flight snapshots (archived), The pattern worth remembering -### Community 436 - "`[2026-09-04]` ERP run 3c trained on pfi-gx10 and GATED — the 20x LR cut erased the diversity gain and did NOT remove the safety regression" -Cohesion: 0.40 -Nodes (4): `[2026-09-04]` ERP run 3c trained on pfi-gx10 and GATED — the 20x LR cut erased the diversity gain and did NOT remove the safety regression, The lora_B signal worth carrying forward, The verdict (brokkr-smithy-dev, run03-gate/RESULT-run03c-gate.md), What the port proved about the box - -### Community 440 - "category" +### Community 440 - "max" Cohesion: 0.50 -Nodes (4): enum, title, type, category +Nodes (4): anyOf, default, title, max ### Community 441 - "pfi-gx10 — ERP-seat SFT run 3c, staged and ready to launch" Cohesion: 0.05 Nodes (33): Editing UniFi port overrides, ⚠ Order is the whole safety argument, pfi-gx10 — desk to rack, network, The shape, and why it is this shape, Wi-Fi, Launch, Layout on the box, Not verified (+25 more) -### Community 442 - "2026-09-05-floor-claim-n2-retraction.md" -Cohesion: 0.40 -Nodes (3): `[2026-09-05]` A peer's "2.7x stack effect" was a coin flip — the operator caught it and the arithmetic is worth keeping, `[2026-09-04/05]` vLLM RUNS on sm_121 — and the blocker was `ninja` not being on PATH, not the silicon, Run 4 — the corpus arm - -### Community 443 - "step" -Cohesion: 0.50 -Nodes (4): step, anyOf, default, title +### Community 442 - "name" +Cohesion: 0.67 +Nodes (3): title, type, name ### Community 445 - "`[2026-09-15]` ESPHome modernised for ha-dev; `kb` search tool for the personal Worldtree KB" Cohesion: 0.50 @@ -2134,20 +2146,12 @@ Cohesion: 0.07 Nodes (26): `[2026-09-10]` MeroMero — both quants landed; the v2 dense took five attempts, A4B — SUCCEEDED, MTP — checked on the SOURCES, not inferred, Published quants: none of them are our scheme, Still owed, The A4B reached a live seat while broken — and looked healthy doing it, The attempt-4 trap — the toolchain moved, the config was fine, The family, because I got it wrong first (+18 more) ### Community 450 - "Current state / in-flight" -Cohesion: 0.14 -Nodes (13): ✅ BUILT — the ops log (`scripts/ops-log`), 2026-09-19, Current state / in-flight, ⚠ FIRST — lv-mccarthy's run outcome is UNVERIFIED by this session, OPEN LOOP — `ravenpen.com`, and a reply hamr-dev is waiting on, Other standing items, unchanged, Persistent memory — eshpfi-management, PLANNED (operator) — move `dragonfireacoustics.com` to Namecheap, DNS to Cloudflare, Recent decisions (+5 more) +Cohesion: 0.17 +Nodes (11): Current state / in-flight, Draupnir engine — COMPLETE on irv-ml1, acceptance passing, ✅ lv-mccarthy SHIPPED — ckpt300 is the fourth voice on `voices-seat`, NEXT UP — the Booth (`services/booth`), Open, low-urgency, Persistent memory — eshpfi-management, Recent decisions, Repo purpose (+3 more) -### Community 451 - "`[2026-09-04]` `gen` moved to ana-ml2 GPU0 — and I sized it against the wrong number, twice in one hour" +### Community 454 - "max_length" Cohesion: 0.50 -Nodes (3): `[2026-09-04]` `gen` moved to ana-ml2 GPU0 — and I sized it against the wrong number, twice in one hour, Final state and what it cost, ⚠ THE SIZING ERROR, WHICH IS THE POINT OF THIS ENTRY - -### Community 453 - "`[2026-09-04]` Forcing 10G on the ESH-Media DAC FAILED progressively — and I called the plateau too early" -Cohesion: 0.50 -Nodes (3): `[2026-09-04]` Forcing 10G on the ESH-Media DAC FAILED progressively — and I called the plateau too early, ⚠ IT DEGRADED, AND MY "PLATEAU" READ WAS WRONG, Method notes worth keeping - -### Community 454 - "content_type" -Cohesion: 0.50 -Nodes (4): default, title, type, content_type +Nodes (4): anyOf, default, title, max_length ### Community 456 - "remote-ssh" Cohesion: 0.50 @@ -2157,13 +2161,13 @@ Nodes (3): SSH_MCP_STRICT_ALLOWLIST, node, remote-ssh Cohesion: 0.29 Nodes (6): [2026-09-18] althing search returned ZERO for every hyphenated query — silently, for everyone, Rollout mechanics worth keeping, Still open — two postbox fixes proposed to forseti, Symptom 1 — "never delivered": NOT A DEFECT, Symptom 2 — A REAL DEFECT, and far wider than one message, The fix, and the boundary that held -### Community 462 - "description" -Cohesion: 0.50 -Nodes (4): anyOf, default, title, description +### Community 462 - "Path" +Cohesion: 0.17 +Nodes (16): Path, load_pairs(), main(), Do the GENERATED BEATS name characters the rename removed? `leak_gate.py` reads…, load_text(), main(), Audit a stoplist for entries that are actually CHARACTERS. ⭐ WHY THIS IS A…, load_copies() (+8 more) -### Community 463 - "2026-09-17-esh-fiber-outages.md" -Cohesion: 0.25 -Nodes (5): 2026-09-06 — Headscale cutover COMPLETE: all three site-pairs on the mesh, Exit nodes (2026-09-06, operator-requested), ESH static WAN follow-ups landed; YTVC chase; gitea webhook repointed (2026-09-08), `[2026-09-17]` ESH: Cityside Fiber failed twice in six hours; site ran on Verizon failover, RESOLVED 2026-09-17 ~12:30 PT — the static is back, confirmed on four axes +### Community 463 - "2026-09-08-esh-static-wan-followups-and-ytvc.md" +Cohesion: 0.40 +Nodes (3): ESH static WAN follow-ups landed; YTVC chase; gitea webhook repointed (2026-09-08), `[2026-09-17]` ESH: Cityside Fiber failed twice in six hours; site ran on Verizon failover, RESOLVED 2026-09-17 ~12:30 PT — the static is back, confirmed on four axes ### Community 464 - "4. Pipeline shape" Cohesion: 0.40 @@ -2181,17 +2185,17 @@ Nodes (6): `[2026-09-10]` althing 3.6.2 rolled — post office + both heralds, a Cohesion: 0.25 Nodes (7): Backup and rollback, Broadened to the whole FV LAN, same session, Evidence and verification, ⭐ EXTENDED 2026-09-15 — NH3, ESH and Irvine now have the same rule, FV to Anaheim: scoped Tailscale NAT repair, Persistent configuration, What this looks like when it breaks, so the next person skips the evening -### Community 469 - "max_length" +### Community 469 - "[2026-08-24] ana-gw public admin surface closed to zero, ACME listener included" Cohesion: 0.50 -Nodes (4): anyOf, default, title, max_length +Nodes (4): [2026-08-24] ana-gw public admin surface closed to zero, ACME listener included, Final state, Port 80 was the FortiOS ACME listener, and I got it wrong first, Retracted in the same pass: the "four all-port VIPs" alarm -### Community 474 - "invocation.py" -Cohesion: 0.19 -Nodes (14): Path, test_published_relative_path(), test_published_relative_path_explicit_train_id_wins(), ValueError, InvalidTrainRequest, published_relative_path(), Fixed-invocation command builder — the enforcement point for INV-T7. arbo hands, The ComfyUI-relative loras path for a succeeded LoRA (Phase 2 publish step). (+6 more) +### Community 474 - "lv-mccarthy-4b-v1" +Cohesion: 0.29 +Nodes (6): ✅ AXIS B: the cleanest safety result in the lv- line, ⚠⚠ AXIS C: THE COST. READ THIS BEFORE PUTTING IT ON A LENGTH-SENSITIVE PATH, Corpus provenance, lv-mccarthy-4b-v1, The gate result — 5 arms, 60 held-out beats × 4 seeds = 240 generations per arm, Why checkpoint-300 and NOT ckpt900, the eval-loss minimum -### Community 483 - "method" -Cohesion: 0.50 -Nodes (4): default, title, type, method +### Community 483 - "`[2026-09-21]` lv-mccarthy: gate run, NOT SHIPPED — the voice works, the memorisation is the cleanest in the line, and the length discipline is gone" +Cohesion: 0.20 +Nodes (10): `[2026-09-21]` lv-mccarthy: gate run, NOT SHIPPED — the voice works, the memorisation is the cleanest in the line, and the length discipline is gone, ⚠ A bug in my own confound detector, and the order I found it in, ⭐⭐ AXIS B: the corrected control turned a 12× red flag into a clean pass, ❌ AXIS C: the damage is real, and reading it confirms the criterion, First: the run WAS complete, and had been for three days, Instruments built, each validated against a documented finding, ⚠ THE COMPLICATION, AND WHY IT DID NOT BECOME A SHIP, The result — 3 arms × 60 held-out beats × 4 seeds = 720 generations (+2 more) ### Community 485 - "Parakeet STT on fv-ml1 GPU 3 (2026-09-15)" Cohesion: 0.20 @@ -2209,9 +2213,9 @@ Nodes (5): char-rp-fast — swapping the MeroMero A4B onto the erp-seat seat (20 Cohesion: 0.33 Nodes (3): png(), Verify the swapped char-rp-fast seat before the gateway alias points at it. The…, Minimal solid-colour PNG, built here so the test needs no asset on disk. -### Community 503 - "Path" -Cohesion: 0.11 -Nodes (26): load(), main(), tensor key -> shard path, read from headers (no mmap; safe_open ENOMEMs on ZFS)., Recover (unit direction, {layer: weight}) from the trunk deltas., recover_direction(), shard_map(), Path, load_pairs() (+18 more) +### Community 503 - "content_type" +Cohesion: 0.50 +Nodes (4): default, title, type, content_type ### Community 506 - "Plex hardware transcoding on the Arc A580 (esh-pve-nas, LXC 105)" Cohesion: 0.25 @@ -2247,7 +2251,7 @@ Nodes (21): describe(), die(), main(), Return (frontmatter_dict, body). Absent o ### Community 528 - "$defs" Cohesion: 0.12 -Nodes (16): additionalProperties, description, required, title, type, additionalProperties, title, type (+8 more) +Nodes (16): additionalProperties, title, type, additionalProperties, description, required, title, type (+8 more) ### Community 529 - "Does MTP actually hurt Qwen3.8-Flash-Next on one Blackwell card?" Cohesion: 0.29 @@ -2289,9 +2293,9 @@ Nodes (17): chunk(), ctx_block(), drop_leading_heading(), main(), names_not_in_r Cohesion: 0.20 Nodes (9): Claude-Code-only, Credentials, FleetTools — what this system can do, and where the details live, LLM and inference, Rule zero: query live inventories, never a written list, Standing conventions that are not tools but will bite you, Talking to people and other agents, Working on code (+1 more) -### Community 541 - "BaseModel" -Cohesion: 0.31 -Nodes (7): ACEStepInput, ACEStepOutput, generate_audio(), initialize_pipeline(), Generate music; respond with the audio bytes inline. Pre-2026-05-11 this re, ACEStepPipeline, BaseModel +### Community 541 - "rename.py" +Cohesion: 0.60 +Nodes (4): build_pool(), main(), R49 Stage D2 (final) + D3 — entity resolution and deterministic rename…, title_gender() ### Community 543 - "Arbo — catalog-driven image generation engine" Cohesion: 0.33 @@ -2318,8 +2322,8 @@ Cohesion: 0.33 Nodes (5): Deploy, fablefusion-charrp-probe — THROWAWAY evaluation seat (fv-ml1 GPU1), ⚠️ GPU1 is zero-sum — this seat CANNOT run beside Dark-Scarlett, MTP depth: 3, not the card's 5, Why this candidate ### Community 556 - "The fleet ops log" -Cohesion: 0.18 -Nodes (11): 1. The log lives centrally on nh3-dev — not per-host, not on the post office, 2. The claim is advisory, and enforced in the tooling, 3. The writers are automatic, 4. There is a detector, not just a rule, Design decisions, and why, Exit codes, Open follow-ons, The fleet ops log (+3 more) +Cohesion: 0.29 +Nodes (7): Exit codes, How this instrument has failed — the pattern, not just the fixes, Open follow-ons, The fleet ops log, Using it, What it deliberately does not do, Why it exists ### Community 557 - "gen-seat — the fleet `gen` seat (fv-ml1 GPU0, :8015)" Cohesion: 0.50 @@ -2337,9 +2341,9 @@ Nodes (6): ⚠⚠ FOUR MORE LIVE BREAKAGES on OTHER hosts — the sweep's real f Cohesion: 0.29 Nodes (6): Current state — ⛔ SHELVED, deliberately, 2026-09-16, For a consumer, grok-token-broker, Invariants, The problem, ⛔ Why the loop is disarmed until measured -### Community 564 - "optional" -Cohesion: 0.50 -Nodes (4): default, title, type, optional +### Community 564 - "recover_direction" +Cohesion: 0.52 +Nodes (6): load(), main(), tensor key -> shard path, read from headers (no mmap; safe_open ENOMEMs on ZFS)., Recover (unit direction, {layer: weight}) from the trunk deltas., recover_direction(), shard_map() ### Community 582 - "build_corpus_hemingway.py" Cohesion: 0.20 @@ -2373,25 +2377,25 @@ Nodes (3): Did the quant actually do what the recipe says, on the tensors it cla Cohesion: 0.40 Nodes (5): [2026-08-24] ESH DNS fixed at the IPv6 layer, and the naming scheme went live, Root cause, The naming scheme became real, The three wrong turns, What landed -### Community 593 - "booth/app.py" -Cohesion: 0.08 -Nodes (28): FastAPI, fixture, Pattern, create_app(), _from_env(), human_dur(), _insert_after(), _insert_before() (+20 more) - -### Community 596 - "source_jsonpath" +### Community 595 - "memorization_check.py" Cohesion: 0.50 -Nodes (4): source_jsonpath, anyOf, default, title +Nodes (3): longest_match(), Did an arm learn the voice, or learn the text? delta_cb cannot tell them apart.…, row() -### Community 597 - "score_beats.py" -Cohesion: 0.39 -Nodes (7): load(), main(), per_seed(), Score beat arms against a decision rule fixed BEFORE any arm's output was read.…, Return (text, words, ran_on) for the chosen measurement surface. The harness…, summarise(), view() +### Community 596 - "`[2026-09-17]` lv-hemingway: SHIPPED on ckpt850 — the line's first clean voice pass, and one axis that needs reading" +Cohesion: 0.29 +Nodes (7): `[2026-09-17]` lv-hemingway: SHIPPED on ckpt850 — the line's first clean voice pass, and one axis that needs reading, ⚠ A SECOND corpus defect, measured and NOT acted on, Artefacts, ⚠⚠ AXIS B — THE COMFORTABLE EXPLANATION WAS WRONG, AND THE CONTROL IS THE ARTIFACT, Pre-flight: the beat leak IS present in Hemingway, and the fixture is clean, The gate result — 3 arms × 60 held-out beats × 4 seeds = 240 generations per arm, Why ckpt850 and NOT ckpt1750, the loss minimum + +### Community 597 - "description" +Cohesion: 0.50 +Nodes (4): anyOf, default, title, description ### Community 598 - "Direct inference seats + the Asset Engine control plane" Cohesion: 0.33 Nodes (5): Asset Engine — `http://10.250.50.70:8200`, Direct inference seats + the Asset Engine control plane, GPU hosts, ⚠ Query the live inventory, never a written list, Traps -### Community 599 - "properties" -Cohesion: 0.15 -Nodes (13): default, title, type, anyOf, default, title, properties, catalog_version (+5 more) +### Community 599 - "lv-mccarthy v2 gate — pre-registration" +Cohesion: 0.08 +Nodes (25): 1. What is being gated, 1a. Provenance defects found on first read, and their disposition, 2. Arms — three, and the third is here for a different reason than Hemingway's was, 3. Fixture and seeds — fixed now, 4. The floor rule — inherited, unchanged, 5. ⭐ THE VOICE AXIS — the open question, settled here before any number exists, 6. The three axes, and what passes, 7. The decision rule — frozen (+17 more) ### Community 600 - "LiteLLM gateway — the fleet's OpenAI-compatible front door" Cohesion: 0.33 @@ -2413,29 +2417,29 @@ Nodes (8): Authoritative vs. mirror, Backup pipeline, Common tasks, Coverage sta Cohesion: 0.50 Nodes (4): source_url, anyOf, default, title -### Community 606 - "$ref" -Cohesion: 0.15 -Nodes (13): items, title, type, $ref, fields, section_groups, services, items (+5 more) +### Community 606 - "section_groups" +Cohesion: 0.50 +Nodes (4): section_groups, items, title, type ### Community 608 - "Speech — TTS and ASR endpoints" Cohesion: 0.40 Nodes (4): ASR, Speech — TTS and ASR endpoints, Traps, TTS -### Community 609 - "type" -Cohesion: 0.50 -Nodes (4): type, enum, title, type +### Community 609 - "`[2026-09-17]` lv-mccarthy D1→D3 — built, gated, and every stage caught a defect in the stage before it" +Cohesion: 0.29 +Nodes (7): `[2026-09-17]` lv-mccarthy D1→D3 — built, gated, and every stage caught a defect in the stage before it, D1: the job was protecting a style that reads as damage, D2 caught a D1 defect: three small-caps manglings, D2's own gates: and `audit_stoplist` was scanning its own rationale, D3: three calls, and the holdout fix that matters most, The human gender pass is an auditable file, The shared splitter: choose by SIZE, not by count ### Community 612 - "2026-09-17-lv-bronte-gate.md" -Cohesion: 0.05 -Nodes (41): `[2026-09-16]` lv-hemingway corpus: half the work was EXCLUSION, and the gate found what a hand count would not, `[2026-09-16]` The lv-* voice line: Option C proved, lv-yarros shipped, lv-hemingway training, `[2026-09-16]` voices-seat: LoRA over merge, measured — and GPU 0 is now full, `[2026-09-17]` The beat-writing model restores the author's real names — a leak the corpus gate structurally cannot see, `[2026-09-17]` lv-bronte: corpus gated for real, adapter trained, SHIPPED with a FAILED voice axis on the record, ⚠ A DEFECT IN THE v2 RULE ITSELF, worth fixing before the next author, ⚠⚠ AMENDED 2026-09-17 — THE VOICE AXIS PASSES UNDER THE CORRECTED FLOOR RULE, Artefacts (+33 more) +Cohesion: 0.29 +Nodes (4): `[2026-09-16]` lv-hemingway corpus: half the work was EXCLUSION, and the gate found what a hand count would not, `[2026-09-16]` The lv-* voice line: Option C proved, lv-yarros shipped, lv-hemingway training, `[2026-09-16]` voices-seat: LoRA over merge, measured — and GPU 0 is now full, `[2026-09-17]` The beat-writing model restores the author's real names — a leak the corpus gate structurally cannot see -### Community 613 - "endpoint" -Cohesion: 0.67 -Nodes (3): title, type, endpoint +### Community 613 - "properties" +Cohesion: 0.17 +Nodes (12): default, title, type, anyOf, default, title, properties, catalog_version (+4 more) -### Community 614 - "estimated_latency" -Cohesion: 0.67 -Nodes (3): anyOf, default, estimated_latency +### Community 614 - "method" +Cohesion: 0.50 +Nodes (4): default, title, type, method ### Community 620 - "althing — inter-agent message bus" Cohesion: 0.50 @@ -2445,13 +2449,13 @@ Nodes (3): althing — inter-agent message bus, Commands, Traps Cohesion: 0.33 Nodes (6): Git / gitea, Known gotchas (save yourself time), Naming / identifiers, NFS / storage, systemd, Workflow -### Community 622 - "`[2026-09-01]` A named failure class: a correct check aimed at the wrong object" +### Community 622 - "show_memorisation_matches.py" Cohesion: 0.50 -Nodes (4): `[2026-09-01]` A named failure class: a correct check aimed at the wrong object, Disposition, Related lessons banked the same day, The instances +Nodes (4): main(), Print the actual verbatim runs an arm shares with the training corpus, for…, Return (cased, lowered) token lists that index in lockstep., tokens() -### Community 623 - "Refusal retention — the axis the gate did not have, and the axis I measured wrong" -Cohesion: 0.50 -Nodes (4): ⚠⚠ But it is the WRONG AXIS — brokkr's catch, and it is the better one, Refusal retention — the axis the gate did not have, and the axis I measured wrong, The measurement — controlled, single instrument, both arms, Why it exists +### Community 623 - "services.schema.json" +Cohesion: 0.25 +Nodes (7): additionalProperties, description, $id, required, $schema, title, type ### Community 625 - "min" Cohesion: 0.50 @@ -2461,45 +2465,121 @@ Nodes (4): anyOf, default, title, min Cohesion: 0.50 Nodes (3): elway — SSH playbook runner (CHANGING things), The split — pick by what you are doing, Traps -### Community 628 - "host" +### Community 628 - "`[2026-09-17]` Romantasy measured as a register — it is real, we already took its best voice, and the obvious next pick is its worst" Cohesion: 0.67 -Nodes (3): title, type, host +Nodes (3): `[2026-09-17]` Romantasy measured as a register — it is real, we already took its best voice, and the obvious next pick is its worst, If the lane gets a second seat it is Kenyon, Two findings that survive either floor reading -### Community 633 - "license" -Cohesion: 0.67 -Nodes (3): title, type, license +### Community 633 - "`[2026-09-17]` lv-bronte: corpus gated for real, adapter trained, SHIPPED with a FAILED voice axis on the record" +Cohesion: 0.33 +Nodes (6): `[2026-09-17]` lv-bronte: corpus gated for real, adapter trained, SHIPPED with a FAILED voice axis on the record, ⚠ A DEFECT IN THE v2 RULE ITSELF, worth fixing before the next author, ⚠⚠ AMENDED 2026-09-17 — THE VOICE AXIS PASSES UNDER THE CORRECTED FLOOR RULE, Artefacts, The gate result, in full, Which checkpoint, if it ships: **ckpt475** -### Community 634 - "version" -Cohesion: 0.67 -Nodes (3): version, title, type +### Community 638 - "`[2026-09-17]` McCarthy and Krakauer D1 — both corpora built, and one of them should not be trained yet" +Cohesion: 0.40 +Nodes (5): `[2026-09-17]` McCarthy and Krakauer D1 — both corpora built, and one of them should not be trained yet, lv-krakauer — builds clean, and the open question is not technical, lv-mccarthy — the job was protecting a style that reads as damage, ⚠⚠ THE OPEN CALL: an unmeasured fraction of Krakauer is not his prose, The shared blocker: a splitter that chooses by SIZE + +### Community 639 - "source_jsonpath" +Cohesion: 0.50 +Nodes (4): source_jsonpath, anyOf, default, title ### Community 641 - "Backup architecture (two layers)" Cohesion: 0.50 Nodes (4): Backup architecture (two layers), Cross-site restic mirror (rsync), Layer 1 — PBS (VM-image, block-level dedupe), Layer 2 — restic (file-level, encrypted, DB-consistent) -### Community 646 - "license_warning" +### Community 642 - "license_warning" Cohesion: 0.50 Nodes (4): anyOf, default, title, license_warning +### Community 643 - "`[2026-09-17]` The leak gate passed with five protagonist names still in every copy" +Cohesion: 0.40 +Nodes (5): `[2026-09-17]` The leak gate passed with five protagonist names still in every copy, Fixed at three levels, and all three must stay, The chain was recovered, not remembered — and is now written down, The mechanism, The shipped corpora, checked with the committed instrument + +### Community 644 - "`[2026-09-04]` ERP run 3c trained on pfi-gx10 and GATED — the 20x LR cut erased the diversity gain and did NOT remove the safety regression" +Cohesion: 0.50 +Nodes (4): `[2026-09-04]` ERP run 3c trained on pfi-gx10 and GATED — the 20x LR cut erased the diversity gain and did NOT remove the safety regression, The lora_B signal worth carrying forward, The verdict (brokkr-smithy-dev, run03-gate/RESULT-run03c-gate.md), What the port proved about the box + +### Community 645 - "Design decisions, and why" +Cohesion: 0.40 +Nodes (5): 1. The log lives centrally on nh3-dev — not per-host, not on the post office, 2. The claim is advisory, and enforced in the tooling, 3. The writers are automatic, 4. There is a detector, not just a rule, Design decisions, and why + +### Community 646 - "category" +Cohesion: 0.50 +Nodes (4): enum, title, type, category + +### Community 647 - "$ref" +Cohesion: 0.20 +Nodes (10): items, title, type, $ref, fields, services, items, items (+2 more) + +### Community 648 - "AMENDMENT-3 ROUND: five arms, and ckpt300 wins every axis — held at the gate by 0.02" +Cohesion: 0.40 +Nodes (5): AMENDMENT-3 ROUND: five arms, and ckpt300 wins every axis — held at the gate by 0.02, The complete ladder, The gate's verdict on ckpt300, stated exactly, ⚠ THREE CLAIMS I MADE EARLIER THAT THE FIVE-ARM DATA REFUTES, What is actually true about this adapter + +### Community 649 - "options" +Cohesion: 0.50 +Nodes (4): anyOf, default, title, options + +### Community 651 - "BaseModel" +Cohesion: 0.19 +Nodes (11): ACEStepInput, ACEStepOutput, generate_audio(), initialize_pipeline(), Generate music; respond with the audio bytes inline. Pre-2026-05-11 this re, ACEStepPipeline, BaseModel, FastAPI (+3 more) + +### Community 652 - "streamable" +Cohesion: 0.50 +Nodes (4): streamable, default, title, type + +### Community 653 - "required" +Cohesion: 0.50 +Nodes (4): required, default, title, type + +### Community 655 - "SHIPPED 2026-09-21 18:01 PT — `lv-mccarthy` = checkpoint-300 on `voices-seat` (fv-ml1 GPU0 :8027)" +Cohesion: 0.50 +Nodes (4): ⚠ A SEAT-WIDE DEFECT FOUND BY THE SMOKE TEST, LIVE SINCE 2026-09-16, SHIPPED 2026-09-21 18:01 PT — `lv-mccarthy` = checkpoint-300 on `voices-seat` (fv-ml1 GPU0 :8027), The cost that shipped with it, unglossed, ⚠ THE PROCESS FAILURE, RECORDED BECAUSE IT IS THE LESSON + +### Community 656 - "estimated_latency" +Cohesion: 0.67 +Nodes (3): anyOf, default, estimated_latency + +### Community 657 - "host" +Cohesion: 0.67 +Nodes (3): title, type, host + +### Community 660 - "`[2026-09-17]` Which voices earn a training seat next — measured against the catalogue, not chosen by taste" +Cohesion: 0.50 +Nodes (4): `[2026-09-17]` Which voices earn a training seat next — measured against the catalogue, not chosen by taste, Held, and why, ⭐ The size ranking INVERTS the voice ranking at the top, The three that clear both bars + +### Community 661 - "lifecycle" +Cohesion: 0.67 +Nodes (3): anyOf, default, lifecycle + +### Community 664 - "adapter/server.py" +Cohesion: 0.29 +Nodes (7): _get_model(), OpenAI-ish /v1/audio/speech adapter in front of the Zonos Python SDK. Why this, _speaker_embedding(), speech(), SpeechRequest, JSONResponse, Zonos + +### Community 668 - "record_to_ops_log" +Cohesion: 0.40 +Nodes (5): print_summary(), Best-effort. A failure here must never change elway's exit code or output — the…, record_to_ops_log(), StepResult, _tally() + +### Community 670 - "sanitize_tts_text" +Cohesion: 0.50 +Nodes (3): Language-safe text sanitizer for the OmniVoice TTS wrapper. Strips artifacts th, Return ``text`` cleaned of artifacts that shouldn't be spoken, language-safe., sanitize_tts_text() + ## Knowledge Gaps -- **3057 isolated node(s):** `Active migration — docker.io 20.10 → docker-ce 29.x`, `Architecture decisions (durable)`, `Audio generation on irv-ml1 (A6000)`, `autorestic decommission — redundant with the new two-layer pipeline`, `Backup coverage (2-layer, fully operational)` (+3052 more) +- **3105 isolated node(s):** `Why it exists — the lane split`, `Rebuilt from scratch, 2026-09-21`, `Scripting it`, `The Homepage widget is deliberately absent`, `Data` (+3100 more) These have ≤1 connection - possible missing edges or undocumented components. -- **156 thin communities (<3 nodes) omitted from report** — run `graphify query` to explore isolated nodes. +- **161 thin communities (<3 nodes) omitted from report** — run `graphify query` to explore isolated nodes. ## Suggested Questions _Questions this graph is uniquely positioned to answer:_ -- **Why does `create_app()` connect `booth/app.py` to `test_booth.py`, `Path`?** - _High betweenness centrality (0.004) - this node is a cross-community bridge._ -- **Why does `Tensor` connect `convert_hf_to_native.py` to `abliterate.py`?** +- **Why does `Tensor` connect `convert_hf_to_native.py` to `adapter/server.py`, `abliterate.py`?** + _High betweenness centrality (0.003) - this node is a cross-community bridge._ +- **Why does `divergences()` connect `abliterate.py` to `convert_hf_to_native.py`?** _High betweenness centrality (0.002) - this node is a cross-community bridge._ -- **What connects `Active migration — docker.io 20.10 → docker-ce 29.x`, `Architecture decisions (durable)`, `Audio generation on irv-ml1 (A6000)` to the rest of the system?** - _3057 weakly-connected nodes found - possible documentation gaps or missing edges._ +- **Why does `_speaker_embedding()` connect `adapter/server.py` to `convert_hf_to_native.py`?** + _High betweenness centrality (0.002) - this node is a cross-community bridge._ +- **What connects `Why it exists — the lane split`, `Rebuilt from scratch, 2026-09-21`, `Scripting it` to the rest of the system?** + _3105 weakly-connected nodes found - possible documentation gaps or missing edges._ - **Should `Status + Open Issues` be split into smaller, more focused modules?** _Cohesion score 0.05555555555555555 - nodes in this community are weakly interconnected._ - **Should `Fleet disaster-recovery runbook` be split into smaller, more focused modules?** _Cohesion score 0.06666666666666667 - nodes in this community are weakly interconnected._ - **Should `properties` be split into smaller, more focused modules?** - _Cohesion score 0.08 - nodes in this community are weakly interconnected._ -- **Should `1. Locked architectural decisions` be split into smaller, more focused modules?** - _Cohesion score 0.07407407407407407 - nodes in this community are weakly interconnected._ \ No newline at end of file + _Cohesion score 0.08 - nodes in this community are weakly interconnected._ \ No newline at end of file diff --git a/playbooks/althing-alert-bridge.yaml b/playbooks/althing-alert-bridge.yaml new file mode 100644 index 0000000..162cc68 --- /dev/null +++ b/playbooks/althing-alert-bridge.yaml @@ -0,0 +1,33 @@ +steps: + # The service was `beszel-althing` until 2026-09-21, when it gained a /kuma + # route and the Beszel-specific name became a lie. Stop and remove the old + # unit FIRST: both bind 10.100.10.50:8096, so they cannot run side by side. + - name: Stop and remove the superseded beszel-althing unit + sudo: true + shell: >- + systemctl disable --now beszel-althing.service 2>/dev/null || true; + rm -f /etc/systemd/system/beszel-althing.service; + rm -rf /opt/beszel-althing; + systemctl daemon-reload + - name: Install fleet alert bridge + sudo: true + upload: + src: services/althing-alert-bridge/bridge.py + dest: /opt/althing-alert-bridge/bridge.py + mode: '0644' + - name: Install fleet alert bridge unit + sudo: true + upload: + src: services/althing-alert-bridge/althing-alert-bridge.service + dest: /etc/systemd/system/althing-alert-bridge.service + mode: '0644' + - name: Start fleet alert bridge + sudo: true + shell: systemctl daemon-reload && systemctl enable althing-alert-bridge.service && systemctl restart althing-alert-bridge.service +verify: + - name: Verify bridge process and BOTH routes are advertised + shell: >- + systemctl is-active althing-alert-bridge.service && + curl --retry 5 --retry-connrefused --retry-delay 1 -fsS http://10.100.10.50:8096/healthz | grep -q '"/kuma"' && + curl -fsS http://10.100.10.50:8096/healthz | grep -q '"/beszel"' + changed_when: 'false' diff --git a/playbooks/beszel-althing.yaml b/playbooks/beszel-althing.yaml deleted file mode 100644 index 36c2760..0000000 --- a/playbooks/beszel-althing.yaml +++ /dev/null @@ -1,20 +0,0 @@ -steps: - - name: Install Beszel alert bridge - sudo: true - upload: - src: services/beszel-althing/bridge.py - dest: /opt/beszel-althing/bridge.py - mode: '0644' - - name: Install Beszel alert bridge unit - sudo: true - upload: - src: services/beszel-althing/beszel-althing.service - dest: /etc/systemd/system/beszel-althing.service - mode: '0644' - - name: Start Beszel alert bridge - sudo: true - shell: systemctl daemon-reload && systemctl enable beszel-althing.service && systemctl restart beszel-althing.service -verify: - - name: Verify bridge process - shell: systemctl is-active beszel-althing.service && curl --retry 5 --retry-connrefused --retry-delay 1 -fsS http://10.100.10.50:8096/healthz - changed_when: 'false' diff --git a/playbooks/deploy-althing-chamber.yaml b/playbooks/deploy-althing-chamber.yaml deleted file mode 100644 index 73026cc..0000000 --- a/playbooks/deploy-althing-chamber.yaml +++ /dev/null @@ -1,147 +0,0 @@ -# Deploy althing-chamber (https://gitea.phasefinal.com/vh/althing) to a -# Docker host following the PFI /opt/docker/ convention (ana-docker by -# default, but the playbook works against any host with Docker in place). -# -# Brings up FOUR compose services: -# althing-chamber — FastAPI/HTMX web UI, port 7881 host → 8000 container -# althing-forseti — moderator daemon, no port -# althing-agent-runner — Phase 2 worldtree-driver dispatcher, no port -# althing-valkey — Phase 3.1 valkey 8 alpine, pub/sub bridge (internal only) -# -# The three althing services use the same image (built from vh/althing); -# valkey is a stock upstream image. SQLite-backed state shares between the -# three althing services via the bind-mount under /opt/docker/conf/althing-chamber/data; -# streaming events ride pub/sub on the docker default network via valkey. -# -# Idempotent: rerunning is safe. Creates-gates skip work that's already -# done; `docker compose up -d` is itself idempotent (no restart unless -# compose content or env changed). -# -# Usage: -# scripts/elway ana-docker --playbook playbooks/deploy-althing-chamber.yaml -# scripts/elway ana-docker --playbook playbooks/deploy-althing-chamber.yaml --var ref=v0.1.0 -# -# Prereqs on the target host: -# - Docker + docker compose plugin -# - Target user (lkraven) has git SSH access to gitea.phasefinal.com -# — either SSH key authorized in gitea, or the repo is HTTPS-reachable -# if you swap `repo_url` below. -# - Target user is in the `docker` group. - -vars: - repo_url: git@gitea.phasefinal.com:vh/althing.git - ref: main - build_dir: /opt/docker/build/althing-chamber - image_tag: althing-chamber:local - compose_dir: /opt/docker/compose/althing-chamber - data_dir: /opt/docker/conf/althing-chamber/data - host_port: "7881" - -steps: - # ── host-side directory prep ───────────────────────────────────────── - - name: Ensure /opt/docker/build parent exists - shell: mkdir -p /opt/docker/build - sudo: true - creates: /opt/docker/build - - - name: Chown /opt/docker/build to lkraven (only if mkdir'd by root above) - shell: chown lkraven:lkraven /opt/docker/build - sudo: true - when: '[ "$(stat -c %U /opt/docker/build)" != lkraven ]' - - # ── fetch / sync source ───────────────────────────────────────────── - - name: Clone althing repo if absent - # Auto-accept the first-run host key so the playbook doesn't hang - # prompting for yes/no. - shell: GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=accept-new" git clone {{ repo_url }} {{ build_dir }} - creates: "{{ build_dir }}/.git" - - - name: Fetch from origin - shell: cd {{ build_dir }} && git fetch --quiet origin - - - name: Reset working tree to {{ ref }} - # Accept either a branch name (resolves via origin/) or a - # full/short SHA (resolves directly). CI passes the triggering - # commit SHA via --var ref=${{ github.sha }}; manual runs pass - # branch names like main / v0.1.0. - shell: | - cd {{ build_dir }} - if sha=$(git rev-parse --verify --quiet "origin/{{ ref }}^{commit}"); then :; - elif sha=$(git rev-parse --verify --quiet "{{ ref }}^{commit}"); then :; - else echo "elway: ref not found: {{ ref }}" >&2; exit 1; fi - git reset --hard "$sha" - # Report ok (no-change) when the tree was already at the requested - # ref — saves a noisy CHANGED status line on no-op reruns. - changed_when: '[ "$(cd {{ build_dir }} && git rev-parse HEAD)" != "$(cd {{ build_dir }} && (git rev-parse --verify --quiet "origin/{{ ref }}^{commit}" || git rev-parse --verify --quiet "{{ ref }}^{commit}"))" ]' - - # ── image build ───────────────────────────────────────────────────── - - name: Build image {{ image_tag }} - shell: cd {{ build_dir }} && docker build -t {{ image_tag }} . - # Docker build reuses layer cache and is fast on reruns, but it - # always runs — we can't cheaply know up-front whether anything - # downstream has changed. Leave it in the always-run lane; Docker - # itself handles the no-op efficiently. - - # ── compose + data dirs ───────────────────────────────────────────── - - name: Ensure compose dir exists - shell: mkdir -p {{ compose_dir }} - creates: "{{ compose_dir }}" - - - name: Ensure data dir exists - # Single bind-mount shared between chamber + forseti. Created as - # lkraven (uid 1000 on these hosts), matching the container's `app` - # user — no chown dance needed. - shell: mkdir -p {{ data_dir }} - creates: "{{ data_dir }}" - - # ── deploy compose files ──────────────────────────────────────────── - - name: Upload compose.yaml - upload: - src: stacks/althing-chamber/compose.yaml - dest: "{{ compose_dir }}/compose.yaml" - mode: "0644" - - - name: Seed .env from template (only if absent) - upload: - src: stacks/althing-chamber/.env.example - dest: "{{ compose_dir }}/.env" - mode: "0644" - when: "[ ! -f {{ compose_dir }}/.env ]" - - # ── bring up + wait for ready ─────────────────────────────────────── - - name: docker compose up -d - shell: cd {{ compose_dir }} && docker compose up -d - - - name: Wait for chamber /health to respond - # Chamber's healthcheck is internal (inside the container's network); - # this host-side poll confirms the published port is reachable too. - # Short retry loop — docker compose up returns before the FastAPI - # app finishes booting. - shell: | - for i in $(seq 1 30); do - curl -sf -o /dev/null http://localhost:{{ host_port }}/health && exit 0 - sleep 2 - done - exit 1 - changed_when: "false" - -verify: - - name: chamber /health returns 200 - shell: curl -sf -o /dev/null http://localhost:{{ host_port }}/health - changed_when: "false" - - - name: chamber container running - shell: docker ps --filter name=^/althing-chamber$ --format '{{.Status}}' | grep -q '^Up' - changed_when: "false" - - - name: forseti container running - shell: docker ps --filter name=^/althing-forseti$ --format '{{.Status}}' | grep -q '^Up' - changed_when: "false" - - - name: agent-runner container running - shell: docker ps --filter name=^/althing-agent-runner$ --format '{{.Status}}' | grep -q '^Up' - changed_when: "false" - - - name: valkey container running + healthy - shell: docker ps --filter name=^/althing-valkey$ --format '{{.Status}}' | grep -q 'healthy' - changed_when: "false" diff --git a/scripts/kuma b/scripts/kuma index d99c7a0..7fcb95f 100755 --- a/scripts/kuma +++ b/scripts/kuma @@ -73,6 +73,18 @@ MONITOR_DEFAULTS = { } +# A channel registered with isDefault + applyExisting attaches to monitors that +# ALREADY exist, not merely to ones created afterwards -- without it, seeding +# monitors before channels silently leaves the existing board unwired. +NOTIFICATION_DEFAULTS = { + "type": "webhook", + "isDefault": True, + "applyExisting": True, + "webhookContentType": "json", + "active": True, +} + + def vault_get(key: str) -> dict: """{'username','password'} from the vault. Loud on failure -- a silent empty credential would show up as a confusing auth error much later.""" @@ -106,10 +118,16 @@ class Kuma: # updating it. Capture the push; never trust the ack for data. self._monitor_list: dict | None = None + self._notification_list: list | None = None + @self.sio.on("monitorList") def _on_monitor_list(data): self._monitor_list = data or {} + @self.sio.on("notificationList") + def _on_notification_list(data): + self._notification_list = data or [] + def __enter__(self): self.sio.connect(self.url, transports=["websocket"], wait_timeout=20) self._connected = True @@ -157,6 +175,29 @@ class Kuma: def delete(self, monitor_id: int): return self.call("deleteMonitor", monitor_id, False) + def notifications(self, timeout: int = 10) -> list: + """Registered notification channels. + + ⚠ DIFFERENT PUSH RULE FROM monitorList, and the difference bit once. + There is no `getNotificationList` event to ask with -- the server pushes + `notificationList` ONCE, at login, and again after any change. So this + must NOT clear the captured value before waiting: doing that discards + the only copy it will ever be sent and then blocks for the full timeout + before reporting an empty list, which reads exactly like "no channels + are configured" and is a lie. + """ + deadline = time.time() + timeout + while self._notification_list is None and time.time() < deadline: + self.sio.sleep(0.2) + return self._notification_list or [] + + def add_notification(self, notification: dict): + """Register (or update) a channel. notificationID None => create.""" + return self.call("addNotification", notification, None) + + def test_notification(self, notification: dict): + return self.call("testNotification", notification) + def add(self, monitor: dict): return self.call("add", {**MONITOR_DEFAULTS, **monitor}) @@ -204,8 +245,20 @@ def cmd_seed(args): same service), and re-running a seed must never fork the board.""" spec = yaml.safe_load(open(args.file)) wanted = spec["monitors"] if isinstance(spec, dict) else spec + channels = spec.get("notifications", []) if isinstance(spec, dict) else [] k = connected_and_logged_in() try: + # Channels FIRST: a monitor created while a default channel exists + # inherits it. Seeding monitors first and alerting second is how you get + # a board that detects and tells nobody -- the exact gap this fleet's + # service layer was built to close. + have = {n.get("name") for n in k.notifications()} + for ch in channels: + if ch["name"] in have: + print(f" = {ch['name']} (channel)") + continue + k.add_notification({**NOTIFICATION_DEFAULTS, **ch}) + print(f" + {ch['name']} (channel)") existing = {m.get("name"): m for m in k.monitors().values()} added = updated = 0 for m in wanted: @@ -223,6 +276,21 @@ def cmd_seed(args): k.__exit__() +def cmd_channels(args): + k = connected_and_logged_in() + try: + ns = k.notifications() + if not ns: + print("no notification channels — THE BOARD ALERTS NOBODY") + return + for n in ns: + cfg = json.loads(n.get("config") or "{}") + print(" %-26s %-9s default=%-5s %s" % ( + n.get("name"), cfg.get("type"), n.get("isDefault"), cfg.get("webhookURL", ""))) + finally: + k.__exit__() + + def cmd_dedupe(args): """Keep the LOWEST id per name, delete the rest. Needed because the first version of this client read the getMonitorList ack instead of the pushed @@ -274,6 +342,7 @@ def main(): a.add_argument("--port", type=int) a.set_defaults(fn=cmd_add) sub.add_parser("dedupe").set_defaults(fn=cmd_dedupe) + sub.add_parser("channels").set_defaults(fn=cmd_channels) args = p.parse_args() try: args.fn(args) diff --git a/services/althing-alert-bridge/README.md b/services/althing-alert-bridge/README.md new file mode 100644 index 0000000..fd7a74c --- /dev/null +++ b/services/althing-alert-bridge/README.md @@ -0,0 +1,71 @@ +# Fleet alert bridge — monitoring tools into the althing inbox + +`althing-alert-bridge.service` runs on nh3-dev as lkraven, listening at +`10.100.10.50:8096`. It turns an HTTP alert into a `postbox send` to the +**infra-ops** inbox, using the established automation identity. + +| route | sender | payload | +|---|---|---| +| `POST /beszel` | Beszel (Shoutrrr generic JSON) | `{title, message}` | +| `POST /kuma` | Uptime Kuma (webhook, JSON) | `{heartbeat, monitor, msg}` | + +```sh +scripts/elway infra-ops@10.100.10.50 --playbook playbooks/althing-alert-bridge.yaml +systemctl status althing-alert-bridge +curl -fsS http://10.100.10.50:8096/healthz # lists the routes it serves +``` + +## Why one bridge with a route registry + +It was `beszel-althing` until 2026-09-21, with the `[Beszel]` subject prefix and +the Beszel hub footer hardcoded. When Uptime Kuma was rebuilt as the fleet's +service layer it needed the same path — and routing it through unchanged would +have delivered Kuma outages labelled `[Beszel]`, pointing the reader at the +wrong dashboard. **An alert that lies about its own source is worse than no +alert.** + +Generalising cost a dict. A sibling service would have cost a second unit, a +second port, and a second thing to notice had died. The name went with it: a +service called `beszel-althing` that also carries Kuma alerts is exactly the +invisible coupling that makes a future session look in the wrong place. + +⚠ **`/beszel` is frozen.** Its prefix, footer and default title must stay +byte-identical — that path was verified end-to-end in production (2026-09-10, +thread `01M25Z0WFDJM92GPTJQF769HJ7`) and a refactor is not allowed to quietly +change what it emits. `deliver()` still defaults to the Beszel route so the +original three tests exercise it unchanged, and a test asserts the Kuma footer +never leaks into a Beszel body or vice versa. + +## Payload shapes are handled here, not in the sending tool + +Uptime Kuma can render a custom webhook body, which would have let the bridge +stay dumb. It is done here instead, because Kuma's notification config lives in +its own database — and that database was destroyed and rebuilt from scratch on +2026-09-21. Anything that lives only in a tool's DB is lost on the next rebuild. +Format knowledge belongs in git, next to a test. + +`parse_kuma` also handles the **monitorless** case: `testNotification` and +certificate-expiry alerts arrive with no monitor and no heartbeat, and +fabricating `unknown monitor is ?` from those makes a real alert read like a +bug — observed live, then fixed and pinned by a test. + +## Operational notes + +- Source-allowlisted by IP (`ALERT_ALLOWED_SOURCES`). `10.250.50.70` is + ana-docker, which now runs **both** the Beszel hub and Uptime Kuma, so one + entry covers both senders. +- Delivery reports success only after `postbox` returns a receipt. Failures + return HTTP 502 and land in the journal; **there is no retry queue**, so a + post-office outage can lose an alert. +- `/healthz` checks the bridge process, not the downstream inbox. End-to-end + verification means sending a real POST and reading the thread back — both + routes were verified that way on 2026-09-21 (`postbox thread` does not + consume the inbox). +- Legacy `BESZEL_*` env names still resolve, so a half-finished deploy starts + instead of crash-looping. New deployments use `ALERT_*`. +- To reroute, change `ALERT_RECIPIENT` in the unit and redeploy. Leave + `ALTHING_HANDLE` as infra-ops so the sender stays identifiable as automation. + +```sh +python3 -m unittest discover -s services/althing-alert-bridge -p 'test_*.py' # 11 tests +``` diff --git a/services/althing-alert-bridge/althing-alert-bridge.service b/services/althing-alert-bridge/althing-alert-bridge.service new file mode 100644 index 0000000..1e76df9 --- /dev/null +++ b/services/althing-alert-bridge/althing-alert-bridge.service @@ -0,0 +1,27 @@ +[Unit] +Description=Fleet alert bridge — Beszel + Uptime Kuma alerts into the althing infra-ops inbox +After=network-online.target +Wants=network-online.target + +[Service] +User=lkraven +Group=lkraven +Environment=PYTHONUNBUFFERED=1 +Environment=ALTHING_HANDLE=infra-ops +Environment=ALTHING_POST_OFFICE=http://10.100.50.40:8390 +Environment=POSTBOX=/home/lkraven/.local/bin/postbox +Environment=ALERT_RECIPIENT=infra-ops +Environment=ALERT_BIND_HOST=10.100.10.50 +# 10.250.50.70 is ana-docker, which now runs BOTH the Beszel hub and Uptime +# Kuma, so one entry covers both senders. 10.100.10.50/127.0.0.1 are local +# diagnostics from nh3-dev itself. +Environment=ALERT_ALLOWED_SOURCES=10.250.50.70,10.100.10.50,127.0.0.1 +ExecStart=/usr/bin/python3 /opt/althing-alert-bridge/bridge.py +Restart=on-failure +RestartSec=5 +NoNewPrivileges=true +ProtectSystem=strict +PrivateTmp=true + +[Install] +WantedBy=multi-user.target diff --git a/services/althing-alert-bridge/bridge.py b/services/althing-alert-bridge/bridge.py new file mode 100644 index 0000000..fe9c689 --- /dev/null +++ b/services/althing-alert-bridge/bridge.py @@ -0,0 +1,162 @@ +#!/usr/bin/env python3 +"""Fleet alert bridge — HTTP alerts from monitoring tools into the althing inbox. + +One listener, one route per SOURCE. Each source declares its own subject prefix, +its own footer link, and how to read its payload, because the tools do not agree +on a shape and the reader needs to know which tool is talking. + + POST /beszel <- Beszel, via Shoutrrr generic JSON: {title, message} + POST /kuma <- Uptime Kuma, via its webhook: {heartbeat, monitor, msg} + +WHY A REGISTRY RATHER THAN A SECOND SERVICE. The prefix and footer were +hardcoded when Beszel was the only caller, so a second tool routed through it +would have arrived labelled "[Beszel]" -- an alert that lies about its own +source is worse than no alert, because it sends you to the wrong dashboard. +Generalising costs a dict; a sibling service costs a second unit, a second port +and a second thing to notice has died. + +⚠ `/beszel` IS FROZEN. Its prefix, footer and default title must stay byte- +identical -- Beszel's delivery path has been verified end-to-end in production +(2026-09-10, thread 01M25Z0WFDJM92GPTJQF769HJ7) and this refactor is not +allowed to quietly change what that path emits. `deliver()` still defaults to +the Beszel route precisely so the original tests exercise it unchanged. +""" +import json +import os +import subprocess +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer + + +def _env(*names, default=None): + """First env var that is set. Accepts the legacy BESZEL_* names so a + half-finished deploy (new bridge, old unit) still starts instead of + crash-looping on a KeyError.""" + for n in names: + if os.environ.get(n): + return os.environ[n] + if default is None: + raise KeyError(' / '.join(names)) + return default + + +def parse_generic(payload): + """Shoutrrr generic JSON — what Beszel sends. {title, message}.""" + return payload.get('title'), payload.get('message') + + +def parse_kuma(payload): + """Uptime Kuma's webhook — {heartbeat, monitor, msg}. + + Kuma's own `msg` already reads like a sentence ("[Homepage] [🔴 Down] + connect ECONNREFUSED"), so it is the body. The title is built from the + monitor name and the heartbeat status so the SUBJECT alone says which + service and which direction -- that is what you see in an inbox list + without opening anything. + """ + monitor = payload.get('monitor') or {} + heartbeat = payload.get('heartbeat') or {} + body = payload.get('msg') or '' + name = monitor.get('name') + # Kuma status: 0 down, 1 up, 2 pending, 3 maintenance. + state = {0: 'DOWN', 1: 'UP', 2: 'PENDING', 3: 'MAINTENANCE'}.get(heartbeat.get('status')) + + if name and state: + title = f'{name} is {state}' + else: + # NOT every Kuma notification is a monitor transition: `testNotification` + # and certificate-expiry alerts arrive with no monitor and no heartbeat. + # Fabricating "unknown monitor is ?" from those makes a real alert read + # like a bug; the message itself is the better subject. + first = body.strip().splitlines()[0] if body.strip() else '' + title = (first[:120] or 'notification') + detail = heartbeat.get('msg') + if detail and detail not in body: + body = f'{body}\n\n{detail}'.strip() + url = monitor.get('url') + if url: + body = f'{body}\n\nTarget: {url}'.strip() + return title, body + + +SOURCES = { + # FROZEN — see the module docstring. + '/beszel': { + 'prefix': '[Beszel] ', + 'default_title': 'Beszel fleet alert', + 'footer': '\n\nHub: http://10.250.50.70:8090\n', + 'parse': parse_generic, + }, + '/kuma': { + 'prefix': '[Uptime Kuma] ', + 'default_title': 'Uptime Kuma alert', + 'footer': '\n\nBoard: http://10.250.50.70:3001\n', + 'parse': parse_kuma, + }, +} + + +def deliver(payload, route='/beszel'): + source = SOURCES[route] + title, message = source['parse'](payload) + if title is None: + title = source['default_title'] + if not isinstance(title, str) or not isinstance(message, str) or not message.strip(): + raise ValueError('Expected a nonempty message and string title') + result = subprocess.run( + [_env('POSTBOX'), '--json', 'send', + '--to', _env('ALERT_RECIPIENT', 'BESZEL_ALERT_RECIPIENT'), + '--subject', source['prefix'] + title], + input=message + source['footer'], + text=True, capture_output=True, timeout=25, + ) + if result.returncode: + raise RuntimeError('postbox delivery failed: ' + result.stderr.strip()) + receipt = json.loads(result.stdout) + print(json.dumps({'event': 'delivered', 'route': route, 'title': title, 'receipt': receipt}), flush=True) + return receipt + + +class Handler(BaseHTTPRequestHandler): + def respond(self, status, body): + data = json.dumps(body).encode() + self.send_response(status) + self.send_header('Content-Type', 'application/json') + self.send_header('Content-Length', str(len(data))) + self.end_headers() + self.wfile.write(data) + + def do_GET(self): + self.respond(200 if self.path == '/healthz' else 404, + {'service': 'althing-alert-bridge', + 'routes': sorted(SOURCES), + 'delivery': 'verified per POST'}) + + def do_POST(self): + if self.path not in SOURCES: + return self.respond(404, {'error': 'Unknown route', 'routes': sorted(SOURCES)}) + allowed = _env('ALERT_ALLOWED_SOURCES', 'BESZEL_ALLOWED_SOURCES').split(',') + if self.client_address[0] not in allowed: + return self.respond(403, {'error': 'Source not allowed'}) + try: + length = int(self.headers.get('Content-Length', '0')) + if not 0 < length <= 65536: + raise ValueError('Invalid body size') + self.connection.settimeout(10) + payload = json.loads(self.rfile.read(length)) + if not isinstance(payload, dict): + raise ValueError('Expected JSON object') + receipt = deliver(payload, self.path) + except (ValueError, TypeError) as exc: + return self.respond(400, {'error': str(exc)}) + except (OSError, RuntimeError, subprocess.TimeoutExpired) as exc: + print(json.dumps({'event': 'delivery_failed', 'route': self.path, 'error': str(exc)}), flush=True) + return self.respond(502, {'error': 'Althing delivery failed; inspect service journal'}) + self.respond(200, {'delivered': True, 'receipt': receipt}) + + +if __name__ == '__main__': + ThreadingHTTPServer( + (_env('ALERT_BIND_HOST', 'BESZEL_BIND_HOST'), + int(_env('ALERT_BIND_PORT', 'BESZEL_BIND_PORT', default='8096'))), + Handler, + ).serve_forever() diff --git a/services/althing-alert-bridge/test_bridge.py b/services/althing-alert-bridge/test_bridge.py new file mode 100644 index 0000000..63a08aa --- /dev/null +++ b/services/althing-alert-bridge/test_bridge.py @@ -0,0 +1,109 @@ +import importlib.util +import os +from pathlib import Path +import subprocess +import unittest +from unittest.mock import patch + +spec=importlib.util.spec_from_file_location('bridge',Path(__file__).with_name('bridge.py')) +bridge=importlib.util.module_from_spec(spec);spec.loader.exec_module(bridge) + +class DeliveryTests(unittest.TestCase): + def setUp(self): + self.env=patch.dict(os.environ,POSTBOX='/bin/postbox',BESZEL_ALERT_RECIPIENT='infra-ops') + self.env.start();self.addCleanup(self.env.stop) + def test_posts_body_as_stdin_and_returns_receipt(self): + with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],0,'{"message_id": 42}','')) as run: + self.assertEqual(bridge.deliver({'title':'Disk 85%','message':'tank is full'})['message_id'],42) + args,kw=run.call_args + self.assertEqual(args[0],['/bin/postbox','--json','send','--to','infra-ops','--subject','[Beszel] Disk 85%']) + self.assertIn('tank is full',kw['input']) + def test_delivery_failure_is_not_success(self): + with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],2,'','post office unavailable')): + with self.assertRaises(RuntimeError):bridge.deliver({'message':'disk full'}) + def test_rejects_empty_message_without_sending(self): + with patch.object(bridge.subprocess,'run') as run: + with self.assertRaises(ValueError):bridge.deliver({'message':''}) + run.assert_not_called() + +class KumaRouteTests(unittest.TestCase): + """The /kuma route. The point of generalising was that an alert must not + lie about its own source -- a Kuma outage labelled [Beszel] sends you to + the wrong dashboard, which is worse than no alert.""" + + def setUp(self): + self.env=patch.dict(os.environ,POSTBOX='/bin/postbox',ALERT_RECIPIENT='infra-ops') + self.env.start();self.addCleanup(self.env.stop) + + DOWN={'monitor':{'name':'Homepage','url':'http://10.0.50.45:5100/'}, + 'heartbeat':{'status':0,'msg':'connect ECONNREFUSED'}, + 'msg':'[Homepage] [Down] connect ECONNREFUSED'} + + def test_subject_names_the_service_and_the_direction(self): + with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],0,'{"message_id":7}','')) as run: + bridge.deliver(self.DOWN,'/kuma') + argv=run.call_args[0][0] + self.assertIn('[Uptime Kuma] Homepage is DOWN',argv) + self.assertNotIn('[Beszel] Homepage is DOWN',argv) + + def test_up_transition_reads_as_up(self): + up={'monitor':{'name':'Gitea'},'heartbeat':{'status':1,'msg':'200 - OK'},'msg':'[Gitea] [Up] 200 - OK'} + with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],0,'{"message_id":8}','')) as run: + bridge.deliver(up,'/kuma') + self.assertIn('[Uptime Kuma] Gitea is UP',run.call_args[0][0]) + + def test_body_carries_the_target_and_the_kuma_board_not_the_beszel_hub(self): + with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],0,'{"message_id":9}','')) as run: + bridge.deliver(self.DOWN,'/kuma') + body=run.call_args[1]['input'] + self.assertIn('http://10.0.50.45:5100/',body) + self.assertIn('Board: http://10.250.50.70:3001',body) + self.assertNotIn('Hub: http://10.250.50.70:8090',body) # the Beszel footer must not leak + + def test_beszel_footer_is_unchanged_by_the_refactor(self): + with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],0,'{"message_id":10}','')) as run: + bridge.deliver({'title':'Disk 85%','message':'tank is full'},'/beszel') + body=run.call_args[1]['input'] + self.assertIn('Hub: http://10.250.50.70:8090',body) + self.assertNotIn('Board:',body) + + def test_a_nameless_alert_prefers_its_message_over_a_fabricated_subject(self): + """Superseded an earlier assertion that this should read "unknown + monitor is DOWN". It should not: a real monitor always carries a name, + so a nameless payload is degenerate, and "something broke" tells the + reader more than a placeholder that looks like a bug.""" + with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],0,'{"message_id":11}','')) as run: + argv=None + bridge.deliver({'heartbeat':{'status':0},'msg':'something broke'},'/kuma') + argv=run.call_args[0][0] + self.assertIn('[Uptime Kuma] something broke',argv) + self.assertNotIn('unknown monitor',' '.join(argv)) + + def test_empty_kuma_message_is_refused_without_sending(self): + with patch.object(bridge.subprocess,'run') as run: + with self.assertRaises(ValueError): + bridge.deliver({'monitor':{'name':'X'},'heartbeat':{'status':0},'msg':''},'/kuma') + run.assert_not_called() + + def test_a_monitorless_notification_uses_its_own_message_as_the_subject(self): + """testNotification and cert-expiry arrive with no monitor and no + heartbeat. Fabricating "unknown monitor is ?" from those makes a real + alert read like a bug -- observed live on 2026-09-21.""" + with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],0,'{"message_id":13}','')) as run: + bridge.deliver({'msg':'althing (infra-ops) Testing'},'/kuma') + argv=run.call_args[0][0] + self.assertIn('[Uptime Kuma] althing (infra-ops) Testing',argv) + self.assertNotIn('unknown monitor',' '.join(argv)) + self.assertNotIn('is ?',' '.join(argv)) + + def test_legacy_beszel_env_names_still_resolve(self): + """A half-finished deploy (new bridge, old unit) must start, not crash.""" + with patch.dict(os.environ,{'ALERT_RECIPIENT':'','BESZEL_ALERT_RECIPIENT':'infra-ops'}): + with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],0,'{"message_id":12}','')) as run: + bridge.deliver({'title':'t','message':'m'}) + self.assertIn('infra-ops',run.call_args[0][0]) + + + +if __name__ == '__main__': + unittest.main() diff --git a/services/beszel-althing/README.md b/services/beszel-althing/README.md deleted file mode 100644 index af0709b..0000000 --- a/services/beszel-althing/README.md +++ /dev/null @@ -1,36 +0,0 @@ -# Beszel to althing - -`beszel-althing.service` runs on nh3-dev as lkraven, listening at -`10.100.10.50:8096`. Beszel sends Shoutrrr generic JSON to `/beszel`; the -bridge invokes the supported `postbox send` command with the body on stdin. -It uses the established infra-ops automation identity and sends to infra-ops. - -Deploy from the repository root: - -```sh -scripts/elway infra-ops@10.100.10.50 --playbook playbooks/beszel-althing.yaml -``` - -The service accepts requests only from ana-docker and local diagnostic -addresses. There is no public listener or new althing handle. It reports -success only after postbox returns a delivery receipt. Failures return HTTP -502 and appear in the system journal; there is no hidden retry queue. A -post-office outage can therefore lose an alert, and needs independent health -monitoring if guaranteed delivery during such outages becomes a requirement. - -```sh -systemctl status beszel-althing -sudo -n journalctl -u beszel-althing --since '1 hour ago' -curl -fsS http://10.100.10.50:8096/healthz -``` - -`/healthz` checks the bridge process, not the downstream inbox. End-to-end -verification requires a real Beszel threshold transition plus its althing -receipt. The first verified alert is recorded in `stacks/beszel/README.md`. - -To reroute later, change `BESZEL_ALERT_RECIPIENT` in the canonical unit to -`miranda`, deploy, and trigger another end-to-end test. Leave `ALTHING_HANDLE` -as infra-ops so the sender remains identifiable as infrastructure automation. -The operator explicitly chose infra-ops for now. - -Run `python3 -m unittest discover -s services/beszel-althing -p 'test_*.py'`. diff --git a/services/beszel-althing/beszel-althing.service b/services/beszel-althing/beszel-althing.service deleted file mode 100644 index 9e4d541..0000000 --- a/services/beszel-althing/beszel-althing.service +++ /dev/null @@ -1,24 +0,0 @@ -[Unit] -Description=Beszel alerts to the althing infra-ops inbox -After=network-online.target -Wants=network-online.target - -[Service] -User=lkraven -Group=lkraven -Environment=PYTHONUNBUFFERED=1 -Environment=ALTHING_HANDLE=infra-ops -Environment=ALTHING_POST_OFFICE=http://10.100.50.40:8390 -Environment=POSTBOX=/home/lkraven/.local/bin/postbox -Environment=BESZEL_ALERT_RECIPIENT=infra-ops -Environment=BESZEL_BIND_HOST=10.100.10.50 -Environment=BESZEL_ALLOWED_SOURCES=10.250.50.70,10.100.10.50,127.0.0.1 -ExecStart=/usr/bin/python3 /opt/beszel-althing/bridge.py -Restart=on-failure -RestartSec=5 -NoNewPrivileges=true -ProtectSystem=strict -PrivateTmp=true - -[Install] -WantedBy=multi-user.target diff --git a/services/beszel-althing/bridge.py b/services/beszel-althing/bridge.py deleted file mode 100644 index 22d389b..0000000 --- a/services/beszel-althing/bridge.py +++ /dev/null @@ -1,63 +0,0 @@ -#!/usr/bin/env python3 -"""Receive Beszel's Shoutrrr JSON and deliver through the supported postbox CLI.""" -import json -import os -import subprocess -from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer - - -def deliver(payload): - title = payload.get('title', 'Beszel fleet alert') - message = payload.get('message') - if not isinstance(title, str) or not isinstance(message, str) or not message.strip(): - raise ValueError('Expected a nonempty message and string title') - result = subprocess.run( - [os.environ['POSTBOX'], '--json', 'send', '--to', os.environ['BESZEL_ALERT_RECIPIENT'], - '--subject', '[Beszel] ' + title], - input=message + '\n\nHub: http://10.250.50.70:8090\n', - text=True, capture_output=True, timeout=25, - ) - if result.returncode: - raise RuntimeError('postbox delivery failed: ' + result.stderr.strip()) - receipt = json.loads(result.stdout) - print(json.dumps({'event': 'delivered', 'title': title, 'receipt': receipt}), flush=True) - return receipt - - -class Handler(BaseHTTPRequestHandler): - def respond(self, status, body): - data = json.dumps(body).encode() - self.send_response(status) - self.send_header('Content-Type', 'application/json') - self.send_header('Content-Length', str(len(data))) - self.end_headers() - self.wfile.write(data) - - def do_GET(self): - self.respond(200 if self.path == '/healthz' else 404, - {'service': 'beszel-althing', 'delivery': 'verified per POST'}) - - def do_POST(self): - if self.path != '/beszel': - return self.respond(404, {'error': 'Unknown route'}) - if self.client_address[0] not in os.environ['BESZEL_ALLOWED_SOURCES'].split(','): - return self.respond(403, {'error': 'Source not allowed'}) - try: - length = int(self.headers.get('Content-Length', '0')) - if not 0 < length <= 65536: - raise ValueError('Invalid body size') - self.connection.settimeout(10) - payload = json.loads(self.rfile.read(length)) - if not isinstance(payload, dict): - raise ValueError('Expected JSON object') - receipt = deliver(payload) - except (ValueError, TypeError) as exc: - return self.respond(400, {'error': str(exc)}) - except (OSError, RuntimeError, subprocess.TimeoutExpired) as exc: - print(json.dumps({'event': 'delivery_failed', 'error': str(exc)}), flush=True) - return self.respond(502, {'error': 'Althing delivery failed; inspect service journal'}) - self.respond(200, {'delivered': True, 'receipt': receipt}) - - -if __name__ == '__main__': - ThreadingHTTPServer((os.environ['BESZEL_BIND_HOST'], int(os.environ.get('BESZEL_BIND_PORT', '8096'))), Handler).serve_forever() diff --git a/services/beszel-althing/test_bridge.py b/services/beszel-althing/test_bridge.py deleted file mode 100644 index bd859dd..0000000 --- a/services/beszel-althing/test_bridge.py +++ /dev/null @@ -1,30 +0,0 @@ -import importlib.util -import os -from pathlib import Path -import subprocess -import unittest -from unittest.mock import patch - -spec=importlib.util.spec_from_file_location('bridge',Path(__file__).with_name('bridge.py')) -bridge=importlib.util.module_from_spec(spec);spec.loader.exec_module(bridge) - -class DeliveryTests(unittest.TestCase): - def setUp(self): - self.env=patch.dict(os.environ,POSTBOX='/bin/postbox',BESZEL_ALERT_RECIPIENT='infra-ops') - self.env.start();self.addCleanup(self.env.stop) - def test_posts_body_as_stdin_and_returns_receipt(self): - with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],0,'{"message_id": 42}','')) as run: - self.assertEqual(bridge.deliver({'title':'Disk 85%','message':'tank is full'})['message_id'],42) - args,kw=run.call_args - self.assertEqual(args[0],['/bin/postbox','--json','send','--to','infra-ops','--subject','[Beszel] Disk 85%']) - self.assertIn('tank is full',kw['input']) - def test_delivery_failure_is_not_success(self): - with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],2,'','post office unavailable')): - with self.assertRaises(RuntimeError):bridge.deliver({'message':'disk full'}) - def test_rejects_empty_message_without_sending(self): - with patch.object(bridge.subprocess,'run') as run: - with self.assertRaises(ValueError):bridge.deliver({'message':''}) - run.assert_not_called() - -if __name__ == '__main__': - unittest.main() diff --git a/stacks/althing-chamber/.env.example b/stacks/althing-chamber/.env.example deleted file mode 100644 index abfddb2..0000000 --- a/stacks/althing-chamber/.env.example +++ /dev/null @@ -1,27 +0,0 @@ -# althing-chamber stack tunables. Copy to `.env` on ana-docker before deploying. -# -# The deploy playbook seeds `.env` from this template on first run only — -# it won't clobber an existing `.env`. - -# Image tag. Built locally from the vh/althing git repo by the playbook. -ALTHING_IMAGE=althing-chamber:local - -# Host port exposing the chamber UI (container always listens on 8000 -# internally). Internal-only — no Traefik. -# Chamber's compiled-in default port is 7878 but that collides with -# task-board's 7878 on the same host. 7881 is the canonical fleet slot, -# adjacent to task-board:7878 and vor:7879. -ALTHING_PORT=7881 - -# Bind address for the host port. 0.0.0.0 = LAN-reachable (default for an -# internal-only tool). -ALTHING_BIND=0.0.0.0 - -# Host path for SQLite + state. Container runs as uid 1000 (matches -# lkraven on these hosts) so the playbook's mkdir without sudo produces -# a writable dir. -# -# This single dir is the only persistent state — althing's SQLite DB, -# read_cursors, notification_state, hand_queue, floor_grants all live -# here. Restic backs it up via the standard /opt/docker tree. -ALTHING_DATA_DIR=/opt/docker/conf/althing-chamber/data diff --git a/stacks/althing-chamber/README.md b/stacks/althing-chamber/README.md deleted file mode 100644 index 70c03b5..0000000 --- a/stacks/althing-chamber/README.md +++ /dev/null @@ -1,110 +0,0 @@ -# althing-chamber - -Web UI + moderator daemon + agent-runner + Valkey IPC bridge for the -althing inter-agent message bus. Four services on the compose default -network. The chamber serves FastAPI/HTMX, the forseti daemon runs the -moderation + curation loops, the agent-runner daemon claims floor -grants for worldtree-driver agents and dispatches them to Worldtree's -conversation API, and Valkey carries the Phase 3.1 cross-process -streaming events (msg_start/delta/thinking/complete/curated) that are -too high-volume to flow through SQLite. SQLite-backed state still -shares between chamber + forseti + agent-runner via the bind-mount; -streaming events ride pub/sub on the docker default network. - -**Server:** ana-docker -**URL:** `http://10.250.50.70:7881` (configurable via `.env`) -**Upstream repo:** [vh/althing](https://gitea.phasefinal.com/vh/althing) -**Image:** `althing-chamber:local` — built on the host from the git repo -by the deploy playbook. Not pulled from a registry. - -## Services in this stack - -| Container | Role | Port | Healthcheck | -|---|---|---|---| -| `althing-chamber` | FastAPI/HTMX web UI; SSE subscribers; `/health` endpoint | host 7881 → container 8000 | `python urllib /health` | -| `althing-forseti` | Moderator + curator daemon; writes events that chamber's bridge picks up | — (no HTTP) | none (process-up signal only) | -| `althing-agent-runner` | Phase 2: claims worldtree-driver floor grants and dispatches to Worldtree's conversation API | — (no HTTP) | none (process-up signal only) | -| `althing-valkey` | Phase 3.1: Valkey 8 redis-protocol pub/sub bridge — carries streaming events from agent-runner → chamber | — (internal-only, `valkey:6379` on default network) | `valkey-cli ping` | - -The three althing services (chamber/forseti/agent-runner) use the same -`${ALTHING_IMAGE}`; the `command:` line in compose picks which entrypoint -runs in each container. Valkey is a stock upstream image (no custom build). -chamber + agent-runner `depends_on: valkey: service_healthy` so the pub/sub -bridge is up before either side starts publishing or subscribing. - -The agent-runner is safe to enable preemptively — when no -`driver=worldtree` handles are declared in config, it polls -`floor_grants` and sleeps when the query returns empty. Multi-instance -safe via the atomic `UPDATE … WHERE consumed_at IS NULL` claim primitive. - -## Deploy - -Two paths — automated (preferred) and manual (escape hatch / first-time). - -### Automated (Gitea Actions, push-to-main) - -The vh/althing repo ships `.gitea/workflows/deploy.yaml`. Every push to -main + manual `workflow_dispatch` triggers the elway playbook below -pinned to the triggering commit SHA. A reference copy of the workflow -lives next to this README at -[`gitea-workflow-deploy.yaml.example`](gitea-workflow-deploy.yaml.example); -the canonical source is in the vh/althing repo. The example header -lists the two repo secrets required (`DEPLOY_SSH_KEY`, -`MGMT_REPO_TOKEN`). - -### Manual (elway from a workstation) - -```bash -# First deploy (or update to latest main) -scripts/elway ana-docker --playbook playbooks/deploy-althing-chamber.yaml - -# Pin to a specific ref (tag, branch, or commit SHA) -scripts/elway ana-docker --playbook playbooks/deploy-althing-chamber.yaml --var ref=v0.1.0 -``` - -## Path layout (on ana-docker) - -| Host path | Container path | Purpose | Restic? | -|---|---|---|---| -| `/opt/docker/build/althing-chamber/` | — | git checkout used as docker build context | excluded | -| `/opt/docker/compose/althing-chamber/` | — | compose.yaml + .env | included (via `/opt/docker`) | -| `/opt/docker/conf/althing-chamber/data/` | `/app/data` | SQLite + state for BOTH services | **included** | - -## Network model - -Internal tooling, LAN-only. Chamber's container port 8000 is published -on the host at `0.0.0.0:7881` (configurable via `ALTHING_BIND` / -`ALTHING_PORT`); access is direct via `http://10.250.50.70:7881`. No -Traefik, no TLS terminator, no public hostname. - -Forseti has no port — it's a daemon. The two services communicate only -via the shared SQLite file under the `${ALTHING_DATA_DIR}` bind-mount; -no docker network coupling beyond compose's default bridge. - -## Env-var contract - -| Var | In env? | In `~/.althing/config.yaml`? | Notes | -|---|---|---|---| -| `ALTHING_ROOT` | ✓ (set in compose) | n/a | Overrides the default `~/.althing/` dir. Container sets `/app/data`. | -| `ALTHING_DB` | ✓ (set in compose) | n/a | Explicit SQLite path. Defaults to `${ALTHING_ROOT}/althing.db`. | -| `ALTHING_BIND` | ✓ (deploy-time) | also accepted | Bind address for the chamber HTTP server. Container always sets 0.0.0.0 internally. | -| `ALTHING_PORT` | ✓ (deploy-time) | also accepted | Chamber listens here internally (always 8000 inside the container). | - -Env-var support for `ALTHING_BIND` / `ALTHING_PORT` was added on the -galdrabok side as part of the container-deploy cycle (env > config.yaml -> defaults precedence). - -## First-deploy sequence (when galdrabok's Dockerfile lands) - -1. Generate a deploy keypair on ana-docker (private stays on host; - pubkey lands in `~lkraven/.ssh/authorized_keys`). -2. Wire two secrets in `vh/althing` Actions settings: - - `DEPLOY_SSH_KEY` — the private key from step 1. - - `MGMT_REPO_TOKEN` — Gitea PAT with `read:repository` on this repo, - used by the workflow to clone the management repo for the playbook. -3. Push to vh/althing's main branch (or trigger `workflow_dispatch`); - the Actions runner clones both repos, configures SSH, runs - `scripts/elway ana-docker --playbook playbooks/deploy-althing-chamber.yaml --var ref=$SHA`. -4. Playbook: clones into `/opt/docker/build/althing-chamber`, builds - the image, uploads compose + .env (one-time seed), brings both - services up, polls `/health` until 200. diff --git a/stacks/althing-chamber/compose.yaml b/stacks/althing-chamber/compose.yaml deleted file mode 100644 index 5388c03..0000000 --- a/stacks/althing-chamber/compose.yaml +++ /dev/null @@ -1,124 +0,0 @@ -# althing-chamber stack — chamber + forseti + agent-runner + valkey. -# -# Four services on the compose default network: -# althing-chamber — FastAPI/HTMX app (port 7881 host → 8000 container) -# althing-forseti — moderator daemon (no port; cross-process glue via the DB) -# althing-agent-runner — Phase 2 daemon: claims worldtree-driver grants and -# dispatches to Worldtree's conversation API (no port) -# valkey — Phase 3.1 sibling service: Valkey 8 alpine, redis-protocol -# pub/sub bridge for cross-process streaming events from -# agent-runner → chamber SSE subscribers. Reached via -# docker DNS at `valkey:6379` on the default network. -# No exposed port; internal-only. -# -# Three storage / IPC channels: -# - SQLite bind-mount at /app/data backs the althing.db state shared by -# chamber + forseti + agent-runner (eventbus.bridge_from_db drains DB -# commits into chamber's SSE). -# - Valkey pub/sub on docker-default network carries the Phase 3 streaming -# events (msg_start/thinking/delta/complete/curated) that don't go -# through SQLite — too high-volume + ephemeral for the DB. -# - chamber's healthcheck-blocked startup gate on valkey ensures the -# subscriber side is up before chamber begins handling SSE traffic. -# -# Image is built on the host from the vh/althing git repo by the deploy -# playbook (`playbooks/deploy-althing-chamber.yaml`), which clones into -# /opt/docker/build/althing-chamber and runs `docker build -t -# althing-chamber:local .` before installing this compose and bringing -# both services up. No registry. -# -# Internal tooling — accessed directly at http://10.250.50.70:7881 over -# the LAN; does NOT traverse Traefik. State persists under -# /opt/docker/conf/althing-chamber/data on the host. -# -# All tunables live in .env — edit that, not this file. - -services: - valkey: - image: valkey/valkey:8-alpine - container_name: althing-valkey - restart: unless-stopped - # No exposed port — chamber + agent-runner reach via docker DNS - # at `valkey:6379` on the compose default network. No volume — - # the pub/sub channel doesn't persist anything between restarts. - healthcheck: - test: ["CMD", "valkey-cli", "ping"] - interval: 5s - timeout: 3s - retries: 5 - - althing-chamber: - image: ${ALTHING_IMAGE} - container_name: althing-chamber - restart: unless-stopped - depends_on: - valkey: - condition: service_healthy - ports: - - "${ALTHING_BIND:-0.0.0.0}:${ALTHING_PORT}:8000" - environment: - # ALTHING_ROOT moves the entire ~/.althing dir; ALTHING_DB additionally - # pins the SQLite path explicitly. Both point inside the bind-mount. - - ALTHING_ROOT=/app/data - - ALTHING_DB=/app/data/althing.db - # ALTHING_BIND / ALTHING_PORT — galdrabok-side env-var precedence - # (env > config.yaml > defaults) is being added in the same cycle - # as this scaffold lands. Container always listens on 8000 internally; - # the host port mapping above is the only externally-visible knob. - - ALTHING_BIND=0.0.0.0 - - ALTHING_PORT=8000 - volumes: - - ${ALTHING_DATA_DIR}:/app/data - command: ["althing-chamber"] - healthcheck: - # Liveness probe — chamber's /health endpoint returns 200 with no DB - # read (true liveness, not readiness). galdrabok adds this endpoint - # in the same cycle as this scaffold; container will crashloop on - # healthcheck until that lands. - test: ["CMD-SHELL", "python -c 'import urllib.request,sys; r=urllib.request.urlopen(\"http://127.0.0.1:8000/health\",timeout=3); sys.exit(0 if r.status==200 else 1)' || exit 1"] - interval: 30s - timeout: 5s - retries: 3 - start_period: 30s - labels: - - homepage.group=Toolchain - - homepage.name=althing chamber - - homepage.icon=mdi-bullhorn - - homepage.description=Web UI for the althing inter-agent message bus - - homepage.href=http://10.250.50.70:${ALTHING_PORT} - - althing-forseti: - image: ${ALTHING_IMAGE} - container_name: althing-forseti - restart: unless-stopped - environment: - - ALTHING_ROOT=/app/data - - ALTHING_DB=/app/data/althing.db - volumes: - - ${ALTHING_DATA_DIR}:/app/data - command: ["althing-forseti"] - # No healthcheck — the forseti CLI doesn't expose one. Liveness signal - # for ops is "container hasn't exited" + chamber-side observation that - # bridge_from_db events are flowing. - - althing-agent-runner: - image: ${ALTHING_IMAGE} - container_name: althing-agent-runner - restart: unless-stopped - depends_on: - valkey: - condition: service_healthy - environment: - - ALTHING_ROOT=/app/data - - ALTHING_DB=/app/data/althing.db - volumes: - - ${ALTHING_DATA_DIR}:/app/data - command: ["althing-agent-runner"] - # Phase 2 daemon (added 2026-05-16). Polls floor_grants WHERE - # consumed_at IS NULL AND agents.driver='worldtree', claims via - # atomic UPDATE, calls Worldtree's conversation API, posts the - # agent's response back through the bus as a broadcast. Harmless - # when no driver=worldtree handles are declared — runner sleeps - # at poll_interval_seconds. Multi-instance safe via claim_grant's - # atomic UPDATE; no flock required. Same lack-of-healthcheck story - # as forseti (CLI doesn't expose one). diff --git a/stacks/althing-chamber/gitea-workflow-deploy.yaml.example b/stacks/althing-chamber/gitea-workflow-deploy.yaml.example deleted file mode 100644 index 91480b8..0000000 --- a/stacks/althing-chamber/gitea-workflow-deploy.yaml.example +++ /dev/null @@ -1,91 +0,0 @@ -# Gitea Actions workflow for althing-chamber. -# -# THIS FILE LIVES IN THE VH/ALTHING REPO, NOT HERE. -# Copy to vh/althing:.gitea/workflows/deploy.yaml and commit. -# (The canonical copy lives in the althing repo; this file is a -# reference for what shape the workflow takes.) -# -# What it does on every push to main (and on manual workflow_dispatch): -# 1. Checks out althing itself (the triggering repo). -# 2. Checks out vh/esh-pfi-infrastructure to pick up the elway -# playbook and helper scripts. -# 3. Configures SSH so elway can reach ana-docker. -# 4. Runs `scripts/elway ana-docker --playbook playbooks/deploy-althing-chamber.yaml` -# pinning to the commit SHA that triggered the workflow. -# -# Required Actions secrets (configure under -# https://gitea.phasefinal.com/vh/althing/settings/actions/secrets, -# or org-level for reuse across repos): -# -# DEPLOY_SSH_KEY Private SSH key whose pubkey is in -# ~lkraven/.ssh/authorized_keys on ana-docker. -# Used by the runner to invoke the elway playbook. -# Generate fresh; don't reuse a personal key. -# -# MGMT_REPO_TOKEN Gitea PAT (read:repository scope) on -# vh/esh-pfi-infrastructure, used to clone the -# management repo. Generate at -# https://gitea.phasefinal.com/-/user/settings/applications. - -name: Deploy althing-chamber - -on: - push: - branches: [main] - workflow_dispatch: - -jobs: - deploy: - # `pfi-fleet` matches the central runner on ana-docker. Pin to - # `ana-docker` instead if you want to refuse running on a future - # site-local runner. The runner's label embeds a default image - # (node:20-bookworm-slim) — has node + git out of the box, so - # actions/checkout@v4 (a JS action) works without a custom - # container. We just apt-install python3 + pyyaml for elway. - runs-on: pfi-fleet - - steps: - - name: Install playbook prerequisites - run: | - apt-get update -qq - apt-get install -y --no-install-recommends \ - python3 python3-yaml openssh-client - rm -rf /var/lib/apt/lists/* - - - name: Checkout althing (triggering repo) - uses: actions/checkout@v4 - - - name: Checkout management repo (eshpfi-management) - uses: actions/checkout@v4 - with: - repository: vh/esh-pfi-infrastructure - token: ${{ secrets.MGMT_REPO_TOKEN }} - path: _mgmt - - - name: Configure SSH to ana-docker - run: | - mkdir -p ~/.ssh - # The DEPLOY_SSH_KEY secret is the full private key contents, - # newline-terminated. ssh refuses keys that aren't 0600. - printf '%s\n' "${{ secrets.DEPLOY_SSH_KEY }}" > ~/.ssh/id_ed25519 - chmod 600 ~/.ssh/id_ed25519 - - # ssh_config alias so elway resolves "ana-docker" the same - # way it would on a workstation. accept-new is fine for a - # fresh job container — host key gets cached for the lifetime - # of this job only. - cat > ~/.ssh/config <<'EOF' - Host ana-docker - HostName 10.250.50.70 - User lkraven - IdentityFile ~/.ssh/id_ed25519 - StrictHostKeyChecking accept-new - EOF - chmod 600 ~/.ssh/config - - - name: Deploy althing-chamber (elway playbook, pinned to this commit) - working-directory: _mgmt - run: | - scripts/elway ana-docker \ - --playbook playbooks/deploy-althing-chamber.yaml \ - --var ref=${{ github.sha }} diff --git a/stacks/beszel/README.md b/stacks/beszel/README.md index bb773d5..36a1fee 100644 --- a/stacks/beszel/README.md +++ b/stacks/beszel/README.md @@ -120,7 +120,7 @@ corviduo-dev remains monitored but its alert policy was not changed. Notification URL: `generic://10.100.10.50:8096/beszel?disabletls=yes&template=json` -The bridge at `services/beszel-althing/` forwards through `postbox` to the +The bridge at `services/althing-alert-bridge/` forwards through `postbox` to the **infra-ops inbox**, as the operator requested. Existing unused email delivery was replaced with this verified route. Miranda is a later cutover, not enabled. See that service's README for operation and recipient changes. diff --git a/stacks/uptimekuma/monitors.yaml b/stacks/uptimekuma/monitors.yaml index 6d55a88..9b0c9cd 100644 --- a/stacks/uptimekuma/monitors.yaml +++ b/stacks/uptimekuma/monitors.yaml @@ -17,10 +17,22 @@ # 2026-09-21. A seed that ships red on day one teaches everyone to ignore the # board, which is how you end up with a monitor nobody reads. # -# NOT SEEDED, deliberately: "althing chamber" (10.250.50.70:7881) is on the -# Homepage dashboard but refuses connections right now. It is either genuinely -# down or retired; seeding it would put a red row on a brand-new board before -# anyone has decided which. Resolve it, then add it. +# "althing chamber" was a candidate and was RETIRED instead (operator, +# 2026-09-21): three of its four containers had never started since being +# created on 2026-09-19, so :7881 refused. Stack, host dirs and image removed. + +# ---- where alerts GO --------------------------------------------------------- +# Seeded BEFORE the monitors, and `applyExisting` attaches the channel to rows +# that already exist. A board that detects and notifies nobody is precisely the +# failure this service layer was built to close -- Homepage's own healthcheck +# caught its 2026-09-18 death correctly and nothing was subscribed. +# +# Route: Kuma -> althing-alert-bridge (/kuma) -> postbox -> infra-ops inbox. +# Same path Beszel uses, different route, so the subject says which tool spoke: +# "[Uptime Kuma] Homepage is DOWN" rather than a Beszel-labelled lie. +notifications: + - name: althing (infra-ops) + webhookURL: http://10.100.10.50:8096/kuma monitors: # ---- fleet toolchain: dead = agents and the operator are blocked ----