vh
5960526c3f
NH3 site visit done: Secure Boot off, iGPU restored as boot VGA, AMT port cabled. - nh3-pve: NVIDIA 580.178.04 (DKMS, open modules) via pve-nvidia-host.yaml. - nh3-ml1 = CT 109 @ 10.100.50.80 via gpu-lxc.yaml; embed-rerank (TEI 1.9.4) deployed with HOST_NAME/HOST_IP labels. - Parity vs esh-ml1 (1,126 texts, 2 runs/host, controls): embed cosine min 0.999993 = own noise floor; overlap@10 1.000 vs MRL-256 positive control 0.684; rerank top-1 1.00, max diff 0.0014 vs floor 0.0020. On-box speed identical within rep spread. - gpu-lxc.yaml: first step upgrades lxc-pve to >= 6.0.0-2 (Proxmox fix #7006). With 6.0.0-1 every docker run in a nesting CT failed on runc 1.5's sysctl reopen; applied on nh3-pve (one package). - pve-nvidia-host.yaml: document that the headers meta drags in the newest kernel (nh3-pve went 6.8.12-11 -> -43 at the next reboot). - Monitoring: Beszel NVIDIA agent + 5 alerts, Kuma #29/#30, Homepage nh3-ml1-docker, Dozzle agent (hub 8 clients). DNS nh3-ml1.nh3.internal. - nh3-pve README: SB/IGFX/driver/kernel state, btmtk oops on -4x kernels, AMT cabled but unreachable on the network. Gateway routing to nh3-ml1 is not changed.
134 lines
6.0 KiB
YAML
134 lines
6.0 KiB
YAML
# Uptime Kuma — the fleet's service-layer monitors, as code.
|
|
#
|
|
# scripts/kuma seed stacks/uptimekuma/monitors.yaml
|
|
#
|
|
# Idempotent: keyed on NAME, so re-running edits rather than duplicating. Name
|
|
# and not URL, because a URL legitimately repeats (a root and a /healthz on one
|
|
# service) and a re-run must never fork the board.
|
|
#
|
|
# WHAT BELONGS HERE — the lane rule, so this file does not sprawl to 115 rows:
|
|
# IN — services that should ALWAYS be up, whose silent death costs us work.
|
|
# OUT — inference seats (they come and go BY DESIGN; a dormant seat is normal
|
|
# and alerting on it is pure noise), hosts and hypervisors and firewalls
|
|
# (Beszel's lane: 18 hosts x Status/CPU/Memory/Disk/Temp), and Uptime
|
|
# Kuma itself (it cannot report its own death — that is Beszel's job).
|
|
#
|
|
# NAMES COME FROM HOMEPAGE, VERBATIM. Homepage already answers "what is this
|
|
# service called", and a second naming authority is how drift starts: an alert
|
|
# reading "[Uptime Kuma] Beszel hub is DOWN" sends you looking for a card called
|
|
# "Beszel hub" that does not exist. So the monitor name IS the dashboard name --
|
|
# which is why this normalisation pass only moved two rows. The remaining mixed
|
|
# case (talk, vor, task-board against Gitea, Backrest) is NOT an inconsistency to
|
|
# fix: those are the products' own names, lowercase on Homepage and lowercase in
|
|
# their own repos. Title-casing them here would make this board disagree with
|
|
# both. `rename_from` exists for exactly this operation -- see scripts/kuma.
|
|
#
|
|
# Every URL below was probed before being written here: all returned 200 on
|
|
# 2026-09-21. A seed that ships red on day one teaches everyone to ignore the
|
|
# board, which is how you end up with a monitor nobody reads.
|
|
#
|
|
# "althing chamber" was a candidate and was RETIRED instead (operator,
|
|
# 2026-09-21): three of its four containers had never started since being
|
|
# created on 2026-09-19, so :7881 refused. Stack, host dirs and image removed.
|
|
|
|
# ---- where alerts GO ---------------------------------------------------------
|
|
# Seeded BEFORE the monitors, and `applyExisting` attaches the channel to rows
|
|
# that already exist. A board that detects and notifies nobody is precisely the
|
|
# failure this service layer was built to close -- Homepage's own healthcheck
|
|
# caught its 2026-09-18 death correctly and nothing was subscribed.
|
|
#
|
|
# Route: Kuma -> althing-alert-bridge (/kuma) -> postbox -> infra-ops inbox.
|
|
# Same path Beszel uses, different route, so the subject says which tool spoke:
|
|
# "[Uptime Kuma] Homepage is DOWN" rather than a Beszel-labelled lie.
|
|
# ---- the published status page -----------------------------------------------
|
|
# Exists so Homepage's `uptimekuma` widget has something to read: it calls
|
|
# /api/status-page/<slug> and /api/status-page/heartbeat/<slug>, and without a
|
|
# published page at that slug it polls a 404 forever. The widget labels on
|
|
# stacks/uptimekuma/compose.yaml were deliberately held back until this existed.
|
|
# Slug kept as `nethealth` -- the same one the pre-rebuild instance used, so any
|
|
# bookmark or older reference still resolves.
|
|
status_page:
|
|
slug: nethealth
|
|
title: PFI fleet services
|
|
group: Services
|
|
|
|
notifications:
|
|
- name: althing (infra-ops)
|
|
webhookURL: http://10.100.10.50:8096/kuma
|
|
|
|
monitors:
|
|
# ---- fleet toolchain: dead = agents and the operator are blocked ----
|
|
- name: althing post office
|
|
url: http://10.100.50.40:8390/
|
|
description: the bus every agent session reads mail from
|
|
|
|
- name: The Booth
|
|
url: http://10.100.10.50:8090/healthz
|
|
description: operator-review surface; real healthz, not a root page
|
|
|
|
- name: vor
|
|
url: http://10.250.50.70:7879
|
|
|
|
# ---- the dashboard that started all this ----
|
|
# Dead for three days in Sept 2026 while Beszel correctly reported its host
|
|
# UP. This row is the entire reason the service layer exists.
|
|
- name: Homepage
|
|
url: http://10.0.50.45:5100/
|
|
description: fleet dashboard (esh-docker-vm) — the 2026-09-18 silent death
|
|
|
|
# ---- credentials + code: dead = nothing ships ----
|
|
- name: Gitea
|
|
url: https://gitea.phasefinal.com
|
|
|
|
- name: Vaultwarden
|
|
url: https://vaultwarden.phasefinal.com
|
|
description: the vault every agent reads credentials from
|
|
|
|
# ---- AI control plane (the gateways, NOT the seats behind them) ----
|
|
- name: LiteLLM Gateway
|
|
url: http://10.250.50.70:4000/health/liveliness
|
|
description: liveliness endpoint, so a model outage does not read as a gateway outage
|
|
|
|
- name: Asset Engine
|
|
url: http://10.250.50.70:8200/api/v1/services
|
|
|
|
# ---- fleet embed/rerank: the one EXCEPTION to "seats are OUT" ----
|
|
# Since 2026-09-25 these back the gateway's `qwen3-embedding` and `reranker`
|
|
# (TEI on esh-ml1; nh3-ml1 is the second RTX 2000, parity-verified the same day,
|
|
# gateway routing pending Prime). They are not come-and-go seats: dead =
|
|
# Worldtree recall, nevermore clustering and Open WebUI RAG all fail. TEI's
|
|
# /health runs the backend, so a loaded-but-broken model reads DOWN, not UP. The
|
|
# reward seat on esh-ml1 stays OUT: it has no working consumer
|
|
# (stacks/reward-seat/README.md).
|
|
- name: Embed — Qwen3 0.6B (TEI, esh-ml1)
|
|
url: http://10.0.50.80:8001/health
|
|
description: gateway `qwen3-embedding` backend (esh-ml1)
|
|
|
|
- name: Rerank — bge-v2-m3 (TEI, esh-ml1)
|
|
url: http://10.0.50.80:8013/health
|
|
description: gateway `reranker` backend (esh-ml1)
|
|
|
|
- name: Embed — Qwen3 0.6B (TEI, nh3-ml1)
|
|
url: http://10.100.50.80:8001/health
|
|
description: second `qwen3-embedding` backend (nh3-ml1)
|
|
|
|
- name: Rerank — bge-v2-m3 (TEI, nh3-ml1)
|
|
url: http://10.100.50.80:8013/health
|
|
description: second `reranker` backend (nh3-ml1)
|
|
|
|
- name: talk
|
|
url: https://talk.nh3.phasefinal.com:8092/
|
|
description: fleet voice bench
|
|
|
|
# ---- monitoring + backup: a blind monitor is worse than none ----
|
|
- name: Beszel
|
|
url: http://10.250.50.70:8090
|
|
description: the host layer; if this is down we are blind to 18 hosts
|
|
|
|
- name: Backrest
|
|
url: http://10.250.50.70:9898
|
|
description: restic orchestration — a silent backup failure is the expensive kind
|
|
|
|
- name: Dozzle
|
|
url: http://10.250.50.70:8088
|