vh
50c85e0e8a
- qwen3-embedding and reranker: second deployment on nh3-ml1 (config); reranker-a3-bge-v2-m3: second DB deployment via /model/new. - router_settings.enable_weighted_failover: true. Only the three multi-deployment groups are affected. - Measured by stopping nh3-ml1 TEI: without failover 7/40 embeds 500'd; with it rerank 80/80, embed 38/40 at onset and 60/60 over a 34 s outage; nh3 rejoins rotation after restart. Split 12/28 embed, 22/18 rerank.
134 lines
6.0 KiB
YAML
134 lines
6.0 KiB
YAML
# Uptime Kuma — the fleet's service-layer monitors, as code.
|
|
#
|
|
# scripts/kuma seed stacks/uptimekuma/monitors.yaml
|
|
#
|
|
# Idempotent: keyed on NAME, so re-running edits rather than duplicating. Name
|
|
# and not URL, because a URL legitimately repeats (a root and a /healthz on one
|
|
# service) and a re-run must never fork the board.
|
|
#
|
|
# WHAT BELONGS HERE — the lane rule, so this file does not sprawl to 115 rows:
|
|
# IN — services that should ALWAYS be up, whose silent death costs us work.
|
|
# OUT — inference seats (they come and go BY DESIGN; a dormant seat is normal
|
|
# and alerting on it is pure noise), hosts and hypervisors and firewalls
|
|
# (Beszel's lane: 18 hosts x Status/CPU/Memory/Disk/Temp), and Uptime
|
|
# Kuma itself (it cannot report its own death — that is Beszel's job).
|
|
#
|
|
# NAMES COME FROM HOMEPAGE, VERBATIM. Homepage already answers "what is this
|
|
# service called", and a second naming authority is how drift starts: an alert
|
|
# reading "[Uptime Kuma] Beszel hub is DOWN" sends you looking for a card called
|
|
# "Beszel hub" that does not exist. So the monitor name IS the dashboard name --
|
|
# which is why this normalisation pass only moved two rows. The remaining mixed
|
|
# case (talk, vor, task-board against Gitea, Backrest) is NOT an inconsistency to
|
|
# fix: those are the products' own names, lowercase on Homepage and lowercase in
|
|
# their own repos. Title-casing them here would make this board disagree with
|
|
# both. `rename_from` exists for exactly this operation -- see scripts/kuma.
|
|
#
|
|
# Every URL below was probed before being written here: all returned 200 on
|
|
# 2026-09-21. A seed that ships red on day one teaches everyone to ignore the
|
|
# board, which is how you end up with a monitor nobody reads.
|
|
#
|
|
# "althing chamber" was a candidate and was RETIRED instead (operator,
|
|
# 2026-09-21): three of its four containers had never started since being
|
|
# created on 2026-09-19, so :7881 refused. Stack, host dirs and image removed.
|
|
|
|
# ---- where alerts GO ---------------------------------------------------------
|
|
# Seeded BEFORE the monitors, and `applyExisting` attaches the channel to rows
|
|
# that already exist. A board that detects and notifies nobody is precisely the
|
|
# failure this service layer was built to close -- Homepage's own healthcheck
|
|
# caught its 2026-09-18 death correctly and nothing was subscribed.
|
|
#
|
|
# Route: Kuma -> althing-alert-bridge (/kuma) -> postbox -> infra-ops inbox.
|
|
# Same path Beszel uses, different route, so the subject says which tool spoke:
|
|
# "[Uptime Kuma] Homepage is DOWN" rather than a Beszel-labelled lie.
|
|
# ---- the published status page -----------------------------------------------
|
|
# Exists so Homepage's `uptimekuma` widget has something to read: it calls
|
|
# /api/status-page/<slug> and /api/status-page/heartbeat/<slug>, and without a
|
|
# published page at that slug it polls a 404 forever. The widget labels on
|
|
# stacks/uptimekuma/compose.yaml were deliberately held back until this existed.
|
|
# Slug kept as `nethealth` -- the same one the pre-rebuild instance used, so any
|
|
# bookmark or older reference still resolves.
|
|
status_page:
|
|
slug: nethealth
|
|
title: PFI fleet services
|
|
group: Services
|
|
|
|
notifications:
|
|
- name: althing (infra-ops)
|
|
webhookURL: http://10.100.10.50:8096/kuma
|
|
|
|
monitors:
|
|
# ---- fleet toolchain: dead = agents and the operator are blocked ----
|
|
- name: althing post office
|
|
url: http://10.100.50.40:8390/
|
|
description: the bus every agent session reads mail from
|
|
|
|
- name: The Booth
|
|
url: http://10.100.10.50:8090/healthz
|
|
description: operator-review surface; real healthz, not a root page
|
|
|
|
- name: vor
|
|
url: http://10.250.50.70:7879
|
|
|
|
# ---- the dashboard that started all this ----
|
|
# Dead for three days in Sept 2026 while Beszel correctly reported its host
|
|
# UP. This row is the entire reason the service layer exists.
|
|
- name: Homepage
|
|
url: http://10.0.50.45:5100/
|
|
description: fleet dashboard (esh-docker-vm) — the 2026-09-18 silent death
|
|
|
|
# ---- credentials + code: dead = nothing ships ----
|
|
- name: Gitea
|
|
url: https://gitea.phasefinal.com
|
|
|
|
- name: Vaultwarden
|
|
url: https://vaultwarden.phasefinal.com
|
|
description: the vault every agent reads credentials from
|
|
|
|
# ---- AI control plane (the gateways, NOT the seats behind them) ----
|
|
- name: LiteLLM Gateway
|
|
url: http://10.250.50.70:4000/health/liveliness
|
|
description: liveliness endpoint, so a model outage does not read as a gateway outage
|
|
|
|
- name: Asset Engine
|
|
url: http://10.250.50.70:8200/api/v1/services
|
|
|
|
# ---- fleet embed/rerank: the one EXCEPTION to "seats are OUT" ----
|
|
# Since 2026-09-25 these back the gateway's `qwen3-embedding` and `reranker`
|
|
# (TEI on esh-ml1; nh3-ml1 is the second RTX 2000, parity-verified the same day,
|
|
# gateway routing pending Prime). They are not come-and-go seats: dead =
|
|
# Worldtree recall, nevermore clustering and Open WebUI RAG all fail. TEI's
|
|
# /health runs the backend, so a loaded-but-broken model reads DOWN, not UP. The
|
|
# reward seat on esh-ml1 stays OUT: it has no working consumer
|
|
# (stacks/reward-seat/README.md).
|
|
- name: Embed — Qwen3 0.6B (TEI, esh-ml1)
|
|
url: http://10.0.50.80:8001/health
|
|
description: gateway `qwen3-embedding` backend (esh-ml1)
|
|
|
|
- name: Rerank — bge-v2-m3 (TEI, esh-ml1)
|
|
url: http://10.0.50.80:8013/health
|
|
description: gateway `reranker` backend (esh-ml1)
|
|
|
|
- name: Embed — Qwen3 0.6B (TEI, nh3-ml1)
|
|
url: http://10.100.50.80:8001/health
|
|
description: gateway `qwen3-embedding` backend (nh3-ml1)
|
|
|
|
- name: Rerank — bge-v2-m3 (TEI, nh3-ml1)
|
|
url: http://10.100.50.80:8013/health
|
|
description: gateway `reranker` backend (nh3-ml1)
|
|
|
|
- name: talk
|
|
url: https://talk.nh3.phasefinal.com:8092/
|
|
description: fleet voice bench
|
|
|
|
# ---- monitoring + backup: a blind monitor is worse than none ----
|
|
- name: Beszel
|
|
url: http://10.250.50.70:8090
|
|
description: the host layer; if this is down we are blind to 18 hosts
|
|
|
|
- name: Backrest
|
|
url: http://10.250.50.70:9898
|
|
description: restic orchestration — a silent backup failure is the expensive kind
|
|
|
|
- name: Dozzle
|
|
url: http://10.250.50.70:8088
|