vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(esh-ml1): wire telemetry and monitoring; fix Dozzle's stale agent list

Browse Source 
- Beszel: NVIDIA agent (stacks/beszel hosts/esh-ml1.yaml), hub system
  registered with Status/Disk/CPU/Memory/Temperature alerts; GPU samples
  verified (RTX 2000E Ada util, VRAM, power, temp).
- Uptime Kuma: health monitors for the TEI embed (:8001) and rerank (:8013)
  services — the sole backends behind the gateway, so the one exception to
  "seats are out of Kuma's lane". Reward seat excluded (no consumer).
- Homepage: dockerd on tcp/2375 bound to 10.0.50.80 (playbook step);
  esh-ml1-docker added to docker.yaml.
- Dozzle: agent v10.4.1 on esh-ml1. The hub's DOZZLE_REMOTE_AGENT still
  named ana-ml2's 10.250.50.54 and irv-ml1's retired 10.100.79.3; repointed
  to 10.251.50.54 / 10.6.110.50 (fv-ml1 and irv-ml1 logs were missing).
- Documented nh3-dev's Beszel agent failing on NH3 cold start (bind of an
  automounted NAS share); revived by hand, fix still open.
This commit is contained in:
vh
2026-09-25 09:20:36 -07:00
parent 52612cbe96
commit 65dc586497
9 changed files with 165 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/dozzle/README.md
+12 -2
View File
@@ -3,9 +3,19 @@
Container log viewer. One UI on **ana-docker** aggregates logs from every Docker host via remote agents.
**Deploys to:**
- **ana-docker** (hub) — UI at `http://10.250.50.70:8088`
- **ana-docker** (hub) — UI at `http://10.250.50.70:8088`; live compose dir `/opt/docker/compose/dozzle-hub`
- **fv-ml1** (agent) — listens on `10.251.50.54:7007`
- **nh3-docker** (agent, cross-site) — listens on `10.100.50.40:7007`
- **esh-docker-vm** / **vm-esh-nas** (agents) — `10.0.50.45:7007`, `10.0.50.154:7007`
- **irv-ml1** (agent) — `10.6.110.50:7007` (mesh address)
- **esh-ml1** (agent, added 2026-09-25) — `10.0.50.80:7007`, pinned `v10.4.1` = the hub's version; compose dir `dozzle-agent`. The host needed an (empty) `traefik-net` network because the compose declares it external.
- **nh3-docker** (agent, cross-site) — `10.100.50.40:7007`. ⚠ **Stopped by hand ~2026-04 (Exited 0) and left that way**; the hub logs a refused connection for it. Revive it or drop it from the list deliberately.
⚠ **The hub's agent list silently rots when a host moves.** Until 2026-09-25 it
still named ana-ml2's `10.250.50.54` and irv-ml1's retired `10.100.79.3`, so fv-ml1
and irv-ml1 logs had been missing from Dozzle since their moves (2026-09-12 and
2026-09-06). Nothing reported it. After any re-IP, check the hub's
`DOZZLE_REMOTE_AGENT`, and check that `docker logs dozzle` shows `"clients":N` equal to
reachable agents + 1.
- **corviduo-dev** (agent) — listens on `10.250.50.152:7007`. Compose at `/home/vh/docker/compose/dozzle-agent/` (not `/opt/docker/compose/` — see `servers/corviduo-dev/README.md` for why)
One compose.yaml lives on each host. The per-host `.env` sets `COMPOSE_PROFILES=hub` or `COMPOSE_PROFILES=agent` so `docker compose up -d` brings up the right service. On the hub, add every agent to `DOZZLE_REMOTE_AGENT` as a comma-separated list (e.g. `10.251.50.54:7007,10.100.50.40:7007`).