Files
esh-pfi-infrastructure/stacks/dozzle/README.md
T
vh 65dc586497 feat(esh-ml1): wire telemetry and monitoring; fix Dozzle's stale agent list
- Beszel: NVIDIA agent (stacks/beszel hosts/esh-ml1.yaml), hub system
  registered with Status/Disk/CPU/Memory/Temperature alerts; GPU samples
  verified (RTX 2000E Ada util, VRAM, power, temp).
- Uptime Kuma: health monitors for the TEI embed (:8001) and rerank (:8013)
  services — the sole backends behind the gateway, so the one exception to
  "seats are out of Kuma's lane". Reward seat excluded (no consumer).
- Homepage: dockerd on tcp/2375 bound to 10.0.50.80 (playbook step);
  esh-ml1-docker added to docker.yaml.
- Dozzle: agent v10.4.1 on esh-ml1. The hub's DOZZLE_REMOTE_AGENT still
  named ana-ml2's 10.250.50.54 and irv-ml1's retired 10.100.79.3; repointed
  to 10.251.50.54 / 10.6.110.50 (fv-ml1 and irv-ml1 logs were missing).
- Documented nh3-dev's Beszel agent failing on NH3 cold start (bind of an
  automounted NAS share); revived by hand, fix still open.
2026-09-25 09:20:36 -07:00

3.8 KiB

dozzle

Container log viewer. One UI on ana-docker aggregates logs from every Docker host via remote agents.

Deploys to:

  • ana-docker (hub) — UI at http://10.250.50.70:8088; live compose dir /opt/docker/compose/dozzle-hub
  • fv-ml1 (agent) — listens on 10.251.50.54:7007
  • esh-docker-vm / vm-esh-nas (agents) — 10.0.50.45:7007, 10.0.50.154:7007
  • irv-ml1 (agent) — 10.6.110.50:7007 (mesh address)
  • esh-ml1 (agent, added 2026-09-25) — 10.0.50.80:7007, pinned v10.4.1 = the hub's version; compose dir dozzle-agent. The host needed an (empty) traefik-net network because the compose declares it external.
  • nh3-docker (agent, cross-site) — 10.100.50.40:7007. ⚠ Stopped by hand ~2026-04 (Exited 0) and left that way; the hub logs a refused connection for it. Revive it or drop it from the list deliberately.

⚠ The hub's agent list silently rots when a host moves. Until 2026-09-25 it still named ana-ml2's 10.250.50.54 and irv-ml1's retired 10.100.79.3, so fv-ml1 and irv-ml1 logs had been missing from Dozzle since their moves (2026-09-12 and 2026-09-06). Nothing reported it. After any re-IP, check the hub's DOZZLE_REMOTE_AGENT, and check that docker logs dozzle shows "clients":N equal to reachable agents + 1.

  • corviduo-dev (agent) — listens on 10.250.50.152:7007. Compose at /home/vh/docker/compose/dozzle-agent/ (not /opt/docker/compose/ — see servers/corviduo-dev/README.md for why)

One compose.yaml lives on each host. The per-host .env sets COMPOSE_PROFILES=hub or COMPOSE_PROFILES=agent so docker compose up -d brings up the right service. On the hub, add every agent to DOZZLE_REMOTE_AGENT as a comma-separated list (e.g. 10.251.50.54:7007,10.100.50.40:7007).

Auth / TLS note

Dozzle agents and hub auto-generate mTLS certificates on first run. On the trusted LAN (10.250.0.0/16) the default config is fine. If you ever expose an agent beyond the LAN, generate and pin certificates explicitly per the Dozzle docs (dozzle generate). The web UI itself is unauthenticated by default — flip DOZZLE_AUTH_PROVIDER=simple and set DOZZLE_USERNAME/DOZZLE_PASSWORD in the hub .env if you want a login gate.

Deploy — hub (ana-docker)

ssh ana-docker
sudo mkdir -p /opt/docker/compose/dozzle
sudo chown $USER /opt/docker/compose/dozzle
cd /opt/docker/compose/dozzle

# scp compose.yaml + .env.example from this workspace, then:
cp .env.example .env
# Ensure:
#   COMPOSE_PROFILES=hub
#   DOZZLE_HOSTNAME=ana-docker
#   DOZZLE_REMOTE_AGENT=10.251.50.54:7007
#   DOZZLE_PORT=8088

docker compose config
docker compose up -d
docker compose logs -f

Deploy — agent (fv-ml1)

ssh fv-ml1
sudo mkdir -p /opt/docker/compose/dozzle
sudo chown $USER /opt/docker/compose/dozzle
cd /opt/docker/compose/dozzle

# scp the same compose.yaml + .env.example, then:
cp .env.example .env
# Edit to:
#   COMPOSE_PROFILES=agent
#   DOZZLE_HOSTNAME=fv-ml1
#   DOZZLE_AGENT_PORT=7007

docker compose config
docker compose up -d
docker compose logs -f

Verify

# Hub health (from anywhere on LAN)
curl -s http://10.250.50.70:8088/healthz

# Agent reachable from the hub's perspective
ssh ana-docker 'nc -zv 10.251.50.54 7007'

# Open http://10.250.50.70:8088 — you should see two tabs:
# "ana-docker" (local containers) and "fv-ml1" (via agent).

Troubleshooting

  • Hub shows only local containers: agent is unreachable. Check firewall rules on fv-ml1 (port 7007 must be open from 10.250.50.70) and that the agent is actually listening (ss -tlnp | grep 7007).
  • Agent keeps restarting: verify the docker.sock bind mount is read-only and the socket exists.
  • Certificate mismatch after image upgrade: delete the dozzle_data and dozzle_agent_data volumes on both hosts and redeploy to regenerate.