Files
esh-pfi-infrastructure/stacks/dozzle/README.md
T
vh 5960526c3f feat(nh3-ml1): second TEI embed/rerank backend live on nh3-pve; parity-verified vs esh-ml1
NH3 site visit done: Secure Boot off, iGPU restored as boot VGA, AMT port cabled.

- nh3-pve: NVIDIA 580.178.04 (DKMS, open modules) via pve-nvidia-host.yaml.
- nh3-ml1 = CT 109 @ 10.100.50.80 via gpu-lxc.yaml; embed-rerank (TEI 1.9.4)
  deployed with HOST_NAME/HOST_IP labels.
- Parity vs esh-ml1 (1,126 texts, 2 runs/host, controls): embed cosine min
  0.999993 = own noise floor; overlap@10 1.000 vs MRL-256 positive control
  0.684; rerank top-1 1.00, max diff 0.0014 vs floor 0.0020. On-box speed
  identical within rep spread.
- gpu-lxc.yaml: first step upgrades lxc-pve to >= 6.0.0-2 (Proxmox fix #7006).
  With 6.0.0-1 every docker run in a nesting CT failed on runc 1.5's sysctl
  reopen; applied on nh3-pve (one package).
- pve-nvidia-host.yaml: document that the headers meta drags in the newest
  kernel (nh3-pve went 6.8.12-11 -> -43 at the next reboot).
- Monitoring: Beszel NVIDIA agent + 5 alerts, Kuma #29/#30, Homepage
  nh3-ml1-docker, Dozzle agent (hub 8 clients). DNS nh3-ml1.nh3.internal.
- nh3-pve README: SB/IGFX/driver/kernel state, btmtk oops on -4x kernels,
  AMT cabled but unreachable on the network.

Gateway routing to nh3-ml1 is not changed.
2026-09-25 15:55:33 -07:00

4.0 KiB

dozzle

Container log viewer. One UI on ana-docker aggregates logs from every Docker host via remote agents.

Deploys to:

  • ana-docker (hub) — UI at http://10.250.50.70:8088; live compose dir /opt/docker/compose/dozzle-hub
  • fv-ml1 (agent) — listens on 10.251.50.54:7007
  • esh-docker-vm / vm-esh-nas (agents) — 10.0.50.45:7007, 10.0.50.154:7007
  • irv-ml1 (agent) — 10.6.110.50:7007 (mesh address)
  • esh-ml1 (agent, added 2026-09-25) — 10.0.50.80:7007, pinned v10.4.1 = the hub's version; compose dir dozzle-agent. The host needed an (empty) traefik-net network because the compose declares it external.
  • nh3-ml1 (agent, added 2026-09-25) — 10.100.50.80:7007, same shape as esh-ml1 (v10.4.1, dozzle-agent, empty traefik-net). Hub clients went 7 → 8.
  • nh3-docker (agent, cross-site) — 10.100.50.40:7007. ⚠ Stopped by hand ~2026-04 (Exited 0) and left that way; the hub logs a refused connection for it. Revive it or drop it from the list deliberately.

⚠ The hub's agent list silently rots when a host moves. Until 2026-09-25 it still named ana-ml2's 10.250.50.54 and irv-ml1's retired 10.100.79.3, so fv-ml1 and irv-ml1 logs had been missing from Dozzle since their moves (2026-09-12 and 2026-09-06). Nothing reported it. After any re-IP, check the hub's DOZZLE_REMOTE_AGENT, and check that docker logs dozzle shows "clients":N equal to reachable agents + 1.

  • corviduo-dev (agent) — listens on 10.250.50.152:7007. Compose at /home/vh/docker/compose/dozzle-agent/ (not /opt/docker/compose/ — see servers/corviduo-dev/README.md for why)

One compose.yaml lives on each host. The per-host .env sets COMPOSE_PROFILES=hub or COMPOSE_PROFILES=agent so docker compose up -d brings up the right service. On the hub, add every agent to DOZZLE_REMOTE_AGENT as a comma-separated list (e.g. 10.251.50.54:7007,10.100.50.40:7007).

Auth / TLS note

Dozzle agents and hub auto-generate mTLS certificates on first run. On the trusted LAN (10.250.0.0/16) the default config is fine. If you ever expose an agent beyond the LAN, generate and pin certificates explicitly per the Dozzle docs (dozzle generate). The web UI itself is unauthenticated by default — flip DOZZLE_AUTH_PROVIDER=simple and set DOZZLE_USERNAME/DOZZLE_PASSWORD in the hub .env if you want a login gate.

Deploy — hub (ana-docker)

ssh ana-docker
sudo mkdir -p /opt/docker/compose/dozzle
sudo chown $USER /opt/docker/compose/dozzle
cd /opt/docker/compose/dozzle

# scp compose.yaml + .env.example from this workspace, then:
cp .env.example .env
# Ensure:
#   COMPOSE_PROFILES=hub
#   DOZZLE_HOSTNAME=ana-docker
#   DOZZLE_REMOTE_AGENT=10.251.50.54:7007
#   DOZZLE_PORT=8088

docker compose config
docker compose up -d
docker compose logs -f

Deploy — agent (fv-ml1)

ssh fv-ml1
sudo mkdir -p /opt/docker/compose/dozzle
sudo chown $USER /opt/docker/compose/dozzle
cd /opt/docker/compose/dozzle

# scp the same compose.yaml + .env.example, then:
cp .env.example .env
# Edit to:
#   COMPOSE_PROFILES=agent
#   DOZZLE_HOSTNAME=fv-ml1
#   DOZZLE_AGENT_PORT=7007

docker compose config
docker compose up -d
docker compose logs -f

Verify

# Hub health (from anywhere on LAN)
curl -s http://10.250.50.70:8088/healthz

# Agent reachable from the hub's perspective
ssh ana-docker 'nc -zv 10.251.50.54 7007'

# Open http://10.250.50.70:8088 — you should see two tabs:
# "ana-docker" (local containers) and "fv-ml1" (via agent).

Troubleshooting

  • Hub shows only local containers: agent is unreachable. Check firewall rules on fv-ml1 (port 7007 must be open from 10.250.50.70) and that the agent is actually listening (ss -tlnp | grep 7007).
  • Agent keeps restarting: verify the docker.sock bind mount is read-only and the socket exists.
  • Certificate mismatch after image upgrade: delete the dozzle_data and dozzle_agent_data volumes on both hosts and redeploy to regenerate.