NH3 site visit done: Secure Boot off, iGPU restored as boot VGA, AMT port cabled. - nh3-pve: NVIDIA 580.178.04 (DKMS, open modules) via pve-nvidia-host.yaml. - nh3-ml1 = CT 109 @ 10.100.50.80 via gpu-lxc.yaml; embed-rerank (TEI 1.9.4) deployed with HOST_NAME/HOST_IP labels. - Parity vs esh-ml1 (1,126 texts, 2 runs/host, controls): embed cosine min 0.999993 = own noise floor; overlap@10 1.000 vs MRL-256 positive control 0.684; rerank top-1 1.00, max diff 0.0014 vs floor 0.0020. On-box speed identical within rep spread. - gpu-lxc.yaml: first step upgrades lxc-pve to >= 6.0.0-2 (Proxmox fix #7006). With 6.0.0-1 every docker run in a nesting CT failed on runc 1.5's sysctl reopen; applied on nh3-pve (one package). - pve-nvidia-host.yaml: document that the headers meta drags in the newest kernel (nh3-pve went 6.8.12-11 -> -43 at the next reboot). - Monitoring: Beszel NVIDIA agent + 5 alerts, Kuma #29/#30, Homepage nh3-ml1-docker, Dozzle agent (hub 8 clients). DNS nh3-ml1.nh3.internal. - nh3-pve README: SB/IGFX/driver/kernel state, btmtk oops on -4x kernels, AMT cabled but unreachable on the network. Gateway routing to nh3-ml1 is not changed.
4.0 KiB
dozzle
Container log viewer. One UI on ana-docker aggregates logs from every Docker host via remote agents.
Deploys to:
- ana-docker (hub) — UI at
http://10.250.50.70:8088; live compose dir/opt/docker/compose/dozzle-hub - fv-ml1 (agent) — listens on
10.251.50.54:7007 - esh-docker-vm / vm-esh-nas (agents) —
10.0.50.45:7007,10.0.50.154:7007 - irv-ml1 (agent) —
10.6.110.50:7007(mesh address) - esh-ml1 (agent, added 2026-09-25) —
10.0.50.80:7007, pinnedv10.4.1= the hub's version; compose dirdozzle-agent. The host needed an (empty)traefik-netnetwork because the compose declares it external. - nh3-ml1 (agent, added 2026-09-25) —
10.100.50.80:7007, same shape as esh-ml1 (v10.4.1,dozzle-agent, emptytraefik-net). Hubclientswent 7 → 8. - nh3-docker (agent, cross-site) —
10.100.50.40:7007. ⚠ Stopped by hand ~2026-04 (Exited 0) and left that way; the hub logs a refused connection for it. Revive it or drop it from the list deliberately.
⚠ The hub's agent list silently rots when a host moves. Until 2026-09-25 it
still named ana-ml2's 10.250.50.54 and irv-ml1's retired 10.100.79.3, so fv-ml1
and irv-ml1 logs had been missing from Dozzle since their moves (2026-09-12 and
2026-09-06). Nothing reported it. After any re-IP, check the hub's
DOZZLE_REMOTE_AGENT, and check that docker logs dozzle shows "clients":N equal to
reachable agents + 1.
- corviduo-dev (agent) — listens on
10.250.50.152:7007. Compose at/home/vh/docker/compose/dozzle-agent/(not/opt/docker/compose/— seeservers/corviduo-dev/README.mdfor why)
One compose.yaml lives on each host. The per-host .env sets COMPOSE_PROFILES=hub or COMPOSE_PROFILES=agent so docker compose up -d brings up the right service. On the hub, add every agent to DOZZLE_REMOTE_AGENT as a comma-separated list (e.g. 10.251.50.54:7007,10.100.50.40:7007).
Auth / TLS note
Dozzle agents and hub auto-generate mTLS certificates on first run. On the trusted LAN (10.250.0.0/16) the default config is fine. If you ever expose an agent beyond the LAN, generate and pin certificates explicitly per the Dozzle docs (dozzle generate). The web UI itself is unauthenticated by default — flip DOZZLE_AUTH_PROVIDER=simple and set DOZZLE_USERNAME/DOZZLE_PASSWORD in the hub .env if you want a login gate.
Deploy — hub (ana-docker)
ssh ana-docker
sudo mkdir -p /opt/docker/compose/dozzle
sudo chown $USER /opt/docker/compose/dozzle
cd /opt/docker/compose/dozzle
# scp compose.yaml + .env.example from this workspace, then:
cp .env.example .env
# Ensure:
# COMPOSE_PROFILES=hub
# DOZZLE_HOSTNAME=ana-docker
# DOZZLE_REMOTE_AGENT=10.251.50.54:7007
# DOZZLE_PORT=8088
docker compose config
docker compose up -d
docker compose logs -f
Deploy — agent (fv-ml1)
ssh fv-ml1
sudo mkdir -p /opt/docker/compose/dozzle
sudo chown $USER /opt/docker/compose/dozzle
cd /opt/docker/compose/dozzle
# scp the same compose.yaml + .env.example, then:
cp .env.example .env
# Edit to:
# COMPOSE_PROFILES=agent
# DOZZLE_HOSTNAME=fv-ml1
# DOZZLE_AGENT_PORT=7007
docker compose config
docker compose up -d
docker compose logs -f
Verify
# Hub health (from anywhere on LAN)
curl -s http://10.250.50.70:8088/healthz
# Agent reachable from the hub's perspective
ssh ana-docker 'nc -zv 10.251.50.54 7007'
# Open http://10.250.50.70:8088 — you should see two tabs:
# "ana-docker" (local containers) and "fv-ml1" (via agent).
Troubleshooting
- Hub shows only local containers: agent is unreachable. Check firewall rules on fv-ml1 (port 7007 must be open from 10.250.50.70) and that the agent is actually listening (
ss -tlnp | grep 7007). - Agent keeps restarting: verify the docker.sock bind mount is read-only and the socket exists.
- Certificate mismatch after image upgrade: delete the
dozzle_dataanddozzle_agent_datavolumes on both hosts and redeploy to regenerate.