vh
5960526c3f
NH3 site visit done: Secure Boot off, iGPU restored as boot VGA, AMT port cabled. - nh3-pve: NVIDIA 580.178.04 (DKMS, open modules) via pve-nvidia-host.yaml. - nh3-ml1 = CT 109 @ 10.100.50.80 via gpu-lxc.yaml; embed-rerank (TEI 1.9.4) deployed with HOST_NAME/HOST_IP labels. - Parity vs esh-ml1 (1,126 texts, 2 runs/host, controls): embed cosine min 0.999993 = own noise floor; overlap@10 1.000 vs MRL-256 positive control 0.684; rerank top-1 1.00, max diff 0.0014 vs floor 0.0020. On-box speed identical within rep spread. - gpu-lxc.yaml: first step upgrades lxc-pve to >= 6.0.0-2 (Proxmox fix #7006). With 6.0.0-1 every docker run in a nesting CT failed on runc 1.5's sysctl reopen; applied on nh3-pve (one package). - pve-nvidia-host.yaml: document that the headers meta drags in the newest kernel (nh3-pve went 6.8.12-11 -> -43 at the next reboot). - Monitoring: Beszel NVIDIA agent + 5 alerts, Kuma #29/#30, Homepage nh3-ml1-docker, Dozzle agent (hub 8 clients). DNS nh3-ml1.nh3.internal. - nh3-pve README: SB/IGFX/driver/kernel state, btmtk oops on -4x kernels, AMT cabled but unreachable on the network. Gateway routing to nh3-ml1 is not changed.
87 lines
4.0 KiB
Markdown
87 lines
4.0 KiB
Markdown
# dozzle
|
|
|
|
Container log viewer. One UI on **ana-docker** aggregates logs from every Docker host via remote agents.
|
|
|
|
**Deploys to:**
|
|
- **ana-docker** (hub) — UI at `http://10.250.50.70:8088`; live compose dir `/opt/docker/compose/dozzle-hub`
|
|
- **fv-ml1** (agent) — listens on `10.251.50.54:7007`
|
|
- **esh-docker-vm** / **vm-esh-nas** (agents) — `10.0.50.45:7007`, `10.0.50.154:7007`
|
|
- **irv-ml1** (agent) — `10.6.110.50:7007` (mesh address)
|
|
- **esh-ml1** (agent, added 2026-09-25) — `10.0.50.80:7007`, pinned `v10.4.1` = the hub's version; compose dir `dozzle-agent`. The host needed an (empty) `traefik-net` network because the compose declares it external.
|
|
- **nh3-ml1** (agent, added 2026-09-25) — `10.100.50.80:7007`, same shape as esh-ml1 (`v10.4.1`, `dozzle-agent`, empty `traefik-net`). Hub `clients` went 7 → 8.
|
|
- **nh3-docker** (agent, cross-site) — `10.100.50.40:7007`. ⚠ **Stopped by hand ~2026-04 (Exited 0) and left that way**; the hub logs a refused connection for it. Revive it or drop it from the list deliberately.
|
|
|
|
⚠ **The hub's agent list silently rots when a host moves.** Until 2026-09-25 it
|
|
still named ana-ml2's `10.250.50.54` and irv-ml1's retired `10.100.79.3`, so fv-ml1
|
|
and irv-ml1 logs had been missing from Dozzle since their moves (2026-09-12 and
|
|
2026-09-06). Nothing reported it. After any re-IP, check the hub's
|
|
`DOZZLE_REMOTE_AGENT`, and check that `docker logs dozzle` shows `"clients":N` equal to
|
|
reachable agents + 1.
|
|
- **corviduo-dev** (agent) — listens on `10.250.50.152:7007`. Compose at `/home/vh/docker/compose/dozzle-agent/` (not `/opt/docker/compose/` — see `servers/corviduo-dev/README.md` for why)
|
|
|
|
One compose.yaml lives on each host. The per-host `.env` sets `COMPOSE_PROFILES=hub` or `COMPOSE_PROFILES=agent` so `docker compose up -d` brings up the right service. On the hub, add every agent to `DOZZLE_REMOTE_AGENT` as a comma-separated list (e.g. `10.251.50.54:7007,10.100.50.40:7007`).
|
|
|
|
## Auth / TLS note
|
|
|
|
Dozzle agents and hub auto-generate mTLS certificates on first run. On the trusted LAN (10.250.0.0/16) the default config is fine. If you ever expose an agent beyond the LAN, generate and pin certificates explicitly per the Dozzle docs (`dozzle generate`). The web UI itself is unauthenticated by default — flip `DOZZLE_AUTH_PROVIDER=simple` and set `DOZZLE_USERNAME`/`DOZZLE_PASSWORD` in the hub `.env` if you want a login gate.
|
|
|
|
## Deploy — hub (ana-docker)
|
|
|
|
```bash
|
|
ssh ana-docker
|
|
sudo mkdir -p /opt/docker/compose/dozzle
|
|
sudo chown $USER /opt/docker/compose/dozzle
|
|
cd /opt/docker/compose/dozzle
|
|
|
|
# scp compose.yaml + .env.example from this workspace, then:
|
|
cp .env.example .env
|
|
# Ensure:
|
|
# COMPOSE_PROFILES=hub
|
|
# DOZZLE_HOSTNAME=ana-docker
|
|
# DOZZLE_REMOTE_AGENT=10.251.50.54:7007
|
|
# DOZZLE_PORT=8088
|
|
|
|
docker compose config
|
|
docker compose up -d
|
|
docker compose logs -f
|
|
```
|
|
|
|
## Deploy — agent (fv-ml1)
|
|
|
|
```bash
|
|
ssh fv-ml1
|
|
sudo mkdir -p /opt/docker/compose/dozzle
|
|
sudo chown $USER /opt/docker/compose/dozzle
|
|
cd /opt/docker/compose/dozzle
|
|
|
|
# scp the same compose.yaml + .env.example, then:
|
|
cp .env.example .env
|
|
# Edit to:
|
|
# COMPOSE_PROFILES=agent
|
|
# DOZZLE_HOSTNAME=fv-ml1
|
|
# DOZZLE_AGENT_PORT=7007
|
|
|
|
docker compose config
|
|
docker compose up -d
|
|
docker compose logs -f
|
|
```
|
|
|
|
## Verify
|
|
|
|
```bash
|
|
# Hub health (from anywhere on LAN)
|
|
curl -s http://10.250.50.70:8088/healthz
|
|
|
|
# Agent reachable from the hub's perspective
|
|
ssh ana-docker 'nc -zv 10.251.50.54 7007'
|
|
|
|
# Open http://10.250.50.70:8088 — you should see two tabs:
|
|
# "ana-docker" (local containers) and "fv-ml1" (via agent).
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
- **Hub shows only local containers:** agent is unreachable. Check firewall rules on fv-ml1 (port 7007 must be open from 10.250.50.70) and that the agent is actually listening (`ss -tlnp | grep 7007`).
|
|
- **Agent keeps restarting:** verify the docker.sock bind mount is read-only and the socket exists.
|
|
- **Certificate mismatch after image upgrade:** delete the `dozzle_data` and `dozzle_agent_data` volumes on both hosts and redeploy to regenerate.
|