Initial commit: PFI fleet inventory, stacks, tooling, and backup pipeline

Captures the full workspace state built up to this point: - CLAUDE.md + README.md describing conventions and the four-host fleet (ana-ml2, ana-docker, nh3-docker, esh-docker-vm). - Per-host notes under servers/<host>/ with ssh-target fallback files and latest system-details snapshots (two in-compose credential leaks scrubbed; the upstream compose files still need to move those to .env). - scripts/: server_inspect.sh (read-only remote diagnostic), refresh-server-info.sh (dir-driven discovery + snapshot capture with validation warnings), add-host.sh, sync-stacks.sh (pull compose/conf trees), deploy-stack.sh (push with per-file diff + prompt). - stacks/: canonical compose for backrest, beszel, dozzle, llama-swap, rest-server-ana, rest-server-nh3, vllm-qwen3, plus the retired infinity reference. All use the .env-driven + traefik-net + homepage label pattern. - configs/restic/ana-docker/: first resticprofile config + pre-backup hook (Synapse pg_dump, Seafile mysqldump, Vaultwarden SQLite); templates for the other three hosts to come. - docs/pfi/: general infrastructure reference carried over. - .gitignore excludes .env, stacks-mirror/, and assorted secret/state filenames to prevent re-leaks on later commits.
2026-04-20 14:29:48 -07:00
commit e376d0aec9
55 changed files with 9101 additions and 0 deletions
@@ -0,0 +1,52 @@
+# nh3-docker
+
+General-purpose Docker host for the New Hampshire (nh3) site. Small VM, no GPU. Separate LAN from the Anaheim colo.
+
+## Network
+
+- **LAN IP:** 10.100.50.40
+- **LAN subnet:** 10.100.0.0/16 (NH site)
+- **FQDN:** `nh3-docker.phasefinal.com`
+- **SSH:** standard port 22
+
+## Hardware
+
+- **CPU:** 8 vCPU (QEMU virtual — VM)
+- **RAM:** 7.8 GB
+- **GPUs:** none
+- **Storage:** 125 GB root (ext4) + NFS mounts from `10.100.50.50` (Synology)
+- **OS:** Debian 12 (bookworm), kernel 6.1.x
+- **Docker:** running (daemon socket at `/var/run/docker.sock`; also listens on `:2375` — be aware if firewalling)
+
+## Key paths
+
+| Path | Purpose |
+|------|---------|
+| `/opt/docker/compose/<stack>/` | Compose files (also a git repo) |
+| `/opt/docker/conf/<stack>/` | Config bind mounts (empty as of last inspection) |
+| `/mnt/compose` | NFS — `10.100.50.50:/volume1/compose` |
+| `/mnt/backup` | NFS — `10.100.50.50:/volume1/Backup` |
+
+## Running stacks
+
+| Stack | Port(s) | Notes |
+|-------|---------|-------|
+| adguard | 53 / 853 / 8080 / 8443 / 3000 / 784 | AdGuard Home — primary DNS for the NH site |
+| dockge | 5001 | Docker stack management UI |
+| dozzle-agent | 7007 | Log agent; hub on ana-docker pulls from here |
+| beszel-nh3-docker (agent) | 45876 | Metrics agent; hub on ana-docker |
+| portainer (+ portainer_agent) | 9443 / 8000 / 9001 | Container management UI (ad-hoc, no compose dir under `/opt/docker/compose/`) |
+
+No Traefik / TLS terminator on this host — services are accessed on raw LAN ports. If that changes, mirror the ana-docker pattern (`/opt/docker/conf/traefik-nh3/…` + `anaprod`-style cert resolver).
+
+## Refresh state
+
+```bash
+scripts/refresh-server-info.sh nh3-docker
+```
+
+Latest snapshot: `system-details.txt` (regenerate as needed).
+
+## Placement rule
+
+No GPU → this is the default target for NH-site Docker workloads. GPU-only stacks still go to **ana-ml2**; non-GPU Anaheim-specific services stay on **ana-docker**.
@@ -0,0 +1 @@
+10.100.50.40
@@ -0,0 +1,383 @@
+
+===== HOST =====
+
+Hostname:   nh3-docker.phasefinal.com
+Date:       2026-04-19T22:16:00-07:00
+Uptime:     up 5 weeks, 4 days, 15 hours, 4 minutes
+OS:         Debian GNU/Linux 12 (bookworm)
+Kernel:     6.1.0-21-amd64
+Arch:       x86_64
+
+===== HARDWARE =====
+
+CPU cores:  8
+CPU model:  QEMU Virtual CPU version 2.5+
+MemTotal:   7.8 GB
+MemAvailable: 6.5 GB
+
+===== GPUS =====
+
+nvidia-smi not present (no NVIDIA GPUs or driver not installed)
+
+===== FILESYSTEMS (df) =====
+
+Filesystem      Size  Used Avail Use% Mounted on
+/dev/sda1       125G  6.5G  112G   6% /
+
+===== PERSISTENT MOUNTS (/etc/fstab, non-comment) =====
+
+UUID=dc0e74f7-3973-4501-b04e-1d9a3888f739 /               ext4    errors=remount-ro 0       1
+UUID=cc35604a-fcb3-41e8-9a30-1b85e28eea99 none            swap    sw              0       0
+/dev/sr0        /media/cdrom0   udf,iso9660 user,noauto     0       0
+10.100.50.50:/volume1/compose /mnt/compose nfs defaults 0 0
+10.100.50.50:/volume1/Backup /mnt/backup nfs defaults 0 0
+
+===== TARGETED DATA PATHS =====
+
+/opt  (total: 1.6M)
+  total 12
+  drwxr-xr-x  3 root root 4096 2024-05-30 00:24 .
+  drwxr-xr-x 18 root root 4096 2024-05-14 14:44 ..
+  drwxrwxrwx  5 root root 4096 2024-05-30 19:12 docker
+
+/opt/docker  (total: 1.6M)
+  total 32
+  drwxrwxrwx 5 root    root    4096 2024-05-30 19:12 .
+  drwxr-xr-x 3 root    root    4096 2024-05-30 00:24 ..
+  drwxr-xr-x 6 lkraven lkraven 4096 2026-04-19 01:03 compose
+  drwxr-xr-x 2 lkraven lkraven 4096 2024-05-30 19:11 conf
+  drwxr-xr-x 8 lkraven lkraven 4096 2024-05-31 19:05 .git
+  -rw-r--r-- 1 lkraven lkraven   14 2024-05-30 19:12 .gitignore
+  -rw-r--r-- 1 lkraven lkraven 1059 2024-05-30 00:27 LICENSE
+  -rw-r--r-- 1 lkraven lkraven   60 2024-05-30 00:27 README.md
+
+/opt/docker/compose  (total: 56K)
+  total 28
+  drwxr-xr-x 6 lkraven lkraven 4096 2026-04-19 01:03 .
+  drwxrwxrwx 5 root    root    4096 2024-05-30 19:12 ..
+  drwxr-xr-x 2 lkraven lkraven 4096 2024-05-30 19:14 adguard
+  drwxr-xr-x 2 root    root    4096 2026-04-19 01:03 beszel-nh3-docker
+  drwxr-xr-x 2 lkraven lkraven 4096 2024-05-30 19:14 dockge
+  drwxr-xr-x 2 root    root    4096 2026-04-19 00:44 dozzle-agent
+  -rw-r--r-- 1 lkraven lkraven   25 2024-05-30 00:27 README.md
+
+/opt/docker/conf  (total: 4.0K)
+  total 8
+  drwxr-xr-x 2 lkraven lkraven 4096 2024-05-30 19:11 .
+  drwxrwxrwx 5 root    root    4096 2024-05-30 19:12 ..
+
+/var/lib/docker  (total: 4.0K)
+
+/srv  (total: 4.0K)
+  total 8
+  drwxr-xr-x  2 root root 4096 2024-05-14 14:43 .
+  drwxr-xr-x 18 root root 4096 2024-05-14 14:44 ..
+
+
+===== DOCKER =====
+
+Server: 20.10.24+dfsg1   Client: 20.10.24+dfsg1
+
+----- docker info -----
+Containers:      6 (running 6, paused 0, stopped 0)
+Images:          13
+Runtimes:        map[io.containerd.runc.v2:{runc [] <nil>} io.containerd.runtime.v1.linux:{runc [] <nil>} runc:{runc [] <nil>}]
+Default runtime: runc
+Storage driver:  overlay2
+Root dir:        /var/lib/docker
+Server version:  20.10.24+dfsg1
+
+----- running containers -----
+NAMES             IMAGE                           STATUS                 PORTS
+beszel-agent      henrygd/beszel-agent:latest     Up 21 hours            
+dozzle-agent      amir20/dozzle:latest            Up 22 hours            0.0.0.0:7007->7007/tcp, 8080/tcp
+dockge-dockge-1   louislam/dockge:latest          Up 5 weeks (healthy)   0.0.0.0:5001->5001/tcp, :::5001->5001/tcp
+adguardhome       adguard/adguardhome             Up 5 weeks             67-68/udp, 0.0.0.0:53->53/udp, :::53->53/udp, 443/udp, 0.0.0.0:53->53/tcp, :::53->53/tcp, 853/udp, 0.0.0.0:853->853/tcp, :::853->853/tcp, 3000/udp, 5443/tcp, 0.0.0.0:3000->3000/tcp, 0.0.0.0:784->784/udp, :::3000->3000/tcp, :::784->784/udp, 5443/udp, 6060/tcp, 0.0.0.0:8080->80/tcp, :::8080->80/tcp, 0.0.0.0:8443->443/tcp, :::8443->443/tcp
+portainer_agent   portainer/agent:2.19.4          Up 5 weeks             0.0.0.0:9001->9001/tcp, :::9001->9001/tcp
+portainer         portainer/portainer-ce:latest   Up 5 weeks             0.0.0.0:8000->8000/tcp, :::8000->8000/tcp, 0.0.0.0:9443->9443/tcp, :::9443->9443/tcp, 9000/tcp
+
+----- all containers -----
+NAMES             IMAGE                           STATUS
+beszel-agent      henrygd/beszel-agent:latest     Up 21 hours
+dozzle-agent      amir20/dozzle:latest            Up 22 hours
+dockge-dockge-1   louislam/dockge:latest          Up 5 weeks (healthy)
+adguardhome       adguard/adguardhome             Up 5 weeks
+portainer_agent   portainer/agent:2.19.4          Up 5 weeks
+portainer         portainer/portainer-ce:latest   Up 5 weeks
+
+----- networks -----
+NAME          DRIVER    SCOPE
+bridge        bridge    local
+host          host      local
+none          null      local
+traefik-net   bridge    local
+
+----- networks (external, non-default — worth knowing for compose external: true) -----
+traefik-net
+
+----- named volumes -----
+VOLUME NAME                           DRIVER
+adguard_adguard-confdir               local
+adguard_adguard-workdir               local
+beszel-nh3-docker_beszel_agent_data   local
+dockge_dockge_data                    local
+dozzle-agent_dozzle_agent_data        local
+portainer_data                        local
+
+----- compose projects currently running -----
+adguard
+beszel-nh3-docker
+dockge
+dozzle-agent
+
+===== COMPOSE FILES (/opt/docker/compose/) =====
+
+
+>>> /opt/docker/compose/adguard/docker-compose.yml
+#version: "3"
+services:
+  adguardhome:
+    image: adguard/adguardhome
+    container_name: adguardhome
+    ports:
+      - 53:53/tcp
+      - 53:53/udp
+      - 784:784/udp
+      - 853:853/tcp
+      - 3000:3000/tcp
+      - 8080:80/tcp
+      - 8443:443/tcp
+    volumes:
+      - adguard-workdir:/opt/adguardhome/work
+      - adguard-confdir:/opt/adguardhome/conf
+    restart: unless-stopped
+    labels:
+      - homepage.group=PFI-NH3
+      - homepage.name=AdGuard Home
+      - homepage.icon=si-adguard
+      - homepage.description=DNS
+      - homepage.href=http://10.100.50.40:8080
+      - homepage.widget.type=adguard
+      - homepage.widget.url=http://10.100.50.40:8080
+      - homepage.widget.username=lkraven
+      - homepage.widget.password=${userpass}
+    networks:
+      - tnet
+    env_file:
+      - .env
+volumes:
+  adguard-workdir: null
+  adguard-confdir: null
+networks:
+  tnet:
+    name: traefik-net
+    external: true
+
+>>> /opt/docker/compose/beszel-nh3-docker/compose.yaml
+# Beszel — lightweight server/container monitoring.
+#
+# Hub: single web UI with the SQLite store. Agents: per-host metric collectors
+# that the hub pulls from over SSH.
+#
+# Multi-host layout via compose profiles:
+#   COMPOSE_PROFILES=hub          → hub only        (ana-docker)
+#   COMPOSE_PROFILES=hub,agent    → hub + local agent on the same host
+#   COMPOSE_PROFILES=agent        → agent only      (ana-ml2)
+#
+# The agent uses network_mode: host so it sees real host CPU/mem/net/disk
+# counters rather than container-scoped ones — that's why it can't share
+# the tnet network with the hub.
+#
+# All tunables live in .env — edit that, not this file.
+
+services:
+  beszel:
+    image: henrygd/beszel:${BESZEL_VERSION}
+    container_name: beszel
+    profiles: [hub]
+    restart: unless-stopped
+    ports:
+      - "${BESZEL_PORT}:8090"
+    volumes:
+      - beszel_data:/beszel_data
+    healthcheck:
+      test: ["CMD", "wget", "-qO-", "http://localhost:8090/api/health"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 15s
+    networks:
+      - tnet
+    labels:
+      - homepage.group=PFI-ANA
+      - homepage.name=Beszel
+      - homepage.icon=mdi-chart-line
+      - homepage.description=Server + container monitoring
+      - homepage.href=http://10.250.50.70:${BESZEL_PORT}
+
+  beszel-agent:
+    image: henrygd/beszel-agent:${BESZEL_VERSION}
+    container_name: beszel-agent
+    profiles: [agent]
+    restart: unless-stopped
+    network_mode: host
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - beszel_agent_data:/var/lib/beszel-agent
+    environment:
+      - PORT=${BESZEL_AGENT_PORT:-45876}
+      - KEY=${BESZEL_HUB_KEY}
+      - HUB_URL=${HUB_URL}
+      - TOKEN=${BESZEL_TOKEN}
+      - EXTRA_FILESYSTEMS=${BESZEL_EXTRA_FS:-}
+
+volumes:
+  beszel_data:
+  beszel_agent_data:
+
+networks:
+  tnet:
+    name: traefik-net
+    external: true
+>>> /opt/docker/compose/dockge/compose.yaml
+services:
+  dockge:
+    image: louislam/dockge:latest
+    restart: unless-stopped
+    ports:
+      # Host Port : Container Port
+      - 5001:5001
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock
+      - dockge_data:/app/data
+      - /opt/docker/compose:/opt/docker/compose
+    environment:
+      # Tell Dockge where is your stacks directory
+      - DOCKGE_STACKS_DIR=/opt/docker/compose
+    labels:
+      - homepage.group=PFI-NH3
+      - homepage.name=Dockge
+      - homepage.icon=si-portainer
+      - homepage.description=Docker
+      - homepage.href=http://10.100.50.40:5001
+    networks:
+      - tnet
+volumes:
+  dockge_data: null
+networks:
+  tnet:
+    name: traefik-net
+    external: true
+
+>>> /opt/docker/compose/dozzle-agent/compose.yaml
+# Dozzle — container log viewer.
+#
+# Multi-host layout via compose profiles:
+#   COMPOSE_PROFILES=hub   → runs the web UI (deploy on ana-docker)
+#   COMPOSE_PROFILES=agent → runs the remote agent (deploy on ana-ml2)
+#
+# Same compose.yaml on both servers; per-host `.env` picks the profile.
+#
+# All tunables live in .env — edit that, not this file.
+
+services:
+  dozzle:
+    image: amir20/dozzle:${DOZZLE_VERSION}
+    container_name: dozzle
+    profiles:
+      - hub
+    restart: unless-stopped
+    ports:
+      - ${DOZZLE_PORT}:8080
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - dozzle_data:/data
+    environment:
+      - DOZZLE_HOSTNAME=${DOZZLE_HOSTNAME}
+      - DOZZLE_REMOTE_AGENT=${DOZZLE_REMOTE_AGENT:-}
+      - DOZZLE_AUTH_PROVIDER=${DOZZLE_AUTH_PROVIDER:-none}
+      - DOZZLE_USERNAME=${DOZZLE_USERNAME:-}
+      - DOZZLE_PASSWORD=${DOZZLE_PASSWORD:-}
+    healthcheck:
+      test:
+        - CMD
+        - /dozzle
+        - healthcheck
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 15s
+    networks:
+      - tnet
+    labels:
+      - homepage.group=PFI-ANA
+      - homepage.name=Dozzle
+      - homepage.icon=mdi-text-box-search
+      - homepage.description=Container logs (ana-docker + ana-ml2)
+      - homepage.href=http://10.250.50.70:${DOZZLE_PORT}
+  dozzle-agent:
+    image: amir20/dozzle:${DOZZLE_VERSION}
+    container_name: dozzle-agent
+    profiles:
+      - agent
+    restart: unless-stopped
+    command: agent
+    ports:
+      - ${DOZZLE_AGENT_BIND:-0.0.0.0}:${DOZZLE_AGENT_PORT}:7007
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - dozzle_agent_data:/data
+    environment:
+      - DOZZLE_HOSTNAME=${DOZZLE_HOSTNAME}
+    networks:
+      - tnet
+volumes:
+  dozzle_data: null
+  dozzle_agent_data: null
+networks:
+  tnet:
+    name: traefik-net
+    external: true
+===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) =====
+
+/opt/docker/conf
+
+===== LISTENING PORTS =====
+
+0.0.0.0:111
+0.0.0.0:22
+0.0.0.0:3000
+0.0.0.0:5001
+0.0.0.0:53
+0.0.0.0:7007
+0.0.0.0:8000
+0.0.0.0:8080
+0.0.0.0:8443
+0.0.0.0:853
+0.0.0.0:9001
+0.0.0.0:9443
+[::]:111
+127.0.0.1:40185
+[::]:22
+*:2375
+[::]:3000
+[::]:5001
+[::]:53
+[::]:8000
+[::]:8080
+[::]:8443
+[::]:853
+[::]:9001
+[::]:9443
+
+===== MODEL / HUGGINGFACE CACHES =====
+
+
+===== DOCKER-ADJACENT SYSTEMD SERVICES =====
+
+containerd.service running
+docker.service running
+
+===== DONE =====
+
+Paste the above back into the chat, or pass a path as argv[1] to save.