Files

T

vh 60367b307f servers: add new host dirs, refresh fleet snapshots, orientation doc

Bundles the inventory expansion since 2026-04-22:

- New host dirs (READMEs + ssh-target where dir name doesn't resolve):
    ana-nas, ana-wg, esh-vm-db, nh3-nas, pbs-ana, pbs-nh3.
- New PFI VM snapshots (registered + key-installed 2026-04-23):
    ana-filebot, pfi-ana-webhost, pfi-postgres, pfi-pteradactyl,
    pfi-tacticalrmm, sf-ana-container, sfsrv-ana (system + proxmox).
- servers/irv-ml1: ONBOARDING.md (the first-time setup notes from when
  the host was brought into the fleet) + ssh-target (10.100.79.3 over
  the WG tunnel — name doesn't DNS-resolve from this workstation).
- servers/{ana-ml2,pfi-pve,sf-r630}/README.md: updates to capture BMC
  IPs, the iDRAC vs OS hostname distinction (sf-r630 hardware =
  sfsrv-ana OS), and the ana-ml2 Supermicro BMC (10.250.250.50,
  distinct from the Dell R750xs iDRAC).
- configs/homepage/docker.yaml: irv-ml1-docker provider added so
  homepage auto-discovers irv-ml1's stacks over the WG tunnel.
- docs/orientation.md: narrative fleet overview written for fresh
  Claude sessions — sites, backup architecture, governing principles,
  gotchas, where-to-look guide. Pointed at from CLAUDE.md.

2026-04-24 21:56:46 -07:00

4.3 KiB

Raw Blame History

irv-ml1 onboarding checklist

Steps to bring irv-ml1 fully into the fleet, matching the conventions used by the other hosts.

Already done (repo-side)

servers/irv-ml1/ registered, ssh-target + system-details captured
SSH config alias ssh irv-ml1
CLAUDE.md Servers table + Placement rules updated
configs/restic/irv-ml1/profiles.yaml + README drafted
Homepage card under new Infra - IRV group (IRV-ML1 ping monitor)
Homepage docker.yaml entry staged (commented) for Docker auto-discovery
Homepage settings.yaml — Infra - IRV group added to Infrastructure tab

To do on the host

1. Rename hostname (optional, non-blocking)

ssh -t irv-ml1 'sudo hostnamectl set-hostname irv-ml1 && \
                sudo sed -i "s/ana-ml1/irv-ml1/g" /etc/hosts && \
                hostname && cat /etc/hostname'
scripts/refresh-server-info.sh irv-ml1  # capture new identity

2. Add lkraven to docker group (blocks server_inspect full output)

ssh -t irv-ml1 'sudo usermod -aG docker lkraven'
# then logout + login for group to take effect

3. Install Docker + Dockge (your call on convention adoption)

Once installed, create the fleet-standard directory layout:

ssh -t irv-ml1 'sudo mkdir -p /opt/docker/{compose,conf} && \
                sudo chown -R lkraven:lkraven /opt/docker'

Create the external traefik-net (even if no traefik here yet — stacks reference it as external: true):

ssh -t irv-ml1 'docker network create traefik-net'

4. Expose Docker socket on port 2375 (for homepage auto-discovery)

On irv-ml1:

sudo mkdir -p /etc/systemd/system/docker.service.d
sudo tee /etc/systemd/system/docker.service.d/override.conf > /dev/null <<'EOF'
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// -H tcp://0.0.0.0:2375
EOF
sudo systemctl daemon-reload && sudo systemctl restart docker

Then uncomment the irv-ml1-docker: block in configs/homepage/docker.yaml and push homepage config.

Trust model: 2375 is unencrypted; only safe because it's bound to the WG tunnel interface on the remote side. Confirm your firewall rules prevent exposure of 2375 on any non-WG interface.

5. Deploy Dockge

Standard fleet pattern — see stacks/dockge/ or copy from esh-docker-vm/nh3-docker. Port 5001. After Dockge is up:

Uncomment the IRV-ML1-Dockge entry in configs/homepage/services.yaml
Push homepage config

6. Deploy monitoring agents

Cross-site Beszel + Dozzle agents report to hubs on ana-docker:

beszel-agent → ana-docker's Beszel hub
dozzle-agent → ana-docker's Dozzle hub

Copy compose files from another host (stacks/beszel-agent/, stacks/dozzle-agent/) and adjust HUB_URL / TOKEN / display name for irv-ml1.

7. Set up restic

Follow configs/restic/irv-ml1/README.md exactly:

Create irv-ml1 htpasswd entry on rest-server-nh3 (not rest-server-ana — we target the NH3 side because it's the WG termination endpoint)
Install /etc/restic/restic.env + /etc/restic/password on irv-ml1
restic init
Install resticprofile
Deploy profiles.yaml
resticprofile schedule --all
First manual backup
Verify snapshot appears in Backrest UI on ana-docker

8. Push homepage config to make cards live

After each homepage YAML edit in this repo:

rsync -av --exclude='.env' --exclude='.env.*' \
  configs/homepage/ esh-docker-vm:/opt/docker/conf/homepage/

Verification after everything

# Host reachable through WG
ssh irv-ml1 hostname

# Docker up, 2375 listening
ssh irv-ml1 'docker ps && ss -tlnp | grep 2375'

# Dockge up
curl -fsS http://10.100.79.3:5001/ >/dev/null && echo DOCKGE_OK

# Agents reporting (check from ana-docker)
ssh ana-docker 'docker exec beszel docker logs 2>&1 | grep irv-ml1 | tail'

# Restic timer armed
ssh irv-ml1 'systemctl list-timers "resticprofile*"'

# First restic snapshot visible
# -> Backrest UI → repos → irv-ml1

WireGuard health considerations

Everything this host does depends on the WG tunnel. Worth adding:

Beszel alert on ping loss to 10.100.50.50 (WG peer)
Or a simple cron on ana-docker that pings 10.100.79.3 every 5 min and alerts via mailrise on failure

Not blocking onboarding, but the tunnel is the hidden SPOF for every irv-ml1 service.

4.3 KiB Raw Blame History