diff --git a/configs/homepage/docker.yaml b/configs/homepage/docker.yaml index 17986ca..f47ca10 100644 --- a/configs/homepage/docker.yaml +++ b/configs/homepage/docker.yaml @@ -21,6 +21,13 @@ ana-ml2-docker: host: 10.250.50.54 port: 2375 +# irv-ml1 — reachable ONLY via WireGuard tunnel, so docker auto-discovery +# traffic traverses the tunnel. If WG drops, homepage will show irv-ml1 +# services as offline until reconnect. +irv-ml1-docker: + host: 10.100.79.3 + port: 2375 + # Example TLS socket (if/when a host moves off plaintext 2375): # ana-pfi-docker: # host: 10.250.50.70 diff --git a/docs/orientation.md b/docs/orientation.md new file mode 100644 index 0000000..670f139 --- /dev/null +++ b/docs/orientation.md @@ -0,0 +1,258 @@ +# Fleet orientation — start here + +Read this first if you're a fresh session joining this project. It +summarizes the fleet, the conventions, the known gotchas, and points +at the deeper docs / memory files that cover each area in detail. + +## What this repo is + +A **reference + management workspace** for PFI infrastructure — a +small multi-site fleet of Proxmox hypervisors, Linux VMs/LXCs, a +bare-metal GPU host, and two NAS boxes. Authoritative copies of +Docker compose files, backup configs, and homepage dashboard config +live **on the hosts themselves**; this repo mirrors them for version +control, planning, and cross-host orchestration. + +Primary file you'll read automatically on session start: + +- `CLAUDE.md` — conventions, servers table, scripts tour, placement rules. +- `~/.claude/projects/.../memory/MEMORY.md` + linked memory files — + durable cross-session facts. + +This file (`docs/orientation.md`) is the narrative overview that ties +them together. + +## Sites + hosts at a glance + +Three physical locations plus one WireGuard-tunneled satellite: + +| Site | Subnet | Physical boxes | Notes | +|---|---|---|---| +| **Anaheim (ANA)** | `10.250.0.0/16` | Dell R750xs (pfi-pve hypervisor, iDRAC 10.250.250.30), Supermicro (ana-ml2 bare-metal GPU, BMC 10.250.250.50), Dell R630 (sfsrv-ana SureFire tenant, iDRAC 10.250.250.110) | Primary production site | +| **NH3** | `10.100.0.0/16` | nh3-pve hypervisor, Synology RS2418+ (`nh3-nas`, 42 TB) | Secondary + DR target | +| **ESH home lab** | `esteban.net`, `10.0.50.0/24` | esh-pve, esh-pve-nas (split-role hypervisors), esh-vm-db (DB VM) | Non-production, shares backup tooling | +| **Irvine (IRV)** | `10.100.79.0/24` (WG tunnel only) | irv-ml1 bare-metal GPU box | Reachable ONLY via WireGuard tunnel from NH3 | + +Full host list with roles: see the **Servers table** in `CLAUDE.md`. + +## Backup architecture (two layers) + +This is the most fleet-impacting system. Every host gets backed up +via one or both layers. + +### Layer 1 — PBS (VM-image, block-level dedupe) + +- **Primary:** `pbs-ana` (VM on pfi-pve, NFS datastore on ana-nas). +- **DR mirror:** `pbs-nh3` (VM on nh3-pve, NFS datastore on nh3-nas Synology). +- **Sync:** pbs-nh3 pulls from pbs-ana at 06:00 daily. One-way. +- **Namespaces:** per-hypervisor (`pfi-pve`, `nh3-pve`, `esh-pve`, + `esh-pve-nas`, `sfsrv-ana`) because VMIDs are NOT globally unique. +- **Scope:** all 5 hypervisors back up to pbs-ana nightly. + +Deployment runbook: `docs/runbooks/pbs-deployment.md` (9 phases, +captured during the 2026-04-22 deployment). + +### Layer 2 — restic (file-level, encrypted, DB-consistent) + +- **Target A:** `rest-server-ana` container on ana-docker. Data dir + NFS-mounted from ana-nas. Used by: ana-docker, ana-ml2, + esh-docker-vm, vm-esh-nas, esh-vm-db. +- **Target B:** `rest-server-nh3` container on nh3-nas (Synology). + Used by: nh3-docker, nh3-dev, irv-ml1. +- **Mode:** both run `--append-only --private-repos` — clients can't + delete their own history, and each client's URL path must match + their HTTP basic-auth user. +- **Scope:** 8 hosts have resticprofile + systemd timers firing at + 01:00 daily. +- **DB-consistent dumps:** hosts with databases have a + `pre-backup.sh` that runs pg_dump/mongodump/sqlite3 .backup into + `/var/lib/restic/stage/` before restic sweeps. + +### Cross-site restic mirror (rsync) + +- **ana-nas → nh3-nas** daily at 04:00 (runs on ana-nas as lkraven). +- **nh3-nas → ana-nas** daily at 05:00 (runs on nh3-nas as root — + rest-server-nh3's container writes files mode 400 owned by admin, + only root can read them on the Synology side). + +Tracked at `configs/rsync/{ana-nas-to-nh3,nh3-nas-to-ana}/`. + +## Governing principles (durable rules) + +Saved in feedback-type memory files; apply by default. + +1. **Backups must not risk production.** If a backup process could + crash the host it's backing up, change the strategy or skip the + backup. Reference: CT 109 (ana-nas) self-backup crash of + 2026-04-23. Memory: `feedback_backups_must_not_risk_production.md`. +2. **DB data lives on local disk, not NFS.** Decision 2026-04-23 + after pfi-postgres migration off `/mnt/db`. New DB deployments + default to local storage; shared NFS for DB data is an anti-pattern + in this environment. Memory: `project_db_migrate_off_nfs.md`. +3. **Use `ssh -t` when the remote runs sudo.** Read-only probes and + piped scripts can stay plain `ssh`. Memory: `feedback_ssh_sudo.md`. +4. **Handle git commands autonomously.** Stage + commit at natural + boundaries without asking; verify risky changes first; never + force-push or push without explicit approval. Memory: + `feedback_git_autonomous.md`. +5. **No Claude attribution in commits.** No `Co-Authored-By`, + "Generated with...", robot emojis, etc. Memory: + `feedback_git_commits.md`. + +## Known gotchas (save yourself time) + +These caught us once; don't let them catch you twice. + +### NFS / storage + +- **ana-nas is `CT 109` (LXC), not `VM 100`.** Persistent confusion + historically — the Proxmox VM name was `PFI-ANA-TRUENAS` for ages + but the host was reprovisioned to Debian, and it's an LXC container + not a VM. Memory: `storage_ana_nas.md`. +- **ana-nas is a SPOF for rest-server-ana + PBS-ANA datastore.** When + multiple fleet services fail together, check `ping 10.250.50.50` + FIRST. Memory: `incident_ana_nas_spof.md`. Recovery runbook: + `docs/runbooks/disaster-recovery.md`. +- **NFS root_squash bites in two directions:** + - **Sudo-root on the client writing to an NFS file** gets mapped to + nobody. Doesn't matter who owns the file. + - **rsync-as-root of NFS-owned data** fails the same way. Workaround + for postgres-on-NFS migration: run rsync as the `postgres` user, + whose uid has legitimate NFS access. +- **ZFS with `casesensitivity=insensitive` + NFSv4 = EACCES on writes.** + PBS-ANA's datastore uses NFSv3 mount to work around this. Don't + "upgrade" to NFSv4 without fixing the dataset first. +- **Synology `.htpasswd` edits must be done as the file owner** (uid + 1000 = lkraven for rest-server-ana), NOT via sudo. sudo-root gets + squashed to nobody and can't read/write the file. +- **Synology Advanced Permissions (syno_acl)** imposes NFSv4 ACLs that + shadow POSIX perms. "Enable Advanced Permissions = off" doesn't + reset it once set. Fix for PBS data: `chmod 777` on the share to + flatten to "Linux mode" (confirm with `synoacltool -get`). +- **DSM Container Manager** doesn't expose `sudo docker` in root's + PATH. Use DSM web UI to restart containers, or find the actual + binary via `find`. +- **scp to DSM** requires `-O` flag (legacy scp protocol) — default + sftp subsystem is disabled. + +### systemd + +- **DSM ships a pre-v247 systemd.** Refuses `Restart=on-failure` on + `Type=oneshot`, doesn't recognize `StartLimitBurst/IntervalSec`, + `systemctl --now` flag absent. Write unit files with plain oneshot + semantics for DSM; the daily timer retry compensates for missed + runs. +- **fstab with bare `defaults`** doesn't auto-retry a failed NFS + mount. If the NFS server wasn't up at boot, `mnt-*.mount` sits in + failed state until manual `systemctl reset-failed` + `systemctl start` + (or simpler: `mount -a`). +- **Ghost files on NFS mount points.** When NFS unmounts while writes + are still landing at the mount point path, the writes go to the + underlying local directory. When NFS remounts, the local ghost is + shadowed. When NFS later unmounts again, the ghost reappears and + breaks whatever was reading that path. See `incident_ana_nas_spof.md` + for the exact case on ana-docker. + +### Naming / identifiers + +- **Hardware vs OS distinction** for servers with iDRAC/BMC: + - `pfi-pve` (Dell R750xs) ↔ `PFI-r750xs-iDRAC` at 10.250.250.30 + - `sfsrv-ana` (Dell R630) ↔ `SF-R630-iDRAC` at 10.250.250.110 + - `ana-ml2` (Supermicro) ↔ `PFI-ANA-ML2 BMC` at 10.250.250.50 + Each `servers//` dir documents the OS side; hardware/iDRAC + entries exist in homepage but are separate logical identities. +- **irv-ml1 was `ana-ml1`** before a physical move; OS hostname still + says `ana-ml1` pending an explicit rename. Doesn't affect services. + +### Workflow + +- **Terminal word-wrap breaks long pasted commands.** Never embed a + 580-char pubkey or long URL inline in a multi-line shell command — + terminal wraps the line, shell captures literal newlines, and the + command fails in bizarre ways. Use `cat` from a file, `printf`, or + a tempfile + scp. +- **zsh interactive mode doesn't parse `#` as comment by default.** + Paste multi-line blocks with comment lines and zsh tries to execute + `#` as a command. Either tell user to `setopt interactivecomments` + or keep blocks comment-free. +- **`ssh -t` with piped stdin fails** — `-t` allocates a tty, stdin + from a pipe can't be a tty. Options: don't use `-t` (give up + interactive sudo prompt), or scp the file first then `ssh -t` for + the sudo bits. + +## Where to look for what + +| Need | Go here | +|---|---| +| Is host X alive? | `servers//system-details.txt` (snapshot). Refresh with `scripts/refresh-server-info.sh `. | +| What does host X run? | `servers//README.md` | +| What's the backup strategy for host X? | `configs/restic//README.md` + `profiles.yaml` | +| How do I recover from host X going down? | `docs/runbooks/disaster-recovery.md` | +| How was PBS deployed? | `docs/runbooks/pbs-deployment.md` | +| What's currently open / in-flight? | `STATUS.md` | +| What do I need to know that isn't in current code? | `MEMORY.md` + the `.md` files it links | +| Why did we do X? | Check memory files + `STATUS.md` session milestones at the bottom | + +## Inventory + automation scripts + +Under `scripts/`: + +- `server_inspect.sh` — pipes over SSH to generate `system-details.txt` + (Docker hosts mostly; works on any Debian-ish Linux). +- `refresh-server-info.sh ` or `all` — fleet-wide refresh + driver. `--validate-only` for dry-run. +- `proxmox_inspect.sh` + `refresh-proxmox-info.sh` — Proxmox-flavored + variant that captures `qm list`, `pct list`, zpool, storage. +- `discover-fortigate.sh`, `discover-unifi.sh`, `discover-gaps.sh` — + network-level inventory diff against `servers/*/`. +- `sync-stacks.sh` / `deploy-stack.sh` — compose mirror pull/push. +- `add-host.sh ` — canonical host registration. + +## Current state (2026-04-23 snapshot) + +- **Host inventory:** ~22 tracked hosts under `servers/` across ANA, + NH3, ESH, IRV. Every host has SSH config alias for convenience. +- **Backup coverage:** 8 restic clients, 5 hypervisors on PBS with + DR mirror, cross-site rsync both directions operational. +- **Recent migrations:** pfi-postgres → local disk (removes ana-nas + cascade); offen sidecars retired on esh-docker-vm (16 GB reclaimed); + irv-ml1 + esh-vm-db fully onboarded. +- **Recent secret rotations:** 6 credentials — all gitea/vaultwarden/ + paperless DB passwords + 3 rest-server htpasswd + repo passphrases. + +Active watchlist (see `STATUS.md` for full detail): + +- pfi-pve backplane repair pending tech visit +- Postgres NFS retirement (block 4+5) pending 24h burn-in +- Tomorrow 01:00: first scheduled restic runs for esh-vm-db + irv-ml1 +- PBS Phase 7-8 (retire legacy vzdump) gated on 2026-04-29 + +## First commands a fresh session should be comfortable running + +```bash +# Fleet validation (read-only, safe) +scripts/refresh-server-info.sh --validate-only all + +# Specific host refresh +scripts/refresh-server-info.sh + +# Look at current fleet state +cat STATUS.md + +# Explore memory +ls /home/lkraven/.claude/projects/-home-lkraven-development-eshpfi-management/memory/ +cat /home/lkraven/.claude/projects/-home-lkraven-development-eshpfi-management/memory/MEMORY.md + +# Probe any host +ssh # alias resolves via ~/.ssh/config +``` + +## When in doubt + +- Read `CLAUDE.md` for conventions and the Servers table. +- Read `STATUS.md` for what's in-flight right now. +- Check `memory/MEMORY.md` for durable facts about the user's + preferences + project history. +- For recovery questions: `docs/runbooks/disaster-recovery.md`. +- Ask. User is experienced — doesn't need hand-holding, but a clear + question is faster than a wrong assumption. diff --git a/servers/ana-filebot/system-details.txt b/servers/ana-filebot/system-details.txt new file mode 100644 index 0000000..fcb9c8b --- /dev/null +++ b/servers/ana-filebot/system-details.txt @@ -0,0 +1,78 @@ + +===== HOST ===== + +Hostname: ana-filebot +Date: 2026-04-23T08:04:59+00:00 +Uptime: up 34 weeks, 4 days, 7 hours, 50 minutes +OS: Debian GNU/Linux 12 (bookworm) +Kernel: 6.8.12-8-pve +Arch: x86_64 + +===== HARDWARE ===== + +CPU cores: 4 +CPU model: Intel(R) Xeon(R) Silver 4310 CPU @ 2.10GHz +MemTotal: 2.0 GB +MemAvailable: 2.0 GB + +===== GPUS ===== + +nvidia-smi not present (no NVIDIA GPUs or driver not installed) + +===== FILESYSTEMS (df) ===== + +Filesystem Size Used Avail Use% Mounted on +ospool/subvol-112-disk-0 80G 813M 80G 1% / +ospool/backup 11T 240M 11T 1% /mnt/ospool-backup +NASPool/backupStore 20T 70G 20T 1% /mnt/backup +NASPool/db 20T 282M 20T 1% /mnt/db +NASPool/docker 20T 384K 20T 1% /mnt/docker +NASPool/pve-VMStorage 22T 2.4T 20T 12% /mnt/pve-VMStorage +NASPool/webdav 20T 256K 20T 1% /mnt/webdav + +===== PERSISTENT MOUNTS (/etc/fstab, non-comment) ===== + + +===== TARGETED DATA PATHS ===== + +/opt (total: 512) + total 9 + drwxr-xr-x 2 root root 2 2023-10-10 13:26 . + drwxr-xr-x 17 root root 21 2025-08-24 00:14 .. + +/srv (total: 512) + total 9 + drwxr-xr-x 2 root root 2 2023-10-10 13:26 . + drwxr-xr-x 17 root root 21 2025-08-24 00:14 .. + + +===== DOCKER ===== + +docker not installed + +===== COMPOSE FILES (/opt/docker/compose/) ===== + +/opt/docker/compose not present + +===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) ===== + +/opt/docker/conf not present + +===== LISTENING PORTS ===== + +*:22 +*:8000 +127.0.0.1:25 +[::1]:25 + +===== MODEL / HUGGINGFACE CACHES ===== + + +===== DOCKER-ADJACENT SYSTEMD SERVICES ===== + +container-getty@1.service running +container-getty@2.service running + +===== DONE ===== + +Paste the above back into the chat, or pass a path as argv[1] to save. diff --git a/servers/ana-ml2/README.md b/servers/ana-ml2/README.md index 45f8bd8..1609ac0 100644 --- a/servers/ana-ml2/README.md +++ b/servers/ana-ml2/README.md @@ -4,11 +4,15 @@ Primary AI inference host for PFI. ## Network -- **LAN IP:** 10.250.50.54 -- **SSH:** standard port 22 +- **LAN IP:** 10.250.50.54 (in-band, OS-side) +- **BMC (OOB):** 10.250.250.50 — Supermicro IPMI web UI + at (homepage card: *PFI-ANA-ML2 BMC*) +- **SSH:** standard port 22 on 10.250.50.54 ## Hardware +- **Chassis:** Supermicro mid-range inferencing server (bare metal, + NOT Dell / not the same box as sf-r630 / sfsrv-ana) - **CPU:** AMD EPYC 9254 24-core (96 threads) - **RAM:** 566 GB - **GPUs:** 2x NVIDIA RTX 6000 Ada Generation (46 GB VRAM each, GPU 0 and GPU 1) diff --git a/servers/ana-nas/README.md b/servers/ana-nas/README.md new file mode 100644 index 0000000..60ae3f6 --- /dev/null +++ b/servers/ana-nas/README.md @@ -0,0 +1,77 @@ +# ana-nas + +Debian NFS/SMB file server at the Anaheim site — **CT 109** (LXC) on +pfi-pve. Serves the fleet's primary storage layer: + +- `/mnt/backup` → restic target for `rest-server-ana` on ana-docker + + backing storage for PBS-ANA's datastore +- `/mnt/docker` → compose/config shares +- `/mnt/webdav` → WebDAV data (if still active) +- Historically `/mnt/db` (retired 2026-04-23 after the DB-off-NFS + migration; pfi-postgres now uses local disk) + +## Network + +- **LAN IP:** 10.250.50.50 +- **SSH:** `ssh ana-nas` (config alias → `lkraven@10.250.50.50`, + key auth). lkraven is a standard user — sudo for root operations. + +## Container context + +This is **CT 109**, not a VM. Bind-mounts into the LXC from pfi-pve's +`ospool` ZFS datasets: + +| Inside CT | Host path on pfi-pve | Purpose | +|---|---|---| +| `/mnt/ospool-backup` | ZFS dataset | ospool snapshots staging | +| `/mnt/backup` | ZFS dataset | rest-server-ana + PBS-ANA data | +| `/mnt/db` | ZFS dataset | **retired 2026-04-23** (postgres moved to local disk) | +| `/mnt/docker` | ZFS dataset | compose files + conf shares | +| `/mnt/pve-VMStorage` | ZFS dataset | alt VM storage pool | +| `/mnt/webdav` | ZFS dataset | WebDAV data | + +Because these are bind-mounts, CT 109's own vzdump backup (LXC rootfs +only) does NOT capture the data payload — the data lives on pfi-pve's +ospool and is captured via the host's own backup chain. + +## Why CT 109 matters (SPOF warning) + +As of 2026-04-23, ana-nas still affects these fleet services if it +goes offline: + +- **rest-server-ana** (on ana-docker) — its data dir is NFS-mounted + from here; an outage kills file-level backups fleet-wide +- **PBS-ANA datastore** — NFS-mounted from here; an outage kills + VM-level backups fleet-wide +- **Cross-site restic rsync source** (ana-nas → nh3-nas daily) + +**Not** affected anymore (as of 2026-04-23 DB migration): + +- pfi-postgres (VM 105) — now on local disk, ana-nas outages + don't cascade into vaultwarden/gitea/paperless/zammad + +Full recovery procedure: `memory/incident_ana_nas_spof.md` + +`docs/runbooks/disaster-recovery.md`. + +## DSM / rescue access + +Under the hood this is vanilla Debian 12 with hand-configured NFS/SMB. +No appliance tooling — share tweaks go through `/etc/exports` + +`systemctl reload nfs-server`. Historical confusion about this box +being a TrueNAS appliance persists in naming (the Proxmox label was +`PFI-ANA-TRUENAS` for a long time) but it is plain Debian. + +## History note + +ID-over-time: this host has been called VM 100, PFI-ANA-TRUENAS, +and now CT 109 / ana-nas. Memory file `storage_ana_nas.md` records +the VM→CT correction that was clarified 2026-04-23 from the vzdump +log output. + +## Refresh + +``` +scripts/refresh-server-info.sh ana-nas +``` + +Snapshot at `system-details.txt`. diff --git a/servers/ana-nas/ssh-target b/servers/ana-nas/ssh-target new file mode 100644 index 0000000..2cd30f3 --- /dev/null +++ b/servers/ana-nas/ssh-target @@ -0,0 +1 @@ +lkraven@10.250.50.50 diff --git a/servers/ana-nas/system-details.txt b/servers/ana-nas/system-details.txt new file mode 100644 index 0000000..1aa093a --- /dev/null +++ b/servers/ana-nas/system-details.txt @@ -0,0 +1,96 @@ + +===== HOST ===== + +Hostname: ana-nas +Date: 2026-04-23T06:53:27+00:00 +Uptime: up 34 weeks, 3 days, 11 hours, 22 minutes +OS: Debian GNU/Linux 12 (bookworm) +Kernel: 6.8.12-8-pve +Arch: x86_64 + +===== HARDWARE ===== + +CPU cores: 4 +CPU model: Intel(R) Xeon(R) Silver 4310 CPU @ 2.10GHz +MemTotal: 2.0 GB +MemAvailable: 1.9 GB + +===== GPUS ===== + +nvidia-smi not present (no NVIDIA GPUs or driver not installed) + +===== FILESYSTEMS (df) ===== + +Filesystem Size Used Avail Use% Mounted on +ospool/subvol-109-disk-0 80G 813M 80G 1% / +ospool/backup 11T 240M 11T 1% /mnt/ospool-backup +NASPool/backupStore 20T 67G 20T 1% /mnt/backup +NASPool/db 20T 282M 20T 1% /mnt/db +NASPool/docker 20T 384K 20T 1% /mnt/docker +NASPool/pve-VMStorage 22T 2.4T 20T 12% /mnt/pve-VMStorage +NASPool/webdav 20T 256K 20T 1% /mnt/webdav + +===== PERSISTENT MOUNTS (/etc/fstab, non-comment) ===== + + +===== TARGETED DATA PATHS ===== + +/opt (total: 512) + total 9 + drwxr-xr-x 2 root root 2 2023-10-10 13:26 . + drwxr-xr-x 17 root root 21 2025-08-24 19:31 .. + +/srv (total: 512) + total 9 + drwxr-xr-x 2 root root 2 2023-10-10 13:26 . + drwxr-xr-x 17 root root 21 2025-08-24 19:31 .. + + +===== DOCKER ===== + +docker not installed + +===== COMPOSE FILES (/opt/docker/compose/) ===== + +/opt/docker/compose not present + +===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) ===== + +/opt/docker/conf not present + +===== LISTENING PORTS ===== + +*:22 +*:9090 +0.0.0.0:111 +0.0.0.0:139 +0.0.0.0:2049 +0.0.0.0:39347 +0.0.0.0:40609 +0.0.0.0:445 +0.0.0.0:48585 +0.0.0.0:49363 +0.0.0.0:57311 +127.0.0.1:25 +[::1]:25 +[::]:111 +[::]:139 +[::]:2049 +[::]:41319 +[::]:445 +[::]:47023 +[::]:50729 +[::]:50951 +[::]:55887 + +===== MODEL / HUGGINGFACE CACHES ===== + + +===== DOCKER-ADJACENT SYSTEMD SERVICES ===== + +container-getty@1.service running +container-getty@2.service running + +===== DONE ===== + +Paste the above back into the chat, or pass a path as argv[1] to save. diff --git a/servers/ana-wg/system-details.txt b/servers/ana-wg/system-details.txt new file mode 100644 index 0000000..92515ce --- /dev/null +++ b/servers/ana-wg/system-details.txt @@ -0,0 +1,73 @@ + +===== HOST ===== + +Hostname: ana-wg +Date: 2026-04-23T08:05:00+00:00 +Uptime: up 34 weeks, 4 days, 7 hours, 50 minutes +OS: Debian GNU/Linux 12 (bookworm) +Kernel: 6.8.12-8-pve +Arch: x86_64 + +===== HARDWARE ===== + +CPU cores: 4 +CPU model: Intel(R) Xeon(R) Silver 4310 CPU @ 2.10GHz +MemTotal: 2.0 GB +MemAvailable: 2.0 GB + +===== GPUS ===== + +nvidia-smi not present (no NVIDIA GPUs or driver not installed) + +===== FILESYSTEMS (df) ===== + +Filesystem Size Used Avail Use% Mounted on +ospool/subvol-113-disk-0 8.0G 623M 7.4G 8% / + +===== PERSISTENT MOUNTS (/etc/fstab, non-comment) ===== + + +===== TARGETED DATA PATHS ===== + +/opt (total: 1.0K) + total 10 + drwxr-xr-x 2 root root 3 2024-10-30 18:08 . + drwxr-xr-x 17 root root 21 2025-08-24 00:14 .. + lrwxrwxrwx 1 root root 29 2024-10-30 18:08 pivpn -> /usr/local/src/pivpn/scripts/ + +/srv (total: 512) + total 9 + drwxr-xr-x 2 root root 2 2023-10-10 13:26 . + drwxr-xr-x 17 root root 21 2025-08-24 00:14 .. + + +===== DOCKER ===== + +docker not installed + +===== COMPOSE FILES (/opt/docker/compose/) ===== + +/opt/docker/compose not present + +===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) ===== + +/opt/docker/conf not present + +===== LISTENING PORTS ===== + +0.0.0.0:22 +127.0.0.1:25 +[::1]:25 +[::]:22 + +===== MODEL / HUGGINGFACE CACHES ===== + + +===== DOCKER-ADJACENT SYSTEMD SERVICES ===== + +container-getty@1.service running +container-getty@2.service running + +===== DONE ===== + +Paste the above back into the chat, or pass a path as argv[1] to save. diff --git a/servers/esh-vm-db/README.md b/servers/esh-vm-db/README.md new file mode 100644 index 0000000..b5e6981 --- /dev/null +++ b/servers/esh-vm-db/README.md @@ -0,0 +1,96 @@ +# esh-vm-db + +ESH-side database VM at `esteban.net`. Hosts **two database servers**: + +- **PostgreSQL 15** on port 5432 — primary consumer is paperless-ngx + on esh-docker-vm (`paperless-ng` user/db). +- **MongoDB** on port 27017 — consumers unknown; document at next + inventory pass. + +Discovered 2026-04-23 during the paperless-ngx password rotation — +paperless's `PAPERLESS_DBHOST: 10.0.50.60` pointed here rather than +the assumed pfi-postgres (VM 105). Up until that point this host was +invisible to inventory. + +## Network + +- **LAN IP:** 10.0.50.60 +- **FQDN:** `esh-vm-db.esteban.net` +- **SSH:** `ssh esh-vm-db` (config alias → `lkraven@10.0.50.60`, + key auth). +- **Listening ports:** 22 (ssh), 111 (rpcbind — NFS mount support), + 5432 (postgres), 27017 (mongo). + +## Hardware + +- **Hypervisor:** presumably esh-pve or esh-pve-nas (to be confirmed + — not yet queried from the PVE side). +- **CPU:** Intel Core i9-13900H (16 cores, mobile) — indicates this + VM lives on a mini-PC / NUC-class ESH host. +- **RAM:** 7.8 GB +- **Root disk:** 250 GB ext4 on `/dev/sda1`, 5.1 GB used (2%). Plenty + of headroom for both DBs. +- **OS:** Debian 12 (bookworm), kernel 6.1.0-23 +- **Uptime:** 3+ weeks at last snapshot. + +## What it runs + +- **PostgreSQL 15 (apt package, `postgresql`)** — data directory + default (`/var/lib/postgresql/15/main` presumably — confirm with + `pg_lsclusters` on a future refresh). Local disk, not NFS — + consistent with the "DBs must live on local disk" principle we + adopted 2026-04-23. +- **MongoDB** — port 27017 bound to 0.0.0.0. DataPath + version to + be captured. +- **No Docker.** `server_inspect.sh` reports `docker not installed` + and `/opt/docker` absent — this host is bare-metal DB service, + not a container host. + +## Storage + +- `/dev/sda1` (250 GB ext4) — DB data + OS +- `/mnt/backup` mounted from `10.0.50.50:/mnt/backup` via NFS + (`defaults`). Purpose TBD — possibly for pg_dump outputs or other + backup staging. Review on next pass. + +## Consumers + +- **paperless-ngx** on esh-docker-vm — uses postgres user + `paperless-ng` against DB `paperless-ng` (confirmed 2026-04-23, + password rotated). +- **MongoDB consumers** — unknown; map these by checking connection + logs or by inventorying the other ESH-side Docker stacks for + `27017`/`mongodb` references. + +## Backup coverage + +**Not yet configured.** Same category as ana-ml2/irv-ml1/esh-vm-db +now (bare-metal DB host without vzdump awareness from the PVE side). + +Next steps: + +- Add to fleet PBS vzdump schedule (its hypervisor backs it up via + PBS-ANA if the hypervisor is onboarded). +- Draft `configs/restic/esh-vm-db/profiles.yaml` for file-level + + per-DB dumps (`pg_dumpall` for postgres, `mongodump` for mongo). + Target: `rest-server-ana` via the ESH → ANA cross-WAN path, same + as other ESH-side restic clients. +- `rest-server-ana`'s `.htpasswd` will need an `esh-vm-db` entry. + +## Refresh state + +```bash +scripts/refresh-server-info.sh esh-vm-db +``` + +## Follow-ups (do soon) + +- Confirm hypervisor (likely esh-pve or esh-pve-nas) — `qm list` + on each and find VM with this name. +- Map MongoDB consumers. +- Configure backup (restic profile + per-DB dumps + vzdump coverage + on the hypervisor side). +- Audit postgres `pg_hba.conf` — is 5432 bound to 0.0.0.0 with + proper host-based auth, or is it over-exposed? paperless connects + from 10.0.50.45 (esh-docker-vm); other hosts shouldn't be able + to reach in. diff --git a/servers/esh-vm-db/ssh-target b/servers/esh-vm-db/ssh-target new file mode 100644 index 0000000..34940dd --- /dev/null +++ b/servers/esh-vm-db/ssh-target @@ -0,0 +1 @@ +lkraven@10.0.50.60 diff --git a/servers/esh-vm-db/system-details.txt b/servers/esh-vm-db/system-details.txt new file mode 100644 index 0000000..e90d856 --- /dev/null +++ b/servers/esh-vm-db/system-details.txt @@ -0,0 +1,80 @@ + +===== HOST ===== + +Hostname: esh-vm-db.esteban.net +Date: 2026-04-23T21:53:16-07:00 +Uptime: up 3 weeks, 2 days, 23 hours, 22 minutes +OS: Debian GNU/Linux 12 (bookworm) +Kernel: 6.1.0-23-amd64 +Arch: x86_64 + +===== HARDWARE ===== + +CPU cores: 16 +CPU model: 13th Gen Intel(R) Core(TM) i9-13900H +MemTotal: 7.8 GB +MemAvailable: 7.2 GB + +===== GPUS ===== + +nvidia-smi not present (no NVIDIA GPUs or driver not installed) + +===== FILESYSTEMS (df) ===== + +Filesystem Size Used Avail Use% Mounted on +/dev/sda1 250G 5.1G 233G 3% / +10.0.50.50:/mnt/backup 92T 11G 92T 1% /mnt/backup + +===== PERSISTENT MOUNTS (/etc/fstab, non-comment) ===== + +UUID=9022f0b6-4299-452a-a4a4-85e1623243ae / ext4 errors=remount-ro 0 1 +UUID=6d90bc64-6f3e-41b9-8295-640ac854e5e5 none swap sw 0 0 +/dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0 +10.0.50.50:/mnt/backup /mnt/backup nfs defaults 0 0 + +===== TARGETED DATA PATHS ===== + +/opt (total: 4.0K) + total 8 + drwxr-xr-x 2 root root 4096 2024-05-20 23:47 . + drwxr-xr-x 18 root root 4096 2024-08-13 17:35 .. + +/srv (total: 2.8M) + total 12 + drwxr-xr-x 3 root root 4096 2024-06-13 10:05 . + drwxr-xr-x 18 root root 4096 2024-08-13 17:35 .. + drwxrwxrwx 4 root root 4096 2024-06-13 10:44 backups + + +===== DOCKER ===== + +docker not installed + +===== COMPOSE FILES (/opt/docker/compose/) ===== + +/opt/docker/compose not present + +===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) ===== + +/opt/docker/conf not present + +===== LISTENING PORTS ===== + +0.0.0.0:111 +0.0.0.0:22 +0.0.0.0:27017 +0.0.0.0:5432 +[::]:111 +[::]:22 +[::]:5432 + +===== MODEL / HUGGINGFACE CACHES ===== + + +===== DOCKER-ADJACENT SYSTEMD SERVICES ===== + +(none matching) + +===== DONE ===== + +Paste the above back into the chat, or pass a path as argv[1] to save. diff --git a/servers/irv-ml1/ONBOARDING.md b/servers/irv-ml1/ONBOARDING.md new file mode 100644 index 0000000..bfdc84c --- /dev/null +++ b/servers/irv-ml1/ONBOARDING.md @@ -0,0 +1,144 @@ +# irv-ml1 onboarding checklist + +Steps to bring irv-ml1 fully into the fleet, matching the conventions +used by the other hosts. + +## Already done (repo-side) + +- [x] `servers/irv-ml1/` registered, ssh-target + system-details captured +- [x] SSH config alias `ssh irv-ml1` +- [x] CLAUDE.md Servers table + Placement rules updated +- [x] `configs/restic/irv-ml1/profiles.yaml` + README drafted +- [x] Homepage card under new `Infra - IRV` group (IRV-ML1 ping monitor) +- [x] Homepage `docker.yaml` entry staged (commented) for Docker auto-discovery +- [x] Homepage `settings.yaml` — `Infra - IRV` group added to Infrastructure tab + +## To do on the host + +### 1. Rename hostname (optional, non-blocking) + +```bash +ssh -t irv-ml1 'sudo hostnamectl set-hostname irv-ml1 && \ + sudo sed -i "s/ana-ml1/irv-ml1/g" /etc/hosts && \ + hostname && cat /etc/hostname' +scripts/refresh-server-info.sh irv-ml1 # capture new identity +``` + +### 2. Add lkraven to docker group (blocks server_inspect full output) + +```bash +ssh -t irv-ml1 'sudo usermod -aG docker lkraven' +# then logout + login for group to take effect +``` + +### 3. Install Docker + Dockge (your call on convention adoption) + +Once installed, create the fleet-standard directory layout: + +```bash +ssh -t irv-ml1 'sudo mkdir -p /opt/docker/{compose,conf} && \ + sudo chown -R lkraven:lkraven /opt/docker' +``` + +Create the external traefik-net (even if no traefik here yet — stacks +reference it as `external: true`): + +```bash +ssh -t irv-ml1 'docker network create traefik-net' +``` + +### 4. Expose Docker socket on port 2375 (for homepage auto-discovery) + +On irv-ml1: + +```bash +sudo mkdir -p /etc/systemd/system/docker.service.d +sudo tee /etc/systemd/system/docker.service.d/override.conf > /dev/null <<'EOF' +[Service] +ExecStart= +ExecStart=/usr/bin/dockerd -H fd:// -H tcp://0.0.0.0:2375 +EOF +sudo systemctl daemon-reload && sudo systemctl restart docker +``` + +Then uncomment the `irv-ml1-docker:` block in +`configs/homepage/docker.yaml` and push homepage config. + +**Trust model:** 2375 is unencrypted; only safe because it's bound to +the WG tunnel interface on the remote side. Confirm your firewall +rules prevent exposure of 2375 on any non-WG interface. + +### 5. Deploy Dockge + +Standard fleet pattern — see `stacks/dockge/` or copy from +esh-docker-vm/nh3-docker. Port 5001. After Dockge is up: + +- Uncomment the `IRV-ML1-Dockge` entry in `configs/homepage/services.yaml` +- Push homepage config + +### 6. Deploy monitoring agents + +Cross-site Beszel + Dozzle agents report to hubs on ana-docker: + +- `beszel-agent` → ana-docker's Beszel hub +- `dozzle-agent` → ana-docker's Dozzle hub + +Copy compose files from another host (`stacks/beszel-agent/`, +`stacks/dozzle-agent/`) and adjust `HUB_URL` / TOKEN / display name +for irv-ml1. + +### 7. Set up restic + +Follow `configs/restic/irv-ml1/README.md` exactly: + +1. Create `irv-ml1` htpasswd entry on rest-server-nh3 (not rest-server-ana + — we target the NH3 side because it's the WG termination endpoint) +2. Install `/etc/restic/restic.env` + `/etc/restic/password` on irv-ml1 +3. `restic init` +4. Install resticprofile +5. Deploy `profiles.yaml` +6. `resticprofile schedule --all` +7. First manual backup +8. Verify snapshot appears in Backrest UI on ana-docker + +### 8. Push homepage config to make cards live + +After each homepage YAML edit in this repo: + +```bash +rsync -av --exclude='.env' --exclude='.env.*' \ + configs/homepage/ esh-docker-vm:/opt/docker/conf/homepage/ +``` + +## Verification after everything + +```bash +# Host reachable through WG +ssh irv-ml1 hostname + +# Docker up, 2375 listening +ssh irv-ml1 'docker ps && ss -tlnp | grep 2375' + +# Dockge up +curl -fsS http://10.100.79.3:5001/ >/dev/null && echo DOCKGE_OK + +# Agents reporting (check from ana-docker) +ssh ana-docker 'docker exec beszel docker logs 2>&1 | grep irv-ml1 | tail' + +# Restic timer armed +ssh irv-ml1 'systemctl list-timers "resticprofile*"' + +# First restic snapshot visible +# -> Backrest UI → repos → irv-ml1 +``` + +## WireGuard health considerations + +Everything this host does depends on the WG tunnel. Worth adding: + +- Beszel alert on ping loss to 10.100.50.50 (WG peer) +- Or a simple cron on ana-docker that pings 10.100.79.3 every 5 min + and alerts via mailrise on failure + +Not blocking onboarding, but the tunnel is the hidden SPOF for every +irv-ml1 service. diff --git a/servers/irv-ml1/ssh-target b/servers/irv-ml1/ssh-target new file mode 100644 index 0000000..97a4a69 --- /dev/null +++ b/servers/irv-ml1/ssh-target @@ -0,0 +1 @@ +lkraven@10.100.79.3 diff --git a/servers/nh3-nas/README.md b/servers/nh3-nas/README.md new file mode 100644 index 0000000..6982208 --- /dev/null +++ b/servers/nh3-nas/README.md @@ -0,0 +1,75 @@ +# nh3-nas + +Synology RS2418+ at the NH3 site (`PFI-NH3-NAS`, `10.100.50.50`). +Primary role is fleet storage: VM-image hosting for nh3-pve, the +NH3-side restic target, and as of 2026-04-22 the PBS-NH3 datastore +backend. + +## Network + +- **LAN IP:** 10.100.50.50 +- **DSM web UI:** +- **SSH:** `ssh nh3-nas` (config alias → `syncuser@10.100.50.50`, + key auth). The `syncuser` account was created specifically for + automation/tooling access; `admin` retained as the DSM primary. + +## Hardware + +- **Model:** Synology RS2418+ (denverton platform) +- **CPU:** Intel Atom C3538 @ 2.10 GHz, 4 cores +- **RAM:** 31.3 GB +- **Kernel:** `4.4.302+` — DSM's custom kernel, not a vanilla Debian + base. Implications: very old bash features only (4.x), BusyBox-ish + userland for some tools, syno_acl layer on Btrfs. +- **Volume:** `/volume1`, Btrfs on `cachedev_0` (SSD-cached LVM). + 42 TB total, 27 TB used (64%), 16 TB free. + +## Services running on the box + +- **DSM built-in NFS server** — exports under `/etc/exports` managed + via DSM Shared Folder UI. Active exports include: + - `/volume1/Shared`, `/volume1/Media`, `/volume1/NetBackup`, + `/volume1/Backup`, `/volume1/compose`, `/volume1/VMStorage`, + `/volume1/devstor` (legacy home-lab exports; `all_squash`) + - `/volume1/pbs` — dedicated share for PBS-NH3's datastore mount. + Linux/POSIX mode 777, no syno_acl, `no_root_squash + no_all_squash`. + Do NOT enable Advanced Permissions on this share — see + `docs/runbooks/pbs-deployment.md` Phase 5.3 for the history. +- **rest-server-nh3** (Docker via DSM ContainerManager) — restic + HTTP endpoint on port 8000, writes under `/volume1/Backup`. Serves + nh3-docker and nh3-dev restic clients. + +## What backs up to it + +- nh3-docker + nh3-dev resticprofile timers (via rest-server-nh3) +- PBS-NH3 datastore (`backups-mirror`), which receives the nightly + sync pull from PBS-ANA + +## What backs up FROM it + +Not currently backed up itself — the DSM side is the source-of-truth +for its own config. Future work: cross-site rsync of +`/volume1/Backup/restic/...` to the Ana NAS (`10.250.50.50`), blocked +previously on SSH-access to this host (now resolved with `syncuser`). + +## DSM-specific gotchas + +- **Docker:** runs via DSM's ContainerManager package, not a native + `docker` CLI. `docker ps` over SSH returns empty; use + `sudo synopkg list | grep -i container` and + `sudo docker ...` (DSM aliases the binary into root's PATH). +- **Home directories:** live under `/var/services/homes//` (not + `/home/`). SSH public keys go in `~/.ssh/authorized_keys` + which maps to the DSM home. Home dir permission must be 755 (not + 700) or sshd rejects keys silently after DSM updates reset it. +- **Btrfs mount options** include `synoacl` — POSIX permissions are + projected through Synology's ACL layer. For share permissions to + behave as plain POSIX, either turn off "Advanced Permissions" in + DSM *and* flatten with `chmod 777` (which converts the share to + "Linux mode", `synoacltool -get` will confirm), or grant explicit + ACL entries with `synoacltool -add`. + +## Refresh state + +Snapshot captured via `scripts/refresh-server-info.sh nh3-nas` — +updates `system-details.txt` alongside this README. diff --git a/servers/nh3-nas/ssh-target b/servers/nh3-nas/ssh-target new file mode 100644 index 0000000..62d1c3b --- /dev/null +++ b/servers/nh3-nas/ssh-target @@ -0,0 +1 @@ +syncuser@10.100.50.50 diff --git a/servers/nh3-nas/system-details.txt b/servers/nh3-nas/system-details.txt new file mode 100644 index 0000000..57959f4 --- /dev/null +++ b/servers/nh3-nas/system-details.txt @@ -0,0 +1,126 @@ + +===== HOST ===== + +Hostname: PFI-NH3-NAS +Date: 2026-04-22T23:46:21-07:00 +Uptime: up 3 days, 9 hours, 26 minutes +Kernel: 4.4.302+ +Arch: x86_64 + +===== HARDWARE ===== + +CPU cores: 4 +CPU model: Intel(R) Atom(TM) CPU C3538 @ 2.10GHz +MemTotal: 31.3 GB +MemAvailable: 29.5 GB + +===== GPUS ===== + +nvidia-smi not present (no NVIDIA GPUs or driver not installed) + +===== FILESYSTEMS (df) ===== + +Filesystem Size Used Avail Use% Mounted on +/dev/md0 2.3G 1.5G 749M 67% / +/dev/mapper/cachedev_0 42T 27T 16T 64% /volume1 +/dev/loop0 27M 767K 24M 4% /tmp/SynologyAuthService +/dev/mapper/cachedev_0 42T 27T 16T 64% /volume1/@appdata/ContainerManager/all_shares/Backup +/dev/mapper/cachedev_0 42T 27T 16T 64% /volume1/@appdata/ContainerManager/all_shares/compose +/dev/mapper/cachedev_0 42T 27T 16T 64% /volume1/@appdata/ContainerManager/all_shares/devstor +/dev/mapper/cachedev_0 42T 27T 16T 64% /volume1/@appdata/ContainerManager/all_shares/docker +/dev/mapper/cachedev_0 42T 27T 16T 64% /volume1/@appdata/ContainerManager/all_shares/homes +/dev/mapper/cachedev_0 42T 27T 16T 64% /volume1/@appdata/ContainerManager/all_shares/Media +/dev/mapper/cachedev_0 42T 27T 16T 64% /volume1/@appdata/ContainerManager/all_shares/Music +/dev/mapper/cachedev_0 42T 27T 16T 64% /volume1/@appdata/ContainerManager/all_shares/NetBackup +/dev/mapper/cachedev_0 42T 27T 16T 64% /volume1/@appdata/ContainerManager/all_shares/Shared +/dev/mapper/cachedev_0 42T 27T 16T 64% /volume1/@appdata/ContainerManager/all_shares/VMStorage +/dev/mapper/cachedev_0 42T 27T 16T 64% /volume1/@appdata/ContainerManager/all_shares/web +/dev/mapper/cachedev_0 42T 27T 16T 64% /volume1/@appdata/ContainerManager/all_shares/web_packages +/dev/mapper/cachedev_0 42T 27T 16T 64% /volume1/@appdata/ContainerManager/all_shares/pbs + +===== PERSISTENT MOUNTS (/etc/fstab, non-comment) ===== + +none /proc proc defaults 0 0 +/dev/root / ext4 defaults 1 1 +/dev/mapper/cachedev_0 /volume1 btrfs auto_reclaim_space,ssd,synoacl,relatime,nodev 0 0 + +===== TARGETED DATA PATHS ===== + +/opt (total: 4.0K) + + +===== DOCKER ===== + +docker not installed + +===== COMPOSE FILES (/opt/docker/compose/) ===== + +/opt/docker/compose not present + +===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) ===== + +/opt/docker/conf not present + +===== LISTENING PORTS ===== + +0.0.0.0:111 +0.0.0.0:139 +0.0.0.0:161 +0.0.0.0:2022 +0.0.0.0:2049 +0.0.0.0:22 +0.0.0.0:4045 +0.0.0.0:443 +0.0.0.0:445 +0.0.0.0:5000 +0.0.0.0:5001 +0.0.0.0:5357 +0.0.0.0:662 +0.0.0.0:80 +0.0.0.0:8000 +0.0.0.0:873 +0.0.0.0:892 +10.100.250.50:3260 +10.100.50.50:3260 +127.0.0.1:33304 +127.0.0.1:512 +127.0.0.1:5432 +:::111 +:::139 +:::161 +:::2022 +:::2049 +:::22 +:::22000 +:::3261 +:::3263 +:::3264 +:::3265 +:::4045 +:::443 +:::445 +:::5000 +:::5001 +:::5357 +:::6281 +:::662 +:::80 +:::8000 +:::8384 +:::873 +:::892 +fe80::211:32ff:fe9:3260 +fe80::ee0d:9aff:fe:3260 + +===== MODEL / HUGGINGFACE CACHES ===== + + +===== DOCKER-ADJACENT SYSTEMD SERVICES ===== + +pkg-ContainerManager-dockerd.service running +pkg-ContainerManager-event-watcherd.service running +pkg-ContainerManager-termd.service running + +===== DONE ===== + +Paste the above back into the chat, or pass a path as argv[1] to save. diff --git a/servers/pbs-ana/README.md b/servers/pbs-ana/README.md new file mode 100644 index 0000000..1682045 --- /dev/null +++ b/servers/pbs-ana/README.md @@ -0,0 +1,87 @@ +# pbs-ana + +Proxmox Backup Server — **fleet primary**. VM on pfi-pve with an +NFS-mounted datastore on the Ana NAS. All 5 hypervisors (pfi-pve, +nh3-pve, esh-pve, esh-pve-nas, sfsrv-ana) back up here; one-way sync +to pbs-nh3 at 06:00 daily. + +## Network + +- **LAN IP:** 10.250.50.90 +- **FQDN:** `pbs-ana.phasefinal.com` +- **PBS web UI:** +- **SSH:** `ssh pbs-ana` (config alias → `lkraven@10.250.50.90`). + Root ops via sudo from lkraven. +- **Homepage card:** *PBS-ANA* in `Infra - ANA` group. + +## Hardware (VM) + +- **Hypervisor:** pfi-pve +- **vCPU:** 4 (QEMU) +- **RAM:** 7.8 GB +- **Root disk:** `/dev/sda1` 31 GB ext4 — OS only (2 GB used) +- **Datastore:** **NFS from ana-nas** at `/mnt/pbs-datastore` → + `10.250.50.50:/mnt/backup/pbs-ana`, 20 TB (currently 336 GB used) + +## NFS details (the ZFS-case-insensitivity gotcha) + +The ZFS dataset backing `/mnt/backup` on ana-nas is +`casesensitivity=insensitive`, which breaks NFSv4 writes (EACCES). +Mount must be **NFSv3** — that's what `/etc/fstab` already uses: + +``` +10.250.50.50:/mnt/backup/pbs-ana /mnt/pbs-datastore nfs \ + defaults,_netdev,bg,hard,timeo=600,retrans=2,vers=3 0 0 +``` + +Don't upgrade this to vers=4 without fixing the dataset. See +`docs/runbooks/pbs-deployment.md` Phase 0.5 for the full history. + +## What runs here + +- `proxmox-backup-proxy` — HTTPS API + web UI (port 8007) +- `proxmox-backup` — backend service +- Scheduled verify job: weekly Saturday 23:00 (`ignore-verified=true`, + `outdated-after=30 days`) +- Garbage collection: default weekly + +## Namespaces + +Per-hypervisor namespaces to prevent VMID collisions across PVE +hosts (VMIDs are NOT globally unique): + +- `pfi-pve` +- `nh3-pve` +- `esh-pve` +- `esh-pve-nas` +- `sfsrv-ana` + +## API token (for hypervisor clients) + +- `root@pam!fleet-vzdump` — stored in password manager; referenced + by each hypervisor's `/etc/pve/storage.cfg` `pbs-ana` stanza. + +## Sync to DR + +One-way pull: **pbs-nh3** pulls from here at 06:00 daily. pbs-ana +does not push; all coordination is on the DR side. + +## Dependencies / blast radius + +- **Hard dependency on ana-nas** (CT 109). Datastore NFS goes dead + if ana-nas crashes. Fleet-wide vzdump + verify + sync all fail + until NFS recovers. See `memory/incident_ana_nas_spof.md`. +- **Self-backup excluded** from scheduled vzdump — backing up a + PBS VM to itself is a circular dependency. Config-only backup + could be captured via file-level restic if needed; not currently + set up (low priority — the datastore is on NFS, OS is + reinstallable in ~30 min). + +## Refresh + +``` +scripts/refresh-server-info.sh pbs-ana +``` + +Snapshot at `system-details.txt`. Full deployment history: +`docs/runbooks/pbs-deployment.md`. diff --git a/servers/pbs-ana/ssh-target b/servers/pbs-ana/ssh-target new file mode 100644 index 0000000..ff90f9e --- /dev/null +++ b/servers/pbs-ana/ssh-target @@ -0,0 +1 @@ +lkraven@10.250.50.90 diff --git a/servers/pbs-ana/system-details.txt b/servers/pbs-ana/system-details.txt new file mode 100644 index 0000000..5a9070c --- /dev/null +++ b/servers/pbs-ana/system-details.txt @@ -0,0 +1,84 @@ + +===== HOST ===== + +Hostname: pbs-ana.phasefinal.com +Date: 2026-04-23T21:55:34-07:00 +Uptime: up 7 hours, 26 minutes +OS: Debian GNU/Linux 12 (bookworm) +Kernel: 6.1.0-44-amd64 +Arch: x86_64 + +===== HARDWARE ===== + +CPU cores: 4 +CPU model: QEMU Virtual CPU version 2.5+ +MemTotal: 7.8 GB +MemAvailable: 7.4 GB + +===== GPUS ===== + +nvidia-smi not present (no NVIDIA GPUs or driver not installed) + +===== FILESYSTEMS (df) ===== + +Filesystem Size Used Avail Use% Mounted on +/dev/sda1 31G 2.2G 27G 8% / +10.250.50.50:/mnt/backup/pbs-ana 20T 336G 19T 2% /mnt/pbs-datastore + +===== PERSISTENT MOUNTS (/etc/fstab, non-comment) ===== + +UUID=6a2b24f3-4134-40ed-9b64-f708dcb77cef / ext4 errors=remount-ro 0 1 +UUID=bfe22def-87a6-4ca8-8f06-19bbf148559c none swap sw 0 0 +/dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0 +10.250.50.50:/mnt/backup/pbs-ana /mnt/pbs-datastore nfs defaults,_netdev,bg,hard,timeo=600,retrans=2,vers=3 0 0 + +===== TARGETED DATA PATHS ===== + +/opt (total: 4.0K) + total 8 + drwxr-xr-x 2 root root 4096 2026-04-21 21:22 . + drwxr-xr-x 18 root root 4096 2026-04-21 21:25 .. + +/srv (total: 4.0K) + total 8 + drwxr-xr-x 2 root root 4096 2026-04-21 21:22 . + drwxr-xr-x 18 root root 4096 2026-04-21 21:25 .. + + +===== DOCKER ===== + +docker not installed + +===== COMPOSE FILES (/opt/docker/compose/) ===== + +/opt/docker/compose not present + +===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) ===== + +/opt/docker/conf not present + +===== LISTENING PORTS ===== + +0.0.0.0:111 +0.0.0.0:22 +0.0.0.0:36203 +0.0.0.0:36419 +[::]:111 +[::1]:25 +127.0.0.1:25 +127.0.0.1:82 +[::]:22 +[::]:35545 +[::]:56445 +*:8007 + +===== MODEL / HUGGINGFACE CACHES ===== + + +===== DOCKER-ADJACENT SYSTEMD SERVICES ===== + +(none matching) + +===== DONE ===== + +Paste the above back into the chat, or pass a path as argv[1] to save. diff --git a/servers/pbs-nh3/README.md b/servers/pbs-nh3/README.md new file mode 100644 index 0000000..84676f0 --- /dev/null +++ b/servers/pbs-nh3/README.md @@ -0,0 +1,99 @@ +# pbs-nh3 + +Proxmox Backup Server — **DR mirror**. VM on nh3-pve with an +NFS-mounted datastore on the NH3 Synology. Pulls nightly from pbs-ana +(the primary) at 06:00. + +## Network + +- **LAN IP:** 10.100.50.90 +- **FQDN:** `pbs-nh3.phasefinal.com` +- **PBS web UI:** +- **SSH:** `ssh pbs-nh3` (config alias → `lkraven@10.100.50.90`). + Root ops via sudo from lkraven. +- **Homepage card:** *PBS-NH3* in `Infra - NH3` group. + +## Hardware (VM) + +- **Hypervisor:** nh3-pve +- **vCPU:** 4 (QEMU) +- **RAM:** 7.8 GB +- **Root disk:** `/dev/sda1` 31 GB ext4 — OS only (2 GB used) +- **Datastore:** **NFS from nh3-nas** at `/mnt/pbs-datastore` → + `10.100.50.50:/volume1/pbs`. Volume size 42 TB; 27 TB used (64%). + +## NFS details (the Synology ACL gotcha) + +Synology's `/volume1/pbs` share was initially created with "Advanced +Permissions" which imposes a `syno_acl` that denies writes to +non-admin UIDs (even the `backup` uid PBS runs as). Fix was to flatten +the share to pure POSIX mode 777 so the mount becomes "Linux mode" +(`synoacltool -get` confirms). `no_root_squash + no_all_squash` also +set on the NFS export. + +Mount uses NFSv3 (same pattern as pbs-ana, for consistency more than +necessity — Synology's Btrfs doesn't have the ZFS-case-insensitivity +issue): + +``` +10.100.50.50:/volume1/pbs /mnt/pbs-datastore nfs \ + defaults,_netdev,bg,hard,timeo=600,retrans=2,vers=3 0 0 +``` + +Full history: `docs/runbooks/pbs-deployment.md` Phase 5.3. + +## What runs here + +- `proxmox-backup-proxy` — HTTPS API + web UI (port 8007) +- `proxmox-backup` — backend service +- Scheduled verify job: weekly Sunday 12:00 (after nightly sync + completes), `ignore-verified=true`, `outdated-after=30 days` +- Scheduled sync from pbs-ana: daily at 06:00 UTC + +## Remote config + +- Remote name: `pbs-ana` +- Remote host: `10.250.50.90:8007` +- Auth: `root@pam!fleet-vzdump` token (same token as clients use on + the primary — shared here for simplicity; could be split to a + sync-only read-only token) +- Cert fingerprint: in password manager under "pbs-ana fingerprint" + +## Namespaces + +Namespaces are auto-replicated from pbs-ana via the sync job — you do +NOT need to pre-create them on the mirror. Expected namespaces after +first sync: same 5 as pbs-ana (`pfi-pve`, `nh3-pve`, `esh-pve`, +`esh-pve-nas`, `sfsrv-ana`). + +## Dependencies / blast radius + +- **Hard dependency on nh3-nas** (Synology). Datastore goes dead if + the Synology crashes; sync fails, verify fails, restore ops fail. + But: ANA primary keeps operating fine, so the fleet's backup SLA + is preserved — only DR redundancy is temporarily lost. +- **Sync source**: pbs-ana going offline means no new snapshots pull + here, but existing ones remain restorable. Sync resumes + automatically once pbs-ana recovers. + +## DR failover (hypothetical) + +If pbs-ana is permanently lost and we needed to restore fleet VMs +from pbs-nh3: + +1. Re-point each PVE host's `pbs-ana` storage entry at pbs-nh3 + (`pve-datacenter.cfg` → change `server` to 10.100.50.90) +2. Fingerprint + token still work (same token) +3. Cross-WAN restore is slower than local but functional + +Documented as future work — not tested. Run a trial restore of a +small CT periodically to validate. + +## Refresh + +``` +scripts/refresh-server-info.sh pbs-nh3 +``` + +Snapshot at `system-details.txt`. Full deployment history: +`docs/runbooks/pbs-deployment.md`. diff --git a/servers/pbs-nh3/ssh-target b/servers/pbs-nh3/ssh-target new file mode 100644 index 0000000..bffba21 --- /dev/null +++ b/servers/pbs-nh3/ssh-target @@ -0,0 +1 @@ +lkraven@10.100.50.90 diff --git a/servers/pbs-nh3/system-details.txt b/servers/pbs-nh3/system-details.txt new file mode 100644 index 0000000..ca90c9e --- /dev/null +++ b/servers/pbs-nh3/system-details.txt @@ -0,0 +1,84 @@ + +===== HOST ===== + +Hostname: pbs-nh3.phasefinal.com +Date: 2026-04-23T21:56:13-07:00 +Uptime: up 22 hours, 41 minutes +OS: Debian GNU/Linux 12 (bookworm) +Kernel: 6.1.0-44-amd64 +Arch: x86_64 + +===== HARDWARE ===== + +CPU cores: 4 +CPU model: QEMU Virtual CPU version 2.5+ +MemTotal: 7.8 GB +MemAvailable: 7.3 GB + +===== GPUS ===== + +nvidia-smi not present (no NVIDIA GPUs or driver not installed) + +===== FILESYSTEMS (df) ===== + +Filesystem Size Used Avail Use% Mounted on +/dev/sda1 31G 2.0G 27G 7% / +10.100.50.50:/volume1/pbs 42T 27T 16T 64% /mnt/pbs-datastore + +===== PERSISTENT MOUNTS (/etc/fstab, non-comment) ===== + +UUID=34a61420-a414-4789-8a3d-065a8905eb11 / ext4 errors=remount-ro 0 1 +UUID=518afc03-e17d-4e1a-a5d4-84b96bee1c60 none swap sw 0 0 +/dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0 +10.100.50.50:/volume1/pbs /mnt/pbs-datastore nfs defaults,_netdev,bg,hard,timeo=600,retrans=2,vers=3 0 0 + +===== TARGETED DATA PATHS ===== + +/opt (total: 4.0K) + total 8 + drwxr-xr-x 2 root root 4096 2026-04-22 21:04 . + drwxr-xr-x 18 root root 4096 2026-04-22 21:06 .. + +/srv (total: 4.0K) + total 8 + drwxr-xr-x 2 root root 4096 2026-04-22 21:04 . + drwxr-xr-x 18 root root 4096 2026-04-22 21:06 .. + + +===== DOCKER ===== + +docker not installed + +===== COMPOSE FILES (/opt/docker/compose/) ===== + +/opt/docker/compose not present + +===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) ===== + +/opt/docker/conf not present + +===== LISTENING PORTS ===== + +0.0.0.0:111 +0.0.0.0:22 +0.0.0.0:41669 +0.0.0.0:46653 +[::]:111 +[::1]:25 +127.0.0.1:25 +127.0.0.1:82 +[::]:22 +[::]:37707 +[::]:54621 +*:8007 + +===== MODEL / HUGGINGFACE CACHES ===== + + +===== DOCKER-ADJACENT SYSTEMD SERVICES ===== + +(none matching) + +===== DONE ===== + +Paste the above back into the chat, or pass a path as argv[1] to save. diff --git a/servers/pfi-ana-webhost/system-details.txt b/servers/pfi-ana-webhost/system-details.txt new file mode 100644 index 0000000..a7b4c11 --- /dev/null +++ b/servers/pfi-ana-webhost/system-details.txt @@ -0,0 +1,104 @@ + +===== HOST ===== + +Hostname: pfi-ana-webhost.phasefinal.com +Date: 2026-04-23T01:00:53-07:00 +Uptime: up 34 weeks, 4 days, 11 hours, 35 minutes +OS: Debian GNU/Linux 11 (bullseye) +Kernel: 5.10.0-34-amd64 +Arch: x86_64 + +===== HARDWARE ===== + +CPU cores: 16 +CPU model: Common KVM processor +MemTotal: 0.8 GB +MemAvailable: 0.3 GB + +===== GPUS ===== + +nvidia-smi not present (no NVIDIA GPUs or driver not installed) + +===== FILESYSTEMS (df) ===== + +Filesystem Size Used Avail Use% Mounted on +/dev/sda1 245G 8.6G 224G 4% / + +===== PERSISTENT MOUNTS (/etc/fstab, non-comment) ===== + +UUID=110a7e2b-aaf8-4a55-a064-408b665ee322 / ext4 rw,grpquota,errors=remount-ro,relatime,quota,usrquota 0 1 +UUID=f788f710-a3ec-457e-8053-2eeca8c3f029 none swap sw 0 0 +/dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0 +/home/dragonfirepro /home/chroot/169559808468512/home/dragonfirepro bind defaults,bind 0 + +===== TARGETED DATA PATHS ===== + +/opt (total: 4.0K) + total 8 + drwxr-xr-x 2 root root 4096 2023-09-24 15:42 . + drwxr-xr-x 18 root root 4096 2025-04-12 13:58 .. + +/srv (total: 12K) + total 12 + drwxr-xr-x 3 root root 4096 2023-09-24 16:15 . + drwxr-xr-x 18 root root 4096 2025-04-12 13:58 .. + drwxr-xr-x 2 ftp nogroup 4096 2023-09-24 16:15 ftp + + +===== DOCKER ===== + +docker not installed + +===== COMPOSE FILES (/opt/docker/compose/) ===== + +/opt/docker/compose not present + +===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) ===== + +/opt/docker/conf not present + +===== LISTENING PORTS ===== + +0.0.0.0:10000 +0.0.0.0:110 +0.0.0.0:143 +0.0.0.0:20000 +0.0.0.0:22 +0.0.0.0:993 +0.0.0.0:995 +[::]:10000 +10.250.50.52:53 +[::]:110 +[::1]:10023 +[::1]:25 +127.0.0.1:10023 +127.0.0.1:25 +127.0.0.1:3306 +127.0.0.1:465 +127.0.0.1:53 +127.0.0.1:587 +127.0.0.1:953 +[::]:143 +[::1]:465 +[::1]:53 +[::1]:587 +[::1]:953 +*:21 +[::]:22 +*:2222 +*:443 +*:80 +[::]:993 +[::]:995 +[fe80::e4f9:3aff:fec9:612a]%ens18:53 + +===== MODEL / HUGGINGFACE CACHES ===== + + +===== DOCKER-ADJACENT SYSTEMD SERVICES ===== + +(none matching) + +===== DONE ===== + +Paste the above back into the chat, or pass a path as argv[1] to save. diff --git a/servers/pfi-postgres/system-details.txt b/servers/pfi-postgres/system-details.txt new file mode 100644 index 0000000..7b8a940 --- /dev/null +++ b/servers/pfi-postgres/system-details.txt @@ -0,0 +1,80 @@ + +===== HOST ===== + +Hostname: pfi-postgres.phasefinal.com +Date: 2026-04-23T01:01:01-07:00 +Uptime: up 34 weeks, 4 days, 5 hours, 12 minutes +OS: Debian GNU/Linux 12 (bookworm) +Kernel: 6.1.0-38-amd64 +Arch: x86_64 + +===== HARDWARE ===== + +CPU cores: 16 +CPU model: Common KVM processor +MemTotal: 7.8 GB +MemAvailable: 7.2 GB + +===== GPUS ===== + +nvidia-smi not present (no NVIDIA GPUs or driver not installed) + +===== FILESYSTEMS (df) ===== + +Filesystem Size Used Avail Use% Mounted on +/dev/sda1 78G 4.2G 70G 6% / +10.250.50.50:/mnt/db 20T 281M 20T 1% /mnt/db + +===== PERSISTENT MOUNTS (/etc/fstab, non-comment) ===== + +UUID=e4f13bda-0320-43e7-be36-8df370f5c099 / ext4 errors=remount-ro 0 1 +UUID=f9efdef0-743c-46b9-8131-ae0a3e07e40a none swap sw 0 0 +/dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0 +10.250.50.50:/mnt/db /mnt/db nfs defaults 0 0 + +===== TARGETED DATA PATHS ===== + +/opt (total: 4.0K) + total 8 + drwxr-xr-x 2 root root 4096 2023-04-30 10:11 . + drwxr-xr-x 18 root root 4096 2025-08-23 19:46 .. + +/srv (total: 4.0K) + total 8 + drwxr-xr-x 2 root root 4096 2023-04-30 10:11 . + drwxr-xr-x 18 root root 4096 2025-08-23 19:46 .. + + +===== DOCKER ===== + +docker not installed + +===== COMPOSE FILES (/opt/docker/compose/) ===== + +/opt/docker/compose not present + +===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) ===== + +/opt/docker/conf not present + +===== LISTENING PORTS ===== + +0.0.0.0:111 +0.0.0.0:22 +0.0.0.0:5432 +[::]:111 +127.0.0.1:5433 +[::1]:5433 +[::]:22 +[::]:5432 + +===== MODEL / HUGGINGFACE CACHES ===== + + +===== DOCKER-ADJACENT SYSTEMD SERVICES ===== + +(none matching) + +===== DONE ===== + +Paste the above back into the chat, or pass a path as argv[1] to save. diff --git a/servers/pfi-pteradactyl/system-details.txt b/servers/pfi-pteradactyl/system-details.txt new file mode 100644 index 0000000..3ceaf39 --- /dev/null +++ b/servers/pfi-pteradactyl/system-details.txt @@ -0,0 +1,100 @@ + +===== HOST ===== + +Hostname: pfi-pteradactyl.phasefinal.com +Date: 2026-04-23T01:00:55-07:00 +Uptime: up 34 weeks, 4 days, 11 hours, 33 minutes +OS: Debian GNU/Linux 12 (bookworm) +Kernel: 6.1.0-34-amd64 +Arch: x86_64 + +===== HARDWARE ===== + +CPU cores: 8 +CPU model: Intel(R) Xeon(R) Silver 4310 CPU @ 2.10GHz +MemTotal: 7.8 GB +MemAvailable: 0.3 GB + +===== GPUS ===== + +nvidia-smi not present (no NVIDIA GPUs or driver not installed) + +===== FILESYSTEMS (df) ===== + +Filesystem Size Used Avail Use% Mounted on +/dev/sda1 250G 67G 172G 28% / + +===== PERSISTENT MOUNTS (/etc/fstab, non-comment) ===== + +UUID=028d611a-2cbc-4523-b1f3-2aa04d74302e / ext4 errors=remount-ro 0 1 +UUID=578e5418-a021-407d-8162-4b725807ee13 none swap sw 0 0 +/dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0 + +===== TARGETED DATA PATHS ===== + +/opt (total: 4.0K) + total 8 + drwxr-xr-x 2 root root 4096 2023-05-14 13:22 . + drwxr-xr-x 18 root root 4096 2025-04-28 17:36 .. + +/var/lib/docker (total: 4.0K) + +/srv (total: 4.0K) + total 8 + drwxr-xr-x 2 root root 4096 2023-05-14 13:22 . + drwxr-xr-x 18 root root 4096 2025-04-28 17:36 .. + + +===== DOCKER ===== + +Server: +docker daemon not reachable by current user + +----- docker info ----- +Containers: + +----- running containers ----- + +----- all containers ----- + +----- networks ----- + +----- networks (external, non-default — worth knowing for compose external: true) ----- + +----- named volumes ----- + +----- compose projects currently running ----- +(none) + +===== COMPOSE FILES (/opt/docker/compose/) ===== + +/opt/docker/compose not present + +===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) ===== + +/opt/docker/conf not present + +===== LISTENING PORTS ===== + +0.0.0.0:22 +0.0.0.0:80 +10.250.50.55:7777 +127.0.0.1:3306 +127.0.0.1:34135 +127.0.0.1:6379 +[::1]:6379 +*:2022 +[::]:22 +*:8080 + +===== MODEL / HUGGINGFACE CACHES ===== + + +===== DOCKER-ADJACENT SYSTEMD SERVICES ===== + +containerd.service running +docker.service running + +===== DONE ===== + +Paste the above back into the chat, or pass a path as argv[1] to save. diff --git a/servers/pfi-pve/README.md b/servers/pfi-pve/README.md index 97321f7..6014a27 100644 --- a/servers/pfi-pve/README.md +++ b/servers/pfi-pve/README.md @@ -4,13 +4,16 @@ Proxmox VE hypervisor for the Anaheim colo (`pve.phasefinal.com`). Hosts every V ## Network -- **LAN IP:** 10.250.250.31 +- **LAN IP:** 10.250.250.31 (in-band, OS-side) +- **iDRAC (OOB):** 10.250.250.30 — Dell iDRAC web UI at + (homepage card: *PFI-r750xs-iDRAC*) - **FQDN:** `pve.phasefinal.com` - **Web UI:** `https://10.250.250.31:8006` - **SSH:** `root@pfi-pve` (key auth; `ssh-target` file uses the hostname) ## Hardware +- **Chassis:** Dell PowerEdge R750xs (2U rackmount) - **CPU:** Intel Xeon Silver 4310 @ 2.10 GHz - **RAM:** 188 GB - **Kernel:** `6.8.12-8-pve` (Proxmox 8.x) @@ -18,9 +21,24 @@ Proxmox VE hypervisor for the Anaheim colo (`pve.phasefinal.com`). Hosts every V ## What it runs -The VMs in [`docs/pfi/proxmox-vms.md`](../../docs/pfi/proxmox-vms.md) live here — VM 100 (PFI-ANA-TRUENAS, now Debian NFS/SMB), VM 101 (DC), VM 102 (PFI-ANA-Docker = `ana-docker`), 103–111 (Mongo, Postgres, Tailscale, Pteradactyl, Webhost, tacticalrmm, etc.). +The VMs/CTs in [`docs/pfi/proxmox-vms.md`](../../docs/pfi/proxmox-vms.md) +live here. Known as of 2026-04-23 (from vzdump logs + `servers/` +dirs; not exhaustive): -`ana-ml2` (`10.250.50.54`) is a separate bare-metal box, not on this hypervisor. +- **VM 102** — `ana-docker` (PFI-ANA-Docker) +- **VM 105** — `pfi-postgres` (PFI-Postgres; PGDATA on NFS from ana-nas) +- **VM 107** — `pfi-pteradactyl` (game panel) +- **VM 110** — `pfi-ana-webhost` +- **VM 111** — `pfi-tacticalrmm` +- **VM (TBD)** — `pbs-ana` (Proxmox Backup Server; 10.250.50.90) +- **CT 109** — `ana-nas` (Debian LXC serving NFS/SMB; bind-mounts + from ospool; data-serving SPOF — see `memory/incident_ana_nas_spof.md`) +- **CT 112** — `ana-filebot` (file-task automation) +- **CT 113** — `ana-wg` (WireGuard VPN) + +`ana-ml2` (`10.250.50.54`) is a separate bare-metal Supermicro box, +not on this hypervisor. `sfsrv-ana` (`10.250.250.115`) runs on a +separate Dell R630, also not on this hypervisor. ## Refresh state diff --git a/servers/pfi-tacticalrmm/system-details.txt b/servers/pfi-tacticalrmm/system-details.txt new file mode 100644 index 0000000..454d265 --- /dev/null +++ b/servers/pfi-tacticalrmm/system-details.txt @@ -0,0 +1,87 @@ + +===== HOST ===== + +Hostname: pfi-tacticalrmm.phasefinal.com +Date: 2026-04-23T01:01:00-07:00 +Uptime: up 2 days, 6 hours, 0 minutes +OS: Debian GNU/Linux 12 (bookworm) +Kernel: 6.1.0-32-amd64 +Arch: x86_64 + +===== HARDWARE ===== + +CPU cores: 16 +CPU model: Common KVM processor +MemTotal: 7.8 GB +MemAvailable: 6.0 GB + +===== GPUS ===== + +nvidia-smi not present (no NVIDIA GPUs or driver not installed) + +===== FILESYSTEMS (df) ===== + +Filesystem Size Used Avail Use% Mounted on +/dev/sda1 250G 7.1G 231G 3% / + +===== PERSISTENT MOUNTS (/etc/fstab, non-comment) ===== + +UUID=ce4c34ea-d1fa-46ea-90c9-5538c1a077c4 / ext4 errors=remount-ro 0 1 +UUID=125637a3-f2eb-492f-85ad-96c4bbda1387 none swap sw 0 0 +/dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0 + +===== TARGETED DATA PATHS ===== + +/opt (total: 9.8M) + total 16 + drwxr-xr-x 4 root root 4096 2023-11-02 12:34 . + drwxr-xr-x 20 root root 4096 2025-04-07 14:39 .. + drwxr-xr-x 3 tactical tactical 4096 2023-11-02 12:34 tactical + drwxr-xr-x 8 tactical tactical 4096 2026-04-20 20:37 trmm-community-scripts + +/srv (total: 4.0K) + total 8 + drwxr-xr-x 2 root root 4096 2023-11-02 11:18 . + drwxr-xr-x 20 root root 4096 2025-04-07 14:39 .. + + +===== DOCKER ===== + +docker not installed + +===== COMPOSE FILES (/opt/docker/compose/) ===== + +/opt/docker/compose not present + +===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) ===== + +/opt/docker/conf not present + +===== LISTENING PORTS ===== + +0.0.0.0:22 +0.0.0.0:443 +0.0.0.0:80 +*:1024 +127.0.0.1:4222 +127.0.0.1:5432 +127.0.0.1:6379 +127.0.0.1:9235 +[::1]:5432 +[::1]:6379 +[::]:22 +[::]:443 +*:4430 +*:4433 +[::]:80 + +===== MODEL / HUGGINGFACE CACHES ===== + + +===== DOCKER-ADJACENT SYSTEMD SERVICES ===== + +(none matching) + +===== DONE ===== + +Paste the above back into the chat, or pass a path as argv[1] to save. diff --git a/servers/sf-ana-container/system-details.txt b/servers/sf-ana-container/system-details.txt new file mode 100644 index 0000000..e0aa120 --- /dev/null +++ b/servers/sf-ana-container/system-details.txt @@ -0,0 +1,249 @@ + +===== HOST ===== + +Hostname: sf-ana-container.surefire.com +Date: 2026-04-21T16:57:55-07:00 +Uptime: up 3 weeks, 6 days, 1 hour, 17 minutes +OS: Debian GNU/Linux 12 (bookworm) +Kernel: 6.1.0-21-amd64 +Arch: x86_64 + +===== HARDWARE ===== + +CPU cores: 16 +CPU model: QEMU Virtual CPU version 2.5+ +MemTotal: 15.6 GB +MemAvailable: 14.4 GB + +===== GPUS ===== + +nvidia-smi not present (no NVIDIA GPUs or driver not installed) + +===== FILESYSTEMS (df) ===== + +Filesystem Size Used Avail Use% Mounted on +/dev/sda1 250G 7.1G 231G 3% / + +===== PERSISTENT MOUNTS (/etc/fstab, non-comment) ===== + +UUID=fcc00570-1585-41d4-8c64-8e5d59af40e0 / ext4 errors=remount-ro 0 1 +UUID=35ad80f9-5952-489a-9178-575be588420c none swap sw 0 0 +/dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0 + +===== TARGETED DATA PATHS ===== + +/opt (total: 8.3M) + total 12 + drwxr-xr-x 3 root root 4096 2024-06-07 10:42 . + drwxr-xr-x 18 root root 4096 2024-06-07 10:35 .. + drwxrwxrwx 5 root root 4096 2024-06-07 10:46 docker + +/opt/docker (total: 8.3M) + total 20 + drwxrwxrwx 5 root root 4096 2024-06-07 10:46 . + drwxr-xr-x 3 root root 4096 2024-06-07 10:42 .. + drwxr-xr-x 6 vhoang vhoang 4096 2024-06-07 11:09 compose + drwxr-xr-x 3 vhoang vhoang 4096 2024-06-07 10:46 conf + drwxr-xr-x 2 vhoang vhoang 4096 2024-06-07 10:42 data + +/opt/docker/compose (total: 8.3M) + total 24 + drwxr-xr-x 6 vhoang vhoang 4096 2024-06-07 11:09 . + drwxrwxrwx 5 root root 4096 2024-06-07 10:46 .. + drwxr-xr-x 2 vhoang vhoang 4096 2024-06-07 10:43 dockge + drwxr-xr-x 4 root root 4096 2024-06-07 10:58 nginx-proxy + drwxr-xr-x 2 root root 4096 2024-06-07 10:46 traefik + drwxr-xr-x 3 root root 4096 2024-06-07 11:09 vaultwarden + +/opt/docker/conf (total: 8.0K) + total 12 + drwxr-xr-x 3 vhoang vhoang 4096 2024-06-07 10:46 . + drwxrwxrwx 5 root root 4096 2024-06-07 10:46 .. + drwxr-xr-x 2 root root 4096 2024-06-07 10:46 traefik-esh + +/var/lib/docker (total: 4.0K) + +/srv (total: 4.0K) + total 8 + drwxr-xr-x 2 root root 4096 2023-11-09 17:45 . + drwxr-xr-x 18 root root 4096 2024-06-07 10:35 .. + + +===== DOCKER ===== + +Server: +docker daemon not reachable by current user + +----- docker info ----- +Containers: + +----- running containers ----- + +----- all containers ----- + +----- networks ----- + +----- networks (external, non-default — worth knowing for compose external: true) ----- + +----- named volumes ----- + +----- compose projects currently running ----- +(none) + +===== COMPOSE FILES (/opt/docker/compose/) ===== + + +>>> /opt/docker/compose/dockge/compose.yaml +services: + dockge: + image: louislam/dockge:latest + restart: unless-stopped + ports: + # Host Port : Container Port + - 5001:5001 + volumes: + - /var/run/docker.sock:/var/run/docker.sock + - dockge_data:/app/data + - /opt/docker/compose:/opt/docker/compose + environment: + # Tell Dockge where is your stacks directory + - DOCKGE_STACKS_DIR=/opt/docker/compose + networks: + - tnet + +volumes: + dockge_data: null +networks: + tnet: + name: traefik-net + external: true + +>>> /opt/docker/compose/nginx-proxy/compose.yaml +version: "3.8" +services: + app: + image: jc21/nginx-proxy-manager:latest + restart: unless-stopped + ports: + # These ports are in format : + - 80:80 # Public HTTP Port + - 443:443 # Public HTTPS Port + - 81:81 # Admin Web Port + # Add any other Stream port you want to expose + # - '21:21' # FTP + + # Uncomment the next line if you uncomment anything in the section + # environment: + # Uncomment this if you want to change the location of + # the SQLite DB file within the container + # DB_SQLITE_FILE: "/data/database.sqlite" + + # Uncomment this if IPv6 is not enabled on your host + # DISABLE_IPV6: 'true' + + volumes: + - ./data:/data + - ./letsencrypt:/etc/letsencrypt + +networks: + default: + external: true + name: nginx-proxy_default + +>>> /opt/docker/compose/traefik/compose.yaml +#version: "3.3" + +services: + traefik: + image: traefik:latest + container_name: traefik + command: + - --log.level=DEBUG + - --configFile=/etc/traefik/traefik.yml + - --api.insecure=true + - --providers.docker=true + - --providers.docker.exposedbydefault=false + - --entrypoints.web.address=:80 + environment: + - AWS_ACCESS_KEY_ID=${AWS_KEY} + - AWS_SECRET_ACCESS_KEY=${AWS_SECRET} + - AWS_REGION=${AWS_REGION} + - AWS_HOSTED_ZONE_ID=${AWS_ZONEID} + ports: + - 80:80 + - 8380:8080 + - 443:443 + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - /opt/docker/conf/traefik-esh:/etc/traefik + networks: + - tnet + env_file: + - .env +networks: + tnet: + name: traefik-net + external: true + +>>> /opt/docker/compose/vaultwarden/compose.yaml +version: "3" +services: + vaultwarden: + image: vaultwarden/server:latest + container_name: vaultwarden + restart: unless-stopped + ports: + - 9445:80 #map any custom port to use (replace 8445 not 80) + volumes: + - ./bitwarden:/data:rw + environment: + - ADMIN_TOKEN=${ADMIN_TOKEN} + - WEBSOCKET_ENABLED=true + - SIGNUPS_ALLOWED=true + - DOMAIN=${DOMAIN} + - SMTP_HOST=surefire-com.mail.protection.outlook.com + - SMTP_FROM=vaultwarden@surefire.com + - SMTP_PORT=25 +# - SMTP_USERNAME=relay@surefire.com +# - SMTP_PASSWORD=5Gb&237N@p%6 +networks: + default: + external: true + name: nginx-proxy_default + +===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) ===== + +/opt/docker/conf +/opt/docker/conf/traefik-esh + +===== LISTENING PORTS ===== + +0.0.0.0:22 +0.0.0.0:443 +0.0.0.0:5001 +0.0.0.0:80 +0.0.0.0:8000 +0.0.0.0:81 +0.0.0.0:9443 +0.0.0.0:9445 +127.0.0.1:46567 +[::]:22 +[::]:443 +[::]:5001 +[::]:80 +[::]:8000 +[::]:81 +[::]:9443 +[::]:9445 + +===== MODEL / HUGGINGFACE CACHES ===== + + +===== DOCKER-ADJACENT SYSTEMD SERVICES ===== + +containerd.service running +docker.service running + +===== DONE ===== + +Paste the above back into the chat, or pass a path as argv[1] to save. diff --git a/servers/sf-r630/README.md b/servers/sf-r630/README.md index 15d7a8b..d6b451c 100644 --- a/servers/sf-r630/README.md +++ b/servers/sf-r630/README.md @@ -3,43 +3,52 @@ SureFire-client physical server at the Anaheim colo — Dell PowerEdge R630. **PFI-managed** under the hosting agreement (hardware + OS ops). +**This directory documents the hardware / iDRAC side only.** The OS +running on this box is tracked separately as +[`servers/sfsrv-ana/`](../sfsrv-ana/README.md) — Proxmox VE at +`10.250.250.115`. Refer to that for OS-level snapshots, SSH access, +and backup coverage. + ## Client context - **Client:** SureFire -- **PFI role:** full-service managed host +- **PFI role:** full-service managed host (iDRAC + Proxmox) ## Network -- **iDRAC BMC IP:** 10.250.250.110 (on the management subnet) - - Web UI: https://10.250.250.110/ -- **OS-side LAN IP:** **unknown** — not in FortiGate DHCP (static - config?). Fill this in once identified. -- **SSH:** no `ssh-target` set yet because the OS IP isn't known. Add - one (e.g. `root@` or `lkraven@`) once identified. +- **iDRAC BMC:** `10.250.250.110` (dedicated OOB management NIC) + - Web UI: (currently 404s; use the + vSphere-style Redfish API or racadm from iDRAC shell for + out-of-band ops) + - Homepage monitors this via ICMP ping rather than HTTP +- **OS-side (Proxmox VE):** see `servers/sfsrv-ana/` — `10.250.250.115` -## Infrastructure +## Hardware -- **Type:** Physical Dell PowerEdge R630 +- **Type:** Physical Dell PowerEdge R630, 2U - **Site:** Anaheim (PFI colo) +- Detailed hardware / CPU / RAM / disk data comes from `sfsrv-ana`'s + `system-details.txt` (the OS can read all of this via `dmidecode`, + `lscpu`, etc. — no need to duplicate here). -## Backup coverage +## Why this dir exists separately from sfsrv-ana -- **VM-image:** N/A (bare-metal physical) -- **File-level restic:** not yet configured - - Pending OS identification + SSH access - - Follow the `configs/restic/ana-ml2/` template when setting up - (both are bare-metal hosts; similar source list + no DB hooks by - default unless SF apps run databases) +- The iDRAC is a distinct management surface with its own credentials, + protocol stack, and firmware lifecycle. Power events, BIOS updates, + and console access live here. +- Inventory audits that track *hardware* (for hosting-agreement + billing, depreciation, etc.) should see the physical box as its own + entry independent of whatever OS happens to be running on it. ## Refresh state -Once the OS IP is known and an `ssh-target` is in place: -```bash -scripts/refresh-server-info.sh sf-r630 -``` +**No `ssh-target` and no `system-details.txt`** — iDRAC doesn't expose +a useful sshd for our inspect scripts. Host metadata comes from +`sfsrv-ana`'s snapshot instead. ## Discovered via `scripts/discover-fortigate.sh 10.250.250.1` on 2026-04-21 — iDRAC's DHCP lease on the management interface (MAC `74:e6:e2:fe:2c:7c`, -VCI `iDRAC`). +VCI `iDRAC`). Matched to sfsrv-ana via iDRAC Service Tag / physical +co-location on 2026-04-23. diff --git a/servers/sfsrv-ana/proxmox-details.txt b/servers/sfsrv-ana/proxmox-details.txt new file mode 100644 index 0000000..c2c849b --- /dev/null +++ b/servers/sfsrv-ana/proxmox-details.txt @@ -0,0 +1,164 @@ + +===== HOST ===== + +Hostname: sfsrv-ana.surefire.com +Date: 2026-04-21T16:57:44-07:00 +Uptime: up 3 weeks, 6 days, 1 hour, 17 minutes +PVE: pve-manager/8.2.2/9355359cd7afbae4 (running kernel: 6.8.4-3-pve) +Kernel: 6.8.4-3-pve +Arch: x86_64 + +===== HARDWARE ===== + +CPU model: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz +CPU cores: 24 +MemTotal: 110.0 GB +MemAvail: 104.1 GB + +===== CLUSTER ===== + +(single-node host (or pvecm returned no cluster info)) + +===== CLUSTER RESOURCES (pvesh) ===== + +┌─────────────────────────────┬─────────┬─────────────┬───────────────────┬───────┬───────────┬─────────┬───────┬────────┬────────────┬────────────┬──────────┬──────────────────┬───────────┬────────────┬──────┬───────────┬───────────┬──────────┬──────┐ +│ id │ type │ cgroup-mode │ content │ cpu │ disk │ hastate │ level │ maxcpu │ maxdisk │ maxmem │ mem │ name │ node │ plugintype │ pool │ status │ storage │ uptime │ vmid │ +╞═════════════════════════════╪═════════╪═════════════╪═══════════════════╪═══════╪═══════════╪═════════╪═══════╪════════╪════════════╪════════════╪══════════╪══════════════════╪═══════════╪════════════╪══════╪═══════════╪═══════════╪══════════╪══════╡ +│ node/sfsrv-ana │ node │ 2 │ │ 0.32% │ 6.71 GiB │ │ │ 24 │ 93.93 GiB │ 110.05 GiB │ 5.05 GiB │ │ sfsrv-ana │ │ │ online │ │ 3w 6d 1h │ │ +├─────────────────────────────┼─────────┼─────────────┼───────────────────┼───────┼───────────┼─────────┼───────┼────────┼────────────┼────────────┼──────────┼──────────────────┼───────────┼────────────┼──────┼───────────┼───────────┼──────────┼──────┤ +│ qemu/100 │ qemu │ │ │ 0.33% │ 0.00 B │ │ │ 16 │ 256.00 GiB │ 16.00 GiB │ 2.99 GiB │ sf-ana-container │ sfsrv-ana │ │ │ running │ │ 3w 6d 1h │ 100 │ +├─────────────────────────────┼─────────┼─────────────┼───────────────────┼───────┼───────────┼─────────┼───────┼────────┼────────────┼────────────┼──────────┼──────────────────┼───────────┼────────────┼──────┼───────────┼───────────┼──────────┼──────┤ +│ sdn/sfsrv-ana/localnetwork │ sdn │ │ │ │ │ │ │ │ │ │ │ │ sfsrv-ana │ │ │ ok │ │ │ │ +├─────────────────────────────┼─────────┼─────────────┼───────────────────┼───────┼───────────┼─────────┼───────┼────────┼────────────┼────────────┼──────────┼──────────────────┼───────────┼────────────┼──────┼───────────┼───────────┼──────────┼──────┤ +│ storage/sfsrv-ana/local │ storage │ │ backup,iso,vztmpl │ │ 6.71 GiB │ │ │ │ 93.93 GiB │ │ │ │ sfsrv-ana │ dir │ │ available │ local │ │ │ +├─────────────────────────────┼─────────┼─────────────┼───────────────────┼───────┼───────────┼─────────┼───────┼────────┼────────────┼────────────┼──────────┼──────────────────┼───────────┼────────────┼──────┼───────────┼───────────┼──────────┼──────┤ +│ storage/sfsrv-ana/local-lvm │ storage │ │ rootdir,images │ │ 13.97 GiB │ │ │ │ 793.79 GiB │ │ │ │ sfsrv-ana │ lvmthin │ │ available │ local-lvm │ │ │ +└─────────────────────────────┴─────────┴─────────────┴───────────────────┴───────┴───────────┴─────────┴───────┴────────┴────────────┴────────────┴──────────┴──────────────────┴───────────┴────────────┴──────┴───────────┴───────────┴──────────┴──────┘ + +===== VMs (qm list) ===== + + VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID + 100 sf-ana-container running 16384 256.00 1361 + +===== VM CONFIGS (qm config per id) ===== + + +----- VMID 100 ----- +agent: 1 +balloon: 0 +boot: order=scsi0;ide2;net0 +cores: 4 +ide2: local:iso/debian-12.2.0-amd64-netinst.iso,media=cdrom,size=628M +memory: 16384 +name: sf-ana-container +net0: virtio=6A:09:9A:8A:08:A5,bridge=vmbr0,firewall=1,tag=150 +onboot: 1 +ostype: l26 +scsi0: local-lvm:vm-100-disk-0,iothread=1,size=256G + +===== LXC CONTAINERS (pct list) ===== + + +===== LXC CONFIGS (pct config per id) ===== + +(no LXC containers found) + +===== STORAGE (pvesm status) ===== + +Name Type Status Total Used Available % +local dir active 98497780 7034220 86414012 7.14% +local-lvm lvmthin active 832352256 14649399 817702856 1.76% + +===== DATASTORE CONFIG (/etc/pve/storage.cfg) ===== + +dir: local + path /var/lib/vz + content iso,vztmpl,backup + +lvmthin: local-lvm + thinpool data + vgname pve + content rootdir,images + +===== ZFS POOLS (zpool list -v) ===== + +no pools available + +===== FILESYSTEMS (df -h, local fs only) ===== + +Filesystem Type Size Used Avail Use% Mounted on +/dev/mapper/pve-root ext4 94G 6.8G 83G 8% / +/dev/fuse fuse 128M 24K 128M 1% /etc/pve + +===== MOUNTED NFS / CIFS ===== + + +===== BACKUP JOBS (/etc/pve/jobs.cfg + /etc/vzdump.conf) ===== + +(/etc/pve/jobs.cfg not readable (no scheduled backups?)) + +----- vzdump defaults ----- + +===== BACKUP COVERAGE ANALYSIS ===== + +(/etc/pve/jobs.cfg not readable — no coverage data) + +----- per-guest coverage ----- +ID VERDICT REASON +100 NO no job targets this id + +===== RECENT BACKUPS ON LOCAL DUMP STORAGES ===== + + +----- /var/lib/vz/dump/ ----- + +===== PBS TARGETS ===== + +(no PBS storage configured on this node) + +===== REPLICATION JOBS (pvesr status) ===== + +JobID Enabled Target LastSync NextSync Duration FailCount State + +===== LISTENING PORTS ===== + +0.0.0.0:111 +0.0.0.0:22 +[::]:111 +[::1]:25 +127.0.0.1:25 +127.0.0.1:85 +[::]:22 +*:3128 +*:8006 + +===== SUBSCRIPTION / UPDATES ===== + +status: notfound + +----- apt upgradable (top 20) ----- +base-files/oldstable 12.4+deb12u13 amd64 [upgradable from: 12.4+deb12u5] +bash/oldstable 5.2.15-2+b10 amd64 [upgradable from: 5.2.15-2+b2] +bind9-dnsutils/oldstable-security 1:9.18.47-1~deb12u1 amd64 [upgradable from: 1:9.18.24-1] +bind9-host/oldstable-security 1:9.18.47-1~deb12u1 amd64 [upgradable from: 1:9.18.24-1] +bind9-libs/oldstable-security 1:9.18.47-1~deb12u1 amd64 [upgradable from: 1:9.18.24-1] +bsdextrautils/oldstable 2.38.1-5+deb12u3 amd64 [upgradable from: 2.38.1-5+deb12u1] +bsdutils/oldstable 1:2.38.1-5+deb12u3 amd64 [upgradable from: 1:2.38.1-5+deb12u1] +btrfs-progs/oldstable 6.2-1+deb12u2 amd64 [upgradable from: 6.2-1] +busybox/oldstable 1:1.35.0-4+b7 amd64 [upgradable from: 1:1.35.0-4+b3] +ca-certificates/oldstable,oldstable-updates 20230311+deb12u1 all [upgradable from: 20230311] +ceph-common/oldstable,oldstable-security 16.2.15+ds-0+deb12u1 amd64 [upgradable from: 16.2.11+ds-2] +ceph-fuse/oldstable,oldstable-security 16.2.15+ds-0+deb12u1 amd64 [upgradable from: 16.2.11+ds-2] +corosync/stable 3.1.10-pve2~bpo12+1 amd64 [upgradable from: 3.1.7-pve3] +criu/oldstable 3.17.1-2+deb12u2 amd64 [upgradable from: 3.17.1-2] +curl/oldstable 7.88.1-10+deb12u14 amd64 [upgradable from: 7.88.1-10+deb12u5] +debian-archive-keyring/oldstable 2023.3+deb12u2 all [upgradable from: 2023.3+deb12u1] +dirmngr/oldstable 2.2.40-1.1+deb12u2 amd64 [upgradable from: 2.2.40-1.1] +distro-info-data/oldstable 0.58+deb12u6 all [upgradable from: 0.58+deb12u1] +e2fsprogs/oldstable 1.47.0-2+b2 amd64 [upgradable from: 1.47.0-2] +eject/oldstable 2.38.1-5+deb12u3 amd64 [upgradable from: 2.38.1-5+deb12u1] + +===== DONE ===== + +Review this output for backup gaps (VERDICT=NO in coverage section) +and anomalies in storage / PBS / replication sections. diff --git a/servers/sfsrv-ana/system-details.txt b/servers/sfsrv-ana/system-details.txt new file mode 100644 index 0000000..7f2a75d --- /dev/null +++ b/servers/sfsrv-ana/system-details.txt @@ -0,0 +1,80 @@ + +===== HOST ===== + +Hostname: sfsrv-ana.surefire.com +Date: 2026-04-23T00:52:46-07:00 +Uptime: up 4 weeks, 9 hours, 12 minutes +OS: Debian GNU/Linux 12 (bookworm) +Kernel: 6.8.4-3-pve +Arch: x86_64 + +===== HARDWARE ===== + +CPU cores: 24 +CPU model: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz +MemTotal: 110.0 GB +MemAvailable: 104.1 GB + +===== GPUS ===== + +nvidia-smi not present (no NVIDIA GPUs or driver not installed) + +===== FILESYSTEMS (df) ===== + +Filesystem Size Used Avail Use% Mounted on +/dev/mapper/pve-root 94G 6.8G 83G 8% / +/dev/fuse 128M 24K 128M 1% /etc/pve + +===== PERSISTENT MOUNTS (/etc/fstab, non-comment) ===== + +/dev/pve/root / ext4 errors=remount-ro 0 1 +/dev/pve/swap none swap sw 0 0 +proc /proc proc defaults 0 0 + +===== TARGETED DATA PATHS ===== + +/opt (total: 4.0K) + total 8 + drwxr-xr-x 2 root root 4096 2023-03-22 07:38 . + drwxr-xr-x 18 root root 4096 2024-05-08 19:29 .. + +/srv (total: 4.0K) + total 8 + drwxr-xr-x 2 root root 4096 2023-03-22 07:38 . + drwxr-xr-x 18 root root 4096 2024-05-08 19:29 .. + + +===== DOCKER ===== + +docker not installed + +===== COMPOSE FILES (/opt/docker/compose/) ===== + +/opt/docker/compose not present + +===== CONFIG LAYOUT (/opt/docker/conf/ — top 200 entries) ===== + +/opt/docker/conf not present + +===== LISTENING PORTS ===== + +0.0.0.0:111 +0.0.0.0:22 +[::]:111 +[::1]:25 +127.0.0.1:25 +127.0.0.1:85 +[::]:22 +*:3128 +*:8006 + +===== MODEL / HUGGINGFACE CACHES ===== + + +===== DOCKER-ADJACENT SYSTEMD SERVICES ===== + +(none matching) + +===== DONE ===== + +Paste the above back into the chat, or pass a path as argv[1] to save.