diff --git a/docs/runbooks/pbs-pbs4-upgrade-plan.md b/docs/runbooks/pbs-pbs4-upgrade-plan.md new file mode 100644 index 0000000..afc0184 --- /dev/null +++ b/docs/runbooks/pbs-pbs4-upgrade-plan.md @@ -0,0 +1,87 @@ +# Proxmox Backup Server 3 → 4: pbs-nh3 then pbs-ana + +Requested by Prime via Miranda, 2026-10-04 ~1113 (thread `01M441X5MR7A8GYC99W7NWAR9S`): "plan and execute … checkers +first, backups of the PBS VMs themselves before you start … ordered plan with an honest rollback story, and report state +as you go. Coordinate with the nh3-pve-2 follow-ups so the two don't collide." Official guide: +pbs.proxmox.com/wiki/Upgrade_from_3_to_4 (read 2026-10-04). + +## The two servers (read live 2026-10-04 ~1120-1130) + +| | pbs-ana | pbs-nh3 | +|---|---|---| +| Role | fleet primary: every PVE host backs up here (5 namespaces incl. SureFire's sfsrv-ana) | DR mirror: pulls pbs-ana daily 06:30 (remote `pba-ana`) | +| VM | 100 on pfi-pve, disk `local-lvm` 32G, 8 GB, agent on | 105 on nh3-pve, disk `local-zfs` 32G, 8 GB, agent on | +| OS / PBS | Debian 12.13, PBS 3.4.8 (3.4.9-2 available) | same | +| Boot | SeaBIOS + `grub-pc` (correct for BIOS; no UEFI trap) | same | +| Network | `ens18` DHCP, allow-hotplug | same | +| Datastore | `backups` on NFSv3 `10.250.50.50:/mnt/backup/pbs-ana` (must stay v3: case-insensitive ZFS) | `backups-mirror` on NFSv3 `10.100.50.50:/volume1/pbs` | +| Jobs | verify 02:30, prune 05:30, GC Sun 06:00 | verify 02:00, sync 06:30, prune + GC daily | +| Repos | **pbs-enterprise enabled and 401-ing** (no subscription) + no-subscription | same | + +Clients writing to pbs-ana on a schedule: nh3-pve 21:00, pfi-pve 22:00 (CT 109) and 03:00, esh-pve 03:30, sfsrv-ana +04:00 (PVE 8). Midday has no backup, sync, verify, prune or GC job on either server. + +**Access:** infra-ops has no login on either VM (tier-3 hosts were never bootstrapped, `reference_infra_ops_sudo_identity`), +and `lkraven` needs a password for sudo. So every step runs as root through the hypervisor's qemu-guest-agent +(`qm guest exec`), driven by `scripts/pbs4-upgrade-guest.sh `. That path needs no guest network, so it is also +the way back in if a trixie boot loses `ens18`. + +**Compatibility (release notes, PBS 4.0):** "you can safely back up from Proxmox VE 8 to Proxmox Backup Server 4, or +from Proxmox VE 9 to Proxmox Backup Server 3." So sfsrv-ana (PVE 8) keeps working against pbs-ana on 4, and a pbs-nh3 +on 4 can pull from a pbs-ana still on 3 in between. + +## Order + +1. **VM backups first** (Prime's ask; nobody backs up pbs-ana today): one-off `vzdump` of VM 100 to pfi-pve's local + `naspool-vmstorage` and of VM 105 to nh3-pve's `local`, both `--prune-backups keep-all=1` so nothing old is pruned. +2. **Prep, both, no downtime** (`prep`): tar `/etc/proxmox-backup` + `/etc`, enterprise repo off, NIC pinned by MAC, + latest 3.4 (3.4.9), `pbs3to4 --full` must show 0 FAIL. +3. **pbs-nh3 first** (the mirror: lower stakes, and it proves the procedure): `qm snapshot 105 pre-pbs4`, then + `upgrade` (datastore read-only, trixie sources, plan read, DETACHED dist-upgrade), `postcheck`, `qm reboot 105`, + `postcheck` again, maintenance mode off, then a manual sync from pbs-ana (still on 3) to prove 3 → 4 pull. +4. **pbs-ana** the same way (`qm snapshot 100 pre-pbs4` …). Then a manual sync pbs-ana (4) → pbs-nh3 (4) and a small + real backup from a PVE 9 host to prove clients write. +5. **Not touched during this window:** nh3-pve-2 (its follow-ups are still waiting on Prime) and nh3-pve itself beyond + VM 105. Nothing on the hypervisors changes. + +## Rollback (honest) + +- **Any time before the datastore leaves read-only maintenance:** stop the VM, `qm rollback pre-pbs4`, start. + That returns the OS disk (PBS 3.4.9, `/etc/proxmox-backup`) exactly; seconds, no data involved. The datastore is on + NFS, outside the VM disk, and nothing writes to it while it is read-only. +- **After it is back to read-write:** the same rollback still works for the OS. Backups written by PBS 4 in between stay + in the datastore; Proxmox states 3 and 4 interoperate (PVE 9 clients write to PBS 3), so a rolled-back 3.4.9 reads them. +- **If the snapshot itself were lost:** restore the step-1 vzdump (minutes), or rebuild PBS and re-add the datastore + path: the chunks and indexes on NFS are the backups; the VM only holds config. +- Snapshots are deleted after a day of clean running. + +## Log (PT, 2026-10-04) — DONE: both on PBS 4.2.7 / Debian 13.7 / kernel 6.12.111 + +- 1117-1119 **VM backups:** `vzdump-qemu-100-2026_10_04-11_17_49.vma.zst` (3.4 GB, pfi-pve `naspool-vmstorage`) and + `vzdump-qemu-105-2026_10_04-11_17_48.vma.zst` (2.2 GB, nh3-pve `local`). Both listed by `pvesm list`. +- 1120-1122 ⚠ **Self-inflicted, fixed:** the first `prep` ran `apt-get dist-upgrade` IN-BAND through qemu-guest-agent. + qemu-guest-agent was in the upgrade set; its postinst restarted the agent, systemd killed the agent's cgroup, and apt + died mid-dpkg on BOTH VMs (~50 packages unpacked-not-configured incl. proxmox-backup-server, the agent half-configured). + PBS kept serving on the old in-memory binaries throughout. Repaired 1123-1126 with `dpkg --configure -a` + `apt-get -f + install` + `dist-upgrade` run under `systemd-run` (a transient unit outside the agent's cgroup): dpkg audit clean, + PBS 3.4.9 running. The script now runs every apt step detached. **Lesson: never run apt in-band through the channel + it may restart.** +- `pbs3to4 --full` on both: 1 FAIL "could not match the 'proxmox-backup' package version": these are PBS-on-plain-Debian + installs (`proxmox-backup-server` on the Debian kernel, no `proxmox-backup` meta package) so the check has nothing to + read. Tolerated explicitly and only that one; the real requirement (PBS >= 3.4.2 installed and running) is asserted + directly. All else PASS/WARN (running kernel; systemd-timesyncd). No `systemd-boot`, BIOS + grub-pc (correct). +- **pbs-nh3:** snapshot `pre-pbs4` 1127; datastore read-only; dist-upgrade 1128-1130 (400 packages; removals only + python3-distutils/lib2to3) EXIT 0; `qm reboot` 1130:37, agent back 1131:54; PBS 4.2.7, kernel 6.12.111, `ens18` kept + (.90 by DHCP), datastore mounted, no failed units, :8007 200; maintenance cleared 1132; **manual sync from pbs-ana + still on 3.4.9: TASK OK (2.36 GiB)**, i.e. a v4 mirror pulls from a v3 primary. +- **pbs-ana:** snapshot `pre-pbs4` 1135 (thin LV `snap_vm-100-disk-0_pre-pbs4`); no running tasks; datastore read-only; + dist-upgrade 1136-1140 EXIT 0; `qm reboot` 1140:15, agent back 1140:38; same checks pass; maintenance cleared 1141. + **All four PVE 9 hosts see `pbs-ana` active.** **Write test: `vzdump 113` (ana-wg) from pfi-pve → pbs-ana, OK + (1141-1143).** **Sync v4 → v4: TASK OK, the new ct/113 snapshot mirrored to pbs-nh3.** +- Both: `pbs-enterprise.sources` (added by the upgrade) set `Enabled: false`; `apt update` clean; Debian 13.7. +- Outage actually paid: each PBS answered nothing for ~1-2 min at its reboot; each datastore was read-only ~5 min. + No scheduled backup, sync, verify, prune or GC fell in the window. +- **Untested combination:** sfsrv-ana (SureFire, PVE 8) → pbs-ana on 4. Supported per the PBS 4.0 release notes; its + first real run is the 04:00 job on 2026-10-05; tomorrow's 08:00 backup-freshness check covers `pbs/sfsrv-pve`. +- Open: drop both `pre-pbs4` snapshots on/after 2026-10-05 after that 04:00 run lands; keep the two one-off vzdumps + until ~2026-10-11. diff --git a/persistent-memory.md b/persistent-memory.md index 5913273..c85d62f 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-10-04 ~0750 PT (pfi-pve + esh-pve on PVE 9; albok-service 0.1.6 live after Prime's push go; esh-nas-pve + nh3-pve on PVE 9; QNAP LCD fixed, awaiting Prime's look; NH3 Mac mini asleep, fixed on site today.)_ +_Last updated: 2026-10-04 ~1145 PT (PBS 4 on both; pfi-pve + esh-pve on PVE 9; albok-service 0.1.6 live after Prime's push go; esh-nas-pve + nh3-pve on PVE 9; QNAP LCD fixed, awaiting Prime's look; NH3 Mac mini asleep, fixed on site today.)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight @@ -119,6 +119,7 @@ _As of 2026-10-04 ~0546 PT. The newest subsection is first; older subsections ca ### 2026-10-03/04: live now (as of 2026-10-04 ~0546 PT) +- **PBS 3 → 4: DONE 2026-10-04 1144** (Prime via Miranda): pbs-nh3 then pbs-ana, both PBS 4.2.7 / Debian 13.7. VM backups first (pbs-ana had none anywhere), disk snapshots `pre-pbs4`, read-only datastores, detached dist-upgrades. Proven: v3→v4 sync, PVE 9→PBS 4 write (ct/113), v4→v4 sync. ⚠ My first prep killed apt mid-dpkg on both (qemu-guest-agent restarted itself under the in-band apt); repaired in ~5 min, PBS never stopped serving. **Open:** sfsrv-ana (PVE 8) 04:00 run on 10-05 is the one untested pair; drop `pre-pbs4` snapshots after it; vzdumps until ~10-11. → `docs/runbooks/pbs-pbs4-upgrade-plan.md` - **pfi-pve (Anaheim): DONE, PVE 9.2.21 / kernel 7.0.14-20 since 0656 2026-10-04** (Prime: "drive it", "proceed"). Prep fixed a real boot trap: the box boots UEFI but had `grub-pc`, so the ESP GRUB was the 2023 install's; now `grub-efi-amd64`. NIC pins, microcode, a PVE 8 reboot test (passed), then trixie. All 14 guests + services verified 0715. ⚠ Each pfi-pve reboot = ~4 min of NH3/ESH↔ANA (ana-scale) and **~20 min of LiteLLM, which also silences Miranda**. Break-glass = FortiGate WAN SSH `infra-ops@38.120.12.42` (verified). ⚠ ifupdown2 3.3 `ifreload -a -s` exits 1 on PVE's own `bridge-fd 0` — cosmetic (same on nh3-pve/esh-nas-pve). **Open: `lvremove -y pve/root-pre-pve9` on/after 2026-10-05** (asked infra-hermes); no `zpool upgrade`. → `docs/runbooks/pfi-pve-pve9-upgrade-plan.md` - **nh3-pve: DONE. ON PVE 9.2.21 / kernel 7.0.14-20 (booted 1402).** Post-check: every guest is back (the stopped 104/108 are as before), ZFS healthy, DNS, post office, albok, svos/hermes, mesh and AMT all OK. ⚠ **nh3-ml1 (CT 109) is DOWN:** no `proxmox-headers-7.0` was installed, so the nvidia 580.178.04 DKMS module was never built for 7.0, `nvidia-persistenced` failed, and CT 109 has no /dev/nvidia*. **FIXED 1409 (Prime go, `playbooks/nh3-pve-pve9-postfix.yaml`):** installed `proxmox-default-headers` + headers-7.0.14-20, nvidia DKMS built for 7.0 (580.178.04 compiles fine), nh3-ml1 back (TEI :8001/:8013 healthy, embed dim 1024); intel-microcode installed (non-free-firmware added; takes effect next boot); `pve-enterprise.sources` off; pve8to9 now 0 FAIL / 1 WARN (dkms, benign). VM 102 snapshot `pre-rootgrow-20261002` DELETED (Prime go). No `pre-pve9` snapshot had been taken. **POST-CHECK DONE: nh3-pve clean.** ⚠ Lesson: on a PVE host with DKMS, check that `proxmox-default-headers` is installed BEFORE a major upgrade, or the new kernel boots with no module. - **nh3-pve-2** (MS-03, NH3) is live at 10.100.250.62 on nh3-mgmt: diff --git a/scripts/pbs4-upgrade-guest.sh b/scripts/pbs4-upgrade-guest.sh new file mode 100755 index 0000000..9f13dcb --- /dev/null +++ b/scripts/pbs4-upgrade-guest.sh @@ -0,0 +1,106 @@ +#!/usr/bin/env bash +# Proxmox Backup Server 3 → 4, run AS ROOT INSIDE the PBS VM, one phase per call. +# +# scripts/pbs4-upgrade-guest.sh phases: prep | prepcheck | upgrade | status | postcheck +# +# How it is driven (2026-10-04): infra-ops has no login on the PBS VMs (tier-3 hosts were never bootstrapped), +# so the phases run through the hypervisor's qemu-guest-agent, which is root in the guest and needs no network: +# ssh infra-ops@ "sudo -n qm guest exec --timeout 600 -- bash -c '…this script…'" +# The same path is the recovery path if a trixie boot loses its network (the agent does not use it). +# +# Plan + rollback: docs/runbooks/pbs-pbs4-upgrade-plan.md. Official guide: pbs.proxmox.com/wiki/Upgrade_from_3_to_4. +# Nothing here reboots: the reboot is a separate, deliberate `qm reboot` from the hypervisor. +set -euo pipefail +phase=${1:?phase} +STAGE=/root/pbs4-prep-20261004 +ds=$(proxmox-backup-manager datastore list --output-format json | python3 -c 'import json,sys; print(json.load(sys.stdin)[0]["name"])') + +case "$phase" in +prep) + # no downtime: back up config, latest 3.4, enterprise repo off, NIC pinned, pbs3to4 --full + mkdir -p "$STAGE" + [ -f "$STAGE/etc-proxmox-backup.tar.gz" ] || tar czf "$STAGE/etc-proxmox-backup.tar.gz" -C /etc proxmox-backup + [ -f "$STAGE/etc.tar.gz" ] || tar czf "$STAGE/etc.tar.gz" -C / etc 2>/dev/null + [ -f "$STAGE/sources.list.before" ] || cp -p /etc/apt/sources.list "$STAGE/sources.list.before" + # the enterprise repo answers 401 (no subscription) and makes every `apt update` fail + if [ -f /etc/apt/sources.list.d/pbs-enterprise.list ]; then mv /etc/apt/sources.list.d/pbs-enterprise.list "$STAGE/"; fi + # pin the only NIC by MAC so Debian 13's udev cannot rename it (ens18 is DHCP, allow-hotplug) + mac=$(cat /sys/class/net/ens18/address) + mkdir -p /etc/systemd/network + if [ ! -f /etc/systemd/network/10-pin-ens18.link ]; then + printf '# Pin ens18 by MAC for the PBS 4 / Debian 13 upgrade. eshpfi scripts/pbs4-upgrade-guest.sh\n[Match]\nMACAddress=%s\nType=ether\n[Link]\nName=ens18\n' "$mac" > /etc/systemd/network/10-pin-ens18.link + update-initramfs -u -k all >/dev/null 2>&1 + fi + if apt-get update -q 2>&1 | grep -E '^(E|Err):'; then echo "REFUSING: apt update errors"; exit 1; fi # an if: set -e ignores a negated pipeline + # ⚠ DETACHED, never in-band: this script arrives through qemu-guest-agent, and qemu-guest-agent is itself in the + # upgrade set. Its postinst restarts the agent, systemd kills the agent's cgroup, and an in-band apt dies mid-dpkg. + # That happened on both VMs at 1120-1122 on 2026-10-04 (≈50 packages unpacked-not-configured, repaired with + # `dpkg --configure -a` run under systemd-run). A transient unit is outside the agent's cgroup and survives it. + if [ ! -f /root/pbs-prep-upgrade.log ]; then + systemd-run --unit=pbs-prep-upgrade --setenv=DEBIAN_FRONTEND=noninteractive \ + /bin/sh -c 'apt-get dist-upgrade -y -q -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold > /root/pbs-prep-upgrade.log 2>&1; echo "EXIT $?" >> /root/pbs-prep-upgrade.log' + fi + echo "latest-3.4 dist-upgrade started detached (log /root/pbs-prep-upgrade.log); then run: prepcheck" + ;; +prepcheck) + test -z "$(dpkg --audit 2>&1)" || { dpkg --audit; exit 1; } + proxmox-backup-manager versions + pbs3to4 --full 2>&1 | sed 's/\x1b\[[0-9;]*m//g' | grep -E '^(FAIL|WARN|NOTICE):|TOTAL|PASSED|SKIPPED|WARNINGS|FAILURES' || true + ;; +upgrade) + # refuse while a task is running; datastore to read-only maintenance; trixie sources; DETACHED dist-upgrade + running=$(proxmox-backup-manager task list --output-format json 2>/dev/null | python3 -c 'import json,sys; print(sum(1 for t in json.load(sys.stdin) if not t.get("endtime")))') + [ "$running" = 0 ] || { echo "REFUSING: $running task(s) running"; exit 1; } + out=$(pbs3to4 --full 2>&1 | sed 's/\x1b\[[0-9;]*m//g'); echo "$out" | grep -E '^(FAIL|WARN):' || true + # ONE known FAIL is tolerated, and only that one: these VMs are "PBS on plain Debian" (proxmox-backup-server on the + # Debian kernel, a documented install mode), so the `proxmox-backup` META package the checker looks for is absent by + # design and pbs3to4 says "could not match the 'proxmox-backup' package version". The requirement that check exists + # for (PBS >= 3.4.2 installed AND running) is asserted directly below instead. Any OTHER FAIL refuses. + if echo "$out" | grep '^FAIL:' | grep -qv "could not match the 'proxmox-backup' package version"; then + echo "REFUSING: pbs3to4 reports a FAIL other than the known meta-package one"; exit 1; fi + inst=$(dpkg-query -W -f='${Version}' proxmox-backup-server) + dpkg --compare-versions "$inst" ge 3.4.2 || { echo "REFUSING: proxmox-backup-server $inst < 3.4.2"; exit 1; } + proxmox-backup-manager versions | grep -q "running version: 3\.4\.[2-9]" || { echo "REFUSING: running PBS is not >= 3.4.2"; exit 1; } + proxmox-backup-manager datastore update "$ds" --maintenance-mode read-only + if grep -q bookworm /etc/apt/sources.list; then + sed -i 's/bookworm/trixie/g' /etc/apt/sources.list + if [ -f /etc/apt/sources.list.d/pbs-no-subscription.list ]; then mv /etc/apt/sources.list.d/pbs-no-subscription.list "$STAGE/"; fi + printf 'Types: deb\nURIs: http://download.proxmox.com/debian/pbs\nSuites: trixie\nComponents: pbs-no-subscription\nSigned-By: /usr/share/keyrings/proxmox-archive-keyring.gpg\n' > /etc/apt/sources.list.d/proxmox.sources + fi + if apt-get update -q 2>&1 | grep -E '^(E|Err):'; then echo "REFUSING: apt update errors"; exit 1; fi # an if: set -e ignores a negated pipeline + apt-get -s dist-upgrade > "$STAGE/dist-upgrade-plan.txt" 2>&1 + echo "installs/upgrades: $(grep -c '^Inst ' "$STAGE/dist-upgrade-plan.txt")" + grep '^Remv ' "$STAGE/dist-upgrade-plan.txt" | awk '{print $2}' | grep -v '^lib' | tr '\n' ' '; echo + if grep -qE '^Remv (proxmox-backup-server|proxmox-backup-client|grub-pc|ifupdown|isc-dhcp-client|openssh-server|nfs-common|qemu-guest-agent|linux-image-amd64) ' "$STAGE/dist-upgrade-plan.txt"; then + echo "REFUSING: the plan removes a package this VM cannot run, boot or be reached without"; exit 1 + fi + if [ ! -f /root/pbs4-upgrade.log ]; then + systemd-run --unit=pbs4-dist-upgrade --setenv=DEBIAN_FRONTEND=noninteractive \ + /bin/sh -c 'apt-get dist-upgrade -y -q -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold > /root/pbs4-upgrade.log 2>&1; echo "EXIT $?" >> /root/pbs4-upgrade.log' + fi + echo started + ;; +status) + tail -2 /root/pbs4-upgrade.log 2>/dev/null || echo "no upgrade log" + ;; +postcheck) + # after the dist-upgrade (before AND after the reboot): enterprise repo off, the things a boot needs + tail -1 /root/pbs4-upgrade.log | grep -qx 'EXIT 0' + test -z "$(dpkg --audit 2>&1)" + f=/etc/apt/sources.list.d/pbs-enterprise.sources + if [ -f $f ] && ! grep -q '^Enabled: *false' $f; then + if grep -q '^Enabled:' $f; then sed -i 's/^Enabled:.*/Enabled: false/' $f; else echo 'Enabled: false' >> $f; fi + fi + dpkg-query -W -f='${Status}\n' grub-pc qemu-guest-agent isc-dhcp-client nfs-common | sort | uniq -c + new=$(ls /boot/vmlinuz-* | sed 's#/boot/vmlinuz-##' | sort -V | tail -1); echo "newest kernel $new, running $(uname -r)" + # to a file first: `lsinitramfs | grep -q` SIGPIPEs zstd ("zstd failed") and pipefail then hides a real match + lsinitramfs /boot/initrd.img-$new > /tmp/pbs4-initrd.lst 2>/dev/null; grep -q 10-pin-ens18.link /tmp/pbs4-initrd.lst && echo "pin in $new initramfs" + grep -q "vmlinuz-$new" /boot/grub/grub.cfg && echo "grub.cfg boots $new" + proxmox-backup-manager versions + findmnt -no SOURCE,FSTYPE /mnt/pbs-datastore || echo "DATASTORE NOT MOUNTED" + proxmox-backup-manager datastore show "$ds" --output-format json | python3 -c 'import json,sys; d=json.load(sys.stdin); print("maintenance:", d.get("maintenance-mode","none"))' + systemctl is-active proxmox-backup proxmox-backup-proxy qemu-guest-agent + pbs3to4 --full 2>&1 | sed 's/\x1b\[[0-9;]*m//g' | grep -E '^(FAIL|WARN):|FAILURES' || true + ;; +*) echo "unknown phase $phase"; exit 2 ;; +esac diff --git a/servers/pbs-ana/README.md b/servers/pbs-ana/README.md index 1682045..a51d278 100644 --- a/servers/pbs-ana/README.md +++ b/servers/pbs-ana/README.md @@ -1,5 +1,7 @@ # pbs-ana +> **2026-10-04: PBS 4.2.7 on Debian 13.7 (kernel 6.12.111)**, upgraded from 3.4.8 (`docs/runbooks/pbs-pbs4-upgrade-plan.md`). PBS-on-plain-Debian install (no `proxmox-backup` meta package, Debian kernel), so `pbs3to4` FAILs one check by design. infra-ops has no login here: root ops go through the hypervisor's guest agent (`qm guest exec`). + Proxmox Backup Server — **fleet primary**. VM on pfi-pve with an NFS-mounted datastore on the Ana NAS. All 5 hypervisors (pfi-pve, nh3-pve, esh-pve, esh-pve-nas, sfsrv-ana) back up here; one-way sync diff --git a/servers/pbs-nh3/README.md b/servers/pbs-nh3/README.md index 84676f0..49e5d46 100644 --- a/servers/pbs-nh3/README.md +++ b/servers/pbs-nh3/README.md @@ -1,5 +1,7 @@ # pbs-nh3 +> **2026-10-04: PBS 4.2.7 on Debian 13.7 (kernel 6.12.111)**, upgraded from 3.4.8 (`docs/runbooks/pbs-pbs4-upgrade-plan.md`). PBS-on-plain-Debian install (no `proxmox-backup` meta package, Debian kernel), so `pbs3to4` FAILs one check by design. infra-ops has no login here: root ops go through the hypervisor's guest agent (`qm guest exec`). + Proxmox Backup Server — **DR mirror**. VM on nh3-pve with an NFS-mounted datastore on the NH3 Synology. Pulls nightly from pbs-ana (the primary) at 06:00.