esh-pfi-infrastructure/docs/runbooks/pbs-pbs4-upgrade-plan.md

Proxmox Backup Server 3 → 4: pbs-nh3 then pbs-ana

Requested by Prime via Miranda, 2026-10-04 ~1113 (thread 01M441X5MR7A8GYC99W7NWAR9S): "plan and execute … checkers first, backups of the PBS VMs themselves before you start … ordered plan with an honest rollback story, and report state as you go. Coordinate with the nh3-pve-2 follow-ups so the two don't collide." Official guide: pbs.proxmox.com/wiki/Upgrade_from_3_to_4 (read 2026-10-04).

The two servers (read live 2026-10-04 ~1120-1130)

	pbs-ana	pbs-nh3
Role	fleet primary: every PVE host backs up here (5 namespaces incl. SureFire's sfsrv-ana)	DR mirror: pulls pbs-ana daily 06:30 (remote pba-ana)
VM	100 on pfi-pve, disk local-lvm 32G, 8 GB, agent on	105 on nh3-pve, disk local-zfs 32G, 8 GB, agent on
OS / PBS	Debian 12.13, PBS 3.4.8 (3.4.9-2 available)	same
Boot	SeaBIOS + grub-pc (correct for BIOS; no UEFI trap)	same
Network	ens18 DHCP, allow-hotplug	same
Datastore	backups on NFSv3 10.250.50.50:/mnt/backup/pbs-ana (must stay v3: case-insensitive ZFS)	backups-mirror on NFSv3 10.100.50.50:/volume1/pbs
Jobs	verify 02:30, prune 05:30, GC Sun 06:00	verify 02:00, sync 06:30, prune + GC daily
Repos	pbs-enterprise enabled and 401-ing (no subscription) + no-subscription	same

Clients writing to pbs-ana on a schedule: nh3-pve 21:00, pfi-pve 22:00 (CT 109) and 03:00, esh-pve 03:30, sfsrv-ana 04:00 (PVE 8). Midday has no backup, sync, verify, prune or GC job on either server.

Access: infra-ops has no login on either VM (tier-3 hosts were never bootstrapped, reference_infra_ops_sudo_identity), and lkraven needs a password for sudo. So every step runs as root through the hypervisor's qemu-guest-agent (qm guest exec), driven by scripts/pbs4-upgrade-guest.sh <phase>. That path needs no guest network, so it is also the way back in if a trixie boot loses ens18.

Compatibility (release notes, PBS 4.0): "you can safely back up from Proxmox VE 8 to Proxmox Backup Server 4, or from Proxmox VE 9 to Proxmox Backup Server 3." So sfsrv-ana (PVE 8) keeps working against pbs-ana on 4, and a pbs-nh3 on 4 can pull from a pbs-ana still on 3 in between.

Order

1. VM backups first (Prime's ask; nobody backs up pbs-ana today): one-off vzdump of VM 100 to pfi-pve's local naspool-vmstorage and of VM 105 to nh3-pve's local, both --prune-backups keep-all=1 so nothing old is pruned.
2. Prep, both, no downtime (prep): tar /etc/proxmox-backup + /etc, enterprise repo off, NIC pinned by MAC, latest 3.4 (3.4.9), pbs3to4 --full must show 0 FAIL.
3. pbs-nh3 first (the mirror: lower stakes, and it proves the procedure): qm snapshot 105 pre-pbs4, then upgrade (datastore read-only, trixie sources, plan read, DETACHED dist-upgrade), postcheck, qm reboot 105, postcheck again, maintenance mode off, then a manual sync from pbs-ana (still on 3) to prove 3 → 4 pull.
4. pbs-ana the same way (qm snapshot 100 pre-pbs4 …). Then a manual sync pbs-ana (4) → pbs-nh3 (4) and a small real backup from a PVE 9 host to prove clients write.
5. Not touched during this window: nh3-pve-2 (its follow-ups are still waiting on Prime) and nh3-pve itself beyond VM 105. Nothing on the hypervisors changes.

Rollback (honest)

• Any time before the datastore leaves read-only maintenance: stop the VM, qm rollback <vmid> pre-pbs4, start. That returns the OS disk (PBS 3.4.9, /etc/proxmox-backup) exactly; seconds, no data involved. The datastore is on NFS, outside the VM disk, and nothing writes to it while it is read-only.
• After it is back to read-write: the same rollback still works for the OS. Backups written by PBS 4 in between stay in the datastore; Proxmox states 3 and 4 interoperate (PVE 9 clients write to PBS 3), so a rolled-back 3.4.9 reads them.
• If the snapshot itself were lost: restore the step-1 vzdump (minutes), or rebuild PBS and re-add the datastore path: the chunks and indexes on NFS are the backups; the VM only holds config.
• Snapshots are deleted after a day of clean running.

Log (PT, 2026-10-04) — DONE: both on PBS 4.2.7 / Debian 13.7 / kernel 6.12.111

• 1117-1119 VM backups: vzdump-qemu-100-2026_10_04-11_17_49.vma.zst (3.4 GB, pfi-pve naspool-vmstorage) and vzdump-qemu-105-2026_10_04-11_17_48.vma.zst (2.2 GB, nh3-pve local). Both listed by pvesm list.
• 1120-1122 ⚠ Self-inflicted, fixed: the first prep ran apt-get dist-upgrade IN-BAND through qemu-guest-agent. qemu-guest-agent was in the upgrade set; its postinst restarted the agent, systemd killed the agent's cgroup, and apt died mid-dpkg on BOTH VMs (~50 packages unpacked-not-configured incl. proxmox-backup-server, the agent half-configured). PBS kept serving on the old in-memory binaries throughout. Repaired 1123-1126 with dpkg --configure -a + apt-get -f install + dist-upgrade run under systemd-run (a transient unit outside the agent's cgroup): dpkg audit clean, PBS 3.4.9 running. The script now runs every apt step detached. Lesson: never run apt in-band through the channel it may restart.
• pbs3to4 --full on both: 1 FAIL "could not match the 'proxmox-backup' package version": these are PBS-on-plain-Debian installs (proxmox-backup-server on the Debian kernel, no proxmox-backup meta package) so the check has nothing to read. Tolerated explicitly and only that one; the real requirement (PBS >= 3.4.2 installed and running) is asserted directly. All else PASS/WARN (running kernel; systemd-timesyncd). No systemd-boot, BIOS + grub-pc (correct).
• pbs-nh3: snapshot pre-pbs4 1127; datastore read-only; dist-upgrade 1128-1130 (400 packages; removals only python3-distutils/lib2to3) EXIT 0; qm reboot 1130:37, agent back 1131:54; PBS 4.2.7, kernel 6.12.111, ens18 kept (.90 by DHCP), datastore mounted, no failed units, :8007 200; maintenance cleared 1132; manual sync from pbs-ana still on 3.4.9: TASK OK (2.36 GiB), i.e. a v4 mirror pulls from a v3 primary.
• pbs-ana: snapshot pre-pbs4 1135 (thin LV snap_vm-100-disk-0_pre-pbs4); no running tasks; datastore read-only; dist-upgrade 1136-1140 EXIT 0; qm reboot 1140:15, agent back 1140:38; same checks pass; maintenance cleared 1141. All four PVE 9 hosts see pbs-ana active. Write test: vzdump 113 (ana-wg) from pfi-pve → pbs-ana, OK (1141-1143). Sync v4 → v4: TASK OK, the new ct/113 snapshot mirrored to pbs-nh3.
• Both: pbs-enterprise.sources (added by the upgrade) set Enabled: false; apt update clean; Debian 13.7.
• Outage actually paid: each PBS answered nothing for ~1-2 min at its reboot; each datastore was read-only ~5 min. No scheduled backup, sync, verify, prune or GC fell in the window.
• Untested combination: sfsrv-ana (SureFire, PVE 8) → pbs-ana on 4. Supported per the PBS 4.0 release notes; its first real run is the 04:00 job on 2026-10-05; tomorrow's 08:00 backup-freshness check covers pbs/sfsrv-pve.
• Open: drop both pre-pbs4 snapshots on/after 2026-10-05 after that 04:00 run lands; keep the two one-off vzdumps until ~2026-10-11.
• 1636 (Prime: "give yourself a login on the backup servers"): infra-ops bootstrapped on both (uid/gid 850, fleet key, NOPASSWD sudo with log_output, same content as playbooks/bootstrap-infra-ops-user.yaml, applied through the guest agent). ssh infra-ops@10.250.50.90 / @10.100.50.90 + sudo -n verified; servers/pbs-*/ssh-target repointed.