# Proxmox Backup Server 3 → 4: pbs-nh3 then pbs-ana

Requested by Prime via Miranda, 2026-10-04 ~1113 (thread `01M441X5MR7A8GYC99W7NWAR9S`): "plan and execute … checkers
first, backups of the PBS VMs themselves before you start … ordered plan with an honest rollback story, and report state
as you go. Coordinate with the nh3-pve-2 follow-ups so the two don't collide." Official guide:
pbs.proxmox.com/wiki/Upgrade_from_3_to_4 (read 2026-10-04).

## The two servers (read live 2026-10-04 ~1120-1130)

| | pbs-ana | pbs-nh3 |
|---|---|---|
| Role | fleet primary: every PVE host backs up here (5 namespaces incl. SureFire's sfsrv-ana) | DR mirror: pulls pbs-ana daily 06:30 (remote `pba-ana`) |
| VM | 100 on pfi-pve, disk `local-lvm` 32G, 8 GB, agent on | 105 on nh3-pve, disk `local-zfs` 32G, 8 GB, agent on |
| OS / PBS | Debian 12.13, PBS 3.4.8 (3.4.9-2 available) | same |
| Boot | SeaBIOS + `grub-pc` (correct for BIOS; no UEFI trap) | same |
| Network | `ens18` DHCP, allow-hotplug | same |
| Datastore | `backups` on NFSv3 `10.250.50.50:/mnt/backup/pbs-ana` (must stay v3: case-insensitive ZFS) | `backups-mirror` on NFSv3 `10.100.50.50:/volume1/pbs` |
| Jobs | verify 02:30, prune 05:30, GC Sun 06:00 | verify 02:00, sync 06:30, prune + GC daily |
| Repos | **pbs-enterprise enabled and 401-ing** (no subscription) + no-subscription | same |

Clients writing to pbs-ana on a schedule: nh3-pve 21:00, pfi-pve 22:00 (CT 109) and 03:00, esh-pve 03:30, sfsrv-ana
04:00 (PVE 8). Midday has no backup, sync, verify, prune or GC job on either server.

**Access:** infra-ops has no login on either VM (tier-3 hosts were never bootstrapped, `reference_infra_ops_sudo_identity`),
and `lkraven` needs a password for sudo. So every step runs as root through the hypervisor's qemu-guest-agent
(`qm guest exec`), driven by `scripts/pbs4-upgrade-guest.sh <phase>`. That path needs no guest network, so it is also
the way back in if a trixie boot loses `ens18`.

**Compatibility (release notes, PBS 4.0):** "you can safely back up from Proxmox VE 8 to Proxmox Backup Server 4, or
from Proxmox VE 9 to Proxmox Backup Server 3." So sfsrv-ana (PVE 8) keeps working against pbs-ana on 4, and a pbs-nh3
on 4 can pull from a pbs-ana still on 3 in between.

## Order

1. **VM backups first** (Prime's ask; nobody backs up pbs-ana today): one-off `vzdump` of VM 100 to pfi-pve's local
   `naspool-vmstorage` and of VM 105 to nh3-pve's `local`, both `--prune-backups keep-all=1` so nothing old is pruned.
2. **Prep, both, no downtime** (`prep`): tar `/etc/proxmox-backup` + `/etc`, enterprise repo off, NIC pinned by MAC,
   latest 3.4 (3.4.9), `pbs3to4 --full` must show 0 FAIL.
3. **pbs-nh3 first** (the mirror: lower stakes, and it proves the procedure): `qm snapshot 105 pre-pbs4`, then
   `upgrade` (datastore read-only, trixie sources, plan read, DETACHED dist-upgrade), `postcheck`, `qm reboot 105`,
   `postcheck` again, maintenance mode off, then a manual sync from pbs-ana (still on 3) to prove 3 → 4 pull.
4. **pbs-ana** the same way (`qm snapshot 100 pre-pbs4` …). Then a manual sync pbs-ana (4) → pbs-nh3 (4) and a small
   real backup from a PVE 9 host to prove clients write.
5. **Not touched during this window:** nh3-pve-2 (its follow-ups are still waiting on Prime) and nh3-pve itself beyond
   VM 105. Nothing on the hypervisors changes.

## Rollback (honest)

- **Any time before the datastore leaves read-only maintenance:** stop the VM, `qm rollback <vmid> pre-pbs4`, start.
  That returns the OS disk (PBS 3.4.9, `/etc/proxmox-backup`) exactly; seconds, no data involved. The datastore is on
  NFS, outside the VM disk, and nothing writes to it while it is read-only.
- **After it is back to read-write:** the same rollback still works for the OS. Backups written by PBS 4 in between stay
  in the datastore; Proxmox states 3 and 4 interoperate (PVE 9 clients write to PBS 3), so a rolled-back 3.4.9 reads them.
- **If the snapshot itself were lost:** restore the step-1 vzdump (minutes), or rebuild PBS and re-add the datastore
  path: the chunks and indexes on NFS are the backups; the VM only holds config.
- Snapshots are deleted after a day of clean running.

## Log (PT, 2026-10-04) — DONE: both on PBS 4.2.7 / Debian 13.7 / kernel 6.12.111

- 1117-1119 **VM backups:** `vzdump-qemu-100-2026_10_04-11_17_49.vma.zst` (3.4 GB, pfi-pve `naspool-vmstorage`) and
  `vzdump-qemu-105-2026_10_04-11_17_48.vma.zst` (2.2 GB, nh3-pve `local`). Both listed by `pvesm list`.
- 1120-1122 ⚠ **Self-inflicted, fixed:** the first `prep` ran `apt-get dist-upgrade` IN-BAND through qemu-guest-agent.
  qemu-guest-agent was in the upgrade set; its postinst restarted the agent, systemd killed the agent's cgroup, and apt
  died mid-dpkg on BOTH VMs (~50 packages unpacked-not-configured incl. proxmox-backup-server, the agent half-configured).
  PBS kept serving on the old in-memory binaries throughout. Repaired 1123-1126 with `dpkg --configure -a` + `apt-get -f
  install` + `dist-upgrade` run under `systemd-run` (a transient unit outside the agent's cgroup): dpkg audit clean,
  PBS 3.4.9 running. The script now runs every apt step detached. **Lesson: never run apt in-band through the channel
  it may restart.**
- `pbs3to4 --full` on both: 1 FAIL "could not match the 'proxmox-backup' package version": these are PBS-on-plain-Debian
  installs (`proxmox-backup-server` on the Debian kernel, no `proxmox-backup` meta package) so the check has nothing to
  read. Tolerated explicitly and only that one; the real requirement (PBS >= 3.4.2 installed and running) is asserted
  directly. All else PASS/WARN (running kernel; systemd-timesyncd). No `systemd-boot`, BIOS + grub-pc (correct).
- **pbs-nh3:** snapshot `pre-pbs4` 1127; datastore read-only; dist-upgrade 1128-1130 (400 packages; removals only
  python3-distutils/lib2to3) EXIT 0; `qm reboot` 1130:37, agent back 1131:54; PBS 4.2.7, kernel 6.12.111, `ens18` kept
  (.90 by DHCP), datastore mounted, no failed units, :8007 200; maintenance cleared 1132; **manual sync from pbs-ana
  still on 3.4.9: TASK OK (2.36 GiB)**, i.e. a v4 mirror pulls from a v3 primary.
- **pbs-ana:** snapshot `pre-pbs4` 1135 (thin LV `snap_vm-100-disk-0_pre-pbs4`); no running tasks; datastore read-only;
  dist-upgrade 1136-1140 EXIT 0; `qm reboot` 1140:15, agent back 1140:38; same checks pass; maintenance cleared 1141.
  **All four PVE 9 hosts see `pbs-ana` active.** **Write test: `vzdump 113` (ana-wg) from pfi-pve → pbs-ana, OK
  (1141-1143).** **Sync v4 → v4: TASK OK, the new ct/113 snapshot mirrored to pbs-nh3.**
- Both: `pbs-enterprise.sources` (added by the upgrade) set `Enabled: false`; `apt update` clean; Debian 13.7.
- Outage actually paid: each PBS answered nothing for ~1-2 min at its reboot; each datastore was read-only ~5 min.
  No scheduled backup, sync, verify, prune or GC fell in the window.
- **Untested combination:** sfsrv-ana (SureFire, PVE 8) → pbs-ana on 4. Supported per the PBS 4.0 release notes; its
  first real run is the 04:00 job on 2026-10-05; tomorrow's 08:00 backup-freshness check covers `pbs/sfsrv-pve`.
- Open: drop both `pre-pbs4` snapshots on/after 2026-10-05 after that 04:00 run lands; keep the two one-off vzdumps
  until ~2026-10-11.
- 1636 (Prime: "give yourself a login on the backup servers"): **infra-ops bootstrapped on both** (uid/gid 850, fleet key,
  NOPASSWD sudo with `log_output`, same content as `playbooks/bootstrap-infra-ops-user.yaml`, applied through the guest
  agent). `ssh infra-ops@10.250.50.90` / `@10.100.50.90` + `sudo -n` verified; `servers/pbs-*/ssh-target` repointed.