ops(pbs): pbs-nh3 + pbs-ana upgraded to PBS 4.2.7 / Debian 13; guest-agent driver script, plan and log

This commit is contained in:
vh
2026-10-04 11:44:57 -07:00
parent 4d4851acd2
commit d0c1bc628a
5 changed files with 199 additions and 1 deletions
+87
View File
@@ -0,0 +1,87 @@
# Proxmox Backup Server 3 → 4: pbs-nh3 then pbs-ana
Requested by Prime via Miranda, 2026-10-04 ~1113 (thread `01M441X5MR7A8GYC99W7NWAR9S`): "plan and execute … checkers
first, backups of the PBS VMs themselves before you start … ordered plan with an honest rollback story, and report state
as you go. Coordinate with the nh3-pve-2 follow-ups so the two don't collide." Official guide:
pbs.proxmox.com/wiki/Upgrade_from_3_to_4 (read 2026-10-04).
## The two servers (read live 2026-10-04 ~1120-1130)
| | pbs-ana | pbs-nh3 |
|---|---|---|
| Role | fleet primary: every PVE host backs up here (5 namespaces incl. SureFire's sfsrv-ana) | DR mirror: pulls pbs-ana daily 06:30 (remote `pba-ana`) |
| VM | 100 on pfi-pve, disk `local-lvm` 32G, 8 GB, agent on | 105 on nh3-pve, disk `local-zfs` 32G, 8 GB, agent on |
| OS / PBS | Debian 12.13, PBS 3.4.8 (3.4.9-2 available) | same |
| Boot | SeaBIOS + `grub-pc` (correct for BIOS; no UEFI trap) | same |
| Network | `ens18` DHCP, allow-hotplug | same |
| Datastore | `backups` on NFSv3 `10.250.50.50:/mnt/backup/pbs-ana` (must stay v3: case-insensitive ZFS) | `backups-mirror` on NFSv3 `10.100.50.50:/volume1/pbs` |
| Jobs | verify 02:30, prune 05:30, GC Sun 06:00 | verify 02:00, sync 06:30, prune + GC daily |
| Repos | **pbs-enterprise enabled and 401-ing** (no subscription) + no-subscription | same |
Clients writing to pbs-ana on a schedule: nh3-pve 21:00, pfi-pve 22:00 (CT 109) and 03:00, esh-pve 03:30, sfsrv-ana
04:00 (PVE 8). Midday has no backup, sync, verify, prune or GC job on either server.
**Access:** infra-ops has no login on either VM (tier-3 hosts were never bootstrapped, `reference_infra_ops_sudo_identity`),
and `lkraven` needs a password for sudo. So every step runs as root through the hypervisor's qemu-guest-agent
(`qm guest exec`), driven by `scripts/pbs4-upgrade-guest.sh <phase>`. That path needs no guest network, so it is also
the way back in if a trixie boot loses `ens18`.
**Compatibility (release notes, PBS 4.0):** "you can safely back up from Proxmox VE 8 to Proxmox Backup Server 4, or
from Proxmox VE 9 to Proxmox Backup Server 3." So sfsrv-ana (PVE 8) keeps working against pbs-ana on 4, and a pbs-nh3
on 4 can pull from a pbs-ana still on 3 in between.
## Order
1. **VM backups first** (Prime's ask; nobody backs up pbs-ana today): one-off `vzdump` of VM 100 to pfi-pve's local
`naspool-vmstorage` and of VM 105 to nh3-pve's `local`, both `--prune-backups keep-all=1` so nothing old is pruned.
2. **Prep, both, no downtime** (`prep`): tar `/etc/proxmox-backup` + `/etc`, enterprise repo off, NIC pinned by MAC,
latest 3.4 (3.4.9), `pbs3to4 --full` must show 0 FAIL.
3. **pbs-nh3 first** (the mirror: lower stakes, and it proves the procedure): `qm snapshot 105 pre-pbs4`, then
`upgrade` (datastore read-only, trixie sources, plan read, DETACHED dist-upgrade), `postcheck`, `qm reboot 105`,
`postcheck` again, maintenance mode off, then a manual sync from pbs-ana (still on 3) to prove 3 → 4 pull.
4. **pbs-ana** the same way (`qm snapshot 100 pre-pbs4` …). Then a manual sync pbs-ana (4) → pbs-nh3 (4) and a small
real backup from a PVE 9 host to prove clients write.
5. **Not touched during this window:** nh3-pve-2 (its follow-ups are still waiting on Prime) and nh3-pve itself beyond
VM 105. Nothing on the hypervisors changes.
## Rollback (honest)
- **Any time before the datastore leaves read-only maintenance:** stop the VM, `qm rollback <vmid> pre-pbs4`, start.
That returns the OS disk (PBS 3.4.9, `/etc/proxmox-backup`) exactly; seconds, no data involved. The datastore is on
NFS, outside the VM disk, and nothing writes to it while it is read-only.
- **After it is back to read-write:** the same rollback still works for the OS. Backups written by PBS 4 in between stay
in the datastore; Proxmox states 3 and 4 interoperate (PVE 9 clients write to PBS 3), so a rolled-back 3.4.9 reads them.
- **If the snapshot itself were lost:** restore the step-1 vzdump (minutes), or rebuild PBS and re-add the datastore
path: the chunks and indexes on NFS are the backups; the VM only holds config.
- Snapshots are deleted after a day of clean running.
## Log (PT, 2026-10-04) — DONE: both on PBS 4.2.7 / Debian 13.7 / kernel 6.12.111
- 1117-1119 **VM backups:** `vzdump-qemu-100-2026_10_04-11_17_49.vma.zst` (3.4 GB, pfi-pve `naspool-vmstorage`) and
`vzdump-qemu-105-2026_10_04-11_17_48.vma.zst` (2.2 GB, nh3-pve `local`). Both listed by `pvesm list`.
- 1120-1122 ⚠ **Self-inflicted, fixed:** the first `prep` ran `apt-get dist-upgrade` IN-BAND through qemu-guest-agent.
qemu-guest-agent was in the upgrade set; its postinst restarted the agent, systemd killed the agent's cgroup, and apt
died mid-dpkg on BOTH VMs (~50 packages unpacked-not-configured incl. proxmox-backup-server, the agent half-configured).
PBS kept serving on the old in-memory binaries throughout. Repaired 1123-1126 with `dpkg --configure -a` + `apt-get -f
install` + `dist-upgrade` run under `systemd-run` (a transient unit outside the agent's cgroup): dpkg audit clean,
PBS 3.4.9 running. The script now runs every apt step detached. **Lesson: never run apt in-band through the channel
it may restart.**
- `pbs3to4 --full` on both: 1 FAIL "could not match the 'proxmox-backup' package version": these are PBS-on-plain-Debian
installs (`proxmox-backup-server` on the Debian kernel, no `proxmox-backup` meta package) so the check has nothing to
read. Tolerated explicitly and only that one; the real requirement (PBS >= 3.4.2 installed and running) is asserted
directly. All else PASS/WARN (running kernel; systemd-timesyncd). No `systemd-boot`, BIOS + grub-pc (correct).
- **pbs-nh3:** snapshot `pre-pbs4` 1127; datastore read-only; dist-upgrade 1128-1130 (400 packages; removals only
python3-distutils/lib2to3) EXIT 0; `qm reboot` 1130:37, agent back 1131:54; PBS 4.2.7, kernel 6.12.111, `ens18` kept
(.90 by DHCP), datastore mounted, no failed units, :8007 200; maintenance cleared 1132; **manual sync from pbs-ana
still on 3.4.9: TASK OK (2.36 GiB)**, i.e. a v4 mirror pulls from a v3 primary.
- **pbs-ana:** snapshot `pre-pbs4` 1135 (thin LV `snap_vm-100-disk-0_pre-pbs4`); no running tasks; datastore read-only;
dist-upgrade 1136-1140 EXIT 0; `qm reboot` 1140:15, agent back 1140:38; same checks pass; maintenance cleared 1141.
**All four PVE 9 hosts see `pbs-ana` active.** **Write test: `vzdump 113` (ana-wg) from pfi-pve → pbs-ana, OK
(1141-1143).** **Sync v4 → v4: TASK OK, the new ct/113 snapshot mirrored to pbs-nh3.**
- Both: `pbs-enterprise.sources` (added by the upgrade) set `Enabled: false`; `apt update` clean; Debian 13.7.
- Outage actually paid: each PBS answered nothing for ~1-2 min at its reboot; each datastore was read-only ~5 min.
No scheduled backup, sync, verify, prune or GC fell in the window.
- **Untested combination:** sfsrv-ana (SureFire, PVE 8) → pbs-ana on 4. Supported per the PBS 4.0 release notes; its
first real run is the 04:00 job on 2026-10-05; tomorrow's 08:00 backup-freshness check covers `pbs/sfsrv-pve`.
- Open: drop both `pre-pbs4` snapshots on/after 2026-10-05 after that 04:00 run lands; keep the two one-off vzdumps
until ~2026-10-11.
+2 -1
View File
@@ -1,6 +1,6 @@
# Persistent memory — eshpfi-management
_Last updated: 2026-10-04 ~0750 PT (pfi-pve + esh-pve on PVE 9; albok-service 0.1.6 live after Prime's push go; esh-nas-pve + nh3-pve on PVE 9; QNAP LCD fixed, awaiting Prime's look; NH3 Mac mini asleep, fixed on site today.)_
_Last updated: 2026-10-04 ~1145 PT (PBS 4 on both; pfi-pve + esh-pve on PVE 9; albok-service 0.1.6 live after Prime's push go; esh-nas-pve + nh3-pve on PVE 9; QNAP LCD fixed, awaiting Prime's look; NH3 Mac mini asleep, fixed on site today.)_
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
@@ -119,6 +119,7 @@ _As of 2026-10-04 ~0546 PT. The newest subsection is first; older subsections ca
### 2026-10-03/04: live now (as of 2026-10-04 ~0546 PT)
- **PBS 3 → 4: DONE 2026-10-04 1144** (Prime via Miranda): pbs-nh3 then pbs-ana, both PBS 4.2.7 / Debian 13.7. VM backups first (pbs-ana had none anywhere), disk snapshots `pre-pbs4`, read-only datastores, detached dist-upgrades. Proven: v3→v4 sync, PVE 9→PBS 4 write (ct/113), v4→v4 sync. ⚠ My first prep killed apt mid-dpkg on both (qemu-guest-agent restarted itself under the in-band apt); repaired in ~5 min, PBS never stopped serving. **Open:** sfsrv-ana (PVE 8) 04:00 run on 10-05 is the one untested pair; drop `pre-pbs4` snapshots after it; vzdumps until ~10-11. → `docs/runbooks/pbs-pbs4-upgrade-plan.md`
- **pfi-pve (Anaheim): DONE, PVE 9.2.21 / kernel 7.0.14-20 since 0656 2026-10-04** (Prime: "drive it", "proceed"). Prep fixed a real boot trap: the box boots UEFI but had `grub-pc`, so the ESP GRUB was the 2023 install's; now `grub-efi-amd64`. NIC pins, microcode, a PVE 8 reboot test (passed), then trixie. All 14 guests + services verified 0715. ⚠ Each pfi-pve reboot = ~4 min of NH3/ESH↔ANA (ana-scale) and **~20 min of LiteLLM, which also silences Miranda**. Break-glass = FortiGate WAN SSH `infra-ops@38.120.12.42` (verified). ⚠ ifupdown2 3.3 `ifreload -a -s` exits 1 on PVE's own `bridge-fd 0` — cosmetic (same on nh3-pve/esh-nas-pve). **Open: `lvremove -y pve/root-pre-pve9` on/after 2026-10-05** (asked infra-hermes); no `zpool upgrade`. → `docs/runbooks/pfi-pve-pve9-upgrade-plan.md`
- **nh3-pve: DONE. ON PVE 9.2.21 / kernel 7.0.14-20 (booted 1402).** Post-check: every guest is back (the stopped 104/108 are as before), ZFS healthy, DNS, post office, albok, svos/hermes, mesh and AMT all OK. ⚠ **nh3-ml1 (CT 109) is DOWN:** no `proxmox-headers-7.0` was installed, so the nvidia 580.178.04 DKMS module was never built for 7.0, `nvidia-persistenced` failed, and CT 109 has no /dev/nvidia*. **FIXED 1409 (Prime go, `playbooks/nh3-pve-pve9-postfix.yaml`):** installed `proxmox-default-headers` + headers-7.0.14-20, nvidia DKMS built for 7.0 (580.178.04 compiles fine), nh3-ml1 back (TEI :8001/:8013 healthy, embed dim 1024); intel-microcode installed (non-free-firmware added; takes effect next boot); `pve-enterprise.sources` off; pve8to9 now 0 FAIL / 1 WARN (dkms, benign). VM 102 snapshot `pre-rootgrow-20261002` DELETED (Prime go). No `pre-pve9` snapshot had been taken. **POST-CHECK DONE: nh3-pve clean.** ⚠ Lesson: on a PVE host with DKMS, check that `proxmox-default-headers` is installed BEFORE a major upgrade, or the new kernel boots with no module.
- **nh3-pve-2** (MS-03, NH3) is live at 10.100.250.62 on nh3-mgmt:
+106
View File
@@ -0,0 +1,106 @@
#!/usr/bin/env bash
# Proxmox Backup Server 3 → 4, run AS ROOT INSIDE the PBS VM, one phase per call.
#
# scripts/pbs4-upgrade-guest.sh <phase> phases: prep | prepcheck | upgrade | status | postcheck
#
# How it is driven (2026-10-04): infra-ops has no login on the PBS VMs (tier-3 hosts were never bootstrapped),
# so the phases run through the hypervisor's qemu-guest-agent, which is root in the guest and needs no network:
# ssh infra-ops@<pve> "sudo -n qm guest exec <vmid> --timeout 600 -- bash -c '…this script…'"
# The same path is the recovery path if a trixie boot loses its network (the agent does not use it).
#
# Plan + rollback: docs/runbooks/pbs-pbs4-upgrade-plan.md. Official guide: pbs.proxmox.com/wiki/Upgrade_from_3_to_4.
# Nothing here reboots: the reboot is a separate, deliberate `qm reboot` from the hypervisor.
set -euo pipefail
phase=${1:?phase}
STAGE=/root/pbs4-prep-20261004
ds=$(proxmox-backup-manager datastore list --output-format json | python3 -c 'import json,sys; print(json.load(sys.stdin)[0]["name"])')
case "$phase" in
prep)
# no downtime: back up config, latest 3.4, enterprise repo off, NIC pinned, pbs3to4 --full
mkdir -p "$STAGE"
[ -f "$STAGE/etc-proxmox-backup.tar.gz" ] || tar czf "$STAGE/etc-proxmox-backup.tar.gz" -C /etc proxmox-backup
[ -f "$STAGE/etc.tar.gz" ] || tar czf "$STAGE/etc.tar.gz" -C / etc 2>/dev/null
[ -f "$STAGE/sources.list.before" ] || cp -p /etc/apt/sources.list "$STAGE/sources.list.before"
# the enterprise repo answers 401 (no subscription) and makes every `apt update` fail
if [ -f /etc/apt/sources.list.d/pbs-enterprise.list ]; then mv /etc/apt/sources.list.d/pbs-enterprise.list "$STAGE/"; fi
# pin the only NIC by MAC so Debian 13's udev cannot rename it (ens18 is DHCP, allow-hotplug)
mac=$(cat /sys/class/net/ens18/address)
mkdir -p /etc/systemd/network
if [ ! -f /etc/systemd/network/10-pin-ens18.link ]; then
printf '# Pin ens18 by MAC for the PBS 4 / Debian 13 upgrade. eshpfi scripts/pbs4-upgrade-guest.sh\n[Match]\nMACAddress=%s\nType=ether\n[Link]\nName=ens18\n' "$mac" > /etc/systemd/network/10-pin-ens18.link
update-initramfs -u -k all >/dev/null 2>&1
fi
if apt-get update -q 2>&1 | grep -E '^(E|Err):'; then echo "REFUSING: apt update errors"; exit 1; fi # an if: set -e ignores a negated pipeline
# ⚠ DETACHED, never in-band: this script arrives through qemu-guest-agent, and qemu-guest-agent is itself in the
# upgrade set. Its postinst restarts the agent, systemd kills the agent's cgroup, and an in-band apt dies mid-dpkg.
# That happened on both VMs at 1120-1122 on 2026-10-04 (≈50 packages unpacked-not-configured, repaired with
# `dpkg --configure -a` run under systemd-run). A transient unit is outside the agent's cgroup and survives it.
if [ ! -f /root/pbs-prep-upgrade.log ]; then
systemd-run --unit=pbs-prep-upgrade --setenv=DEBIAN_FRONTEND=noninteractive \
/bin/sh -c 'apt-get dist-upgrade -y -q -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold > /root/pbs-prep-upgrade.log 2>&1; echo "EXIT $?" >> /root/pbs-prep-upgrade.log'
fi
echo "latest-3.4 dist-upgrade started detached (log /root/pbs-prep-upgrade.log); then run: prepcheck"
;;
prepcheck)
test -z "$(dpkg --audit 2>&1)" || { dpkg --audit; exit 1; }
proxmox-backup-manager versions
pbs3to4 --full 2>&1 | sed 's/\x1b\[[0-9;]*m//g' | grep -E '^(FAIL|WARN|NOTICE):|TOTAL|PASSED|SKIPPED|WARNINGS|FAILURES' || true
;;
upgrade)
# refuse while a task is running; datastore to read-only maintenance; trixie sources; DETACHED dist-upgrade
running=$(proxmox-backup-manager task list --output-format json 2>/dev/null | python3 -c 'import json,sys; print(sum(1 for t in json.load(sys.stdin) if not t.get("endtime")))')
[ "$running" = 0 ] || { echo "REFUSING: $running task(s) running"; exit 1; }
out=$(pbs3to4 --full 2>&1 | sed 's/\x1b\[[0-9;]*m//g'); echo "$out" | grep -E '^(FAIL|WARN):' || true
# ONE known FAIL is tolerated, and only that one: these VMs are "PBS on plain Debian" (proxmox-backup-server on the
# Debian kernel, a documented install mode), so the `proxmox-backup` META package the checker looks for is absent by
# design and pbs3to4 says "could not match the 'proxmox-backup' package version". The requirement that check exists
# for (PBS >= 3.4.2 installed AND running) is asserted directly below instead. Any OTHER FAIL refuses.
if echo "$out" | grep '^FAIL:' | grep -qv "could not match the 'proxmox-backup' package version"; then
echo "REFUSING: pbs3to4 reports a FAIL other than the known meta-package one"; exit 1; fi
inst=$(dpkg-query -W -f='${Version}' proxmox-backup-server)
dpkg --compare-versions "$inst" ge 3.4.2 || { echo "REFUSING: proxmox-backup-server $inst < 3.4.2"; exit 1; }
proxmox-backup-manager versions | grep -q "running version: 3\.4\.[2-9]" || { echo "REFUSING: running PBS is not >= 3.4.2"; exit 1; }
proxmox-backup-manager datastore update "$ds" --maintenance-mode read-only
if grep -q bookworm /etc/apt/sources.list; then
sed -i 's/bookworm/trixie/g' /etc/apt/sources.list
if [ -f /etc/apt/sources.list.d/pbs-no-subscription.list ]; then mv /etc/apt/sources.list.d/pbs-no-subscription.list "$STAGE/"; fi
printf 'Types: deb\nURIs: http://download.proxmox.com/debian/pbs\nSuites: trixie\nComponents: pbs-no-subscription\nSigned-By: /usr/share/keyrings/proxmox-archive-keyring.gpg\n' > /etc/apt/sources.list.d/proxmox.sources
fi
if apt-get update -q 2>&1 | grep -E '^(E|Err):'; then echo "REFUSING: apt update errors"; exit 1; fi # an if: set -e ignores a negated pipeline
apt-get -s dist-upgrade > "$STAGE/dist-upgrade-plan.txt" 2>&1
echo "installs/upgrades: $(grep -c '^Inst ' "$STAGE/dist-upgrade-plan.txt")"
grep '^Remv ' "$STAGE/dist-upgrade-plan.txt" | awk '{print $2}' | grep -v '^lib' | tr '\n' ' '; echo
if grep -qE '^Remv (proxmox-backup-server|proxmox-backup-client|grub-pc|ifupdown|isc-dhcp-client|openssh-server|nfs-common|qemu-guest-agent|linux-image-amd64) ' "$STAGE/dist-upgrade-plan.txt"; then
echo "REFUSING: the plan removes a package this VM cannot run, boot or be reached without"; exit 1
fi
if [ ! -f /root/pbs4-upgrade.log ]; then
systemd-run --unit=pbs4-dist-upgrade --setenv=DEBIAN_FRONTEND=noninteractive \
/bin/sh -c 'apt-get dist-upgrade -y -q -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold > /root/pbs4-upgrade.log 2>&1; echo "EXIT $?" >> /root/pbs4-upgrade.log'
fi
echo started
;;
status)
tail -2 /root/pbs4-upgrade.log 2>/dev/null || echo "no upgrade log"
;;
postcheck)
# after the dist-upgrade (before AND after the reboot): enterprise repo off, the things a boot needs
tail -1 /root/pbs4-upgrade.log | grep -qx 'EXIT 0'
test -z "$(dpkg --audit 2>&1)"
f=/etc/apt/sources.list.d/pbs-enterprise.sources
if [ -f $f ] && ! grep -q '^Enabled: *false' $f; then
if grep -q '^Enabled:' $f; then sed -i 's/^Enabled:.*/Enabled: false/' $f; else echo 'Enabled: false' >> $f; fi
fi
dpkg-query -W -f='${Status}\n' grub-pc qemu-guest-agent isc-dhcp-client nfs-common | sort | uniq -c
new=$(ls /boot/vmlinuz-* | sed 's#/boot/vmlinuz-##' | sort -V | tail -1); echo "newest kernel $new, running $(uname -r)"
# to a file first: `lsinitramfs | grep -q` SIGPIPEs zstd ("zstd failed") and pipefail then hides a real match
lsinitramfs /boot/initrd.img-$new > /tmp/pbs4-initrd.lst 2>/dev/null; grep -q 10-pin-ens18.link /tmp/pbs4-initrd.lst && echo "pin in $new initramfs"
grep -q "vmlinuz-$new" /boot/grub/grub.cfg && echo "grub.cfg boots $new"
proxmox-backup-manager versions
findmnt -no SOURCE,FSTYPE /mnt/pbs-datastore || echo "DATASTORE NOT MOUNTED"
proxmox-backup-manager datastore show "$ds" --output-format json | python3 -c 'import json,sys; d=json.load(sys.stdin); print("maintenance:", d.get("maintenance-mode","none"))'
systemctl is-active proxmox-backup proxmox-backup-proxy qemu-guest-agent
pbs3to4 --full 2>&1 | sed 's/\x1b\[[0-9;]*m//g' | grep -E '^(FAIL|WARN):|FAILURES' || true
;;
*) echo "unknown phase $phase"; exit 2 ;;
esac
+2
View File
@@ -1,5 +1,7 @@
# pbs-ana
> **2026-10-04: PBS 4.2.7 on Debian 13.7 (kernel 6.12.111)**, upgraded from 3.4.8 (`docs/runbooks/pbs-pbs4-upgrade-plan.md`). PBS-on-plain-Debian install (no `proxmox-backup` meta package, Debian kernel), so `pbs3to4` FAILs one check by design. infra-ops has no login here: root ops go through the hypervisor's guest agent (`qm guest exec`).
Proxmox Backup Server — **fleet primary**. VM on pfi-pve with an
NFS-mounted datastore on the Ana NAS. All 5 hypervisors (pfi-pve,
nh3-pve, esh-pve, esh-pve-nas, sfsrv-ana) back up here; one-way sync
+2
View File
@@ -1,5 +1,7 @@
# pbs-nh3
> **2026-10-04: PBS 4.2.7 on Debian 13.7 (kernel 6.12.111)**, upgraded from 3.4.8 (`docs/runbooks/pbs-pbs4-upgrade-plan.md`). PBS-on-plain-Debian install (no `proxmox-backup` meta package, Debian kernel), so `pbs3to4` FAILs one check by design. infra-ops has no login here: root ops go through the hypervisor's guest agent (`qm guest exec`).
Proxmox Backup Server — **DR mirror**. VM on nh3-pve with an
NFS-mounted datastore on the NH3 Synology. Pulls nightly from pbs-ana
(the primary) at 06:00.