vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
c4b2278e7ddcfe5778c0c9292ea3a36efa929fba
esh-pfi-infrastructure/servers/esh-pve-nas
T
History
vh c4b2278e7d feat(esh-pve-nas): stage the PVE root migration off the USB DOM 
Everything but the reboot. Two rerunnable elway playbooks; the host is
still running from the ext4 root and its boot path is byte-identical to
the last 140 days, because grub-install is deliberately held back to the
cutover window.

Phase 1 (esh-pve-nas-stage-zfs-root.yaml): carve a 512 MB /boot LV out
of the 768 MB swap LV, populate it, rsync the 4.3 GB ext4 root into
nvme/ROOT/pve-1, write the copy's fstab.

Phase 2 (esh-pve-nas-stage-bootloader.yaml): ZFS initramfs, grub.cfg,
explicit pve-zfs-root and pve-ext4-rollback entries with stable ids,
grubenv pinned to the rollback so cutover's grub-reboot is a one-shot.

Three landmines the plan did not predict, all caught by verify steps
asserting effective state rather than by reading the plan:

- The /boot LV had nowhere to live. VG pve had 4 MB free and mounted
  ext4 cannot shrink; freeing space from root needs a rescue boot, which
  costs the one-reboot property. Space came from swap (768M -> 256M).

- The runbook's `zpool set cachefile=... nvme` would have broken the
  NAS. Populating a cachefile flips the host from import-by-scan to
  import-by-cache, so a one-pool cache leaves ssd and tank unimported --
  and CT 103 esh-nas has twelve bind mounts spanning all three pools.
  Set on all three instead, verified in the resulting cache.

- update-grub silently emitted a pool-less root=ZFS=/ROOT/pve-1, which
  boots to an initramfs prompt. Debian's 10_linux builds ${rpool}${bootfs}
  and rpool comes from grub-probe --target=fs_label, which returns empty
  because GRUB's ZFS reader cannot open a pool with encryption,
  large_dnode and zstd_compress -- the same feature set that forced /boot
  to stay ext4. The probe failure is swallowed by `2>/dev/null || true`.
  Fixed with a /etc/default/grub.d drop-in plus explicit menu entries.

The transferable lesson: the original verify grepped for the correct
root= string appearing somewhere in grub.cfg, which passes while every
menu entry is still broken. Assert the effective value, not the presence
of a substring.
2026-08-17 22:11:14 -07:00
..
proxmox-details.txt
scripts: refresh-proxmox-info.sh wrapper for PVE snapshots
2026-04-20 23:01:48 -07:00
README.md
feat(esh-pve-nas): stage the PVE root migration off the USB DOM
2026-08-17 22:11:14 -07:00
ssh-target
Add the four Proxmox hypervisors to the inventory
2026-04-20 14:58:17 -07:00
system-details.txt
Add the four Proxmox hypervisors to the inventory
2026-04-20 14:58:17 -07:00

README.md

esh-pve-nas

Second Proxmox VE host at the ESH home lab (esh-nas-pve.esteban.net), dedicated to NAS / storage duty.

Network

  • LAN IP: 10.0.50.55
  • FQDN: esh-nas-pve.esteban.net
  • Web UI: https://10.0.50.55:8006
  • SSH: root@esh-pve-nas (key auth)

Media services on this box sit at 10.0.50.56 (Plex) and 10.0.50.57 (Jellyfin) — references to esh-nas-pve in the homepage config point at those addresses.

Hardware

  • CPU: Intel Xeon W-1250 @ 3.30 GHz

  • RAM: 125.6 GB

  • Kernel: 6.8.12-13-pve (Proxmox 8.x)

  • Storage: ⚠ PVE root is on a USB Disk-on-Module (sdq, 7.3 GB, ID_BUS=usb, NORELSYS 1081), whose 512 MB ESP is the host's only boot path. Not a wear problem (a DOM is SLC/pSLC), but it is on the USB bus, has no headroom, and is unmirrored. Mitigated 2026-08-17 (90% → 76%: journald capped, journal relocated to ZFS). Migration STAGED 2026-08-18 — awaiting only the reboot window; runbook docs/runbooks/esh-pve-nas-boot-migration.md.

    DOM LVM layout after staging (was root 6.04 G + swap 768 M):

    LV size role
    pve-root 6.04 G ext4 — the live root today, and the rollback after cutover
    pve-boot 512 M ext4 — the new /boot, carved out of swap
    pve-swap 256 M swap, shrunk to make room

    Post-cutover root is nvme/ROOT/pve-1 on the mirrored NVMe pool. /boot stays ext4 on the DOM on purpose: GRUB cannot read the nvme pool, which has encryption, large_dnode and zstd_compress enabled.

  • ⚠ Never set the ZFS cachefile on one pool. zpool set cachefile=… flips the host from import-by-scan to import-by-cache; a cache holding only nvme leaves ssd and tank unimported at boot, which empties every CT 103 export. Set it on all three or none.

  • Pools: nvme (2× 931 GB NVMe mirror — 32 G used, 867 G free, holds every guest rootfs), ssd (4× 894 GB Intel SATA, 2 mirrors — 1.42 T free), tank (12× 14.6 TB raidz2 ×2 — 40 T of 175 T). Plus NFS /mnt/pve/tank-vmbu for VM backups.

  • ⚠ CT 103 esh-nas (10.0.50.50) runs on THIS host and serves hard NFS to esh-docker-vm (/mnt/books, /mnt/backup) and to esh-pve (/mnt/pve/esh-nas, /mnt/pve/tank-vmbu). Taking this box down stalls esh-pve's storage layer and wedges esh-docker-vm into unkillable D-state unless those clients are quiesced first. Never reboot this host casually.

Root-fs pressure: 77% of a 5.9 GB root (1.3 GB free) after the 2026-08-17 mitigation. Still not enough for the pending upgrade — 225 packages, 161 carrying deb12uN/security bumps, including a ~250 MB signed kernel that lands in /boot. ⚠ Migrate first, patch after: unpacking that into 1.3 GB of headroom risks wedging dpkg on a hypervisor running five guests. The host is on pve-manager/8.4.11 vs esh-pve's 8.4.14 for exactly this reason.

What it runs

Media + storage-adjacent workloads for the ESH site. Not yet catalogued here; qm list / pct list on the host is authoritative.

The ESH-FileBot entry in homepage (10.0.50.70, labeled "Restic Backup Target and File Sync — esh-nas-pve") is a workload running here — mentioned in the backup project as a possible additional restic endpoint; confirm role before integrating into the fleet plan.

Refresh state

scripts/refresh-server-info.sh esh-pve-nas

Placement rule

Hypervisor for ESH storage/media VMs. Non-PFI scope.

Reference in New Issue View Git Blame Copy Permalink