vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
5f11d1b3cb0c8cefe39ab7545586f3a6847d6b99
esh-pfi-infrastructure/servers/esh-pve-nas
T
History
vh 5f11d1b3cb feat(esh-pve-nas): cut PVE root over to ZFS on the mirrored NVMe 
Root is now nvme/ROOT/pve-1. The USB DOM keeps the ESP and /boot but is
out of the runtime I/O path, so a bus reset can no longer drop root from
under a running hypervisor. All five guests healthy, three pools ONLINE,
system running, ext4 pve-root intact and unmounted as the rollback with
its own kernel and initrd. zfs-import-cache is now the active import
path -- the all-three-pools cachefile fix doing its job.

The window cost an unplanned outage, and the cause was this repo's own
tooling rather than the migration.

The staging chroot ran `mount --rbind /dev` and /sys with no
--make-rslave. On systemd `/` has shared propagation, so the cutover's
`umount -R` propagated back into the live host and removed the real
/sys/fs/cgroup, /dev/pts and /dev/shm. With cgroup2 gone systemd-logind
could not create a session: ping fine, TCP fine, SSH authentication
succeeded, resident daemons kept serving -- and every new exec hung,
including /sbin/reboot, so the reboot never ran at all.

It impersonates failing root-disk I/O almost perfectly, and I called it
as the DOM dying. That was wrong. dmesg had the answer throughout: the
DOM attached cleanly with no errors, and the last log timestamp was
12114881s -- 140 days -- meaning this was still the original boot. A
down-detector had also never reported the host down, which I read as a
fast reboot rather than as no reboot.

Fixes and guards:

- --make-rslave after every rbind, plus a guard that refuses to proceed
  while any chroot bind still reports shared propagation.
- Confirm a reboot by observing the host DOWN, not by watching for it to
  come back. Those two states are indistinguishable otherwise.
- Blast radius now measured from the server: `ss` inside CT 103 found
  five NFS clients, not the two documented. The new one that mattered is
  esh-vm-db, hard-mounted and unreachable by ssh. Left mounted on
  purpose and it came through read-write.
- grub-reboot's one-shot does NOT work here: grubenv sits on an LVM LV
  which GRUB reads but cannot write, so next_entry survived the boot
  that consumed it. Steady state is saved_entry=pve-zfs-root with no
  next_entry. There is no auto-fallback on this host and no IPMI.

Recovery needed no console: an idempotent cgroup2/devpts/shm remount
landed in the brief windows where exec succeeded. No data was lost, and
neither the DOM nor any pool was ever at risk.
2026-08-18 06:39:34 -07:00
..
proxmox-details.txt
scripts: refresh-proxmox-info.sh wrapper for PVE snapshots
2026-04-20 23:01:48 -07:00
README.md
feat(esh-pve-nas): cut PVE root over to ZFS on the mirrored NVMe
2026-08-18 06:39:34 -07:00
ssh-target
Add the four Proxmox hypervisors to the inventory
2026-04-20 14:58:17 -07:00
system-details.txt
Add the four Proxmox hypervisors to the inventory
2026-04-20 14:58:17 -07:00

README.md

esh-pve-nas

Second Proxmox VE host at the ESH home lab (esh-nas-pve.esteban.net), dedicated to NAS / storage duty.

Network

  • LAN IP: 10.0.50.55
  • FQDN: esh-nas-pve.esteban.net
  • Web UI: https://10.0.50.55:8006
  • SSH: root@esh-pve-nas (key auth)

Media services on this box sit at 10.0.50.56 (Plex) and 10.0.50.57 (Jellyfin) — references to esh-nas-pve in the homepage config point at those addresses.

Hardware

  • CPU: Intel Xeon W-1250 @ 3.30 GHz

  • RAM: 125.6 GB

  • Kernel: 6.8.12-13-pve (Proxmox 8.x)

  • Storage: root is nvme/ROOT/pve-1 on the mirrored NVMe pool since the 2026-08-18 migration. The USB Disk-on-Module (7.3 GB, ID_BUS=usb, NORELSYS 1081) still holds the ESP and /boot, but is out of the runtime I/O path — a USB bus reset no longer drops root from under a running hypervisor. Runbook: docs/runbooks/esh-pve-nas-boot-migration.md.

    DOM LVM layout:

    LV size role
    pve-root 6.04 G ext4 — the rollback, intact and unmounted, keeps its own kernel + initrd
    pve-boot 512 M ext4 — /boot, carved out of swap
    pve-swap 256 M swap, shrunk to make room

    /boot stays ext4 on the DOM on purpose: GRUB cannot read the nvme pool, which has encryption, large_dnode and zstd_compress enabled.

    ⚠ Device letters are not stable — the DOM was sdq before the reboot and sdl after. fstab uses /dev/pve/* and UUIDs; never write a rule against a bare sdX on this host.

    ⚠ There is no auto-fallback if a boot fails. grubenv lives on an LVM LV, which GRUB can read but not write, so grub-reboot's one-shot degrades to a sticky default (verified 2026-08-18: next_entry survived the boot that consumed it). Steady state is saved_entry=pve-zfs-root with no next_entry. A failed boot needs the console — and this box has no IPMI, no BMC, no serial console. Recovery is selecting Proxmox VE - ROLLBACK: ext4 root on the USB DOM at the GRUB menu.

  • ⚠ Never set the ZFS cachefile on one pool. zpool set cachefile=… flips the host from import-by-scan to import-by-cache; a cache holding only nvme leaves ssd and tank unimported at boot, which empties every CT 103 export. Set it on all three or none.

  • Pools: nvme (2× 931 GB NVMe mirror — 32 G used, 867 G free, holds every guest rootfs), ssd (4× 894 GB Intel SATA, 2 mirrors — 1.42 T free), tank (12× 14.6 TB raidz2 ×2 — 40 T of 175 T). Plus NFS /mnt/pve/tank-vmbu for VM backups.

  • ⚠ CT 103 esh-nas (10.0.50.50) runs on THIS host and serves hard NFS. Never reboot this host casually — quiesce the clients first. Measured from the server on 2026-08-18 (ss -tn '( sport = :2049 )' inside CT 103), there are five clients, not the two long documented here:

    client mounts opts
    esh-docker-vm 10.0.50.45 /mnt/books, /mnt/backup hard — quiesce
    esh-pve 10.0.250.35 /mnt/pve/esh-nas, /mnt/pve/tank-vmbu hard — quiesce
    esh-vm-db 10.0.50.60 /mnt/backup hard — no ssh; reach it via qm guest exec 101 on esh-pve
    vm-esh-nas 10.0.50.154 — is VM 104 on this host; stops with it
    nh3-dev 10.100.10.50 /mnt/books soft,ro — safe, errors rather than blocks

    Ask the server who its clients are; don't trust this table. It rots. ss on CT 103 is ground truth, and that is how the three undocumented clients surfaced. Playbooks: esh-cutover-1-quiesce-docker-vm.yaml, esh-cutover-2-quiesce-esh-pve.yaml, esh-cutover-5-restore-docker-vm.yaml.

Root-fs pressure: 77% of a 5.9 GB root (1.3 GB free) after the 2026-08-17 mitigation. Still not enough for the pending upgrade — 225 packages, 161 carrying deb12uN/security bumps, including a ~250 MB signed kernel that lands in /boot. ⚠ Migrate first, patch after: unpacking that into 1.3 GB of headroom risks wedging dpkg on a hypervisor running five guests. The host is on pve-manager/8.4.11 vs esh-pve's 8.4.14 for exactly this reason.

What it runs

Media + storage-adjacent workloads for the ESH site. Not yet catalogued here; qm list / pct list on the host is authoritative.

The ESH-FileBot entry in homepage (10.0.50.70, labeled "Restic Backup Target and File Sync — esh-nas-pve") is a workload running here — mentioned in the backup project as a possible additional restic endpoint; confirm role before integrating into the fleet plan.

Refresh state

scripts/refresh-server-info.sh esh-pve-nas

Placement rule

Hypervisor for ESH storage/media VMs. Non-PFI scope.

Reference in New Issue View Git Blame Copy Permalink