vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
8758c359bc2c7bfbc0a78e18bfd0316efd435f51
esh-pfi-infrastructure/servers/esh-pve-nas/README.md
T
vh 30a7220b2c ops(esh-nas): infra-ops identity bootstrapped in CT 103 via pct from esh-pve-nas 
Prime's ruling 2026-10-13. esh-nas (10.0.50.50) had no infra-ops user, so key
login was refused. The same steps as bootstrap-infra-ops-user.yaml (uid/gid
850, fleet key, NOPASSWD sudoers with log_output and the BULK_DATA carve-out,
visudo-validated) ran inside the container from its host. Verified with a
direct ssh infra-ops@10.0.50.50 and sudo -n whoami = root.
2026-10-13 13:25:21 -07:00

5.6 KiB
Raw Blame History

esh-pve-nas

Second Proxmox VE host at the ESH home lab (esh-nas-pve.esteban.net), dedicated to NAS / storage duty.

Network

  • LAN IP: 10.0.50.55
  • FQDN: esh-nas-pve.esteban.net
  • Web UI: https://10.0.50.55:8006
  • SSH: root@esh-pve-nas (key auth)

Media services on this box sit at 10.0.50.56 (Plex) and 10.0.50.57 (Jellyfin) — references to esh-nas-pve in the homepage config point at those addresses.

Hardware

  • CPU: Intel Xeon W-1250 @ 3.30 GHz

  • RAM: 125.6 GB

  • Kernel: 6.8.12-13-pve (Proxmox 8.x)

  • Storage: root is nvme/ROOT/pve-1 on the mirrored NVMe pool since the 2026-08-18 migration. The USB Disk-on-Module (7.3 GB, ID_BUS=usb, NORELSYS 1081) still holds the ESP and /boot, but is out of the runtime I/O path — a USB bus reset no longer drops root from under a running hypervisor. Runbook: docs/runbooks/esh-pve-nas-boot-migration.md.

    DOM LVM layout:

    LV size role
    pve-root 6.04 G ext4 — the rollback, intact and unmounted, keeps its own kernel + initrd
    pve-boot 512 M ext4 — /boot, carved out of swap
    pve-swap 256 M swap, shrunk to make room

    /boot stays ext4 on the DOM on purpose: GRUB cannot read the nvme pool, which has encryption, large_dnode and zstd_compress enabled.

    ⚠ Device letters are not stable — the DOM was sdq before the reboot and sdl after. fstab uses /dev/pve/* and UUIDs; never write a rule against a bare sdX on this host.

    ⚠ There is no auto-fallback if a boot fails. grubenv lives on an LVM LV, which GRUB can read but not write, so grub-reboot's one-shot degrades to a sticky default (verified 2026-08-18: next_entry survived the boot that consumed it). Steady state is saved_entry=pve-zfs-root with no next_entry. A failed boot needs the console — and this box has no IPMI, no BMC, no serial console. Recovery is selecting Proxmox VE - ROLLBACK: ext4 root on the USB DOM at the GRUB menu.

  • ⚠ Never set the ZFS cachefile on one pool. zpool set cachefile=… flips the host from import-by-scan to import-by-cache; a cache holding only nvme leaves ssd and tank unimported at boot, which empties every CT 103 export. Set it on all three or none.

  • Pools: nvme (2× 931 GB NVMe mirror — 32 G used, 867 G free, holds every guest rootfs), ssd (4× 894 GB Intel SATA, 2 mirrors — 1.42 T free), tank (12× 14.6 TB raidz2 ×2 — 40 T of 175 T). Plus NFS /mnt/pve/tank-vmbu for VM backups.

  • ⚠⚠ THIS HOST IS HALF OF A 2-NODE PROXMOX CLUSTER — esh-pve-cluster, nodes pve (esh-pve, 10.0.250.35, nodeid 1) and esh-nas-pve (this host, nodeid 2). Expected votes: 2, Quorum: 2, no qdevice. Taking either node down drops the survivor below quorum and makes its /etc/pve read-only — guests keep running, but no config change, no VM start/stop, no storage edit works until the partner returns. This was undocumented and nearly bit the 2026-08-18 migration, which rebooted this node without accounting for it. Before any planned reboot of either node, either accept the read-only window on the survivor or set pvecm expected 1 on it for the duration. Discovered 2026-08-18.

  • ⚠ CT 103 esh-nas (10.0.50.50) runs on THIS host and serves hard NFS. Since 2026-10-13 it has the infra-ops identity (uid 850, NOPASSWD sudo, Prime's ruling): ssh infra-ops@10.0.50.50 works directly. Before that, the only way in was pct exec 103 from this host. Never reboot this host casually — quiesce the clients first. Measured from the server on 2026-08-18 (ss -tn '( sport = :2049 )' inside CT 103), there are five clients, not the two long documented here:

    client mounts opts
    esh-docker-vm 10.0.50.45 /mnt/books, /mnt/backup hard — quiesce
    esh-pve 10.0.250.35 /mnt/pve/esh-nas, /mnt/pve/tank-vmbu hard — quiesce
    esh-vm-db 10.0.50.60 /mnt/backup hard — no ssh; reach it via qm guest exec 101 on esh-pve
    vm-esh-nas 10.0.50.154 — is VM 104 on this host; stops with it
    nh3-dev 10.100.10.50 /mnt/books soft,ro — safe, errors rather than blocks

    Ask the server who its clients are; don't trust this table. It rots. ss on CT 103 is ground truth, and that is how the three undocumented clients surfaced. Playbooks: esh-cutover-1-quiesce-docker-vm.yaml, esh-cutover-2-quiesce-esh-pve.yaml, esh-cutover-5-restore-docker-vm.yaml.

Root-fs pressure: 77% of a 5.9 GB root (1.3 GB free) after the 2026-08-17 mitigation. Still not enough for the pending upgrade — 225 packages, 161 carrying deb12uN/security bumps, including a ~250 MB signed kernel that lands in /boot. ⚠ Migrate first, patch after: unpacking that into 1.3 GB of headroom risks wedging dpkg on a hypervisor running five guests. The host is on pve-manager/8.4.11 vs esh-pve's 8.4.14 for exactly this reason.

What it runs

Media + storage-adjacent workloads for the ESH site. Not yet catalogued here; qm list / pct list on the host is authoritative.

The ESH-FileBot entry in homepage (10.0.50.70, labeled "Restic Backup Target and File Sync — esh-nas-pve") is a workload running here — mentioned in the backup project as a possible additional restic endpoint; confirm role before integrating into the fleet plan.

Refresh state

scripts/refresh-server-info.sh esh-pve-nas

Placement rule

Hypervisor for ESH storage/media VMs. Non-PFI scope.

Reference in New Issue View Git Blame Copy Permalink