# esh-pve-nas

Second Proxmox VE host at the ESH home lab (`esh-nas-pve.esteban.net`), dedicated to NAS / storage duty.

## Network

- **LAN IP:** 10.0.50.55
- **FQDN:** `esh-nas-pve.esteban.net`
- **Web UI:** `https://10.0.50.55:8006`
- **SSH:** `root@esh-pve-nas` (key auth)

Media services on this box sit at `10.0.50.56` (Plex) and `10.0.50.57` (Jellyfin) — references to `esh-nas-pve` in the homepage config point at those addresses.

## Hardware

- **CPU:** Intel Xeon W-1250 @ 3.30 GHz
- **RAM:** 125.6 GB
- **Kernel:** `6.8.12-13-pve` (Proxmox 8.x)
- **Storage: root is `nvme/ROOT/pve-1` on the mirrored NVMe pool** since the
  2026-08-18 migration. The USB Disk-on-Module (7.3 GB, `ID_BUS=usb`, NORELSYS
  1081) still holds the ESP and `/boot`, but is **out of the runtime I/O path** —
  a USB bus reset no longer drops root from under a running hypervisor. Runbook:
  [`docs/runbooks/esh-pve-nas-boot-migration.md`](../../docs/runbooks/esh-pve-nas-boot-migration.md).

  DOM LVM layout:

  | LV | size | role |
  |---|---|---|
  | `pve-root` | 6.04 G | ext4 — **the rollback**, intact and unmounted, keeps its own kernel + initrd |
  | `pve-boot` | 512 M | ext4 — `/boot`, carved out of swap |
  | `pve-swap` | 256 M | swap, shrunk to make room |

  `/boot` stays ext4 on the DOM on purpose: GRUB cannot read the `nvme` pool,
  which has `encryption`, `large_dnode` and `zstd_compress` enabled.

  ⚠ **Device letters are not stable** — the DOM was `sdq` before the reboot and
  `sdl` after. `fstab` uses `/dev/pve/*` and UUIDs; never write a rule against a
  bare `sdX` on this host.

  ⚠ **There is no auto-fallback if a boot fails.** grubenv lives on an LVM LV,
  which GRUB can read but not write, so `grub-reboot`'s one-shot degrades to a
  sticky default (verified 2026-08-18: `next_entry` survived the boot that
  consumed it). Steady state is `saved_entry=pve-zfs-root` with no `next_entry`.
  A failed boot needs the console — and this box has **no IPMI, no BMC, no serial
  console**. Recovery is selecting `Proxmox VE - ROLLBACK: ext4 root on the USB
  DOM` at the GRUB menu.
- ⚠ **Never set the ZFS cachefile on one pool.** `zpool set cachefile=…` flips the
  host from import-by-scan to import-by-cache; a cache holding only `nvme` leaves
  `ssd` and `tank` unimported at boot, which empties every CT 103 export. Set it on
  all three or none.
- **Pools:** `nvme` (2× 931 GB NVMe mirror — 32 G used, 867 G free, holds every guest
  rootfs), `ssd` (4× 894 GB Intel SATA, 2 mirrors — 1.42 T free), `tank`
  (12× 14.6 TB raidz2 ×2 — 40 T of 175 T). Plus NFS `/mnt/pve/tank-vmbu` for VM backups.
- ⚠⚠ **THIS HOST IS HALF OF A 2-NODE PROXMOX CLUSTER** — `esh-pve-cluster`, nodes
  `pve` (esh-pve, 10.0.250.35, nodeid 1) and `esh-nas-pve` (this host, nodeid 2).
  `Expected votes: 2`, `Quorum: 2`, no qdevice. **Taking either node down drops the
  survivor below quorum and makes its `/etc/pve` read-only** — guests keep running,
  but no config change, no VM start/stop, no storage edit works until the partner
  returns. This was undocumented and nearly bit the 2026-08-18 migration, which
  rebooted this node without accounting for it. Before any planned reboot of either
  node, either accept the read-only window on the survivor or set
  `pvecm expected 1` on it for the duration. Discovered 2026-08-18.
- ⚠ **CT 103 `esh-nas` (10.0.50.50) runs on THIS host and serves `hard` NFS.**
  Never reboot this host casually — quiesce the clients first. Measured from the
  server on 2026-08-18 (`ss -tn '( sport = :2049 )'` inside CT 103), there are
  **five** clients, not the two long documented here:

  | client | mounts | opts |
  |---|---|---|
  | esh-docker-vm `10.0.50.45` | `/mnt/books`, `/mnt/backup` | **hard** — quiesce |
  | esh-pve `10.0.250.35` | `/mnt/pve/esh-nas`, `/mnt/pve/tank-vmbu` | **hard** — quiesce |
  | esh-vm-db `10.0.50.60` | `/mnt/backup` | **hard** — no ssh; reach it via `qm guest exec 101` on esh-pve |
  | vm-esh-nas `10.0.50.154` | — | is VM 104 *on this host*; stops with it |
  | nh3-dev `10.100.10.50` | `/mnt/books` | soft,ro — safe, errors rather than blocks |

  **Ask the server who its clients are; don't trust this table.** It rots. `ss` on
  CT 103 is ground truth, and that is how the three undocumented clients surfaced.
  Playbooks: `esh-cutover-1-quiesce-docker-vm.yaml`,
  `esh-cutover-2-quiesce-esh-pve.yaml`, `esh-cutover-5-restore-docker-vm.yaml`.

> **Root-fs pressure:** 77% of a 5.9 GB root (1.3 GB free) after the 2026-08-17
> mitigation. Still not enough for the pending upgrade — **225 packages, 161
> carrying `deb12uN`/security bumps, including a ~250 MB signed kernel that lands
> in `/boot`.** ⚠ **Migrate first, patch after:** unpacking that into 1.3 GB of
> headroom risks wedging dpkg on a hypervisor running five guests. The host is on
> `pve-manager/8.4.11` vs esh-pve's 8.4.14 for exactly this reason.

## What it runs

Media + storage-adjacent workloads for the ESH site. Not yet catalogued here; `qm list` / `pct list` on the host is authoritative.

The `ESH-FileBot` entry in homepage (`10.0.50.70`, labeled "Restic Backup Target and File Sync — esh-nas-pve") is a workload running here — mentioned in the backup project as a possible additional restic endpoint; confirm role before integrating into the fleet plan.

## Refresh state

```bash
scripts/refresh-server-info.sh esh-pve-nas
```

## Placement rule

Hypervisor for ESH storage/media VMs. Non-PFI scope.