vh
b84f8a996f
Two findings from chasing node `pve` showing dark in the UI. esh-pve and esh-pve-nas are a 2-node cluster (esh-pve-cluster, expected votes 2, quorum 2, no qdevice). This was undocumented, and last night's migration rebooted one of the members without accounting for it. Nothing broke -- quorum is intact and both nodes report the same ring id -- but that was luck. Rebooting either node drops the survivor below quorum and makes its /etc/pve read-only until the partner returns. It matters for the pending confirmation reboot and the 225-package upgrade, both of which take a node down. The dark tile itself was NOT last night's doing. pvestatd SEGV'd on 2026-05-28 and had been dead 82 days; the journal has nothing between that crash and the restart today. It is only the reporting daemon, so the node stayed quorate and healthy with all services active and all three guests running the whole time -- the UI simply had nothing telling it the node was alive. Fourth SEGV in that unit's history, so treat a recurrence as expected and consider a watchdog: nothing alerts on it, and the sole symptom is cosmetic enough to go unnoticed for months.
103 lines
5.4 KiB
Markdown
103 lines
5.4 KiB
Markdown
# esh-pve-nas
|
||
|
||
Second Proxmox VE host at the ESH home lab (`esh-nas-pve.esteban.net`), dedicated to NAS / storage duty.
|
||
|
||
## Network
|
||
|
||
- **LAN IP:** 10.0.50.55
|
||
- **FQDN:** `esh-nas-pve.esteban.net`
|
||
- **Web UI:** `https://10.0.50.55:8006`
|
||
- **SSH:** `root@esh-pve-nas` (key auth)
|
||
|
||
Media services on this box sit at `10.0.50.56` (Plex) and `10.0.50.57` (Jellyfin) — references to `esh-nas-pve` in the homepage config point at those addresses.
|
||
|
||
## Hardware
|
||
|
||
- **CPU:** Intel Xeon W-1250 @ 3.30 GHz
|
||
- **RAM:** 125.6 GB
|
||
- **Kernel:** `6.8.12-13-pve` (Proxmox 8.x)
|
||
- **Storage: root is `nvme/ROOT/pve-1` on the mirrored NVMe pool** since the
|
||
2026-08-18 migration. The USB Disk-on-Module (7.3 GB, `ID_BUS=usb`, NORELSYS
|
||
1081) still holds the ESP and `/boot`, but is **out of the runtime I/O path** —
|
||
a USB bus reset no longer drops root from under a running hypervisor. Runbook:
|
||
[`docs/runbooks/esh-pve-nas-boot-migration.md`](../../docs/runbooks/esh-pve-nas-boot-migration.md).
|
||
|
||
DOM LVM layout:
|
||
|
||
| LV | size | role |
|
||
|---|---|---|
|
||
| `pve-root` | 6.04 G | ext4 — **the rollback**, intact and unmounted, keeps its own kernel + initrd |
|
||
| `pve-boot` | 512 M | ext4 — `/boot`, carved out of swap |
|
||
| `pve-swap` | 256 M | swap, shrunk to make room |
|
||
|
||
`/boot` stays ext4 on the DOM on purpose: GRUB cannot read the `nvme` pool,
|
||
which has `encryption`, `large_dnode` and `zstd_compress` enabled.
|
||
|
||
⚠ **Device letters are not stable** — the DOM was `sdq` before the reboot and
|
||
`sdl` after. `fstab` uses `/dev/pve/*` and UUIDs; never write a rule against a
|
||
bare `sdX` on this host.
|
||
|
||
⚠ **There is no auto-fallback if a boot fails.** grubenv lives on an LVM LV,
|
||
which GRUB can read but not write, so `grub-reboot`'s one-shot degrades to a
|
||
sticky default (verified 2026-08-18: `next_entry` survived the boot that
|
||
consumed it). Steady state is `saved_entry=pve-zfs-root` with no `next_entry`.
|
||
A failed boot needs the console — and this box has **no IPMI, no BMC, no serial
|
||
console**. Recovery is selecting `Proxmox VE - ROLLBACK: ext4 root on the USB
|
||
DOM` at the GRUB menu.
|
||
- ⚠ **Never set the ZFS cachefile on one pool.** `zpool set cachefile=…` flips the
|
||
host from import-by-scan to import-by-cache; a cache holding only `nvme` leaves
|
||
`ssd` and `tank` unimported at boot, which empties every CT 103 export. Set it on
|
||
all three or none.
|
||
- **Pools:** `nvme` (2× 931 GB NVMe mirror — 32 G used, 867 G free, holds every guest
|
||
rootfs), `ssd` (4× 894 GB Intel SATA, 2 mirrors — 1.42 T free), `tank`
|
||
(12× 14.6 TB raidz2 ×2 — 40 T of 175 T). Plus NFS `/mnt/pve/tank-vmbu` for VM backups.
|
||
- ⚠⚠ **THIS HOST IS HALF OF A 2-NODE PROXMOX CLUSTER** — `esh-pve-cluster`, nodes
|
||
`pve` (esh-pve, 10.0.250.35, nodeid 1) and `esh-nas-pve` (this host, nodeid 2).
|
||
`Expected votes: 2`, `Quorum: 2`, no qdevice. **Taking either node down drops the
|
||
survivor below quorum and makes its `/etc/pve` read-only** — guests keep running,
|
||
but no config change, no VM start/stop, no storage edit works until the partner
|
||
returns. This was undocumented and nearly bit the 2026-08-18 migration, which
|
||
rebooted this node without accounting for it. Before any planned reboot of either
|
||
node, either accept the read-only window on the survivor or set
|
||
`pvecm expected 1` on it for the duration. Discovered 2026-08-18.
|
||
- ⚠ **CT 103 `esh-nas` (10.0.50.50) runs on THIS host and serves `hard` NFS.**
|
||
Never reboot this host casually — quiesce the clients first. Measured from the
|
||
server on 2026-08-18 (`ss -tn '( sport = :2049 )'` inside CT 103), there are
|
||
**five** clients, not the two long documented here:
|
||
|
||
| client | mounts | opts |
|
||
|---|---|---|
|
||
| esh-docker-vm `10.0.50.45` | `/mnt/books`, `/mnt/backup` | **hard** — quiesce |
|
||
| esh-pve `10.0.250.35` | `/mnt/pve/esh-nas`, `/mnt/pve/tank-vmbu` | **hard** — quiesce |
|
||
| esh-vm-db `10.0.50.60` | `/mnt/backup` | **hard** — no ssh; reach it via `qm guest exec 101` on esh-pve |
|
||
| vm-esh-nas `10.0.50.154` | — | is VM 104 *on this host*; stops with it |
|
||
| nh3-dev `10.100.10.50` | `/mnt/books` | soft,ro — safe, errors rather than blocks |
|
||
|
||
**Ask the server who its clients are; don't trust this table.** It rots. `ss` on
|
||
CT 103 is ground truth, and that is how the three undocumented clients surfaced.
|
||
Playbooks: `esh-cutover-1-quiesce-docker-vm.yaml`,
|
||
`esh-cutover-2-quiesce-esh-pve.yaml`, `esh-cutover-5-restore-docker-vm.yaml`.
|
||
|
||
> **Root-fs pressure:** 77% of a 5.9 GB root (1.3 GB free) after the 2026-08-17
|
||
> mitigation. Still not enough for the pending upgrade — **225 packages, 161
|
||
> carrying `deb12uN`/security bumps, including a ~250 MB signed kernel that lands
|
||
> in `/boot`.** ⚠ **Migrate first, patch after:** unpacking that into 1.3 GB of
|
||
> headroom risks wedging dpkg on a hypervisor running five guests. The host is on
|
||
> `pve-manager/8.4.11` vs esh-pve's 8.4.14 for exactly this reason.
|
||
|
||
## What it runs
|
||
|
||
Media + storage-adjacent workloads for the ESH site. Not yet catalogued here; `qm list` / `pct list` on the host is authoritative.
|
||
|
||
The `ESH-FileBot` entry in homepage (`10.0.50.70`, labeled "Restic Backup Target and File Sync — esh-nas-pve") is a workload running here — mentioned in the backup project as a possible additional restic endpoint; confirm role before integrating into the fleet plan.
|
||
|
||
## Refresh state
|
||
|
||
```bash
|
||
scripts/refresh-server-info.sh esh-pve-nas
|
||
```
|
||
|
||
## Placement rule
|
||
|
||
Hypervisor for ESH storage/media VMs. Non-PFI scope.
|