vh
30a7220b2c
Prime's ruling 2026-10-13. esh-nas (10.0.50.50) had no infra-ops user, so key login was refused. The same steps as bootstrap-infra-ops-user.yaml (uid/gid 850, fleet key, NOPASSWD sudoers with log_output and the BULK_DATA carve-out, visudo-validated) ran inside the container from its host. Verified with a direct ssh infra-ops@10.0.50.50 and sudo -n whoami = root.
103 lines
5.6 KiB
Markdown
103 lines
5.6 KiB
Markdown
# esh-pve-nas
|
||
|
||
Second Proxmox VE host at the ESH home lab (`esh-nas-pve.esteban.net`), dedicated to NAS / storage duty.
|
||
|
||
## Network
|
||
|
||
- **LAN IP:** 10.0.50.55
|
||
- **FQDN:** `esh-nas-pve.esteban.net`
|
||
- **Web UI:** `https://10.0.50.55:8006`
|
||
- **SSH:** `root@esh-pve-nas` (key auth)
|
||
|
||
Media services on this box sit at `10.0.50.56` (Plex) and `10.0.50.57` (Jellyfin) — references to `esh-nas-pve` in the homepage config point at those addresses.
|
||
|
||
## Hardware
|
||
|
||
- **CPU:** Intel Xeon W-1250 @ 3.30 GHz
|
||
- **RAM:** 125.6 GB
|
||
- **Kernel:** `6.8.12-13-pve` (Proxmox 8.x)
|
||
- **Storage: root is `nvme/ROOT/pve-1` on the mirrored NVMe pool** since the
|
||
2026-08-18 migration. The USB Disk-on-Module (7.3 GB, `ID_BUS=usb`, NORELSYS
|
||
1081) still holds the ESP and `/boot`, but is **out of the runtime I/O path** —
|
||
a USB bus reset no longer drops root from under a running hypervisor. Runbook:
|
||
[`docs/runbooks/esh-pve-nas-boot-migration.md`](../../docs/runbooks/esh-pve-nas-boot-migration.md).
|
||
|
||
DOM LVM layout:
|
||
|
||
| LV | size | role |
|
||
|---|---|---|
|
||
| `pve-root` | 6.04 G | ext4 — **the rollback**, intact and unmounted, keeps its own kernel + initrd |
|
||
| `pve-boot` | 512 M | ext4 — `/boot`, carved out of swap |
|
||
| `pve-swap` | 256 M | swap, shrunk to make room |
|
||
|
||
`/boot` stays ext4 on the DOM on purpose: GRUB cannot read the `nvme` pool,
|
||
which has `encryption`, `large_dnode` and `zstd_compress` enabled.
|
||
|
||
⚠ **Device letters are not stable** — the DOM was `sdq` before the reboot and
|
||
`sdl` after. `fstab` uses `/dev/pve/*` and UUIDs; never write a rule against a
|
||
bare `sdX` on this host.
|
||
|
||
⚠ **There is no auto-fallback if a boot fails.** grubenv lives on an LVM LV,
|
||
which GRUB can read but not write, so `grub-reboot`'s one-shot degrades to a
|
||
sticky default (verified 2026-08-18: `next_entry` survived the boot that
|
||
consumed it). Steady state is `saved_entry=pve-zfs-root` with no `next_entry`.
|
||
A failed boot needs the console — and this box has **no IPMI, no BMC, no serial
|
||
console**. Recovery is selecting `Proxmox VE - ROLLBACK: ext4 root on the USB
|
||
DOM` at the GRUB menu.
|
||
- ⚠ **Never set the ZFS cachefile on one pool.** `zpool set cachefile=…` flips the
|
||
host from import-by-scan to import-by-cache; a cache holding only `nvme` leaves
|
||
`ssd` and `tank` unimported at boot, which empties every CT 103 export. Set it on
|
||
all three or none.
|
||
- **Pools:** `nvme` (2× 931 GB NVMe mirror — 32 G used, 867 G free, holds every guest
|
||
rootfs), `ssd` (4× 894 GB Intel SATA, 2 mirrors — 1.42 T free), `tank`
|
||
(12× 14.6 TB raidz2 ×2 — 40 T of 175 T). Plus NFS `/mnt/pve/tank-vmbu` for VM backups.
|
||
- ⚠⚠ **THIS HOST IS HALF OF A 2-NODE PROXMOX CLUSTER** — `esh-pve-cluster`, nodes
|
||
`pve` (esh-pve, 10.0.250.35, nodeid 1) and `esh-nas-pve` (this host, nodeid 2).
|
||
`Expected votes: 2`, `Quorum: 2`, no qdevice. **Taking either node down drops the
|
||
survivor below quorum and makes its `/etc/pve` read-only** — guests keep running,
|
||
but no config change, no VM start/stop, no storage edit works until the partner
|
||
returns. This was undocumented and nearly bit the 2026-08-18 migration, which
|
||
rebooted this node without accounting for it. Before any planned reboot of either
|
||
node, either accept the read-only window on the survivor or set
|
||
`pvecm expected 1` on it for the duration. Discovered 2026-08-18.
|
||
- ⚠ **CT 103 `esh-nas` (10.0.50.50) runs on THIS host and serves `hard` NFS.** Since 2026-10-13 it has the `infra-ops` identity (uid 850, NOPASSWD sudo, Prime's ruling): `ssh infra-ops@10.0.50.50` works directly. Before that, the only way in was `pct exec 103` from this host.
|
||
Never reboot this host casually — quiesce the clients first. Measured from the
|
||
server on 2026-08-18 (`ss -tn '( sport = :2049 )'` inside CT 103), there are
|
||
**five** clients, not the two long documented here:
|
||
|
||
| client | mounts | opts |
|
||
|---|---|---|
|
||
| esh-docker-vm `10.0.50.45` | `/mnt/books`, `/mnt/backup` | **hard** — quiesce |
|
||
| esh-pve `10.0.250.35` | `/mnt/pve/esh-nas`, `/mnt/pve/tank-vmbu` | **hard** — quiesce |
|
||
| esh-vm-db `10.0.50.60` | `/mnt/backup` | **hard** — no ssh; reach it via `qm guest exec 101` on esh-pve |
|
||
| vm-esh-nas `10.0.50.154` | — | is VM 104 *on this host*; stops with it |
|
||
| nh3-dev `10.100.10.50` | `/mnt/books` | soft,ro — safe, errors rather than blocks |
|
||
|
||
**Ask the server who its clients are; don't trust this table.** It rots. `ss` on
|
||
CT 103 is ground truth, and that is how the three undocumented clients surfaced.
|
||
Playbooks: `esh-cutover-1-quiesce-docker-vm.yaml`,
|
||
`esh-cutover-2-quiesce-esh-pve.yaml`, `esh-cutover-5-restore-docker-vm.yaml`.
|
||
|
||
> **Root-fs pressure:** 77% of a 5.9 GB root (1.3 GB free) after the 2026-08-17
|
||
> mitigation. Still not enough for the pending upgrade — **225 packages, 161
|
||
> carrying `deb12uN`/security bumps, including a ~250 MB signed kernel that lands
|
||
> in `/boot`.** ⚠ **Migrate first, patch after:** unpacking that into 1.3 GB of
|
||
> headroom risks wedging dpkg on a hypervisor running five guests. The host is on
|
||
> `pve-manager/8.4.11` vs esh-pve's 8.4.14 for exactly this reason.
|
||
|
||
## What it runs
|
||
|
||
Media + storage-adjacent workloads for the ESH site. Not yet catalogued here; `qm list` / `pct list` on the host is authoritative.
|
||
|
||
The `ESH-FileBot` entry in homepage (`10.0.50.70`, labeled "Restic Backup Target and File Sync — esh-nas-pve") is a workload running here — mentioned in the backup project as a possible additional restic endpoint; confirm role before integrating into the fleet plan.
|
||
|
||
## Refresh state
|
||
|
||
```bash
|
||
scripts/refresh-server-info.sh esh-pve-nas
|
||
```
|
||
|
||
## Placement rule
|
||
|
||
Hypervisor for ESH storage/media VMs. Non-PFI scope.
|