fix(esh): Plex hardware transcoding on the Arc A580, and the two ways it hid

Plex never hardware-transcoded on esh-pve-nas LXC 105 despite correct passthrough,
cgroups, group membership, authenticated HuC firmware, a lifetime Plex Pass,
HardwareAcceleratedCodecs=1, and the Arc already selected as HardwareDevicePath.

Root cause sat below all of that: intel-media-va-driver 22.3.1 (Apr 2023, stock
jammy) predates Arc/DG2 support and exports only __vaDriverInit_1_14, against the
libva 2.22 that Plex bundles and loads via RPATH rather than the system one. A
half-finished prior attempt at the same fix was also present -- libva and libva-drm
hand-installed at 2.22 with libva-x11 left at 2.14, breaking every X11 VA-API
consumer on va_fool_postp.

Fixed with Intel's client-GPU repo: iHD 24.3.4 (__vaDriverInit_1_22, an exact ABI
match) plus a consistent libva 2.22.0.2-87 set, which also brings the orphaned
manual install back under dpkg. The repo track is rolling, so the six packages are
pinned in /etc/apt/preferences.d/intel-gpu-pin and apt-mark held; verified by a
simulated upgrade that moves 152 packages and touches none of them.

Two findings worth more than the fix:

pct snapshot refuses on a guest with a bind mount and still exits 0, so a script
guarding a change with it proceeds without the rollback point it believes it has.
The ZFS dataset snapshot is the working path, verified by reading it back.

A synthetic Plex Transcoder invocation is not a valid test of Plex's transcode
path. Plex bundles its own libc among 61 libraries; the harness produced three
distinct failure modes that were artifacts of not reproducing that runtime, and it
failed identically before and after a fix that worked. With no positive control its
negatives carried no information. Only a forced transcode settles it, and PASS is
recognisable by Plex naming the device. The original empty decoder/encoder line was
an absence of evidence rather than evidence of failure -- TranscodeSession was 0.

Jellyfin LXC 107 has the same stale stack and the same Arc available; left alone
per the operator, and it ships its own ffmpeg so this may not transfer verbatim.

docs/runbooks/plex-arc-vaapi-jammy.md

@@ -0,0 +1,122 @@
+# Plex hardware transcoding on the Arc A580 (esh-pve-nas, LXC 105)
+
+**Fixed 2026-09-11.** Plex never hardware-transcoded on the Arc A580 despite every
+setting being correct. The fault was one layer below the settings, and the whole
+diagnosis is reusable for any Intel-GPU VA-API consumer on Ubuntu 22.04.
+
+## The symptom, and why it misleads
+
+Plex logged, on every transcode decision:
+
+    TPU: hardware transcoding: final decoder: , final encoder:
+
+Both fields empty. That reads like "hardware transcoding failed", but on its own it is
+**also** what you get when nothing transcoded at all — and `TranscodeSession` count was
+zero, so the log was not evidence of failure. Twenty of those lines landed inside one
+second on one thread: a capability probe loop, not twenty sessions.
+
+⚠ **Do not diagnose this from configuration.** Every one of these read correct while
+hardware transcoding was dead:
+
+| checked | state |
+|---|---|
+| Arc A580 present, DMC/GuC/HuC firmware | ✅ HuC "authenticated for all workloads" |
+| LXC passthrough: both render nodes + cgroup allows | ✅ |
+| `plex` user in `video(44)` + `render(104)` | ✅ |
+| Plex 1.43.2, lifetime Plex Pass | ✅ |
+| `HardwareAcceleratedCodecs=1` (via API, not the file) | ✅ |
+| `HardwareDevicePath` = `…@0000:03:00.0` (the Arc) | ✅ already selected |
+
+⚠ `HardwareAcceleratedCodecs` is **absent from `Preferences.xml` when enabled** — Plex
+only persists non-defaults. Read it from the API, never from the file:
+
+    curl -s "http://127.0.0.1:32400/:/prefs?X-Plex-Token=$TOKEN"
+
+## Root cause
+
+`intel-media-va-driver` **22.3.1** (Apr 2023, stock jammy) — predates Arc/DG2 support
+entirely, and exports only `__vaDriverInit_1_14` against Plex's **bundled** libva 2.22
+(`/usr/lib/plexmediaserver/lib/libva.so.2`, loaded via RPATH, not the system one).
+
+Compounding it: someone had previously hand-installed libva 2.22 over the packaged
+2.14 and left `libva-x11` behind at 2.14, so `vainfo` died with
+`undefined symbol: va_fool_postp` — a half-finished run at this same fix.
+
+## The fix
+
+```bash
+# 1. Snapshot. `pct snapshot` REFUSES on a guest with a bind mount (mp0: /tank/media)
+#    AND STILL EXITS 0 — snapshot the ZFS dataset directly and read it back.
+zfs snapshot nvme/subvol-105-disk-0@pre-vaapi-20260911
+zfs list -t snapshot nvme/subvol-105-disk-0@pre-vaapi-20260911   # VERIFY, don't trust rc
+
+# 2. Intel client-GPU repo (inside LXC 105)
+curl -fsS https://repositories.intel.com/gpu/intel-graphics.key \
+  | gpg --yes --dearmor -o /usr/share/keyrings/intel-graphics.gpg
+echo "deb [arch=amd64 signed-by=/usr/share/keyrings/intel-graphics.gpg] \
+https://repositories.intel.com/gpu/ubuntu jammy client" \
+  > /etc/apt/sources.list.d/intel-gpu-jammy.list
+apt-get update
+
+# 3. Consistent set — the driver AND all four libva packages together
+apt-get install -y intel-media-va-driver-non-free libva2 libva-drm2 libva-x11-2
+
+# 4. Pinned + held, because `jammy client` is a ROLLING track
+#    /etc/apt/preferences.d/intel-gpu-pin  +  apt-mark hold
+
+systemctl restart plexmediaserver
+```
+
+Landed: iHD **24.3.4** (`__vaDriverInit_1_22`), libva set **2.22.0.2-87**, libigdgmm12
+22.5.2 — and the orphaned manual libva is now dpkg-owned.
+
+## Verification — behaviour, not config
+
+⚠ **A synthetic `Plex Transcoder` invocation is not a valid test.** Running it from a
+shell produced three different failure modes (`unknown libva error`, then a libstdc++
+`__wmemmove_chk` relocation error) that were **artifacts of not reproducing Plex's
+bundled Conan runtime** — Plex ships its own libc among 61 bundled libraries. It failed
+identically before and after a fix that worked. No positive control existed, so its
+negatives were worthless.
+
+Force a real transcode and read Plex's own log:
+
+```bash
+TOKEN=$(sed -n 's/.*PlexOnlineToken="\([^"]*\)".*/\1/p' \
+  "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Preferences.xml")
+KEY=<a ratingKey from /library/sections/<n>/all>
+curl -s -o /dev/null "http://127.0.0.1:32400/video/:/transcode/universal/start.m3u8\
+?path=%2Flibrary%2Fmetadata%2F$KEY&mediaIndex=0&partIndex=0&protocol=hls\
+&directPlay=0&directStream=0&videoQuality=20&maxVideoBitrate=1500\
+&X-Plex-Token=$TOKEN&X-Plex-Client-Identifier=vaapi-verify&session=vaapiverify1"
+sleep 12
+grep -i "hardware transcoding: testing API" "…/Logs/Plex Media Server.log" | tail -3
+curl -s -o /dev/null "http://127.0.0.1:32400/video/:/transcode/universal/stop\
+?session=vaapiverify1&X-Plex-Token=$TOKEN"
+```
+
+PASS looks like this — the device is **named**, and both encoder and decoder are tested:
+
+    Codecs: testing h264_vaapi (encoder)
+    Codecs: hardware transcoding: testing API vaapi for device '/dev/dri/renderD129' (Intel DG2 [Arc A580])
+    Codecs: testing h264 (decoder) with hwdevice vaapi
+
+followed by `[FFMPEG] - Format 0x… -> bgra` surface enumeration, which only occurs
+after a successful `vaInitialize`. FAIL is the empty `final decoder: , final encoder:`.
+
+`vainfo --display drm --device /dev/dri/renderD129` is a useful *secondary* check (it
+now reports iHD 24.3.4 with H.264/HEVC VLD **and** EncSliceLP) but it exercises the
+system libva, not Plex's bundled one — so it can pass while Plex fails.
+
+## Rollback
+
+    pct stop 105
+    zfs rollback nvme/subvol-105-disk-0@pre-vaapi-20260911
+    pct start 105
+
+## Not done
+
+- **Jellyfin (LXC 107)** on the same host has the same stale stack and the same Arc
+  available. Left alone 2026-09-11 — operator: not actively used. It ships its own
+  ffmpeg, so this fix may not transfer verbatim.
+- LXC 105 has **152 packages pending upgrade** unrelated to this work.

persistent-memory.d/2026-09-11-plex-arc-vaapi.md

@@ -0,0 +1,80 @@
+# `[2026-09-11]` Plex never hardware-transcoded on the Arc, and every setting said it should
+
+Operator: *"I believe plex is running but I guess it's using cpu quicksync — anything to
+gain by moving it to a gpu?"* The premise had a fold in it (**Quick Sync *is* a GPU** —
+it's the iGPU's media engine), and the real answer was that the GPU was already wired up
+and Plex had been unable to use it.
+
+## The configuration was correct the entire time
+
+Plex is **LXC 105 (`vm-plex`) on esh-pve-nas**, 10.0.50.56. Verified good *before* any
+change: Arc A580 present with DMC/GuC/**HuC authenticated for all workloads**; both
+render nodes bind-mounted into the LXC with cgroup allows; `plex` in `video(44)` +
+`render(104)`; Plex 1.43.2; lifetime Plex Pass; `HardwareAcceleratedCodecs=1`; and
+`HardwareDevicePath` **already pointed at the Arc** (`…@0000:03:00.0`), not the iGPU.
+
+⚠ **`HardwareAcceleratedCodecs` is ABSENT from `Preferences.xml` when it is ENABLED** —
+Plex only persists non-defaults. I read its absence as "off" and was wrong; the API
+(`GET /:/prefs`) reported `1`. Read Plex settings from the API, never the file.
+
+## Root cause, one layer below every setting
+
+`intel-media-va-driver` **22.3.1 (Apr 2023, stock jammy)** — predates Arc/DG2 support
+entirely and exports only `__vaDriverInit_1_14`, against the libva **2.22** that Plex
+**bundles** and loads via RPATH (`/usr/lib/plexmediaserver/lib/libva.so.2`), not the
+system one.
+
+⚠ And a **half-finished prior attempt at this same fix** was sitting there: libva and
+libva-drm hand-installed at 2.22 (not dpkg-owned), `libva-x11` left at 2.14, so every
+X11 VA-API consumer died on `undefined symbol: va_fool_postp`. Upgrading a library
+without its companions is how you get a stack that is broken in a way no single package
+version explains.
+
+## Fix + pin
+
+Intel client-GPU repo (`https://repositories.intel.com/gpu/ubuntu jammy client`, a
+**rolling** track) → `intel-media-va-driver-non-free` **24.3.4** (`__vaDriverInit_1_22`,
+exact ABI match for Plex's libva), libva set **2.22.0.2-87**, libigdgmm12 22.5.2. The
+orphaned manual libva is now dpkg-owned. **Pinned** in
+`/etc/apt/preferences.d/intel-gpu-pin` **and** `apt-mark hold`, verified by a simulated
+upgrade moving 152 packages and touching none of the six.
+
+## ⚠⚠ Two footguns, both of the silent-wrong class
+
+**`pct snapshot` REFUSES on a guest with a bind mount AND STILL EXITS 0.** LXC 105 has
+`mp0: /tank/media`, so Proxmox printed `snapshot feature is not available` and returned
+`rc=0`. A script trusting that exit code believes it has a backup it does not have. The
+rootfs is on ZFS, so the working path is `zfs snapshot nvme/subvol-105-disk-0@<tag>` —
+**and read it back**, per [[feedback_unfalsifiable_at_write_time]].
+
+**A synthetic `Plex Transcoder` invocation is NOT a valid test of Plex's transcode
+path, and I burned several rounds proving it.** Plex bundles its own libc among 61
+libraries; running its ffmpeg from a shell produced three different failure modes
+(`unknown libva error`, then a libstdc++ `__wmemmove_chk` relocation error) that were
+artifacts of the harness, and it **failed identically before and after a fix that
+worked**. No positive control existed, so its negatives carried no information — the
+exact shape of the tag-detection specimen in the measurement-discipline rule.
+
+## What actually settles it
+
+A forced transcode, reading Plex's own log. PASS **names the device**:
+
+    Codecs: testing h264_vaapi (encoder)
+    Codecs: hardware transcoding: testing API vaapi for device '/dev/dri/renderD129' (Intel DG2 [Arc A580])
+    Codecs: testing h264 (decoder) with hwdevice vaapi
+
+plus `[FFMPEG] - Format 0x… -> bgra` surface enumeration, which only follows a
+successful `vaInitialize`. FAIL is `final decoder: , final encoder:` with the device
+never named. ⚠ And an empty-fields line is **also** what a server that never transcoded
+emits — `TranscodeSession` count was 0, so the original log was an *absence of
+evidence*, not evidence of failure. I called it failure first and had to withdraw that.
+
+`vainfo` is a secondary check only (now: iHD 24.3.4, H.264/HEVC VLD **and** EncSliceLP
+on the Arc) — it exercises the **system** libva, so it can pass while Plex fails.
+
+Runbook: `docs/runbooks/plex-arc-vaapi-jammy.md`.
+Rollback: `pct stop 105; zfs rollback nvme/subvol-105-disk-0@pre-vaapi-20260911; pct start 105`.
+
+**Left alone:** Jellyfin LXC 107 on the same host has the same stale stack and the same
+Arc available — operator 2026-09-11: not actively used. It ships its own ffmpeg so this
+fix may not transfer verbatim. LXC 105 also has 152 unrelated pending package upgrades.

persistent-memory.md

@@ -150,6 +150,8 @@ _As of 2026-09-11 ~17:45 PT._
 
 ## Recent decisions
 
+- `[2026-09-11]` ⭐ **Plex hardware transcoding on the Arc A580 FIXED (esh-pve-nas LXC 105) — every setting was already correct and the fault was one layer below them.** `intel-media-va-driver` **22.3.1** (Apr 2023, stock jammy) predates Arc/DG2 support and exports only `__vaDriverInit_1_14`, against the libva **2.22 Plex BUNDLES** and loads via RPATH. Passthrough, cgroups, `plex` in video+render, HuC authenticated, Plex Pass, `HardwareAcceleratedCodecs=1` and the Arc already selected as `HardwareDevicePath` — all good the whole time. Fixed with Intel's client-GPU repo (rolling `jammy client`) → iHD **24.3.4** (`__vaDriverInit_1_22`) + a consistent libva **2.22.0.2-87** set, now **pinned + `apt-mark hold`** (verified: a simulated upgrade moves 152 packages, touches none of the six). Also repaired a **half-finished prior attempt** — libva/libva-drm hand-installed at 2.22 with `libva-x11` left at 2.14, killing every X11 VA-API app on `va_fool_postp`. ⚠⚠ **`pct snapshot` REFUSES on a bind-mounted guest AND STILL EXITS 0** (LXC 105 has `mp0: /tank/media`) — use `zfs snapshot nvme/subvol-105-disk-0@<tag>` and read it back. ⚠⚠ **A synthetic `Plex Transcoder` run is NOT a valid test** (Plex bundles its own libc among 61 libs; my harness failed identically before and after a fix that worked — no positive control, so its negatives were worthless). Only a **forced transcode** settles it: PASS *names the device* (`testing API vaapi for device '/dev/dri/renderD129' (Intel DG2 [Arc A580])`). ⚠ The original empty `final decoder: , final encoder:` was an **absence of evidence**, not failure — `TranscodeSession` was 0. Jellyfin LXC 107 left alone (operator: not actively used). → `persistent-memory.d/2026-09-11-plex-arc-vaapi.md`, runbook `docs/runbooks/plex-arc-vaapi-jammy.md`
+
 - `[2026-09-11]` **Beszel priority 2 complete:** both DB hosts and both PBS hosts verified, 16 new alerts; fleet 17/18 up (ana-ml2 down). → `persistent-memory.d/2026-09-11-beszel-priority2.md`
 
 - `[2026-09-11]` **Beszel priority 1 complete: all six installed and verified.** After Anaheim recovery, live Synology samples and alerts verified; fleet 13/14 up, only known ana-ml2 outage remains. Configs not committed. → `persistent-memory.d/2026-09-11-beszel-priority1.md`