diff --git a/docs/runbooks/plex-arc-vaapi-jammy.md b/docs/runbooks/plex-arc-vaapi-jammy.md new file mode 100644 index 0000000..6346405 --- /dev/null +++ b/docs/runbooks/plex-arc-vaapi-jammy.md @@ -0,0 +1,122 @@ +# Plex hardware transcoding on the Arc A580 (esh-pve-nas, LXC 105) + +**Fixed 2026-09-11.** Plex never hardware-transcoded on the Arc A580 despite every +setting being correct. The fault was one layer below the settings, and the whole +diagnosis is reusable for any Intel-GPU VA-API consumer on Ubuntu 22.04. + +## The symptom, and why it misleads + +Plex logged, on every transcode decision: + + TPU: hardware transcoding: final decoder: , final encoder: + +Both fields empty. That reads like "hardware transcoding failed", but on its own it is +**also** what you get when nothing transcoded at all — and `TranscodeSession` count was +zero, so the log was not evidence of failure. Twenty of those lines landed inside one +second on one thread: a capability probe loop, not twenty sessions. + +⚠ **Do not diagnose this from configuration.** Every one of these read correct while +hardware transcoding was dead: + +| checked | state | +|---|---| +| Arc A580 present, DMC/GuC/HuC firmware | ✅ HuC "authenticated for all workloads" | +| LXC passthrough: both render nodes + cgroup allows | ✅ | +| `plex` user in `video(44)` + `render(104)` | ✅ | +| Plex 1.43.2, lifetime Plex Pass | ✅ | +| `HardwareAcceleratedCodecs=1` (via API, not the file) | ✅ | +| `HardwareDevicePath` = `…@0000:03:00.0` (the Arc) | ✅ already selected | + +⚠ `HardwareAcceleratedCodecs` is **absent from `Preferences.xml` when enabled** — Plex +only persists non-defaults. Read it from the API, never from the file: + + curl -s "http://127.0.0.1:32400/:/prefs?X-Plex-Token=$TOKEN" + +## Root cause + +`intel-media-va-driver` **22.3.1** (Apr 2023, stock jammy) — predates Arc/DG2 support +entirely, and exports only `__vaDriverInit_1_14` against Plex's **bundled** libva 2.22 +(`/usr/lib/plexmediaserver/lib/libva.so.2`, loaded via RPATH, not the system one). + +Compounding it: someone had previously hand-installed libva 2.22 over the packaged +2.14 and left `libva-x11` behind at 2.14, so `vainfo` died with +`undefined symbol: va_fool_postp` — a half-finished run at this same fix. + +## The fix + +```bash +# 1. Snapshot. `pct snapshot` REFUSES on a guest with a bind mount (mp0: /tank/media) +# AND STILL EXITS 0 — snapshot the ZFS dataset directly and read it back. +zfs snapshot nvme/subvol-105-disk-0@pre-vaapi-20260911 +zfs list -t snapshot nvme/subvol-105-disk-0@pre-vaapi-20260911 # VERIFY, don't trust rc + +# 2. Intel client-GPU repo (inside LXC 105) +curl -fsS https://repositories.intel.com/gpu/intel-graphics.key \ + | gpg --yes --dearmor -o /usr/share/keyrings/intel-graphics.gpg +echo "deb [arch=amd64 signed-by=/usr/share/keyrings/intel-graphics.gpg] \ +https://repositories.intel.com/gpu/ubuntu jammy client" \ + > /etc/apt/sources.list.d/intel-gpu-jammy.list +apt-get update + +# 3. Consistent set — the driver AND all four libva packages together +apt-get install -y intel-media-va-driver-non-free libva2 libva-drm2 libva-x11-2 + +# 4. Pinned + held, because `jammy client` is a ROLLING track +# /etc/apt/preferences.d/intel-gpu-pin + apt-mark hold + +systemctl restart plexmediaserver +``` + +Landed: iHD **24.3.4** (`__vaDriverInit_1_22`), libva set **2.22.0.2-87**, libigdgmm12 +22.5.2 — and the orphaned manual libva is now dpkg-owned. + +## Verification — behaviour, not config + +⚠ **A synthetic `Plex Transcoder` invocation is not a valid test.** Running it from a +shell produced three different failure modes (`unknown libva error`, then a libstdc++ +`__wmemmove_chk` relocation error) that were **artifacts of not reproducing Plex's +bundled Conan runtime** — Plex ships its own libc among 61 bundled libraries. It failed +identically before and after a fix that worked. No positive control existed, so its +negatives were worthless. + +Force a real transcode and read Plex's own log: + +```bash +TOKEN=$(sed -n 's/.*PlexOnlineToken="\([^"]*\)".*/\1/p' \ + "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Preferences.xml") +KEY=/all> +curl -s -o /dev/null "http://127.0.0.1:32400/video/:/transcode/universal/start.m3u8\ +?path=%2Flibrary%2Fmetadata%2F$KEY&mediaIndex=0&partIndex=0&protocol=hls\ +&directPlay=0&directStream=0&videoQuality=20&maxVideoBitrate=1500\ +&X-Plex-Token=$TOKEN&X-Plex-Client-Identifier=vaapi-verify&session=vaapiverify1" +sleep 12 +grep -i "hardware transcoding: testing API" "…/Logs/Plex Media Server.log" | tail -3 +curl -s -o /dev/null "http://127.0.0.1:32400/video/:/transcode/universal/stop\ +?session=vaapiverify1&X-Plex-Token=$TOKEN" +``` + +PASS looks like this — the device is **named**, and both encoder and decoder are tested: + + Codecs: testing h264_vaapi (encoder) + Codecs: hardware transcoding: testing API vaapi for device '/dev/dri/renderD129' (Intel DG2 [Arc A580]) + Codecs: testing h264 (decoder) with hwdevice vaapi + +followed by `[FFMPEG] - Format 0x… -> bgra` surface enumeration, which only occurs +after a successful `vaInitialize`. FAIL is the empty `final decoder: , final encoder:`. + +`vainfo --display drm --device /dev/dri/renderD129` is a useful *secondary* check (it +now reports iHD 24.3.4 with H.264/HEVC VLD **and** EncSliceLP) but it exercises the +system libva, not Plex's bundled one — so it can pass while Plex fails. + +## Rollback + + pct stop 105 + zfs rollback nvme/subvol-105-disk-0@pre-vaapi-20260911 + pct start 105 + +## Not done + +- **Jellyfin (LXC 107)** on the same host has the same stale stack and the same Arc + available. Left alone 2026-09-11 — operator: not actively used. It ships its own + ffmpeg, so this fix may not transfer verbatim. +- LXC 105 has **152 packages pending upgrade** unrelated to this work. diff --git a/persistent-memory.d/2026-09-11-plex-arc-vaapi.md b/persistent-memory.d/2026-09-11-plex-arc-vaapi.md new file mode 100644 index 0000000..1851d22 --- /dev/null +++ b/persistent-memory.d/2026-09-11-plex-arc-vaapi.md @@ -0,0 +1,80 @@ +# `[2026-09-11]` Plex never hardware-transcoded on the Arc, and every setting said it should + +Operator: *"I believe plex is running but I guess it's using cpu quicksync — anything to +gain by moving it to a gpu?"* The premise had a fold in it (**Quick Sync *is* a GPU** — +it's the iGPU's media engine), and the real answer was that the GPU was already wired up +and Plex had been unable to use it. + +## The configuration was correct the entire time + +Plex is **LXC 105 (`vm-plex`) on esh-pve-nas**, 10.0.50.56. Verified good *before* any +change: Arc A580 present with DMC/GuC/**HuC authenticated for all workloads**; both +render nodes bind-mounted into the LXC with cgroup allows; `plex` in `video(44)` + +`render(104)`; Plex 1.43.2; lifetime Plex Pass; `HardwareAcceleratedCodecs=1`; and +`HardwareDevicePath` **already pointed at the Arc** (`…@0000:03:00.0`), not the iGPU. + +⚠ **`HardwareAcceleratedCodecs` is ABSENT from `Preferences.xml` when it is ENABLED** — +Plex only persists non-defaults. I read its absence as "off" and was wrong; the API +(`GET /:/prefs`) reported `1`. Read Plex settings from the API, never the file. + +## Root cause, one layer below every setting + +`intel-media-va-driver` **22.3.1 (Apr 2023, stock jammy)** — predates Arc/DG2 support +entirely and exports only `__vaDriverInit_1_14`, against the libva **2.22** that Plex +**bundles** and loads via RPATH (`/usr/lib/plexmediaserver/lib/libva.so.2`), not the +system one. + +⚠ And a **half-finished prior attempt at this same fix** was sitting there: libva and +libva-drm hand-installed at 2.22 (not dpkg-owned), `libva-x11` left at 2.14, so every +X11 VA-API consumer died on `undefined symbol: va_fool_postp`. Upgrading a library +without its companions is how you get a stack that is broken in a way no single package +version explains. + +## Fix + pin + +Intel client-GPU repo (`https://repositories.intel.com/gpu/ubuntu jammy client`, a +**rolling** track) → `intel-media-va-driver-non-free` **24.3.4** (`__vaDriverInit_1_22`, +exact ABI match for Plex's libva), libva set **2.22.0.2-87**, libigdgmm12 22.5.2. The +orphaned manual libva is now dpkg-owned. **Pinned** in +`/etc/apt/preferences.d/intel-gpu-pin` **and** `apt-mark hold`, verified by a simulated +upgrade moving 152 packages and touching none of the six. + +## ⚠⚠ Two footguns, both of the silent-wrong class + +**`pct snapshot` REFUSES on a guest with a bind mount AND STILL EXITS 0.** LXC 105 has +`mp0: /tank/media`, so Proxmox printed `snapshot feature is not available` and returned +`rc=0`. A script trusting that exit code believes it has a backup it does not have. The +rootfs is on ZFS, so the working path is `zfs snapshot nvme/subvol-105-disk-0@` — +**and read it back**, per [[feedback_unfalsifiable_at_write_time]]. + +**A synthetic `Plex Transcoder` invocation is NOT a valid test of Plex's transcode +path, and I burned several rounds proving it.** Plex bundles its own libc among 61 +libraries; running its ffmpeg from a shell produced three different failure modes +(`unknown libva error`, then a libstdc++ `__wmemmove_chk` relocation error) that were +artifacts of the harness, and it **failed identically before and after a fix that +worked**. No positive control existed, so its negatives carried no information — the +exact shape of the tag-detection specimen in the measurement-discipline rule. + +## What actually settles it + +A forced transcode, reading Plex's own log. PASS **names the device**: + + Codecs: testing h264_vaapi (encoder) + Codecs: hardware transcoding: testing API vaapi for device '/dev/dri/renderD129' (Intel DG2 [Arc A580]) + Codecs: testing h264 (decoder) with hwdevice vaapi + +plus `[FFMPEG] - Format 0x… -> bgra` surface enumeration, which only follows a +successful `vaInitialize`. FAIL is `final decoder: , final encoder:` with the device +never named. ⚠ And an empty-fields line is **also** what a server that never transcoded +emits — `TranscodeSession` count was 0, so the original log was an *absence of +evidence*, not evidence of failure. I called it failure first and had to withdraw that. + +`vainfo` is a secondary check only (now: iHD 24.3.4, H.264/HEVC VLD **and** EncSliceLP +on the Arc) — it exercises the **system** libva, so it can pass while Plex fails. + +Runbook: `docs/runbooks/plex-arc-vaapi-jammy.md`. +Rollback: `pct stop 105; zfs rollback nvme/subvol-105-disk-0@pre-vaapi-20260911; pct start 105`. + +**Left alone:** Jellyfin LXC 107 on the same host has the same stale stack and the same +Arc available — operator 2026-09-11: not actively used. It ships its own ffmpeg so this +fix may not transfer verbatim. LXC 105 also has 152 unrelated pending package upgrades. diff --git a/persistent-memory.md b/persistent-memory.md index 8f69be3..8f6f26a 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -150,6 +150,8 @@ _As of 2026-09-11 ~17:45 PT._ ## Recent decisions +- `[2026-09-11]` ⭐ **Plex hardware transcoding on the Arc A580 FIXED (esh-pve-nas LXC 105) — every setting was already correct and the fault was one layer below them.** `intel-media-va-driver` **22.3.1** (Apr 2023, stock jammy) predates Arc/DG2 support and exports only `__vaDriverInit_1_14`, against the libva **2.22 Plex BUNDLES** and loads via RPATH. Passthrough, cgroups, `plex` in video+render, HuC authenticated, Plex Pass, `HardwareAcceleratedCodecs=1` and the Arc already selected as `HardwareDevicePath` — all good the whole time. Fixed with Intel's client-GPU repo (rolling `jammy client`) → iHD **24.3.4** (`__vaDriverInit_1_22`) + a consistent libva **2.22.0.2-87** set, now **pinned + `apt-mark hold`** (verified: a simulated upgrade moves 152 packages, touches none of the six). Also repaired a **half-finished prior attempt** — libva/libva-drm hand-installed at 2.22 with `libva-x11` left at 2.14, killing every X11 VA-API app on `va_fool_postp`. ⚠⚠ **`pct snapshot` REFUSES on a bind-mounted guest AND STILL EXITS 0** (LXC 105 has `mp0: /tank/media`) — use `zfs snapshot nvme/subvol-105-disk-0@` and read it back. ⚠⚠ **A synthetic `Plex Transcoder` run is NOT a valid test** (Plex bundles its own libc among 61 libs; my harness failed identically before and after a fix that worked — no positive control, so its negatives were worthless). Only a **forced transcode** settles it: PASS *names the device* (`testing API vaapi for device '/dev/dri/renderD129' (Intel DG2 [Arc A580])`). ⚠ The original empty `final decoder: , final encoder:` was an **absence of evidence**, not failure — `TranscodeSession` was 0. Jellyfin LXC 107 left alone (operator: not actively used). → `persistent-memory.d/2026-09-11-plex-arc-vaapi.md`, runbook `docs/runbooks/plex-arc-vaapi-jammy.md` + - `[2026-09-11]` **Beszel priority 2 complete:** both DB hosts and both PBS hosts verified, 16 new alerts; fleet 17/18 up (ana-ml2 down). → `persistent-memory.d/2026-09-11-beszel-priority2.md` - `[2026-09-11]` **Beszel priority 1 complete: all six installed and verified.** After Anaheim recovery, live Synology samples and alerts verified; fleet 13/14 up, only known ana-ml2 outage remains. Configs not committed. → `persistent-memory.d/2026-09-11-beszel-priority1.md`