Video offload: Mesa etnaviv in the image, forgectrl pinned, the H.264 stream in the catalog

The stream conversion now runs on the GC880 when the image carries the
GL stack, and /cam/h264 serves the same picture as fragmented MP4 in a
fraction of MJPEG's bytes (forgectrl 6573abd). This side supplies what
that needs and holds it to account:

- Distro: opengl stays a feature, with mesa trimmed to exactly the
  path used (gallium etnaviv, GLES/EGL/GBM, no GLX, no display
  platforms). Image: libegl-mesa, libgles2-mesa, libgbm,
  mesa-megadriver. forgectrl dlopens them, so an image without them
  still streams on NEON. Platform change: the next campaign is full.
- forgetest: camera.h264-stream exercises the endpoint end to end
  (codec header, init segment, media fragments, /cam/status encoder
  state) and accepts a clean 503 on a machine without the stack;
  the camera covers name the new forgectrl files.
- VIDEO.md 5.6 now describes both streams and why H.264 earns its
  place (the WiFi transmit path measures about 7 percent of the core
  per MB/s sent); BRINGUP records the bench-validation checklist as
  Next work item 20, including the release-size watch: mesa must fit
  the 200 MiB slot gate.

docs/BRINGUP.md

@@ -484,6 +484,10 @@ bounce copy; daemon ~41 % CPU with one viewer. Full-res snapshot 2.4 s warm /
 copy, needed on a kernel without the `allow_cache_hints` patch — detected via
 the `MMAP_CACHE_HINTS` capability bit), `FORGECTRL_NO_NEON` (scalar convert,
 bit-identical), `FORGECTRL_NO_VPU` (libjpeg; also the snapshot path).
+Newer and **not yet bench-run** (Next work item 20): the GC880 GPU demosaic,
+the `/cam/h264` CODA960 H.264 stream, and CSI hardware frame skip, with
+`FORGECTRL_NO_GPU` / `FORGECTRL_NO_H264` / `FORGECTRL_NO_HW_SKIP` to strip
+them individually; all fall back to the measured paths above.
 `/cam/status` reports `encoder` and `buffers`. A CSI glitch frame can out-size
 the coda driver's default JPEG capture buffer, so forgectrl requests 3 B/px and
 drops error-flagged dequeues as single bad frames. **LightBurn consumes the
@@ -1442,6 +1446,30 @@ Open items only. Anything closed is in `CAMPAIGN-LOG.md`.
     are established. A pause on contact, on the factory's shape, would be
     the first use.
 
+20. **Video pipeline offload — bench validation.** Code-complete and
+    host-proven (CI: `mp4mux_test`; catalog: `camera.h264-stream`), not yet
+    run on hardware: the GC880 GLES2 demosaic (`src/gpu_debayer.c`,
+    dlopen'd Mesa, capture dmabuf in, encoder dmabuf out), the CODA960
+    H.264 stream (`/cam/h264`, fragmented MP4, panel MSE player,
+    ~1.5 Mbit/s vs MJPEG's ~9), and the CSI hardware frame skip behind
+    `FORGECTRL_STREAM_FPS`. Falls back to the NEON path wherever a piece
+    is missing, so the existing stream is not at risk. To settle on the
+    bench, in order: (1) the image builds with Mesa etnaviv inside the
+    200 MiB slot cap (distro `PACKAGECONFIG:pn-mesa`, image adds
+    `libegl-mesa libgles2-mesa libgbm mesa-megadriver`); (2) surfaceless
+    EGL comes up and dmabuf import works both directions (the open logs
+    say exactly which probe refused); (3) `FORGECTRL_GPU_CHECK` reports a
+    max delta of a couple of counts against the scalar demosaic;
+    (4) `GL_MAX_TEXTURE_SIZE` covers 1944 rows (5 MP single-tile; the
+    8 MP tiling path can only be exercised on an HD machine or by
+    forcing tiles); (5) coda H.264 rate control and picture quality at
+    1296x972p15, and the measured CPU with a panel H.264 viewer vs the
+    41 % MJPEG baseline; (6) the stream-during-jog coexistence drill
+    rerun with the GPU path active. Switches to strip a suspect layer:
+    `FORGECTRL_NO_GPU`, `FORGECTRL_NO_H264`, `FORGECTRL_NO_HW_SKIP`,
+    plus the existing `FORGECTRL_NO_VPU` / `FORGECTRL_NO_NEON` /
+    `FORGECTRL_NO_CACHED_BUFS`.
+
 **Deliberately not gated:** an armed GRBL job after an underrun cuts at the
 stale origin unless homing is required (GRBL mode permits unhomed cutting; the
 underrun itself alarms and unlinks the anchor). Not in the acceptance catalog

docs/VIDEO.md

@@ -1,10 +1,11 @@
 # Video and the cameras
 
 The machine has two cameras — one in the lid looking down at the bed, one in the
-print head looking at the material under the lens — and ForgeFIRM serves both as
+print head looking at the material under the lens — and ForgeFIRM serves both
-plain **MJPEG over HTTP** from the web control panel. There is no app, no cloud
+over plain HTTP from the web control panel: **MJPEG** for anything that can read
-relay, and no proprietary protocol: a browser, LightBurn, or anything else that
+a stream of JPEGs, and an **H.264** live stream for clients that decode video
-can read an MJPEG stream or fetch a JPEG can use them.
+(the panel uses it when the browser can). There is no app, no cloud relay, and
+no proprietary protocol.
 
 One rule governs all of it: **the cameras only capture with the lid closed**
 (§2). Everything else here assumes that condition is met.
@@ -115,6 +116,7 @@ buttons; **Live** switches the same frame to the running stream.
 |---|---|
 | `/cam/stream?cam=lid` | continuous MJPEG (`multipart/x-mixed-replace`) |
 | `/cam/stream?cam=head` | the same, from the head camera |
+| `/cam/h264?cam=lid` | continuous H.264 as fragmented MP4 (see §5.6): the same picture in a fraction of the bytes, for clients that decode video |
 | `/cam/snapshot?cam=lid` | one full-resolution JPEG |
 | `/cam/snapshot?cam=lid&res=half` | one half-resolution JPEG (much faster) |
 | `/cam/status` | JSON: which sensor, which camera, frame rate, frame sizes, whether the lid currently permits capture |
@@ -145,7 +147,7 @@ practice: paste the URL into any local client and it works.
 | Live stream | 1296 × 972 | 1632 × 1224 |
 | Full snapshot | 2592 × 1944 | 3264 × 2448 |
 | Half snapshot | 1296 × 972 | 1632 × 1224 |
-| Format | JPEG, quality 75 by default | same |
+| Stream formats | MJPEG (quality 75 by default) and H.264 (~1.5 Mbit/s by default) | same |
 | Frame rate | **15 fps** sustained | not yet measured (§10) |
 
 Measured on a 5 MP machine: 15.0 fps with a viewer attached, which is the rate
@@ -161,6 +163,11 @@ sensor pixels becomes exactly one output pixel, which is why the stream is
 precisely half the capture in each axis and why it is cheap enough to run
 continuously.
 
+The 41 % figure is the NEON demosaic feeding the hardware JPEG encoder. When
+the GPU demosaic and the H.264 stream carry the load instead (§5.6), the
+stream's CPU cost drops to bookkeeping; those two paths are newer than the
+figure above and their own numbers will be measured on the bench the same way.
+
 ---
 
 ## 5. What the sensor can do versus what ForgeFIRM sends
@@ -177,7 +184,7 @@ reason.
 | Bit depth | 10 bits per pixel | 8 bits | JPEG is 8-bit, and 8-bit is what makes §5.2 fit |
 | Exposure / color | auto exposure and auto white balance | fixed values | a bed image has to look the same frame to frame; §5.4 |
 | Lens | — | no correction applied | correction belongs in the client; §5.5 |
-| Encoding | — | MJPEG only, nothing recorded | §5.6 |
+| Encoding | — | MJPEG and H.264, nothing recorded | §5.6 |
 | Mirroring | a mirror register | mirrored in software instead | the register breaks capture on this board; §5.7 |
 
 ### 5.1 Resolution and frame rate on a 5 MP machine
@@ -273,20 +280,43 @@ correction on the host, which is more accurate than a fixed correction baked
 into the firmware and costs the machine's CPU nothing. Run that calibration
 before trusting the camera overlay for placement.
 
-### 5.6 MJPEG only — and nothing is recorded
+### 5.6 Two streams, one picture: MJPEG and H.264. Nothing is recorded.
 
-The stream is a sequence of complete JPEG frames, not H.264 or any other
+The same live picture is served two ways, and **the machine never writes video
-inter-frame codec, and **the machine never writes video to disk**.
+to disk**.
 
-MJPEG is the right trade here: every frame stands alone, so a viewer can join
+**MJPEG** (`/cam/stream`) is the universal one: every frame is a complete
-or leave at any moment and a dropped frame costs nothing; browsers and sender
+JPEG, so a viewer can join or leave at any moment, a dropped frame costs
-software consume it with no plugin; and stream frames are encoded by the
+nothing, and browsers, LightBurn and mjpg-streamer clients consume it with no
-board's **hardware JPEG encoder**, which is what makes 15 fps affordable while
+plugin. It stays, unchanged, and it is what anything that cannot decode video
-the machine is also running a job. (Stills are encoded in software instead,
+should use.
-which is most of why a full-resolution one takes a couple of seconds.) An
+
-inter-frame codec would need buffering and a container, would break the "any
+**H.264** (`/cam/h264`) exists because bytes on this machine are not free.
-client, any time" property, and would buy bandwidth savings that a LAN does
+MJPEG re-sends the whole scene fifteen times a second, roughly 9 Mbit/s, and
-not need.
+the WiFi transmit path runs on the machine's single CPU core, where measured
+cost is about 7 % of the core per MB/s sent. A bed camera's scene barely
+changes between frames, which is exactly what an inter-frame codec exploits:
+the H.264 stream carries the same picture in roughly 1.5 Mbit/s and gives most
+of that CPU back. It arrives as fragmented MP4, the form a browser's Media
+Source Extensions accept, with the codec named in an `X-H264-Codec` response
+header; the panel's **Live** button uses it automatically where the browser
+can and falls back to MJPEG where it cannot. Latency is a beat behind MJPEG
+(under a second), which is why LightBurn keeps consuming the MJPEG stream.
+
+Both encoders are hardware: JPEG frames come from the CODA960's JPEG unit and
+H.264 from its BIT processor, two independent engines, so serving both at once
+does not double any cost that matters. The demosaic that feeds them runs as
+fragment shaders on the SoC's GC880 GPU when the image ships the GL stack
+(reported as `"convert": "gpu"` in `/cam/status`), reading the sensor frame
+and writing the encoder's buffer directly, so a stream frame never crosses the
+CPU at all; without the GPU it falls back to the NEON demosaic. (Stills are
+still demosaiced and encoded on the CPU, which is most of why a
+full-resolution one takes a couple of seconds.)
+
+One more consumer of nothing: with a frame-rate cap set (`FORGECTRL_STREAM_FPS`
+of 1 or more), the cap is programmed into the CSI receiver's frame-skip
+hardware, and skipped frames are dropped before they are ever written to
+memory. `/cam/status` reports `"hw_fps_skip": true` when that is in effect.
 
 If you want a recording, record the stream on the computer watching it. The
 machine stores its firmware, settings and logs on a small internal flash device

forgetest/forgetest/suite/camera.py

@@ -5,6 +5,8 @@ from ..baseline import LID_LAMP_ATTR
 
 _CAM_COVERS = [("forgectrl", "src/cam.*"), ("forgectrl", "src/camhealth.*"),
                ("forgectrl", "src/debayer.*"), ("forgectrl", "src/vpu_jpeg.*"),
+               ("forgectrl", "src/vpu_h264.*"), ("forgectrl", "src/mp4mux.*"),
+               ("forgectrl", "src/gpu_debayer.*"),
                ("forgectrl", "src/main.c"),
                ("python3-gfhardware", "gfhardware/src/**"), ("python3-gfhardware", "gfhardware/cam*")]
 
@@ -158,6 +160,68 @@ def sensor_profile(ctx):
         ctx.check(got == (w, h), "%s snapshot is %s, not %dx%d", res, got, w, h)
 
 
+@test("camera.h264-stream", title="H.264 live stream", subsystem="camera",
+      kind="auto", est_min=2,
+      covers=_CAM_COVERS, requires=["forgectrl.panel-serves"],
+      steps=["Setup: lid closed."],
+      description="/cam/h264 answers a fragmented MP4: a codec header naming the SPS profile, an "
+                  "init segment (ftyp+moov) followed by media fragments (moof+mdat) within a few "
+                  "seconds, and /cam/status reporting the H.264 encoder active with this client "
+                  "counted. Also records which demosaic path (GPU or CPU) served the stream. On a "
+                  "machine whose image lacks Mesa or whose encoder refused, the endpoint must "
+                  "answer 503 rather than hang - that is a pass for the endpoint but is recorded "
+                  "in the evidence.")
+def h264_stream(ctx):
+    import urllib.error
+    import urllib.request
+
+    fc = ctx.forgectrl
+    ev = ctx.evidence
+
+    req = urllib.request.Request(fc.base + "/cam/h264?cam=lid",
+                                 headers={"Host": fc.host_header()})
+    try:
+        r = urllib.request.urlopen(req, timeout=20)
+    except urllib.error.HTTPError as e:
+        ev["status"] = e.code
+        ev["body"] = e.read()[:200].decode("utf-8", "replace")
+        ctx.check(e.code == 503, "H.264 refused with %s, not 503: %s", e.code, ev["body"])
+        ctx.log("H.264 unavailable on this machine (503): %s", ev["body"])
+        return
+    with r:
+        codec = r.headers.get("X-H264-Codec", "")
+        ctype = r.headers.get("Content-Type", "")
+        ev["codec"] = codec
+        ev["content_type"] = ctype
+        ctx.log("codec %s, content-type %s", codec, ctype)
+        ctx.check(ctype == "video/mp4", "content type is %r", ctype)
+        ctx.check(codec.startswith("avc1."), "codec header is %r", codec)
+
+        # Init segment, then at least one media fragment.
+        data = b""
+        while len(data) < 512 * 1024:
+            chunk = r.read(65536)
+            if not chunk:
+                break
+            data += chunk
+            if b"moof" in data and b"mdat" in data:
+                break
+    ev["bytes"] = len(data)
+    ctx.check(data[4:8] == b"ftyp", "stream does not start with ftyp")
+    ctx.check(b"moov" in data, "no moov (init segment)")
+    ctx.check(b"avcC" in data, "no avcC in the init segment")
+    ctx.check(b"moof" in data and b"mdat" in data,
+              "no media fragment arrived in %d bytes", len(data))
+
+    st, body = fc.get("/cam/status")
+    ctx.check(st == 200 and isinstance(body, dict), "GET /cam/status -> %s", st)
+    ev["cam_status"] = body
+    ctx.log("convert=%s h264=%s", body.get("convert"), body.get("h264"))
+    h = body.get("h264") or {}
+    ctx.check(h.get("active") is True, "/cam/status does not report the H.264 encoder active: %s",
+              body)
+
+
 @test("camera.frame-health", title="Capture delivers whole frames", subsystem="camera",
       kind="auto", est_min=1,
       covers=_CAM_COVERS, requires=["forgectrl.panel-serves"],

meta-forgefirm/conf/distro/forgefirm.conf

@@ -5,9 +5,17 @@ DISTRO_NAME = "OpenGlow/ForgeFIRM"
 DISTRO_VERSION = "0.0.0"
 
 DISTRO_FEATURES:remove = " \
-  3g alsa avahi bluetooth bluez5 ext2 ipv6 irda nfc nfs opengl pci pcmcia \
+  3g alsa avahi bluetooth bluez5 ext2 ipv6 irda nfc nfs pci pcmcia \
   pulseaudio vulkan wayland x11 zeroconf "
 
+# opengl stays: forgectrl's camera demosaic runs as GLES2 fragment
+# shaders on the GC880 (etnaviv), reached through surfaceless EGL with
+# no display stack. Mesa is trimmed to exactly that: the etnaviv
+# gallium driver, GLES/EGL/GBM, no GLX, no X11/Wayland platforms
+# (both remain removed above), so the cost is the libraries and the one
+# driver, not a graphics stack.
+PACKAGECONFIG:pn-mesa = "opengl gles egl gbm gallium etnaviv"
+
 # System logger: rsyslog replaces busybox syslogd/klogd. Every ForgeFIRM
 # process logs through it and it is the only log writer (one directory
 # per logger under /data/log/forgefirm; per-logger levels and the remote

meta-forgefirm/recipes-forgefirm/forgectrl/forgectrl-pin.inc

@@ -2,5 +2,5 @@
 # only SRCREV and PV here - the image manifest leaves *-pin.inc out of the
 # layer content hash because the component entry already identifies the
 # pinned source (forgefirm-image-manifest.bbclass).
-SRCREV = "3acd66425dd53b3a8b16ae175ec442a4d057b9ac"
+SRCREV = "6573abd725e885f0a56463cc83bcb1863525ff77"
 PV = "0.1.0"

meta-forgefirm/recipes-forgefirm/images/forgefirm-image.bb

@@ -35,6 +35,11 @@ IMAGE_INSTALL:remove = "gfui-client"
 # VIRTUAL-RUNTIME_base-utils-syslog (conf/distro/forgefirm.conf).
 IMAGE_INSTALL:append = " grblhal-glowforge forgectrl gfhome gfcloud v4l-utils fwup ffboot slotmigrate forgefirm-logging"
 
+# Mesa GLES2/EGL on etnaviv for forgectrl's GPU demosaic (loaded with
+# dlopen at runtime; forgectrl itself has no build-time GL dependency,
+# and without these packages it falls back to the NEON path).
+IMAGE_INSTALL:append = " libegl-mesa libgles2-mesa libgbm mesa-megadriver"
+
 # NXP's firmware EULA covers the i.MX VPU/EPDC blobs the BSP installs, so the
 # image ships the license text with them (/usr/share/licenses/firmware-imx).
 # The SDMA firmware brings its own -license package through linux-firmware.