Video offload: Mesa etnaviv in the image, forgectrl pinned, the H.264 stream in the catalog

The stream conversion now runs on the GC880 when the image carries the
GL stack, and /cam/h264 serves the same picture as fragmented MP4 in a
fraction of MJPEG's bytes (forgectrl 6573abd). This side supplies what
that needs and holds it to account:

- Distro: opengl stays a feature, with mesa trimmed to exactly the
  path used (gallium etnaviv, GLES/EGL/GBM, no GLX, no display
  platforms). Image: libegl-mesa, libgles2-mesa, libgbm,
  mesa-megadriver. forgectrl dlopens them, so an image without them
  still streams on NEON. Platform change: the next campaign is full.
- forgetest: camera.h264-stream exercises the endpoint end to end
  (codec header, init segment, media fragments, /cam/status encoder
  state) and accepts a clean 503 on a machine without the stack;
  the camera covers name the new forgectrl files.
- VIDEO.md 5.6 now describes both streams and why H.264 earns its
  place (the WiFi transmit path measures about 7 percent of the core
  per MB/s sent); BRINGUP records the bench-validation checklist as
  Next work item 20, including the release-size watch: mesa must fit
  the 200 MiB slot gate.
This commit is contained in:
ScottW514
2026-08-24 08:10:33 -04:00
parent a2b391b8f6
commit a47c0c7f6d
6 changed files with 155 additions and 20 deletions
+48 -18
View File
@@ -1,10 +1,11 @@
# Video and the cameras
The machine has two cameras — one in the lid looking down at the bed, one in the
print head looking at the material under the lens — and ForgeFIRM serves both as
plain **MJPEG over HTTP** from the web control panel. There is no app, no cloud
relay, and no proprietary protocol: a browser, LightBurn, or anything else that
can read an MJPEG stream or fetch a JPEG can use them.
print head looking at the material under the lens — and ForgeFIRM serves both
over plain HTTP from the web control panel: **MJPEG** for anything that can read
a stream of JPEGs, and an **H.264** live stream for clients that decode video
(the panel uses it when the browser can). There is no app, no cloud relay, and
no proprietary protocol.
One rule governs all of it: **the cameras only capture with the lid closed**
(§2). Everything else here assumes that condition is met.
@@ -115,6 +116,7 @@ buttons; **Live** switches the same frame to the running stream.
|---|---|
| `/cam/stream?cam=lid` | continuous MJPEG (`multipart/x-mixed-replace`) |
| `/cam/stream?cam=head` | the same, from the head camera |
| `/cam/h264?cam=lid` | continuous H.264 as fragmented MP4 (see §5.6): the same picture in a fraction of the bytes, for clients that decode video |
| `/cam/snapshot?cam=lid` | one full-resolution JPEG |
| `/cam/snapshot?cam=lid&res=half` | one half-resolution JPEG (much faster) |
| `/cam/status` | JSON: which sensor, which camera, frame rate, frame sizes, whether the lid currently permits capture |
@@ -145,7 +147,7 @@ practice: paste the URL into any local client and it works.
| Live stream | 1296 × 972 | 1632 × 1224 |
| Full snapshot | 2592 × 1944 | 3264 × 2448 |
| Half snapshot | 1296 × 972 | 1632 × 1224 |
| Format | JPEG, quality 75 by default | same |
| Stream formats | MJPEG (quality 75 by default) and H.264 (~1.5 Mbit/s by default) | same |
| Frame rate | **15 fps** sustained | not yet measured (§10) |
Measured on a 5 MP machine: 15.0 fps with a viewer attached, which is the rate
@@ -161,6 +163,11 @@ sensor pixels becomes exactly one output pixel, which is why the stream is
precisely half the capture in each axis and why it is cheap enough to run
continuously.
The 41 % figure is the NEON demosaic feeding the hardware JPEG encoder. When
the GPU demosaic and the H.264 stream carry the load instead (§5.6), the
stream's CPU cost drops to bookkeeping; those two paths are newer than the
figure above and their own numbers will be measured on the bench the same way.
---
## 5. What the sensor can do versus what ForgeFIRM sends
@@ -177,7 +184,7 @@ reason.
| Bit depth | 10 bits per pixel | 8 bits | JPEG is 8-bit, and 8-bit is what makes §5.2 fit |
| Exposure / color | auto exposure and auto white balance | fixed values | a bed image has to look the same frame to frame; §5.4 |
| Lens | — | no correction applied | correction belongs in the client; §5.5 |
| Encoding | — | MJPEG only, nothing recorded | §5.6 |
| Encoding | — | MJPEG and H.264, nothing recorded | §5.6 |
| Mirroring | a mirror register | mirrored in software instead | the register breaks capture on this board; §5.7 |
### 5.1 Resolution and frame rate on a 5 MP machine
@@ -273,20 +280,43 @@ correction on the host, which is more accurate than a fixed correction baked
into the firmware and costs the machine's CPU nothing. Run that calibration
before trusting the camera overlay for placement.
### 5.6 MJPEG only — and nothing is recorded
### 5.6 Two streams, one picture: MJPEG and H.264. Nothing is recorded.
The stream is a sequence of complete JPEG frames, not H.264 or any other
inter-frame codec, and **the machine never writes video to disk**.
The same live picture is served two ways, and **the machine never writes video
to disk**.
MJPEG is the right trade here: every frame stands alone, so a viewer can join
or leave at any moment and a dropped frame costs nothing; browsers and sender
software consume it with no plugin; and stream frames are encoded by the
board's **hardware JPEG encoder**, which is what makes 15 fps affordable while
the machine is also running a job. (Stills are encoded in software instead,
which is most of why a full-resolution one takes a couple of seconds.) An
inter-frame codec would need buffering and a container, would break the "any
client, any time" property, and would buy bandwidth savings that a LAN does
not need.
**MJPEG** (`/cam/stream`) is the universal one: every frame is a complete
JPEG, so a viewer can join or leave at any moment, a dropped frame costs
nothing, and browsers, LightBurn and mjpg-streamer clients consume it with no
plugin. It stays, unchanged, and it is what anything that cannot decode video
should use.
**H.264** (`/cam/h264`) exists because bytes on this machine are not free.
MJPEG re-sends the whole scene fifteen times a second, roughly 9 Mbit/s, and
the WiFi transmit path runs on the machine's single CPU core, where measured
cost is about 7 % of the core per MB/s sent. A bed camera's scene barely
changes between frames, which is exactly what an inter-frame codec exploits:
the H.264 stream carries the same picture in roughly 1.5 Mbit/s and gives most
of that CPU back. It arrives as fragmented MP4, the form a browser's Media
Source Extensions accept, with the codec named in an `X-H264-Codec` response
header; the panel's **Live** button uses it automatically where the browser
can and falls back to MJPEG where it cannot. Latency is a beat behind MJPEG
(under a second), which is why LightBurn keeps consuming the MJPEG stream.
Both encoders are hardware: JPEG frames come from the CODA960's JPEG unit and
H.264 from its BIT processor, two independent engines, so serving both at once
does not double any cost that matters. The demosaic that feeds them runs as
fragment shaders on the SoC's GC880 GPU when the image ships the GL stack
(reported as `"convert": "gpu"` in `/cam/status`), reading the sensor frame
and writing the encoder's buffer directly, so a stream frame never crosses the
CPU at all; without the GPU it falls back to the NEON demosaic. (Stills are
still demosaiced and encoded on the CPU, which is most of why a
full-resolution one takes a couple of seconds.)
One more consumer of nothing: with a frame-rate cap set (`FORGECTRL_STREAM_FPS`
of 1 or more), the cap is programmed into the CSI receiver's frame-skip
hardware, and skipped frames are dropped before they are ever written to
memory. `/cam/status` reports `"hw_fps_skip": true` when that is in effect.
If you want a recording, record the stream on the computer watching it. The
machine stores its firmware, settings and logs on a small internal flash device