mirror of
https://github.com/openglow-org/forgefirm.git
synced 2026-09-29 09:31:17 -07:00
BRINGUP: present-state sweep for the shared-services architecture
The runbook operational sections now describe the machine as it runs: forgectrl as the machine-services daemon (supervisor, pulse-device broker, motion-liveness gate, cooling engine, cameras), controllers spawned and supervised rather than init-started, standalone driver runs as the bench/debug exception, diagnostics suspending the active controller through the supervisor, the cooling engine as the thermal owner with the drivers as thin clients, and the new /mode and /cool endpoints. A dated correction closes the no-motion record with the DRV8825 wedge diagnosis and the liveness/homing hardening; the cold-start section gains the first-light and shared-services milestones.
This commit is contained in:
+125
-45
@@ -76,6 +76,25 @@ motion constants were extracted from the `_RESOURCES` pulse files
|
|||||||
`GLOWFORGE_DEFAULTS` values, run `$RST=$` once on the board (the sim
|
`GLOWFORGE_DEFAULTS` values, run `$RST=$` once on the board (the sim
|
||||||
persists settings in its eeprom file in /data).
|
persists settings in its eeprom file in /data).
|
||||||
|
|
||||||
|
**First light: 2026-08-11** — first GRBL-mode burn (operator-run
|
||||||
|
LightBurn job, chain armed; details in the laser item under Next work).
|
||||||
|
|
||||||
|
**Shared machine services: complete and bench-verified 2026-08-11.**
|
||||||
|
forgectrl is the machine-services daemon: the cooling engine (single
|
||||||
|
thermal-hardware owner for both controller modes, flow verification and
|
||||||
|
over-temp policy behind the `/cool/state` + verdict-file channels), the
|
||||||
|
controller-mode supervisor (one managed child, live `POST /mode`
|
||||||
|
switching, crash respawn with machine safing, a respawn wrapper on
|
||||||
|
forgectrl itself with retake-at-idle), the pulse-device broker (one
|
||||||
|
exclusive `/dev/glowforge` hold for the daemon's lifetime — handovers
|
||||||
|
and respawns never cycle the 40 V rail), and the **motion-liveness
|
||||||
|
gate**: the head accelerometer is the only truth about physical motion
|
||||||
|
(the DRV8825 drivers can wedge unserviceably on rail glitches with
|
||||||
|
counters running normally — see the hardware facts bank), so the
|
||||||
|
supervisor probes real motion before each session's first controller
|
||||||
|
spawn and gfhome refuses to report a homing the accelerometer did not
|
||||||
|
witness. The contract for all of it is `forgectrl/docs/SERVICES.md`.
|
||||||
|
|
||||||
## The bench
|
## The bench
|
||||||
|
|
||||||
- **Board**: SSH `root@172.16.1.97` (fixed DHCP lease since 2026-08-02;
|
- **Board**: SSH `root@172.16.1.97` (fixed DHCP lease since 2026-08-02;
|
||||||
@@ -149,12 +168,18 @@ grblHAL driver repo** (github.com/ScottW514/grblHAL-glowforge, branch
|
|||||||
`forgefirm` = **upstream master + the step_us_min buffer fix pending
|
`forgefirm` = **upstream master + the step_us_min buffer fix pending
|
||||||
upstream**; the settings-write crash fix merged upstream 2026-08-04 as
|
upstream**; the settings-write crash fix merged upstream 2026-08-04 as
|
||||||
grblHAL/core PR #999), `driver.c` implementing the HAL, machine
|
grblHAL/core PR #999), `driver.c` implementing the HAL, machine
|
||||||
constants in `src/boards/glowforge.h`. **The controller autostarts at boot** since
|
constants in `src/boards/glowforge.h`. **The controller is spawned and
|
||||||
2026-08-03: the `grblhal-glowforge` recipe (meta-forgefirm; gitsm pin +
|
supervised by forgectrl**: the supervisor starts the controller selected
|
||||||
sysvinit script `grblhal`, defaults 92) is installed in both images, and
|
by `controller_mode` (grbl | cloud) as a direct child, respawns it on a
|
||||||
the same init script is installed on the current bench rootfs
|
crash (after safing the machine), and switches modes live via
|
||||||
(reboot-verified: controller + forgectrl both up unattended, Grbl
|
`POST /mode` / the Status-tab selector. The `grblhal` and `gfcloud` init
|
||||||
answering on :23). The manual start below remains the bench/debug path. Architecture: a wall-paced producer thread runs
|
scripts defer to it (they remain only as manual emergency stops). The
|
||||||
|
pulse device arrives as a broker-inherited fd (`GF_PULSE_FD`; see the
|
||||||
|
pulse-device ownership section of forgectrl `docs/SERVICES.md`) — the
|
||||||
|
device never closes across mode switches, homing handovers, or respawns,
|
||||||
|
so the 40 V rail never cycles as a side effect, and the supervisor
|
||||||
|
verifies **physical motion** (head-accelerometer liveness probe) before
|
||||||
|
the first controller spawn of each session. Architecture: a wall-paced producer thread runs
|
||||||
the core stepper ISR against a virtual step clock (1000× machine tick)
|
the core stepper ISR against a virtual step clock (1000× machine tick)
|
||||||
and maps step events to pulse bytes; a SCHED_FIFO shipper feeds
|
and maps step events to pulse bytes; a SCHED_FIFO shipper feeds
|
||||||
`/dev/glowforge` with the bounded queue; a recursive core mutex stands in
|
`/dev/glowforge` with the bounded queue; a recursive core mutex stands in
|
||||||
@@ -167,18 +192,25 @@ hardware I/O — host testing).
|
|||||||
force-included into the core: 53.333 µsteps/mm XY @ ×8, 2.832
|
force-included into the core: 53.333 µsteps/mm XY @ ×8, 2.832
|
||||||
half-steps/mm Z, 0.417" Z travel, 12000 mm/min max, 700/590 mm/s²
|
half-steps/mm Z, 0.417" Z travel, 12000 mm/min max, 700/590 mm/s²
|
||||||
accel — factory-derived, see `puls_profile.py`).
|
accel — factory-derived, see `puls_profile.py`).
|
||||||
2. Deploy to `/usr/bin/grblHAL_glowforge` on the board (kill the running
|
2. Deploy: move the new binary over `/usr/bin/grblHAL_glowforge` (mv
|
||||||
instance first — the binary can't be overwritten while executing).
|
replaces the inode, so the running instance is untouched), then kill
|
||||||
3. Start: `cd /data && GFSINK=/dev/glowforge grblHAL_glowforge -p 23 -e
|
the running controller — the supervisor respawns it on the new binary
|
||||||
/data/EEPROM-glowforge.DAT` (no `-t` — real-time pacing is intrinsic
|
within about a second.
|
||||||
now). Env knobs: `GFSINK_RATE` (machine tick, default 28160 Hz =
|
3. Standalone start (bench/debug only — requires forgectrl stopped,
|
||||||
factory travel tick), `GFSINK_DEPTH_MS` (queue depth = feed-hold
|
since the broker's exclusive hold on `/dev/glowforge` makes any
|
||||||
latency, default 200). The driver applies the full analog machine
|
self-open fail EBUSY): `cd /data && GFSINK=/dev/glowforge
|
||||||
config itself at init (×8 modes, decay 1, motor_lock 8, laser latched,
|
grblHAL_glowforge -p 23 -e /data/EEPROM-glowforge.DAT`. Env knobs:
|
||||||
PIC hold currents) and swaps PIC run/hold currents around motion. If
|
`GFSINK_RATE` (machine tick, default 28160 Hz = factory travel tick),
|
||||||
the baked $-defaults changed since the last run, `$RST=$` once (stored
|
`GFSINK_DEPTH_MS` (queue depth = feed-hold latency, default 200).
|
||||||
settings win). Each motion run logs a producer-stats line to stderr
|
Standalone, the driver opens the device itself and every takeover
|
||||||
(callbacks, µs/call, max-behind, clamped) — clamped should stay 0.
|
runs the `rail_settle_s` off-period; under the broker it inherits
|
||||||
|
the fd and skips the settle (the rail never dropped). The driver
|
||||||
|
applies the full analog machine config at init either way (×8 modes,
|
||||||
|
decay 1, motor_lock 8, laser latched, PIC hold currents) and swaps
|
||||||
|
PIC run/hold currents around motion. If the baked $-defaults changed
|
||||||
|
since the last run, `$RST=$` once (stored settings win). Each motion
|
||||||
|
run logs a producer-stats line to stderr (callbacks, µs/call,
|
||||||
|
max-behind, clamped) — clamped should stay 0.
|
||||||
4. Connect LightBurn/UGS to `172.16.1.97:23`, or jog raw:
|
4. Connect LightBurn/UGS to `172.16.1.97:23`, or jog raw:
|
||||||
`$J=G91X40F1200`. `^X` mid-motion aborts via kernel `cnc/stop`
|
`$J=G91X40F1200`. `^X` mid-motion aborts via kernel `cnc/stop`
|
||||||
(controlled decel) and raises an alarm; TCP disconnects never kill the
|
(controlled decel) and raises an alarm; TCP disconnects never kill the
|
||||||
@@ -212,28 +244,40 @@ image binaries — grblHAL (fortified) serves at 1.0 ms RTT with exact
|
|||||||
jogs, $0 min 35.5 intact, $H rejected ($22=0); forgectrl streams
|
jogs, $0 min 35.5 intact, $H rejected ($22=0); forgectrl streams
|
||||||
15.0 fps, `"buffers":"cached"`, vpu, 41% CPU; grblHAL idle 2.1%.
|
15.0 fps, `"buffers":"cached"`, vpu, 41% CPU; grblHAL idle 2.1%.
|
||||||
|
|
||||||
## The camera service (forgectrl, port 8080)
|
## The machine-services daemon (forgectrl, port 8080)
|
||||||
|
|
||||||
Source: `C:\dev\openglow-forgefirm\forgectrl` — the **canonical repo**
|
Source: `C:\dev\openglow-forgefirm\forgectrl` — the **canonical repo**
|
||||||
(github.com/ScottW514/forgectrl, branch `main`, MIT). forgectrl is the
|
(github.com/ScottW514/forgectrl, branch `main`, MIT). forgectrl is the
|
||||||
ForgeFIRM control daemon: camera service today; realtime hardware
|
ForgeFIRM machine-services daemon: **controller-mode supervision** (it
|
||||||
status/settings, hardware control, and GRBL-vs-cloud mode selection are
|
spawns exactly one of grblHAL / gfcloud as a direct child, respawns on
|
||||||
its planned scope. The meta-forgefirm recipe pins its SRCREV (bump
|
crash after safing the machine, and switches live via `POST /mode`),
|
||||||
|
the **pulse-device broker** (one exclusive hold on `/dev/glowforge` for
|
||||||
|
its lifetime; controllers inherit the fd, the rail never cycles on
|
||||||
|
handovers, and the supervisor is the writers' dead-man), the
|
||||||
|
**motion-liveness gate** (head-accelerometer probe before the first
|
||||||
|
spawn of each session, with a rail-off recovery ladder for wedged
|
||||||
|
DRV8825 drivers and a loud `motion-fault` state), the **cooling
|
||||||
|
engine** (single owner of fans/pump/TEC/heater for both modes:
|
||||||
|
`POST /cool/state` job reports in, the `/run/forgefirm/cooling.state`
|
||||||
|
verdict file out), plus cameras, telemetry, settings, diagnostics, the
|
||||||
|
web panel, and updates. It runs under a respawn wrapper (its init
|
||||||
|
script) and a restarted daemon retakes supervision automatically once
|
||||||
|
the machine is idle. The meta-forgefirm recipe pins its SRCREV (bump
|
||||||
deliberately after pushing) and installs the sysvinit script from the
|
deliberately after pushing) and installs the sysvinit script from the
|
||||||
repo's `init/`; bench builds cross-compile with
|
repo's `init/`; bench builds cross-compile with
|
||||||
`forgefirm/scripts/bench/build-forgectrl.sh` (same toolchain-borrow
|
`forgefirm/scripts/bench/build-forgectrl.sh` (same toolchain-borrow
|
||||||
pattern as build-glowforge.sh). The **machine-services contract** —
|
pattern as build-glowforge.sh). The **machine-services contract** —
|
||||||
the EV_SW switch map, the authoritative sensor conversions, the
|
the EV_SW switch map, the authoritative sensor conversions, the
|
||||||
hardware single-writer ownership matrix, and the interface for the
|
hardware single-writer ownership matrix, the cooling channels, mode
|
||||||
shared cooling service both controller modes will use — is
|
supervision, and pulse-device ownership — is
|
||||||
`forgectrl/docs/SERVICES.md` in the forgectrl repo. One ulfius daemon
|
`forgectrl/docs/SERVICES.md` in the forgectrl repo. One ulfius daemon
|
||||||
exposes both OV5648 cameras as MJPEG over the mainline imx-media
|
serves it all, including both OV5648 cameras as MJPEG over the
|
||||||
pipeline:
|
mainline imx-media pipeline:
|
||||||
|
|
||||||
- `GET /` — the tabbed machine control panel (Status / Machine /
|
- `GET /` — the tabbed machine control panel (Status / Machine /
|
||||||
GF Cloud / GRBL / Diagnostics; ui.c): status page with the
|
GF Cloud / GRBL / Diagnostics; ui.c): status page with the
|
||||||
controller-mode selector (GRBL active; factory cloud disabled until
|
controller-mode selector (live switch through the supervisor; the
|
||||||
implemented), the operational dashboard, a scaled lid snapshot +
|
setting persists for boot), the operational dashboard, a scaled lid snapshot +
|
||||||
on-demand live stream, and the settings forms for display units,
|
on-demand live stream, and the settings forms for display units,
|
||||||
homing method, home-position calibration, the nine cooling
|
homing method, home-position calibration, the nine cooling
|
||||||
tunables, identity overrides, and the session timeout. All
|
tunables, identity overrides, and the session timeout. All
|
||||||
@@ -265,6 +309,16 @@ pipeline:
|
|||||||
**Writes 409 unless cnc/state is idle** (the controller and homing
|
**Writes 409 unless cnc/state is idle** (the controller and homing
|
||||||
runner read the file mid-run) — live-verified during a jog — **and
|
runner read the file mid-run) — live-verified during a jog — **and
|
||||||
409 while a diagnostic owns the hardware**.
|
409 while a diagnostic owns the hardware**.
|
||||||
|
- `GET /mode` / `POST /mode?controller=grbl|cloud` — the supervisor:
|
||||||
|
current mode, controller state (`running | stopped | standby |
|
||||||
|
motion-fault`), pid, and the motion-liveness verdict
|
||||||
|
(`verified | unverified | fault`); the POST is the live idle-gated
|
||||||
|
mode switch and the retry lever after a motion fault.
|
||||||
|
- `POST /cool/state` (job-state reports from the active controller,
|
||||||
|
level-triggered ~1 Hz) and `GET /cool/status` (engine phase, verdict,
|
||||||
|
temps, report age) — the cooling engine's channels; the verdict the
|
||||||
|
controllers enforce is the `/run/forgefirm/cooling.state` file, per
|
||||||
|
the SERVICES.md contract.
|
||||||
- `POST /diag/flow-verify`, `POST /diag/flow-calibrate`,
|
- `POST /diag/flow-verify`, `POST /diag/flow-calibrate`,
|
||||||
`POST /diag/abort`, `GET /diag/status` — the diagnostics runner
|
`POST /diag/abort`, `GET /diag/status` — the diagnostics runner
|
||||||
(own section below). `GET /status` carries a `diag` flag for the
|
(own section below). `GET /status` carries a `diag` flag for the
|
||||||
@@ -390,15 +444,18 @@ max. Images from 20260807204056 carry forgectrl at the bumped SRCREV
|
|||||||
## Diagnostics (forgectrl-owned hardware tests)
|
## Diagnostics (forgectrl-owned hardware tests)
|
||||||
|
|
||||||
The Diagnostics tab runs tools that **take the hardware over**: the
|
The Diagnostics tab runs tools that **take the hardware over**: the
|
||||||
runner (forgectrl diag.c, one slot) stops the `grblhal` service
|
runner (forgectrl diag.c, one slot) suspends the active controller
|
||||||
(launch is gated on cnc idle + no diagnostic), drives the loop
|
through the supervisor (launch is gated on cnc idle + no diagnostic),
|
||||||
directly through sysfs — the same model as the bench characterization
|
drives the loop directly through sysfs — the same model as the bench
|
||||||
scripts — and restarts the service on every exit path (completion,
|
characterization scripts — and resumes the controller on every exit
|
||||||
tool error, operator abort via `POST /diag/abort`, safety ceiling).
|
path (completion, tool error, operator abort via `POST /diag/abort`,
|
||||||
`/run/forgefirm-diag.active` marks the ownership; forgectrl startup
|
safety ceiling); the controller that returns is the selected mode's,
|
||||||
recovers a stale marker (stand-down + controller start), covering a
|
whichever that is. The cooling engine suspends its own writes for the
|
||||||
daemon crash mid-diagnostic. The laser is untouched throughout (latch
|
duration and publishes fire-blocked. `/run/forgefirm-diag.active`
|
||||||
stays locked). While a diagnostic runs: settings POSTs 409, `/status`
|
marks the ownership; forgectrl startup recovers a stale marker
|
||||||
|
(stand-down + controller resume), covering a daemon crash
|
||||||
|
mid-diagnostic. The laser is untouched throughout (latch stays
|
||||||
|
locked). While a diagnostic runs: settings POSTs 409, `/status`
|
||||||
reports `diag:true`, and the whole panel locks with a banner. Live
|
reports `diag:true`, and the whole panel locks with a banner. Live
|
||||||
progress (phase, elapsed, both coolant temps, a scrolling log) streams
|
progress (phase, elapsed, both coolant temps, a scrolling log) streams
|
||||||
through `GET /diag/status` on a 2.5 s poll; results persist on the
|
through `GET /diag/status` on a 2.5 s poll; results persist on the
|
||||||
@@ -418,14 +475,18 @@ windows hard-abort at 48 °C downstream):
|
|||||||
refuses when the gap is under 3 °C (raise the duty and rerun) —
|
refuses when the gap is under 3 °C (raise the duty and rerun) —
|
||||||
the per-machine path for replacement coolant or a swapped pump.
|
the per-machine path for replacement coolant or a swapped pump.
|
||||||
|
|
||||||
**Cooling tunables are conf-backed since 2026-08-08**: the nine
|
**Cooling tunables are conf-backed**: the nine `cool_*` keys
|
||||||
`cool_*` keys (flow_rise, flow_heater_pct, flow_check_s, recheck_s,
|
(flow_rise, flow_heater_pct, flow_check_s, recheck_s, confirm_max_s,
|
||||||
confirm_max_s, temp_max, temp_resume, cooldown_s, cooldown_max_s) live
|
temp_max, temp_resume, cooldown_s, cooldown_max_s) live in
|
||||||
in `/data/forgefirm.conf` (forgectrl Machine tab, validated ranges),
|
`/data/forgefirm.conf` (forgectrl Machine tab, validated ranges), and
|
||||||
the driver re-reads them at **every flood start** (env `GFCOOL_*` >
|
the **cooling engine** (forgectrl cool.c — the single fan/pump/TEC/
|
||||||
conf > compiled default; env stays the bench-override path — it wins
|
heater owner for both controller modes) re-reads them at **every run
|
||||||
for the process lifetime), and the conf parser now lives in
|
start** (env `GFCOOL_*` > conf > compiled default; env stays the
|
||||||
`glowforge_io.c` shared with homing.
|
bench-override path — it wins for the process lifetime). The GRBL
|
||||||
|
driver is a thin client of the engine: it reports job state, enforces
|
||||||
|
the published verdict in-process (fire gate, hold/resume, the
|
||||||
|
compiled-duty emergency fallback), and touches no thermal hardware
|
||||||
|
otherwise; the cloud client works the same way.
|
||||||
|
|
||||||
Bench record 2026-08-08 (hot-deployed binaries, all through the HTTP
|
Bench record 2026-08-08 (hot-deployed binaries, all through the HTTP
|
||||||
API): **conf plumbing** — `cool_flow_rise=8` posted, next M8's healthy
|
API): **conf plumbing** — `cool_flow_rise=8` posted, next M8's healthy
|
||||||
@@ -778,6 +839,25 @@ accordingly ("Automatic — AP country, else World").
|
|||||||
head accelerometer); until then the first-light procedure is:
|
head accelerometer); until then the first-light procedure is:
|
||||||
operator watches from the first commanded move and stops the
|
operator watches from the first commanded move and stops the
|
||||||
job on any no-motion.
|
job on any no-motion.
|
||||||
|
- **2026-08-11 (later, same day): root cause corrected and the
|
||||||
|
liveness gate landed.** The supply is fine — the **DRV8825
|
||||||
|
stepper drivers wedge on rail glitches** (operator diagnosis;
|
||||||
|
see the hardware facts bank): whether a given power-up leaves
|
||||||
|
them unserviceable is chance, which is why one clean-settle
|
||||||
|
baseline still failed. The mitigation stack is now: the
|
||||||
|
pulse-device broker (the rail never cycles on handovers), the
|
||||||
|
supervisor's **head-accelerometer liveness probe** before each
|
||||||
|
session's first controller spawn (+X-first per the cable rule,
|
||||||
|
laser latched; rail-off recovery ladder 5/15/30 s on a dead
|
||||||
|
verdict; `motion-fault` state when the drivers won't recover),
|
||||||
|
and **gfhome's hardened completion** (a run of near-identical
|
||||||
|
cloud corrections aborts the session; quiet without an
|
||||||
|
accel-witnessed motion window is a failure, not a homing —
|
||||||
|
proven the hard way when the service repeated one correction
|
||||||
|
eleven times into a motionless gantry, gave up, and the old
|
||||||
|
quiet heuristic reported homed). A genuine accel-witnessed
|
||||||
|
homing (8 motion windows, head at the corner,
|
||||||
|
operator-confirmed) closed the episode.
|
||||||
- **LASER_PWM waveform: PASSED 2026-08-02** (scope on the physical
|
- **LASER_PWM waveform: PASSED 2026-08-02** (scope on the physical
|
||||||
pin). Method: direct PWMSAR duty steps (`scripts/bench/pwm_sweep.py`
|
pin). Method: direct PWMSAR duty steps (`scripts/bench/pwm_sweep.py`
|
||||||
/ `pwm_hold.py`) with the controller stopped, cnc `disabled`
|
/ `pwm_hold.py`) with the controller stopped, cnc `disabled`
|
||||||
|
|||||||
Reference in New Issue
Block a user