The hardware watchdog is armed by the bootloader and kept fed by the
kernel core; nothing in userspace opens it. Until now nothing in the
catalog read that it is running. image.health reads watchdog0's state,
timeout and bootstatus (CONFIG_WATCHDOG_SYSFS, meta-openglow) and
requires active at 60 s; bootstatus goes into the evidence, so a campaign
that follows a watchdog reset says so.
Bench: on 2026-09-02 the forced hang ended in the factory recovery after
the 60 s timeout; the register read WCR 0x771f. The sysfs view itself
arrives with the next image, so this check is proven there.
Catalog consequence: image.health is always-run.
CAMPAIGN-LOG: the forced hang on image 20260902144848 with its times,
the watchdog register read back, U-Boot's recovery branch (the purple
button), the recovery image on the lease, and the power cycle back; the
pad measurement dropped by decision. BRINGUP item 9: both measurements
closed; the pushes, the pins, one build and the campaign on that image
remain.
CAMPAIGN-LOG: the attended set green on image 20260902144848, the three
defects the live runs and the takeover restarts found (the dwell-gap
latch rule, the log drops, the busy-start airflow) and their fixes, and
what is owed. BRINGUP item 9: both queues green; the measurements, the
pushes, the pins, one build, and the campaign on that image are what is
left.
A forgectrl started while the kernel is not idle takes the cooldown
airflow (forgectrl's busy-start rule), and on the bench it kept it: after
kernel.fire-line's takeover restarted the daemon with the kernel in the
drill's safe state, the exhaust ran at 6200 rpm on an idle machine until
the daemon was restarted by hand. cooling.fans-quiet-after-motion gains
the case: forgectrl stopped, cnc/disable written, forgectrl started, and
within 90 s the controller must be running with the idle duties applied.
The host replay stubs the init script and holds both outcomes: the idle
duties after the start, and a daemon that keeps the cooldown duties.
Bench: with forgectrl 522cdb2, the busy start logged, idle airflow one
tick later, the duties idle 15 s after the start, PASS.
Catalog consequence: the cooling.* implementation hashes move.
laser.emission-witness required the hardware button latch clear in every
sample the engine reported armed, and, after a first fix, in every sample
up to the last nonzero emission count. Both windows were drawn from
lagging signals: the engine's armed flag follows the controller's next
report, and the emission counter latches once per second and reads
nonzero about two seconds past the relock. Both reached into the tail
where the job-end relock sets the button latch by design, and the rule
refused three clean runs on image 20260902144848 (all four sides
burned; the trail shows the latch clear from the press to the relock,
emission through the fourth side, HV_ENABLE's dip in the dwell and its
return).
The rule now uses the window the hardware defines: from the first
emission, in every sample whose readback word shows the laser latch
unlocked, the button-latch bit of that same word must be clear. That
spans the kernel-run gap of the dwell and ends at the relock, and no
lagging flag can misplace it. dwell_gap() is a pure function;
tests/test_laser_dwell.py holds the relocked tail, a set inside the gap,
and a trail without emission. The recorded trail of the third run
replays to a pass (47 unlocked samples, none set).
The live runs keep a per-sample trail in the evidence (TRAIL_FIELDS: the
readback word, the switches, the lock flag, the controller's state and
messages), so a run's timeline can be read back without a rerun.
A fourth run then errored on a name the refactor had removed and one
later check still used; py_compile does not catch it and a live drill
never executes on the host, so the CI job now fails on any undefined
name in the harness (pyflakes).
Catalog consequence: the laser implementation hashes move.
The daemons log to /dev/log with non-blocking datagrams and drop what
the socket will not take. The kernel's default queue for a unix datagram
socket is 10; forgectrl's arm-time burst alone was about 20, so the
lines around every job start were lost, and the daemon reported 18 to
46 dropped per job on the bench. The logging init sets
net.unix.max_dgram_qlen to 512 before rsyslog and the daemons start;
nothing on the image applied sysctl files before.
Bench: set at runtime on 20260902144848 for the rest of the session.
Catalog consequence: a layer change; the meta-forgefirm content hash
moves and nothing inherits on the image that ships it.
CAMPAIGN-LOG: the unattended set on image 20260902144848, the six
harness and diagnostic defects it found and their fixes, the numbers,
and the board's state at the end of the pass. BRINGUP: item 9 names the
reset in the hang case and the session under way; the facts bank gains
the PIC read-pattern measurement; Next work item 10 is a pacing of PIC
reads in the kernel.
update.slots-and-signature's apply section required 200 from
POST /update/apply, where the daemon answers 202 with started, like
every job endpoint, so its first bench run on image 20260902144848
ended before the job did; the cleanup then deleted the staged archive
under the running job. The drill requires 202 and started, and looks
for the daemon's refusal ("archive is not signed with the ForgeFIRM
release key"). Bench: the apply started, the job ended with that
refusal, PASS.
A queue started 4 s after a forgetest restart ran 7 tests instead of
10. The bench page's /state poll had a fixture probe in flight (an mDNS
answer), probe_fixture stamped its time at its start, and the queue
start read the stale fixture, none, so the three operator tests the
fixture runs in the unattended queue were routed to nobody. The probe
now runs under a lock and is stamped when it completes: a caller that
arrives during a probe waits for its answer. tests/test_fixture.py
holds the race with a slow scripted probe; it fails on the old code.
Catalog consequence: the update implementation hash moves; the runner
change is dev-only.
A queue runs the catalog in registration order among tests with the
same prerequisites, and cooling.aa-offset-calibrate followed
cooling.flow-verify. A flow-verify trial heats the tube water (a no-flow
trial by 17 C on the bench) and the warm slug circulates past the
coolant sensors for minutes afterward; the calibration's stationary gate
passed 44 s after the trial on image 20260902144848 and the edges read
the wave as disagreement.
The calibration is registered first now, with the reason beside it, and
tests/test_cooling_order.py holds the order.
Catalog consequence: the cooling.* implementation hashes move (the suite
file changed).
The hang case resumed the controller and sent $X alone. The stream
fault raises Alarm 17 (motor fault), which the core treats as a critical
event: $X is refused with error:79 until a soft reset, and the reset is
what the stream takes as the operator's acknowledgment of the fault
(grblHAL 38b450e: the kernel stopped and re-armed, the stale ring
cleared, the producer armed again). The drill's first bench run on image
20260902144848 therefore ended in Alarm.
The drill now records $X before the reset and requires the error:79
refusal, sends the soft reset, unlocks, requires Idle, requires the ring
back at its idle free count (the stale bytes of the interrupted move are
gone), and then jogs. The final assertion read the state off the report
dict returned by wait_idle as a string; it never ran before because the
TIMEOUT test short-circuited it.
Bench: motion.deadman PASS on 20260902144848 (kill respawn 1.2 s, hang
to underrun 0.21 s, $X -> ALARM:17 error:79, reset + $X -> Idle, ring
33521664 of 33521664, jog Jog -> Idle, restart retook supervision).
Catalog consequence: the motion.* implementation hashes move (the suite
file changed) and the set re-ran and passed. The failed first run had
closed the campaign, so the always-required core ran again in the new
one, as the campaign rules require.
The gate that refuses a latch unlock while the chain may hold HV_ENABLE
up (charge_pump_alive or a pulse engine not idle) ran at the start of
phases B, U and K3 of kernel.fire-line, within a second of the previous
phase's run. A run feeds the charge-pump watchdog every 200 ms and the
one-shot holds ALIVE for 0.45 s after the last feed, so the gate read
alive=1 and refused: the first bench run of the gate (forgefirm
64f552fc; the laser_pgood gate before it was vacuous) failed phase B on
image 20260902144848.
wait_hv_off() polls the chain for up to 3 s before it refuses, logs the
release when it was not immediate and records every wait in the
evidence (hv_release_s). require_hv_off and check_hv_off use it. The
bench scripts that copy the gate (fire_test.py per phase,
gate_a_kernel_drills.py K3 after K2) get the same wait.
Bench: kernel.fire-line PASS on 20260902144848 with the chain released
after 0.41 s at each of the three phase boundaries. Host:
tests/test_kernel_suite.py covers release inside the window, a chain
held past it, and a chain already off.
Catalog consequence: the kernel.* implementation hashes move (the suite
file changed); the kernel set re-ran and passed.
The manifest lists the modules directory without the kernel's
LOCALVERSION_AUTO hash (the hash does not reproduce across a re-patch of
the same source, so the image manifest strips it). image.health still
compared the full running release against that list and failed on the
first post-flash run of image 20260902144848 with the kernel
6.12.20-fslc-fslc-g72a0b1431a9d against the manifest's 6.12.20-fslc-fslc.
kernel_ident() strips the same suffix from both sides, so a manifest with
or without the hash matches the running kernel, and a different base
release still fails. tests/test_image.py covers both forms.
Catalog consequence: only image.health's own implementation hash moves;
it is an always-run test, so no inherited result is affected.
BRINGUP describes the present: the 54-test catalog and its seven-test
always core, the tier counts, the shipped low-temperature gates, the
density floor ($35 = 10), the two local core commits, the ffboot env
write, the aa-offset route, the current bench image, and the bench
measurements the audit asks for (pooled into the next session). The
workstation shell notes and every em dash are gone.
forgetest: the takeover waits for the cloud client too (found by its
command line); the unauthenticated /boot probe names the endpoint's
parameter; the UI prose is American English. Recipes: forgetest
fetches its package directory and init script only and drops
__pycache__ at unpack; the dev image no longer re-adds forgectrl; the
release image's remove list drops the gfui-client the BSP no longer
has; the platform identity strips the kernel's local-version hash
from the modules directory name, so a re-patched kernel keeps its
fingerprints. grblhal restart is stop then start. release.sh --dev
packs the dev image. fixture.sh refuses a readable env file.
Bench tools: the live-fire drills measure the lid-IR baseline before
every run and point at the fire-watch thresholds the engine reads;
one thermistor conversion (gfbench.degc) serves every drill; the six
dated measurement records leave the tool directory; feeder.c names the
two sysfs writes its caller makes.
Host tests: forgetest 258 pass; the coverage lint reports no uncovered
path across 54 tests. Acceptance: forgectrl.auth covers the /boot
probe; update.* cover ffboot and the manifest identity; the runbook
and bench-tool changes have no catalog consequence.
A read-only mount of an ext4 slot still replays its journal, which
writes to the partition the probe was not meant to touch. The probe
mounts with noload: the journal is left as it is and the slot's
content is read as it stands.
Acceptance: update.slots covers ffboot -l (covers map ffboot/**).
A test's fingerprint covered its own text and its module's shared text
only, so a judge imported from a sibling suite module (laser.py takes
its motion judges from motion.py) could change without moving the
fingerprints of the tests that call it. The shared text of every sibling
module a module imports now rides along, transitively; unit test.
update.slots-and-signature claimed to refuse a tampered signature but
fed fwup one garbage file. It now makes a throwaway key pair on the
machine, signs a tiny archive, checks that the archive verifies with its
own key and fails against the shipped release key, and asks the update
job to apply it without confirm_unsigned: the job refuses it for its
signature before touching the slot.
BRINGUP named nine cool_* tunables (there are thirty), a cool_fire_ir_delta
key nothing reads, and panel source files that no longer exist; the
inventory is the present one. The bench page's help popovers pointed at a
documentation host and paths that do not exist; they open the site.
FORGEFIRM_RELEASE said 0.1.0, the first non-beta number by the settled
rule, so the first cut was either refused as 0.0.1 or shipped as a
non-beta. The recipe now says 0.0.1, and release.sh refuses a version at
or above 0.1.0 while the README carries the beta banner.
The faultpos live-fire drill armed and commanded a cut at an origin it
called stale to test a refusal the design decided not to gate: its only
outcome was an emission at an unknown position. Removed from the script
and the bench page.
live_fire_drills.py discarded every drill's return value, so the bench
page recorded a failed live-fire drill as OK. The exit status is the
drill's.
laser_pgood is the supply's power-good, high on every healthy machine;
fire_test.py and the K3 drill aborted on it and pgood_probe.py inverted
it. The latch-unlock drills now gate on the safety chain holding HV off
(charge-pump watchdog dead, pulse engine idle), as the kernel suite
does, and the probe reports the pin as the kernel publishes it.
motion.deadman: the controller resumed from its hang recovers on $X and
moves again without a restart (the stream's fault acknowledgment).
The archive was written straight to its final name and a rerun accepted
any non-empty file as complete, so a run interrupted mid-archive left a
truncated image that the next run kept and then overwrote the slot. The
archive is now written as .part and renamed on success; an existing
archive counts only with its manifest line present and a whole gzip
stream, otherwise it is archived again.
With the gate skipped the script still staged and attached whatever
acceptance.json existed, indistinguishable from a gated release, and
sha256sums.txt never listed the artifact. A skipped gate now stages
NO-ACCEPTANCE.txt instead and publishes as a prerelease; every attached
file is listed in the sums.
The inheritance walk skipped every record that was not a PASS on the
current fingerprint, so a FAIL or ERROR recorded after a PASS on the
same image was stepped over and the older PASS inherited into the next
campaign. The newest record on the fingerprint now decides: a PASS is
inherited, a FAIL or ERROR blocks it (reason failed-since), an ABORTED
run says nothing. Unit tests for all three orders.
ffboot, the tool that rewrites the boot environment on every install and
slot switch, was packaged from scripts/ outside every fingerprint. It
now lives in the recipe's files and the recipe inherits the manifest
class; the tree manifest tool fingerprints file components the same
way, and the update tests cover the component.
forgectrl.settings-bounds fell back to ui_units=mm, which the whitelist
refuses, so the always-required test failed on a fresh machine; the
fallback is metric.
Rule 22: M3 S1000 with the window open, then jogs from Idle (a sender's
Fire button plus its Move panel) ship every jog tick dark, and the cut
after them is lit. Rule 23: two cuts at S300 and S1000 queued together
render the S300 cruise at the S300 density; the rolloff shapes against
the segment's own velocity ratio, never the newest S over the block's.
Both fail on the driver before its fix.
The start gate set just above the coolant opens the armed session under
the warm-up; the cloud client waits it out after the button, nothing
runs, and the release starts the print, which completes. The mid-run
hold and its bound are host-tested: the gates apply at session open, so
no setting can produce a hold mid-run on the bench. Bench-excerpt unit
tests: the pass, and the failure when a run starts under the hold.
A pulse device closed while locked and running is the kernel dead man's
switch. The drill enables the lens driver at low current, runs a padded
stream under the takeover, drops the device with the lock held, and
checks that the engine halts at once, the latch is locked, and the head
is in its safe state (z_enable 1, z_current 1, measure laser and UV LED
off). Always required; the kernel covers map already names src/head.c.
A job longer than the ring keeps its feeder alive through the button
wait, so the cancel there has to stop the feeder before the park clears
the ring. The drill now loads such a job, requires the "longer than the
ring" line, and after the cancel reads the program total twice over the
feeder's retry period (zero both times) and cnc/streaming (zero). The
bench-excerpt unit test carries the long-job line and the new evidence.
Covers map unchanged: the drill already names gfhardware/machine.py.
The recipe URLs, the release and install URLs, the vendor check and the CI checkouts name openglow-org, and the grblHAL core fork is openglow-org/grblHAL-core. No catalog change is owed: the recipe edits move the meta-forgefirm content hash, which every test fingerprint folds in through the platform block, so the whole catalog re-runs on its own.
The supply's power-good line is active high, static across HV enable and emission, and driven; the facts bank and CAMPAIGN-LOG carry the measurement and the item closes. The kernel-drill latch-unlock guard read the old inverted value as HV not good, a check that was vacuous and would refuse every run once the module reads the line correctly; it now uses the chain's own witnesses, the charge-pump watchdog and the engine state. pgood_probe.py watches the line beside the chain through the kernel readbacks and is registered on the bench page. The dev image lists python3-mmap and python3-ctypes again for the pad-level bench tools the python trim had left without them.
Stream harness rule 21: a feed hold leaves no dark ground in either mode (lit into the hold, dark while held, lit from the first step out), with realtime and wait-state steps in the session runner. Lifecycle harness: the hold a sender change puts a running job into, the resume that re-arms a held job from the sender and from the button, a reset from a held job, and the resume after the grace closed the window in Hold. Live-fire drills: holdres (the pause as a corner in time, the re-arm after the grace); senderchg follows the hold. BRINGUP: the gapless-pause and sender-change items close, the facts bank gains the measured hold and resume behavior; CAMPAIGN-LOG records the proof and the bench runs. Acceptance: the pause-resume-lid-cancel text follows the behavior; the driver stays covered by src/**.
INSTALL.md, SERIAL.md, docs/COOLING.md, docs/LIGHTBURN.md,
docs/MOTION.md, docs/SAFETY.md, docs/UPDATE-SYSTEM.md, docs/VIDEO.md and
their images are pages on https://docs.forgefirm.org/ now. Every
reference in the README, BRINGUP, the kas config, the cold-build
workflow, forgetest, and the bench scripts points to the site page. The
README carries the beta banner. docs/ keeps BRINGUP.md and
CAMPAIGN-LOG.md.
No catalog consequence: the deleted files are documents, and the code
changes are comment and help-text repoints only.
forgectrl 161686f adds the head-accelerometer crash watch (two tiers
on the LIS2HH12's interrupt generators, armed inside the laser's
armed window). The catalog grows cooling.crash-watch-plumbing - the
unarmed half of the watch provable without a button press: accel_watch
stays at watch through an unarmed session, three zeros read as the
three crash gates off, an out-of-range threshold is refused - and
_COOL_COVERS widens to src/accel.*. The armed-tier bench drill is a
BRINGUP next-work item.
The de-risk drill script's CTRL7 write moves from bit 0 (4D_IG1) to
bit 2 (LIR1), the latch the datasheet actually puts there; the drill's
findings did not depend on latching (a continuous gravity signal).
The captured pulse headers carry the factory's IG programming: hunts
off, travels abort-only (HAar=133 at +/-4 g), the cut alert-only
(HAxr=132, HAyr=112), Z and idle never armed. The factory abort value
itself rules out an FS/128 LSB (4.16 g would sit over the measurable
range), and two Z-only bench windows bracket gravity between
thresholds 100 and 150, which only FS/256 explains. The drill script's
printed g conversion is corrected, and the stale motion-profile
reading of HAxr/HAyr/HAar as accel limits is corrected in the facts
bank.
No catalog consequence: a bench script's printed conversion and the
docs record; no shipped behavior changes.
The de-risk drill ran on the bench: the IG registers program and poll
over i2c-dev (I2C_SLAVE_FORCE) with st_accel bound, raw reads intact,
so the crash detector is forgectrl-only (BRINGUP item 6, CAMPAIGN-LOG
2026-08-31). The drill found the IG needs a running ODR: st_accel
leaves the part in power-down between one-shot reads. The script now
saves CTRL1, runs the window at 800 Hz, and restores it on exit; its
old ODR test checked the axis-enable bits instead of the ODR bits.
No catalog consequence: the change is a bench drill script, its bench
page text, and the docs record of the drill; no shipped behavior
changes.
accel_crash_probe.py arms the head LIS2HH12's on-chip interrupt
generator (IG_CFG1/IG_THS/IG_DUR1) and polls the latched IG_SRC1 for a
strike, reporting the axes and raw magnitude. It settles the bench fact
the crash detector rides on: coexist mode reaches the IG registers over
i2c-dev with I2C_SLAVE_FORCE while st_accel stays bound, so it proves
whether the detector can be forgectrl-only with the liveness path
untouched, or whether the accel must move under glowforge.ko. It touches
only the IG registers (0x30-0x35) plus the CTRL7 latch bit, never the
full scale, so st_accel's raw scaling is undisturbed; no emission, no
commanded motion by default.
Registered on the bench page (dry, board) and in the bench README.
BRINGUP item 6 now stages the drill as the committed first step, with
the readout path, per-state thresholds and the two-tier wiring owed
after it. Tooling only: no shipped component source changed, so no
acceptance-catalog consequence (the detector feature gets its case when
it is built); the bench registry test and coverage lint pass.
A firmware decode of the head MCU (recorded in CAMPAIGN-LOG) shows the
factory head crash detector and the head IRQ accel_irq source are one
mechanism: the head accelerometer (LIS2HH12) has an on-chip interrupt
generator the factory arms per job from the HA* header tags (bit-exact
onto its registers) and reads by polling IG_SRC1; its INT wires to the
head MCU and surfaces as reg 0x05 b1 behind HEAD_IRQ, which is
level-driven off the MCU's latched reg 0x02 and dormant until the SoC
arms an edge in reg 0x03/0x04.
Facts bank: add "The head MCU flag register and HEAD_IRQ", "The head
accelerometer", "Beam detect in the head MCU" (correcting the thinner
entry: reg 0x05 has a fourth input and a b7 processed verdict we do not
expose; beam_detect_digital is the raw comparator, not the verdict).
Next work: merge items 6 and 8, renumber 12 to 11, repoint the trailer.
CAMPAIGN-LOG: dated entry recording the decode. Docs only.
Facts bank: add the head-IRQ/beam-detect entry and the coolant-ADC
offset entry; point the SDIO CRC note at the closed watch record.
Next work: trim items 6-10 to the open work with facts-bank citations,
remove the stale bench-page sentence from item 3, and retitle item 10
to the second-machine re-measure (the sensor-line scope work is
dropped by decision). Docs only, no catalog consequence.
Load/unload under DEBUG_MUTEXES (three clean cycles) and the forced
-EPROBE_DEFER unwind both passed on the debug-kernel image, no lock
splat, machine recovered (CAMPAIGN-LOG has the run). BRINGUP: the item
closes, items 4 and up move down one.
Three fixes found running the drills: the splat regex no longer matches
the benign "RCU lockdep checking is enabled" boot banner; the 40 V
regulator search reaches its reg-fixed-voltage driver (regulators:40v)
so the forced-defer drill resolves the resource instead of skipping;
and require_idle waits out the run-through-'running' a forgectrl restart
passes through instead of refusing.
forgefirm-image-dev has one deploy slot, so the debug variant was
clobbering the normal dev image (and a dev rebuild clobbered the debug
one back). A distinct IMAGE_BASENAME (forgefirm-image-dev-debug) lets
both coexist in the deploy directory.
kas/forgefirm-glowforge-debug.yml builds one dev image on the debug
kernel (FORGEFIRM_KERNEL_DEBUG=1, tagged dev-debug) beside the closing
release and dev images; the debug options never touch either.
scripts/bench/debug_kernel_drills.py runs the two drills (three
load/unload cycles under DEBUG_MUTEXES, a forced -EPROBE_DEFER unwind),
each read against dmesg for lock splats, refusing on a non-debug kernel
or a non-idle machine; it is registered as a board bench tool. BRINGUP
item 3 names the mechanism and stays remaining work (run it on the
closing burn); the site's Build page documents the variant.
A head reset (its own 0xc9<-0x5a register) made the present head NAK
through its reboot; the witness reads errored (K-11 propagation) with no
spoofed positive, and the head recovered on a driver rebind
(CAMPAIGN-LOG has the drill). BRINGUP: the last physical-evidence
negative closes, items 4 and up move down one.
ScottW514