The record of what the first attended run really found and what it cost:
the verdict had no run-session identity, so the arm opened its window on
the verdict computed for the idle session before it. The build entry
lists the four components taken from local commits and the checks made
on the built images.
BRINGUP moves to the new campaign image and says why nothing inherits.
The cooling verdict now carries the engine's own armed flag, so the
stand-in engines in both null-sink harnesses publish it. The lifecycle
harness gains two cases: an engine that never takes the armed window
must produce a refused arm and no emission, and one that takes it a
couple of seconds late must produce a wait and then a normal arm. The
late case is the one that proves the controller keeps reading the
verdict while it is blocked in the arm; without that every job would
fail there.
The emission witness gains the bench form of the same rule: no sample
may show the laser firing while the cooling engine reports a phase that
runs the fans at their idle duty. That is what a burn with no airflow
looks like from the outside, and nothing in the catalog looked for it.
The record of the live-fire failure on 2026-09-03: a job fired for about a
second with idle airflow under a warm-up hold that arrived one tick later,
the leftover start-gate setting behind it, the engine's arm-time gate gap it
exposed, and what is owed before any further live fire.
The record of the campaign's unattended set on p32, the numbers the two new
kernel drills and the air-assist offset diagnostic gave under the settle,
and the forgetest restart that interrupted the last test, with the rule
that keeps it from recurring.
The kernel's spin is its own load level, a count or two under a Python
spin, so the third check no longer compares the settled level with the
Python-spin control; it requires the settled idle reader to have moved at
least half the control's split off the idle regime. Bench: PASS on image
20260903011655 (control split +6, settled split +1).
BRINGUP item 9: items 10 and 11 are on image 20260903011655; owed is the
flash, the fresh-boot baseline, and the full campaign, then the push and
the pin bumps. CAMPAIGN-LOG: the p32 build and its checks.
The PIC16F1713 converts its inputs in a free-running loop (10 channels,
about 0.30 ms a loop) and a read returns the last conversion of that
channel; the count follows the SoC's load at conversion time (idle 659,
busy 665 on the coolant thermistors, both tight; every channel shifts in
proportion to its count; the step lands within one PIC loop of the CPU
changing state, with a regulator's overshoot in each direction). The
kernel.pic-soc-load drill replaces kernel.pic-pacing: 200 reads after 3 ms
of sleep and 200 after 3 ms of spinning, with the module's settle off
(the control, reported) and on (the claim: the two agree). The catalog
counts 56 tests, 0 uncovered.
BRINGUP item 10 and the facts bullet describe the mechanism and the fix;
the CAMPAIGN-LOG entries record the first pass of the campaign, the PIC
study, and the mechanism's proof.
The sysfs state of watchdog0 says whether a process holds the device, and
none does: the kernel's core feeds the boot-armed hardware. The check
reads WCR through /dev/mem (WDE set, a 60 s period) and expects the state
to read inactive. Bench: PASS on image 20260903003213 with WCR 0x771f. The
BRINGUP facts bullet says the same.
BRINGUP item 9: items 10 and 11 are on image 20260903003213; owed is the
flash, the fresh-boot baseline, and the full campaign, then the push and
the pin bumps. CAMPAIGN-LOG: the p31 build and its checks.
kernel.pic-pacing reads a coolant thermistor twice back to back, 300
pairs, with the module's pacing off (the control, reported) and on (the
claim: the second read agrees with the first). The bench proof for the
module's pic_gap_us pacing; the catalog counts 56 tests, 0 uncovered.
BRINGUP item 10 and the facts bullet describe the pacing as it is; the
CAMPAIGN-LOG entry records the change and its proof.
BRINGUP item 9: the deferred batch is on image 20260903000529; owed is the
flash, the fresh-boot baseline, and the full campaign, then the push and
the pin bumps. CAMPAIGN-LOG: the p30 build and its checks.
B-16: BB_SIGNATURE_LOCAL_DIRS_EXCLUDE in the distro conf names __pycache__
and .pytest_cache, so a workstation's bytecode caches never enter a
file:// checksum (proven in the build VM: a cache under the package leaves
the fetch task alone, a source change reruns it).
kernel.resume-lead: two phases behind one takeover at a 1 kHz tick. E, a
resume whose lead is longer than the data ends at end-of-data within a few
ticks (a lost end-of-data would show as 255 ms). L, a 1000-byte lead over
FIRE bits with the latch unlocked and the chain unarmed keeps the FIRE
line low through the lead and drives it from the waypoint byte on. The
bench proof for the module's K-4 and K-8 fixes; the catalog counts 55
tests, 0 uncovered.
BRINGUP items 9 and 11 and the CAMPAIGN-LOG entry record the batch and the
campaign rule: no campaign until every audit finding is on one image.
CAMPAIGN-LOG: the push in CI order, the pin bumps, and build p29 with its
checks. BRINGUP item 9: the remediation is pushed, pinned, and built; owed
now is the flash, the fresh-boot baseline, and the full campaign on the
image, which the release gate asks for.
B-16: the forgetest recipe fetches the package directory whole, and the
file fetcher's checksum is taken before do_unpack drops __pycache__, so a
host `__pycache__` written by running the tests moves the recipe's task
hash with no source change. The three python steps run with -B now;
`src-sync` to the build VM excludes the caches as well. A developer who
runs the tests locally without -B still creates them; there is no exclude
on the file fetcher, so this is the floor, not a cure (BRINGUP item 11).
The audit remediation's commits and the bench session's fixes in the
three repositories, pushed and bench-proven on image 20260902144848
(built from the same commits through a local pin overlay). Each PV
moves with its SRCREV; the manifest records the PV.
The hardware watchdog is armed by the bootloader and kept fed by the
kernel core; nothing in userspace opens it. Until now nothing in the
catalog read that it is running. image.health reads watchdog0's state,
timeout and bootstatus (CONFIG_WATCHDOG_SYSFS, meta-openglow) and
requires active at 60 s; bootstatus goes into the evidence, so a campaign
that follows a watchdog reset says so.
Bench: on 2026-09-02 the forced hang ended in the factory recovery after
the 60 s timeout; the register read WCR 0x771f. The sysfs view itself
arrives with the next image, so this check is proven there.
Catalog consequence: image.health is always-run.
CAMPAIGN-LOG: the forced hang on image 20260902144848 with its times,
the watchdog register read back, U-Boot's recovery branch (the purple
button), the recovery image on the lease, and the power cycle back; the
pad measurement dropped by decision. BRINGUP item 9: both measurements
closed; the pushes, the pins, one build and the campaign on that image
remain.
CAMPAIGN-LOG: the attended set green on image 20260902144848, the three
defects the live runs and the takeover restarts found (the dwell-gap
latch rule, the log drops, the busy-start airflow) and their fixes, and
what is owed. BRINGUP item 9: both queues green; the measurements, the
pushes, the pins, one build, and the campaign on that image are what is
left.
A forgectrl started while the kernel is not idle takes the cooldown
airflow (forgectrl's busy-start rule), and on the bench it kept it: after
kernel.fire-line's takeover restarted the daemon with the kernel in the
drill's safe state, the exhaust ran at 6200 rpm on an idle machine until
the daemon was restarted by hand. cooling.fans-quiet-after-motion gains
the case: forgectrl stopped, cnc/disable written, forgectrl started, and
within 90 s the controller must be running with the idle duties applied.
The host replay stubs the init script and holds both outcomes: the idle
duties after the start, and a daemon that keeps the cooldown duties.
Bench: with forgectrl 522cdb2, the busy start logged, idle airflow one
tick later, the duties idle 15 s after the start, PASS.
Catalog consequence: the cooling.* implementation hashes move.
laser.emission-witness required the hardware button latch clear in every
sample the engine reported armed, and, after a first fix, in every sample
up to the last nonzero emission count. Both windows were drawn from
lagging signals: the engine's armed flag follows the controller's next
report, and the emission counter latches once per second and reads
nonzero about two seconds past the relock. Both reached into the tail
where the job-end relock sets the button latch by design, and the rule
refused three clean runs on image 20260902144848 (all four sides
burned; the trail shows the latch clear from the press to the relock,
emission through the fourth side, HV_ENABLE's dip in the dwell and its
return).
The rule now uses the window the hardware defines: from the first
emission, in every sample whose readback word shows the laser latch
unlocked, the button-latch bit of that same word must be clear. That
spans the kernel-run gap of the dwell and ends at the relock, and no
lagging flag can misplace it. dwell_gap() is a pure function;
tests/test_laser_dwell.py holds the relocked tail, a set inside the gap,
and a trail without emission. The recorded trail of the third run
replays to a pass (47 unlocked samples, none set).
The live runs keep a per-sample trail in the evidence (TRAIL_FIELDS: the
readback word, the switches, the lock flag, the controller's state and
messages), so a run's timeline can be read back without a rerun.
A fourth run then errored on a name the refactor had removed and one
later check still used; py_compile does not catch it and a live drill
never executes on the host, so the CI job now fails on any undefined
name in the harness (pyflakes).
Catalog consequence: the laser implementation hashes move.
The daemons log to /dev/log with non-blocking datagrams and drop what
the socket will not take. The kernel's default queue for a unix datagram
socket is 10; forgectrl's arm-time burst alone was about 20, so the
lines around every job start were lost, and the daemon reported 18 to
46 dropped per job on the bench. The logging init sets
net.unix.max_dgram_qlen to 512 before rsyslog and the daemons start;
nothing on the image applied sysctl files before.
Bench: set at runtime on 20260902144848 for the rest of the session.
Catalog consequence: a layer change; the meta-forgefirm content hash
moves and nothing inherits on the image that ships it.
CAMPAIGN-LOG: the unattended set on image 20260902144848, the six
harness and diagnostic defects it found and their fixes, the numbers,
and the board's state at the end of the pass. BRINGUP: item 9 names the
reset in the hang case and the session under way; the facts bank gains
the PIC read-pattern measurement; Next work item 10 is a pacing of PIC
reads in the kernel.
update.slots-and-signature's apply section required 200 from
POST /update/apply, where the daemon answers 202 with started, like
every job endpoint, so its first bench run on image 20260902144848
ended before the job did; the cleanup then deleted the staged archive
under the running job. The drill requires 202 and started, and looks
for the daemon's refusal ("archive is not signed with the ForgeFIRM
release key"). Bench: the apply started, the job ended with that
refusal, PASS.
A queue started 4 s after a forgetest restart ran 7 tests instead of
10. The bench page's /state poll had a fixture probe in flight (an mDNS
answer), probe_fixture stamped its time at its start, and the queue
start read the stale fixture, none, so the three operator tests the
fixture runs in the unattended queue were routed to nobody. The probe
now runs under a lock and is stamped when it completes: a caller that
arrives during a probe waits for its answer. tests/test_fixture.py
holds the race with a slow scripted probe; it fails on the old code.
Catalog consequence: the update implementation hash moves; the runner
change is dev-only.
A queue runs the catalog in registration order among tests with the
same prerequisites, and cooling.aa-offset-calibrate followed
cooling.flow-verify. A flow-verify trial heats the tube water (a no-flow
trial by 17 C on the bench) and the warm slug circulates past the
coolant sensors for minutes afterward; the calibration's stationary gate
passed 44 s after the trial on image 20260902144848 and the edges read
the wave as disagreement.
The calibration is registered first now, with the reason beside it, and
tests/test_cooling_order.py holds the order.
Catalog consequence: the cooling.* implementation hashes move (the suite
file changed).
The hang case resumed the controller and sent $X alone. The stream
fault raises Alarm 17 (motor fault), which the core treats as a critical
event: $X is refused with error:79 until a soft reset, and the reset is
what the stream takes as the operator's acknowledgment of the fault
(grblHAL 38b450e: the kernel stopped and re-armed, the stale ring
cleared, the producer armed again). The drill's first bench run on image
20260902144848 therefore ended in Alarm.
The drill now records $X before the reset and requires the error:79
refusal, sends the soft reset, unlocks, requires Idle, requires the ring
back at its idle free count (the stale bytes of the interrupted move are
gone), and then jogs. The final assertion read the state off the report
dict returned by wait_idle as a string; it never ran before because the
TIMEOUT test short-circuited it.
Bench: motion.deadman PASS on 20260902144848 (kill respawn 1.2 s, hang
to underrun 0.21 s, $X -> ALARM:17 error:79, reset + $X -> Idle, ring
33521664 of 33521664, jog Jog -> Idle, restart retook supervision).
Catalog consequence: the motion.* implementation hashes move (the suite
file changed) and the set re-ran and passed. The failed first run had
closed the campaign, so the always-required core ran again in the new
one, as the campaign rules require.
The gate that refuses a latch unlock while the chain may hold HV_ENABLE
up (charge_pump_alive or a pulse engine not idle) ran at the start of
phases B, U and K3 of kernel.fire-line, within a second of the previous
phase's run. A run feeds the charge-pump watchdog every 200 ms and the
one-shot holds ALIVE for 0.45 s after the last feed, so the gate read
alive=1 and refused: the first bench run of the gate (forgefirm
64f552fc; the laser_pgood gate before it was vacuous) failed phase B on
image 20260902144848.
wait_hv_off() polls the chain for up to 3 s before it refuses, logs the
release when it was not immediate and records every wait in the
evidence (hv_release_s). require_hv_off and check_hv_off use it. The
bench scripts that copy the gate (fire_test.py per phase,
gate_a_kernel_drills.py K3 after K2) get the same wait.
Bench: kernel.fire-line PASS on 20260902144848 with the chain released
after 0.41 s at each of the three phase boundaries. Host:
tests/test_kernel_suite.py covers release inside the window, a chain
held past it, and a chain already off.
Catalog consequence: the kernel.* implementation hashes move (the suite
file changed); the kernel set re-ran and passed.
The manifest lists the modules directory without the kernel's
LOCALVERSION_AUTO hash (the hash does not reproduce across a re-patch of
the same source, so the image manifest strips it). image.health still
compared the full running release against that list and failed on the
first post-flash run of image 20260902144848 with the kernel
6.12.20-fslc-fslc-g72a0b1431a9d against the manifest's 6.12.20-fslc-fslc.
kernel_ident() strips the same suffix from both sides, so a manifest with
or without the hash matches the running kernel, and a different base
release still fails. tests/test_image.py covers both forms.
Catalog consequence: only image.health's own implementation hash moves;
it is an always-run test, so no inherited result is affected.
BRINGUP describes the present: the 54-test catalog and its seven-test
always core, the tier counts, the shipped low-temperature gates, the
density floor ($35 = 10), the two local core commits, the ffboot env
write, the aa-offset route, the current bench image, and the bench
measurements the audit asks for (pooled into the next session). The
workstation shell notes and every em dash are gone.
forgetest: the takeover waits for the cloud client too (found by its
command line); the unauthenticated /boot probe names the endpoint's
parameter; the UI prose is American English. Recipes: forgetest
fetches its package directory and init script only and drops
__pycache__ at unpack; the dev image no longer re-adds forgectrl; the
release image's remove list drops the gfui-client the BSP no longer
has; the platform identity strips the kernel's local-version hash
from the modules directory name, so a re-patched kernel keeps its
fingerprints. grblhal restart is stop then start. release.sh --dev
packs the dev image. fixture.sh refuses a readable env file.
Bench tools: the live-fire drills measure the lid-IR baseline before
every run and point at the fire-watch thresholds the engine reads;
one thermistor conversion (gfbench.degc) serves every drill; the six
dated measurement records leave the tool directory; feeder.c names the
two sysfs writes its caller makes.
Host tests: forgetest 258 pass; the coverage lint reports no uncovered
path across 54 tests. Acceptance: forgectrl.auth covers the /boot
probe; update.* cover ffboot and the manifest identity; the runbook
and bench-tool changes have no catalog consequence.
A read-only mount of an ext4 slot still replays its journal, which
writes to the partition the probe was not meant to touch. The probe
mounts with noload: the journal is left as it is and the slot's
content is read as it stands.
Acceptance: update.slots covers ffboot -l (covers map ffboot/**).
A test's fingerprint covered its own text and its module's shared text
only, so a judge imported from a sibling suite module (laser.py takes
its motion judges from motion.py) could change without moving the
fingerprints of the tests that call it. The shared text of every sibling
module a module imports now rides along, transitively; unit test.
update.slots-and-signature claimed to refuse a tampered signature but
fed fwup one garbage file. It now makes a throwaway key pair on the
machine, signs a tiny archive, checks that the archive verifies with its
own key and fails against the shipped release key, and asks the update
job to apply it without confirm_unsigned: the job refuses it for its
signature before touching the slot.
BRINGUP named nine cool_* tunables (there are thirty), a cool_fire_ir_delta
key nothing reads, and panel source files that no longer exist; the
inventory is the present one. The bench page's help popovers pointed at a
documentation host and paths that do not exist; they open the site.
FORGEFIRM_RELEASE said 0.1.0, the first non-beta number by the settled
rule, so the first cut was either refused as 0.0.1 or shipped as a
non-beta. The recipe now says 0.0.1, and release.sh refuses a version at
or above 0.1.0 while the README carries the beta banner.
The faultpos live-fire drill armed and commanded a cut at an origin it
called stale to test a refusal the design decided not to gate: its only
outcome was an emission at an unknown position. Removed from the script
and the bench page.
live_fire_drills.py discarded every drill's return value, so the bench
page recorded a failed live-fire drill as OK. The exit status is the
drill's.
laser_pgood is the supply's power-good, high on every healthy machine;
fire_test.py and the K3 drill aborted on it and pgood_probe.py inverted
it. The latch-unlock drills now gate on the safety chain holding HV off
(charge-pump watchdog dead, pulse engine idle), as the kernel suite
does, and the probe reports the pin as the kernel publishes it.
motion.deadman: the controller resumed from its hang recovers on $X and
moves again without a restart (the stream's fault acknowledgment).
The archive was written straight to its final name and a rerun accepted
any non-empty file as complete, so a run interrupted mid-archive left a
truncated image that the next run kept and then overwrote the slot. The
archive is now written as .part and renamed on success; an existing
archive counts only with its manifest line present and a whole gzip
stream, otherwise it is archived again.
With the gate skipped the script still staged and attached whatever
acceptance.json existed, indistinguishable from a gated release, and
sha256sums.txt never listed the artifact. A skipped gate now stages
NO-ACCEPTANCE.txt instead and publishes as a prerelease; every attached
file is listed in the sums.
The inheritance walk skipped every record that was not a PASS on the
current fingerprint, so a FAIL or ERROR recorded after a PASS on the
same image was stepped over and the older PASS inherited into the next
campaign. The newest record on the fingerprint now decides: a PASS is
inherited, a FAIL or ERROR blocks it (reason failed-since), an ABORTED
run says nothing. Unit tests for all three orders.
ffboot, the tool that rewrites the boot environment on every install and
slot switch, was packaged from scripts/ outside every fingerprint. It
now lives in the recipe's files and the recipe inherits the manifest
class; the tree manifest tool fingerprints file components the same
way, and the update tests cover the component.
forgectrl.settings-bounds fell back to ui_units=mm, which the whitelist
refuses, so the always-required test failed on a fresh machine; the
fallback is metric.
Rule 22: M3 S1000 with the window open, then jogs from Idle (a sender's
Fire button plus its Move panel) ship every jog tick dark, and the cut
after them is lit. Rule 23: two cuts at S300 and S1000 queued together
render the S300 cruise at the S300 density; the rolloff shapes against
the segment's own velocity ratio, never the newest S over the block's.
Both fail on the driver before its fix.
The start gate set just above the coolant opens the armed session under
the warm-up; the cloud client waits it out after the button, nothing
runs, and the release starts the print, which completes. The mid-run
hold and its bound are host-tested: the gates apply at session open, so
no setting can produce a hold mid-run on the bench. Bench-excerpt unit
tests: the pass, and the failure when a run starts under the hold.
A pulse device closed while locked and running is the kernel dead man's
switch. The drill enables the lens driver at low current, runs a padded
stream under the takeover, drops the device with the lock held, and
checks that the engine halts at once, the latch is locked, and the head
is in its safe state (z_enable 1, z_current 1, measure laser and UV LED
off). Always required; the kernel covers map already names src/head.c.
A job longer than the ring keeps its feeder alive through the button
wait, so the cancel there has to stop the feeder before the park clears
the ring. The drill now loads such a job, requires the "longer than the
ring" line, and after the cancel reads the program total twice over the
feeder's retry period (zero both times) and cnc/streaming (zero). The
bench-excerpt unit test carries the long-job line and the new evidence.
Covers map unchanged: the drill already names gfhardware/machine.py.
The recipe URLs, the release and install URLs, the vendor check and the CI checkouts name openglow-org, and the grblHAL core fork is openglow-org/grblHAL-core. No catalog change is owed: the recipe edits move the meta-forgefirm content hash, which every test fingerprint folds in through the platform block, so the whole catalog re-runs on its own.
ScottW514