The fan floors proven on a pinned image; a fan fault ends with its session

The dated record of dev image 20260822135848: the campaign of every
non-operator, non-live test at 18 of 18 PASS with the measured floors and
the operating-point rule (cooling.fan-gate-trips and the hunt leg of
cloud.mode-switch as recorded), and the unplugged-exhaust-fan drill:
AIRFLOW at the grace plus three ticks with the exhaust dead, the other
fans held, the reason relayed on the Grbl port, the replugged fan ok
inside the next session's grace.

The drill showed the fault riding into idle, where the hold canceled
jogs and would have refused the cloud print that re-proves the fan.
forgectrl pin d51dbdb: the fault ends with its run session, and the next
session judges every fan afresh. cooling.fan-gate-trips checks the
verdict is OK with no hold once the tripped session is over (the unit
fake mirrors it); its covers, and the cooling tests' shared covers, gain
src/coolfmt.* (the tree manifest carries the new files at the bumped pin,
and the lint was right to ask). cloud.mode-switch samples the hunt's gate
rows twice a second: a hunt's run phase is a few seconds long.

Docs: COOLING 3a, BRINGUP item 19, CAMPAIGN-LOG.
This commit is contained in:
ScottW514
2026-08-22 10:46:41 -04:00
parent adcd1adb9a
commit c1591e4f69
7 changed files with 107 additions and 15 deletions
+14 -6
View File
@@ -1358,12 +1358,20 @@ Open items only. Anything closed is in `CAMPAIGN-LOG.md`.
and unarmed whenever it is commanded at the run duty; a cloud hunt, and unarmed whenever it is commanded at the run duty; a cloud hunt,
sent with the extraction fans off, is measured and not judged. sent with the extraction fans off, is measured and not judged.
`cooling.fan-gate-trips` and the hunt leg of `cloud.mode-switch` in `cooling.fan-gate-trips` and the hunt leg of `cloud.mode-switch` in
the catalog. The shipped floors are 55 percent of the bench machine's the catalog, both PASS on the pinned dev image `20260822135848`
measured steady speeds (facts bank: exhaust 6400, intake 2290, air (campaign 18 of 18). The shipped floors are 55 percent of the bench
assist 6000 rpm, purge current 300, grace 15 s); the bench run of the machine's measured steady speeds (facts bank: exhaust 6400, intake
gate tests on an image carrying them, and the unplugged-tach drill, 2290, air assist 6000 rpm, purge current 300, grace 15 s). The
are still owed. The coolant critical tier and the watch-only board unplugged-exhaust-fan drill PASS on the same image: `AIRFLOW` at
temperatures follow. grace plus three ticks with the exhaust dead, the other fans held,
the reason relayed on the Grbl port, and the replugged fan `ok` inside
the next session's grace (CAMPAIGN-LOG). From the drill: the fault
ends with its run session (decided; a standing hold at idle canceled
GRBL jogs and would have refused the cloud print that re-proves the
fan), since every session judges every fan afresh after the grace;
`cooling.fan-gate-trips` checks it, bench run owed on the next image.
The coolant critical tier and the watch-only board temperatures
follow.
Nothing here can put energy where it was not commanded: the hardware chain Nothing here can put energy where it was not commanded: the hardware chain
is the emission boundary and no header field touches it, and forgectrl runs is the emission boundary and no header field touches it, and forgectrl runs
+57
View File
@@ -3308,6 +3308,63 @@ purge in the purge leg, the exhaust read `off` with floor 0 in the off leg,
and the restore showed every fan `ok` at the shipped floors (exhaust 11723, and the restore showed every fan `ok` at the shipped floors (exhaust 11723,
intakes 4157 and 4162, air assist 11078 rpm, purge 628 counts). intakes 4157 and 4162, air assist 11078 rpm, purge 628 counts).
## 2026-08-22: the measured floors and the operating-point rule on a pinned image
Dev image `20260822135848` (forgectrl 47e4256 pinned by forgefirm adcd1ad;
release `20260822135751` built alongside), flashed after a fetch-verified
both-image build. Campaign `c-20260822140659-2f25`, every auto test the
pin bump invalidated plus the rest of the non-operator, non-live catalog:
**18 of 18 PASS** (cooling.flow-verify, image.health, kernel.latch-locked-idle,
kernel.k1-k2, kernel.backtrack-bounds, forgectrl.auth,
forgectrl.settings-bounds, forgectrl.panel-serves, logs.tree-tail-export,
logs.level-settings, motion.deadman, cooling.fans-quiet-after-motion,
cooling.gate-off, cooling.fan-gate-trips, camera.sensor-profile,
camera.frame-health, cloud.mode-switch, kernel.fire-line).
The two that carry this change, as recorded: `cooling.fan-gate-trips` in
58 s with only the exhaust `TRIPPED` in its leg, only the purge in its,
the exhaust `off` at floor 0 in the same tick `gates_off` named it, and
the restore reading exhaust 11726, intakes 4160 and 4193, air assist
11095 rpm, purge 627 counts, every gate `ok` at the shipped floors;
`cloud.mode-switch` in 29 s with the connect-time hunt's run tick reading
the exhaust at 0 rpm, `unjudged`, the air assist `unjudged`, verdict `OK`
throughout, and the hunt finishing `:completed`. The hunt's run phase is a
few seconds long and gave one sample at a 1 s poll, so the watcher now
samples twice a second.
## 2026-08-22: the unplugged-exhaust-fan drill
The gate on the real failure path, not a settings override: the operator
unplugged the exhaust fan's whole connector at the Interconnect PCB (fan
dead, tach silent), the machine idle in GRBL mode, lid closed, nothing
armed. One `M8` session from the board, `/cool/status` read once a second
(dev image `20260822135848`):
- 1 to 14 s: every gate `grace` (the shipped 15 s), exhaust reading 0.
- 15 and 16 s: exhaust `under` at 0 rpm; intakes, air assist and purge
`ok`, up to speed inside the grace.
- 17 s: verdict **`AIRFLOW`**, `fire_ok false`, `hold true`, exhaust
`TRIPPED`, reason `AIRFLOW: exhaust 0 under the 6400 floor for 3 s -
hold, no resume this job`; the other four fans held at run duty around
the dead one. grblHAL relayed the reason on the Grbl port as a
`[MSG:Warning: ...]`, and forgectrl logged the `WARNING` line.
- `M9` ended the session into the smoke-clear phase with the fault still
named; the operator replugged the connector.
- The next `M8` session: exhaust 4112 rpm at 1 s, 6623 at 2 s (past the
floor), 11640 at 7 s (the measured time to 90 percent), `ok` with every
gate at the end of the grace, verdict `OK`, clean end.
One observation for a decision: between the two sessions the engine sat
at idle with the verdict still `AIRFLOW`, `hold=true`, `fire_ok=false`.
The fan fault latches for the run session and clears at the next session
start, the same shape as the fire alarm; at idle the hold cancels GRBL
jogs and the cloud client's print pre-check refuses a print before a
session could re-prove the fan (a hunt clears it). Decision (operator,
the same day): the fault ends with its session, since every session
judges every fan afresh after the grace before anything can fire;
`cooling.fan-gate-trips` now checks the verdict is `OK` with no hold once
the tripped session is over. The fire alarm keeps its idle hold.
## Superseded status notes ## Superseded status notes
### Shared machine services — remaining polish, as listed 2026-08-13 ### Shared machine services — remaining polish, as listed 2026-08-13
+4 -1
View File
@@ -170,7 +170,10 @@ floor.
session: a fan that has stopped moving air is not a condition to cut session: a fan that has stopped moving air is not a condition to cut
through. The fans stay at run duty (a stalled extraction fan needs every through. The fans stay at run duty (a stalled extraction fan needs every
other fan around it running), and the reason names the fan, the reading other fan around it running), and the reason names the fan, the reading
and the floor. The next job starts the gates fresh. and the floor. The fault ends with the session: at idle the verdict is
`OK` again (a standing hold would cancel jogs and refuse the next job
before it could re-prove the fan), and the next session judges every fan
afresh after the grace.
- **A floor of zero is that gate off** (§8a). It still measures: the first - **A floor of zero is that gate off** (§8a). It still measures: the first
reading in a job that would have tripped the shipped default is logged. reading in a job that would have tripped the shipped default is logged.
+3 -3
View File
@@ -372,8 +372,8 @@ def wait_action_finished(ctx, offset, action, timeout, poll=0.5):
def watch_hunt_gates(ctx, offset, timeout): def watch_hunt_gates(ctx, offset, timeout):
"""Wait for the action's terminal line like wait_action_finished, """Wait for the action's terminal line like wait_action_finished,
sampling /cool/status once a second meanwhile. Returns (line or sampling /cool/status twice a second meanwhile (a hunt's run phase
None, samples).""" can be a few seconds). Returns (line or None, samples)."""
fc = ctx.forgectrl fc = ctx.forgectrl
samples = [] samples = []
t0 = time.time() t0 = time.time()
@@ -386,7 +386,7 @@ def watch_hunt_gates(ctx, offset, timeout):
i = action_finish_index(lines, "hunt") i = action_finish_index(lines, "hunt")
if i is not None: if i is not None:
return lines[i], samples return lines[i], samples
time.sleep(1) time.sleep(0.5)
return None, samples return None, samples
+24 -4
View File
@@ -10,7 +10,7 @@ import time
from ..catalog import test from ..catalog import test
from .. import hw from .. import hw
_COOL_COVERS = [("forgectrl", "src/cool.*"), ("forgectrl", "src/diag.*"), _COOL_COVERS = [("forgectrl", "src/cool.*"), ("forgectrl", "src/coolfmt.*"), ("forgectrl", "src/diag.*"),
("forgectrl", "src/gates.*"), ("forgectrl", "src/airflow.*"), ("forgectrl", "src/gates.*"), ("forgectrl", "src/airflow.*"),
("forgectrl", "src/settings.*"), ("forgectrl", "src/settings.*"),
("forgectrl", "src/status.*"), ("forgectrl", "src/ui/**"), ("forgectrl", "src/status.*"), ("forgectrl", "src/ui/**"),
@@ -239,6 +239,18 @@ def _run_session(ctx, g, fc, until, what, wait=None):
_session_ended(ctx, fc, what) _session_ended(ctx, fc, what)
def _after_session(ctx, fc, wait=5):
"""The engine's state a few ticks after a session ended."""
c = {}
t0 = time.time()
while time.time() - t0 < wait:
ctx.sleep(1)
c = _cool(fc)
if c.get("phase") != "run" and c.get("verdict") == "OK":
break
return c
def _tail_has(fc, needle): def _tail_has(fc, needle):
st, body = fc.get("/logs/tail", params={"name": "forgectrl", "lines": GATE_LOG_LINES}) st, body = fc.get("/logs/tail", params={"name": "forgectrl", "lines": GATE_LOG_LINES})
text = body.get("text", "") if st == 200 and isinstance(body, dict) else "" text = body.get("text", "") if st == 200 and isinstance(body, dict) else ""
@@ -358,9 +370,10 @@ def _fan_gate(c, name):
description="The airflow gates judge a fan commanded at the cut fan profile, which a bare M8 " description="The airflow gates judge a fan commanded at the cut fan profile, which a bare M8 "
"applies, armed or not. An exhaust floor no fan " "applies, armed or not. An exhaust floor no fan "
"can meet must trip AIRFLOW after the grace plus three ticks (hold, fire blocked, no " "can meet must trip AIRFLOW after the grace plus three ticks (hold, fire blocked, no "
"resume while the session lasts); a purge current floor at the ADC rail must trip the " "resume while the session lasts) and the fault must end with the session (verdict OK, "
"same way; a floor of zero must read off in gates_off and trip nothing; restored, the " "no hold, once the session is over); a purge current floor at the ADC rail must trip "
"next session runs OK with every fan reading at or above its floor.") "the same way; a floor of zero must read off in gates_off and trip nothing; restored, "
"the next session runs OK with every fan reading at or above its floor.")
def fan_gate_trips(ctx): def fan_gate_trips(ctx):
fc = ctx.forgectrl fc = ctx.forgectrl
ev = ctx.evidence ev = ctx.evidence
@@ -397,6 +410,13 @@ def fan_gate_trips(ctx):
ctx.check(_fan_gate(c, "exhaust").get("state") == "TRIPPED", ctx.check(_fan_gate(c, "exhaust").get("state") == "TRIPPED",
"the exhaust gate does not read TRIPPED: %s", c.get("fan_gates")) "the exhaust gate does not read TRIPPED: %s", c.get("fan_gates"))
ctx.check("exhaust" in (c.get("reason") or ""), "the reason does not name the fan: %r", c.get("reason")) ctx.check("exhaust" in (c.get("reason") or ""), "the reason does not name the fan: %r", c.get("reason"))
# The fault is the session's: with the session over, the
# verdict is back to OK and nothing holds (jogs and the next
# job's pre-check must not see a fan fault at idle).
c = _after_session(ctx, fc)
ev["exhaust_trip_after"] = c
ctx.check(c.get("verdict") == "OK" and c.get("hold") is False,
"the fan fault outlived its run session: %s", c)
# Leg 2: the purge fan by current, floor at the ADC rail. # Leg 2: the purge fan by current, floor at the ADC rail.
_set_gates(ctx, fc, {"cool_tach_exhaust_min_rpm": orig["cool_tach_exhaust_min_rpm"], _set_gates(ctx, fc, {"cool_tach_exhaust_min_rpm": orig["cool_tach_exhaust_min_rpm"],
+4
View File
@@ -448,6 +448,10 @@ class FanGateTests(unittest.TestCase):
def end(): def end():
time.sleep(0.3) time.sleep(0.3)
cool["phase"] = "idle" cool["phase"] = "idle"
# the fan fault is the session's: it ends with it
cool.update(verdict="OK", fire_ok=True, hold=False, reason="")
for g in cool.get("fan_gates", {}).values():
g["state"] = "idle"
threading.Thread(target=end, daemon=True).start() threading.Thread(target=end, daemon=True).start()
return return
if line != "M8": if line != "M8":
@@ -2,5 +2,5 @@
# only SRCREV and PV here - the image manifest leaves *-pin.inc out of the # only SRCREV and PV here - the image manifest leaves *-pin.inc out of the
# layer content hash because the component entry already identifies the # layer content hash because the component entry already identifies the
# pinned source (forgefirm-image-manifest.bbclass). # pinned source (forgefirm-image-manifest.bbclass).
SRCREV = "47e42569e8def2ed6fe10f61e9ab9ff4f9c8807d" SRCREV = "d51dbdbcc6b3a12d2f7963eb94514ce52493645f"
PV = "0.1.0" PV = "0.1.0"