The coolant critical tier, with its catalog test

forgectrl pin a1875a8: cool_temp_critical_c (default 38 C) is the fail
tier above the coolant ceiling's pause: at or over it in a run session the
verdict is CRITICAL (fire blocked, hold, no resume this job), the fault
ends with the session, the settings API keeps it above the ceiling, and
its top turns the gate off.

Catalog: cooling.critical-tier (auto, 45 tests), driven through the
settings API like cooling.gate-off: a critical line at the ceiling is
refused (and undone should it ever be accepted); with the ceiling, the
resume gate and the critical line all under the coolant's temperature a
session reads CRITICAL rather than OVERTEMP with no resume and the reason
naming the tier and the coolant; after the session the ceiling alone
holds (OVERTEMP); with the critical line at its top the gate is off
(gates_off, the run-start log line) and the ceiling alone pauses;
restored, OK with nothing off. Four unit cases against a scripted engine.
_after_session takes the condition to wait for.

Docs: COOLING (verdict table, section 5, the settings table, the quick
reference), BRINGUP item 19 and the catalog count, the CAMPAIGN-LOG
paragraph for the fan-fault session rule on dev image 20260822145201.

docs/BRINGUP.md

@@ -579,7 +579,7 @@ under the domain model from the day's earlier dev images) and the export reads "
 YES" for that image's manifest. That authorizes a release; it is not one
 until `releases/v<version>/acceptance.json` is committed.
 
-- **Catalog: 44 tests** in `forgetest/forgetest/suite/`, every one a port of a
+- **Catalog: 45 tests** in `forgetest/forgetest/suite/`, every one a port of a
   proven bench drill or a bench-verified check — the always-required core
   (`image.health`, `kernel.latch-locked-idle`, `kernel.k1-k2`,
   `kernel.fire-line`), `forgectrl.*`, `logs.*`, `update.*`, `motion.*`
@@ -1369,9 +1369,15 @@ Open items only. Anything closed is in `CAMPAIGN-LOG.md`.
     ends with its run session (decided; a standing hold at idle canceled
     GRBL jogs and would have refused the cloud print that re-proves the
     fan), since every session judges every fan afresh after the grace;
-    `cooling.fan-gate-trips` checks it, bench run owed on the next image.
+    `cooling.fan-gate-trips` checks it, PASS on dev image `20260822145201`.
-    The coolant critical tier and the watch-only board temperatures
+    **The coolant critical tier is in:** `cool_temp_critical_c` (default
-    follow.
+    38 C, 6 to 70, off at 70, kept above the ceiling by the settings
+    cross-check) is the fail tier above the ceiling's pause: at or over it
+    in a run session the verdict is `CRITICAL` (fire blocked, hold, no
+    resume this job), the fault ends with the session, no header touches
+    it; `cooling.critical-tier` in the catalog (45), bench run owed on the
+    next image, plus the heater-driven warm-loop drill to the critical
+    line. The watch-only board temperatures follow.
 
     Nothing here can put energy where it was not commanded: the hardware chain
     is the emission boundary and no header field touches it, and forgectrl runs

docs/CAMPAIGN-LOG.md

@@ -3363,7 +3363,10 @@ session could re-prove the fan (a hunt clears it). Decision (operator,
 the same day): the fault ends with its session, since every session
 judges every fan afresh after the grace before anything can fire;
 `cooling.fan-gate-trips` now checks the verdict is `OK` with no hold once
-the tripped session is over. The fire alarm keeps its idle hold.
+the tripped session is over. The fire alarm keeps its idle hold. Built
+and flashed the same day (dev image `20260822145201`, forgectrl d51dbdb):
+`cooling.fan-gate-trips` PASS in 60 s, the engine reading `OK`, no hold,
+fire allowed in the smoke-clear phase right after the tripped session.
 
 ## Superseded status notes
 

docs/COOLING.md

@@ -261,6 +261,8 @@ A clean check from the fault state logs a recovery.
 | `SUSPECT` | Hold requested, cut airflow held; auto-resumes on a clean re-check. |
 | `FAULT` | Fire gated and the hold stands — for the operator to resolve. |
 | `OVERTEMP` | Hold with forced cooling airflow; auto-resumes below the resume gate (§5). |
+| `CRITICAL` | The coolant at or over the critical line in a run session: fire blocked, hold, no resume this job (§5). |
+| `AIRFLOW` | A fan under its floor: fire blocked, hold, no resume this job (§3a). |
 | `FIRE` | Motion stopped, latch locked, hold until the next run session (§7). |
 
 Practical note: **expect a legitimate suspicion on the first checks after
@@ -277,6 +279,16 @@ The engine uses the factory's coolant windows:
   request and cooling airflow forced on.
 - **Resume gate 31 °C** — below this, recovery is signaled and the controller
   resumes automatically.
+- **Critical line 38 °C** (`cool_temp_critical_c`, §8) — a second tier above
+  the ceiling, and a different kind: at or over it during a run session the
+  verdict goes `CRITICAL`, fire is blocked, the job holds, and there is no
+  resume for the rest of that session, because a loop that ran through the
+  pause tier and kept climbing is not a condition to cut through. The fault
+  ends with the session; the ceiling's pause keeps holding while the loop is
+  hot, and the next session judges the line afresh. A cloud job's header
+  carries no critical line for the coolant, so this one is always the local
+  setting; the settings API keeps it above the ceiling, and at its top
+  (70 °C) it is the gate turned off (§8a).
 
 The **upstream** sensor gates, because it reads the coolant actually entering
 the tube.
@@ -368,6 +380,7 @@ start of every run, so a change takes effect on your next job.
 | `cool_confirm_max_s` | 480 s | 60 to 3600 s | | How long a suspicion may stay unresolved before it escalates to a fault. |
 | `cool_temp_max` | 33 °C | 5 to 60 °C | 25 to 38 °C | Run ceiling: above it, hold. `60` turns the gate off (§8a). |
 | `cool_temp_resume` | 31 °C | 5 to 59 °C | 20 to 36 °C | Resume gate: below it, continue. Always kept below the ceiling. |
+| `cool_temp_critical_c` | 38 °C | 6 to 70 °C | 36 to 45 °C | Critical line: a fault with no resume in the job (§5). Always kept above the ceiling; `70` turns the gate off. |
 | `cool_cooldown_s` | 15 s | 0 to 1800 s | | Smoke-clear phase at run duty after a job. |
 | `cool_cooldown_max_s` | 300 s | 0 to 1800 s | | Cap on the thermal cooldown phase. |
 | `cool_tach_exhaust_min_rpm` | 6400 rpm | 0 to 20000 | 5800 to 7000 | Exhaust fan floor at run duty (§3a). `0` turns the gate off. |
@@ -440,6 +453,7 @@ Stated plainly so nobody counts on them:
 | Suspicion unresolved past the budget | Escalates to `FAULT`. |
 | Three cleared suspicions in one job | Aggregated "check your coolant" warning. |
 | Upstream coolant above 33 °C | `OVERTEMP`: hold + forced cooling; auto-resume under 31 °C. |
+| Upstream coolant at or over 38 °C during a job | `CRITICAL`: fire blocked, hold, no resume this job; the ceiling's hold stands until the loop is under 31 °C. |
 | A fan under its floor inside the spin-up grace | Nothing yet: the gate reads `grace`. |
 | A fan under its floor for three seconds after the grace | `AIRFLOW`: fire blocked, hold, no resume this job; fans held at run duty; the next job starts the gates fresh. |
 | Purge-air current absent at run duty | `AIRFLOW`, the same way. |

forgetest/forgetest/suite/cooling.py

@@ -239,14 +239,17 @@ def _run_session(ctx, g, fc, until, what, wait=None):
         _session_ended(ctx, fc, what)
 
 
-def _after_session(ctx, fc, wait=5):
+def _after_session(ctx, fc, wait=5, until=None):
-    """The engine's state a few ticks after a session ended."""
+    """The engine's state a few ticks after a session ended: the first
+    sample out of phase run that satisfies `until` (default: verdict OK),
+    or the last sample taken."""
+    until = until or (lambda c: c.get("verdict") == "OK")
     c = {}
     t0 = time.time()
     while time.time() - t0 < wait:
         ctx.sleep(1)
         c = _cool(fc)
-        if c.get("phase") != "run" and c.get("verdict") == "OK":
+        if c.get("phase") != "run" and until(c):
             break
     return c
 
@@ -353,6 +356,125 @@ def gate_off(ctx):
     ctx.check(fc.wait_idle(60, abort=ctx.aborted), "machine did not return to idle")
 
 
+CRIT_KEYS = ("cool_temp_max", "cool_temp_resume", "cool_temp_critical_c")
+
+
+@test("cooling.critical-tier", title="The coolant critical line is a fault above the ceiling's pause",
+      subsystem="cooling", kind="auto", mode="grbl", est_min=3,
+      covers=_COOL_COVERS + [("forgectrl", "src/main.c")], requires=["cooling.gate-off"],
+      steps=["Machine idle, coolant at room temperature (above 8 C). The test writes the coolant "
+             "ceiling, the resume gate and the critical line and restores them; three short M8/M9 "
+             "cycles spin the fans."],
+      description="Two tiers on the upstream coolant sensor: the ceiling pauses (OVERTEMP, resume "
+                  "below the resume gate), the critical line above it is a fault. With the ceiling "
+                  "and the critical line both under the coolant's temperature a run session must "
+                  "read CRITICAL rather than OVERTEMP (fire blocked, hold, no resume), the fault "
+                  "must end with the session, and the settings API must refuse a critical line at "
+                  "or below the ceiling; with the critical line at its top the gate is off by "
+                  "value (gates_off names it) and the ceiling alone pauses; restored, the next "
+                  "session runs OK with nothing off.")
+def critical_tier(ctx):
+    fc = ctx.forgectrl
+    ev = ctx.evidence
+    before = fc.settings()
+    orig = {k: before.get(k, "") for k in CRIT_KEYS}
+    ev["orig"] = orig
+    ctx.log("original: %s", orig)
+    up = (fc.status().get("coolant") or {}).get("up_c")
+    ctx.check(up is not None and up > 8.0, "coolant too cold for the trip leg (up_c %s)", up)
+    c0 = _cool(fc)
+    ctx.check(c0.get("verdict") == "OK" and c0.get("gates_off") == [],
+              "engine is not at OK with every gate on before the test: %s", c0)
+    gates = before.get("gates") or {}
+    ceil = gates.get("cool_temp_max") or {}
+    crit = gates.get("cool_temp_critical_c") or {}
+    ctx.check(crit.get("gate") == "coolant_critical" and crit.get("off") == "high",
+              "settings reply does not describe the critical line as the coolant_critical gate, off at "
+              "its top: %s", crit)
+    bottom, top = ceil.get("lo"), crit.get("hi")
+    ctx.check(isinstance(bottom, (int, float)) and isinstance(top, (int, float)),
+              "no ranges in the reply: %s %s", ceil, crit)
+    ctx.check(isinstance(crit.get("lo"), (int, float)) and crit.get("lo") > bottom,
+              "the critical line's floor %s is not above the ceiling's %s", crit.get("lo"), bottom)
+
+    # The cross-check: a critical line at or below the ceiling is refused
+    # before anything is written.
+    st, body = fc.post("/settings", params={"cool_temp_max": "33", "cool_temp_critical_c": "33"})
+    ev["cross_check"] = {"status": st, "body": body}
+    if st == 200:
+        fc.post("/settings", params=orig)       # undo before failing
+    ctx.check(st == 400, "a critical line equal to the ceiling was accepted: %s %s", st, body)
+    after = fc.settings()
+    ctx.check(all(after.get(k, "") == orig[k] for k in CRIT_KEYS),
+              "the refused POST changed a setting: %s", {k: after.get(k) for k in CRIT_KEYS})
+
+    restored = False
+    with ctx.grbl() as grbl:
+        st = grbl.status_report()
+        ctx.check(st["state"].startswith("Idle"), "controller is %s", st["state"])
+        try:
+            # Leg 1: ceiling and critical line both under the coolant's
+            # temperature. The fail tier wins: CRITICAL, not OVERTEMP.
+            low = {"cool_temp_max": str(bottom + 1), "cool_temp_resume": str(bottom),
+                   "cool_temp_critical_c": str(bottom + 2)}
+            _set_gates(ctx, fc, low)
+            c = _run_session(ctx, grbl, fc, lambda c: c.get("verdict") == "CRITICAL", "critical leg")
+            ev["critical"] = c
+            ctx.check(c.get("verdict") == "CRITICAL",
+                      "critical line %s C with coolant at %.1f C did not fault (verdict %s): %s",
+                      bottom + 2, up, c.get("verdict"), c)
+            ctx.check(c.get("fire_ok") is False and c.get("hold") is True,
+                      "CRITICAL without fire blocked and a hold: %s", c)
+            ctx.check(c.get("resume_ok") is not True, "a coolant fault offered a resume: %s", c)
+            ctx.check("CRITICAL" in (c.get("reason") or "") and "coolant" in (c.get("reason") or ""),
+                      "the reason does not name the tier and the coolant: %r", c.get("reason"))
+            ctx.check(c.get("gates_off") == [], "a tripped gate is not an off gate: %s", c.get("gates_off"))
+            # The fault ends with the session; the ceiling, still under the
+            # coolant, keeps its pause (OVERTEMP), never CRITICAL.
+            c = _after_session(ctx, fc, until=lambda c: c.get("verdict") != "CRITICAL")
+            ev["critical_after"] = c
+            ctx.check(c.get("verdict") == "OVERTEMP",
+                      "after the faulted session the engine reads %s, expected the ceiling's OVERTEMP: %s",
+                      c.get("verdict"), c)
+
+            # Leg 2: the critical line at its top is the gate off; the
+            # ceiling alone pauses, and gates_off says so.
+            _set_gates(ctx, fc, {"cool_temp_critical_c": str(top)})
+            c = _run_session(ctx, grbl, fc,
+                             lambda c: c.get("verdict") == "OVERTEMP" and "coolant_critical" in (c.get("gates_off") or []),
+                             "critical off leg")
+            ev["critical_off"] = c
+            ctx.check(c.get("verdict") == "OVERTEMP",
+                      "with the critical line off the ceiling did not pause (verdict %s): %s", c.get("verdict"), c)
+            ctx.check("coolant_critical" in (c.get("gates_off") or []),
+                      "gates_off %s lacks coolant_critical", c.get("gates_off"))
+            ctx.check(_tail_has(fc, "gate coolant_critical OFF: cool_temp_critical_c = %g" % top),
+                      "the run-start log line for the off gate is missing from the forgectrl log")
+
+            # Restore, and prove it: OK, nothing off.
+            _set_gates(ctx, fc, orig)
+            restored = True
+            c = _run_session(ctx, grbl, fc, lambda c: c.get("verdict") == "OK" and not c.get("gates_off"),
+                             "restored")
+            ev["restored"] = c
+            ctx.check(c.get("verdict") == "OK" and c.get("gates_off") == [],
+                      "engine did not return to OK with every gate on after the restore: %s", c)
+        finally:
+            if not restored:
+                st, body = fc.post("/settings", params=orig)
+                ctx.log("restore on failure: POST /settings %s -> %s", orig, st)
+                try:
+                    c = _run_session(ctx, grbl, fc,
+                                     lambda c: c.get("verdict") == "OK" and not c.get("gates_off"),
+                                     "restore on failure")
+                    ctx.log("restore on failure: engine %s gates_off %s", c.get("verdict"), c.get("gates_off"))
+                except Exception as e:      # the original failure is the one to report
+                    ctx.log("restore on failure: run session did not complete (%s)", e)
+    after = fc.settings()
+    ctx.check(all(after.get(k, "") == orig[k] for k in CRIT_KEYS),
+              "settings not restored: %s", {k: after.get(k) for k in CRIT_KEYS})
+
+
 FAN_KEYS = ("cool_tach_exhaust_min_rpm", "cool_purge_min_current", "cool_fan_grace_s")
 FAN_GRACE_S = "8"           # past the intakes' 7 s to 90 percent, so only the leg's floor trips
 FAN_TRIP_WAIT_S = 20        # grace + three ticks, with slack for the 1 Hz pipeline

forgetest/tests/test_cooling_suite.py

@@ -376,6 +376,156 @@ class GateOffTests(unittest.TestCase):
         self.assertEqual(self.fc.state["settings"]["cool_temp_max"], "30")
 
 
+class CriticalTierTests(unittest.TestCase):
+    """cooling.critical-tier against a scripted engine: two tiers on the
+    coolant, the fail tier winning over the pause tier in a run session
+    and ending with it, the critical line off at its top, and the settings
+    cross-check refusing a critical line at or below the ceiling."""
+
+    UP = 22.3
+    ROWS = {  # key: (gate, default, lo, hi, band_lo, band_hi, off)
+        "cool_temp_max": ("coolant_max", 33.0, 5.0, 60.0, 25.0, 38.0, "high"),
+        "cool_temp_resume": (None, 31.0, 5.0, 59.0, 20.0, 36.0, "none"),
+        "cool_temp_critical_c": ("coolant_critical", 38.0, 6.0, 70.0, 36.0, 45.0, "high"),
+    }
+
+    def setUp(self):
+        self.fc = helpers.FakeForgectrl().start()
+        self.grbl = FakeGrbl()
+        self.saved = (cooling.VERDICT_WAIT_S, cooling.SESSION_END_WAIT_S)
+        cooling.VERDICT_WAIT_S = 3
+        cooling.SESSION_END_WAIT_S = 3
+        self.fc.state["status"] = dict(self.fc.state["status"],
+                                       coolant={"down_c": 22.4, "up_c": self.UP, "pump": True, "tec": False},
+                                       gates_off=[])
+        self.fc.state["cool"] = {"phase": "idle", "verdict": "OK", "fire_ok": False, "hold": False,
+                                 "gates_off": []}
+        for k in self.ROWS:
+            self.fc.state["settings"].setdefault(k, "")
+        self.fc.state["logs_tail"] = {"text": ""}
+        self.faults = True          # the engine has the critical tier
+        self.ends_with_session = True
+        self.cross_checks = True    # the settings API refuses crit <= max
+        self._describe()
+        self.grbl.on_command = self._engine
+        self.fc.on_post = self._on_post
+
+    def tearDown(self):
+        cooling.VERDICT_WAIT_S, cooling.SESSION_END_WAIT_S = self.saved
+        self.grbl.close()
+        self.fc.stop()
+
+    def setting(self, key):
+        v = self.fc.state["settings"].get(key) or ""
+        return float(v) if v else self.ROWS[key][1]
+
+    def _describe(self):
+        gates = {}
+        for k, (gate, d, lo, hi, blo, bhi, off) in self.ROWS.items():
+            v = self.setting(k)
+            state = "off" if (off == "high" and v >= hi) else ("ok" if blo <= v <= bhi else "warn")
+            gates[k] = {"gate": gate, "def": d, "lo": lo, "hi": hi, "band": [blo, bhi], "off": off,
+                        "value": v, "state": state}
+        self.fc.state["settings"]["gates"] = gates
+
+    def _on_post(self, path, form):
+        if path != "/settings":
+            return None
+        if self.cross_checks:
+            merged = dict(self.fc.state["settings"])
+            merged.update(form)
+            tmax = float(merged.get("cool_temp_max") or 33.0)
+            tcrit = float(merged.get("cool_temp_critical_c") or 38.0)
+            if tcrit <= tmax:
+                return (400, {"error": "cool_temp_critical_c must be above cool_temp_max"})
+        self.fc.state["settings"].update(form)
+        self._describe()
+        return (200, self.fc.state["settings"])
+
+    def _verdict_idle(self):
+        """Outside a run only the ceiling's pause tier stands."""
+        return "OVERTEMP" if self.UP > self.setting("cool_temp_max") else "OK"
+
+    def _engine(self, line):
+        self._describe()
+        cool = self.fc.state["cool"]
+        if line == "M9":
+            def end():
+                time.sleep(0.3)
+                cool["phase"] = "idle"
+                if self.ends_with_session or cool["verdict"] != "CRITICAL":
+                    v = self._verdict_idle()
+                    cool.update(verdict=v, fire_ok=v == "OK", hold=v != "OK", reason="")
+            threading.Thread(target=end, daemon=True).start()
+            return
+        if line != "M8":
+            return
+        cool["phase"] = "run"
+        tmax, tcrit = self.setting("cool_temp_max"), self.setting("cool_temp_critical_c")
+        crit_off = tcrit >= self.ROWS["cool_temp_critical_c"][3]
+        off = ["coolant_critical"] if crit_off else []
+        if crit_off:
+            self.fc.state["logs_tail"]["text"] += (
+                "Aug 22 12:00:00 forgectrl: cool: gate coolant_critical OFF: cool_temp_critical_c = 70 "
+                "(the high end of 6 to 70; recommended 36 to 45, default 38)\n")
+        if self.faults and not crit_off and self.UP >= tcrit:
+            cool.update(verdict="CRITICAL", fire_ok=False, hold=True, resume_ok=False,
+                        reason="CRITICAL: coolant %.1f C at or over the %.0f C critical line - hold, "
+                               "no resume this job" % (self.UP, tcrit), gates_off=off)
+        elif self.UP > tmax:
+            cool.update(verdict="OVERTEMP", fire_ok=False, hold=True, resume_ok=False,
+                        reason="coolant over the ceiling", gates_off=off)
+        else:
+            cool.update(verdict="OK", fire_ok=True, hold=False, resume_ok=True, reason="", gates_off=off)
+        self.fc.state["status"]["gates_off"] = off
+
+    def run_test(self):
+        run = Run("test", "cooling.critical-tier", "t")
+        ctx = Context(run, None, helpers.make_test("cooling.critical-tier", []))
+        cooling.critical_tier(ctx)
+        return run
+
+    def settings_posts(self):
+        return [f for p, f in self.fc.posts if p == "/settings"]
+
+    # -- cases ------------------------------------------------------------------
+    def test_fault_then_off_then_restored_passes(self):
+        run = self.run_test()
+        self.assertEqual(run.evidence["cross_check"]["status"], 400)
+        self.assertEqual(run.evidence["critical"]["verdict"], "CRITICAL")
+        self.assertEqual(run.evidence["critical_after"]["verdict"], "OVERTEMP")
+        self.assertEqual(run.evidence["critical_off"]["gates_off"], ["coolant_critical"])
+        self.assertEqual(run.evidence["restored"]["verdict"], "OK")
+        posts = self.settings_posts()
+        self.assertEqual(posts[1], {"cool_temp_max": "6.0", "cool_temp_resume": "5.0",
+                                    "cool_temp_critical_c": "7.0"})
+        self.assertEqual(posts[-1], {"cool_temp_max": "", "cool_temp_resume": "", "cool_temp_critical_c": ""})
+        self.assertEqual(self.grbl.commands.count("M8"), 3)
+        self.assertEqual(self.grbl.commands.count("M9"), 3)
+
+    def test_an_engine_without_the_tier_fails_and_restores(self):
+        self.faults = False
+        with self.assertRaises(Failed) as cm:
+            self.run_test()
+        self.assertIn("did not fault", str(cm.exception))
+        self.assertEqual(self.settings_posts()[-1],
+                         {"cool_temp_max": "", "cool_temp_resume": "", "cool_temp_critical_c": ""})
+        self.assertEqual(self.fc.state["cool"]["verdict"], "OK")
+
+    def test_a_fault_that_outlives_its_session_fails(self):
+        self.ends_with_session = False
+        with self.assertRaises(Failed) as cm:
+            self.run_test()
+        self.assertIn("after the faulted session", str(cm.exception))
+
+    def test_a_missing_cross_check_fails_before_any_session(self):
+        self.cross_checks = False
+        with self.assertRaises(Failed) as cm:
+            self.run_test()
+        self.assertIn("was accepted", str(cm.exception))
+        self.assertEqual(self.grbl.commands.count("M8"), 0)
+
+
 class FanGateTests(unittest.TestCase):
     """cooling.fan-gate-trips against a scripted engine: the fake reads the
     floors at every M8 (the engine reloads at run start), holds every fan

meta-forgefirm/recipes-forgefirm/forgectrl/forgectrl-pin.inc

@@ -2,5 +2,5 @@
 # only SRCREV and PV here - the image manifest leaves *-pin.inc out of the
 # layer content hash because the component entry already identifies the
 # pinned source (forgefirm-image-manifest.bbclass).
-SRCREV = "d51dbdbcc6b3a12d2f7963eb94514ce52493645f"
+SRCREV = "a1875a8113b45c42e252caa3e4b409c1ba4b39fe"
 PV = "0.1.0"