The critical tier proven on a rising loop; the board temperatures watched

forgectrl pin 76115fd: the chassis LM75 and the supply sensor ride /status
as temps (degrees and a raw count), the engine ranges them over every run
session into one run-end line, and a critical fault that clears with its
session yields the reason to the standing hold.

Bench: critical_tier_drill.py (a bench tool now, registered as
critical-tier) sets the ceiling, the resume gate and the critical line a
few tenths above the live upstream reading and lets the engine's own
flow-check heater warm the loop through them inside one M8 session;
temp_calibrate.py gains supply-watch, supply-point and supply-fit for the
supply sensor against a thermometer on its heatsink, the fit shown beside
UAPI.md's unverified guess.

Catalog: cooling.gate-off checks the /status temps fields and the run-end
board-temperature line (the unit fake mirrors both, one new failure
case); cooling.critical-tier checks the reason after a faulted session.

Docs: CAMPAIGN-LOG entries for cooling.critical-tier on dev image
20260822154257 and the warm-loop drill (OVERTEMP at 10 s, CRITICAL at
14 s, the fault ending with the session); BRINGUP item 19, the facts bank
(board temperatures at idle), COOLING section 9, the bench README.

scripts/bench/README.md

@@ -40,6 +40,8 @@ page's takeover does that; from a host, stop them first.
 | `flow_characterize.py` | Coolant flow characterization using the factory temperature curve (board or host; forgectrl and controller stopped): baseline → flow → no-flow → recovery, printing the ΔT bands and their separation. Takes the heater duty as an argument (`flow_characterize.py 30`); aborts if downstream passes 45 °C. |
 | `flow_matrix.py` | **The flow-detection design matrix** (board or host; forgectrl and controller stopped; with `flow_sampler.py` from `/usr/share/forgetest/bench/`): duty × duration × flow/no-flow, every run from a common cooled baseline, interleaved repeats. One heating trace yields the metric at every candidate duration, so cost and precision come from the same 60 runs. Prints a cost table, a precision table (mean±sd, worst-case margin, d′) and a ranked shortlist. `flow_matrix.py [duties] [repeats]` (or env `FM_DUTIES`, `FM_REPEATS`, `FM_RESULTS`); results/log in the bench data directory, resumable. |
 | `flow_sustained.py` | Long-run test of the real re-check cadence via M8 (board or host; controller running): counts verdicts/false faults against the configured `cool_flow_rise` and tracks whether the loop accumulates heat. `flow_sustained.py [minutes]`. |
+| `temp_calibrate.py` (`supply-*` modes) | The power supply's sensor (`pic/pwr_temp`, raw) against a thermometer on its heatsink: `supply-watch`, `supply-point <C>`, `supply-fit`; the fit is printed beside `UAPI.md`'s unverified guess. Three points during a long cut settle it. |
+| `critical_tier_drill.py` | The coolant critical tier on a rising temperature (board or host): sets the ceiling, the resume gate and the critical line a few tenths above the live upstream reading and lets the engine's own flow-check heater warm the loop through them inside one `M8` session, expecting `OVERTEMP` at the ceiling and then `CRITICAL` (fire blocked, hold, no resume) with the fault ending at `M9`; restores the settings and cycles a session so the engine re-reads them. Results as JSON in the bench data directory. |
 | `flow_warm_validate.py` | Runs the real check from a heater-warmed baseline (board or host; forgectrl and controller stopped; `flow_warm_validate.py [cycles_per_case]`; results/log in the bench data directory; exit 1 if any run is misclassified). Note the ceiling: 100 % duty pushes the downstream sensor past 50 °C in 30 s while the bulk barely moves, so warm-loop validation above ~23 °C needs the laser, not the heater. |
 | `flow_recheck_char.py` | Characterizes short in-run re-checks and the differential metric (board or host; forgectrl and controller stopped; `flow_recheck_char.py [heater_pct] [window_s]`); shows why over-temp cannot see a stopped pump and why passive warming trends are ambiguous. |
 | `flow_confirm_drill.py` | Coolant flow suspicion/confirmation drill (runs on the board): one continuous M8 session walks the verdict state machine through real pump-off transients — verified → SUSPECT (+ immediate re-check) → cleared → SUSPECT → FAULT (consecutive) → recovered — printing PASS/FAIL per transition. Leaves the machine idle (M9, pump on, heater off). |

scripts/bench/critical_tier_drill.py

@@ -0,0 +1,179 @@
+#!/usr/bin/env python3
+"""Coolant critical-tier drill: the loop warmed through the lines by the
+engine itself, inside one run session.
+
+The loop heater reaches the high twenties at most, so the drill sets the
+coolant ceiling, the resume gate and the critical line a few tenths above
+the live upstream reading and lets the engine's own flow-check heater
+(duty 100 percent, 300 s windows, rechecks every 30 s, the suspect
+threshold parked at its top so the warm-up is not read as a flow fault)
+carry the coolant through them during a bare M8 session. Expected, in
+order: OVERTEMP at the ceiling (the pause tier), CRITICAL at the critical
+line (the fail tier: fire blocked, hold, no resume), the fault ending with
+the session and the ceiling's hold standing after it. The settings are
+restored at the end and a short session makes the engine re-read them.
+
+No laser, no motion, nothing armed. Runs on the board (the bench page) or
+from a LAN host (gfbench: GF_HOST). Results as JSON in the bench data
+directory.
+
+Usage: critical_tier_drill.py [--max-seconds N]    (default 1200)
+"""
+import json
+import socket
+import sys
+import time
+
+from gfbench import HOST, data_path, forgectrl_get, forgectrl_post
+
+KEYS = ("cool_temp_max", "cool_temp_resume", "cool_temp_critical_c",
+        "cool_flow_heater_pct", "cool_flow_check_s", "cool_recheck_s", "cool_flow_rise")
+ABOVE = {"cool_temp_max": 0.4, "cool_temp_resume": 0.2, "cool_temp_critical_c": 0.7}
+HEAT = {"cool_flow_heater_pct": "100", "cool_flow_check_s": "300", "cool_recheck_s": "30",
+        "cool_flow_rise": "40"}
+
+
+class Grbl:
+    def __init__(self):
+        self.s = socket.create_connection((HOST, 23), timeout=5)
+        self.s.settimeout(0.15)
+        time.sleep(0.5)
+        self.drain()
+
+    def drain(self):
+        out = b""
+        try:
+            while True:
+                d = self.s.recv(4096)
+                if not d:
+                    break
+                out += d
+        except socket.timeout:
+            pass
+        return out.decode("ascii", "replace")
+
+    def cmd(self, line, wait=0.4):
+        self.s.sendall(line.encode() + b"\n")
+        time.sleep(wait)
+        return self.drain().strip()
+
+    def close(self):
+        self.s.close()
+
+
+def cool():
+    st, c = forgectrl_get("/cool/status")
+    if st != 200 or not isinstance(c, dict):
+        raise RuntimeError("/cool/status -> %s %r" % (st, c))
+    return c
+
+
+def row(c, t):
+    return "%5.0f s  %-6s %-8s fire_ok=%-5s hold=%-5s up %.2f down %.2f | %s" % (
+        t, c["phase"], c["verdict"], c["fire_ok"], c["hold"], c["up_c"], c["down_c"], c.get("reason", ""))
+
+
+def session_ended(wait=60):
+    for _ in range(wait):
+        time.sleep(1)
+        c = cool()
+        if c["phase"] != "run":
+            return c
+    return cool()
+
+
+def main(argv):
+    max_s = 1200
+    args = argv[1:]
+    while args:
+        a = args.pop(0)
+        if a == "--max-seconds":
+            max_s = int(args.pop(0))
+        else:
+            print(__doc__)
+            return 2
+    st, before = forgectrl_get("/settings")
+    if st != 200:
+        print("GET /settings -> %s" % st)
+        return 2
+    orig = {k: before.get(k, "") for k in KEYS}
+    print("original settings: %s" % orig)
+    c = cool()
+    if c["phase"] == "run":
+        print("a run session is already open; nothing done")
+        return 2
+    t0_up = c["up_c"]
+    lines = {k: "%.1f" % (t0_up + d) for k, d in ABOVE.items()}
+    lines.update(HEAT)
+    print("upstream %.2f C; drill settings: %s" % (t0_up, lines))
+    st, body = forgectrl_post("/settings", params=lines)
+    if st != 200:
+        print("POST /settings -> %s %r" % (st, body))
+        return 2
+    result = {"started": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), "upstream_start_c": t0_up,
+              "settings": lines, "transitions": [], "critical_s": None}
+    g = None
+    rc = 1
+    try:
+        g = Grbl()
+        print("M8 -> %s" % g.cmd("M8"))
+        t0 = time.time()
+        last = None
+        crit_at = None
+        while time.time() - t0 < max_s:
+            time.sleep(1)
+            c = cool()
+            t = time.time() - t0
+            key = (c["verdict"], c["fire_ok"], c["hold"])
+            if key != last or int(t) % 15 == 0:
+                print(row(c, t))
+            if key != last:
+                result["transitions"].append({"t_s": round(t), "verdict": c["verdict"], "up_c": c["up_c"],
+                                              "down_c": c["down_c"], "reason": c.get("reason", "")})
+                last = key
+            if c["verdict"] == "CRITICAL" and crit_at is None:
+                crit_at = t
+                result["critical_s"] = round(t)
+            if crit_at is not None and t - crit_at >= 10:
+                break
+        if crit_at is None:
+            print("no CRITICAL within %d s (upstream %.2f C)" % (max_s, c["up_c"]))
+        print("M9 -> %s" % g.cmd("M9"))
+        c = session_ended()
+        print("after the session: %s" % row(c, time.time() - t0))
+        result["after_session"] = {k: c.get(k) for k in ("phase", "verdict", "fire_ok", "hold", "reason", "up_c")}
+        rc = 0 if crit_at is not None else 1
+    finally:
+        if g:
+            try:
+                g.cmd("M9")
+            except OSError:
+                pass
+            g.close()
+        st, body = forgectrl_post("/settings", params=orig)
+        print("restore POST /settings -> %s" % st)
+        result["restored"] = st == 200
+        try:
+            session_ended()
+            g2 = Grbl()
+            g2.cmd("M8")
+            time.sleep(3)
+            c = cool()
+            print("re-read session: %s %s limits %s" % (c["phase"], c["verdict"], c.get("limits")))
+            g2.cmd("M9")
+            g2.close()
+            c = session_ended()
+            print("final: %s" % row(c, 0))
+            result["final"] = {k: c.get(k) for k in ("phase", "verdict", "fire_ok", "hold", "up_c")}
+        except OSError as e:
+            print("re-read session failed: %s" % e)
+    out = data_path("critical_tier_drill_%s.json" % time.strftime("%Y%m%d-%H%M%S"))
+    with open(out, "w") as f:
+        json.dump(result, f, indent=1)
+    print("wrote %s" % out)
+    print("RESULT: %s" % ("PASS" if rc == 0 else "FAIL"))
+    return rc
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv))

scripts/bench/temp_calibrate.py

@@ -14,6 +14,14 @@ Usage:
   temp_calibrate.py point <measured_C> [note]  record a calibration point
   temp_calibrate.py fit                        fit and print the calibration
 
+  temp_calibrate.py supply-watch [seconds]     the same for the power supply
+  temp_calibrate.py supply-point <C> [note]    sensor (pic/pwr_temp, raw):
+  temp_calibrate.py supply-fit                 thermometer on the heatsink vs
+                                               the raw count; UAPI.md's guess
+                                               raw * 0.08715 - 21 is shown
+                                               beside it. Points go to
+                                               supply_calibration.json.
+
 Points accumulate in temp_calibration.json in the bench data directory
 (gfbench.data_path: next to this script, or FORGETEST_BENCH_DATA). Take
 at least two points as far apart in temperature as practical (e.g. cold
@@ -28,6 +36,22 @@ import time
 from gfbench import board, degc, data_path
 
 STORE = data_path('temp_calibration.json')
+SUPPLY_STORE = data_path('supply_calibration.json')
+
+
+def supply_guess_c(raw):
+    """UAPI.md's unverified guess for pic/pwr_temp."""
+    return raw * 0.08715 - 21
+
+
+def supply_raw(samples=5, delay=1.0):
+    acc, n = 0, 0
+    for _ in range(samples):
+        out = board('cat /sys/glowforge/pic/pwr_temp').strip()
+        if out.isdigit():
+            acc += int(out); n += 1
+        time.sleep(delay)
+    return acc / n if n else None
 
 
 def uapi_c(raw):
@@ -72,8 +96,71 @@ def fit(points, key):
     return slope, my - slope * mx
 
 
+def supply_main(mode):
+    if mode == 'supply-watch':
+        seconds = float(sys.argv[2]) if len(sys.argv) > 2 else 60.0
+        print('pwr_temp raw   guess-C   (%.0f s)' % seconds)
+        t0 = time.time()
+        while time.time() - t0 < seconds:
+            r = supply_raw(1, 0)
+            if r is None:
+                print('  (no reading)')
+            else:
+                print('  %6.1f      %.2f' % (r, supply_guess_c(r)), flush=True)
+            time.sleep(2)
+        return 0
+    if mode == 'supply-point':
+        try:
+            measured = float(sys.argv[2])
+        except (IndexError, ValueError):
+            print('supply-point needs the thermometer reading in C (value)')
+            return 2
+        note = sys.argv[3] if len(sys.argv) > 3 else ''
+        print('sampling pwr_temp (5 s)...', flush=True)
+        r = supply_raw()
+        if r is None:
+            print('no reading from the machine')
+            return 1
+        data = {'points': []}
+        if os.path.exists(SUPPLY_STORE):
+            with open(SUPPLY_STORE) as f:
+                data = json.load(f)
+        data['points'].append({'measured_c': measured, 'raw': r, 'note': note,
+                               'when': time.strftime('%Y-%m-%d %H:%M:%S')})
+        with open(SUPPLY_STORE, 'w') as f:
+            json.dump(data, f, indent=2)
+        print('recorded: measured %.2f C  raw=%.1f (guess %.1f C)  (%d points total in %s)'
+              % (measured, r, supply_guess_c(r), len(data['points']), SUPPLY_STORE))
+        return 0
+    if mode == 'supply-fit':
+        if not os.path.exists(SUPPLY_STORE):
+            print('no points yet')
+            return 1
+        with open(SUPPLY_STORE) as f:
+            pts = json.load(f)['points']
+        if len(pts) < 2:
+            print('need at least 2 points (have %d)' % len(pts))
+            return 1
+        print('points:')
+        for p in pts:
+            print('  %6.2f C   raw=%.1f  guess %.1f C   %s %s'
+                  % (p['measured_c'], p['raw'], supply_guess_c(p['raw']), p['when'], p['note']))
+        slope, offset = fit(pts, 'raw')
+        if slope is None:
+            print('points share one raw value; no fit')
+            return 1
+        print('fit: degC = %.5f * raw + %.2f   (UAPI.md guess: 0.08715 * raw - 21)' % (slope, offset))
+        worst = max(abs(p['measured_c'] - supply_guess_c(p['raw'])) for p in pts)
+        print('the guess is off by at most %.1f C at these points' % worst)
+        return 0
+    print(__doc__)
+    return 2
+
+
 def main():
     mode = sys.argv[1] if len(sys.argv) > 1 else 'watch'
+    if mode.startswith('supply-'):
+        return supply_main(mode)
 
     if mode == 'watch':
         seconds = float(sys.argv[2]) if len(sys.argv) > 2 else 60.0