feat(semif): 0.1.3 — order averaging, fast kernels, bug-hunt hardening (Prime)

Order averaging (Prime, after the 739aa03 spike):
- A decision may set orderings: rotations|all (all only for <= 4 options). Every
  ordering goes to the engine in one shared batch.
- The reply keeps each native result and adds combined {probabilities (log-mean),
  top, agreement, spread}.
- Through the service on SemIf's labelled sets (252 rows): 78.6% -> 88.1%
  (group-bootstrap 95% CI +5.1..+14.3). Unanimous agreement is 94.5% accurate.

Fast kernels: flash-linear-attention 0.5.2 and causal-conv1d 1.7.0 are now the
default build. A/B on the empty GPU 3:
- parity with upstream went from 142/144 to 144/144;
- a ~2k-token /decide went from 169 to 92 ms server-side;
- short 3-rotation batches cost ~3-6 ms more.
triton builds a C shim at runtime, so the image carries gcc. Without it the
warm-up failed and startup failed closed.

Heid bug-hunt panel (4/4 arms, thread 01M3H3F4RR7XBP90KQ3A39H4SX), folded:
- Startup validation: VRAM cap 0 no longer means uncapped (C1); limits must be
  >= 1 (S1); the token must be visible ASCII (S2); the calibration file must
  exist and parse, with T in [0.05, 20] (S8, and C3's NaN leg).
- The body limit is checked before a chunk is kept, and a Unicode-digit
  Content-Length no longer crashes (C2, S3).
- Failures while building the response now get the 500 envelope (C3).
- 429 busy past SEMIF_MAX_QUEUE requests in progress (C6).
- The engine releases memory on every non-validation failure, unchained after
  gc; an empty OOM message is handled; 'out of memory' RuntimeErrors map to 503
  (C4, C5, S9).
- The entry point forces HF_HUB_OFFLINE (S10). README wording fixed (S5, S6).
- New guard tests close the gaps the arms' mutation grids exposed: early stop of
  the body read, a shared-route lock, calibration pass-through, the gc cycle,
  the exact caps, TorchEngine.load's arch and device checks, and the offline
  entry point.
86 tests.

Deployed on fv-ml1 GPU 1: parity 144/144, OOM and burst release verified, shared
capacity 63/51/26/16 rows at ~140/520/1960/3900 prefix tokens.

services/semif-serve/src/semif_serve/engine.py

@@ -7,10 +7,14 @@ unit-tested against a fake torch (tests/test_engine.py).
 from __future__ import annotations
 
 import gc
+import logging
+import traceback
 from typing import Any, Callable
 
 from .config import Settings
-from .errors import OutOfMemory
+from .errors import OutOfMemory, ScoringFailed
+
+log = logging.getLogger("semif_serve.engine")
 
 RELEASE_SLACK_BYTES = 512 * 2**20
 WARMUP_ROW = {
@@ -24,6 +28,11 @@ WARMUP_ROW = {
 }
 
 
+def _first_line(exc: BaseException) -> str:
+    lines = str(exc).splitlines()
+    return lines[0] if lines else ""
+
+
 class TorchEngine:
     def __init__(self, torch: Any, model: Any, tokenizer: Any, metadata: dict, settings: Settings,
                  direct_fn: Callable, shared_fn: Callable, release_above_bytes: int | None = None):
@@ -46,7 +55,7 @@ class TorchEngine:
             arch = f"sm_{major}{minor}"
             if arch not in torch.cuda.get_arch_list():            # INV-3: no silent PTX/CPU fallback
                 raise RuntimeError(f"torch {torch.__version__} has no kernels for {arch}: {torch.cuda.get_arch_list()}")
-            if settings.vram_cap_gib:                              # INV-4: cap BEFORE the weights land
+            if settings.vram_cap_gib is not None:                  # INV-4: cap BEFORE the weights land
                 total = torch.cuda.get_device_properties(0).total_memory
                 fraction = settings.vram_cap_gib * 2**30 / total
                 if not 0 < fraction <= 1:
@@ -82,17 +91,28 @@ class TorchEngine:
     def _guard(self, fn, *args):
         try:
             result = fn(*args)
+        except ValueError:
+            raise                            # validation: SemIf raises it before any GPU work
         except self._torch.cuda.OutOfMemoryError as exc:
-            message = str(exc).splitlines()[0]
+            failure, message = OutOfMemory, _first_line(exc) or "CUDA out of memory"
+        except Exception as exc:  # noqa: BLE001 — every other failure is released and reported below
+            message = _first_line(exc)
+            if "out of memory" in message.lower():      # cuBLAS/cuDNN allocation failures
+                failure = OutOfMemory
+            else:
+                failure, message = ScoringFailed, f"{type(exc).__name__}: {message}"
+                # Formatted text, not exc_info: a log record that keeps the traceback object alive
+                # (pytest's capture handler does; so would any buffering handler) pins the tensors.
+                log.error("scorer failed:\n%s", traceback.format_exc())
         else:
             self._release_burst()
             return result
-        # INV-4, outside the except block on purpose: the torch exception's traceback holds the
-        # failed scorer's frames, and with them its tensors (the replicated prefix cache). Raising
-        # inside the block, or `from exc`, would chain to it and keep GiBs allocated after the 503.
+        # INV-4, outside the except block on purpose: the exception's traceback holds the failed
+        # scorer's frames, and with them its tensors (the replicated prefix cache). Raising inside
+        # the block, or `from exc`, would chain to it and keep GiBs allocated after the response.
         gc.collect()
         self._torch.cuda.empty_cache()
-        raise OutOfMemory(message)
+        raise failure(message)
 
     def direct(self, row: dict) -> dict:
         return self._guard(self._direct, self._model, self._tokenizer, row, self._metadata, self._settings.max_tokens)