vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
66034cc69e2a480577258d1323a548e682f6e9cf
esh-pfi-infrastructure/services/semif-serve/bench-jev-2026-09-30/tables.md
T
vh 475d6d6bcb docs: Jev candidate bench vs SemIf (fv-ml1 GPU 3) — Intern-Decision-4B is the replacement if SemIf is displaced 
Operator ask relayed by brokkr-smithy-dev. Positive control (SemIf 187/231,
hard 0.613) reproduced exactly; negative control and a 4-restart noise floor
(0 flips) measured. On our replaced-baseline sets no candidate beats SemIf-with-
rotations beyond the ~4-pt floor; Intern-Decision-4B native matches it at one
ordering, is better on Wyrd, fits 9.7/10.3 GB and is 1.5-2.3x faster. JevBench
rank does not transfer. Raw per-item data kept out of git.
2026-09-30 05:00:45 -07:00

281 lines
25 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
### Key table
| system | fits 12 GiB? rest / peak MiB | JevBench all /231 · hard /111 | pooled /259 single · rot | Wyrd /84 single · rot | Δ pooled vs SemIf single (95% CI) | Δ pooled vs SemIf rot (95% CI) | 21 criteria, ms | 16 × ~3,900 tok, ms |
|---|---|---|---|---|---|---|---|---|
| SemIf (Qwen3.5-4B), baseline | yes: 9242 (9242–9250) / 12918 (12886–12918) | 187 · 68 | 217 · 232 | 71 · 71 | baseline | baseline | 131 (131–132) | 505 (502–519) |
| Plumb-4B, native | yes: 12064 / 12064 | 207 (206–207) · 89 (88–89) | 224 (224–226) · 227 (227–228) | 65 · 67 | +2.7 (-2.7..+7.9) | -1.9 (-5.4..+1.5) | 294 (292–295) | 3378 (3374–3384) |
| Plumb-4B, drop-in | yes: 9242 / 12918 | 206 · 88 | 221 · 224 | 66 · 67 | +1.5 (-3.7..+6.6) | -3.1 (-6.8..+0.4) | 131 (131–132) | 506 (506–507) |
| Imajev-4B, native | yes: 10404 / 11056 | 199 · 80 | 235 · 236 | 68 · 69 | +6.9 (+1.6..+12.2) | +1.5 (-2.5..+5.6) | 1219 (1212–1230) | 5125 (5054–5140) |
| Intern-Decision-4B, native | yes: 9736 / 10290 | 202 · 83 | 240 · 236 | 79 · 77 | +8.9 (+4.7..+13.2) | +1.5 (-2.0..+5.0) | 88 (88–89) | 215 (213–216) |
| Intern-Decision-4B, drop-in | yes: 9242 / 12918 | 202 · 85 | 232 · 233 | 74 · 73 | +5.8 (+2.7..+9.0) | +0.4 (-2.3..+3.1) | 132 | 508 (500–516) |
| JevK5 v0.2, native | yes: 12064 / 12064 | 198 (198–199) · 81 (81–82) | 229 (228–229) · 228 | 67 (66–67) · 67 | +4.6 (-0.8..+9.9) | -1.5 (-4.9..+1.9) | 291 (291–292) | 3360 (3358–3368) |
| JevK5 v0.2, drop-in | yes: 9242 / 12918 | 200 · 82 | 232 · 237 | 72 · 74 | +5.8 (+1.1..+10.7) | +1.9 (-1.1..+5.1) | 132 | 518 (505–519) |
| JevK5 v0.3, native | yes: 12064 / 12064 | 202 · 86 | 227 · 226 | 66 · 65 | +3.9 (-0.9..+8.5) | -2.3 (-6.0..+1.4) | 291 (291–292) | 3363 (3355–3370) |
| JevK5 v0.3, drop-in | yes: 9242 / 12918 | 201 · 85 | 227 · 227 | 69 · 66 | +3.9 (-0.8..+8.2) | -1.9 (-5.5..+1.8) | 133 (131–133) | 518 (506–521) |
### JevBench v1.2.16 public items (positive control + candidate score)
| system | repeats | easy /48 | standard /72 | hard /111 | all /231 | all | p50 ms |
|---|---|---|---|---|---|---|---|
| SemIf (Qwen3.5-4B), baseline | 4 | 48 | 71 | 68 | 187 | 0.810 | 36 |
| Plumb-4B, native | 3 | 48 | 70 | 89 (88–89) | 207 (206–207) | 0.896 (0.892–0.896) | 20 |
| Plumb-4B, drop-in | 3 | 48 | 70 | 88 | 206 | 0.892 | 36 |
| Imajev-4B, native | 4 | 48 | 71 | 80 | 199 | 0.861 | 62 (62–63) |
| Intern-Decision-4B, native | 4 | 48 | 71 | 83 | 202 | 0.874 | 40 |
| Intern-Decision-4B, drop-in | 3 | 48 | 69 | 85 | 202 | 0.874 | 36 |
| JevK5 v0.2, native | 3 | 48 | 69 | 81 (81–82) | 198 (198–199) | 0.857 (0.857–0.861) | 20 (20–21) |
| JevK5 v0.2, drop-in | 3 | 48 | 70 | 82 | 200 | 0.866 | 36 (36–37) |
| JevK5 v0.3, native | 3 | 48 | 68 | 86 | 202 | 0.874 | 20 (20–21) |
| JevK5 v0.3, drop-in | 3 | 48 | 68 | 85 | 201 | 0.870 | 36 |
### Replaced-baseline sets, single (caller order)
| system | pooled /259 | authored144 | perturb108 | Cicada w1 /31 | Cicada w2 /31 | Wyrd /84 | Wyrd place2 /21 | Wyrd exit /21 |
|---|---|---|---|---|---|---|---|---|
| SemIf (Qwen3.5-4B), baseline | 217 | 116 | 83 | 30 | 20 | 71 | 21 | 16 |
| Plumb-4B, native | 224 (224–226) | 130 (130–132) | 84 | 29 | 19 | 65 | 21 | 10 |
| Plumb-4B, drop-in | 221 | 129 | 88 | 26 | 20 | 66 | 21 | 12 |
| Imajev-4B, native | 235 | 138 | 102 | 29 | 22 | 68 | 20 | 13 |
| Intern-Decision-4B, native | 240 | 132 | 100 | 29 | 22 | 79 | 21 | 19 |
| Intern-Decision-4B, drop-in | 232 | 128 | 102 | 30 | 19 | 74 | 21 | 17 |
| JevK5 v0.2, native | 229 (228–229) | 132 | 85 (85–86) | 30 | 16 | 67 (66–67) | 21 | 12 (11–12) |
| JevK5 v0.2, drop-in | 232 | 130 | 92 | 30 | 17 | 72 | 21 | 17 |
| JevK5 v0.3, native | 227 | 135 | 101 (101–102) | 26 | 17 | 66 | 21 | 12 |
| JevK5 v0.3, drop-in | 227 | 132 | 99 | 26 | 21 | 69 | 21 | 15 |
### Replaced-baseline sets, rotations (n rotations averaged)
| system | pooled /259 | authored144 | perturb108 | Cicada w1 /31 | Cicada w2 /31 | Wyrd /84 | Wyrd place2 /21 | Wyrd exit /21 |
|---|---|---|---|---|---|---|---|---|
| SemIf (Qwen3.5-4B), baseline | 232 | 131 | 93 | 30 | 19 | 71 | 21 | 18 |
| Plumb-4B, native | 227 (227–228) | 130 (130–131) | 87 | 30 | 18 (17–18) | 67 | 21 | 11 |
| Plumb-4B, drop-in | 224 | 129 | 92 | 28 | 18 | 67 | 21 | 13 |
| Imajev-4B, native | 236 | 138 | 101 | 29 | 23 | 69 | 20 | 13 |
| Intern-Decision-4B, native | 236 | 130 | 99 | 29 | 22 | 77 | 21 | 19 |
| Intern-Decision-4B, drop-in | 233 | 130 | 102 | 30 | 19 | 73 | 21 | 17 |
| JevK5 v0.2, native | 228 | 131 | 91 (90–91) | 30 | 16 | 67 | 21 | 12 |
| JevK5 v0.2, drop-in | 237 | 133 | 93 | 30 | 16 | 74 | 21 | 20 |
| JevK5 v0.3, native | 226 | 135 | 101 | 26 | 16 | 65 | 20 | 12 |
| JevK5 v0.3, drop-in | 227 | 135 | 101 | 26 | 19 | 66 | 20 | 13 |
### Wyrd as one request per turn (4 decisions over one state: SemIf /decide/shared, native multi-question request)
| system | repeats | Wyrd /84 | place /21 | place2 /21 | exit /21 | exit2 /21 |
|---|---|---|---|---|---|---|
| SemIf (Qwen3.5-4B), baseline | 3 | 71 | 17 | 21 | 16 | 17 |
| Plumb-4B, native | 2 | 65 | 17 | 21 | 10 | 17 |
| Plumb-4B, drop-in | 3 | 66 | 17 | 21 | 12 | 16 |
| Imajev-4B, native | 3 | 68 | 19 | 20 | 13 | 16 |
| Intern-Decision-4B, native | 3 | 77 | 16 | 21 | 20 | 20 |
| Intern-Decision-4B, drop-in | 3 | 74 | 18 | 21 | 17 | 18 |
| JevK5 v0.2, native | 3 | 67 (66–67) | 17 | 21 | 12 (11–12) | 17 |
| JevK5 v0.2, drop-in | 3 | 73 | 18 | 21 | 18 | 16 |
| JevK5 v0.3, native | 3 | 66 | 18 | 21 | 12 | 15 |
| JevK5 v0.3, drop-in | 3 | 69 | 18 | 21 | 15 | 15 |
### Null control (content-free state) and positive controls inside the spike sets, single ordering
| system | Cicada w1 blind /31 | Wyrd blind /84 | Cicada w1 controls /5 | Wyrd controls /28 |
|---|---|---|---|---|
| SemIf (Qwen3.5-4B), baseline | 16 | 60 | 5 | 21 |
| Plumb-4B, native | 15 | 50 | 5 | 20 |
| Plumb-4B, drop-in | 15 | 60 | 4 | 21 |
| Imajev-4B, native | 16 | 48 | 5 | 24 |
| Intern-Decision-4B, native | 16 | 60 | 5 | 27 |
| Intern-Decision-4B, drop-in | 16 | 60 | 5 | 23 |
| JevK5 v0.2, native | 15 | 53 | 5 | 22 (21–22) |
| JevK5 v0.2, drop-in | 15 | 60 | 5 | 23 |
| JevK5 v0.3, native | 16 | 60 | 5 | 20 |
| JevK5 v0.3, drop-in | 16 | 60 | 5 | 22 |
### Paired against SemIf (each row's majority top over the repeats; group bootstrap 95% CI; exact McNemar)
| system | set | cond | SemIf | cand | fixed | broken | Δ pts | 95% CI | p |
|---|---|---|---|---|---|---|---|---|---|
| Plumb-4B, native | pooled | single | 217/259 | 224/259 | 23 | 16 | +2.7 | -2.7..+7.9 | 0.3368 |
| Plumb-4B, native | pooled | rotations | 232/259 | 227/259 | 10 | 15 | -1.9 | -5.4..+1.5 | 0.4244 |
| Plumb-4B, native | authored144 | single | 116/144 | 130/144 | 22 | 8 | +9.7 | +1.4..+18.1 | 0.0161 |
| Plumb-4B, native | authored144 | rotations | 131/144 | 130/144 | 7 | 8 | -0.7 | -5.6..+3.5 | 1.0 |
| Plumb-4B, native | cicada-w1 | single | 30/31 | 29/31 | 0 | 1 | -3.2 | -9.7..+0.0 | 1.0 |
| Plumb-4B, native | cicada-w1 | rotations | 30/31 | 30/31 | 0 | 0 | +0.0 | +0.0..+0.0 | 1.0 |
| Plumb-4B, native | wyrd | single | 71/84 | 65/84 | 1 | 7 | -7.1 | -13.1..-1.2 | 0.0703 |
| Plumb-4B, native | wyrd | rotations | 71/84 | 67/84 | 3 | 7 | -4.8 | -11.9..+2.4 | 0.3438 |
| Plumb-4B, native | perturbations108 | single | 83/108 | 84/108 | 10 | 9 | +0.9 | -9.3..+11.1 | 1.0 |
| Plumb-4B, native | perturbations108 | rotations | 93/108 | 87/108 | 4 | 10 | -5.6 | -15.7..+3.7 | 0.1796 |
| Plumb-4B, native | cicada-w2 | single | 20/31 | 19/31 | 1 | 2 | -3.2 | -12.9..+6.5 | 1.0 |
| Plumb-4B, native | cicada-w2 | rotations | 19/31 | 18/31 | 1 | 2 | -3.2 | -16.1..+6.5 | 1.0 |
| Plumb-4B, native | pooled | cand-single-vs-semif-rotations | 232/259 | 224/259 | 10 | 18 | -3.1 | -7.3..+0.8 | 0.1849 |
| Plumb-4B, native | wyrd | cand-single-vs-semif-rotations | 71/84 | 65/84 | 3 | 9 | -7.1 | -15.5..+1.2 | 0.146 |
| Plumb-4B, drop-in | pooled | single | 217/259 | 221/259 | 21 | 17 | +1.5 | -3.7..+6.6 | 0.6271 |
| Plumb-4B, drop-in | pooled | rotations | 232/259 | 224/259 | 7 | 15 | -3.1 | -6.8..+0.4 | 0.1338 |
| Plumb-4B, drop-in | authored144 | single | 116/144 | 129/144 | 20 | 7 | +9.0 | +1.4..+16.7 | 0.0192 |
| Plumb-4B, drop-in | authored144 | rotations | 131/144 | 129/144 | 5 | 7 | -1.4 | -6.2..+3.5 | 0.7744 |
| Plumb-4B, drop-in | cicada-w1 | single | 30/31 | 26/31 | 0 | 4 | -12.9 | -25.8..-3.2 | 0.125 |
| Plumb-4B, drop-in | cicada-w1 | rotations | 30/31 | 28/31 | 0 | 2 | -6.5 | -16.1..+0.0 | 0.5 |
| Plumb-4B, drop-in | wyrd | single | 71/84 | 66/84 | 1 | 6 | -6.0 | -11.9..+0.0 | 0.125 |
| Plumb-4B, drop-in | wyrd | rotations | 71/84 | 67/84 | 2 | 6 | -4.8 | -11.9..+2.4 | 0.2891 |
| Plumb-4B, drop-in | perturbations108 | single | 83/108 | 88/108 | 12 | 7 | +4.6 | -5.6..+14.8 | 0.3593 |
| Plumb-4B, drop-in | perturbations108 | rotations | 93/108 | 92/108 | 6 | 7 | -0.9 | -10.2..+7.4 | 1.0 |
| Plumb-4B, drop-in | cicada-w2 | single | 20/31 | 20/31 | 1 | 1 | +0.0 | -9.7..+9.7 | 1.0 |
| Plumb-4B, drop-in | cicada-w2 | rotations | 19/31 | 18/31 | 0 | 1 | -3.2 | -9.7..+0.0 | 1.0 |
| Plumb-4B, drop-in | pooled | cand-single-vs-semif-rotations | 232/259 | 221/259 | 8 | 19 | -4.2 | -8.3..-0.4 | 0.0522 |
| Plumb-4B, drop-in | wyrd | cand-single-vs-semif-rotations | 71/84 | 66/84 | 2 | 7 | -6.0 | -13.1..+1.2 | 0.1797 |
| Imajev-4B, native | pooled | single | 217/259 | 235/259 | 30 | 12 | +6.9 | +1.6..+12.2 | 0.0079 |
| Imajev-4B, native | pooled | rotations | 232/259 | 236/259 | 14 | 10 | +1.5 | -2.5..+5.6 | 0.5413 |
| Imajev-4B, native | authored144 | single | 116/144 | 138/144 | 25 | 3 | +15.3 | +9.0..+21.5 | 0.0 |
| Imajev-4B, native | authored144 | rotations | 131/144 | 138/144 | 8 | 1 | +4.9 | +0.7..+9.0 | 0.0391 |
| Imajev-4B, native | cicada-w1 | single | 30/31 | 29/31 | 0 | 1 | -3.2 | -9.7..+0.0 | 1.0 |
| Imajev-4B, native | cicada-w1 | rotations | 30/31 | 29/31 | 0 | 1 | -3.2 | -9.7..+0.0 | 1.0 |
| Imajev-4B, native | wyrd | single | 71/84 | 68/84 | 5 | 8 | -3.6 | -13.1..+6.0 | 0.5811 |
| Imajev-4B, native | wyrd | rotations | 71/84 | 69/84 | 6 | 8 | -2.4 | -11.9..+7.1 | 0.7905 |
| Imajev-4B, native | perturbations108 | single | 83/108 | 102/108 | 20 | 1 | +17.6 | +8.3..+27.8 | 0.0 |
| Imajev-4B, native | perturbations108 | rotations | 93/108 | 101/108 | 9 | 1 | +7.4 | +1.9..+13.9 | 0.0215 |
| Imajev-4B, native | cicada-w2 | single | 20/31 | 22/31 | 3 | 1 | +6.5 | -6.5..+19.4 | 0.625 |
| Imajev-4B, native | cicada-w2 | rotations | 19/31 | 23/31 | 4 | 0 | +12.9 | +3.2..+25.8 | 0.125 |
| Imajev-4B, native | pooled | cand-single-vs-semif-rotations | 232/259 | 235/259 | 15 | 12 | +1.2 | -3.2..+5.5 | 0.7011 |
| Imajev-4B, native | wyrd | cand-single-vs-semif-rotations | 71/84 | 68/84 | 6 | 9 | -3.6 | -14.3..+6.0 | 0.6072 |
| Intern-Decision-4B, native | pooled | single | 217/259 | 240/259 | 28 | 5 | +8.9 | +4.7..+13.2 | 0.0001 |
| Intern-Decision-4B, native | pooled | rotations | 232/259 | 236/259 | 12 | 8 | +1.5 | -2.0..+5.0 | 0.5034 |
| Intern-Decision-4B, native | authored144 | single | 116/144 | 132/144 | 20 | 4 | +11.1 | +4.9..+18.1 | 0.0015 |
| Intern-Decision-4B, native | authored144 | rotations | 131/144 | 130/144 | 5 | 6 | -0.7 | -5.6..+4.2 | 1.0 |
| Intern-Decision-4B, native | cicada-w1 | single | 30/31 | 29/31 | 0 | 1 | -3.2 | -9.7..+0.0 | 1.0 |
| Intern-Decision-4B, native | cicada-w1 | rotations | 30/31 | 29/31 | 0 | 1 | -3.2 | -9.7..+0.0 | 1.0 |
| Intern-Decision-4B, native | wyrd | single | 71/84 | 79/84 | 8 | 0 | +9.5 | +3.6..+16.7 | 0.0078 |
| Intern-Decision-4B, native | wyrd | rotations | 71/84 | 77/84 | 7 | 1 | +7.1 | +1.2..+13.1 | 0.0703 |
| Intern-Decision-4B, native | perturbations108 | single | 83/108 | 100/108 | 20 | 3 | +15.7 | +4.6..+26.9 | 0.0005 |
| Intern-Decision-4B, native | perturbations108 | rotations | 93/108 | 99/108 | 9 | 3 | +5.6 | -2.8..+13.0 | 0.146 |
| Intern-Decision-4B, native | cicada-w2 | single | 20/31 | 22/31 | 2 | 0 | +6.5 | +0.0..+16.1 | 0.5 |
| Intern-Decision-4B, native | cicada-w2 | rotations | 19/31 | 22/31 | 4 | 1 | +9.7 | -3.2..+22.6 | 0.375 |
| Intern-Decision-4B, native | pooled | cand-single-vs-semif-rotations | 232/259 | 240/259 | 14 | 6 | +3.1 | -0.4..+6.7 | 0.1153 |
| Intern-Decision-4B, native | wyrd | cand-single-vs-semif-rotations | 71/84 | 79/84 | 9 | 1 | +9.5 | +2.4..+16.7 | 0.0215 |
| Intern-Decision-4B, drop-in | pooled | single | 217/259 | 232/259 | 18 | 3 | +5.8 | +2.7..+9.0 | 0.0015 |
| Intern-Decision-4B, drop-in | pooled | rotations | 232/259 | 233/259 | 8 | 7 | +0.4 | -2.3..+3.1 | 1.0 |
| Intern-Decision-4B, drop-in | authored144 | single | 116/144 | 128/144 | 15 | 3 | +8.3 | +3.5..+13.9 | 0.0075 |
| Intern-Decision-4B, drop-in | authored144 | rotations | 131/144 | 130/144 | 5 | 6 | -0.7 | -4.9..+3.5 | 1.0 |
| Intern-Decision-4B, drop-in | cicada-w1 | single | 30/31 | 30/31 | 0 | 0 | +0.0 | +0.0..+0.0 | 1.0 |
| Intern-Decision-4B, drop-in | cicada-w1 | rotations | 30/31 | 30/31 | 0 | 0 | +0.0 | +0.0..+0.0 | 1.0 |
| Intern-Decision-4B, drop-in | wyrd | single | 71/84 | 74/84 | 3 | 0 | +3.6 | +0.0..+8.3 | 0.25 |
| Intern-Decision-4B, drop-in | wyrd | rotations | 71/84 | 73/84 | 3 | 1 | +2.4 | -2.4..+7.1 | 0.625 |
| Intern-Decision-4B, drop-in | perturbations108 | single | 83/108 | 102/108 | 20 | 1 | +17.6 | +7.4..+28.7 | 0.0 |
| Intern-Decision-4B, drop-in | perturbations108 | rotations | 93/108 | 102/108 | 9 | 0 | +8.3 | +2.8..+15.7 | 0.0039 |
| Intern-Decision-4B, drop-in | cicada-w2 | single | 20/31 | 19/31 | 0 | 1 | -3.2 | -9.7..+0.0 | 1.0 |
| Intern-Decision-4B, drop-in | cicada-w2 | rotations | 19/31 | 19/31 | 0 | 0 | +0.0 | +0.0..+0.0 | 1.0 |
| Intern-Decision-4B, drop-in | pooled | cand-single-vs-semif-rotations | 232/259 | 232/259 | 10 | 10 | +0.0 | -3.0..+3.1 | 1.0 |
| Intern-Decision-4B, drop-in | wyrd | cand-single-vs-semif-rotations | 71/84 | 74/84 | 4 | 1 | +3.6 | -1.2..+8.3 | 0.375 |
| JevK5 v0.2, native | pooled | single | 217/259 | 229/259 | 25 | 13 | +4.6 | -0.8..+9.9 | 0.073 |
| JevK5 v0.2, native | pooled | rotations | 232/259 | 228/259 | 9 | 13 | -1.5 | -4.9..+1.9 | 0.5235 |
| JevK5 v0.2, native | authored144 | single | 116/144 | 132/144 | 23 | 7 | +11.1 | +2.8..+18.8 | 0.0052 |
| JevK5 v0.2, native | authored144 | rotations | 131/144 | 131/144 | 7 | 7 | +0.0 | -4.9..+4.9 | 1.0 |
| JevK5 v0.2, native | cicada-w1 | single | 30/31 | 30/31 | 0 | 0 | +0.0 | +0.0..+0.0 | 1.0 |
| JevK5 v0.2, native | cicada-w1 | rotations | 30/31 | 30/31 | 0 | 0 | +0.0 | +0.0..+0.0 | 1.0 |
| JevK5 v0.2, native | wyrd | single | 71/84 | 67/84 | 2 | 6 | -4.8 | -10.7..+1.2 | 0.2891 |
| JevK5 v0.2, native | wyrd | rotations | 71/84 | 67/84 | 2 | 6 | -4.8 | -10.7..+1.2 | 0.2891 |
| JevK5 v0.2, native | perturbations108 | single | 83/108 | 85/108 | 10 | 8 | +1.9 | -7.4..+11.1 | 0.8145 |
| JevK5 v0.2, native | perturbations108 | rotations | 93/108 | 91/108 | 4 | 6 | -1.9 | -10.2..+5.6 | 0.7539 |
| JevK5 v0.2, native | cicada-w2 | single | 20/31 | 16/31 | 0 | 4 | -12.9 | -25.8..-3.2 | 0.125 |
| JevK5 v0.2, native | cicada-w2 | rotations | 19/31 | 16/31 | 0 | 3 | -9.7 | -19.4..+0.0 | 0.25 |
| JevK5 v0.2, native | pooled | cand-single-vs-semif-rotations | 232/259 | 229/259 | 11 | 14 | -1.2 | -5.0..+2.6 | 0.69 |
| JevK5 v0.2, native | wyrd | cand-single-vs-semif-rotations | 71/84 | 67/84 | 3 | 7 | -4.8 | -13.1..+2.4 | 0.3438 |
| JevK5 v0.2, drop-in | pooled | single | 217/259 | 232/259 | 24 | 9 | +5.8 | +1.1..+10.7 | 0.0135 |
| JevK5 v0.2, drop-in | pooled | rotations | 232/259 | 237/259 | 11 | 6 | +1.9 | -1.1..+5.1 | 0.3323 |
| JevK5 v0.2, drop-in | authored144 | single | 116/144 | 130/144 | 21 | 7 | +9.7 | +2.1..+17.4 | 0.0125 |
| JevK5 v0.2, drop-in | authored144 | rotations | 131/144 | 133/144 | 6 | 4 | +1.4 | -2.8..+5.6 | 0.7539 |
| JevK5 v0.2, drop-in | cicada-w1 | single | 30/31 | 30/31 | 0 | 0 | +0.0 | +0.0..+0.0 | 1.0 |
| JevK5 v0.2, drop-in | cicada-w1 | rotations | 30/31 | 30/31 | 0 | 0 | +0.0 | +0.0..+0.0 | 1.0 |
| JevK5 v0.2, drop-in | wyrd | single | 71/84 | 72/84 | 3 | 2 | +1.2 | -3.6..+6.0 | 1.0 |
| JevK5 v0.2, drop-in | wyrd | rotations | 71/84 | 74/84 | 5 | 2 | +3.6 | -2.4..+10.7 | 0.4531 |
| JevK5 v0.2, drop-in | perturbations108 | single | 83/108 | 92/108 | 13 | 4 | +8.3 | -0.9..+17.6 | 0.049 |
| JevK5 v0.2, drop-in | perturbations108 | rotations | 93/108 | 93/108 | 4 | 4 | +0.0 | -7.4..+6.5 | 1.0 |
| JevK5 v0.2, drop-in | cicada-w2 | single | 20/31 | 17/31 | 0 | 3 | -9.7 | -22.6..+0.0 | 0.25 |
| JevK5 v0.2, drop-in | cicada-w2 | rotations | 19/31 | 16/31 | 0 | 3 | -9.7 | -19.4..+0.0 | 0.25 |
| JevK5 v0.2, drop-in | pooled | cand-single-vs-semif-rotations | 232/259 | 232/259 | 10 | 10 | +0.0 | -3.2..+3.2 | 1.0 |
| JevK5 v0.2, drop-in | wyrd | cand-single-vs-semif-rotations | 71/84 | 72/84 | 3 | 2 | +1.2 | -3.6..+6.0 | 1.0 |
| JevK5 v0.3, native | pooled | single | 217/259 | 227/259 | 25 | 15 | +3.9 | -0.9..+8.5 | 0.1539 |
| JevK5 v0.3, native | pooled | rotations | 232/259 | 226/259 | 9 | 15 | -2.3 | -6.0..+1.4 | 0.3075 |
| JevK5 v0.3, native | authored144 | single | 116/144 | 135/144 | 24 | 5 | +13.2 | +6.9..+19.4 | 0.0005 |
| JevK5 v0.3, native | authored144 | rotations | 131/144 | 135/144 | 7 | 3 | +2.8 | -1.4..+7.6 | 0.3438 |
| JevK5 v0.3, native | cicada-w1 | single | 30/31 | 26/31 | 0 | 4 | -12.9 | -25.8..-3.2 | 0.125 |
| JevK5 v0.3, native | cicada-w1 | rotations | 30/31 | 26/31 | 0 | 4 | -12.9 | -25.8..-3.2 | 0.125 |
| JevK5 v0.3, native | wyrd | single | 71/84 | 66/84 | 1 | 6 | -6.0 | -10.7..-1.2 | 0.125 |
| JevK5 v0.3, native | wyrd | rotations | 71/84 | 65/84 | 2 | 8 | -7.1 | -13.1..-1.2 | 0.1094 |
| JevK5 v0.3, native | perturbations108 | single | 83/108 | 101/108 | 18 | 0 | +16.7 | +8.3..+25.9 | 0.0 |
| JevK5 v0.3, native | perturbations108 | rotations | 93/108 | 101/108 | 8 | 0 | +7.4 | +1.9..+13.9 | 0.0078 |
| JevK5 v0.3, native | cicada-w2 | single | 20/31 | 17/31 | 0 | 3 | -9.7 | -19.4..+0.0 | 0.25 |
| JevK5 v0.3, native | cicada-w2 | rotations | 19/31 | 16/31 | 0 | 3 | -9.7 | -19.4..+0.0 | 0.25 |
| JevK5 v0.3, native | pooled | cand-single-vs-semif-rotations | 232/259 | 227/259 | 11 | 16 | -1.9 | -5.5..+1.5 | 0.4421 |
| JevK5 v0.3, native | wyrd | cand-single-vs-semif-rotations | 71/84 | 66/84 | 2 | 7 | -6.0 | -11.9..+0.0 | 0.1797 |
| JevK5 v0.3, drop-in | pooled | single | 217/259 | 227/259 | 24 | 14 | +3.9 | -0.8..+8.2 | 0.1433 |
| JevK5 v0.3, drop-in | pooled | rotations | 232/259 | 227/259 | 10 | 15 | -1.9 | -5.5..+1.8 | 0.4244 |
| JevK5 v0.3, drop-in | authored144 | single | 116/144 | 132/144 | 22 | 6 | +11.1 | +4.9..+17.4 | 0.0037 |
| JevK5 v0.3, drop-in | authored144 | rotations | 131/144 | 135/144 | 8 | 4 | +2.8 | -1.4..+7.6 | 0.3877 |
| JevK5 v0.3, drop-in | cicada-w1 | single | 30/31 | 26/31 | 0 | 4 | -12.9 | -25.8..-3.2 | 0.125 |
| JevK5 v0.3, drop-in | cicada-w1 | rotations | 30/31 | 26/31 | 0 | 4 | -12.9 | -25.8..-3.2 | 0.125 |
| JevK5 v0.3, drop-in | wyrd | single | 71/84 | 69/84 | 2 | 4 | -2.4 | -7.1..+2.4 | 0.6875 |
| JevK5 v0.3, drop-in | wyrd | rotations | 71/84 | 66/84 | 2 | 7 | -6.0 | -11.9..+0.0 | 0.1797 |
| JevK5 v0.3, drop-in | perturbations108 | single | 83/108 | 99/108 | 17 | 1 | +14.8 | +6.5..+24.1 | 0.0001 |
| JevK5 v0.3, drop-in | perturbations108 | rotations | 93/108 | 101/108 | 8 | 0 | +7.4 | +1.9..+13.9 | 0.0078 |
| JevK5 v0.3, drop-in | cicada-w2 | single | 20/31 | 21/31 | 1 | 0 | +3.2 | +0.0..+9.7 | 1.0 |
| JevK5 v0.3, drop-in | cicada-w2 | rotations | 19/31 | 19/31 | 0 | 0 | +0.0 | +0.0..+0.0 | 1.0 |
| JevK5 v0.3, drop-in | pooled | cand-single-vs-semif-rotations | 232/259 | 227/259 | 10 | 15 | -1.9 | -5.2..+1.2 | 0.4244 |
| JevK5 v0.3, drop-in | wyrd | cand-single-vs-semif-rotations | 71/84 | 69/84 | 3 | 5 | -2.4 | -8.3..+3.6 | 0.7266 |
### Noise floors
| system | A-vs-A in process (authored144): flips, max Δp | across restarts, single (560 rows/pair): flips per pair, max Δp | across restarts, rotations: flips per pair | labelled rows whose top moved in ANY restart pair, single: pooled /259 · perturb /108 · Cicada w2 /31 |
|---|---|---|---|---|
| SemIf (Qwen3.5-4B), baseline | 0/144 Δp≤0.000, 0/144 Δp≤0.000, 0/144 Δp≤0.000, 0/144 Δp≤0.000 | 0 (Δp≤0.000), 0 (Δp≤0.000), 0 (Δp≤0.000), 0 (Δp≤0.000), 0 (Δp≤0.000), 0 (Δp≤0.000) | 0, 0, 0, 0, 0, 0 | 0 · 0 · 0 (4 repeats) |
| Plumb-4B, native | 0/144 Δp≤0.000, 0/144 Δp≤0.000, 0/144 Δp≤0.000 | 0 (Δp≤0.000), 2 (Δp≤0.043), 2 (Δp≤0.043) | 0, 2, 2 | 2 · 0 · 0 (3 repeats) |
| Plumb-4B, drop-in | 0/144 Δp≤0.000, 0/144 Δp≤0.000, 0/144 Δp≤0.000 | 0 (Δp≤0.000), 0 (Δp≤0.000), 0 (Δp≤0.000) | 0, 0, 0 | 0 · 0 · 0 (3 repeats) |
| Imajev-4B, native | 0/144 Δp≤0.000, 0/144 Δp≤0.000, 0/144 Δp≤0.000, 0/144 Δp≤0.000 | 0 (Δp≤0.000), 0 (Δp≤0.000), 0 (Δp≤0.000), 0 (Δp≤0.000), 0 (Δp≤0.000), 0 (Δp≤0.000) | 0, 0, 0, 0, 0, 0 | 0 · 0 · 0 (4 repeats) |
| Intern-Decision-4B, native | 0/144 Δp≤0.000, 0/144 Δp≤0.000, 0/144 Δp≤0.000, 0/144 Δp≤0.000 | 0 (Δp≤0.000), 0 (Δp≤0.000), 0 (Δp≤0.000), 0 (Δp≤0.000), 0 (Δp≤0.000), 0 (Δp≤0.000) | 0, 0, 0, 0, 0, 0 | 0 · 0 · 0 (4 repeats) |
| Intern-Decision-4B, drop-in | 0/144 Δp≤0.000, 0/144 Δp≤0.000, 0/144 Δp≤0.000 | 0 (Δp≤0.128), 0 (Δp≤0.128), 0 (Δp≤0.000) | 0, 0, 0 | 0 · 0 · 0 (3 repeats) |
| JevK5 v0.2, native | 0/144 Δp≤0.000, 0/144 Δp≤0.000, 0/144 Δp≤0.000 | 0 (Δp≤0.000), 3 (Δp≤0.036), 3 (Δp≤0.036) | 0, 2, 2 | 1 · 1 · 0 (3 repeats) |
| JevK5 v0.2, drop-in | 0/144 Δp≤0.000, 0/144 Δp≤0.000, 0/144 Δp≤0.000 | 0 (Δp≤0.000), 0 (Δp≤0.000), 0 (Δp≤0.000) | 0, 0, 0 | 0 · 0 · 0 (3 repeats) |
| JevK5 v0.3, native | 0/144 Δp≤0.000, 0/144 Δp≤0.000, 0/144 Δp≤0.000 | 1 (Δp≤0.051), 1 (Δp≤0.051), 0 (Δp≤0.000) | 1, 1, 0 | 0 · 1 · 0 (3 repeats) |
| JevK5 v0.3, drop-in | 0/144 Δp≤0.000, 0/144 Δp≤0.000, 0/144 Δp≤0.000 | 0 (Δp≤0.000), 0 (Δp≤0.000), 0 (Δp≤0.000) | 0, 0, 0 | 0 · 0 · 0 (3 repeats) |
### Order sensitivity (authored144, single ordering vs the same request reordered) and the negative control
| system | reversed: label changes /144 | reversed: max Δp | shuffled: label changes | shuffled: max Δp | NEG: same top as unrotated /144 | NEG: follows the description | NEG: right vs original gold |
|---|---|---|---|---|---|---|---|
| SemIf (Qwen3.5-4B), baseline | 30 | 0.89 | 29 | 0.89 | 14 | 119 | 17 |
| Plumb-4B, native | 15 | 0.54 (0.54–0.56) | 10 | 0.44 (0.44–0.45) | 24 (24–25) | 106 (105–106) | 27 (27–28) |
| Plumb-4B, drop-in | 12 | 0.73 | 8 | 0.67 | 3 | 126 | 12 |
| Imajev-4B, native | 3 | 0.72 | 4 | 0.68 | 4 | 135 | 5 |
| Intern-Decision-4B, native | 8 | 0.60 | 5 | 0.46 | 10 | 122 | 14 |
| Intern-Decision-4B, drop-in | 10 | 0.98 | 12 (11–12) | 0.94 (0.93–0.94) | 5 | 127 | 10 |
| JevK5 v0.2, native | 13 (12–13) | 0.72 | 10 | 0.64 (0.63–0.64) | 20 | 114 | 21 |
| JevK5 v0.2, drop-in | 15 | 0.70 | 12 | 0.70 | 4 | 128 | 9 |
| JevK5 v0.3, native | 8 (7–8) | 0.61 | 11 | 0.49 | 20 | 118 | 18 |
| JevK5 v0.3, drop-in | 4 | 0.53 | 5 | 0.51 | 4 | 133 | 5 |
### Latency at our shape (loopback on fv-ml1, GPU 3; ms; median of all requests, run-median range)
| system | short1 e2e | crit21 e2e | crit21 server | long1 e2e | long16 e2e | long16 server | tokens crit21 / long16 |
|---|---|---|---|---|---|---|---|
| SemIf (Qwen3.5-4B), baseline | 37 (37–38) | 131 (131–132) | 119 (119–120) | 186 (186–187) | 505 (502–519) | 486 (482–500) | 62 / 3900 |
| Plumb-4B, native | 16 | 294 (292–295) | 292 (290–293) | 215 (212–215) | 3378 (3374–3384) | 3376 (3372–3382) | 2489 / 63302 |
| Plumb-4B, drop-in | 37 | 131 (131–132) | 120 (119–120) | 186 | 506 (506–507) | 487 (487–488) | 62 / 3900 |
| Imajev-4B, native | 60 (60–61) | 1219 (1212–1230) | 1212 (1204–1223) | 315 (310–315) | 5125 (5054–5140) | 5121 (5049–5135) | 374 / 7925 |
| Intern-Decision-4B, native | 39 | 88 (88–89) | 84 (84–85) | 190 (189–191) | 215 (213–216) | 213 (211–214) | 1083 / 4579 |
| Intern-Decision-4B, drop-in | 38 (37–38) | 132 | 120 | 186 (186–188) | 508 (500–516) | 488 (481–497) | 62 / 3900 |
| JevK5 v0.2, native | 16 | 291 (291–292) | 289 (289–290) | 212 | 3360 (3358–3368) | 3358 (3356–3366) | 2489 / 63302 |
| JevK5 v0.2, drop-in | 38 | 132 | 120 (119–120) | 187 (185–188) | 518 (505–519) | 499 (486–500) | 62 / 3900 |
| JevK5 v0.3, native | 16 | 291 (291–292) | 289 (289–290) | 212 | 3363 (3355–3370) | 3361 (3353–3368) | 2489 / 63302 |
| JevK5 v0.3, drop-in | 38 (37–39) | 133 (131–133) | 120 (119–121) | 187 (186–188) | 518 (506–521) | 498 (487–501) | 62 / 3900 |
### VRAM (nvidia-smi, whole GPU 3, only our process on it) and capacity under the 12 GiB cap
| system | rest after warm-up MiB | peak in sets+shapes MiB | peak in capacity sweep MiB | max rows @ short state | max rows @ ~3,900 tok |
|---|---|---|---|---|---|
| SemIf (Qwen3.5-4B), baseline | 9242 (9242–9250) | 12918 (12886–12918) | 12920 | 64 (next 96: 422) | 16 (next 18: 503) |
| Plumb-4B, native | 12064 | 12064 | 12064 | 128 (no failure up to the last step) | 64 (no failure up to the last step) |
| Plumb-4B, drop-in | 9242 | 12918 | 12920 | 64 (next 96: 422) | 16 (next 18: 503) |
| Imajev-4B, native | 10404 | 11056 | 11056 | 128 (no failure up to the last step) | 64 (no failure up to the last step) |
| Intern-Decision-4B, native | 9736 | 10290 | 10290 | 128 (no failure up to the last step) | 64 (no failure up to the last step) |
| Intern-Decision-4B, drop-in | 9242 | 12918 | 12920 | 64 (next 96: 422) | 16 (next 18: 503) |
| JevK5 v0.2, native | 12064 | 12064 | 12064 | 128 (no failure up to the last step) | 64 (no failure up to the last step) |
| JevK5 v0.2, drop-in | 9242 | 12918 | – | – | – |
| JevK5 v0.3, native | 12064 | 12064 | – | – | – |
| JevK5 v0.3, drop-in | 9242 | 12918 | – | – | – |
Reference in New Issue View Git Blame Copy Permalink