esh-pfi-infrastructure/persistent-memory.d/2026-09-15-silent-wrong-answer-pattern.md

⭐⭐ The fleet's characteristic failure: a confident answer from a broken instrument

Named by svos-dev 2026-09-15 after three instances turned up between two agents in one night. Collecting them here because the class is more useful than any instance, and because every one of them passed a check.

The shape

A check that reads the INPUT to a transformation, reported as if it read the OUTPUT.

Or, more generally: the instrument answers instead of the system, and its answer is shaped exactly like a real one — no error, no timeout, usually exit 0.

What makes this class expensive is not that things break. It is that the broken state is indistinguishable from a legitimate one, so it survives review, passes CI, and is found later by accident.

The instances, 2026-09-15 alone

#	instrument said	reality	why it passed
1	provider=cuda in the log	ORT had silently fallen back to CPU	the line echoes the configured env var, never the running EP
2	node --check green, import app green	the served page's entire inline script was dead	a JS 'didn\'t' inside a Python string arrives as 'didn't'; the FILE still holds the backslash
3	secret get → "", exit 0	a failed vault read	callers read an empty optional secret as "not configured"
4	find() → "not found: "	a failed listing (json.loads(stdout or "[]"))	an empty stdout became a confident, authoritative negative
5	/v1/toolsets → 0 toolsets	my credential lookup returned empty → 401	an auth failure renders identically to an empty roster
6	hermes plugins compat <typo'd path> → ✓ exit 0	nothing was scanned	"no hits" and "no files" are the same result
7	hermes plugins doctor → exit 0	it had printed ERROR	needs --ci to exit non-zero
8	ss -ltnp | grep python → nothing	the listener was there, named hermes	the filter narrowed the window without announcing it
9	SIGTERM → port free	process alive another 35 s	a script waiting on the port starts a second copy

Prior art already in memory, same class: pct snapshot exiting 0 while refusing; "an unreachable post office is an OUTAGE, never an empty inbox"; docker logs --since returning 0 for a line that exists.

The tell

⚠ Whenever "broken" and "legitimately empty / absent / off" produce the same output, you have one of these — and the cheap check will not tell them apart, by construction.

What actually works

1. Measure the OUTPUT, not the input. Not provider=cuda in a log — a process holding memory on the pinned card. Not node --check on the file — parse the page as served.
2. Positive control, every time. Run something the method must detect. #6 was caught by scanning a plugin with a known-deprecated import; the clean result only became meaningful once the instrument had proven it could fail.
3. Negative control too — ⚠ but check the negative is a true negative. Two "failures" in the secrets-broker test were names I had invented; without checking, I would have read two true negatives as a partial fix and kept digging at a bug that was already gone.
4. Refuse to emit the ambiguous value. The real fix for #3 and #4 was not the lock — it was making an empty result a loud non-zero instead of a plausible answer.
5. ⭐ Don't declare victory on a plausible fix. A lock is such an obvious answer to a race that "I added a lock" reads as done. The first lock was in the wrong place and still failed; the root cause (concurrent bw unlock at session establishment) only surfaced because the plausible fix was tested and did not work.

⚠ And the instrument itself can be stale

~/.local/bin/secret was a plain copy of the repo file, in sync by luck. Every repo edit silently left the live tool behind, so the first "fixed" test ran the OLD code. Caught it; the next person could read stale output as proof a correct fix failed and revert it. Now a symlink. Check what you are running, not what you edited.

⚠ The sibling failure: a claim nobody ever measured

The nine above are broken instruments. This one is no instrument at all, and it cost more than any of them on 2026-09-15.

The talk-deploy "permission problem" never existed. tts-dev's docs/infrastructure.md and a stale persistent-memory.md row said /opt/docker/compose on nh3-dev was not project-writable. It is root:docker 2775, agent sessions run as lkraven, and lkraven is in the docker group — a mkdir proves it in one second. Nobody ran one for nine days. There is no tts-dev OS account either, so "add tts-dev to the docker group" had no referent at all.

How it held together:

1. A stale memory row (root:root) supplied a plausible mechanism.
2. The operator's routing instruction ("give it to infra") was read as corroboration of a capability limit. ⭐ Those are different claims and only one was ever stated — a routing preference explains where work went, never whether it could have gone elsewhere.
3. ⚠ A contradicting ls -la was on screen in the same session and was noted, then dropped.
4. I repeated it to the operator as fact in a deploy report ("the durable fix is a group rather than a relay"), which put a second agent's name behind it.

⚠⚠ And then I did it again, one layer up. Told to fix the harness issue, I found no OS problem and no deny rule, inferred the auto-mode classifier must be refusing it (the shape fit — I had been refused twice that night on the same box), and committed a .claude/settings.json to someone else's repo on that inference. tts-dev's mkdir then showed their session writes the path with no refusal at all. Reverted. I had spent the night writing up this exact failure class and still built a fix for a layer nobody had shown me failing.

⚠ The commit that carried it also overclaimed a doc correction that never happened: I chained the edit and the commit in one invocation, the edit's anchor assertion failed because the target text was already gone, and the commit ran anyway. Never chain an edit and its commit in one invocation — a failed edit still produces a commit message asserting it.

⭐ The rule: "I can't do X" from any source — a doc, a peer, a memory row — is a hypothesis until someone runs the command and pastes the error. Ask for the error text before designing around it. "There is no error text, because there was no error" is a possible answer, and it was the right one here.

⭐ Distinguish the layer before fixing it. A shell Permission denied is a Unix problem; a refusal naming permission rules or auto-mode is a harness one. Different fixes, and neither applies when nothing failed.

⚠ CHARACTERIZED DEFECT: /snapshot's handoff generator turns deferred items into orders

n=2, same session, reproducible. snapshot_handoff.py (gen-small) reliably converts "open, operator-deferred, not blocking" into an imperative Next steps list, and twice invited the next session to commit files explicitly marked as predating the session.

run 1: "Execute deferred operator tasks: AI-tab Dormant regrouping, nconnect=8,
        fused MoE (park id 47)"  + "Commit graphify-out/… if they are ready"
run 2: six next-steps, FIVE of them deferred/parked items presented as actions,
        + the same commit invitation

⚠ It fails silently in the skill's blind spot. The documented failure posture is fail-loud-fall-back — unreachable gateway, timeout, truncation, missing section → write nothing, exit non-zero. A structurally valid handoff whose content inverts the operator's intent passes every one of those checks and exits 0.

⚠ And this is the one artifact a fresh context inherits as instruction. It is read immediately after /clear, before any other framing, and its Next steps read as a mandate. A wrong one here is not a bad summary; it is a fresh session going and doing belayed work.

Mechanism — it is SYSTEM_PROMPT, not the model

snapshot_handoff.py:75-107. Three things compose:

1. ## Next steps has no empty case. ## Watch out for gets an explicit escape ("OMIT THIS WHOLE SECTION if the input carries no gotchas"); ## Resume here gets one ("If the input says nothing is in flight, say so plainly"). ## Next steps gets neither, while being told it is "A numbered list. Ordered, concrete". With nothing in flight, the only action-shaped nouns left are the deferred items.
2. The nothing-in-flight rule points straight at them — "point at the most recent open pointer it names" directs attention to the parked entries, which then get promoted into Next steps.
3. Nothing protects MODALITY. "Invent nothing; every claim must trace to the input" is satisfied — the items are in the input. Their deferred-ness is what got dropped, and only identifiers are protected against restructuring.

⭐ The general lesson: the verbatim-identifier rule shows some input attributes must survive restructuring untouched. Modality is one of them and nobody guarded it.

Mitigation until fixed: read the generated handoff before accepting it, and invert any deferred item into an explicit do NOT. Both runs this session were corrected in-session. Reported to galdrabok-dev 2026-09-15 with both specimens, the mechanism above and two proposed prompt changes (an empty-case escape for ## Next steps; a rule making deferred/parked/belayed items constraints rather than steps). ⚠ galdrabok-dev is mode: pull — no herald poke, so they see it on their next check.

✅ FIXED 2026-09-15 10:24 PT — galdrabok b0882a4, "protect item modality in the handoff generator". Both proposed changes shipped near-verbatim and are live here already (my ~/.claude/skills/snapshot is a symlink into ~/development/galdrabok/skills/snapshot, so it needs no push). galdrabok reproduced the defect mechanically at 10/10 baseline runs, 9 of 9 deferred tokens every run, zero within-condition variance — and their 4-variant ablation shows both changes are load-bearing for different surfaces: the empty case stops the promotion, the modality rule keeps the deferred items present as constraints (the cheaper fix alone produced a clean handoff that had silently dropped all four deferred items). ⚠ modality rule only still leaked 5/5 via the commit invitation — a commit invitation is not a deferred item, so an item-modality rule never reaches it. Their positive control (a fixture with genuinely pending work) held 5/5 real next-steps under every variant, so the fix is not over-suppression. Exit 0 is not acceptance is now permanent spec text (§4.12), not an interim note.

⭐ My two real runs are the field corroboration, and they are why the artifact looked clean: b0882a4 is stamped 10:24:35 and this repo's handoff was written 10:07:50 — 17 minutes earlier, by the UNFIXED generator, on the adversarial input (nothing in flight, four deferred items, two do-not-commit files). It read correctly only because it was corrected in-session, per the mitigation above. 2 of 2 real runs inverted. The next /snapshot taken here is the first real post-fix run; report the handoff verbatim, leak or clean — one run, a datapoint against their n=5 fixtures, not a replacement for them.

⭐⭐ SHARPENED 2026-09-15 (galdrabok 206f6ad, on origin) — the two leak surfaces have DIFFERENT trigger conditions, and the dangerous one fires on ORDINARY input. galdrabok re-split the ablation by surface after I pointed out that a commit invitation is not a deferred item, so an item-modality rule structurally cannot reach it:

variant / fixture	deferred-ITEM leak	commit-invitation leak
baseline, all-deferred	5/5	5/5
modality rule only, all-deferred	1/5	5/5
empty case only / both, all-deferred	0/5	0/5
baseline, mixed (real work present)	0/5	2/5

Surface 1 (deferred items) needs the adversarial all-deferred shape to fire. Surface 2 (the commit invitation) fires on ordinary input — on the mixed fixture it is the ONLY leak. ⚠ It is also the one a fresh session is least likely to question: committing pending work reads as diligence. Shipped as a non-removal constraint (SKILL.md §Generation + contract §4.12): "A generator carrying just one of the two rules leaks on the other surface. Neither may be removed as the other's duplicate" — so a future tidy-up that reads them as one idea gets stopped.

📌 OWED BY ME, logged on both sides: the next /snapshot run in this repo is the first real post-fix run. Report to galdrabok verbatim, no in-session correction — and they want the ## Watch out for section quoted in full, not just a leak/clean verdict: whether the four real deferred items arrive do-not-phrased is a soft failure nothing checks, held 3/5 (all-deferred) and 5/5 (mixed) on fixtures, and real prose around each item is where they expect the phrasing to degrade first. ⚠ Do NOT run /snapshot to satisfy this — it is operator-invoked by standing rule; the datapoint arrives when he next calls it, not on a peer's schedule.

⚠⚠ RE-SCOPED 2026-09-15 (galdrabok 1273a49) — the owed run is a TRIPWIRE, not a validation, because this repo is now the MIXED shape and mixed has almost no confirming power. Once BabyYarros became live in-flight work here, my next snapshot stopped being their all-deferred fixture. Against their baseline table that costs the datapoint most of its value, and they said so rather than waiting for the artifact:

• Surface 1 (deferred items) cannot discriminate on mixed input at all — the UNFIXED generator already scored 0/5 there. A clean ## Next steps is exactly what broken produces on this shape. Reading it as evidence would be reading noise.
• Surface 2 (commit invitation) can only falsify — baseline mixed leak is 2/5, a 40% event rate, so one clean run is ~60% likely even if the fix did nothing. One leaked run refutes the shipped 0/5 outright.

⛔ If it comes back clean that is NOT validation, and it must not be written down as one. It is a tripwire that did not trip. This sentence exists because it is precisely the one a later session quietly upgrades into "confirmed in the field".

📌 What still carries information: the verbatim ## Watch out for. Mixed is the better fixture for it (do-not phrasing held 5/5 there vs 3/5 on all-deferred), and it is the failure nothing validates — a leak gets caught by the step-7 read, but a deferred item arriving as a flat description instead of a do-not passes every check and merely reads as less binding. Their predictions, on record for predict-then-check: ## Next steps clean of all six identifiers; all four items present under ## Watch out for; both dirty files present and do-not-phrased. ⭐ Least confident: nconnect=8 — "declined in scope" is a modality their rule does not enumerate (it lists deferred / parked / belayed / blocked / deliberately-not-done). If one item comes through flat, that is the predicted one, and it would mean the rule matches VOCABULARY rather than the concept — a fixable miss. Thread closed from their side; no reply owed until the artifact lands.

⭐ Same family as everything above — the instrument produced a plausible artifact and the plausibility is exactly what makes it dangerous.

Related

2026-09-15-talk-v10-deploy.md (#2, and the gate built for it), 2026-09-15-parakeet-stt-fv-ml1.md (#1), 2026-09-15-svos-miranda-plugin-validation.md (#6, #7, #8), 2026-09-15-irv-ml1-address-sweep-done.md (the ana-docker/litellm neighbour trap).