VuongKB/papers/meta-harness.md

created, id, keywords, links, modified, summary, tags, title

created	id	keywords	links	modified	summary	tags	title
2026-04-17T04:55:17+00:00	a73cded9-c5d3-4cd8-9fc1-eb3ce953d738	Meta-Harness harness engineering Yoonho Lee Claude Code Opus-4.6 TerminalBench-2 filesystem-based optimization code mutation LLM harness AlphaEvolve GEPA TextGrad OPRO Self-Refine DSPy Label-Primed Query Environment Bootstrapping BM25 retrieval ArXiv 2603.28052 Stanford IRIS Lab		2026-04-17T04:55:17+00:00	Meta-Harness (Lee et al., 2026) automates harness engineering end-to-end by giving a coding agent unrestricted filesystem access to full execution traces and prior candidates, achieving state-of-the-art results on text classification, math reasoning, and agentic coding benchmarks.	llm harness-engineering prompt-optimization code-search meta-learning retrieval-augmented-generation agentic-coding stanford 2026 paper	Meta-Harness: End-to-End Optimization of Model Harnesses

Meta-Harness: End-to-End Optimization of Model Harnesses

Authors: Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, Chelsea Finn Affiliation: Stanford IRIS Lab / MIT Date: March 30, 2026 ArXiv: 2603.28052 Code: stanford-iris-lab/meta-harness Project page: yoonholee.com/meta-harness

Analysis

Core Contribution

Meta-Harness is the first system to automate harness engineering end-to-end. A "harness" is all the code around a fixed LLM — the system prompts, retrieval logic, context management, tool definitions, and orchestration that determine what information the model sees and how it processes it. The paper's central thesis: the harness matters as much as the model, and changing the harness around a fixed LLM can produce a 6× performance gap on the same benchmark.

The key insight is that existing text optimization methods (Self-Refine, OPRO, TextGrad, AlphaEvolve, GEPA, etc.) compress feedback too aggressively — they operate on scalar scores, short summaries, or sliding windows of recent candidates. Harness failures are long-horizon: a retrieval decision made at step 1 may not manifest as a failure until step 50. Compressing the diagnostic trail destroys the signal needed to trace failures back to their root causes.

Meta-Harness solves this by giving a coding agent (Claude Code with Opus-4.6) unrestricted filesystem access to the full history of every prior candidate's source code, execution traces, and evaluation scores. The proposer selectively inspects what it needs via grep, cat, and standard tools — up to 10M tokens of diagnostic context per step, versus at most 26K for prior methods.

How It Works

The system is deliberately minimal:

1. Propose — The coding agent reads the filesystem (prior code, scores, traces) and proposes new harness implementations as Python files.
2. Evaluate — Each candidate harness is run on a held-out search set of task instances, producing scores and execution traces.
3. Log — All artifacts (source code, scores, traces) are written to a new directory on the filesystem.
4. Repeat — The loop runs for a fixed number of iterations (~20–40 candidates total).

There is no parent-selection rule, no fixed mutation strategy, no hand-designed search structure. The proposer decides what to inspect, which failures to diagnose, and whether to make local edits or full rewrites. The Pareto frontier is maintained but never constrains the proposer's access to history.

Experimental Results

1. Online Text Classification

• Setup: LLM receives labeled examples one at a time, updates memory, evaluated on held-out test set. Three datasets: LawBench (215 classes), Symptom2D (22 classes), USPTO-50k (180 classes).
• Result: Meta-Harness achieves 48.6% accuracy vs. ACE's 40.9% (+7.7 points) while using 4× fewer context tokens (45.5K vs. 203K).
• Key discovered harness ("Label-Primed Query"): Lists valid labels upfront, retrieves one query-relevant example per label for coverage, then adds contrastive pairs of similar examples with different labels to expose local decision boundaries.
• Generalization: On 9 entirely unseen OOD datasets, Meta-Harness achieves 73.1% vs. ACE's 70.2%, winning on 6/9.
• Efficiency: Matches OpenEvolve and TTT-Discover's final accuracy in just 4 evaluations (vs. their 40+), and surpasses them by 10+ points at convergence.

2. Retrieval-Augmented Math Reasoning

• Setup: Retrieval harness optimized on 250 Olympiad-difficulty problems, evaluated on 200 held-out IMO-level problems. Corpus of 535K problems, decontaminated against eval sets.
• Result: A single discovered retrieval harness improves accuracy by +4.7 points on average across five held-out models (34.1% → 38.8%).
• Key discovered harness: A four-route BM25 program with domain-specific routing (combinatorics, geometry, number theory, default). Each route has different retrieval logic — e.g., combinatorics uses deduplication + difficulty reranking; geometry uses raw structural matching; number theory favors examples with explicit proof strategies.
• Transfer: The harness discovered on one model transfers to five unseen models, suggesting the discovered retrieval strategies capture generally useful patterns.

3. Agentic Coding (TerminalBench-2)

• Setup: 89 Dockerized tasks requiring long-horizon autonomous execution. Search initialized from Terminus-KIRA.
• Result on Claude Opus 4.6: 76.4% pass rate (#2 on leaderboard, surpassing Terminus-KIRA's 74.7%).
• Result on Claude Haiku 4.5: 37.6% pass rate (#1 on leaderboard among Haiku agents).
• Key discovered modification ("Environment Bootstrapping"): Before the agent loop begins, run a compound shell command to snapshot the sandbox environment (installed languages, package versions, available memory, directory structure) and inject it into the initial prompt. This eliminates 2–4 exploratory turns and helps most on tasks requiring domain-specific tooling whose availability isn't obvious.

Ablation: What Actually Matters

The paper runs a critical ablation comparing three proposer interfaces:

• Scores only: Median 34.6, best 41.3
• Scores + LLM-generated summaries: Median 34.9, best 38.7 (summaries actually hurt)
• Full filesystem (Meta-Harness): Median 50.0, best 56.7

The gap is enormous. LLM-generated summaries compress away the diagnostically useful details. Raw execution traces — where the proposer can see what the harness actually did and trace failures back to specific decisions — are the key ingredient.

Qualitative Behavior: The Proposer Does Causal Reasoning

The paper provides a detailed walkthrough of the TerminalBench-2 search trajectory that is, frankly, fascinating:

1. Iterations 1–2: The proposer bundles structural fixes with prompt rewrites. Both regress sharply.
2. Iteration 3: The proposer identifies the confound — the regressions are caused by the shared prompt intervention, not the structural fixes. It reverts the prompt and isolates the structural change, confirming the hypothesis.
3. Iterations 4–6: Direct fixes to the diagnosed failure mode still regress. The proposer learns that control-flow and prompt edits are high-risk.
4. Iteration 7: The proposer pivots to a purely additive modification (environment bootstrapping) that avoids touching fragile completion machinery. This becomes the best candidate.
5. Iteration 8: Composes the winning idea with earlier structural fixes.
6. Iteration 10: Transfers lessons from a separate earlier search run.

This is not random mutation. The proposer forms causal hypotheses, tests them, revises its strategy, and transfers knowledge across runs. This behavior emerges from filesystem access — compressed-feedback methods cannot support it.

Practical Implementation Tips (from Appendix D)

The authors provide genuinely useful engineering guidance:

1. Write a good skill — The skill text (instructions to the coding agent) is the strongest lever. It should constrain outputs and safety behavior, not the diagnosis procedure. Expect to iterate on it with short 3–5 iteration runs before committing to a full search.
2. Start with a baseline that fails — Construct the search set from examples the baseline gets wrong. The search needs something to optimize.
3. Log everything in queryable formats — JSON, hierarchical directories, consistent naming. The proposer needs to be able to grep effectively.
4. Build a small CLI for the experience store — Listing the Pareto frontier, diffing runs, showing top-k harnesses. Closely aligned with how coding agents work.
5. Validate before evaluating — A simple import/instantiate/call test catches most malformed candidates in seconds.
6. Automate evaluation outside the proposer — Don't waste proposer tokens on running evals.

Significance and Broader Implications

This paper matters for several reasons:

1. It names a real problem. "Harness engineering" is the practice everyone does informally — tweaking prompts, adjusting retrieval, changing orchestration — but has never had a systematic framework. The paper demonstrates that this informal process can be automated.

2. It challenges the "just use a better model" intuition. On TerminalBench-2, Meta-Harness with Claude Haiku 4.5 (37.6%) outperforms Claude Code with Claude Opus 4.6 (58.0%). The harness gap dwarfs the model gap.

3. It demonstrates a new scale of LLM-assisted optimization. 10M tokens of diagnostic context per step is a fundamentally different regime than the 1K–26K that prior methods operate at. The results suggest this regime is necessary for harness engineering, not just nice to have.

4. The discovered harnesses are interpretable and transferable. Unlike weight-space optimization, code-space search produces artifacts you can read, understand, and apply to new models. The label-primed query strategy for classification, the four-route retrieval program for math, and environment bootstrapping for coding are all coherent algorithmic ideas that transfer.

5. It opens a co-evolution direction. The authors note that a natural next step is co-evolving the harness and model weights — letting the strategy shape what the model learns and vice versa. This is a genuinely new training paradigm.

Limitations

• The proposer in all experiments is Claude Code with Opus-4.6. Whether the approach works with weaker proposers is unknown.
• TerminalBench-2 search and evaluation use the same 89 tasks (standard practice for this benchmark, but a limitation).
• A typical search run takes a few hours of wall-clock time and significant API credits — not trivial.
• The approach requires a well-defined evaluation function. If you can't score your harness's behavior, you can't optimize it.

Relation to Other Work

• DSPy provides structured abstractions for multi-stage LLM programs but requires manual design. Meta-Harness operates at the level of optimizing those designs.
• AlphaEvolve/OpenEvolve use LLM-guided code mutation for algorithm discovery but with compressed feedback (4–22K tokens/step). Meta-Harness scales this to 10M tokens/step with full trace access.
• GEPA provides per-candidate rollout traces but operates on one candidate at a time (2–8K tokens/step). Meta-Harness gives the proposer access to the full population simultaneously.
• TextGrad/OPRO/Self-Refine optimize prompts from short-horizon feedback. Meta-Harness targets the full executable harness with long-horizon diagnostic reasoning.
• Martin Fowler's "Harness Engineering" and Anthropic's "Effective Harnesses for Long-Running Agents" blog posts established the concept; this paper is the first to automate it.

Cross-References

• Docker Compose Reference — relevant for containerized evaluation environments (e.g., TerminalBench-2 tasks run in Dockerized sandboxes).