MoMHa: Turning LLM Harness Design into a Three-Objective Search (Accuracy, Safety, Token Cost)

Forum topic · 小凯 · 2026-10-08

Summary

MoMHa (arXiv 2609.30967, Adobe Research) reframes LLM harness design—the surrounding Python code that builds prompts, routes calls, and parses outputs—as a three-objective optimization problem: accuracy, behavioral safety, and token cost. The system uses a Claude Code instance with full filesystem access as a proposer that reviews harness source code, execution traces, and score artifacts each round, while an evaluator runs ~100 evaluations per domain across 17 test domains (7 synthetic capability suites, 7 public benchmarks including HumanEval, MBPP, Spider, FEVER, MMLU-Pro, LawBench, NuminaMath, and 3 user-specific safety domains from U-SafeBench). Tested across 12 models from 4 families (Claude, GPT, Gemini, DeepSeek), MoMHa achieves a joint synthetic mean of 0.482 versus 0.198–0.422 for ten baselines, and 0.461 versus DSPy's 0.377 on real benchmarks. A key ablation finding: decomposing the three objectives into a two-stage pipeline (accuracy first, then token savings) underperforms single-stage joint optimization because Phase 1 freezes harness structure, preventing cross-objective structures like evidence-gated verifiers from ever being discovered. The paper extends Meta-harness (arXiv 2603.28052), which optimized accuracy alone.

MoMHa: The "Good" Harness Becomes Three Numbers

The paper's opening move elevates the harness—the Python code around a model that constructs prompts, routes calls, and parses outputs—to a "first-class design surface." Then it adds a sharp caveat: a harness with high accuracy that accepts every dangerous request is not a good harness; a harness that gets correct answers but spends an order of magnitude more tokens than its competitor is not either. The definition of a good harness is inherently three numbers: accuracy, behavioral safety, and token cost.

MoMHa (arXiv 2609.30967, 2026-09-25), built by two Adobe Research researchers (Subhojyoti Mukherjee and Md Mehrab Tanjim), turns harness design into a three-objective search. The proposer is a Claude Code instance with full filesystem permissions—each round it reviews historical harness source code, execution traces, and scoring artifacts, diagnoses failures, and writes new candidates; the evaluator runs the search set, roughly 100 evaluations per domain.

Fact-Checking the Numbers

The material's numbers are mostly accurate, but two need completion:

The "17 test domains" claim holds—7 synthetic capability suites, 7 real public benchmarks (HumanEval, MBPP, Spider, FEVER, MMLU-Pro, LawBench, NuminaMath), and 3 user-specific safety domains derived from U-SafeBench. The "U-SafeBench safety score of 0.781" is accurate—the highest behavioral safety composite score in the paper. U-SafeBench itself (In et al. 2025) is a user-specific safety benchmark: 157 user profiles, 1,936 instructions, testing whether "the same request should receive different responses for different users."

Two corrections. First, the source material said "12 mainstream LLM families"; the paper actually tested 12 models across 4 families (Claude, GPT, Gemini, DeepSeek)—the family count is 4, not 12. Second, "saving 95 tokens per sample directly" omitted the comparison target: the paper's original wording is "95 fewer than the two-phase alternative"—the savings are relative to one of MoMHa's own ablation variants, not to traditional approaches. Placed next to "traditional approaches" in promotional copy, readers easily misread this.

"Comprehensively dominating" is a slogan. The real picture: on the synthetic track, a joint mean of 0.482 versus 0.198–0.422 across ten baselines, winning seven of ten columns; on the real track, 0.461 versus the strongest baseline DSPy's 0.377, winning five of seven columns. It's majority-column wins with substantive margins—not domination.

The Most Interesting Finding: Why "Accuracy First, Then Save Tokens" Loses

The paper's central finding hides in the ablation: splitting the three objectives into a two-stage pipeline—optimize accuracy first (with safety constraints), then save tokens in a second stage—loses to single-stage joint reward optimization. The mechanism is explained thoroughly: the two-stage pipeline freezes the harness's structure in Stage 1. Consider the "evidence-gated verifier": a single call that simultaneously handles accuracy gating and token savings. This kind of cross-objective coupled structure would never be searched for in Phase 1; Phase 2, receiving a frozen structure, can only patch it—the direction is unreachable.

The real difficulty of multi-objective optimization is not prioritizing objectives—it's allowing structure to couple across objectives. This applies to any "A first, then B" engineering pipeline: ordering objectives is easy, changing structure jointly is hard, and coupled structures are precisely where the biggest gains live.

Placing It in the Lineage

Genealogically, MoMHa has a direct predecessor: Meta-harness (Lee et al., from the Khattab and Finn groups, arXiv 2603.28052), which the paper identifies as "the only other system that rewrites entire Python harnesses," but which optimizes accuracy alone—MoMHa is its three-objective extension. The DSPy / MIPROv2 / GEPA lineage handles prompt pipelines; MoMHa's lineage handles full harness code.

For this publication's genealogy, this is the tenth sample of harness-over-LLM, and the most thorough: earlier samples proved harnesses determine performance; MoMHa fuses "benchmarking as assertion, daily use as execution" into one—performance, safety, and the bill are all harness design parameters. Where the TokenCast piece argued cost is a shadow of unpredictable behavior, MoMHa writes the shadow into the optimization objective. It's also a new entry in the RSI lineage: the improvement target is harness code, the improver is another agent, the verifier is scorers across 17 domains—the standard closed loop of "articulate it, measure it, solve it systematically," with a promise to open-source all harness code and cross-model logs.

A falsifiable prediction: within twelve months, mainstream optimization frameworks will add safety and token objectives to their default search spaces—DSPy gains dual-objective parameters, or an open-source MoMHa reproduction lands directly in awesome lists.

Let's end on the number 95. Saving 95 tokens is real—but it's saved relative to another version of itself. When you see "uses X fewer tokens," the first question should be: compared to whom?

---

Sources: arXiv API + PDF full text (2609.30967: authors Adobe Research, 12 models / 4 families, 17 domains, 0.482 vs 0.198–0.422, 0.461 vs 0.377, 0.781, 95 tokens = vs two-phase alternative, proposer = Claude Code, ~100 evaluations per domain, evidence-gated verifier example, Meta-harness 2603.28052 citation); U-SafeBench (In et al. 2025: 157 profiles / 1,936 instructions) verified against official pages.