Cursor: Don't Switch Models Mid-Task — Deep Dive into 'Continually Improving Our Agent Harness' (Part 2)

In April 2026, Cursor published an engineering blog post, *Continually Improving Our Agent Harness*. While the first half covers context window evolution and evaluation systems, the most overlooked yet critical conclusion sits in the latter half: don't switch models in the middle of a task. This is not a preference — it's the convergence of real engineering problems.

Key points

1. Methodology: vision → hypothesis → experiment → iteration

• Cursor treats harness building like any ambitious software product: form a vision of the ideal agent experience, derive hypotheses, iterate via quantitative and qualitative signals, and build online/offline observability.
• Onboarding a new model takes weeks of tailoring prompts and tool interfaces to the model's strengths and quirks. Model capability is the base score; harness design is the multiplier — the same Claude Opus 4.6 can vary by ~15 SWE-bench points across harnesses.

2. Context evolution: from packed guardrails to on-demand retrieval

• In late 2024, weak models required heavy guardrails: post-edit lint/type errors, rewriting undersized file reads, tool-call limits, and lots of static context (repo layout, semantic snippets).
• Today most of these are gone. Static context is limited to essentials (OS, git state, current/recent files); everything else is fetched dynamically. Scaffolding built to compensate for weak models must be dismantled as models improve — otherwise it becomes a liability.

3. Evaluation: offline + online

• Offline — CursorBench: fast, standardized, comparable over time, but only approximates real usage.
• Online — A/B tests measuring:
• *Direct metrics*: latency, token efficiency, tool-call counts, cache hit rates.
• *Quality metrics*: Keep Rate (fraction of agent changes still in the codebase after a fixed interval) and an LLM Judge reading user reactions (starting the next feature = strong positive; pasting a stack trace = strong negative).
• A harness variant with a higher benchmark score won't be rolled out if Keep Rate drops. Example: summarizing context with a stronger model offered negligible quality gains for the added cost.

4. Tool reliability: expected vs. unknown errors

• Expected errors: InvalidArguments, UnexpectedEnvironment, ProviderError, UserAborted, Timeout. Anything unclassified is treated as a bug.
• Failed tool calls linger in context, waste tokens, and cause context corruption that degrades subsequent decisions — sometimes stalling or derailing the agent entirely.

5. Per-model harness customization

• Different model families prefer different file-edit formats: OpenAI models favor patch/diff-based edits; Anthropic models favor string replacement. Cursor customizes tool formats and prompts per model, even across versions of the same model.
• New-model onboarding: start from the closest existing harness → offline evals → internal dogfooding → prompt tuning → rollout. Benchmark scores are only a starting point; your own eval suite answers the real question.

6. Why mid-task model switching is risky — three traps

1. Out-of-distribution history: the new model inherits a conversation written by another model — unfamiliar tool-call formats, reasoning style, error recovery — raising error rates. Mitigation: custom instructions telling the model it's taking over mid-conversation. 2. Tool-set mismatch: history may contain tool calls the new model doesn't support. Mitigation: instructions to ignore incompatible tools. 3. Cache invalidation: caches are provider/model-specific, so switching means slower, costlier first responses. Summarizing the conversation helps, but summaries can lose crucial details — fatal for a 20-file, multi-iteration refactor.

7. The better alternative: subagents

• Subagents start from a fresh context window with no inherited baggage. Cursor now lets users request a subagent running a specific model: the main conversation keeps one model for coherence, model-specific tasks run in isolated subagents, and only results — not internal reasoning — flow back into the main context.

8. Practical advice

• Cursor users: keep one model per complex task; use subagents when you need another model; watch your own "Keep Rate" — if you always hand-fix generated code, the model/harness combo isn't optimal.
• Agent builders: support switching technically but steer users to subagents or new sessions; customize tool schemas per model; adopt Keep Rate as a north-star metric; regularly audit and remove compensatory guardrails built for older models.

9. Conclusion

1. Model capability is rapidly commoditizing; competitive edge is shifting from "use the strongest model" to "use the same model best." 2. Multi-model support is user choice, not architectural optimum — isolate switches in subagents rather than mid-conversation. 3. Evaluation infrastructure matters more than any specific trick: without CursorBench + Keep Rate + LLM Judge, harness improvements are just intuition.

References:

• Cursor blog — https://cursor.com/blog/continually-improving-agent-harness
• yage.ai analysis — https://yage.ai/share/cursor-agent-harness-evaluation-first-20260501.html
• juejin.cn Cursor SDK analysis — https://juejin.cn/post/7635874903253991439
• atou.cc harness breakdown — https://www.atou.cc/articles/continually-improving-our-agent-harness/