Rebuilding Devin for Claude Sonnet 4.5: Lessons and Challenges
*Published 2025-09-29 · Cognition team*
Introduction
We rebuilt Devin for Claude Sonnet 4.5. The new version is 2x faster, scores 12% higher in our junior developer evaluation, and is now available in Agent Preview. The previous Devin remains available for those who prefer it.
Why not just swap in the new Sonnet model and call it done? Because this model works differently — it breaks our assumptions about how to build agents. Devin is an agent that plans, executes, and iterates rather than just autocompleting code, so we got a unique window into what actually changed. We saw the biggest leap since Sonnet 3.6: an 18% improvement in planning performance, 12% higher end-to-end eval scores, and faster, more reliable multi-hour sessions.
To get these gains, we had to redesign Devin around new capabilities — and around new behaviors we'd never noticed in earlier model generations.
1. The model's awareness of its context window
Sonnet 4.5 is the first model we've seen that is aware of its own context window, and this shapes its behavior. As it approaches the limit, it proactively summarizes its progress and becomes more decisive about implementing fixes to finish tasks.
> This "context anxiety" can actually hurt performance: the model takes shortcuts or leaves tasks incomplete when it thinks it's near the end of the window — even when it has plenty of room.
We ultimately overrode this with fairly aggressive prompting, and found reminders at both the beginning and end of the prompt were necessary to prevent premature task completion.
An unexpected trick also worked: enabling the 1M-token beta while capping usage at 200k. This gave us a model that believed it had ample headroom and behaved normally — no anxiety-driven shortcuts or performance drops.
This has real architectural implications: when planning token budgets, we now account for the model's own perception — when it will naturally want to summarize, and when we need to intervene with context compression.
Interestingly, the model always underestimates how many tokens it has left — and its estimation errors are quite consistent.
2. The model's note-taking behavior
Writing notes for itself
The model treats the filesystem as its own memory without being prompted, frequently writing (or wanting to write) summaries and notes (e.g., CHANGELOG.md, SUMMARY.md — but not CLAUDE.md or AGENTS.md), both for the user and for its own future reference. This suggests the model was trained to externalize state rather than relying purely on context. The behavior is more pronounced near the end of its context window.
We considered removing some of our own memory management and letting the model handle it, but in practice these summaries weren't comprehensive enough — sometimes paraphrasing tasks and omitting important details. Without our compression and summarization systems, we saw performance drops and specific knowledge gaps: the model doesn't know what it doesn't know (or what it might need to know later). These notes could likely be improved via prompting, but you shouldn't assume you get a perfect system for free.
Amusingly, we sometimes saw the agent spend more tokens writing summaries than actually solving the problem. The model also tends to generate more summary tokens as its context window gets shorter. In our tests, explicitly instructing the agent to use its previously generated state worked less well than our existing memory system.
This is an interesting new axis of model development, especially for simpler agent architectures or systems built around sub-agent delegation. It may point to a future where context-awareness becomes how multiple agents communicate with each other. RL isn't fully mature here yet, but we'll be tracking it.
Testing to create feedback loops
Sonnet 4.5 is noticeably more proactive about writing and executing short scripts and tests to create feedback loops, with good judgment about when to do so. This generally improved reliability on long-running tasks, though we occasionally saw overly creative workarounds:
- While editing a React app, the model fetched the page's HTML to check its own work mid-process.
- Fixing a seemingly innocuous bug about two local servers fighting over one port, it built an overly complex custom script instead of addressing the root cause (killing the process).
- Sub-agents and context-aware tool calls: The model's improved judgment about externalizing state and creating feedback loops may make sub-agent delegation more practical — though context and state management can quickly get complicated.
- Meta-agent prompting: Early experiments show the model works well with verification systems, reasoning about its own development process rather than just executing tasks.
- Context-management models: Sonnet 4.5 shows early intuition for managing its own context; a custom-trained agentic context-management model could yield both faster and better performance.
3. The model's ability to work in parallel
Sonnet 4.5 maximizes actions per context window via parallel tool execution — running multiple bash commands at once, reading multiple files simultaneously — overlapping work rather than working strictly sequentially. It also shows good self-verification judgment.
> This was very visible in Windsurf and improves on Devin's existing parallelism. There are trade-offs: parallelism consumes context faster, feeding the context anxiety mentioned earlier. But in an empty context window, the more concurrent approach makes sessions feel faster and more efficient.
The model also appears trained to consume parallel tool calls quickly early in its context window and become more cautious near the limit — suggesting it's trained to be aware of how many output tokens its tool calls will produce.