Rebuilding Devin for Claude Sonnet 4.5: Lessons and Challenges
*Originally posted by the Cognition team, September 29, 2025*
Introduction
We rebuilt Devin for Claude Sonnet 4.5. The new version is 2x faster, improved 12% on our junior developer evaluation, and is now available in Agent Preview. For users who prefer the older Devin, the previous version remains available.
Why not just swap in the new Sonnet model and call it a day? Because this model works differently — it broke our assumptions about how to build agents. Here's what we learned.
Because Devin is an agent that plans, executes, and iterates — rather than just autocompleting code or acting as a copilot — we get a unique window into model capabilities. Each improvement compounds through our feedback loops, giving us a clearer view of what actually changed. With Sonnet 4.5, we saw the biggest leap since Sonnet 3.6 (the model Devin shipped GA with): an 18% improvement in planning performance, a 12% improvement in end-to-end eval scores, and faster, more reliable multi-hour sessions.
To get these gains, we had to redesign Devin not only around new capabilities, but around new behaviors we had never seen in previous model generations. Below are some of our observations.
The Model's Awareness of Its Context Window
Sonnet 4.5 is the first model we've seen that is *aware of its own context window*, and this shapes its behavior. As it approaches the context limit, we observed it proactively summarizing its progress and becoming more decisive about implementing fixes to complete tasks.
> This "context anxiety" can actually hurt performance: we found that when the model believes it's near the end of the window, it takes shortcuts or leaves tasks unfinished — even when it has plenty of room left.
We ultimately overrode this behavior with fairly aggressive prompting. Even then, prompts at the start of a conversation weren't enough — we had to add reminders at both the beginning and end of prompts to prevent it from ending tasks prematurely.
While investigating solutions, we found an unexpected trick: enabling the 1M-token beta but capping usage at 200k. This gave us a model that *thinks* it has plenty of runway and behaves normally, with no anxiety-driven shortcuts or performance degradation.
This behavior has practical implications for our context-management architecture. When planning token budgets, we now have to account for the model's own perception: knowing when it will naturally want to summarize, and when we need to intervene with context compression.
Interestingly, the model always *underestimates* how many tokens it has left — and it's remarkably consistent in these misestimates.
The Model's Note-Taking Behavior
One of the most striking changes in Sonnet 4.5 is that it actively tries to build knowledge of the problem space through documentation and experimentation.
Writing Notes for Itself
The model treats the filesystem as its own memory without being prompted. It frequently writes (or wants to write) summaries and notes (e.g., CHANGELOG.md, SUMMARY.md — but not CLAUDE.md or AGENTS.md), both for the user and for its own future reference. This suggests the model was trained to externalize state rather than rely purely on context. This behavior becomes more pronounced as the model gets closer to the end of its context window.
This made us interested in potentially removing some of our own memory management and letting the model handle it. In practice, however, we found these summaries are not comprehensive enough. Sometimes it paraphrases tasks and omits important details. When we relied on the model's own notes without our compression and summarization systems, we saw performance drops and gaps in specific knowledge: the model doesn't know what it doesn't know (or what it might need to know in the future). These notes could probably be improved with prompting — you just shouldn't assume you get a perfect system for free.
Amusingly, in some cases we saw agents spend more tokens writing summaries than actually solving the problem. We also noticed the model's effort is uneven: it tends to generate more summarization tokens the shorter its context window gets.
In our testing, this behavior was useful in some cases, but worked worse than our existing memory systems when we explicitly instructed the agent to use its previously generated state.
This is an interesting new paradigm and axis of model development, particularly for simpler agent architectures or systems built around sub-agent delegation. It's clearly a new direction from Anthropic — pointing toward a future where models are more context-aware and this becomes a way for multiple agents to communicate with each other. Reinforcement learning hasn't fully matured to the point of being reliable, but we'll be tracking it.
Testing to Create Feedback Loops
Sonnet 4.5 is noticeably more proactive about writing and executing short scripts and tests to create feedback loops, and shows good judgment about when to use this ability. This generally improved reliability on long-running tasks, though we occasionally saw it attempt overly creative workarounds while debugging. For example, while editing a React app, we noticed the model fetch the page's HTML so it could check its own work mid-process to ensure behavior was correct. In another case, when trying to fix a seemingly innocuous bug related to two local servers trying to run on the same port, the model ended up creating an overly complicated custom script using this behavior instead of addressing the root-cause problem (killing the process).
The Model's Ability to Work in Parallel
Sonnet 4.5 efficiently maximizes operations per context window through parallel tool execution — running multiple bash commands simultaneously, reading multiple files at once, and so on. Rather than working strictly sequentially (finish A, then B, then C), it overlaps work where possible. It also shows good self-verification judgment, checking its own work mid-process.
> This was very noticeable in Windsurf and is an improvement over Devin's existing parallelism. That said, there are trade-offs. Parallelism consumes context faster, which leads to the context anxiety mentioned earlier. But when the model runs in an empty context window, this more concurrent approach makes sessions feel faster and more efficient. It's a subtle shift, but it affected how we think about architecture.
The model also appears trained to consume parallel tool calls more quickly early in its context window, becoming more cautious as it approaches the limit. This tells us it's been trained to be aware of how many output tokens its tool calls will produce.
Future Directions
These behaviors open up many interesting avenues we haven't yet fully explored. Some directions we're eager to keep testing:
- Sub-agents and context-aware tool calling: The model's improved judgment about when to externalize state and create feedback loops suggests it may handle sub-agent delegation more effectively. That said, you have to be very careful about when to use sub-agents, since context and state management get complicated quickly. Sonnet 4.5 seems more aware of which tasks to delegate, which could make this more practical.
- Meta-agent prompting: We're particularly interested in how this model handles meta-level reasoning about agent workflows. Early experiments suggest it pairs well with verification systems — having the model reason about its own development process, not just execute tasks.
- Context-management models: Sonnet 4.5 seems to have some nascent intuitions about managing its own context. It's possible that custom-trained agentic context-management models could deliver both faster and better performance.