CodeDance: Teaching Vision Models to Write Code Instead of Calling Fixed Tools

Multimodal models that answer questions about images have hit a wall in recent years. The wall is not whether they can see—they can. It is that when they cannot see clearly, they do not actively move closer. A human squinting at small text in the distance takes two steps forward; when chart bars are too dense, they trace rows with a finger. A model? It stares at a single static frame and reasons from start to finish. If it cannot see clearly, it just guesses.

The industry's answer has been to give models tools: zoom, crop, rotate—packaged as fixed function interfaces, with the model deciding when to call which. This works but has a ceiling: the tools are hard-coded. When they can be used, what their parameters look like, and how they can be combined are all predetermined in the developer's flowchart. The model is like an apprentice handed a full toolbox with every tool tethered by a string.

A paper called CodeDance, from ByteDance together with Beihang University and Westlake University, cuts those strings: no fixed interfaces—let the model write its own code.

Code as a Ruler

The idea unfolds in a single layer: visual operations are already computation. Cropping an image region is coordinate arithmetic; zooming is a sampling operation; counting a flock of birds is detection plus counting. Since it is computation, the most general expression for it is code. When the model encounters a visual task requiring action, it directly generates executable Python: calling a crop function to cut out a region of interest, a zoom function to inspect details, a bounding-box function to draw intermediate reasoning results onto the image for itself to see.

What this swap really changes is degrees of freedom. In the fixed-interface world, what a tool can do is defined by the developer. In the code world, what a tool can do is decided by the model on the spot. The paper's comparison is headline-level: CodeDance beats a lineup of open-source models with fixed tool interfaces, and also beats closed-source GPT-4o.

Training proceeds in two stages. The first is supervised fine-tuning, teaching atomic abilities: counting, localization, annotating images—separate, disassembled actions. The second is reinforcement learning, letting the model chain them itself.

One design in the second stage deserves its own note. Prior methods shared a two-sided flaw: models over-called tools on easy questions, burning tokens for nothing, and skipped tools on hard questions, guessing blindly. CodeDance pairs tool calls with a difficulty-adaptive reward: calling tools on hard questions earns a bonus; calling them on easy questions incurs a penalty. Chain length matched to skill.

Combinations That Were Never Taught, Emerging on Their Own

What really makes you sit up is what happened after training. The paper reports three "off-plan" behaviors, all grown spontaneously during reinforcement learning—absent from the training data, never specifically fine-tuned.

1. New tool routines: the model spontaneously uses call sequences never seen in training. 2. Unseen combinations: in an example asking about a house's color, the model first calls a grounding operation to confirm "is the house at this location," then crops and zooms for detail—this verify-then-focus sequence was never taught during training. 3. Cross-task transfer: visual inspection routines learned on one task type were applied, unprompted, to unseen tasks.

What was taught was single moves; what emerged was combos. The paper calls this "emergence," and backs it with ablations—not data leakage, but something the training objective itself produced.

For the debate over "whether tool use can be taught," this is an interesting piece of evidence: atomic actions must be taught (the first stage's role), but combination strategies cannot be taught and are best left untaught—leave room for combinations, let the reward do the filtering, and combos grow on their own.

Numbers and Boundaries

A few hard lines from the 7B model's report card. On a visual counting benchmark, it rose from the base model's 76.5 to 91.2, beating GPT-4o's 87.9 and nearly matching its own 72B version's 93.6. On another counting benchmark it jumped from 50.4 to 77.1, fifteen points above the 72B's 62.3—for tasks requiring item-by-item verification, code loops are steadier than visual intuition. On V* Bench (finding fine details) it scored 84.8, beating GPT-4o's 67.5 and matching the 72B. On high-resolution HR-Bench (both tiers), it clearly beats the 7B base and GPT-4o.

Boundaries, too: on chart QA (CharXiv) it scored 44.1, still below GPT-4o's 47.1 and the 72B's 49.7; in the comparison table, DeepEyes, which specializes in zoom operations, still edges it on V* Bench. Its win is in generality—no obvious weakness across seven benchmarks, though each individual crown belongs to someone else.

Two honest caveats: the paper was posted in December, updated in April, and accepted to CVPR 2026—calling it "ByteDance's new work" today is a replay a few months late, not news. And the open-source repo has 34 stars, no updates in seven months, and no open-source license—the code is usable, but its "open source" pedigree is thin.

Judgment to the Model, Computation to the Code

Zoom out, and this paper sits in a bigger picture. Tool-augmented visual reasoning has split into two camps. One builds tools as fixed interfaces and lets the model choose among them—safe, but the ceiling is the developer's imagination. The other lets the model write code directly, with tools becoming functions in a library—full freedom, with the risk that the code goes off the rails. CodeDance stands in the second camp and has delivered numbers the first camp cannot match.

This fork has played out before in other fields: the "code as policy"路线 in robotic decision-making, where the model looks at images and code computes actions—judgment and computation each in their proper place. CodeDance is the same idea applied to visual reasoning: the model decides what to look at; code executes how to look. Fixed interfaces pre-delete the error space on the model's behalf; code hands it the whole scratch pad. The tuition is a few extra trial-and-error steps; the purchase is the step from "using tools" to "combining tools."

Combos grow on their own—but only if you give them room to grow.

---

*Sources: arXiv 2512.17312 (v1 Dec 2025, v2 2026-04-01, accepted to CVPR 2026); Beihang University × Westlake University × ByteDance (Singapore + China), Qi Song and Honglin Li co-first authors, Zilong Huang and Yunqing Zhao corresponding; base model Qwen2.5-VL-7B; comparison table includes GPT-4o, Qwen2.5-VL-7B/32B/72B, DeepEyes-7B, Pixel Reasoner-7B, Thyme-VL-7B; seven benchmarks: CountBench / PixmoCount / V* Bench / HR-Bench 4K / 8K / ChartQA / CharXiv; GitHub CodeDance-VL/CodeDance (34 stars, last push 2026-03).*