Paper Slam 4/19: When a Web Designer Meets a Radiologist — Two Agents, Two Roads
This post compares two agent papers published around the same time: MM-WebAgent (arXiv 2604.15309, Microsoft Research Asia), a hierarchical multimodal web agent for webpage generation, and RadAgent (arXiv 2604.15231, Imperial College London & Stanford), a tool-using agent for stepwise interpretation of chest CT. Both are agents, but they walk entirely different roads: one toward aesthetics and creativity, the other toward medical precision.
Key points
MM-WebAgent (arXiv 2604.15309) — creative integration
- Problem: naively pasting AIGC content (Midjourney images, generated videos, charts) into webpages produces visually incoherent results — elements clash in style, the way a suit paired with slippers clashes.
- Core design: a three-stage pipeline mirroring how human designers work — global planning (layout, style, content structure expressed as structured JSON), local planning (per-section multimodal asset generation guided by the global blueprint), and integration with a three-level reflection mechanism (Local Refine → Context Refine → Global Refine, the last one screenshotting the rendered page for holistic review).
- Architecture: a central LLM (gpt-4o) orchestrating tools — image generation, video generation, chart generation, HTML/CSS code generation, and browser rendering for visual checks. Crucially, multimodal content generation is a first-class action, not a placeholder, enabling joint optimization across code correctness, layout, and visual coherence.
- Evaluation: the team built MM-WebGEN-Bench (120 curated tasks across four intents, multiple visual styles, and varied layout/multimodal complexity) with a six-dimension protocol: layout correctness, style coherence, aesthetics (global) plus image, video, and chart quality (local).
- Results (from the post's table):
- Limitations: dependence on external AIGC tool quality and stability; a fixed tool set (no dynamic tool discovery); no learning — it is a training-free orchestration agent that cannot accumulate design experience.
- Problem: end-to-end 3D VLMs like CT-CHAT compress the whole CT volume into visual tokens for an LLM. This yields black-box reasoning, severe hallucination, adversarial fragility (macro-F1 of only 0.287 on CT-RATE), and zero faithfulness — no way to trace report claims back to image evidence.
- Core design: instead of generating a report directly, the agent calls interpretable tools step by step — region localization, lesion detection, measurement, slice comparison, report templating — with human-readable inputs and outputs. Every step is recorded in a transparent reasoning trace, followed by a faithfulness verification step that checks whether each report claim has visual evidence.
- Results (on CT-RATE, 50,188 chest CT volumes):
- Limitations: faithfulness still has large headroom; the tool set is chest-CT specific; stepwise reasoning is much slower than end-to-end inference — a real issue in time-critical settings like the emergency room.
- A bad webpage is an aesthetic disaster — redo it. So MM-WebAgent can pursue "overall harmony" through iterative, tolerant reflection.
- A wrong diagnosis can be fatal — no redo. So RadAgent must be step-by-step correct and fully explainable, functioning as an automated, mandatory checklist.
- Naming is not understanding: both papers say "Agent," but one is a creative coordinator and the other a reasoning executor. "Agent" is a spectrum, not a unified concept — and the term is suffering inflation as any LLM program with function calling gets labeled an agent.
- Organization over novelty: neither system invents new AI capabilities. Both automate an existing human workflow (designer's top-down process; radiologist's protocol) by organizing LLM reasoning, generation tools, and verification into structured pipelines — arguably the most underrated skill in current AI engineering.
- Open questions: Is MM-WebAgent's lack of learning a flaw or a feature that preserves diversity (and avoids baking in "AI slop" taste)? Is 37% faithfulness enough? Not as a final diagnosis, but valuable as a triage tool that directs physician attention.
- MM-WebAgent: arXiv 2604.15309, Yan Li et al., Microsoft Research Asia
- RadAgent: arXiv 2604.15231, Mélanie Roschewitz et al., Imperial College London & Stanford
| Paradigm | Average | |---|---| | Code-only One-shot | 0.43 | | Code-only Agent | 0.49 | | MM-WebAgent | 0.75 |
The multimodal agent roughly doubles local multimodal quality and gains 20–30% on global dimensions over a code-only agent — a qualitative leap, not an incremental one.
RadAgent (arXiv 2604.15231) — explainable correctness
| Metric | CT-CHAT | RadAgent | |---|---|---| | macro-F1 | 0.287 | 0.347 | | micro-F1 | 0.312 | 0.366 | | Faithfulness | 0% | 37.0% | | Adversarial robustness | — | +24.7 points |
The recall improvement matters most clinically: fewer missed findings. The 37% faithfulness is a from-zero-to-one breakthrough for verifiability, though 63% of claims remain unverifiable.
The key insight: error cost determines design philosophy
The comparison boils down to one question: what does failure cost?
Neither is "better"; different scenarios demand different philosophies.
Feynman-style reflections
Conclusion
Both systems share the same skeleton — a central LLM brain, specialized tools, and a feedback loop — differing only in what feedback means: "does this look coherent?" versus "is this conclusion supported by evidence?" Future agents will likely be hybrids: creative yet constrained, intelligent yet transparent. As Feynman put it: "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." Neither paper is a revolution; both are solid, honest progress.