Paper Slam 4/19: When a Web Designer Meets a Radiologist — Two Agents, Two Roads

Forum topic · 小凯 · 2026-04-28

Summary

This zhichai.net Paper Slam post compares two AI agent papers: MM-WebAgent (arXiv 2604.15309), a hierarchical multimodal web agent from Microsoft Research Asia for generating complete multimedia webpages, and RadAgent (arXiv 2604.15231), a tool-using AI agent from Imperial College London and Stanford for stepwise interpretation of chest CT scans. MM-WebAgent uses global-to-local planning plus a three-level reflection loop (local, context, global refine) and treats AIGC content generation (images, video, charts) as a first-class action, scoring 0.75 average on its self-built MM-WebGEN-Bench versus 0.49 for code-only agents. RadAgent replaces black-box 3D VLM report generation with explicit stepwise tool calls and faithfulness verification, improving macro-F1 on CT-RATE from 0.287 to 0.347, raising adversarial robustness by 24.7 points, and achieving 37.0% faithfulness versus 0% for CT-CHAT. The author argues both papers represent opposite extremes of agent design — creative integration versus verifiable correctness — driven by the cost of errors in their domains, and closes with a Feynman-style reflection on the inflation of the term 'Agent' and the trade-off between explainability and speed.

Overview

This Paper Slam post compares two AI agent papers with fundamentally different design philosophies:

MM-WebAgent (arXiv 2604.15309, Microsoft Research Asia) — a hierarchical multimodal web agent for webpage generation. Core challenge: creative integration and visual consistency.
RadAgent (arXiv 2604.15231, Imperial College London & Stanford) — a tool-using AI agent for stepwise interpretation of chest CT. Core challenge: explainable correctness.

Key points: MM-WebAgent (arXiv 2604.15309)

Problem. Naively assembling AIGC content (Midjourney-style images, generated video, charts) into a webpage produces visually incoherent results — each element may be fine alone, but together they clash.

Approach. Mimic a human designer's top-down workflow:

Global planning: a structured JSON design blueprint (layout, style, color scheme, section structure).
Local planning: per-section multimodal content generation, always checked against the global blueprint, with concrete AIGC call instructions (e.g., image prompts, sizes, style-match constraints).
Three-level reflection: Local Refine (element quality), Context Refine (HTML/CSS integration and alignment), Global Refine (rendered-screenshot review of overall balance and coherence).

The central LLM (gpt-4o) orchestrates tools for image generation, video generation, chart generation, HTML/CSS code generation, and browser rendering for visual checks. Crucially, multimodal generation is a first-class action — not placeholder-based code generation.

Evaluation. The team built MM-WebGEN-Bench: 120 web design tasks spanning four intents (information display, data analysis, creative expression, commercial promotion), multiple visual styles, layout complexities, and multimodal combinations, scored across six dimensions (layout correctness, style coherence, aesthetics, image/video/chart quality).

Results (averages):

| Paradigm | Average | |---|---| | Code-only One-shot | 0.43 | | Code-only Agent | 0.49 | | MM-WebAgent | 0.75 |

The multimodal agent roughly doubles local multimodal quality scores — the key differentiator is generating real content rather than placeholders.

Limitations: dependence on external AIGC tool quality, a fixed toolset (no dynamic tool discovery), and no learning (training-free prompt engineering; no memory of past design experience).

Key points: RadAgent (arXiv 2604.15231)

Problem. End-to-end 3D VLMs for CT report generation (e.g., CT-CHAT) suffer from black-box reasoning, hallucination, adversarial fragility (macro-F1 0.287 on CT-RATE), and zero faithfulness — report claims cannot be traced to image evidence.

Approach. Stepwise, tool-based reasoning with human-readable traces:

1. Visual understanding of the 3D CT volume. 2. Tool calls: anatomy localization, lesion detection, measurement (size, HU density), slice comparison, report templating — all with interpretable inputs/outputs. 3. Stepwise reasoning: each step builds on prior results (overall structure → per-region checks → confirm findings → structured report), recorded in a transparent trace. 4. Faithfulness verification: every report assertion is matched against visual evidence, yielding a faithfulness score.

Results on CT-RATE (50,188 chest CT volumes):

| Metric | CT-CHAT | RadAgent | |---|---|---| | macro-F1 | 0.287 | 0.347 (+6.0) | | micro-F1 | 0.312 | 0.366 | | Adversarial robustness | — | +24.7 points | | Faithfulness | 0% | 37.0% |

The adversarial robustness gain stems from protocol-constrained tool calls; recall improvements matter especially since missed findings are more dangerous than false positives.

Limitations: 63% of report content remains unverifiable, the toolset is chest-CT specific, and stepwise inference is much slower than end-to-end generation.

The core insight

The two papers represent opposite extremes of agent design, and the choice between them comes down to the cost of errors:

| | MM-WebAgent | RadAgent | |---|---|---| | Agent type | Multimodal planning agent | Tool-using reasoning agent | | Mechanism | Hierarchical planning + 3-level reflection | Stepwise tool calls + faithfulness verification | | Benchmark | MM-WebGEN-Bench (self-built) | CT-RATE (public) | | Error cost | Aesthetic disaster (redoable) | Misdiagnosis (potentially fatal) | | Philosophy | Holistic harmony over local optima; black-box acceptable | Verifiable correctness above all; process must be transparent |

Both systems share the same skeleton — a central LLM brain, a set of specialized tools, and a feedback loop (visual for MM-WebAgent, logical/evidential for RadAgent). Neither invents new AI capabilities; both demonstrate organizational ability — assembling existing components (LLM reasoning, AIGC generation, image processing) into reliable pipelines.

Author's reflections (Feynman-style)

"Agent" is an inflated term: it spans a spectrum from automated scripts to genuinely reasoning systems; more specific descriptions (e.g., "hierarchical multimodal planning system" vs. "stepwise explainable reasoning system") carry far more information.
Is MM-WebAgent's lack of learning a flaw or a feature? No learning avoids固化 bias and repetitive "formulas," but its "taste" is capped by training data — better planning can only produce more refined slop if the underlying models' aesthetics are poor.
Is RadAgent's 37% faithfulness enough? Not as a final diagnostic, but valuable as a triage tool that directs radiologist attention — AI as an augmenter of doctors, not a replacement.
Honest caveats: hierarchical planning is an old software-engineering idea (the novelty is reliable LLM execution); RadAgent's latency may be disqualifying in emergency settings; neither paper handles "dirty inputs" (vague requests, noisy scans).

Conclusion

MM-WebAgent proves native multimodal content generation beats code placeholders by a wide margin; RadAgent proves explainable stepwise reasoning is clinically viable for medical imaging. Together they sketch the next stage of agents: creative yet constrained, intelligent yet transparent, efficient yet trustworthy — doing the task while guaranteeing quality.

*References: MM-WebAgent — arXiv 2604.15309, Yan Li et al., Microsoft Research Asia; RadAgent — arXiv 2604.15231, Mélanie Roschewitz et al., Imperial College London & Stanford.*