Paper Slam 4/19: MM-WebAgent vs. RadAgent — Two Agent Design Philosophies for Web Design and Radiology

Forum topic · 小凯 · 2026-04-28

Summary

This forum post compares two AI agent papers: MM-WebAgent (arXiv 2604.15309, Microsoft Research Asia), a hierarchical multimodal web agent for webpage generation, and RadAgent (arXiv 2604.15231, Imperial College London & Stanford), a tool-using agent for stepwise chest CT interpretation. MM-WebAgent uses global-to-local planning plus a three-level reflection loop (local, context, global refinement) to coordinate AIGC tools for images, video, and charts, achieving 0.75 average score on its new MM-WebGEN-Bench versus 0.49 for code-only agents. RadAgent replaces black-box 3D VLM report generation with explicit tool calls and faithfulness verification, improving macro-F1 on CT-RATE from 0.287 (CT-CHAT) to 0.347, boosting adversarial robustness by 24.7 points, and reaching 37.0% faithfulness versus 0% for end-to-end baselines. The author argues the two systems embody opposite design philosophies—creative iteration with tolerant error costs versus verifiable correctness where mistakes can be fatal—and that agent design should be shaped by the stakes of failure. Weaknesses discussed include tool dependency and lack of learning for MM-WebAgent, and limited faithfulness and slower inference for RadAgent.

Paper Slam 4/19: When a Web Designer Meets a Radiologist — Two Agents, Two Roads

This post compares two agent papers through a Feynman-style lens: MM-WebAgent (arXiv 2604.15309, Microsoft Research Asia) and RadAgent (arXiv 2604.15231, Imperial College London & Stanford). One optimizes for creative integration; the other for explainable correctness.

Paper 1: MM-WebAgent — Hierarchical Multimodal Webpage Generation

Problem: Directly pasting AIGC content (images from Midjourney/DALL-E, videos, charts) into generated HTML produces visual chaos — elements clash in style even if each is fine in isolation.

Core design — three-stage planning plus three-level reflection:

1. Global Planning: produce a structured JSON design blueprint (layout, color scheme, style, sections) before writing any code. 2. Local Planning: per-section multimodal asset generation, with prompts constrained by the global style (e.g., "dark-blue-accent", 1920x1080 hero image). 3. Integration & Reflection: a three-tier refinement loop — Local Refine (per-element quality), Context Refine (HTML/CSS alignment and layout bugs), Global Refine (render, screenshot, and judge the whole page, looping back to global planning if needed).

The key architectural choice: multimodal content generation is a first-class agent action, not a placeholder. A central LLM (gpt-4o) orchestrates image/video/chart generation, HTML/CSS writing, and browser rendering for visual verification, enabling joint optimization across code correctness, layout, aesthetics, and style coherence.

Evaluation: The team built MM-WebGEN-Bench — 120 design tasks spanning four intents, multiple visual styles, layout complexities, and multimodal combinations — plus a six-dimension rubric (layout correctness, style coherence, aesthetics; image, video, chart quality).

Results (from the post):

| Paradigm | Average | |---|---| | Code-only One-shot | 0.43 | | Code-only Agent | 0.49 | | MM-WebAgent | 0.75 |

Multimodal generation roughly doubles local asset quality; iteration alone helps only marginally. The agent is training-free, which the author flags as both a limitation (no accumulated experience) and a possible feature (no hardened "design clichés"). Other weaknesses: dependence on external AIGC tool quality and a fixed toolset.

Paper 2: RadAgent — Tool-Using, Stepwise Chest CT Interpretation

Problem: End-to-end 3D VLMs like CT-CHAT compress the CT volume into visual tokens and autoregressively generate reports. This yields black-box reasoning, severe hallucination, poor adversarial robustness (macro-F1 of only 0.287 on CT-RATE), and zero faithfulness — no way to trace report claims back to image evidence.

Core design — explicit, traceable reasoning:

1. Visual Understanding: 3D encoder extracts multi-scale features as raw material. 2. Tool Use: interpretable tools for region localization, lesion detection, measurement, slice comparison, and report templating — each with human-readable outputs (e.g., "right upper lobe: slices 45–120, ~320 ml, −850 HU, no obvious abnormality"). 3. Stepwise Reasoning: sequential steps, each conditioned on the previous, all logged in an inspectable trace. 4. Faithfulness Verification: each report assertion is matched against visual evidence, yielding a faithfulness score.

Results (from the post): macro-F1 0.347 vs. 0.287 (+6.0), micro-F1 0.366 vs. 0.312, recall improved — critical since missed findings are the costliest errors. Adversarial robustness improves by 24.7 points, attributed to protocol-constrained tool calling acting as regularization. Faithfulness reaches 37.0% vs. 0% for CT-CHAT — the author reads this as a from-zero-to-one trust foundation, best used as a triage/screening aid rather than a final diagnosis. Weaknesses: 63% of content remains unverifiable, the toolset is chest-CT-specific, and stepwise inference is far slower than end-to-end generation.

Head-to-Head Comparison

| Dimension | MM-WebAgent | RadAgent | |---|---|---| | Task | Webpage generation (creative) | CT report generation (diagnostic) | | Core challenge | Visual consistency of multimodal content | Explainable, verifiable correctness | | Mechanism | Hierarchical planning + 3-level reflection | Stepwise tool calls + faithfulness verification | | Benchmark | MM-WebGEN-Bench (self-built) | CT-RATE (public) | | Cost of failure | Aesthetic disaster (redoable) | Misdiagnosis (potentially fatal) | | Black-box vs. white-box | Outcome-oriented; black box acceptable | Process must be white box |

Key Insight: Error Cost Determines Philosophy

The author's central claim: the two extremes of agent design are chosen by the cost of being wrong. Web generation can tolerate iterate-and-fix; medical diagnosis cannot. Hence MM-WebAgent's reflection loops automate "redo until harmonious," while RadAgent's tool protocol automates a "mandatory checklist."

Deeper Reflections (Feynman Lens)

Naming ≠ understanding: both are "agents," but they occupy different points on a spectrum from scripted automation to reasoning systems; the term "agent" is inflating toward zero information content.
Neither invents new AI capabilities — both contribute *organizational* ability: assembling existing components (LLMs, ViTs, diffusion models) into structured, evaluated workflows.
Open questions: MM-WebAgent's lack of learning may preserve creative diversity, but its "taste" is capped by training data (AI slop in, refined slop out). RadAgent's 37% faithfulness is insufficient as a final diagnosis but valuable as an attention-guiding screening tool.
Honest caveats: hierarchical design is a software-engineering staple; slow stepwise inference may exclude RadAgent from emergency settings; both papers assume clean inputs.

Conclusion

Both systems share the same skeleton — a central LLM brain, specialized tools, and a feedback loop — differing only in whether feedback is aesthetic ("does this look coherent?") or evidential ("is this claim supported?"). The future likely holds both types, and hybrids that verify factual accuracy while iterating on quality: creative yet constrained, intelligent yet transparent, efficient yet trustworthy.

*References: MM-WebAgent — arXiv 2604.15309, Microsoft Research Asia; RadAgent — arXiv 2604.15231, Imperial College London & Stanford.*

*Note: figures and table values are as reported in the original forum post; consult the papers for verified numbers.*