DiffuSAM: Diffusion-Based Prompt-Free SAM2 Adaptation for Medical Image Segmentation
Paper Overview
- Field: Computer Vision
- Authors: Tal Grossman, Noa Cahan, Lev Ayzenberg
- Published: 2025-04-29
- arXiv: 2504.20597
- A lightweight diffusion prior that converts frozen SAM2 image features into mask-like embeddings compatible with SAM2's mask decoder
- Prompt-free segmentation, removing the reliance on user-provided prompts
- Slice-conditioned generation that enforces spatial consistency across volumetric data (CT/MRI)
- Competitive results on BTCV and CHAOS datasets under SF-UDA and few-shot settings
Original Abstract
Segmentation models such as Segment Anything Model (SAM) and SAM2 achieve strong prompt-driven zero-shot performance. However, their training on natural images limits domain transfer to medical data. Consequently, accurate segmentation typically requires extensive fine-tuning and expert-designed prompts.
We propose DiffuSAM, a diffusion-based adaptation of SAM2 for prompt-free medical image segmentation. Our framework synthesizes SAM2-compatible segmentation mask-like embeddings via a lightweight diffusion-prior from off-the-shelf frozen SAM2 image features. The generated embeddings are integrated into SAM2's mask decoder to produce accurate segmentations, thereby eliminating the need for user prompts. The diffusion prior is further conditioned on previously segmented slices, enforcing spatial consistency across volumes.
Key Contributions
Summary
DiffuSAM shows that generative diffusion priors can adapt general-purpose segmentation models like SAM2 to medical imaging without manual prompting or heavy fine-tuning, making volumetric CT and MRI segmentation more practical in source-free domain adaptation and few-shot scenarios.
Paper link: https://arxiv.org/abs/2504.20597