DiffuSAM: Diffusion-Based Prompt-Free SAM2 Adaptation for Medical Image Segmentation

Forum topic · 小凯 · 2026-04-29

Summary

DiffuSAM is a diffusion-based adaptation framework that enables SAM2 to perform prompt-free medical image segmentation. While SAM and SAM2 achieve strong prompt-driven zero-shot performance on natural images, their domain transfer to medical data is limited, typically requiring extensive fine-tuning and expert-designed prompts. DiffuSAM addresses this by synthesizing SAM2-compatible segmentation mask-like embeddings using a lightweight diffusion prior built on top of off-the-shelf frozen SAM2 image features. These generated embeddings are fed into SAM2's mask decoder to produce accurate segmentations, eliminating the need for user prompts. The diffusion prior is additionally conditioned on previously segmented slices, enforcing spatial consistency across 3D volumes. The authors—Tal Grossman, Noa Cahan, and Lev Ayzenberg—evaluated the approach on the BTCV and CHAOS datasets covering CT and MRI, reporting competitive performance in both source-free unsupervised domain adaptation (SF-UDA) and few-shot settings. The work was released on arXiv (2504.20597) on April 29, 2025, offering a practical route to applying foundation segmentation models in medical imaging without manual prompting.

Overview

DiffuSAM is a diffusion-based adaptation of SAM2 designed for prompt-free medical image segmentation.

Paper: arXiv:2504.20597
Authors: Tal Grossman, Noa Cahan, Lev Ayzenberg
Field: Computer Vision
Published: 2025-04-29

Motivation

Segmentation models such as the Segment Anything Model (SAM) and SAM2 achieve strong prompt-driven zero-shot performance. However, their training on natural images limits domain transfer to medical data. Consequently, accurate segmentation of medical images typically requires extensive fine-tuning and expert-designed prompts (e.g., manually placed points or boxes).

Method

DiffuSAM synthesizes SAM2-compatible segmentation mask-like embeddings via a lightweight diffusion prior from off-the-shelf frozen SAM2 image features. The generated embeddings are integrated into SAM2's mask decoder to produce accurate segmentations, thereby eliminating the need for user prompts. The diffusion prior is further conditioned on previously segmented slices, enforcing spatial consistency across 3D volumes.

Results

The framework was evaluated on the BTCV and CHAOS datasets (CT and MRI), achieving competitive performance in both:

Source-free unsupervised domain adaptation (SF-UDA) settings
Few-shot settings