MedVTok: A General-Purpose Medical Visual Tokenizer Unifying 2D and 3D Medical Imaging

Forum topic · 小凯 · 2026-10-27

Summary

MedVTok is a general-purpose medical visual tokenizer introduced in arXiv paper 2610.35842 by Chenglong Ma, Yuanfeng Ji, Junzhi Ning, Jiyao Liu, Yujie Wei, and colleagues. It maps 2D images and 3D volumes from diverse imaging modalities into a unified token space, addressing two key challenges: preserving anatomical structure across dimensions and capturing clinical semantics across modalities. The method introduces slice differential regularization, which explicitly models inter-slice anatomical coherence absent from prior per-pixel/per-voxel supervision, and multi-expert representation alignment, which integrates knowledge from multiple medical expert encoders while retaining modality-specific complementary information. Trained on 28 million images and 70,000 volumes across 9 imaging modalities, MedVTok supports classification, retrieval, segmentation, synthesis, visual question answering, and medical report generation through a single tokenizer. Across 24 benchmarks it achieves state-of-the-art performance, demonstrating a scalable token interface that unifies medical image dimensions, modalities, and task usage for multimodal medical AI systems.

Paper Overview

Research area: Computer Vision (CV) Authors: Chenglong Ma, Yuanfeng Ji, Junzhi Ning, Jiyao Liu, Yujie Wei et al. arXiv: 2610.35842

Abstract

Multimodal medical AI requires a common visual interface for encoding heterogeneous imaging evidence for patient condition modeling. However, existing medical visual tokenizers are often tied to a specific dimension and imaging modality, forcing multimodal systems to rely on fragmented and poorly reusable representations.

MedVTok is a general-purpose visual tokenizer that maps 2D images and 3D volumes from diverse imaging modalities into a unified token space. Building such a tokenizer poses two key challenges:

1. Preserving anatomical structure across dimensions — addressed via *slice differential regularization*, which explicitly models inter-slice anatomical coherence missing in previous per-pixel/per-voxel supervision. 2. Capturing clinical semantics across modalities — addressed via *multi-expert representation alignment*, which integrates knowledge from multiple medical expert encoders while retaining modality-specific complementary knowledge.

Training and Results

Trained on 28M images and 70K volumes across 9 imaging modalities
Supports classification, retrieval, segmentation, synthesis, visual question answering, and medical report generation through a single tokenizer
Achieves state-of-the-art performance across 24 benchmarks

MedVTok demonstrates a scalable token interface that unifies medical image dimension, modality, and task usage.