CHARM: A Multimodal Graph Foundation Model with Hierarchical Context Modeling for Zero-Shot Transfer

Forum topic · 小凯 · 2026-07-30

Summary

CHARM is a multimodal graph foundation model (GFM) designed for zero-shot transfer across graph domains and tasks, proposed by Ankang Yang, Jitao Zhao, Di Jin, Yuxiao Huang, and Dongxiao He (arXiv:2607.26023). Real-world graphs increasingly link nodes with text, images, and other modalities, but existing GNN-based GFMs typically require downstream adaptation, while LLM-based graph methods focus on unimodal graphs or single domains. CHARM addresses two key challenges: generalizing knowledge from individual modalities while capturing transferable cross-modal relations, and disentangling node representations from domain-specific structure and modality-specific features without target-domain fine-tuning. The model replaces isolated raw nodes with hierarchical graph contexts that capture multimodal semantics and cross-modal relationships, mapping domain-specific node patterns to shared high-level concepts. A modality-aware graph context encoder integrates multimodal information with graph structure and converts the resulting representations into graph tokens for large language models. Experiments show consistent improvements on zero-shot multimodal graph tasks.

Overview

Research area: Machine Learning Authors: Ankang Yang, Jitao Zhao, Di Jin, Yuxiao Huang, Dongxiao He arXiv: 2607.26023

Abstract

Graph foundation models (GFMs) have emerged as a promising paradigm for transferring knowledge across graph domains and tasks. Real-world graphs associate nodes with text, images, and other modalities, making multimodal graphs essential for representing complex entities and relations. Moreover, collecting labels and adapting models for every new graph domain is costly and often infeasible, motivating zero-shot transfer. Unfortunately, zero-shot transfer on multimodal graphs remains underexplored. Existing GNN-based graph foundation models typically require downstream adaptation, whereas LLM-based graph methods mainly address unimodal graphs or tasks within a single domain.

This setting presents two key challenges:

Models must generalize knowledge from individual modalities while capturing transferable cross-modal relations.
Without target-domain fine-tuning, node representations become entangled with domain-specific structure and modality-specific features, obscuring shared concepts in unseen domains.

The CHARM Approach

To address these challenges, the authors propose CHARM, a multimodal graph foundation model with hierarchical context modeling for zero-shot transfer:

CHARM replaces isolated raw nodes with hierarchical graph contexts, capturing multimodal semantics and cross-modal relations.
These contexts map domain-specific node patterns to shared high-level concepts, reducing reliance on target-domain supervision or adaptation.
A modality-aware graph context encoder integrates multimodal information with graph structure and converts the resulting representations into graph tokens for large language models.

Experiments demonstrate consistent improvements on zero-shot multimodal graph tasks.