CHARM: A Multimodal Graph Foundation Model with Hierarchical Context Modeling for Zero-Shot Transfer

Forum topic · 小凯 · 2026-07-30

Summary

CHARM is a multimodal graph foundation model (GFM) introduced by Ankang Yang, Jitao Zhao, Di Jin, Yuxiao Huang, and Dongxiao He on arXiv (2607.26023). It targets zero-shot transfer on multimodal graphs, where nodes are associated with text, images, and other modalities. Existing GNN-based graph foundation models typically require downstream adaptation, while LLM-based graph methods mainly handle unimodal graphs or single-domain tasks. CHARM addresses two key challenges: generalizing knowledge across individual modalities while capturing transferable cross-modal relations, and disentangling node representations from domain-specific structures and modality-specific features without target-domain fine-tuning. It replaces isolated raw nodes with hierarchical graph contexts that capture multimodal semantics and cross-modal relationships, mapping domain-specific node patterns to shared high-level concepts. A modality-aware graph context encoder integrates multimodal information with graph structure and converts the resulting representations into graph tokens for large language models. Experiments show consistent improvements on zero-shot multimodal graph tasks.

论文概要

Research area: Machine Learning Authors: Ankang Yang, Jitao Zhao, Di Jin, Yuxiao Huang, Dongxiao He Posted: 2026-07-28 arXiv: 2607.26023

Summary (English)

Graph foundation models (GFMs) have emerged as a promising paradigm for transferring knowledge across graph domains and tasks. Real-world graphs associate nodes with text, images, and other modalities, making multimodal graphs essential for representing complex entities and relations. Moreover, collecting labels and adapting models for every new graph domain is costly and often infeasible, motivating zero-shot transfer. Unfortunately, zero-shot transfer on multimodal graphs remains underexplored. Existing GNN-based graph foundation models typically require downstream adaptation, whereas LLM-based graph methods mainly address unimodal graphs or tasks within a single domain.

This setting presents two key challenges:

1. Models must generalize knowledge from individual modalities while capturing transferable cross-modal relations. 2. Without target-domain fine-tuning, node representations become entangled with domain-specific structures and modality-specific features, obscuring shared concepts in unseen domains.

The CHARM Approach

To address these challenges, the authors propose CHARM, a multimodal graph foundation model with hierarchical context modeling for zero-shot transfer:

Hierarchical graph contexts replace isolated raw nodes, capturing multimodal semantics and cross-modal relations.
These contexts map domain-specific node patterns to shared high-level concepts, reducing dependence on target-domain supervision or adaptation.
A modality-aware graph context encoder integrates multimodal information with graph structure and converts the resulting representations into graph tokens consumed by a large language model.

Results

Experiments demonstrate consistent improvements on zero-shot multimodal graph tasks.

---

*Auto-collected on 2026-07-30*