PictorialCortex: Zero-Shot Cross-Subject fMRI-to-Image Reconstruction via Compositional Latent Modeling

Forum topic · 小凯 · 2026-05-30

Summary

A joint team from Fudan University, Zhejiang Normal University, and Nanyang Technological University proposes PictorialCortex, a framework for zero-shot cross-subject reconstruction of viewed images from fMRI signals (arXiv:2601.15071v1). The method decomposes fMRI observations into four interpretable factors: stimulus-driven content, subject-specific traits, dataset bias, and trial-level nuisance noise. A 1.27-billion-parameter universal cortical autoencoder is pre-trained on UK Biobank cortical-surface fMRI to learn a shared cortical latent space, while a 195M-parameter Latent Factor Compositional Module (LFCM) performs the factorization. Two training mechanisms—Paired Factorization and Reconstruction (PFR) and Re-Factorizing Consistency Regularization (ReFCR)—enforce true disentanglement. At inference, surrogate latent representations generated under multiple known subjects are re-factorized and aggregated to refine the stimulus code for unseen subjects, which then conditions a diffusion model based on IP-Adapter SDXL Plus features. Evaluated on UniCortex-fMRI, a unified dataset integrating four visual-stimulus datasets, the approach achieves state-of-the-art zero-shot cross-subject reconstruction. The author notes limitations: no pixel-level metrics reported, unverified memory/imagination decoding, high training cost, and unresolved privacy implications.

PictorialCortex: Turning Thoughts into Pictures with Compositional Latent Modeling

In one sentence: A team from Fudan University, Zhejiang Normal University, and Nanyang Technological University decomposes fMRI signals into four interpretable factors—stimulus content, subject traits, dataset bias, and nuisance noise—achieving zero-shot cross-subject visual reconstruction on a 1.27-billion-parameter universal brain-language foundation model. No need to scan a new subject for hours first; the model can infer what they saw.

| Item | Detail | |------|--------| | Paper | The Pictorial Cortex: Zero-Shot Cross-Subject fMRI-to-Image Reconstruction via Compositional Latent Modeling | | arXiv | 2601.15071v1 | | Date | January 21, 2026 | | Team | Fudan University · Zhejiang Normal University · Nanyang Technological University | | Method | PictorialCortex: compositional latent modeling + universal brain-language foundation pre-training | | Model size | 1.27B params (universal autoencoder) + 195M params (LFCM) | | Training hardware | 8× NVIDIA H100 (autoencoder) + 2× NVIDIA H200 (LFCM) | | Datasets | UniCortex-fMRI (integrating 4 visual-stimulus datasets) + UK Biobank pre-training | | Visual features | IP-Adapter SDXL Plus | | Core capability | Zero-shot cross-subject fMRI → image reconstruction, no per-subject training data |

The Core Problem: Same Image, Ten Different Brain Signals

The long-standing challenge in fMRI visual decoding is cross-subject generalization. Two people looking at the same cat produce strikingly different BOLD spatiotemporal patterns in visual cortex: anatomical differences in cortical folding, vascular variability, scanner parameters, even daily fatigue—all mixed into one inseparable cocktail.

Conventional approaches require hours of subject-specific data collection and retraining for every new participant—impractical outside the lab. PictorialCortex tackles the zero-shot cross-subject setting: the model has never seen the subject during training, yet reconstructs what they saw from their fMRI alone.

Three-Stage Architecture

Stage 1: Universal Brain-Language Foundation (1.27B params)

A high-capacity autoencoder is pre-trained on large-scale cortical-surface fMRI from UK Biobank.

Input: 256×256 cortical surface activation maps (anatomically blank regions pre-masked to save tokens)
Encoder: 32-layer Transformer, 1280-dim embeddings, 16 heads
Output: a universal cortical latent representation of 4 CLS tokens

A symmetric decoder replaces masked spatial tokens with learnable guide tokens, forcing the encoder to compress entire cortical activation patterns into just a few CLS tokens via simple MSE reconstruction. To succeed, it must capture cross-subject shared anatomical regularities and abstract away individual differences. The result is a "universal cortical space" into which any new subject's fMRI can be mapped.

Stage 2: Compositional Latent Modeling (LFCM, 195M params)

An fMRI observation is the superposition of four factors:

1. Stimulus-driven factor — the actual visual content (should be similar across subjects) 2. Subject factor — individual cortical anatomy and physiology 3. Dataset factor — systematic bias from protocols, scanners, preprocessing 4. Nuisance factor — trial-level random noise

LFCM comprises two modules:

Factorizer: splits the universal latent representation z (conditioned on subject and dataset embeddings) into a stimulus code c (16×2048) and nuisance code n (1×2048)
Compositor: recombines c and n, conditioned on subject/dataset, to synthesize surrogate fMRI latent representations

The embeddings are simple learnable 1×2048 vectors added to hidden representations (like positional encodings), but conditioning lets the Factorizer strip subject- and dataset-specific deviations, isolating the stimulus-driven core.

Stage 3: Surrogate Aggregation at Inference

For a new, unseen subject:

1. Extract an initial stimulus code using a default subject embedding 2. Use the Compositor to synthesize surrogate latent representations conditioned on multiple known subjects 3. Re-factorize the surrogates with the Factorizer and aggregate into a refined stimulus code 4. Condition a diffusion model (IP-Adapter SDXL Plus feature space) on this code to generate the reconstruction

Key insight: cross-validating stimulus content through multiple known subjects' "perspectives" cancels the bias of any single subject embedding—like averaging translations from ten bilingual speakers rather than trusting one.

Two Training Mechanisms

Compositional modeling's greatest risk is factor entanglement. Two complementary mechanisms lock the boundaries:

PFR (Paired Factorization and Reconstruction): the same stimulus is shown twice, yielding two fMRI observations that share the same stimulus code c but have different nuisance codes n. After factorization, pairwise swapping (c from one, n from the other) must still reconstruct faithfully—forcing c to carry only visual content and n only trial-level variation.
ReFCR (Re-Factorizing Consistency Regularization): surrogate representations synthesized by the Compositor are "fake" fMRI; a robust Factorizer should re-decompose them into components consistent with the original c/n. This guarantees stability at inference, which relies entirely on surrogates.

Together they form a self-consistency loop: real data → factorization → recomposition → re-factorization, with consistency constraints at every link.

Results

Zero-shot cross-subject reconstruction: subjects held out entirely from training achieve state-of-the-art results in semantic content and spatial layout; reconstructions recover the approximate category and structure of the original stimuli.
Multi-dataset gains: training on the unified UniCortex-fMRI significantly improves cross-subject generalization—subject diversity itself acts as regularization.
Known vs. unseen subjects: reconstruction quality is comparable, indicating individual differences were successfully isolated into subject and nuisance factors rather than leaking into the stimulus code.

Honest Unknowns

Reconstruction fidelity? Semantic-level success is shown, but no pixel-level SSIM/LPIPS is reported. fMRI-to-image reconstruction is inherently low-fidelity—a blurry but recognizable sketch, not a "4K brain-interface monitor."
Memory vs. live perception? The work uses real-time stimulus fMRI. Decoding recall, imagination, or dreams lacks ground truth and remains unverified, though the framework could in principle transfer.
Clinical path? 1.27B parameters and 8×H100 training mean this is not deployable in a hospital MRI suite today.
Privacy and ethics: if zero-shot decoding matures, anyone in a scanner could be "mind-read" without prior personal modeling—implications for privacy, interrogation, and forensics far exceed traditional (cooperation-requiring) brain decoding.
UK Biobank representativeness: pre-training draws mainly from a middle-aged/older UK population; generalization to other demographics or neurological patients is unassessed.

Author's Take

PictorialCortex's core contribution is not flashy demos but a methodological decomposition of the multi-source heterogeneity that has plagued visual decoding for two decades. The compositional latent modeling framework—learn a universal space, then perform conditioned factorization within it—could apply to any inverse problem with multi-source confounded signals, from audio source separation to medical imaging denoising.

The author remains skeptical of "mind-reading" hype: reconstructions are still blurry, and fMRI's second-scale temporal resolution rules out real-time BCI—this is post-hoc decoding, not live reading. The genuinely exciting direction may be different: if the stimulus-driven factor is clean enough, it could become a universal coordinate system for human visual representation, whose geometry might reveal deep organizing principles of the visual system—closer to fundamental neuroscience than "reconstructing a cat."

References 1. Fudan University / Zhejiang Normal University / Nanyang Technological University, "The Pictorial Cortex: Zero-Shot Cross-Subject fMRI-to-Image Reconstruction via Compositional Latent Modeling", arXiv:2601.15071v1, 2026. 2. UK Biobank, https://www.ukbiobank.ac.uk/ 3. Ye et al., "IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models", arXiv:2308.06721, 2023.