PictorialCortex: Zero-Shot Cross-Subject fMRI-to-Image Reconstruction via Compositional Latent Modeling

Forum topic · 小凯 · 2026-05-30

Summary

Researchers from Fudan University, Zhejiang Normal University, and Nanyang Technological University propose PictorialCortex, a framework that reconstructs visual images from fMRI signals of subjects never seen during training. The method decomposes fMRI observations into four interpretable factors—stimulus-driven content, subject-specific traits, dataset bias, and nuisance noise—via a Latent Factor Compositional Module (LFCM, 195M parameters) built on a 1.27B-parameter universal cortex autoencoder pretrained on UK Biobank data. Two training mechanisms, Paired Factorization and Reconstruction (PFR) and Re-Factorizing Consistency Regularization (ReFCR), enforce factor disentanglement. At inference, surrogate fMRI latents synthesized under multiple known subjects' conditioning are aggregated to extract a refined stimulus code, which conditions a diffusion model using IP-Adapter SDXL Plus visual features. Trained on the new UniCortex-fMRI dataset integrating four visual stimulation datasets, the model achieves state-of-the-art zero-shot cross-subject reconstruction in semantic content and spatial layout. The paper does not report pixel-level metrics, and the authors note limits regarding memory decoding, compute cost, population representativeness, and privacy implications.

Reading Minds into Images: How PictorialCortex Turns Thoughts into Pictures

> In one sentence: A team from Fudan University, Zhejiang Normal University, and Nanyang Technological University decomposes fMRI signals into four interpretable factors—stimulus content, subject traits, dataset bias, and nuisance noise—achieving zero-shot cross-subject visual reconstruction on a 1.27B-parameter universal brain-language foundation. The model can guess what someone saw without first scanning them for hours.

| Item | Detail | |------|--------| | Paper | The Pictorial Cortex: Zero-Shot Cross-Subject fMRI-to-Image Reconstruction via Compositional Latent Modeling | | arXiv | 2601.15071v1 | | Date | January 21, 2026 | | Team | Fudan University · Zhejiang Normal University · Nanyang Technological University | | Method | Compositional latent modeling + universal brain-language foundation pretraining | | Scale | 1.27B params (universal autoencoder) + 195M params (LFCM factorization module) | | Hardware | 8× NVIDIA H100 (autoencoder) + 2× NVIDIA H200 (LFCM) | | Data | UniCortex-fMRI (integrating 4 visual-stimulus datasets) + UK Biobank pretraining | | Visual target | IP-Adapter SDXL Plus features | | Capability | Zero-shot cross-subject fMRI → image reconstruction, no per-subject training |

The Core Problem: One Image, Ten Brains, Ten Different Signals

The long-standing challenge in fMRI visual decoding is cross-subject generalization. Two people looking at the same cat produce vastly different BOLD spatiotemporal patterns in visual cortex—due to anatomical differences in cortical folds, vascular differences, scanner parameters, and even fatigue. All of these are mixed into the signal like an inseparable cocktail.

The traditional approach requires hours of subject-specific data and retraining for every new participant—impractical outside the lab. PictorialCortex targets the zero-shot cross-subject setting: the model has never seen the subject during training, yet reconstructs what they saw from test-time fMRI alone.

Three-Stage Architecture

Stage 1: Universal Brain-Language Foundation (1.27B params)

A high-capacity autoencoder is pretrained on large-scale cortical-surface fMRI from UK Biobank. Input: 256×256 cortical surface activation maps (anatomically blank regions pre-masked to save tokens). Encoder: 32-layer Transformer, 1280-dim embeddings, 16-head attention. Output: a universal cortical latent of 4 CLS tokens.

A symmetric decoder with learnable mask-replacement tokens forces the encoder to compress whole-cortex patterns into just a few CLS tokens. Trained with simple MSE reconstruction (250k iterations, 8×H100), the encoder must capture cross-subject shared anatomical regularities and abstract away individual differences—yielding a universal cortical space into which any new subject's fMRI can be mapped.

Stage 2: Latent Factor Compositional Module (LFCM, 195M params)

An fMRI observation is modeled as the superposition of four factors:

(i) Stimulus-driven factor — the true visual content; should be similar across subjects viewing the same image.
(ii) Subject factor — individual cortical anatomy and physiology.
(iii) Dataset factor — systematic bias from scanning protocols, hardware, preprocessing.
(iv) Nuisance factor — trial-level random noise.

LFCM comprises:

Factorizer: under subject/dataset embedding conditioning, splits the universal latent z into a stimulus-driven code c (16×2048) and a nuisance code n (1×2048).
Compositor: recombines c and n, conditioned on subject/dataset, to synthesize surrogate fMRI latents.

Conditioning is implemented simply—learnable 1×2048 vectors for subject and dataset, added like positional encodings—but the effect is powerful: the Factorizer knows which subject and dataset a signal comes from and strips those biases away, leaving the stimulus-driven core.

Stage 3: Surrogate Aggregation at Inference

For a new, unseen subject: start with a default subject embedding to extract an initial stimulus code; use the Compositor to synthesize surrogate fMRI latents conditioned on multiple known subjects; feed these surrogates back through the Factorizer; aggregate to obtain a refined stimulus code. This code then conditions a diffusion model (in IP-Adapter SDXL Plus feature space) to generate the reconstruction.

Key insight: cross-validating stimulus content across multiple known subjects' "viewpoints" cancels the bias of any single subject embedding—like taking the consensus of ten translators rather than trusting one.

Two Training Mechanisms That Keep Factors Disentangled

Compositional modeling's greatest risk is factor entanglement—visual content leaking into the nuisance code, or subject traits contaminating visual content.

PFR (Paired Factorization and Reconstruction): the same stimulus presented twice yields two fMRI observations that share one stimulus code c but have different nuisance codes n. After factorization, the Compositor reconstructs each, plus pairwise-swapped combinations (c from one paired with n from the other). This forces c to carry only visual content and n only trial-level variation.
ReFCR (Re-Factorizing Consistency Regularization): surrogate latents synthesized by the Compositor are "fake" fMRI. ReFCR requires the Factorizer to recover the original c/n components from these surrogates, ensuring stability when inference relies entirely on surrogates.

Together they form a self-consistent loop: real data → factorization → recomposition → re-factorization, with consistency constraints at every link.

Results

The team built UniCortex-fMRI, a unified cortical-surface dataset integrating four heterogeneous visual-stimulus datasets with broad subject and stimulus coverage.

Zero-shot cross-subject reconstruction achieves state-of-the-art results in both semantic content and spatial layout; reconstructions identify the approximate category and structure of the original stimulus.
Multi-dataset training gains: integrating four datasets significantly improves generalization—subject diversity itself acts as regularization.
Seen vs. unseen subjects: reconstruction quality is comparable, indicating individual differences were successfully isolated into subject and nuisance factors while the stimulus-driven factor stayed stable across subjects.

Honestly Unclear Things

How accurate is reconstruction? The paper shows semantic-level success but no pixel-level SSIM or LPIPS. fMRI-to-image reconstruction is inherently low-fidelity—a blurry but recognizable sketch, not a 4K brain monitor.
Memory vs. current perception? The work decodes real-time visual stimuli. Decoding recall, imagination, or dreams—where no ground-truth visual input exists—is far harder and untested, though the framework could theoretically transfer.
Clinical path? Training costs (1.27B params, 8×H100) preclude local hospital deployment; even LFCM fine-tuning and inference demand substantial compute.
Privacy and ethics: if mature, zero-shot decoding means anyone in a scanner could be "mind-read" without prior personal modeling—implications for privacy, interrogation, and forensics far exceed traditional brain decoding, and are not discussed in depth.
UK Biobank representativeness: the pretrained population is largely middle-aged/older UK residents; generalization to other demographics or neurological patients is unevaluated.

Our Take

PictorialCortex's contribution is not the visual flashiness of image reconstruction but the methodological decomposition. The heterogeneity of fMRI signals has plagued visual decoding for twenty years; prior work either ignored it (brute-force with data) or avoided it (single-subject only). PictorialCortex confronts it head-on—explicitly modeling heterogeneity as four interpretable factors and enforcing their separation through training mechanisms.

This compositional latent modeling framework extends beyond brain decoding: any inverse problem with multi-source confounded signals—audio source separation, medical image denoising, economic factor extraction—could adopt the recipe of first learning a universal space, then performing conditioned factorization within it.

Still, "mind-reading" hype warrants caution. Reconstructions remain blurry, and fMRI's second-scale temporal resolution rules out real-time brain-computer interfaces—this is post-hoc decoding, not live reading.

The most exciting direction may be different: if the stimulus-driven factor is pure enough, it could become a universal coordinate system for human visual representation, mapping different brains, devices, and experiments into one shared space whose geometry might reveal deep organizing principles of the visual system—closer to fundamental neuroscience than "reconstructing a cat picture."

| Item | Content | |------|---------| | Core contributions | (1) UniCortex-fMRI unified dataset; (2) PictorialCortex compositional latent modeling explicitly decoupling stimulus/subject/dataset/nuisance factors; (3) UK Biobank-pretrained universal foundation supporting cross-subject shared representations; (4) PFR+ReFCR dual mechanisms ensuring disentanglement and inference stability; (5) SOTA zero-shot cross-subject fMRI→image reconstruction | | Key limitations | Pixel-level accuracy unreported; memory/imagination decoding unvalidated; high training cost; UK Biobank representativeness questionable; profound but undiscussed privacy implications |

References: 1. Fudan University / Zhejiang Normal University / Nanyang Technological University, "The Pictorial Cortex: Zero-Shot Cross-Subject fMRI-to-Image Reconstruction via Compositional Latent Modeling", arXiv:2601.15071v1, 2026. 2. UK Biobank, https://www.ukbiobank.ac.uk/ 3. Ye et al., "IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models", arXiv:2308.06721, 2023.