BrainTaskonomy: A Curriculum for Pretraining and Transfer in fMRI Foundation Models

Forum topic · 小凯 · 2026-09-10

Summary

BrainTaskonomy is a two-stage framework for training fMRI foundation models, addressing how to pretrain and what to transfer when data comes from heterogeneous sources such as resting-state, visual, working-memory, and pain-processing fMRI scans. In the first stage, a lightweight difficulty-aware trainer called Brain-DiT estimates each domain's learning difficulty via prediction error and measures pairwise transfer relationships between domains. These signals build a priority-guided cumulative domain curriculum, combined with a high-to-low noise time-step schedule. Compared with uniform sampling, this curriculum reportedly reduces voxel-level NMSE by 6.5%, power spectral density NMSE by 16.3%, and functional connectivity MSE by 10.5%. In the second stage, transfer relationships among 15 downstream tasks are measured and organized into a directed task graph, from which a budgeted integer programming (BIP) algorithm selects optimal transfer paths, including higher-order paths through intermediate tasks. Key findings include that transfer is highly asymmetric and target-dependent, and that indirect paths sometimes outperform direct transfer. The article also discusses connections to curriculum learning theory, brain development, and limitations such as proxy-trainer cost, dynamic data streams, and the correlation-versus-causation problem in transfer measurement.

Overview

Paper: BrainTaskonomy: Learning How to Pretrain and What to Transfer in fMRI Foundation Models Authors: Junfeng Xia, Wenhao Ye, Junxiang Zhang, et al. Fields: Computer Vision / Neuroscience (cs.CV, q-bio.NC) arXiv: to be announced

The Problem: Heterogeneous fMRI Data

Training an AI model to interpret human brain activity from fMRI (BOLD) signals involves data from many heterogeneous sources: resting-state scans, visual task fMRI, working-memory tasks, pain processing, and scans from different labs, scanners, and acquisition protocols.

The conventional approach mixes all data uniformly, like a salad, producing an "average" brain representation. The BrainTaskonomy authors identify why this is suboptimal:

Unequal domain difficulty: models waste capacity re-learning easy domains while under-serving hard ones.
Ignored inter-domain relationships: e.g., visual processing may share mechanisms with working memory, but standard training treats samples as independent.
Temporal noise structure: signal strength varies across task phases; uniform sampling wastes budget on noise.
Negative transfer risk: anomalous data (e.g., drug-intervention scans) can contaminate learned representations.

Stage 1: A Pretraining Curriculum via Brain-DiT

Brain-DiT (Brain Difficulty-aware Trainer) is a lightweight proxy that:

1. Estimates domain difficulty dynamically by measuring the model's prediction error on each domain — higher error means harder. 2. Measures transfer relationships: if pretraining on domain A improves initial performance on domain B, A helps B.

These measurements produce a Priority-guided Cumulative Domain Curriculum: basic domains (easy, broadly helpful) first, advanced domains later, combined with a high-to-low noise time-step schedule (coarse features early, fine features late). The curriculum adjusts dynamically, shifting sampling weight away from mastered domains.

Reported gains over uniform sampling:

v-NMSE (voxel-level normalized MSE): −6.5%
PSD-NMSE (power spectral density error): −16.3%
FC-MSE (functional connectivity error): −10.5%

The PSD improvement is notable because frequency-domain features are critical for understanding brain oscillations (alpha, gamma waves).

Stage 2: Transfer Taskonomy

For downstream adaptation, the framework measures transfer effects between all pairs of 15 downstream tasks (e.g., predicting memory load, pain, attention focus), builds a directed transfer graph, and uses Budgeted Integer Programming (BIP) to select optimal transfer paths given a compute budget — deciding which source tasks to supervise directly and which higher-order paths (A→B→C) to exploit.

Key findings:

Transfer is highly asymmetric and target-dependent: A→B may be strong while B→A is weak; optimal paths differ per target task.
Higher-order transfer: sometimes A→C is weak but A→B→C is strong — intermediate task B acts as a representational "bridge".

Why It Works

Curriculum learning intuition: starting with easy tasks finds a broadly correct region of the loss landscape, making harder tasks smoother to optimize. Brain-DiT extends standard linear curricula to a graph structure modeling mutual domain influence.
Network effects: some tasks act as "hubs" in the transfer graph (e.g., resting-state functional connectivity, reflecting foundational brain architecture). BIP acts like a navigation system finding the best route — sometimes a detour beats the direct path.
Neuroscience parallels: the ordering mirrors cortical development (sensory → association → prefrontal); the joint consolidation mechanism resembles sleep-based memory consolidation; dynamic domain weighting parallels changing neuroplasticity.

Beyond fMRI

The core idea — learning how to pretrain and what to transfer — generalizes to multimodal learning, cross-lingual transfer, lifelong learning, and experiment planning in science (drug discovery, materials science).

Limitations and Open Questions

1. Proxy cost: Brain-DiT requires extra compute for difficulty and transfer measurement; its own cost at scale is unreported. 2. Dynamics: the paper assumes fixed domains/tasks; streaming new data would require curriculum updates. 3. Correlation vs. causation: measured transfer may reflect hidden common factors rather than genuine causal transfer. 4. Interpretability: the taskonomy says A helps B but not *why* — understanding shared representations remains open.

Takeaway

The deepest insight of BrainTaskonomy: learning is not the accumulation of information but the weaving of relationships. Knowing *how* to learn — what to learn first, what to defer, and where knowledge transfers — matters more than raw data volume.

Reference

Xia, J., Ye, W., Zhang, J., et al. (2026). *BrainTaskonomy: Learning How to Pretrain and What to Transfer in fMRI Foundation Models*. arXiv preprint.