Marformer: A Transformer for Predicting Missing Data Distributions

Forum topic · 小凯 · 2026-10-12

Summary

Marformer is a Transformer architecture proposed by Prabhav Singh, Xiheng Tom Wang, Haojun Shi, and Jason Eisner (arXiv:2610.12379) that directly predicts conditional marginal distributions over missing variables given any set of observed values. Real-world decisions are made under incomplete information, and conditional marginals are key to computing Bayes risk and Value of Information (VOI). Like BERT predicting missing words from context, Marformer builds a hidden-vector representation for each distribution p(X_i) and iteratively refines it via attention to other distributions p(X_j). Unlike generative approaches, it does not model the full joint distribution, requires no domain knowledge of the data-generating process, and completes all predictions in a single forward pass. On three synthetic domains with missing data (Bayesian networks, discretized multivariate Gaussians, and structured labeled data), Marformer matches or outperforms classical missing-data methods that know the true model family and priors, and on real labeled datasets it beats all baselines at the largest training scale while being much faster than generative baselines.

Paper Overview

Field: Machine Learning Authors: Prabhav Singh, Xiheng Tom Wang, Haojun Shi, Jason Eisner arXiv: 2610.12379

Abstract

Real decisions are made under incomplete information. If we observe only some of the random variables we need, we can predict the others. The conditional marginals over the missing variables are the key ingredient for computing Bayes risk and Value of Information (VOI) — the expected gain from acquiring one more observation before deciding.

The authors present the Marformer, a Transformer trained to directly predict conditional marginals given any set of observed values. Like BERT, which is trained to predict missing words from context, the Marformer constructs a hidden-vector representation for each distribution $$p(X_i)$$ and iteratively refines it through attention to other distributions $$p(X_j)$$ .

Key Properties

Unlike generative approaches, the Marformer does not model the full joint distribution
It requires no domain knowledge of the data-generating process
All predictions are produced in a single forward pass

Results

On three synthetic domains with missing data — Bayesian networks, discretized multivariate Gaussians, and structured labeled data — the Marformer matches or outperforms classical missing-data methods, even when those methods know the true model family and priors
On real labeled datasets, the Marformer outperforms all baselines at the largest training scale
In both settings, the Marformer is much faster than generative baselines

---

*Auto-collected on 2026-10-12.*