Paper Overview
Field: Machine Learning Authors: Malena Loza, David Chushig-Muzo, Eva Milara, Luis Bote-Curiel, Luis Estrada-Petrocelli, Felipe Grijalva arXiv: 2607.26000
Summary
Tabular Foundation Models (TFMs) have emerged as novel approaches for tabular predictive tasks, demonstrating predictive performance competitive with ensemble tree-based models. However, most TFMs are trained and evaluated on independent and identically distributed (IID) data — an assumption that often breaks down in real-world scenarios due to distribution shifts, compromising model robustness. Research on TFMs under distribution shifts has been limited.
Key Points
- The authors empirically evaluate the Out-Of-Distribution (OOD) performance of nine TFMs spanning diverse pre-training strategies and architectures: TabPFNv2, TabPFNv2.5, TabPFNv2.6, TabPFNv3, TabICL, TabICLv2, Mitra, LimiX, and TabFM.
- Three real-world datasets from the TableShift study were used (HELOC, Voting, Childhood Lead), covering label, socioeconomic, and geographic shift types.
- All evaluated TFMs systematically degrade under distribution shift, regardless of pre-training strategy. Shift gaps range from 0.003 to 0.060 depending on the shift type.
- The relationship between in-distribution and OOD predictive performance documented for classical tabular models also extends to TFMs.
- A scalability gap exists: high-performing models require significant memory and compute resources that standard deployment infrastructure cannot support.
- The study extends existing OOD benchmarks for tabular data and provides evidence to inform the adoption of TFMs in high-stakes domains characterized by structural distribution shift.