LimiX: Tsinghua's 2M-Parameter Model Tackles Tabular Data, Outperforming XGBoost
This post, originally published on zhichai.net, discusses why deep learning has struggled with structured tabular data and introduces LimiX, a lightweight tabular foundation model from Professor Cui Peng's team at Tsinghua University.
Why Deep Learning Struggles with Tabular Data
Structured tables mix numeric and categorical features, contain missing values, and feature hidden inter-column dependencies. Unlike text or images, tabular datasets are often small and noisy, so deep models tend to overfit—memorizing training-set noise instead of learning generalizable decision boundaries. Purpose-built architectures (TabNet, SAINT, FT-Transformer) have repeatedly lost to gradient-boosted trees like XGBoost and CatBoost, which naturally handle mixed types, missing values, and feature importance, and remain robust on small datasets.
The LimiX Approach: Causal Pretraining on a Light Transformer
LimiX is a multi-task family covering classification, regression, imputation, data generation, and causal inference within one framework. Key design elements:
- SCM-based pretraining: synthetic data generated from hierarchical structural causal models teaches the model causal relationships rather than spurious correlations.
- Lightweight architecture: 12-layer Transformer with discriminative feature encoding (DFE) and an asymmetric design balancing feature-level and sample-level attention.
- Zero-shot adaptation: masked joint-distribution pretraining enables in-context learning on new tasks without retraining.
- Scaling laws: loss follows a power-law decline with model size and data volume, mirroring LLM scaling behavior.
- Steel manufacturing: fault prediction improved ~15%, shifting from reactive to proactive maintenance.
- Materials research: reported 5x efficiency gains in R&D workflows.
- Healthcare: imputation accuracy of 0.915 on early-stage diabetes data supports safer record completion.
- Edge deployment: at 2M parameters, LimiX-2M runs on resource-constrained devices.
Benchmark Results
| Benchmark | Task | LimiX-16M | LimiX-2M | XGBoost | CatBoost | AutoGluon | TabPFN-v2 | |---|---|---|---|---|---|---|---| | BCCO-CLS | Classification (AUC) | 0.871 | 0.855 | 0.829 | 0.822 | 0.846 | 0.843 | | OpenML-CC18 | Classification (Acc) | 0.892 | 0.878 | 0.851 | 0.845 | 0.867 | 0.862 | | BCCO-REG | Regression (R²) | 0.794 | 0.772 | 0.764 | 0.758 | 0.781 | 0.777 | | TALENT-REG | Regression (RMSE) | 0.386 | 0.402 | 0.415 | 0.421 | 0.398 | 0.399 | | TableShift | OOD Generalization (AUC) | 0.806 | 0.792 | 0.793 | 0.793 | 0.797 | 0.797 | | Early Diabetes | Imputation (Acc) | 0.915 | 0.902 | N/A | N/A | 0.889 (HyperImpute) | N/A |
Results were validated across 11 benchmarks and 600+ datasets. Robustness tests show LimiX degrades only ~5% with 90% uninformative features or extreme outliers, compared to >15% drops for AutoGluon.
Industrial Applications
Debate and Outlook
Critics note that benchmarks like TableShift may not represent TB-scale industrial data with long-tail distributions, and tree-model interpretability remains an advantage. Suggested hybrid pipelines (LimiX embeddings + XGBoost) may offer the best of both worlds. LimiX is open-sourced under Apache 2.0.
References
1. arXiv: LimiX: Unleashing Structured-Data Modeling Capability for Generalist Intelligence 2. Tsinghua University Team Open-Sources First General Large Model for Structured Data 3. GitHub Repository: limix-ldm/LimiX 4. WiseModel: LimiX-2M Model Page 5. NetEase Article: 仅2M参数！清华LimiX攻克表格数据难题，超越XGBoost