ScoringBench: A Benchmark for Evaluating Tabular Foundation Models with Proper Scoring Rules

Forum topic · 小凯 · 2026-04-02

Summary

Tabular foundation models such as TabPFN and TabICL produce full predictive distributions, yet prevailing regression benchmarks evaluate them almost exclusively with point-estimate metrics like RMSE and R2. These aggregate measures often obscure model performance in the tails of the distribution—a critical gap for high-stakes domains such as finance and clinical research where asymmetric risk profiles are the norm. ScoringBench, introduced by Jonas Landsgesell and Pascal Knoll (arXiv:2603.11115), is an open benchmark that computes a comprehensive suite of proper scoring rules—including CRPS, CRLS, Interval Score, Energy Score, weighted CRPS, and Brier Score—alongside standard point metrics, giving a richer picture of probabilistic forecast quality. The authors evaluate realTabPFNv2.5 fine-tuned with different scoring-rule objectives as well as TabICL against an unadjusted realTabPFNv2.5 baseline across a regression benchmark suite. Results confirm that model rankings depend on the chosen scoring rule, and no single pretraining objective is universally optimal, suggesting practitioners should select models based on the score most aligned with their decision-making needs.

Paper Overview

Field: AI Authors: Jonas Landsgesell, Pascal Knoll Published: 2026-03-31 arXiv: 2603.11115

Abstract

Tabular foundation models such as TabPFN and TabICL already produce full predictive distributions, yet prevailing regression benchmarks evaluate them almost exclusively via point-estimate metrics like RMSE and R2. These aggregate measures often obscure model performance in the tails of the distribution—a critical deficit for high-stakes decision-making in domains like finance and clinical research, where asymmetric risk profiles are the norm.

The authors introduce ScoringBench, an open benchmark that computes a comprehensive suite of proper scoring rules—CRPS, CRLS, Interval Score, Energy Score, weighted CRPS, and Brier Score—alongside standard point metrics, providing a richer picture of probabilistic forecast quality.

Evaluation

They evaluate realTabPFNv2.5 fine-tuned with different scoring-rule objectives, as well as TabICL, relative to an unadjusted realTabPFNv2.5 baseline on a regression benchmark suite.

Findings

Model rankings depend on the selected scoring rule.
No single pretraining objective is universally optimal.
Proper scoring rules reveal tail behavior that point metrics like RMSE and R2 miss, which matters for risk-sensitive applications.