Summary
This arXiv paper (2610.13909) by Rudenko et al. systematically re-evaluates regression objectives for deep learning on tabular data. While the choice of loss function for regression is an understudied degree of freedom compared to classification, prior proposals for replacing Mean Squared Error (MSE) are hard to compare due to differing datasets, backbones, and evaluation protocols. Across a broad collection of established tabular datasets and multiple neural backbones, the authors find that many proposed objectives offer limited or inconsistent gains over a properly tuned MSE baseline, whereas quantile loss delivers consistent aggregate improvements across all tested backbones. They then identify a remaining limitation: regions of the target with different noise levels reach peak predictive performance at different training epochs, so a single global early-stopping checkpoint forces a compromise across regions of varying uncertainty. To address this, the paper introduces a heteroscedasticity-aware extension of quantile loss using input-dependent label smoothing, achieving further improvements over quantile loss on multiple backbones. The work provides a modern empirical picture of regression objectives in tabular deep learning.
Paper Overview
- Field: Machine Learning
- Authors: Danil Rudenko, Georgii Bulgakov, Nikolay Kartashev, Ivan Rubachev, Daniil Krasylnikov, Artem Babenko
- Published: 2026-10-09
- arXiv: 2610.13909
Abstract
While deep learning models for tabular data have recently received significant research attention, the choice of objective functions for regression tasks remains an understudied degree of freedom. Several lines of research offer alternatives to the standard Mean Squared Error (MSE), but differences in datasets, backbones, and evaluation protocols make their practical benefits difficult to compare.
This work addresses that gap by performing a systematic re-evaluation of regression objectives across a broad collection of established tabular datasets and multiple neural backbones. Key findings:
- Many previously proposed loss functions provide limited or inconsistent gains over a properly tuned MSE baseline.
- Quantile loss yields consistent aggregate improvements over MSE across all tested backbones.
The authors then identify a remaining limitation even when using this stronger loss: regions of the target with different noise levels may reach their best predictive performance at different training epochs. Consequently, even when the loss function can characterize heteroscedastic target distributions, a
single global early-stopping checkpoint may impose a compromise across regions of varying certainty.
Motivated by this limitation, the paper introduces a heteroscedasticity-aware extension of quantile loss that uses input-dependent label smoothing as a potential remedy, achieving further improvements over quantile loss across multiple backbones.
Overall, the results provide a modern empirical picture of regression objectives in tabular deep learning and highlight how differences in optimal stopping times across regions of varying certainty limit the effectiveness of global early stopping.
---
*Auto-collected on 2026-10-14*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178635966