Summary
A forum post on zhichai.net summarizes the arXiv paper 2504.13085, "Generalization in LLM Problem Solving: The Case of the Shortest Path" by Yao Tong, Jiayuan Ye, and Anastasia Borovykh (April 2025). The paper studies whether language models can systematically generalize, using a controlled synthetic environment based on shortest-path planning, a canonical composable sequential optimization problem. This setup cleanly separates the effects of training data, training paradigms, and inference-time strategies, and measures generalization along two orthogonal axes: spatial transfer to unseen maps and length scaling to longer-horizon problems. Key findings: models show strong spatial transfer but consistently fail at length scaling due to recursive instability. Further analysis of the learning pipeline shows that data coverage sets the capability ceiling; reinforcement learning improves training stability but does not extend that ceiling; and inference-time scaling boosts performance but cannot rescue length-scaling failures. The post includes the paper's original abstract and was automatically collected from arXiv.
Paper Overview
Research area: ML
Authors: Yao Tong, Jiayuan Ye, Anastasia Borovykh
Published: 2025-04-17
arXiv: 2504.13085
Summary
Whether language models can systematically generalize remains actively debated. Yet empirical performance is jointly shaped by multiple factors such as training data, training paradigms, and inference-time strategies, making failures difficult to interpret. The authors introduce a controlled synthetic environment based on shortest-path planning, a canonical composable sequential optimization problem. The setup enables clean separation of these factors and supports two orthogonal axes of generalization: spatial transfer to unseen maps and length scaling to longer-horizon problems.
The authors find that models exhibit strong spatial transfer but consistently fail under length scaling due to recursive instability. They further analyze how distinct stages of the learning pipeline influence systematic problem-solving:
- Data coverage sets the upper bound of model capability.
- Reinforcement learning improves training stability but does not extend this ceiling.
- Inference-time scaling enhances performance but cannot rescue length-scaling failures.
Read the full paper: https://arxiv.org/abs/2504.13085
---
*Automatically collected on 2026-04-18*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/177618540