DynaSchedBench: Calibrated Dynamic Scheduling Benchmarks with a Schedule Stress Index

Forum topic · 小凯 · 2026-05-29

Summary

DynaSchedBench is a diagnostic benchmark framework for neural combinatorial optimization on the Dynamic Flexible Job Shop Scheduling Problem (DFJSP), introduced to resolve a methodological tension: static benchmarks encourage overfitting, while uncalibrated generators obscure algorithmic capability with stochastic noise. Instead of parameter sampling, the framework uses a Sequential Event-Space Calibrator (SESC) that computes a novel Schedule Stress Index (SSI) to stratify instances by difficulty. The authors show that SESC is substantially more computationally efficient than evolutionary baselines while converging reliably to target metrics. The framework integrates modular components for instance generation, snapshot-based simulation, agents, evaluation, and visualization. Using this calibrated environment, the paper reveals key limitations of LLM scheduling agents: in step-wise online decision-making, an 'observability paradox' emerges where giving agents complete structural information degrades policy performance compared with concise information. Moreover, despite heavy token expenditure, tool augmentation and refinement strategies fail to reliably improve performance, and most LLM agents cannot surpass strong heuristic baselines. The paper is available as arXiv preprint 2605.27566.

Overview

Field: AI Authors: Shijie Cao, Yuan Yuan, Jing Liu Published: 2026-05-28 arXiv: 2605.27566

Abstract

Progress in neural combinatorial optimization for the Dynamic Flexible Job Shop Scheduling Problem (DFJSP) is currently hindered by a methodological tension: static benchmarks encourage benchmark overfitting, while uncalibrated generators obscure algorithmic capability with stochastic noise. To resolve this, the authors introduce DynaSchedBench, a diagnostic framework for DFJSP that rigorously controls the instance-generation process.

Key Contributions

Sequential Event-Space Calibrator (SESC): Instead of relying on parameter sampling, the framework computes a novel Schedule Stress Index (SSI) to stratify instances by difficulty.
Efficiency: SESC is substantially more computationally efficient than evolutionary baselines while converging reliably to the target metrics.
Modular design: The framework integrates components for instance generation, snapshot-based simulation, agents, evaluation, and visualization.

Findings on LLM Scheduling Agents

Using the calibrated environment, the study reveals critical limitations of LLM agents in dynamic scheduling:

Observability paradox: In step-wise online decision-making, giving agents full structural information actually *degrades* policy performance, while concise information performs better.
Diminishing returns from tooling: Despite significant token expenditure, tool augmentation and refinement strategies do not reliably improve performance.
Heuristic baseline dominance: Most LLM agents consistently fail to surpass strong heuristic baselines.