Paper Overview
- Field: AI
- Authors: Arthur Renard, Franck Gabriel, Valentin Hartmann, et al.
- Published: 2026-05-28
- arXiv: 2605.27701
- Better ability to generate high-scoring outputs
- Higher maximum scores in a best-of-k setting
- Faster convergence compared to baselines
Abstract (Original)
> We present Frost Training, a method for improving Monte Carlo-based policy optimization for a large family of LLM-as-a-judge tasks called Cross-Entropy Games. The key idea is to exploit the gradient of the reward function in embedding space. This signal is used in the Greedy Coordinate Gradient (GCG) jailbreaking technique; we demonstrate for the first time that it can also be used to boost model training. We validate our method using GRPO training for maximum-likelihood infilling. Frost Training improves the model's ability to generate high-scoring outputs, reaching higher maximum scores in a best-of-k setting, and does so at an increased speed.
Summary
Frost Training improves Monte Carlo-based policy optimization for Cross-Entropy Games—a broad family of LLM-as-a-judge tasks. Its core insight is to use the gradient of the reward function in embedding space as an additional training signal. Notably, this signal underlies the GCG (Greedy Coordinate Gradient) jailbreaking attack; the paper is the first to show it can instead accelerate model training.
Validation: Experiments use GRPO training on maximum-likelihood infilling.
Findings: