Frost Training: Exploiting Reward Gradients in Cross-Entropy Games for LLM Policy Optimization

Forum topic · 小凯 · 2026-05-29

Summary

Researchers Arthur Renard, Franck Gabriel, Valentin Hartmann, and colleagues introduce Frost Training, a method that improves Monte Carlo-based policy optimization for a broad family of LLM-as-a-judge tasks they call Cross-Entropy Games. The central idea is to exploit the gradient of the reward function in embedding space. This gradient signal is already used in the Greedy Coordinate Gradient (GCG) jailbreaking technique, and the paper demonstrates for the first time that it can also be leveraged to boost model training. The authors validate their approach with GRPO training on a maximum-likelihood infilling task. Results show that Frost Training improves the model's ability to generate high-scoring outputs, achieves higher maximum scores in a best-of-k setting, and trains faster than the baseline. The preprint is available on arXiv as 2605.27701, with a note that it was collected automatically on 2026-05-29 from the zhichai.net forum.

Paper Overview

Research area: AI
Authors: Arthur Renard, Franck Gabriel, Valentin Hartmann, et al.
Published: 2026-05-28
arXiv: 2605.27701

Abstract (translated)

This paper proposes Frost Training, an improved method for Monte Carlo-based policy optimization, targeting a large family of LLM-as-a-judge tasks called Cross-Entropy Games. The core idea is to exploit the gradient of the reward function in embedding space. This signal is already used in the GCG jailbreaking technique; this paper is the first to show that it can also be used to enhance model training. The method is validated using GRPO training for maximum-likelihood infilling. Frost Training improves the model's ability to generate high-scoring outputs, reaches higher maximum scores in a best-of-k setting, and does so at increased speed.

Original Abstract

We present Frost Training, a method for improving Monte Carlo-based policy optimization for a large family of LLM-as-a-judge tasks called Cross-Entropy Games. The key idea is to exploit the gradient of the reward function in embedding space. This signal is used in the Greedy Coordinate Gradient (GCG) jailbreaking technique; we demonstrate for the first time that it can also be used to boost model training. We validate our method using GRPO training for maximum-likelihood infilling. Frost Training improves the model's ability to generate high-scoring outputs, reaching higher maximum scores in a best-of-k setting, and does so at an increased speed.

*Automatically collected on 2026-05-29.*