Cross-Entropy Games and Frost Training: Exploiting Reward Gradients in Embedding Space for LLM Policy Optimization

Forum topic · 小凯 · 2026-05-29

Summary

Frost Training is a method for improving Monte Carlo-based policy optimization for a broad family of LLM-as-a-judge tasks called Cross-Entropy Games. Introduced by Arthur Renard, Franck Gabriel, Valentin Hartmann, and colleagues in an arXiv paper (2605.27701), the approach exploits the gradient of the reward function in embedding space. Notably, this same signal underlies the Greedy Coordinate Gradient (GCG) jailbreaking technique; the authors demonstrate for the first time that it can also be used to boost model training rather than only to attack models. The method is validated using GRPO training for maximum-likelihood infilling tasks. Experimental results show that Frost Training improves a model's ability to generate high-scoring outputs, achieves higher maximum scores in best-of-k evaluation settings, and does so at increased speed compared to standard Monte Carlo policy optimization. This work connects adversarial attack techniques with reward optimization, offering a faster alternative for LLM fine-tuning on judge-scored tasks. The paper was posted on arXiv on May 28, 2026 and falls within the artificial intelligence research domain.

Paper Overview

Research area: AI
Authors: Arthur Renard, Franck Gabriel, Valentin Hartmann, et al.
Published: 2026-05-28
arXiv: 2605.27701

Abstract

We present Frost Training, a method for improving Monte Carlo-based policy optimization for a large family of LLM-as-a-judge tasks called Cross-Entropy Games. The key idea is to exploit the gradient of the reward function in embedding space. This signal is used in the Greedy Coordinate Gradient (GCG) jailbreaking technique; we demonstrate for the first time that it can also be used to boost model training. We validate our method using GRPO training for maximum-likelihood infilling. Frost Training improves the model's ability to generate high-scoring outputs, reaching higher maximum scores in a best-of-k setting, and does so at an increased speed.

Key Points

Frost Training: an improved Monte Carlo-based policy optimization method targeting the Cross-Entropy Games family of LLM-as-a-judge tasks.
Core idea: leverage the gradient of the reward function in embedding space as a training signal.
Novel repurposing: this gradient signal is already used in the GCG jailbreaking attack; the paper is the first to show it can enhance model training as well.
Validation: experiments use GRPO training for maximum-likelihood infilling.
Results: better generation of high-scoring outputs, higher maximum scores in best-of-k settings, and faster training.

---

*Auto-collected on 2026-05-29.*