Improving Generative Ad Text on Facebook Using Reinforcement Learning
- Paper: Improving Generative Ad Text on Facebook using Reinforcement Learning (arXiv, July 2025)
- Authors: Daniel R. Jiang, Alex Nikulkov, Yu-Chia Chen, Yang Bai, Zheqing Zhu (Meta)
- Source: zhichai.net forum entry, Verticals / Response Generation section
- The paper presents a production-scale application of reinforcement learning (RL) to improve advertisement text generated by large language models on Facebook.
- Rather than fine-tuning only on static datasets, the system uses a large language model to generate ad text variants and learns from real-world feedback signals collected on the live platform.
- The approach is described as one of the first large-scale industrial deployments of RL fine-tuning for generative ad text creation.
- For researchers: a rare documented example of RL fine-tuning for generative models evaluated with real online feedback rather than offline metrics alone.
- For engineers: lessons on reward design, deployment constraints, and safety guardrails for LLMs in advertising products.
- For product teams: evidence that advertiser-side engagement signals can drive iterative improvement of generative creative tools.
- Neural headline generation: A comprehensive survey (2025)
- Beyond Relevant Documents: A Knowledge-Intensive Approach for Query-Focused Summarization (arXiv 2408.10357)
- Cite Before You Speak: Enhancing Context-Response Grounding in E-commerce (arXiv 2503.04830)
- Applying Deep Learning to Ads Conversion Prediction in Last Mile Delivery (arXiv 2502.10514)
- Automated Query-Product Relevance Labeling using Large Language Models (arXiv 2502.15990)
Key points
Background
Advertising content creation is a natural fit for generative LLMs, but offline benchmarks do not capture how advertisers actually use, edit, or adopt generated copy in production. The authors frame ad-text generation as a sequential decision-making problem, where the reward is derived from advertiser engagement with generated suggestions on Facebook's ads platform. This connects to the broader trend of applying RLHF-style training beyond chat assistants to vertical industry applications.
Approach
The system follows a general pipeline:
1. Generation: An LLM produces candidate ad text (e.g., headlines and primary text) conditioned on advertiser inputs and campaign context. 2. Deployment: Candidate generations are surfaced to advertisers in the production ads-creation flow. 3. Reward signal: Advertiser interactions with the generated text (such as adoption/acceptance behavior) are logged as rewards. 4. Learning: A reinforcement learning algorithm optimizes the policy (the LLM) against these real engagement-based rewards, improving the quality and usefulness of future generations.
The paper discusses the practical requirements of such a deployment, including latency and cost budgets, content safety and policy compliance for ad text, and the engineering infrastructure needed to close the loop between generation, feedback collection, and model updates.