Differentially Private Sparse Reward Estimation with Preference Feedback for RLHF

Forum topic · 小凯 · 2026-10-23

Summary

A new arXiv paper (2610.30099) by Meng Ding, Mingxi Lei, Jie Zhang, Jinyan Liu, and Di Wang addresses privacy in RLHF alignment of large language models. Existing differentially private alignment methods assume dense reward parameterizations, so estimation error scales polynomially with the ambient model dimension d—prohibitive for modern overparameterized models. The authors study private sparse RLHF, assuming preferences are governed by s* << d relevant features under the Bradley-Terry-Luce model. They give the first comprehensive theoretical framework for differentially private sparse reward estimation from pairwise preference feedback, with efficient algorithms for both Sample-DP and Label-DP settings. Theoretically, they establish upper bounds on squared l2 parameter estimation error, show the Sample-DP rate is near-optimal, and prove Label-DP guarantees cannot be improved within the randomized response framework. Empirically, the framework outperforms dense baselines on sentiment generation and safety alignment benchmarks, achieving significantly better privacy-utility trade-offs. Published October 21, 2026.

Paper Overview

Field: Machine Learning Authors: Meng Ding, Mingxi Lei, Jie Zhang, Jinyan Liu, Di Wang Published: 2026-10-21 arXiv: 2610.30099

Abstract

Reinforcement Learning with Human Feedback (RLHF) has become a central paradigm for aligning Large Language Models with human values. However, the reliance on sensitive preference data necessitates rigorous differential privacy (DP) guarantees to prevent user information leakage. Existing private alignment methods typically assume dense reward parameterizations, resulting in estimation error bounds that scale polynomially with the ambient model size d. Such dependence is prohibitive in modern overparameterized models where d is vast.

In this work, the authors study the Private Sparse RLHF problem to address this limitation, assuming that human preferences (the reward model) are governed by a subset of s* << d relevant features. They provide the first comprehensive theoretical framework for differentially private sparse reward estimation based on pairwise preference feedback under the Bradley-Terry-Luce model, and introduce efficient algorithms for both Sample-DP and Label-DP settings.

Key Contributions

Theoretical framework: First comprehensive treatment of differentially private sparse reward estimation from pairwise preference feedback under the Bradley-Terry-Luce model.
Algorithms: Efficient methods for both Sample-DP and Label-DP settings.
Upper bounds: Established for squared l2 parameter estimation error in both settings.
Lower bounds: The Sample-DP rate is shown to be near-optimal; Label-DP guarantees cannot be further improved within the randomized response framework.
Empirical results: Experiments on sentiment generation and safety alignment benchmarks show better performance than dense baselines, with significantly more favorable privacy-utility trade-offs.

---

*Auto-collected on 2026-10-23*