Paper Overview
Field: Computer Vision Authors: Archer Moore, Mingming Gong, Liam Hodgkinson Published: 2026-06-25 arXiv: 2606.27305
Abstract
Reinforcement learning from human feedback (RLHF) for 3D generation is now established across a number of works, but most existing pipelines optimise explicit surface representations, often by converting radiance fields into meshes and training heavily on surface-supervised data. The authors instead fine-tune a pretrained 3D-aware generative model directly from a learned reward over radiance-field density (sigma) values, with no externally supplied mesh or shape prior. The reward model requires no pretraining, trains easily on a small set of preference samples, and yields robust improvement in 3D geometry.
Working on an unconditional 3D-aware face GAN (EG3D), the reward reads the continuous 3D density field of the neural radiance field (NeRF) directly and supplies a geometry-only learning signal, requiring neither text conditioning, mesh extraction, nor multi-view rendering. A density consistency constraint reshapes geometry while keeping 2D appearance qualitatively similar, at a measurable but bounded distributional cost (FID-50k rises from 4.09 to 6.66). As a proof of concept, a generator fine-tuned from a single annotator's preferences produces face geometry preferred by users in 74.4% of pairwise comparisons.
Key Results
- Direct reward over NeRF density values — no meshes or shape priors needed
- Reward model trains without pretraining on a small preference dataset
- FID-50k trade-off: 4.09 → 6.66 while improving geometry
- 74.4% human preference win rate in pairwise comparisons using a single annotator
*Auto-collected on 2026-06-28*