Paper Overview
- Field: Machine Learning
- Authors: Kevin Kingslin, Anish Natekar, Ashutosh Ranjan
- Published: 2026-06-26
- arXiv: 2606.28294
- Extends Inverse Constitutional AI by replacing single-pass preference summarization with structured persona debate, collecting multiple competing rationales per comparison.
- Derives clearer, more comprehensive steering principles from the richer debate signals.
- Uses these principles to guide decision modeling with both LLM-based judges and decision-tree judges.
- Empirically shows improved preference prediction and more preferred constitutions versus deliberative prompting and principle-based baselines.
- Evaluated on MuCE-Pref and LiTBench, covering multiple creative task categories.
- Paper: https://arxiv.org/abs/2606.28294
Abstract (Original)
Preference-based alignment often struggles to capture the reasoning that underlies human judgments. Many evaluations rely on multiple interacting criteria, yet pairwise labels reveal only the final choice rather than the considerations that shape preferences. Inverse Constitutional AI (ICAI) improves interpretability in decision making by summarizing preferences into natural-language principles, but its single-pass explanations miss much of the nuance involved in complex decisions. We introduce Democratic ICAI, a novel approach that gathers multiple competing rationales through structured persona debate, offering a broader and more expressive account of the factors influencing each comparison. From these richer signals, we derive clearer and more comprehensive steering principles and use them to guide decision modeling through LLM-based and decision-tree judges. Experiments on the creative preference benchmarks MuCE-Pref and LiTBench, across multiple creative task categories, demonstrate that Democratic ICAI produces more faithful preference structures. Relative to deliberative prompting and principle-based baselines, it improves average preference prediction across tasks while producing constitutions preferred by LLM annotators.