Paper Overview
Field: Machine Learning Authors: Di Yang Shi, W. Bradley Knox Published: 2026-08-13 arXiv: 2508.03415
Summary
This paper proposes a formal pipeline that allows non-experts to instantiate and iterate on human-aligned reward functions—reward functions that comply with a given preference ordering over trajectories. Given a natural-language description of a task, the process produces a linear reward function in three steps:
1. Distill task objectives into a set of primitive objectives and derive measurable outcome variables that capture these primitives. The paper describes a guided workflow for this first step. 2. Select reward terms: choose a causally representative subset of the outcome variables. This is reduced to a minimum-cost partial coverage problem on a causal DAG, solvable in polynomial time via max-flow. 3. Fit weights: weight fitting is framed as a convex feasibility problem, iteratively shrunk through preference queries and solved via existing separation-oracle methods.
Key Contributions
- A formal process from natural-language task descriptions to linear reward functions, aimed at non-experts.
- Polynomial-time reward-term selection via minimum-cost partial coverage on causal DAGs.
- A preference-guided convex feasibility formulation for weight fitting.
- To the authors' knowledge, the first reward-design method that preserves a deterministic conflict-free feasible weight region, shrinking to a desired tolerance within O(n log κ) preference queries via a separation oracle.
Original Abstract (excerpt)
> We present a formal process to enable non-experts to instantiate and iterate on human-aligned reward functions...
Full paper: https://arxiv.org/abs/2508.03415
*Auto-collected on 2026-08-14*