FinAutoRubric: Expert-Guided Automatic Rubric Generation for Evaluating Finance Research Agents

Forum topic · 小凯 · 2026-09-30

Summary

FinAutoRubric is a framework for evaluating finance research agents using automatically generated rubrics guided by expert specifications. Instead of costly fixed, per-item rubrics used in expert-reviewed finance benchmarks, experts define reusable evaluation guidance that is enforced through prompts and code-executed rules, with a Task Bank of reusable criteria carried across tasks. In long-horizon loops, a writer agent researches each expected value, a reviewer agent verifies it, and failures escalate to human review. On three expert-authored finance benchmarks, FinAutoRubric's rubrics track expert scores as closely as the strongest evaluation generators while covering more expert-specified criteria; its scores align with human ratings, and internal analysts prefer its rubrics in blind comparisons. The released 100-query FinAutoRubric benchmark derives from internal analysts' critical questions across 78 tasks and eight asset classes, showing that rubrics generated by earlier models still leave room for improvement with newer models.

论文概要

Research area: Machine Learning
Published: 2026-09-28
arXiv: 2609.35744
Authors: Hoyoung Lee, Suyeol Yun, Jack Haverty, Yunju Cho, Meesong Kim, Daekyung Park, Sumin Kim, Jihoon Kwon, Jasmine Jia Geng, Andrew Chin, Yin Luo, Edward Tong, Yu Yu, Zach Golkhou, Minkyu Kim, Igor Halperin, Young Cha, Alejandro Lopez-Lira, Chanyeol Choi, Yongjae Lee

Abstract

Evaluating finance research agents requires rubrics that reflect expert standards and fix the values correct as of an information cutoff. Expert-reviewed finance benchmarks rely on fixed, per-item rubrics, which are costly to extend and cannot encode each institution's own standard. In FinAutoRubric, experts specify reusable evaluation guidance, while agents and code carry out query-specific rubric generation, review, and validation. This expert guidance governs every agent, as prompts and as rules that code enforces, and a Task Bank of reusable criteria carries it across tasks. In long-horizon loops that follow the expert guidance, a writer agent researches every expected value and a reviewer agent verifies it, and failures escalate to a human.

Key findings

On three expert-authored finance benchmarks, FinAutoRubric's rubrics track expert scores as closely as the strongest evaluation generators, while covering more of the expert rubrics' expected criteria.
Its scores align with human ratings, and internal analysts prefer its rubrics in blind comparisons.
The released 100-query FinAutoRubric benchmark is built from internal analysts' critical questions spanning 78 tasks and eight asset classes.
Results indicate that rubrics generated by earlier models still leave room for improvement by newer models.

*Auto-collected on 2026-09-30.*