TasteVal: A Benchmark Measuring the Experimental Research Taste of AI Systems

Forum topic · 小凯 · 2026-10-07

Summary

TasteVal is a benchmark introduced by Oliver Jaffe and Dane Sherburn that evaluates the experimental research taste of frontier AI models—defined as the ability to design experiments and interpret results. It operationalizes taste as compute efficiency: the compute multiplier needed for a model to reach expert-human scores using half the serial experimental compute. The benchmark features 8 novel, open-ended tasks representing frontier AI R&D, with evaluated models acting as researchers designing experiments while a fixed coding agent implements them, running up to 40 H100-hours or 120 wall-clock hours. Baselines come from 24 human experts (at least 2 per task). Across 20 models released between 2023 and 2026, the best model, Opus 5.5, exceeded the expert baseline with a 2.3x compute multiplier (95% CI 1.15–4.37) at roughly 1/30th the average cost per run. The frontier compute multiplier has doubled roughly every 3.0 months since December 2025 (95% CI 1.7–5.0), compared to every 14 months before. Tasks are withheld to prevent contamination. Paper: arXiv:2610.06824.

Paper Overview

Field: AI Evaluation Authors: Oliver Jaffe, Dane Sherburn Published: 2026-10-05 arXiv: 2610.06824

Abstract

We introduce TasteVal, a benchmark for evaluating the experimental research taste of frontier models. Research taste is defined as the ability to select interesting problems, design experiments, and interpret experimental results. TasteVal measures the experimental dimension of research taste: given a fixed research question, it measures a model's ability to iteratively design experiments and draw conclusions from results.

Experimental research taste is operationalized as compute efficiency—a researcher whose taste is twice that of an expert reaches the same score as an expert human using half the serial experimental compute. Experimental taste thus acts as a multiplier on experimental compute, making it a key input for forecasting AI progress.

TasteVal consists of 8 novel, challenging, open-ended tasks representative of frontier AI R&D. To isolate taste from coding ability, the evaluated model acts as a researcher iteratively designing experiments, while a fixed coding agent implements the experiments and reports results. The researcher runs until exhausting a budget of 40 H100-hours or 120 hours of wall-clock time.

24 human experts were recruited (at least 2 per task), with the best expert attempt per task serving as the baseline. The evaluation covers 20 models released between 2023 and 2026. The best model, Opus 5.5, exceeds the expert baseline with a compute multiplier of 2.3x (95% CI 1.15–4.37), at roughly 1/30th the average cost per run of the baseline humans.

The compute multiplier of frontier models has doubled roughly every 3.0 months since December 2025 (95% CI 1.7–5.0), compared to a doubling time of 14 months between 2023 and December 2025. To keep TasteVal uncontaminated, the tasks are not publicly released.

Key Findings

Best model (Opus 5.5) surpasses expert baseline: 2.3x compute multiplier (95% CI 1.15–4.37)
Model cost per run is ~1/30th that of the average baseline human expert
Compute multiplier doubling time: ~3.0 months since Dec 2025 (95% CI 1.7–5.0), vs. 14 months from 2023 to Dec 2025
Evaluation design: researcher model + fixed coding agent, up to 40 H100-hours or 120 wall-clock hours
Tasks withheld from release to prevent benchmark contamination

---

*Auto-collected on 2026-10-07.*