rTaste: Do Research Agents Have Taste? 415 Experiments Say They Can Produce but Not Select

A Question Nobody Asked

In AI labs, agents can already write code, run experiments, read papers, and write reports. Frontier labs claim agents write most of their code. But one step is never delegated to agents: deciding what to research.

METR's risk assessments of four frontier labs found that none let agents set research agendas or make budget decisions.

Why?

Intuitively, "choosing a research topic" requires something hard to articulate — taste. Judging which experiment is worth running before the results come in. This is not a question of "can it be done," but "is it worth doing."

Art Kanke's (University of Minnesota) paper rTaste asks exactly this: Do research agents have taste?

Experimental Design: Agents as PI

The design is elegant:

Seed papers: 7 recent NLP papers as "seeds."

Agent lineup: 6 frontier models (GPT-4, Claude, Gemini, etc.), each acting solo as a research agent or together as a team.

Task: Each agent reads a seed paper and proposes Minimum Viable Experiments (MVEs) — "if you could only run one experiment to validate or extend this paper, what would it be?"

Key steps: 1. Agents propose multiple MVEs and rank them 2. Agents predict whether each MVE will pass their own threshold 3. An independent executor runs all MVEs — including the lowest-ranked ones 4. Three model auditors judge the results

415 MVEs, all executed and audited.

Finding 1: Agents Can Produce, but Quality Varies

Of the 415 MVEs:

• 194 were judged "reasonable, valuable, novel" — a 47% pass rate
• Only 73 were also judged "beyond the seed paper and non-obvious" — an 18% true high-quality rate

In other words, nearly half of agent-proposed experiments are "plausibly good experiments," but fewer than one in five truly go beyond the original paper in a non-obvious way.

Agents' production capability isn't bad — but "producing" and "judging production quality" are two different things.

Finding 2: Agents' Ranking Ability Is Close to Random

This is the most painful finding.

Agents ranked their own MVEs — which is most worth running. Researchers compared agents' "first choice" with the auditors' "most valuable experiment."

Result: Agents' first choice is indeed better than random selection. But —

It is no better than "the first experiment mentioned in the paper."

Seed papers typically mention "future work" or "directions to validate." The agent's top-ranked experiment showed no significant advantage over these already-mentioned directions.

In other words, agents' "taste" ≈ "propose whatever the paper mentions."

Finding 3: Prediction Exists but Is Limited

Agents predicted whether each MVE would pass its threshold:

• Predictions beat the base rate — models do have some predictive ability
• But no better than a fresh instance of the same model given only the paper and registration info

This means the context agents accumulated while proposing MVEs (thinking, reasoning, discussion) did not improve prediction accuracy. A model instance that didn't participate in the deliberation predicted just as well.

The thinking process added no predictive power. This is a blow to the assumption that "agent thinking is valuable."

Finding 4: Teams Are More Expensive but Not Better

Multi-agent teams (discussing like research teams):

• Cost more (more API calls)
• No significant overall quality improvement

This contradicts the intuition that "human teams beat individuals." Possibly because agent discussions lack genuine "collision of perspectives" — they tend to agree with each other rather than challenge each other.

Why Is "Taste" So Hard?

The paper's discussion contains a sharp observation:

> "The step that has not been delegated is close to what researchers call taste."

Taste is judging "which direction is worth it" before results arrive. Such judgment requires:

• Deep domain understanding (what's been done, what's obvious)
• Intuition for "interesting" (what makes eyes light up)
• A sense of "feasibility" (can it produce results under limited resources?)

Agents are okay on the first point (can produce plausible experiments), weak on the second (ranking ≈ random), and have a vague sense on the third (predictions beat base rate) that deliberation doesn't strengthen.

Taste is not about "being able to do" but about "being able to choose." Choice requires metacognition — knowing what you know, what you don't, and what's worth knowing. Agents currently lack this.

My Take: The "Generation-Judgment Gap" in Research Agents

This paper points to a recurring pattern: a systematic gap between agents' generation ability and judgment ability.

• Agents can write code but can't necessarily judge whether it's correct (external audit needed)
• Agents can propose experiments but can't necessarily judge which are worth running (this paper)
• Agents can generate replies but can't necessarily judge whether they're safe (safety alignment needed)

This "generation-judgment gap" and the "judgment-gate decoupling" are two sides of one logic: generation and judgment are different capabilities requiring different mechanisms.

The most striking finding is that "thinking adds no predictive power." It suggests agent "thinking" today is more like "rephrasing" than "real reasoning" — restating what it already knows without producing new cognition.

A practical lesson: if you use agents for research, don't just follow their rankings. Run all the proposed experiments (including the lowest-ranked ones) — the last-ranked may be as good as the first.

That's what 415 experiments tell us: agents can produce, but cannot select.

---

Paper: arXiv:2610.40488 Code and data: github.com/ArtKanke/rTaste