Paper Overview
Field: NLP Author: Timothy Kassis Published: 2026-10-05 arXiv: 2610.00084
Summary
Detailed profession-specific system prompts raise token use and estimated cost per response without a consistent accuracy gain. The authors evaluate Scientific Agents, an open-source corpus of 503 profession-specific profile prompts, running Gemini 3.8 Flash via OpenRouter in the Pi agent harness. Matched profiles are compared against four controls:
- A minimal baseline ("You are a helpful assistant")
- The profile's opening role sentence
- A generic scientific rigor guide
- A profile from an unrelated domain
- Text benchmarks: Across nine text-based science benchmarks (4,531 sampled questions, 100 matched profiles), 4,488 items completed all five conditions after API-error retries, scored with automated rule-based grading. The average profile-baseline accuracy difference was -0.6 percentage points (95% interval [-1.5, +0.2]); no benchmark showed a statistically clear improvement.
- Cost: Matched profiles produced 1.5-2.3x more output tokens than baseline and cost 2.2-4.5x more per successful call.
- Tool-use tasks: On 60 tool-using BioMysteryBench bioinformatics questions (three runs each), average solve rate was 46.7% with profiles vs 56.7% baseline (-10.0 points, 95% interval [-16.7, -3.3]), mainly because profiles caused more frequent token/time-limit interruptions.
- Unexpected reliability benefit: Frequent API dropouts on SuperGPQA meant only 54.0% of items retained a one-pass correct answer with the short baseline, versus 71.6% with profiles. Generic and mismatched prompts were equally reliable, indicating the gain comes from prompt length or format rather than domain expertise.
Findings
Conclusion
For the tested model and tasks, loading full profession-specific profiles by default does not improve accuracy and is substantially more expensive. Whether selectively retrieving profile sections, or open-ended tasks, would change these conclusions remains to be tested.
---
*Auto-collected on 2026-10-05*