Jev for Scientific Decisions: Evaluating Semantic Choices in Scientific Workflows

Forum topic · 小凯 · 2026-09-23

Summary

This paper evaluates Jev as a semantic decision component for scientific workflows, where choosing among known relations (e.g., whether observations share a culture batch, treatment, or reference standard) must occur before deterministic calculation. The authors—Boyuan Deng, Shuyi Fan, Hongyang Zhang, and Xinhong Xie—built a harness that follows Jev's documented guidance and assigns arithmetic to code. The study compares twelve model configurations across twenty source-grounded semantic Choices drawn from ten scientific cases, each repeated five times, measuring semantic selections, downstream outputs, and final claim labels separately. Jev matched five other configurations at complete semantic correctness and achieved the lowest observed median latency among successful responses. Across three comparison models, seven wrong choices on a culture-history question altered downstream counts while preserving correct final labels, showing that evaluating such components requires examining the relations and quantities a workflow will reuse. The arXiv identifier is 2609.24965, published September 21, 2026.

Paper Overview

Field: NLP
Authors: Boyuan Deng, Shuyi Fan, Hongyang Zhang, Xinhong Xie
Published: 2026-09-21
arXiv: 2609.24965

Abstract (English, from source)

Scientific workflows often require choosing among known relations before a deterministic calculation can proceed. Whether observations share a culture, treatment or reference standard can change the scientific meaning of the resulting count or comparison. We evaluate Jev as a semantic decision component using a harness that follows its documented guidance and assigns arithmetic to code. The study compares twelve model configurations on twenty source-grounded Choices across ten scientific cases, each repeated five times. We measure semantic selections, downstream outputs and final claim labels separately. Jev matched five other configurations at complete semantic correctness and achieved the lowest observed median latency among successful responses. Across three comparison models, seven wrong choices on a culture-history question changed downstream counts but preserved correct final labels. These results establish a useful role for Jev in pre-specified scientific decision tasks and indicate that evaluating that role requires examining the relations and quantities a workflow will reuse.

Key Findings

Scientific workflows depend on semantic choices about known relations before any deterministic computation runs.
Jev was evaluated with a harness following its documentation, delegating arithmetic to code.
Benchmark: 12 model configurations, 20 source-grounded Choices, 10 scientific cases, 5 repetitions each.
Semantic selections, downstream outputs, and final claim labels were measured independently.
Jev tied five other configurations on complete semantic correctness and had the lowest observed median latency among successful responses.
In three comparison models, seven wrong choices on a culture-history question changed downstream counts while the final labels remained correct.

*Auto-collected on 2026-09-23.*