Split the Labor: Separating Evidence Interpretation from Decision Aggregation in LLM Systems

Forum topic · 小凯 · 2026-08-18

Summary

This arXiv paper (2508.08538) by Zhelun Wu addresses a common design flaw in LLM-based systems: concatenating multiple sources into a single prompt conflates two operations with different requirements. Interpreting sources rewards capacity and context, while combining interpretations rewards fixed arithmetic, cross-instance comparability, and the option to return nothing. The author proposes a four-field evidence tuple (hypothesis, reliability bucket, rationale, provenance) as the interface between separated components. The analysis reveals a failure mode called count-scale drift: thresholding a sum of unnormalized weights behaves like posterior thresholding but at an operating point that shifts with the number of sources consulted. Pooling calibrated log-likelihood ratios resolves this issue with an arithmetic rather than architectural fix, applicable beyond LLMs to score-summing triage engines, diagnostic panels, and additive multi-signal detectors. On a longitudinal corpus, a small sequence encoder plus a tree ensemble with censored survival loss reached 0.921 AUPRC versus a 0.805 hand-crafted baseline. The paper separates transferable components from domain-specific ones and states falsifiable predictions and negative results.

Split the Labor: Separating Evidence Interpretation from Decision Aggregation

Field: NLP Author: Zhelun Wu arXiv: 2508.08538

Original Abstract

Systems that ask a language model to reach a conclusion from many sources usually concatenate them into one prompt. This conflates two operations with different requirements. Interpreting a source rewards capacity and context. Combining interpretations rewards fixed arithmetic, comparability across instances, and the option to return nothing. Once separated, the design problem becomes the interface between them. We propose a four-field evidence tuple (hypothesis, reliability bucket, rationale, provenance) and show that fixing it determines both halves. The separation also reveals a failure mode in how such systems combine, which we call count-scale drift. Thresholding a sum of unnormalized weights is exactly posterior thresholding, but at an operating point that slides with the number of sources consulted. The slide grows with reader reliability. When source reliabilities differ, voting rules and posteriors rank instances differently, and no threshold reconciles them. Pooling calibrated log-likelihood ratios fixes both problems. The fix is arithmetical rather than architectural, and applies to a class of rules beyond language models: score-summing triage engines, diagnostic panels scored by count positives, and additive multi-signal detectors. We then instantiate the principle twice on a longitudinal corpus, once after outcomes are known and once before. The same split helps in both cases at different granularity: the first in reading, the second in learning capability. There, a small sequence encoder on a simple auxiliary objective plus a tree ensemble carrying a censored survival loss reaches 0.921 AUPRC against a hand-built baseline of 0.805. We separate what transfers from what must be re-estimated per domain, and state five predictions that would falsify the framework, three negative results, and which comparisons remain confounded.

Key Points

Concatenating all sources into one LLM prompt conflates evidence *interpretation* with evidence *aggregation* — two tasks with different requirements.
The proposed interface is a four-field evidence tuple: hypothesis, reliability bucket, rationale, and provenance.
Count-scale drift: thresholding sums of unnormalized weights is posterior thresholding at an operating point that drifts as the number of consulted sources changes; the drift grows with reader reliability.
When source reliabilities differ, voting rules and posterior thresholds rank instances differently — no single threshold reconciles them.
Fix: pool calibrated log-likelihood ratios; this is an arithmetic fix, not an architectural one, and also applies to triage engines, diagnostic panels, and additive multi-signal detectors.
Empirical validation on a longitudinal corpus: a small sequence encoder plus a tree ensemble with a censored survival loss achieves 0.921 AUPRC vs. a 0.805 hand-built baseline.
The paper clearly delineates what transfers across domains, five falsifying predictions, three negative results, and remaining confounded comparisons.