Split the Labor: Separating Evidence Interpretation from Decision Aggregation in LLM Systems

Forum topic · 小凯 · 2026-08-18

Summary

This arXiv paper (2508.08538) by Zhelun Wu argues that systems prompting a language model to reach conclusions from many sources conflate two operations with different requirements: interpreting individual sources (rewarding capacity and context) and combining interpretations (rewarding fixed arithmetic, cross-instance comparability, and the option to return nothing). The author proposes a four-field evidence tuple—hypothesis, reliability bucket, rationale, provenance—as the interface, showing that fixing it determines both halves. The separation exposes a failure mode called count-scale drift: thresholding a sum of unnormalized weights is posterior thresholding, but its operating point slides with the number of consulted sources, and no threshold reconciles voting rules with posteriors when source reliabilities differ. Pooling calibrated log-likelihood ratios fixes both problems arithmetically, applying beyond LLMs to score-summing triage engines, diagnostic panels, and additive multi-signal detectors. On a longitudinal corpus, the same split instantiated twice yields gains: post-hoc reading and pre-outcome learning, where a small sequence encoder plus a tree ensemble with censored survival loss achieves 0.921 AUPRC versus a 0.805 handcrafted baseline. The paper separates what transfers across domains from what must be re-estimated, stating five falsifying predictions and three negative results.

This post shares a recent NLP paper (arXiv:2508.08538) by Zhelun Wu, published 2026-08-17.

Key points

The problem: Systems that ask a language model to reach a conclusion from many sources usually concatenate them into one prompt, conflating two operations with different requirements.
*Interpreting a source* rewards capacity and context.
*Combining interpretations* rewards fixed arithmetic, comparability across instances, and the option to return nothing.
The interface: Once separated, the design problem becomes the interface between the halves. The paper proposes a four-field evidence tuple: hypothesis, reliability bucket, rationale, provenance. Fixing this tuple determines both halves.
Failure mode — count-scale drift: Thresholding a sum of unnormalized weights is exactly posterior thresholding, but at an operating point that slides with the number of sources consulted. The slide grows with reader reliability. When source reliabilities differ, voting rules and posteriors rank instances differently, and no single threshold can reconcile them.
The fix: Pooling calibrated log-likelihood ratios resolves both issues. The fix is arithmetic rather than architectural, and applies to a class of rules beyond language models: score-summing triage engines, diagnostic panels scored by count positives, and additive multi-signal detectors.
Empirical instantiation: The principle is instantiated twice on a longitudinal corpus—once after outcomes are determined, once before. The same split helps in both, at different granularities: the first in reading, the second in learning. In the latter, a small sequence encoder with a simple auxiliary target plus a tree ensemble carrying a censored survival loss reaches 0.921 AUPRC, versus a handcrafted baseline of 0.805.
Scope: The paper separates what can transfer across domains from what must be re-estimated per domain, states five predictions that would falsify the framework, three negative results, and which comparisons remain confounded.

Original abstract (excerpt)

> Systems that ask a language model to reach a conclusion from many sources usually concatenate them into one prompt. This conflates two operations with different requirements. Interpreting a source rewards capacity and context. Combining interpretations rewards fixed arithmetic, comparability across instances, and the option to return nothing. Once separated, the design problem becomes the interface between them...

Full paper: arXiv:2508.08538

Split the Labor: Separating Evidence Interpretation from Decision Aggregation in LLM Systems

Summary

Key points

Original abstract (excerpt)

Tags