Can LLMs in Draft-Verify-Revise Pipelines Resolve Deictic Ambiguity?

Forum topic · 小凯 · 2026-09-15

Summary

Draft-verify-revise is a common LLM orchestration pattern for scaling inference-time compute, but as context cascades between stages, models may resolve context-dependent expressions like "previous" differently, causing a deictic shift. Obinna I. Ekekezie (arXiv:2609.12162) studies this with a synthetic dataset of 10 base examples in three conditions, testing six models from three providers across 21 reasoning-effort configurations using e-value sequential tests, plus an ablation removing error-classification labels from grader feedback and an independent LLM analysis of the meta-evaluator's stated rationales. Balanced accuracy ranged from 0.156 (below chance) to near-perfect: GPT-5.2 rose from 0.156 without context reasoning to 0.942 at the highest effort, while Gemini 3 Pro stayed above 0.94 at every level, outperforming GPT-5.2's xhigh results at roughly 5% of the per-trial cost. When the meta-evaluator erred, it relied on surface cues rather than operational reasoning. Context engineers should be alert to deictic shifts and make referents explicit at every stage.

Overview

Field: NLP Author: Obinna I. Ekekezie Published: 2026-09-15 arXiv: 2609.12162

Abstract

Draft-verify-revise is a common LLM orchestration pattern for scaling inference-time compute. One LLM drafts, a second critiques the draft and provides feedback, and a third uses that feedback to revise the draft into the final output. As context cascades between stages, LLMs at different stages can resolve a context-dependent expression such as "previous" differently. When that happens, the expression undergoes a deictic shift, a change in what it refers to.

This phenomenon was studied with a synthetic dataset of 10 base examples, each rendered in three conditions. Holding shared components constant, the conditions varied whether the draft stage LLM (the assistant) or the verify stage LLM (the grader) resolved the expression correctly, and how much independent reasoning the revise stage LLM (the meta-evaluator) needed to determine which reading was correct. Six models from three providers were tested across 21 reasoning-effort configurations, using e-values for sequential testing. The study included one main experiment and an ablation removing error-classification labels from grader feedback; an independent LLM also analyzed the meta-evaluator's stated rationales for each error.

Key Findings

Balanced accuracy (unweighted mean of sensitivity and specificity) ranged from 0.156 (below chance) to near-perfect.
GPT-5.2 improved from 0.156 without context reasoning to 0.942 at the highest reasoning effort.
Gemini 3 Pro stayed above 0.94 at every level; at low reasoning effort it outperformed GPT-5.2 at xhigh effort, at roughly 5% of the per-trial cost.
When the meta-evaluator erred, it tended to rely on surface cues rather than operational reasoning.

Practical Takeaway

Context engineers building draft-verify-revise pipelines should watch for deictic shifts and make referents explicit at each stage.

--- *Auto-collected on 2026-09-15*