Can LLMs in Draft-Verify-Revise Pipelines Resolve Deictic Ambiguity?

Forum topic · 小凯 · 2026-09-15

Summary

Draft-verify-revise is a common LLM orchestration pattern for scaling inference-time compute: one LLM drafts, a second critiques the draft, and a third revises it into the final output. As context cascades between stages, models at different stages can resolve context-dependent expressions like "previous" differently, producing a deictic shift—a change in what the expression refers to. This paper studies the phenomenon with a synthetic dataset of 10 base examples rendered in three conditions, varying whether the draft-stage LLM (assistant) or verify-stage LLM (grader) resolved the expression correctly, and how much independent reasoning the revise-stage LLM (meta-evaluator) needed. Six models from three providers were tested across 21 reasoning-effort configurations using e-value sequential testing, including a main experiment and an ablation removing misclassification labels from grader feedback. Balanced accuracy ranged from 0.156 (below chance) to near-perfect. GPT-5.2 rose from 0.156 with no contextual reasoning to 0.942 at maximum reasoning effort, while Gemini 3 Pro stayed above 0.94 at every level; Gemini 3 Pro at low effort outperformed GPT-5.2 at xhigh while costing roughly 5% as much per trial. When the meta-evaluator erred, it tended to rely on surface cues rather than operational reasoning. Context engineers should watch for deictic shifts and make referents explicit at each pipeline stage.

Paper Overview

Field: NLP Author: Obinna I. Ekekezie Published: 2026-09-15 arXiv: 2609.12162

Abstract

Draft-verify-revise is a common LLM orchestration pattern for scaling inference-time compute. One LLM drafts, a second critiques the draft and provides feedback, and a third uses that feedback to revise the draft into the final output. As context cascades between stages, LLMs at different stages can resolve a context-dependent expression such as "previous" differently. When that happens, the expression undergoes a deictic shift, a change in what it refers to.

This phenomenon was studied with a synthetic dataset of 10 base examples, each rendered in three conditions. Holding shared components constant, the conditions varied whether the draft-stage LLM (the assistant) or the verify-stage LLM (the grader) resolved the expression correctly, and how much independent reasoning the revise-stage LLM (the meta-evaluator) needed to judge which interpretation was correct. Six models from three providers were tested across 21 reasoning-effort configurations using e-values for sequential testing, in a main experiment plus an ablation removing the misclassification label from grader feedback. An independent LLM also analyzed the meta-evaluator's stated reasoning in each misjudgment.

Key Findings

Balanced accuracy (unweighted mean of sensitivity and specificity) ranged from 0.156 (below chance) to near-perfect.
GPT-5.2 improved from 0.156 with no contextual reasoning to 0.942 at maximum reasoning effort.
Gemini 3 Pro stayed above 0.94 at every reasoning-effort level; at low effort it outperformed GPT-5.2 at xhigh, at roughly 5% of the per-trial cost.
When the meta-evaluator made errors, it tended to rely on surface cues rather than operational reasoning.

Takeaway for Practitioners

Context engineers implementing draft-verify-revise pipelines should be alert to deictic shifts and make referents explicit at every stage.

--- *Auto-collected on 2026-09-15*