The Instruction Tax: Why Base Models Can Beat Instruction-Tuned Models at Factual Recall

Forum topic · ✨步子哥 · 2026-10-21

Summary

A team from the American University of Beirut and Sorbonne Université won the AKBC LM-KBC 2026 Shared Task (0.6836, vs. 0.2964 baseline) with a counterintuitive finding: instruction tuning damages factual recall, a phenomenon they call the 'instruction tax.' Using gemma-3-27b-pt as a base model with a text completion interface scored 12 points higher than the instruction-tuned gemma-3-27b-it chat interface on knowledge graph completion. Sampling with the base model 64-256 times per query and keeping only frequently repeated answers yielded high precision and natural abstention under F1 scoring. The paper argues the instruction tax is a weight-level trade-off, not recoverable via prompt engineering, and cautions that scalar notions of 'model capability' hide this dimension-specific cost.

The Instruction Tax: Base Models Outperform Their Instruction-Tuned Versions at Factual Recall

Instruct tuning a base model into a chat model looks like a strict upgrade: better dialogue, instruction following, safety refusals. But at the AKBC LM-KBC 2026 Shared Task, Christophe Khalil (American University of Beirut) and Aya Mourad (Sorbonne Université) found that the base version of the same model beat the chat version by 12 percentage points on factual recall. Their solution scored 0.6836 on the leaderboard—more than double the 0.2964 baseline. They call this phenomenon the "instruction tax" (instruction tax).

The Task

Given a (subject, relation) pair, generate all possible objects—for example, (Albert Einstein, placeOfDeath) → Princeton, New Jersey, USA. Constraints: models up to 32B parameters, no retrieval, no fine-tuning. Pure parametric knowledge.

The Counterintuitive Finding

The team first used the chat interface of gemma-3-27b-it, scoring 0.5681. Switching to the base model gemma-3-27b-pt via text completion raised this to 0.6836. Across model families, base vs. instruct "sampling ceilings" (fraction of correct answers covered in 256 samples):

| Model | base (completion) | instruct (chat) | Delta | |---|---|---|---| | gemma-3-12b | 0.370 | 0.320 | +0.050 | | gemma-3-27b | 0.590 | 0.580 | +0.010 | | gemma-3-27b (full ceiling) | 0.880 | 0.590 | −0.290 | | Mistral-Nemo-2407 | 0.630 | 0.660 | −0.030 |

Instruction tuning changed the output distribution: instead of continuing the most likely facts, the model "answers"—adding qualifiers, hedging, giving partial answers—harmful in exhaustive-recall tasks. The authors argue this is a weight-level change, not recoverable through prompt engineering.

The Method: Ask Many Times, Keep What Repeats

1. Use the base model with a completion interface, few-shot examples as prefix. 2. Sample 64–256 times per query. 3. Keep answers that repeat often enough (per-query thresholds calibrated on training data). 4. Special handling: grouped sampling for large answer sets (awardWonBy); a dead-or-alive check before predicting placeOfDeath.

This repetition filter is naturally an abstention mechanism—uncertain answers don't repeat, so they're filtered out, protecting F1. As the paper puts it: "where the best answer is often empty, knowing when not to answer helps almost as much as knowing more."

Takeaways

Model capability is not a scalar: instruction tuning improves instruction-following while degrading factual recall.
"Knowing a fact" and "willing to output it" are separate mechanisms; instruction tuning changes the gate, not the judgment.
Caveat: LM-KBC rewards exhaustive recall. In conversational settings, instruction following matters more—but for knowledge graph construction, fact extraction, and retrieval, the instruction tax is a real factor.

Paper: The Instruction Tax on Factual Recall: Base Models Can Beat Their Instruction-Tuned Counterparts Code: https://github.com/aiamourad/lm-kbc-2026/