Keyword Harnesses Fail Open: When Your Evaluation Tool Passes Models That Can't Actually Use Tools

Imagine you're a teacher grading an exam. One question says: "use a calculator to compute 123 × 456." When grading, you mark any answer of "55588" as correct—regardless of whether the student actually used the calculator, did mental math, or guessed.

Seems fine? But suppose a student who cannot use a calculator at all writes the right answer and gets full marks. Your exam has "failed open"—at the exact point where it should detect a missing capability, it signals "pass."

That's the story of this paper, except the protagonist is a language model, the "calculator" is tool calling, and the "teacher" is a keyword-matching evaluation benchmark.

The Main Characters: A "Matched" Pair of Models

The author built a pair of models called the VectraYX series:

• VectraYX-600M: 661.6M parameters, single-stage pretraining, ~65% of training data is code and technical text, no dedicated SFT (supervised fine-tuning) stage. Training was frozen at 64% progress.
• VectraYX-1B: 1.109B parameters, multi-stage training, mostly web text, with a dedicated tool-call SFT stage of roughly 6 billion tokens.

The two share decoder architecture, tokenizer, and special-token layout. The 1B should be stronger—it's bigger and has dedicated tool-use training.

On the series' own benchmark, B4, the scores are nearly identical: 600M gets 0.660, 1B gets 0.650. Looks about even, right?

The Truth: One Can, One Completely Can't

The author did something rarely seen in evaluation papers: verbatim reproduction checks. He took real training samples and had both models actually generate tool calls.

Results:

• VectraYX-600M: 6/6 samples produced well-formed tool calls, with parameters generalizing to new scenarios.
• VectraYX-1B: across all checkpoints, 0/4 to 0/6 samples produced valid tool calls.

Completely unable.

A first-token probability probe confirmed it: the 1B model assigns a probability of 10⁻⁴ to 10⁻⁵ to the tool-call start token. This isn't "occasional mistakes"—it's "cannot do it at all, from the root."

So why did B4 award 0.650? Because B4 uses keyword matching—if certain keywords (tool names, parameter names) appear in the output, it counts as "used a tool." The 1B model can't initiate tool calls, but it saw massive amounts of tool-keyword-laden text during SFT, so its ordinary generated text frequently happens to contain those keywords.

The keyword matcher is grading on a curve—it's detecting a capability that doesn't exist.

Why Is the 600M Stronger? The Hidden Power of Code Pretraining

This is the paper's most interesting finding. The 600M has no dedicated tool-call SFT, but its pretraining data is 65% code. The 1B has 6B tokens of tool-call SFT, but web-text-heavy pretraining.

The author found that the 600M's tool-calling ability comes from compositional generalization acquired via code pretraining. Code is full of "function call" patterns—foo(arg1, arg2). After pretraining on large amounts of code, the model learns a prior for "structured invocation." Tool calling is just a special case of function calling. With as little as 0.1% tool-call signal in the training data, that prior gets activated.

The 1B model, meanwhile, was fed plenty of tool-call data in SFT, but its pretraining prior is too weak—web text doesn't contain enough "structured invocation" patterns. SFT taught it "what to say," but not "how to say it." It knows the keywords of tool calling but can't assemble them into valid calls.

It's like one person who memorized a cookbook (SFT) but never cooked (no code pretraining), versus another who has cooked thousands of dishes (code pretraining) without studying any specific recipe (no SFT)—the latter knows the basic structure "chop → stir-fry → season" and can make new dishes with a little guidance.

The Fix: Precision Surgery on the Embedding Layer

The author also ran a "repair" experiment. He found the 1B model's problem lies in the embedding layer—the embeddings of tool-call special tokens were never properly learned. The fix: re-initialize only a small portion of the embedding table and continue training.

Result: 97.7% of the embedding table's bits remained exactly identical—only 2.3% of parameters changed. Yet that 2.3% change moved the 1B model from "0/6 usable" to generating valid tool calls.

This has two implications: 1. The problem is extremely localized—the model isn't fundamentally broken; a small part of the embedding layer simply never learned. 2. The fix is extremely precise—no full retraining needed, just "scalpel-like" repair of a small parameter subset.

A Diagnostic Ladder: A Cheap Set of Checks

The paper's title mentions "A Cheap Diagnostic Ladder." The author stresses: these diagnostics require no large compute; anyone with model weights can run them.

The ladder includes:

• Verbatim reproduction checks: take samples from training data, generate, and check formatting
• First-token probability probe: check the probability of the tool-call start token
• Trajectory probes: track F1 changes across training

The core idea: don't just look at benchmark scores—look at what the model actually does. Keyword-matched B4 can read 0.650, but a verbatim check immediately exposes the 0/6 truth.

The Bigger Picture: Evaluation False Positives

The paper touches on an issue broader than tool calling: false positives in benchmarks.

A "false positive" is a statistics term: reporting positive when the result should be negative. In AI evaluation, it means the model lacks a capability, but the benchmark says it has it.

Keyword matching is a common source. It assumes "mentioning keywords = using the capability." But language models are probabilistic generators—they can produce text containing a concept's keywords without understanding the concept at all. This is especially true after SFT: SFT teaches the model "what to say," but not necessarily "why to say it." This is the same problem as a "rationalization shell"—models can produce plausibly correct outputs with entirely wrong internal mechanisms. Keyword matching only sees outputs; verbatim checks and probability probes see mechanisms.

The author candidly lists limitations: the 600M was permanently frozen at 64% training progress, has no conversational ability, and all diagnostics use small samples. This isn't a "small model beats big model" story—it's an "evaluation methods can deceive you" story.

Practical Takeaways

If you're evaluating tool-use in small models, the paper offers three actionable lessons:

1. Don't trust benchmark scores alone. B4 said 0.650; verbatim checks said 0/6. Any keyword-matching evaluation needs stricter checks to supplement it. 2. Code pretraining matters more than tool SFT. The 65%-code 600M calls tools with zero SFT; the web-text 1B still can't after 6B tokens of SFT. Tool calling is a special case of "structured invocation," and code pretraining builds that structural prior. 3. The embedding layer is a common failure point. 97.7% of parameters unchanged; fixing 2.3% sufficed. When a model "can't use tools," check the special-token embeddings first.

Finally, the author is a DevOps engineer doing this research in spare time—which perhaps explains the focus on "cheap diagnostics": not everyone has the compute for large-scale evaluation, but everyone can do a verbatim check.

---

Paper link: https://arxiv.org/abs/2610.02142

Code repository: https://github.com/vectrayx/vectrayx-nano-paper