Nautil: Teaching LLM Investigators When to Close a Case
> Paper: *Not Until the Evidence Says So: Teaching LLM Investigators When to Close a Case* > arXiv: 2610.03190 > Code: etigerstudio/Nautil
An Overlooked Decision
Imagine you are an aviation accident investigator. A plane has crashed; you've spent three weeks gathering evidence—black box data, wreckage analysis, witness statements, maintenance records. Now two decisions face you:
1. What caused the accident? 2. Is the evidence enough? Can the case be closed?
The second decision is far harder than the first. A substantial share of official NTSB reports end with "cause undetermined." This isn't laziness—it's honesty: the evidence doesn't support any specific conclusion, so the case must stay open.
But LLMs are bad at saying "I don't know yet." In October 2026, Tingzhu Bi, Ping Wang, and Meng Ma published a paper studying exactly this overlooked decision: when should an LLM investigator close a case? They built Nautil, a dataset and training framework that teaches LLMs something counterintuitive—keep the case open when evidence is insufficient.
Shocking Baseline Numbers
| Model | Correctly identifies cause | "Overstatement" rate | Wrongly closed "undetermined" cases | |---|---|---|---| | 9B (untuned) | 60% | 97% | - | | Gemini 3.8 Flash (frontier) | 84% | 91% | 17/41 |
"Overstating" means the model says "evidence establishes" when it only "suggests." The model isn't fabricating causes—its identified cause is often right—but it converts "possible" into "certain." Worse: of 41 cases officially concluded as "cause undetermined," Gemini 3.8 Flash closed 17. The untuned 9B model is even worse at 97% overstatement. Finding causes ≠ knowing when to stop—these are separate abilities.
Three Tests, Increasingly Strict
Nautil's core contribution is redefining how to evaluate "when to close a case," via three complementary tests:
Test 1: Closure Accuracy
Did the model close cases that should be closed, and keep open those that should stay open? Here's the trap: Nautil's data comes from sources (NTSB aviation, NHTSA vehicle defects, maritime, chemical plants, production server failures) where the share of "undetermined" cases varies hugely.
A rule that looks only at the source label—ignoring all evidence—achieves 83.0 balanced accuracy. So a 90% closure accuracy might just mean your model learned "NTSB cases are undetermined." Nautil therefore requires reporting closure accuracy both (a) as improvement over the source-only rule and (b) per source.
Test 2: Evidence Dependence
> If the model's conclusion truly relies on evidence E, removing E from the case file should change the conclusion.
Method: (1) model concludes C citing evidence E; (2) build a counterfactual version with E removed; (3) re-investigate; (4) if the model still concludes C, then E wasn't a genuine basis—just a rationalization shell. A matched control removes the same amount of irrelevant evidence, controlling for generic degradation.
Test 3: Conclusion and Gap Quality
Beyond binary close/open decisions, a full closure report needs a conclusion (what is asserted) and gaps (what evidence is missing). A judged checklist evaluates whether conclusions are accurate, whether stated gaps are real, and whether gaps were missed.
The Nautil Dataset
731 audited cases from:
- NTSB (aviation accidents)
- NHTSA (vehicle defects)
- Maritime investigations
- Chemical plant safety reports
- Production server failures (real incident reports)
- Overstatement rate dropped sharply from 97%
- On "cause undetermined" cases, the model learned to say "evidence insufficient—keep open"
- The model proactively names specific missing evidence ("need to check X to confirm Y")
- Code agents: when to submit a PR vs. say "I need more info"?
- Customer service agents: when to answer vs. escalate to a human?
- Research agents: when to conclude vs. say "evidence insufficient"?
Each case includes: full evidence files (tens to hundreds of items), official conclusions (including "undetermined"), teacher investigation trajectories (for SFT), counterfactual evidence-removed versions, an out-of-distribution test set, and independent evaluation on 141 cases.
Training Results
After SFT on a 9B model:
Nautil-SFT neither forces conclusions nor mindlessly says "unknown"—it accurately states that current evidence merely suggests X, is insufficient to establish it, and that Y and Z are missing.
Why This Matters
1. "Knowing when to stop" is an independent capability
The central finding: cause-finding and case-closing are separate abilities. A model identifying causes correctly in 84% of cases may overstate in 91% of answers. Judgment and gate must be decoupled. Every agent system faces this:
2. "Source-predictive labels" are an evaluation trap
When data source correlates with labels, raw accuracy massively overestimates capability. A source-only rule hits 83.0 balanced accuracy—any model below that baseline has learned nothing. This trap applies everywhere: medical AI learning "which hospital," legal AI learning "which court," support AI learning "which product line." Every classification task should establish a source baseline first, then report improvement over it.
3. Counterfactual evidence tests counter rationalization
If deleting the model's stated basis doesn't change its conclusion, that basis wasn't the real cause—just post-hoc rationalization. This generalizes to all LLM-as-judge scenarios: remove the cited reason and check whether the answer survives.
Honest Limitations
1. Absolute accuracy remains modest—trained models still lag human investigators on "undetermined" cases. 2. Limited dataset scale—731 cases is small for teaching "when to stop." 3. Teacher trajectories may carry bias into SFT. 4. Counterfactual construction is artificial—degraded performance after deletion may partly reflect awkward documents rather than truly removed evidence.
The Bigger Picture
Nautil highlights a fact the AI world often overlooks: in many professional fields, "I don't know" is a correct answer. Investigators saying "cause undetermined" is honesty; doctors requesting more tests is caution; engineers flagging insufficient root-cause data is responsibility. Current LLMs are trained to always have an answer—even one based on weak evidence. Nautil's contribution isn't making LLMs smarter but teaching a kind of epistemic honesty: admit insufficiency when evidence is insufficient, stay open when the case is unclear. This honesty isn't "capability"—it's "restraint." And restraint is harder to teach than capability.
---
Paper: https://arxiv.org/abs/2610.03190 Code: https://github.com/etigerstudio/Nautil