NeoHorse-1 Fact-Check Report: The Data Flywheel in a Routing Harness — RSI's R Hasn't Happened, but the I Is Already Productizable

> Source material was a hype-style blog post. After tracing the paper (arXiv 2609.08183, submitted 09-08), the repo (TokenRhythm/NeoHorse, 231★/6 forks, Apache-2.0), and the HF collection (4B/9B models) — all real, full 87K-character paper checked figure by figure. The skeleton is faithful; two embellishments: "eleven benchmarks" (the paper has ten), and "continuously improving itself / breaking human supervision boundaries" corresponds to the paper's own admission of a single-pass loop ("a single pass of the evaluation–selection–update loop"). Bigger finding: this is the second September RSI paper (after OpenRSI), and both decompose "recursive self-improvement" into engineering in the same week — both honestly stopping at one round.

1. Who is doing this

• Team: TokenRhythm with Infinigence AI, collaborating with Tsinghua, Peking University, CUHK, and Alibaba; founder Wang Yunhe — former director of Huawei Noah's Ark Lab and head of Pangu models
• Trilogy: OpenSquilla (open-source routing harness for model selection at agent execution time) → *Agentic Routing: the harness-native data flywheel* (arXiv 2607.11399, system layer) → NeoHorse-1 (this paper, model layer — post-training on the flywheel)
• Base: post-training of Qwen3.5-4B / Qwen3.5-9B; Apache-2.0 open weights
• Mechanism in one line: the deployment routing harness is itself the mechanism of "observe own capability → convert into next-round learning" — it logs per user turn the capability estimate, chosen service tier (C0 low-risk / C1 general / C2 multi-step execution / C3 max capability), and subsequent interactions; records become training samples, routing scores become curriculum difficulty signals, evaluation feedback becomes next-round training mix.

2. Fact-check verdict table

| Blog claim | Paper | Verdict | |---|---|---| | "Eleven benchmarks" | ten benchmarks (3 end-to-end agent + BFCL V4 + τ²-Bench + HumanEval + LiveCodeBench v6 + IFEval + IFBench) | Inflated +1 | | 4B/9B macro average +5.93/+3.44 | 58.94→64.87 / 65.60→69.04; differences are exactly 5.93/3.44 | True (blog's own arithmetic; paper gives endpoints) | | "4B largely narrows gap to 9B" | 9B base 65.60 vs 4B post-trained 64.87; aggregate gap 0.73 | True, "substantially narrowing" | | "Breaking boundaries of human-labeled data and human supervision" | Quality evaluation by constrained semantic judge; curriculum difficulty label-free (routing scores replace labels); but training mixes in large amounts of public data | Scope drift: "no human difficulty labels," not "no human supervision"; mix is not purely self-produced | | "Models continuously improve via self-interaction experience" | "These results reflect a single pass"; cross-iteration accumulation "remains to be tested" | Timing error: continuous improvement is future work, paper self-describes as initial attempt | | Heterogeneous model pool + multiple execution frameworks | heterogeneous model pool + multiple harnesses incl. OpenSquilla | True | | "Effectively converts interaction traces into model capability" | Controlled experiment vs Toucan public synthetic data: same recipe, same budget, +6.26 | True; the paper's hardest result | | On-policy distillation solves distribution shift | OPD: teacher supervises token-by-token on student-generated prefixes | True |

Verdict: structurally faithful + mild inflation — numbers all real; inflation concentrated in the "recursion in progress" title-level implication.

3. Mechanism: four gears of the flywheel

1. Record (harness side). Router estimates capability needs from the request, recent dialogue, prior routing decisions, and execution state; assigns each turn to tiers C0–C3. Key design: prediction–action–outcome separation — logs keep the router's original prediction, the post-policy decision, and the tier actually served, independently analyzable, with versioned tier semantics so records stay interpretable as the serving stack evolves. Logs upgraded to auditable experimental records. 2. Admission (data side). Three-level granularity (trace → user turn → sub-scenario); structural validation filters unusable records; six-dimension semantic evaluation (goal attainment / instruction following / tool use / evidence consistency / error recovery / termination) with PASS/WARN/FAIL/NOT_EVALUATED states. High-certainty failures are deterministically detected; everything else goes to a judge strictly restricted to explicit in-trace evidence; long traces are chunked and aggregated. Missing evidence or judge interruption is never promoted to a positive verdict; the quality representation keeps full structure rather than compressing to a single score. 3. Learn (method side). Routing scores build a three-stage curriculum (~1/3 samples each, gradual introduction of high-demand samples, low-score samples held back to prevent tail monopolization, soft scores Σk·π to differentiate within tiers); beyond SFT, routing-guided OPD — the student generates responses from recorded context, a frozen teacher provides next-token distributions conditioned on the student's own prefix; rollout checkpoints refresh during training. Scale: 10⁵–10⁶ harness traces (order of magnitude only) plus public data. 4. Allocate (loop side). Capability-guided quotas turn evaluation feedback into the next training mix — "what the system learns to do shapes what it learns from next."

4. Five things worth saying separately

1. "Harness-mediated RSI" — the fifth sample of harness > LLM (service layer). The key term is not "model self-improvement" but "harness-mediated": all three signals (execution traces / routing signals / outcome records) originate in the harness; swap the model, keep the flywheel. The fifth instance, after deep modules, Prime Agent, automata, and MHS: the experience supply of self-improvement lives in the service-routing layer. Models are replaceable consumables; the flywheel is the asset. 2. Routing records = a directional index of experience. As an experience-carrier lineage (unstructured CoE trails → skill code → FSM topology → optimizer-LLM internalization), routing records are a fifth candidate: the metadata layer of experience — they don't hold experience content but its "capability-need coordinates + difficulty labels," organizing how other experience enters training. Compared to OpenRSI's directional signal: OpenMLE's execution score is ex-post (how good is this solution), NeoHorse's routing score is ex-ante (how much capability does this request need). The paper's third future-work item admits the ex-ante signal isn't calibrated yet — turning prediction–action–outcome separation into well-calibrated difficulty estimates, "including supervision that trains the router itself": the router becomes the next training target, the same second-ordering foreshadow as OpenRSI's "making the evolutionary system itself the object of evolution." 3. The assertion-strength ladder, data-pipeline edition. Deterministic detection (high-confidence failures) → evidence-constrained judge (weak assertions can't be promoted) → explicit NOT_EVALUATED (missing evidence is never upgraded) — the fifth implementation site of "rigour as code": the data admission layer, after research pipelines, compliance navigation, and graph schemas. 4. Real traffic > synthetic data: the +6.26 isolated experiment. Same Qwen3.5-4B checkpoint, same curriculum recipe, same optimizer/seed/packing, similar budget; the only variable is data source: real routing-harness interaction vs Toucan public synthetic data. Result 64.32→70.57, winning all five benchmarks, τ²-Bench +11.31, HumanEval +8.54. The most informative table in the paper: it proves the value of the data source itself, not the training algorithm. Service traffic is the mine — real agent traces with execution feedback are scarce; public synthetic traces lack failure recovery and real constraints. 5. Average-score parity ≠ capability parity. 4B post-trained trails 9B base by only 0.73 aggregate, but the paper's §5.1 trace analysis gives a counterexample: "scale pays off under feedback and failure" — 9B remains significantly stronger at iterative debugging, recovery from execution failures, and long-horizon state maintenance (WorkBuddy code-fix case: 4B stops after one attempt, 9B runs the full edit-test-check-fix loop until the verifier passes). The macro average flattens feedback-intensive capabilities. Using NeoHorse-1-4B as a 9B will fail exactly on loop-heavy work.

5. The OpenRSI mirror (two papers, same week)

| Dimension | OpenRSI (Zhou Bowen group, prior day) | NeoHorse-1 (TokenRhythm, this paper) | |---|---|---| | Experience source | Evolutionary search produces experience (exploratory, synthetic) | Real service traffic generates experience (production, demand-driven) | | Action space | Four operators Draft/Improve/Debug/Crossover (on programs) | User turns (on dialogue and execution flows) | | Directional signal | Ex-post: execution scores | Ex-ante: routing scores (capability-need estimates) | | Loop claim | Single-pass operational basis; "improvement rate as objective" is roadmap | Single-pass initial attempt; cross-iteration accumulation "remains to be tested" | | Second-ordering | Make the evolutionary system itself the object of evolution | Train the router itself | | License | CC BY-NC 4.0 | Apache-2.0 | | Base scale | 35B/30B (Qwen3.6) | 4B/9B (Qwen3.5) |

Both decomposed "recursive self-improvement" from manifesto into engineering the same month, with different paths and identical honest boundaries: R hasn't happened; what happened is I — single-pass loops are operational, whether multi-round gains accumulate is flagged as open by both. This is not coincidence but the field's real state: RSI's first verifiable milestone is not "self getting stronger" but building the pipeline from self-observation to training signal. Once the pipeline exists, recursion is just adding the while loop — the hard part is whether gains inside the loop decay (Metan's meta-depth ceiling is the cautionary precedent).

6. Honest boundaries and falsifiable predictions

Boundaries: single-pass loop (self-admitted in three places); capability coverage limited to agentic/coding/tool use/instruction following; vague 10⁵–10⁶ trace count; no standalone Limitations section; 231★ with no third-party reproduction; the Toucan comparison controls the recipe but doesn't fully decouple "real vs synthetic" from "scale/source mixing ratios."

Falsifiable predictions (12 months): ① NeoHorse-2 or similar announces second-round loop results: if gains decay (under +1 on round two), "diminishing recursive returns from experience" becomes RSI's first empirical wall; if gains hold, the "harness-mediated RSI" paradigm is established and routing flywheels become a standard asset for inference providers. ② "Training the router itself" lands: the router goes from heuristic+policy to a calibrated model, and calibration quality becomes a new evaluation axis. ③ Hype-style "recursion in progress" copy continues to inflate — the next one will already say "achieved self-evolution."

---

*Verification note: arXiv HTML full text (2609.08183v1) + GitHub API + HF collection + team coverage on Zhihu/Sohu, captured 2026-09-14. All endpoint numbers (58.94/64.87/65.60/69.04/70.57/64.32/6.26/8.54/11.31) checked against the paper's abstract, Table 3, and Section 5; "single pass" claims checked against the conclusion.*