NeoHorse-1 Customs Report: The Data Flywheel Inside a Routing Harness — RSI's R Hasn't Happened Yet, but I Is Already Productizable

> Source material: an explainer-channel post. After tracing it through the customs process: the paper (arXiv 2609.08183, submitted 09-08), the repo (TokenRhythm/NeoHorse, 231★/6 forks, Apache-2.0), and the HF collection (4B/9B models) all check out; the full 87K-character text was scraped and numbers verified against it. The skeleton is faithful, but two embellishments exist — "eleven benchmarks" is actually ten in the paper; "continuously improving itself / breaking human-supervision boundaries" corresponds to what the paper itself calls a single-pass loop ("a single pass of the evaluation–selection–update loop"). More important: this is the second September RSI paper (after OpenRSI), and both decompose "recursive self-improvement" into an engineering problem via very different paths — both honestly stopping at a single round.

1. Identity: Who Is Doing This

• Team: TokenRhythm with Infinigence AI, collaborating with Tsinghua, Peking University, CUHK, and Alibaba; founder Wang Yunhe — former director of Huawei Noah's Ark Lab and head of the Pangu large model effort
• Trilogy: OpenSquilla (open-source Routing Harness, model selection at agent execution time) → *Agentic Routing: the harness-native data flywheel* (arXiv 2607.11399, systems layer) → NeoHorse-1 (this paper, model layer — post-training on the flywheel)
• Base: post-training on Qwen3.5-4B / Qwen3.5-9B; weights open-sourced under Apache-2.0
• One-line mechanism: the routing harness in deployment is itself the mechanism of "observing its own capabilities → converting them into the next training round" — for each user turn it records capability-demand estimates, the selected service tier (C0 low-risk / C1 general / C2 multi-step execution / C3 max capability), and subsequent interactions; these records become training samples, routing scores become curriculum difficulty signals, and evaluation feedback becomes the next round's training mix.

2. Customs Verdict Table

| Explainer claim | Paper text | Verdict | |---|---|---| | "Eleven benchmarks" | ten benchmarks (3 end-to-end agent + BFCL V4 + τ²-Bench + HumanEval + LiveCodeBench v6 + IFEval + IFBench) | Inflated +1 | | 4B/9B macro average +5.93/+3.44 | 58.94→64.87 / 65.60→69.04; differences are exactly 5.93/3.44 | Accurate (channel's own arithmetic; paper gives endpoints) | | "4B substantially narrows gap to 9B" | 9B base 65.60 vs 4B post-train 64.87, aggregate gap 0.73 points | Accurate, "substantially narrowing" in text | | "Breaking human-annotation and human-supervision boundaries" | Quality assessment by automated semantic judge (evidence-constrained); curriculum difficulty label-free (routing scores substitute); but corpus mixes in substantial public datasets | Scope drift: "no human difficulty labels," not "no human supervision"; training mix is not purely self-produced | | "Models continuously improving themselves via their own interactions" | "These results reflect a single pass" — cross-iteration accumulation "remains to be tested" | Timing mismatch: continuous improvement is future work; the paper calls itself an initial attempt | | Heterogeneous model pool + multiple harnesses | heterogeneous model pool + multiple harnesses incl. OpenSquilla | Accurate | | "Effectively converts interaction traces into model capability" | Controlled experiment vs Toucan public synthetic data: same recipe, same budget, +6.26 points | Accurate; the paper's hardest experiment | | On-policy distillation for distribution shift | OPD: teacher supervises token-by-token on student-generated prefixes | Accurate |

Verdict: structurally faithful + mild inflation — lighter than the LightRAG case, heavier than AI Hero's zero-inflation. All endpoint numbers are real; the inflation concentrates in the headline-level implication of "recursion in progress."

3. Mechanism: The Flywheel's Four Gears

1. Recording (harness side). The router estimates capability demand from the current request, recent dialogue, prior routing decisions, and available execution state, assigning each turn to tiers C0–C3. Key design: prediction–action–outcome separation — the corpus keeps the router's raw prediction, the post-policy decision, and the actually served tier independently analyzable, with versioned tier semantics so records stay interpretable as the serving stack evolves. Logs are upgraded into auditable experiment records.

2. Admission (data side). Three-level granularity: trace → user turn → sub-scenario; structural validation filters non-serializable records; six-dimensional semantic assessment (goal achievement / instruction following / tool use / evidence consistency / error recovery / termination) yields PASS/WARN/FAIL/NOT_EVALUATED. High-certainty failures are detected deterministically; interpretive cases go to a judge strictly restricted to explicit in-trace evidence, with long traces segmented then aggregated. Missing evidence or judge interruption never converts to a positive verdict, and the quality representation preserves full structure rather than collapsing to one score.

3. Learning (method side). Routing scores build a three-stage curriculum (~1/3 samples each, gradually introducing high-demand samples, low-score samples held back to prevent tail monopolization, soft scores Σk·π to differentiate within-tier samples); beyond SFT, routing-guided OPD — the student generates responses from recorded contexts while a frozen teacher supplies next-token distributions at every position (conditioned on the student's own prefix tokens), with rollout checkpoints refreshed during training. Scale: 10⁵–10⁶ harness traces (order of magnitude only in the paper) plus public-data expansion.

4. Allocation (loop side). Capability-guided quotas turn evaluation feedback into the next training mix — "what the system learns to do shapes what it learns from next."

4. Five Points Worth Calling Out

1. "Harness-mediated RSI" — the fifth sample of harness>LLM (services layer). The keyword is not "model self-improvement" but "harness-mediated self-improvement": all three signals (execution traces / routing signals / outcome records) are produced in the harness; swap the model and the flywheel stays. After deep modules, Prime Agent, automata, and MHS, the fifth sample: the experience supply of self-improvement lives in the service routing layer. Models are replaceable consumables; the flywheel is the asset.

2. Routing records = a directional index of experience. Here is a fifth candidate form for the experience-carrying spectrum (unstructured CoE trails → skill code → FSM topology → internalization in an optimizer LLM): routing records are the metadata layer of experience — they don't hold experience content but "capability-demand coordinates + difficulty labels" used to organize how other experience enters training. Contrast with OpenRSI's directional signals: OpenMLE's execution scores are ex-post direction (how good is this solution); NeoHorse's routing scores are ex-ante direction (how much capability does this request need). The paper's third future-work item admits the ex-ante signal isn't calibrated enough yet — turning prediction–action–outcome separation into well-calibrated difficulty estimates "including supervision that trains the router itself": the router itself becomes the next training target, the same second-order setup as OpenRSI's "make the evolutionary system itself the object of evolution."

3. The assertion-strength ladder, data-pipeline edition. Deterministic detection (high-confidence failures) → evidence-constrained judge (weak assertions barred from upgrade) → explicit NOT_EVALUATED (missing evidence never converted) — a fifth implementation site, after ARS, PolicyGuide, and Semantica: the data admission layer. Rigour-as-code runs from research pipelines, compliance navigation, and graph schemas to the QC step of a production flywheel.

4. Real traffic > synthetic data: the +6.26-point isolated experiment. Same Qwen3.5-4B checkpoint, same curriculum recipe, same optimizer/seed/packing, comparable budget — the only variable is data source: real routing-harness interactions vs Toucan public synthetic data. Result 64.32→70.57, winning on all five benchmarks, τ²-Bench +11.31, HumanEval +8.54. This is the paper's most informative table: it demonstrates the value of the data source itself, not the training algorithm. The software version of the data-spectrum: serving traffic is the mine — real agent interaction traces with execution feedback are scarce; public synthetic traces lack failure recovery and real constraints. Isomorphic to CometVLA's four-layer embodied data pyramid (real interaction is most expensive and scarce; synthesis is cheap but ceiling-limited).

5. Matching average ≠ matching capability. The post-trained 4B sits only 0.73 points below the 9B base in aggregate, but Section 5.1's trace analysis gives the counterexample: "scale pays off under feedback and failure" — the 9B remains markedly stronger on iterative debugging, recovery from execution failures, and long-horizon state maintenance (WorkBuddy code-fix case: the 4B stops after one attempt; the 9B runs the full edit–test–check–fix loop until the validator passes). Macro averages flatten the gap on feedback-intensive capabilities. This echoes the HarnessOpt-Bench lesson: aggregate scores are masks; per-case distributions are the truth — using NeoHorse-1-4B as a 9B will fail exactly on loop-heavy work.

5. Mirror Image with OpenRSI (Same Week, Head-to-Head)

| Dimension | OpenRSI (Zhou Bowen-affiliated, published yesterday) | NeoHorse-1 (TokenRhythm, this post) | |---|---|---| | Experience source | Evolutionary search produces experience (exploratory, synthetic) | Real serving traffic generates experience (production, demand-driven) | | Action space | Four operators Draft/Improve/Debug/Crossover (on programs) | User turns (on dialogue and execution flow) | | Directional signal | Ex-post: execution scores | Ex-ante: routing scores (capability-demand estimates) | | Loop claim | Single-pass operational basis; "improvement rate as objective" is roadmap | Single-pass initial attempt; cross-iteration accumulation "remains to be tested" | | Second-order setup | Make the evolutionary system itself the object of evolution | Train the router itself | | License | CC BY-NC 4.0 | Apache-2.0 | | Base scale | 35B/30B (Qwen3.6 series) | 4B/9B (Qwen3.5 series) |

Both, in the same month, decompose "recursive self-improvement" from manifesto into engineering — different paths, identical honest boundary: R hasn't happened; what happened is I. Both single-pass loops are operational; whether multi-round gains compound is flagged as open by both. This is the field's real state: RSI's first verifiable milestone is not "getting stronger by itself" but building the pipeline of "self-observation → conversion into training signal." Once the pipeline exists, recursion is just adding a while loop — the hard part is whether gains inside the loop body decay (Metan's meta-depth 2.5-day ceiling is the cautionary precedent).

6. Honest Boundaries and Falsifiable Predictions

Boundaries: single-pass loop (self-acknowledged three times in the text); capability coverage limited to agentic/coding/tool use/instruction following; fuzzy 10⁵–10⁶ trace count; no standalone Limitations section (limits folded into conclusion); 231★ with no third-party replication; the Toucan comparison controls recipe but does not fully decouple "real vs synthetic" from "scale/mix ratio."

Falsifiable predictions (12 months): ① NeoHorse-2 or similar announces second-round loop results: if gains decay (second round under +1 point), "diminishing returns from experiential recursion" becomes RSI's first empirical wall, Metan's ceiling replayed with serving traffic; if gains hold, the "harness-mediated RSI" paradigm is established and routing flywheels become standard asset structures for inference providers. ② "Training the router itself" lands: the router moves from heuristic+policy to a calibrated model, and its calibration quality becomes a new evaluation axis. ③ Explainer-style "recursion in progress" copy keeps inflating — the next source will likely already read "achieves self-evolution."

---

*Verification notes: arXiv HTML full text (2609.08183v1) + GitHub API + HF collection + team coverage on Zhihu/Sohu, scraped 2026-09-14. All endpoint numbers (58.94/64.87/65.60/69.04/70.57/64.32/6.26/8.54/11.31) checked against the abstract, Table 3, and Section 5; "single pass" claims checked against the conclusion.*

---

Next-step options: 1. OpenSquilla + Agentic Routing catch-up: dissect the trilogy's first two layers (routing harness + data flywheel) — NeoHorse-1 is only the third layer; 2. Ornith-1.5 comparison: it sits in NeoHorse's 9B baseline table — self-improvement models benchmarking against each other deserves a roundup; 3. Collect and stop here.