FIRE Benchmark and XuanYuan 4.0: The Real State of Financial AI and a 36B Model's 'Comeback' Story
Introduction: From 'Revolutionary Breakthroughs' to the Crucible of Real Finance
News headlines constantly proclaim that some financial AI model has achieved another "revolutionary breakthrough" or aced another exam. Yet when these models enter real financial workflows, they often cannot even read a financial report, let alone understand complex business dynamics and risk control. How do trillion-parameter "super brains" actually perform in real financial scenarios?
The FIRE (Financial Intelligence & Reasoning Evaluation) benchmark, jointly launched by Du Xiaoman, Tsinghua University's PBC School of Finance, and Renmin University of China's School of Finance, together with XuanYuan 4.0 — a 36B-parameter open-source model — have sparked a debate in the financial community about the truth behind the technology. Is this PPT vaporware or genuine, disruptive capability? This article dissects both to reveal the real capability boundaries and value of financial AI.
Capability Boundaries and Hallucinations: High CFA Scores, Fabricated Due Diligence
Many general-purpose LLMs score near-perfect on standardized tests like the CFA exam, yet in real due diligence and risk assessment tasks they frequently fabricate data and produce confused logic. This "high scores, low capability" phenomenon reveals the fundamental difference between lexical recall and logical reasoning.
General models memorize vast amounts of financial terminology from training data, so they excel at vocabulary-level questions such as multiple-choice definitions. But finance demands deep understanding of causal logic and context. When asked to apply knowledge to concrete cases with multi-step reasoning, these models fall back on pattern matching and produce hallucinations — content that contradicts facts or context. A model that aces an exam can still "make things up" in due diligence because it never truly grasped the underlying business logic.
FIRE was designed precisely to expose this capability boundary. By combining theoretical exams with scenario-based practice, it shows that many models score high on certification-style questions but drop sharply on real-world scenario tasks — evidence that today's models' financial "intelligence" largely remains surface-level knowledge.
A 2D Scenario Evaluation Matrix: Beyond Multiple Choice, Into Real Business
FIRE introduces a two-dimensional evaluation matrix: a vertical business dimension crossed with a horizontal capability dimension, ensuring full-coverage, fine-grained assessment.
- Vertical business dimension: eight major sectors — banking, insurance, securities, funds, futures, trust, fintech, and general finance — subdivided into 17 second-level business scenarios (e.g., banking splits into corporate finance, retail finance, and financial markets/treasury; insurance into property, life, and reinsurance).
- Horizontal capability dimension: four core capabilities spanning the full value chain of "insight → creation → delivery → risk control": Insight & Decision-making, Product & Marketing, Service & Operations, and Risk & Compliance.
- 1,000 closed-loop decision questions with definitive standard answers for objective evaluation of decision accuracy and robustness.
- 2,000+ open-ended scenario questions without a single correct answer, graded against professional rubrics that emphasize reasonableness and explainability of outputs.
Based on this matrix, FIRE collected nearly 70 business grids and 3,000 practical questions, including:
Reverse Chain of Thought: Building an Impartial AI Judge
Scoring open-ended answers objectively and consistently is a major challenge. Naive LLM-as-a-Judge approaches are unstable — highly sensitive to prompt wording and answer length, sometimes giving wildly different scores to semantically identical answers.
The FIRE team instead created fine-grained 1–5 point scoring rubrics for every open-ended question, inspired by structured criteria like OpenAI's HealthBench. Rubrics were produced via a three-step pipeline: multi-model collaborative generation → single-model synthesis → expert refinement, ensuring professionalism and consistency.
More critically, the team trained a dedicated judge model using Reverse Chain of Thought (RCoT): instead of the usual forward direction (question → reasoning → answer), the model learns to reason backward — generating the thought process from the question and answer. This trains the model to strictly follow the scoring rubric rather than improvise. Combined with Reinforcement Learning with Verifiable Rewards (RLVR) optimized on real business financial tasks, the resulting AI judge scores objectively, is immune to answer-length or phrasing tricks, and enables reliable, scalable open-ended evaluation.
XuanYuan 4.0's Dimensional Strike: The Rise of a 36B Model
A striking FIRE finding: XuanYuan 4.0, an open-source model with only 36B parameters, outperforms trillion-parameter closed-source giants on multiple real-business metrics.
XuanYuan 4.0 is Du Xiaoman's latest flagship financial LLM. It uses a dense architecture with 36 billion parameters, initialized from the open-source Seed-OSS-36B-Base model, and was trained through a rigorous multi-stage pipeline:
1. Continual Pre-Training (CPT): leveraging the XuanYuan series' accumulated high-quality financial corpus, with simulated-annealing training scheduling and a reference-model-based KL-divergence self-regularization objective. This "unleashes the KL leash" to deepen financial knowledge and reasoning while suppressing distribution drift, preventing catastrophic forgetting. 2. Supervised Fine-Tuning (SFT): on high-fidelity datasets covering math, STEM, and agent tasks, systematically strengthening base reasoning and problem-solving. 3. RLVR alignment: using the DAPO algorithm on real internal business financial tasks, boosting consistency and reliability under complex financial logic and business constraints.
Measured results are remarkable: on FIRE's financial certification exam set, XuanYuan 4.0 averages 91.33, far above its base model Seed-OSS-36B (69.45) and within 1.56 points of GPT-5.2 (92.89). Even more notably, on real financial scenario tasks, XuanYuan 4.0's composite score reportedly surpasses GPT-5.2 — e.g., more business-appropriate suggestions in retail banking customer service, insurance product designs that better fit regulatory requirements, and more insightful securities research reports. These wins come not from parameter scale but from deep business understanding and precise reasoning.
Additionally, XuanYuan 4.0's open-source nature enables low-cost local deployment: financial institutions can run it on-premises, cutting compute costs, avoiding data-leakage risks, and keeping business operations sovereign and controllable.
Conclusion: Bubble or Disruption? The Real Value of Financial AI
Both narratives are true. Bubbles exist: models that shine on multiple-choice tests often fail in real business, their "intelligence" amounting to surface-level word matching. But disruption is real too: XuanYuan 4.0's comeback shows that with deliberate domain training and methodological innovation, a far smaller model can excel in finance's crucible and deliver real business value.
FIRE provides a ruler that links model capability directly to business outcomes, letting institutions rationally assess AI ROI. XuanYuan 4.0 demonstrates a practical path: go deep into the domain, train rigorously, and control hallucination boundaries. In finance — a high-value, high-risk field — genuine competence beats flashy showmanship. AI that can read financial statements, perceive risk, and create value is the revolutionary force finance actually needs.
Key Score Comparison (FIRE Certification Exam Set)
| Model | Parameters | Score | |---|---|---| | Seed-OSS-36B (base) | 36B | 69.45 | | XuanYuan 4.0 | 36B | 91.33 | | GPT-5.2 (closed-source) | ~trillion | 92.89 |