FIRE Benchmark and XuanYuan 4.0: How Good Is Financial AI, Really?

Introduction: From "Revolutionary Breakthroughs" to the Crucible of Real Finance

Headlines constantly announce "revolutionary breakthroughs" in financial AI—models acing exams with near-perfect scores. Yet when deployed in real financial work, many of these models cannot even read a financial statement, let alone understand complex business strategy or risk control. How do trillion-parameter super-brains actually perform in real financial scenarios?

Recently, FIRE (Financial Intelligence & Reasoning Evaluation)—a benchmark jointly released by Du Xiaoman, Tsinghua University's PBC School of Finance, and Renmin University of China's School of Finance—along with XuanYuan 4.0, an open-source model of only 36B parameters, sparked a serious discussion about the true state of financial AI. This article breaks down what FIRE reveals about LLMs' real capability boundaries, and how XuanYuan 4.0 manages to punch far above its weight.

Capability Boundaries and Hallucinations: High CFA Scores, Fabricated Due Diligence

Many general-purpose LLMs score near-perfectly on standardized tests like the CFA exam, yet in real due-diligence and risk-assessment scenarios they fabricate data and produce incoherent logic. This "high scores, low ability" phenomenon exposes the difference between lexical recall and logical reasoning.

Models memorize vast amounts of financial terminology from training data, so they excel at vocabulary-level questions—multiple-choice definitions and principles. But real finance demands deep understanding of causal logic and context. When asked to apply knowledge to concrete cases with multi-step reasoning, these models fall back on pattern matching and produce hallucinations—content that contradicts facts or context. A model that aces the exam fabricates answers in due diligence because it never truly grasped the underlying business logic.

FIRE was designed precisely to expose this boundary. Combining theory exams with scenario practice, it shows that many models score high on qualification-exam dimensions but drop sharply on real-scenario tasks—their "intelligence" remains surface-level knowledge.

A 2D Scenario Evaluation Matrix: Beyond Multiple Choice, Into Real Business

FIRE introduces a two-dimensional evaluation matrix to measure AI's actual business value:

• Vertical (business) dimension: covers eight financial sectors—banking, insurance, securities, funds, futures, trust, fintech, and general finance—further split into 17 secondary business scenarios (e.g., banking divides into corporate finance, retail finance, and financial markets/treasury; insurance into property, life, and reinsurance).
• Horizontal (capability) dimension: four core capabilities spanning all businesses—insight & decision-making, product & marketing, service & operations, and risk & compliance—mapping the full value chain of "insight → creation → delivery → risk control."

Based on this matrix, FIRE collects nearly 70 business grids and 3,000 practical tasks:

• 1,000 closed-loop decision questions with clear standard answers for objective evaluation of decision accuracy and robustness.
• 2,000+ open-ended scenario questions without unique answers, judged against professional scoring rubrics focused on soundness and explainability.

Reverse Chain-of-Thought: Building an Impartial AI Judge

Grading open-ended answers objectively is a major challenge. Conventional "LLM-as-a-Judge" approaches are unstable—scores flip based on prompt wording, answer length, and other superficial factors.

FIRE's solution has three parts:

1. Fine-grained 1–5 scoring rubrics for every open question, inspired by structured rubrics like OpenAI's HealthBench, produced via a multi-model collaboration → single-model synthesis → expert refinement pipeline. 2. Reverse Chain-of-Thought training: instead of generating reasoning then an answer, the judge model is trained to generate the reasoning process first from the question, then derive the answer from question + reasoning—teaching it to follow the rubric rather than improvise. 3. RLVR (Reinforcement Learning with Verifiable Rewards) to align the judge model, optimized on financial tasks from real business scenarios.

The result is an AI judge that scores strictly against predefined professional standards with very low error, immune to answer length or phrasing—enabling objective, scalable evaluation of open-ended tasks.

XuanYuan 4.0: The 36B Underdog's Breakthrough

A striking FIRE finding: XuanYuan 4.0, an open-source model with just 36B parameters, beats much larger closed-source models on multiple real-business metrics.

XuanYuan 4.0 is Du Xiaoman's latest flagship financial LLM—a dense 36B model initialized from the open-source Seed-OSS-36B-Base and trained through a rigorous multi-stage pipeline:

1. Continual Pre-Training (CPT) on high-quality financial corpora, using simulated-annealing-style scheduling and a reference-model-based KL-divergence self-regularization objective. This "unleashes the KL leash"—deepening financial knowledge and reasoning while suppressing distribution drift, avoiding catastrophic forgetting. 2. Supervised Fine-Tuning (SFT) on high-fidelity data covering math, STEM, and agent tasks to strengthen general reasoning. 3. RLVR alignment using the DAPO algorithm, optimized on financial tasks from real internal business scenarios.

Measured results are remarkable: on FIRE's certification-exam set, XuanYuan 4.0 averages 91.33, far above its base model Seed-OSS-36B (69.45) and within 1.56 points of GPT-5.2 (92.89). More impressively, on real financial scenario tasks, its overall score surpasses GPT-5.2—e.g., more practical advice in banking retail customer service, insurance product designs better aligned with regulation, and more insightful securities research reports.

Because it is open source, XuanYuan 4.0 supports low-cost local deployment: financial institutions can run it on-premises, cutting compute costs, avoiding data leakage, and achieving full business autonomy.

Conclusion: Bubble or Disruption?

The bubble is real: models that shine on multiple-choice exams often fail in real business, their "intelligence" being surface-level word matching. But the disruption is real too: XuanYuan 4.0 proves that with careful domain training and methodological innovation, a much smaller model can excel in finance's crucible. FIRE provides the yardstick linking model capability to business outcomes, while XuanYuan 4.0 shows the viable path: deep domain work, disciplined training, and strict boundary control. In a high-value, high-risk field like finance, genuine competence beats flashy showmanship—and AI that can truly read financials, assess risk, and create value is the revolutionary force the industry actually needs.