Evaluating Sampling-Based Reasoning in Foundation Models: Comparative Analysis and Experimental Validation

Forum topic · ✨步子哥 · 2025-11-09

Summary

This report evaluates the sampling-based reasoning capabilities of major foundation models, comparing LLaMA-2 70B, GPT-4, and PaLM 540B across logic reasoning (ARC-Challenge), math word problems (GSM8K), and commonsense reasoning (MMLU). Experiments show larger closed-source models consistently outperform open-source alternatives: GPT-4 achieves roughly 60% on ARC, 70% on GSM8K, and 75% on MMLU, versus about 45%, 30%, and 60% for LLaMA. Sampling strategy experiments on GSM8K reveal a clear trade-off between temperature and accuracy (75% at temperature 0.2 dropping to 60% at 1.0), while Top-k/Top-p truncation, chain-of-thought prompting (lifting accuracy to ~80%), and self-consistency voting each improve reasoning quality or stability. A meta-reasoning experiment shows GPT-4 can verify solutions, detecting ~70% of deliberately introduced errors and locating the first erroneous step in ~60% of cases, suggesting latent metacognitive and self-correction abilities. The report concludes that model scale is a decisive factor for reasoning, open-source models are closing the gap, and evaluation should consider reasoning processes rather than final answers alone.

Key points

This forum post presents an evaluation report on sampling-based reasoning in foundation models, comparing three representative models — LLaMA-2 70B (open source), GPT-4 (closed source), and PaLM 540B (ultra-large scale) — across three benchmark tasks, and investigating how sampling strategies affect performance.

Methodology

Models: LLaMA-2 70B (trained on public data), GPT-4 (state-of-the-art closed model), PaLM 540B (ultra-large Pathways-trained model).
Tasks and datasets:
Logic reasoning: ARC-Challenge (accuracy)
Math problem solving: GSM8K (accuracy plus step-matching rate)
Commonsense reasoning: MMLU, 57 subjects (accuracy)
Sampling strategies tested: greedy decoding (baseline), temperature sampling (0.2 / 0.7 / 1.0), Top-k (k=50) and Top-p (p=0.9) sampling, chain-of-thought (CoT) prompting, and self-consistency (majority voting over 20 samples).
Evaluation emphasizes zero-shot reasoning to expose the models' latent, untuned capabilities.

Experiment 1: Model comparison

ARC: GPT-4 ≈ 60%, PaLM ≈ 55%, LLaMA ≈ 45%.
GSM8K: GPT-4 ≈ 70%, PaLM ≈ 65%, LLaMA ≈ 30% — the largest gap, showing deep multi-step reasoning favors scale. LLaMA often makes calculation errors mid-chain.
MMLU: GPT-4 ≈ 75%, PaLM ≈ 70%, LLaMA ≈ 60% — smallest gap, partly because multiple-choice format allows elimination strategies.
Conclusions: model scale strongly correlates with reasoning ability (emergent abilities), and open-source models reach roughly comparable performance on breadth-of-knowledge tasks.

Experiment 2: Sampling strategy effects (GSM8K, GPT-4)

Temperature: accuracy falls from ~75% (T=0.2) to ~70% (T=0.7) to ~60% (T=1.0). Low temperature favors correctness; high temperature increases path diversity at the cost of errors.
Top-k / Top-p: each adds ~2 points of accuracy and ~3 points of step-matching over unconstrained sampling, by filtering out irrelevant tokens.
Chain-of-thought prompting: the largest gain — accuracy from ~70% to ~80%, step-matching from ~60% to ~75%, most effective at low temperature.
Self-consistency: only ~1 point of accuracy gain over single sampling, but improves robustness on problems where occasional samples are correct.
Best practical combination for precise reasoning: low temperature + candidate-set truncation + chain-of-thought.

Experiment 3: Latent intelligence via meta-reasoning

Inspired by the MR-GSM8K benchmark, the report builds a meta-reasoning test set where correct GSM8K solutions are paired with solutions containing one deliberately injected error. GPT-4 must judge correctness, locate the first erroneous step, and explain the error, scored by an MR-Score.

Correct solutions are judged correct nearly 100% of the time.
~70% of erroneous solutions are detected; the first wrong step is precisely located in ~60% of those cases.
Explanations are often plausible (e.g., identifying that an addition should have been a subtraction), though some misattribute the root error.

The authors interpret this as evidence of latent metacognition and self-correction potential: models can reflect on and evaluate reasoning processes, not just produce answers — implying uses such as automatic graders or self-debugging assistants.

Conclusions

1. Scale matters: larger models (GPT-4, PaLM) dominate on all reasoning tasks, confirming scale effects. 2. Open-source catching up: LLaMA reaches roughly comparable performance on knowledge-breadth tasks despite weaker deep reasoning. 3. Strategies are tunable: sampling parameters and prompts let practitioners trade off accuracy versus diversity. 4. Evaluate the process, not just the answer: models can be right for the wrong reasons, so meta-reasoning benchmarks are needed to measure deep reasoning and latent intelligence.