Statistical Evaluation Playbook for Defensible Results in ECE/CS Papers

> Paper: How to Do Statistical Evaluations in ECE/CS Papers: A Practical Playbook for Defensible Results > Author: Bhaskar Krishnamachari > arXiv: 2605.00428 | 2026-04-29

1. The Trap of the "Impressive but Untrustworthy" Number

Imagine you are reviewing a paper that claims: "Our method improves over SOTA by 5%!"

But you notice:

• Only one experiment run
• No confidence intervals
• No statistical significance testing
• Tested on just one dataset

Is the number credible? Probably not.

2. Why Statistical Evaluation Matters

The tutorial emphasizes:

> "Strong experimental papers do not rely on a single impressive number. They rely on a complete chain of design, measurement, analysis, and validation—choices that together make results credible."

Common mistakes:

1. Missing hypotheses — results presented without stating what is being verified 2. Unclear units — "5% improvement" of what? Absolute or relative? 3. Ignoring variability — reporting only means, no variance or confidence intervals; single runs may reflect random fluctuation 4. Data snooping (p-hacking) — trying many parameter settings and reporting only the best

3. An Evaluation Workflow for Defensible Results

1. State a Claim — clear and verifiable (e.g., "our method is faster than A") 2. Form a Hypothesis — H0: method = baseline; H1: method > baseline 3. Determine the Unit of Analysis — per sample, per user, or per run? This affects test choice 4. Design the Experiment — controlled variables, randomization, blinding where applicable 5. Measure and Report — mean + std/confidence interval, effect size, statistical significance (p-values), and practical significance 6. Validate — cross-dataset validation, ablation studies, sensitivity analysis

Courtroom analogy: Claim = accusation; hypothesis = fact to be proven; experiments = evidence collection; statistical tests = evidence evaluation. Only the complete chain convinces the reader.

4. Why This Paper Matters

• For beginners: avoid common statistical errors, build correct experimental habits, write credible papers
• For reviewers: spot suspicious claims, ask the right questions
• For the field: improve research quality, reduce irreproducible results

5. Feynman-Style Judgment: Honesty Is the Best Strategy

Feynman said:

> "The first principle is that you must not fool yourself—and you are the easiest person to fool."

Statistical evaluation is not for proving you are right, but for honestly testing whether you might be wrong. Good scientists seek evidence that falsifies their claims, not just data that supports them. This embodies the core of the scientific method: falsifiability, transparency, and reproducibility.

6. Takeaways

If you are writing a paper or running experiments, ask yourself:

1. Is my hypothesis explicit? 2. Did I report variability (variance / confidence intervals)? 3. Did my results pass statistical significance testing? 4. Did I avoid data snooping and selective reporting?

The core lesson: good research is not just about getting good results, but about presenting them honestly. Even the most impressive number is meaningless if it cannot withstand scrutiny. Statistical evaluation is not bureaucratic procedure—it is a cornerstone of scientific integrity. In the ocean of data, statistics is a compass: not to take you anywhere you want to go, but to ensure you honestly report where you are.