It Solves a Very Specific Waste
Frontier models keep getting stronger and more expensive, while local and cheap models multiply. What teams actually want to know is usually not "who tops the leaderboard," but: for my kind of task, with my prompt, this tool-calling setup, and this agent harness, which model gives the lowest unit cost and the fewest failure modes?
smevals is a Python CLI that splits evaluation into five objects:
- eval — a set of challenges
- task — a single task
- config — records the model, prompt, and other settings
- run — stores one actual execution
- grade — a score over run results
uvx smevals run . -g— execute tasks and grade immediately;uvx smevals grade . --regrade— re-grade existing runs after updating scoring rules;uvx smevals report .— output a score summary by model and config;uvx smevals serve .— launch a local result browser;uvx smevals build .— build a hostable static HTML report.- https://simonwillison.net/2026/Jul/31/smevals
- https://primeradiant.com/blog/2026/smevals.html
- https://github.com/prime-radiant-inc/smevals
- https://github.com/prime-radiant-inc/smevals/commits/main
Configuration is written in YAML; execution and grading are separated, so the same batch of run results can be re-graded with a different grader.
How It Runs
A minimal evaluation directory roughly contains eval.yaml, tasks/, configs/, graders/, checkers/, and a runner. Commands can be chained like this:
A runner only needs to follow an environment variable contract — it can call the llm CLI, or hook up Codex, Pi, or a custom agent. Checkers can do deterministic checks (e.g., is the XML valid, does the output contain a string) or call another model as a judge, returning structured results such as score, metrics, tags, and notes.
Why It Matters for AI Coding
The hard part of AI coding is shifting from "can it generate a code snippet" to "can it complete the task in a repo, pass tests, respect permissions, and leave auditable artifacts." So what should really be evaluated is the configuration combination: model + system prompt + toolset + harness + test and rollback rules.
This dovetails with Codex's Sol-planning / Luna-execution workflow: don't rely on gut feeling that "a cheap model is good enough." Give the same tasks to different model-and-harness combos, record the run artifacts, then compare them with the same grader.
Know the Limits
smevals is not a ready-made SWE-bench replacement and won't automatically design high-quality tasks for you. Evaluation tasks may be too small; checkers may reward formatting over correctness; LLM judges introduce bias; and randomness across runs, tool permissions, and data leakage must be controlled yourself. Its value is turning evaluation into a small, reproducible engineering directory — instead of picking models by impression.
Sources and evidence: