Category: tip · AI coding / evaluation infrastructure Published: 2026-07-31, repo still being updated as of August 1 Sources: Simon Willison and the Prime Radiant blog; the smevals GitHub repository
The specific waste it addresses
Frontier models keep getting stronger and more expensive, while local and cheap models keep multiplying. What teams actually want to know is usually not "who is number one on the leaderboard" but: for my kind of task, this prompt, this tool set, and this agent harness, which model has the lowest cost per unit of work and the fewest failure modes.
smevals is a Python CLI that splits evaluation into five objects: eval is a set of challenges, task is a single task, config records the model, prompt, and other settings, run stores one actual execution, and grade is the scoring of that run. Configs are written in YAML; execution and grading are separated, so the same batch of runs can be re-scored with a different grader.
How it runs
A minimal eval directory roughly contains eval.yaml, tasks/, configs/, graders/, checkers/, and a runner. Commands can be chained like this:
uvx smevals run . -g: run tasks and grade immediately;uvx smevals grade . --regrade: re-score existing results after updating grading rules;uvx smevals report .: output score summaries by model and config;uvx smevals serve .: start a local results browser;uvx smevals build .: build a hostable static HTML report.- https://simonwillison.net/2026/Jul/31/smevals
- https://primeradiant.com/blog/2026/smevals.html
- https://github.com/prime-radiant-inc/smevals
- https://github.com/prime-radiant-inc/smevals/commits/main
A runner only needs to follow an environment-variable contract, so it can call the llm CLI or hook up Codex, Pi, or a custom agent. Checkers can do deterministic checks — e.g. whether the XML is valid or the output contains a given string — or call another model as a judge, returning structured results such as score, metrics, tags, and notes.
Why it matters for AI coding
The hard part of AI coding is shifting from "can it generate a snippet of code" to "can it complete a task in a repo, pass tests, respect permissions, and leave auditable artifacts." So what should actually be evaluated is the configuration combination: model + system prompt + tool set + harness + test and rollback rules.
This connects neatly with the Codex Sol-planning / Luna-execution workflow: instead of going on gut feeling that "a cheap model is good enough," hand the same tasks to different models and harnesses, record the run artifacts, and compare them with the same grader.
Its limits, stated honestly
smevals is not a ready-made SWE-bench replacement, and it will not design high-quality tasks for you automatically. Eval tasks may be too small; checkers may reward format over correctness; LLM judges introduce their own bias; and randomness across runs, tool permissions, and data leakage all need to be controlled by you. Its value is turning evaluation into a small, reproducible engineering directory instead of picking models by impression.
Original sources and evidence: