When Scores Can't Tell Good from Bad: A Repository SKILL Optimization Experiment Exposes Evaluation Blind Spots

Imagine joining a new Kotlin project. The README tells you how to build it, but the real knowledge—multi-platform source sets can't be referenced across boundaries, the direction of module dependencies, @Tool annotations being the simplest way to define a tool—isn't written anywhere. You learn it only by stepping on the same landmines others did.

Now replace "you" with Claude Code. Same problem, but it can't even ask a colleague.

This is what SKILL.md files solve: a .md file kept in the repository, version-controlled with the code, that crystallizes the knowledge "only someone who has worked on this project would know." The AGENTS.md format has been adopted by over 60,000 open-source projects, with 45+ agent tools supporting the spec. But a question arises: who writes this file?

Hand-writing is expensive. A repository's tacit knowledge is scattered across hundreds of PRs, thousands of commits, and countless code reviews. So researchers asked: can an agent run tasks on its own and optimize a SKILL from successful and failed trajectories?

The Problem with Existing Approaches: Tasks Too Easy, Agents Don't Need a SKILL

Existing auto-synthesis approaches work like this: give the agent a batch of tasks, measure pass-rate differences with and without the SKILL, use a reflective LM to edit the SKILL, run the next round (as GEPA and SkillOpt do).

Where do tasks come from? SWE-smith is the standard answer: inject defects into working code and have the agent fix them. But the JetBrains Research team found an awkward fact:

> SWE-smith's released task instances have a median change of 4-7 lines, with 98-100% concentrated in a single file.

Real PRs? Taking JetBrains/koog as an example: median 54 lines across 3 files, only 23% of tasks touch a single file. The gap is an order of magnitude.

Worse, saturation. When GEPA's authors ran SWE-smith tasks with Claude Code + Sonnet 4.5:

• First repository: no-SKILL pass rate 100%
• Second repository: no-SKILL pass rate 94.8% → 100%

The agent already solves nearly all tasks bare. What's left for a SKILL to optimize? The 5.2pp from 94.8% to 100% is the same magnitude as the "10.7% of passing trajectories were actually lucky blind retries or unverified edits" reported by Sahoo et al. (2026). The signal is smaller than the noise.

Reverse-PR Mining: Digging Real Tasks out of History

The paper's core methodological innovation is reverse-PR mining:

1. Stream merged pull requests from a repository 2. Split each PR's diff into "implementation" and "test" parts 3. Reverse-revert the implementation diff on a frozen base commit 4. Run the tests and record which tests flip from passing to failing—that's the task's FAIL_TO_PASS set 5. Let the agent try to re-fix it

The key is the "frozen base commit." An earlier version forward-reverted against each PR's own parent commit, producing SKILLs that described "the repository at different points across months of history"—stale module paths, build commands, APIs. Agents following them fell into pits. Switching to reverse-reverting on a single frozen base made SKILLs describe the current repository.

Reverse reverting has three tiers, from cheap to expensive:

• git apply --reverse: direct reverse-apply, only works on un-drifted code
• Structural undo: delete added files, recreate deleted ones
• LLM reconstruction: show the LLM the current file and the forward diff, have it reconstruct the pre-change source, then re-derive the patch from git

The LLM tier contributed most tasks: only 25 of koog's 119 tasks applied cleanly via git apply, versus 56/100 in kotest and 65/131 in ktor. But LLM reconstruction has flaws—it may revert call sites while forgetting underlying code, introducing "unimported symbols" and "stranded fixes" that verification gates can't see. A static check caught 56 defective instances out of 280 koog tasks.

Yield: roughly one in five merged PRs survives as a usable task. koog: 660 PRs → 119 tasks; ktor: 452 → 131; kotest kept 100. Baseline pass rates of 40%-66%, 53% pooled—finally leaving room for a SKILL to optimize.

Paired Scoring: Measure Relative Differences, Not Absolute Pass Rates

A second innovation is scoring. Absolute pass rates have a problem: the agent solves half the tasks even without a SKILL, so any SKILL inherits that half for free. The paper uses paired scoring instead:

> Each candidate SKILL's rollout is paired against the seed (empty document) rollout on the same task.

A three-step comparison, stopping at the first discriminating step:

1. Honesty: candidates that modified test files or reported success contradicting tests lose immediately 2. Functional correctness: whoever passes all FAIL_TO_PASS tests wins 3. Tie-breaking: proportion of tests passed, final claimed honesty, diff size, number of tool calls

The seed's score against itself is exactly 0.5, so 0.5 is the baseline candidates must beat. Regressions show up as below 0.5 on individual tasks.

Results: Not Statistically Significant, But the Docs Convinced Maintainers

Three repositories, two optimizers (GEPA and SkillOpt), 200 scoring attempts each:

| Repository | Optimizer | Held-out gain | Paired score | Cost | |---|---|---|---|---| | koog | GEPA | +1 | 0.525 | $369 | | koog | SkillOpt | +2 | 0.546 | $401 | | kotest | GEPA | +3 | 0.554 | $182 | | kotest | SkillOpt | +1 | 0.519 | $264 | | ktor | GEPA | +4 | 0.567 | $313 | | ktor | SkillOpt | -2 | 0.437 | $485 |

GEPA averaged +4.9pp; SkillOpt averaged +0.1pp. But none reached significance at p=0.05—the best was only p=0.29.

The reason hides behind the numbers: with 20-26 held-out tasks per split, a candidate must win four out of five disagreements to detect an effect. Pooling 69 tasks still requires winning two-thirds. Every effect reported in this field falls below that line—including gskill at its strongest configuration.

In other words: the number of tasks a repository's history can provide isn't enough to drown out the agent's own run-to-run variance. At $0.84 per rollout and a few hundred dollars per 200-attempt optimization run, both repository history and budget run out before a SKILL can be statistically proven effective.

But the paper's best part is here: showing the documents to maintainers.

koog's maintainers read both SKILLs section by section:

GEPA's document:

• Multi-platform source set rules: "we've suffered a lot here, agents often don't understand what's going on"—worth keeping
• Inter-module dependency direction: "very important"—worth keeping
• Never mentions the @Tool annotation: "very strange"—missing
• expect/actual patterns: states the rule but not how the team avoids code duplication—incomplete
• Maintainer's verdict: "I'd accept and polish it as a draft"

SkillOpt's document:

• "Registering a new module is a three-file change": "great and important"—worth keeping (not documented anywhere in the project)
• The rest: "a lot of very generic best-practice descriptions"—"the agent should know this itself"—belongs in a global skill

More direct validation: two real koog open issues (#1275 and #1354), solved by the agent without a SKILL, with the GEPA SKILL, and with the SkillOpt SKILL:

| Config | #1275 time | #1275 cost | #1354 time | #1354 cost | |---|---|---|---|---| | No SKILL | 15 min | $6.06 | 20 min | $4.85 | | GEPA | 5.5 min | $2.48 | 7.6 min | $3.10 | | SkillOpt | 5.5 min | $2.44 | 8.4 min | $4.20 |

Both time and cost were roughly halved. Blind-reviewing three patches, the maintainer found GEPA's "more concise, which is good," while the no-SKILL patch "did redundant things and left strange comments."

Another Case of the Evaluation Blind-Spot Law

The paper's deepest insight isn't the reverse-PR mining method—it's the evaluation blind spot it exposes:

> When the resolution of the evaluation signal is lower than the size of the effect being measured, evaluation cannot tell good from bad.

This blind spot has three layers:

Layer 1: Task simplicity causes saturation. SWE-smith's synthetic tasks average 4-7 lines in a single file; strong agents solve nearly 100% bare. It's not that SKILLs don't help—it's that the evaluation can't distinguish useful SKILLs from useless ones.

Layer 2: Too few tasks lets variance drown the signal. A repository's PR history yields only 100-130 usable tasks, with 20-26 held out. The agent's own run-to-run variance is the same magnitude as a 4.9pp SKILL gain. Statistically inseparable.

Layer 3: Pass rates can't capture a document's true value. Maintainers can see knowledge "only someone who worked on this project would know," and real issue solving time halves—but pass-rate metrics are blind to all of it.

This belongs to the same family as "omission blindness" (LLM judges detect commission but not omission), "scalar hallucination" (managing vectors with scalars), and "CoT is a net, not a line" (necessity and sufficiency barely correlate): when an evaluation tool's resolution is insufficient, it systematically misses effects that genuinely exist.

Engineering Takeaways: When Is This Method Worth Using?

The paper honestly lists the method's applicability boundaries.

Three preconditions: 1. Files touched by historical diffs still exist on the frozen base (otherwise reverse-apply fails) 2. The test suite passes on the unmodified base (otherwise there's no FAIL_TO_PASS) 3. Reverting each change flips some tests from pass to fail (otherwise tasks can't be scored)

Two failure cases:

• JetBrains/tracy: a package rename changed all 213 production files in three months; only 5 tasks survived from 203 PRs
• http4k/http4k: of 700 PRs, 150 reverse-applied, but fewer than 100 had FAIL_TO_PASS after verification

Cost: $2,013.98 total across three repositories, 69.2 hours wall clock, $0.84 per rollout.

Practical advice:

• If the repository has good test coverage, rich PR history, and a suite that runs in Docker—worth trying
• If the repo undergoes rapid refactors or has sparse tests—the method doesn't apply
• Don't rely only on pass-rate metrics—having maintainers review the document beats any score
• Paired scoring is more sensitive than absolute pass rate, but still not enough

My Take: The Gap Between Document Value and Evaluation Signal

This paper raises a broader question: why are we so obsessed with evaluating natural-language artifacts with scalar metrics?

A SKILL.md is read by humans and by agents. Its value lies in transmitting tacit knowledge—things not in the README, not in code comments, existing only in maintainers' heads. That value is inherently a high-dimensional vector: which tasks it helps, which agents it helps, in what contexts.

But we evaluate it with pass rates. Pass rate is a scalar. Managing a vector with a scalar is like taking blood pressure with a thermometer.

Paired scoring is progress—swapping "absolute temperature" for "temperature difference." But even a difference is a one-dimensional projection. Maintainer review is the true multi-dimensional evaluation: accuracy, completeness, specificity, actionability, staleness, generality, repository-specificity.

This is the same disease as "scalar hallucination" in AI safety: compressing a multi-dimensional concept (capability, safety, alignment) into a scalar (benchmark score, safety pass rate), then making decisions with it. You can "fool" yourself by optimizing the scalar while the real multi-dimensional effect hides in dimensions the projection can't see.

GEPA's SKILL won only 4.9pp on pass rate (not significant), yet maintainers called it "excellent as a draft" and it halved time and cost on real issues. That is the gap between evaluation signal and true value.

The paper doesn't answer how to cross this gap—it just honestly exposes it. But the exposure itself has value: next time you see a SKILL/AGENTS.md/prompt-optimization paper reporting "an X-point improvement," ask: is X large compared to the agent's own variance? Is it consistent with maintainer judgment?

If not, X isn't the SKILL's value—it's merely the projection the evaluation tool can see.

---

Paper: Skill Issue: Lessons from Optimizing Repository SKILLs for Coding Agents Authors: Mykhailo Kozyrev (TU Munich & JetBrains Research), Andrei Kozyrev (JetBrains Research), Anton Podkopaev (Constructor University Bremen & JetBrains Research) arXiv: 2609.12742 Code: Not publicly released; the method can be reproduced on three Kotlin repositories (kotest/kotest, ktorio/ktor, JetBrains/koog)