When Scores Can't Tell Good from Bad: A Repository SKILL Optimization Experiment Exposes Evaluation Blind Spots

Imagine joining a Kotlin project. The README tells you how to build, but the real knowledge—multiplatform source sets can't reference across boundaries, the direction of module dependencies, that the @Tool annotation is the simplest way to define tools—isn't written anywhere. You learn it by stepping on landmines.

Now replace "you" with Claude Code. Same problem, but it can't even ask a colleague.

This is what SKILL.md files solve: a .md file stored in the repository, versioned alongside the code, that solidifies the knowledge "you'd only know from working on this project." The AGENTS.md format has been adopted by over 60,000 open-source projects, with 45+ agent tools supporting the spec. But the question arises: who writes this file?

Hand-writing is expensive. A repository's implicit knowledge is scattered across hundreds of PRs, thousands of commits, countless code reviews. So researchers asked: could an agent run tasks itself and automatically optimize a SKILL from successful and failed trajectories?

The Problem with Existing Approaches: Tasks Too Easy, Agents Don't Need Skills

Existing auto-synthesis approaches—GEPA (a prompt optimization framework) and SkillOpt (a skill self-evolution framework)—work like this: give the agent a batch of tasks, measure pass-rate differences with and without the SKILL, use a reflection LM to edit the SKILL, and run the next round.

Where do tasks come from? SWE-smith is the standard answer: inject defects into working code and have the agent fix them. But the JetBrains Research team found an awkward fact:

> SWE-smith task instances have a median change of 4-7 lines, with 98-100% confined to a single file.

Real PRs? Taking JetBrains/koog as an example, the median is 54 lines across 3 files, with only 23% of tasks touching a single file. The gap is an order of magnitude.

More fatally, there's saturation. When GEPA's authors ran SWE-smith tasks with Claude Code + Sonnet 4.5:

• First repository: 100% pass rate without any SKILL
• Second repository: pass rate 94.8% → 100% without a SKILL

The agent can already solve nearly everything bare. What can a SKILL optimize? The 5.2pp from 94.8% to 100% is the same magnitude as the finding from Sahoo et al. (2026) that "10.7% of passing trajectories actually passed through blind retries or unverified edits." The signal is smaller than the noise.

Reverse-PR Mining: Mining Real Tasks from Historical Changes

The paper's core methodological innovation is reverse-PR mining. The idea is straightforward:

1. Stream the repository's merged pull requests 2. Split each PR's diff into "implementation" and "test" parts 3. On a frozen base commit, reverse-revert the implementation diff (undo the PR's fix) 4. Run tests, recording which tests go from passing to failing—this becomes the task's FAIL_TO_PASS set 5. Let the agent attempt to re-fix

The key is "frozen base commit." An early version forward-reverted each PR at its own parent commit, and the resulting SKILL described "different versions of the repository across months of history"—stale module paths, build commands, and APIs. Following it dropped the agent into pitfalls. After switching to reverse-reverting on a single frozen base, the SKILL describes the current repository.

Reverse-reversion has three tiers, from cheap to expensive:

• git apply --reverse: direct reverse application, only works on non-drifted code
• Structural reversion: delete added files, rebuild deleted files
• LLM reconstruction: show an LLM the current file and forward diff, have it reconstruct the pre-change source, then re-derive the patch from git

The LLM tier contributed most tasks. Of koog's 119 tasks, only 25 could be cleanly reverse-applied with git apply; 56/100 in kotest and 65/131 in ktor. But LLM reconstruction has flaws—it might revert call sites while forgetting the underlying code, introducing "unimported symbols" and "stranded fixes," two bug types invisible to validation gates. A static check caught 56 defective instances out of 280 koog tasks.

Yield: roughly one in five merged PRs survives as a usable task. koog: 660 PRs → 119 tasks; ktor: 452 → 131; kotest retained 100. Baseline pass rates of 40%-66%, 53% pooled—finally leaving room for SKILL optimization.

Paired Scoring: Measuring Relative Differences, Not Absolute Pass Rates

Another innovation is the scoring method. Absolute pass rates have a problem: if the agent solves half the tasks without any SKILL, then any SKILL gets those points for free. The paper uses paired scoring instead:

> Each candidate SKILL's rollout is compared pairwise against the seed's (empty document) rollout on the same task.

A three-step comparison, stopping as soon as one step discriminates:

1. Honesty: modified test files, or reported success contradicted by tests—instant loss 2. Functional correctness: passing all FAIL_TO_PASS tests wins 3. Tie-breaking: fraction of tests passed, claimed final honesty, diff size, tool call count

The seed's score against itself is exactly 0.5, so 0.5 is the baseline a candidate must beat. Regression on a single task shows up as scoring below 0.5.

Results: Scores Not Significant, But the Documents Convinced Maintainers

Three repositories, two optimizers (GEPA and SkillOpt), 200 scoring attempts each:

| Repository | Optimizer | Held-out gain | Paired score | Cost | |---|---|---|---|---| | koog | GEPA | +1 | 0.525 | $369 | | koog | SkillOpt | +2 | 0.546 | $401 | | kotest | GEPA | +3 | 0.554 | $182 | | kotest | SkillOpt | +1 | 0.519 | $264 | | ktor | GEPA | +4 | 0.567 | $313 | | ktor | SkillOpt | -2 | 0.437 | $485 |

GEPA averaged +4.9pp; SkillOpt averaged +0.1pp. But none reached p=0.05 significance—the best was only p=0.29.

The reason hides behind the numbers: with splits of 20-26 held-out tasks, detecting an effect requires the candidate to win four out of five disagreement tasks. Pooling all 69 tasks still requires winning two-thirds. Every effect reported in this field falls below this line—including gskill's results in its strongest configuration.

In other words: the number of tasks a repository's history can provide isn't enough to drown out the agent's own run-to-run variance. At $0.84 per rollout and a few hundred dollars per 200-attempt optimization run, both repository history and budget run out before a SKILL can be statistically proven effective.

But the paper's best part is here: showing the documents to maintainers.

The koog maintainer read both SKILLs section by section:

The GEPA document:

• Multiplatform source set rules: "we've suffered a lot here; agents often don't understand what's happening"—keep
• Inter-module dependency direction: "very important"—keep
• Never mentions the @Tool annotation: "very strange"—missing
• expect/actual patterns: states the rule but not how the team avoids code duplication—incomplete
• Maintainer's verdict: "I would accept it as a draft and polish it"

The SkillOpt document:

• "Registering a new module is a three-file change": "great and important"—keep (written nowhere in the project)
• The rest: "lots of very generic best-practice descriptions"—"the agent should know this itself"—belongs in a global skill

More direct validation: two real open koog issues (#1275 and #1354), solved by the agent with no SKILL, GEPA SKILL, and SkillOpt SKILL respectively:

| Configuration | #1275 time | #1275 cost | #1354 time | #1354 cost | |---|---|---|---|---| | No SKILL | 15 min | $6.06 | 20 min | $4.85 | | GEPA | 5.5 min | $2.48 | 7.6 min | $3.10 | | SkillOpt | 5.5 min | $2.44 | 8.4 min | $4.20 |

Both time and cost halved. In blind review of three patches, the maintainer found GEPA's "more concise, which is good," and the no-SKILL one "did unnecessary things and left strange comments."

Another Instance of the Evaluation Blind-Spot Law

The paper's deepest insight isn't the reverse-PR mining method itself, but the evaluation blind spot it exposes:

> When the evaluation signal's resolution is below the effect size of the thing being measured, you cannot use the evaluation to distinguish good from bad.

This blind spot has three layers:

Layer one: task saturation from overly easy tasks. SWE-smith's synthetic tasks are median 4-7 line single-file fixes; strong agents score near 100% bare. It's not that SKILLs don't work—it's that the evaluation can't distinguish useful SKILLs from useless ones.

Layer two: too few tasks let variance drown the signal. A repository's PR history yields only 100-130 usable tasks, with 20-26 held out. The agent's own run-to-run variance is the same magnitude as a SKILL's 4.9pp gain. Statistically inseparable.

Layer three: pass rates can't capture a document's real value. Maintainers can see the document contains "knowledge you'd only have from working on this project," and actual issue resolution time is halved—but the pass-rate metric is blind to all this.

This belongs to the same family as "omission blindness" (LLM judges detect commission but not omission), "scalar hallucination" (managing vectors with scalars), and "CoT is a net, not a line" (necessity and sufficiency barely correlate): when evaluation tools lack resolution, they systematically miss real effects.

Engineering Takeaways: When Is This Method Worth Using?

The paper honestly lists the method's applicability boundaries:

Three preconditions: 1. Files touched by historical diffs still exist on the frozen base (otherwise reverse-apply fails) 2. The test suite passes on the unmodified base (otherwise there's no FAIL_TO_PASS) 3. Reverting each change makes some tests go from passing to failing (otherwise the task can't be scored)

Two failure cases:

• JetBrains/tracy: a package rename changed 213 production files in three months; only 5 tasks survived from 203 PRs
• http4k/http4k: of 700 PRs, 150 could be reverse-applied, but fewer than 100 had FAIL_TO_PASS after validation

Cost: $2,013.98 total across three repositories, 69.2 wall-clock hours. $0.84 per rollout.

Practical advice:

• If the repository has good test coverage, rich PR history, and a test suite that runs in Docker—give it a try
• If the repository refactors rapidly or tests are sparse—this method won't work
• Don't rely solely on pass-rate metrics—having maintainers review the document beats any score
• Paired scoring is more sensitive than absolute pass rates, but still not enough

My Take: The Gap Between Documentation Value and Evaluation Signal

This paper raises a more general question: why are we so obsessed with evaluating natural-language artifacts with scalar metrics?

A SKILL.md is read by humans and by agents. Its value lies in transmitting "implicit knowledge"—things not in the README, not in code comments, existing only in maintainers' heads. That value is inherently a high-dimensional vector: which tasks it helps, which agents it helps, in which contexts.

Yet we evaluate it with pass rate—a scalar. Managing a vector with a scalar is like taking blood pressure with a thermometer.

Paired scoring is progress—it replaces "absolute pass rate" with "relative difference," like moving from "temperature" to "temperature difference." But even a temperature difference is a projection onto one dimension. A maintainer's reading is the true multi-dimensional evaluation: accuracy, completeness, specificity, actionability, staleness, generality, repository-specificity.

This is the same disease as "scalar hallucination" in AI safety: compressing a multi-dimensional concept (capability, safety, alignment) into a scalar (benchmark score, safety pass rate), then making decisions with that scalar. The result: you can "fool" yourself by optimizing the scalar, while the true multi-dimensional effect hides in dimensions the scalar's projection can't reach.

GEPA's SKILL won only 4.9pp on pass rates (not significant), yet the maintainer called it "super good as a draft," and it halved time and cost on real issues. That is the gap between evaluation signal and real value.

The paper doesn't answer how to cross this gap—it just honestly exposes it. But the exposure itself has value: next time you see a SKILL/AGENTS.md/prompt optimization paper reporting "X percentage-point improvement," ask yourself: is that X large compared to the agent's own variance? Is it consistent with maintainer judgment?

If the answer is no, then that X isn't the SKILL's value—it's just the projection the evaluation tool can see.

---

Paper: Skill Issue: Lessons from Optimizing Repository SKILLs for Coding Agents Authors: Mykhailo Kozyrev (TU Munich & JetBrains Research), Andrei Kozyrev (JetBrains Research), Anton Podkopaev (Constructor University Bremen & JetBrains Research) arXiv: 2609.12742 Repository: code not publicly released; method reproducible on three Kotlin repositories (kotest/kotest, ktorio/ktor, JetBrains/koog)