62% vs 42%: A 2015 CNN Outperforms Frontier LLMs in a Dermatology Exam

Forum topic · QianXun · 2026-10-26

Summary

Dermatologist Magnus Lynch of Guy's Hospital, London, ran a small controlled exam comparing Inception v3 (2015) against two frontier large language models, GPT-6 Astra and Claude Fable 5.1, on 200 biopsy-confirmed skin lesions from the MILK10k dataset. The decade-old convolutional network, fine-tuned on 68,011 public ISIC images with one H100 GPU in twenty minutes, scored 62% on first-choice answers, versus 42% for both LLMs. The gap was widest on benign lesions: the CNN recognized 37 of 50 nevi while GPT-6 Astra got 16, and Claude Fable 5.1 correctly identified only 6 of 50 seborrheic keratoses. The LLMs also showed systematic bias, over-predicting basal cell carcinoma and actinic keratosis respectively. Lynch notes limitations: small melanoma sample (30 cases), single runs per case, possible training-data exposure, and a looser default threshold for the CNN that narrows its advantage when aligned. The blog post is not peer-reviewed. The results suggest that specialized pipelines remain more robust than general-purpose models on narrow, high-stakes classification tasks.

Dermatologist Magnus Lynch of Guy's Hospital, London, published a blog post on October 25 describing a small controlled exam that pitted an eleven-year-old convolutional network, Inception v3, against two of today's most prominent frontier large language models: GPT-6 Astra and Claude Fable 5.1. The task was a routine dermatology challenge — classifying skin lesions from images.

The scoreline was unflattering. The veteran scored 62%; the two top performers tied at 42%.

A single exam proves little. But the way this exam was constructed lands squarely where large models are most uncomfortable.

How the exam was built

Inception v3 is not new. The Stanford team's famous 2017 *Nature* demonstration of "dermatologist-level" skin cancer classification used exactly this family of architecture. Lynch fine-tuned it on 68,011 images from the public ISIC skin image archive — one H100 GPU, twenty minutes. He noted in his post that this network was deliberately a *non-state-of-the-art* baseline.

The exam paper came from MILK10k, a public dataset released in 2025: two hundred lesions, each biopsy-confirmed, with answers taken from pathology reports. One hundred belonged to the "cannot-send-home" dangerous group — 30 melanomas, 25 basal cell carcinomas, 25 squamous cell carcinomas, plus 20 actinic keratoses. The other 100 were benign — evenly split between nevi and seborrheic keratoses. Because everything was biopsy-confirmed, the exam was hard: none of the typical cases abundant in everyday image archives made it onto the test.

The scorecard

Each model answered independently, scored on its first-choice answer:

| Contestant | First-choice correct | With second choice | Melanomas (of 30) | Melanomas sent home | |---|---|---|---|---| | Inception v3 (2015) | 125/200 | 173 | 13 | Default threshold 18; 2 or 1 when aligned | | GPT-6 Astra | 84/200 | 134 | 8 | 3 | | Claude Fable 5.1 | 85/200 | 138 | 11 | 4 |

The gap came mostly on benign disease. Of 50 nevi, the veteran got 37 right; GPT-6 Astra managed only 16. Of 50 seborrheic keratoses, Claude got only 6. The failure modes had personality, too: GPT-6 Astra labeled 75 lesions as basal cell carcinoma when the true answer was 25; Claude guessed actinic keratosis 53 times when the truth count was 20. Each large model carried its own hammer and hammered plenty of things that were not nails.

First-choice accuracy, out of 100:

Inception v3: 62
GPT-6 Astra: 42
Claude Fable 5.1: 42

Shelf life of this exam

Lynch listed the caveats himself at the end. The sample is small — only 30 melanomas, so confidence intervals are wide. Each model ran once per case, with no repeated sampling. The exam images come from a public dataset that the LLMs' training corpora may have seen, though his judgment is that prior exposure could only *favor* the models, a bias that cannot explain their lagging results.

There is also a fairness question. The veteran's decision threshold was loose: by default it would send home 18 melanomas, far more than the LLMs' 3 and 4. After aligning the release threshold, the missed count dropped to 2 and 1, narrowing the veteran's advantage. A separate control reading: on ordinary archival images the veteran reaches an AUC of 0.96, but only 0.86 on this biopsy-confirmed exam. The paper is genuinely hard.

The blog post is not peer-reviewed; the analysis scripts were written with Claude Code assistance, and scoring was executed against pathology reports by script with no room for manual intervention.

An old question, a new reading

Ten years ago, Inception v3 made *Nature* by "matching dermatologists." Today it returns to the field under a different comparison: not whether it resembles a doctor, but who is more stable in a narrow specialty. General-purpose models know a bit of everything, but lesion classification is a closed task with a tiny answer space and heavy consequences — exactly where specialized networks are cheap, fast, and ready after twenty minutes of fine-tuning.

The narrative of LLMs as generalist doctors is still running forward. This small-sample report card offers at least one reminder: beneath the generalists, specialized pipelines have not retired — if anything, they look sturdier. As for when the two top performers can hold their ground in narrow specialties, Lynch's exam paper is sitting there — 200 questions, ready for a retake any time.

---

Source: Magnus Lynch, Lynch Lab, "A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis" (2026-10-25); ISIC Archive; MILK10k dataset documentation; Esteva et al., *Nature* 2017 (background).

Caveats: All figures come from that blog post — single author, not peer-reviewed; one run per model per case; 30 melanoma samples with limited statistical power; LLM versions and prompting settings as self-reported in the post; this article contains no medical advice.