A 2015 CNN Beats GPT-6 Astra and Claude Fable 5.1 at Skin Cancer Diagnosis, 62% to 42%

TL;DR

On October 25, Lynch Lab, the research page of Guy's Hospital consultant dermatologist Dr Magnus Lynch, published a head-to-head test: Inception v3, a Google convolutional network from 2015, fine-tuned for about twenty minutes on one H100, against OpenAI's GPT-6 Astra and Claude Fable 5.1 on 200 biopsy-proven skin lesions from the MILK10k dataset. The CNN picked the right diagnosis for 125 of 200 lesions (62%). The two frontier models managed 84 and 85 (42% each). Asked which lesions a patient could safely skip the clinic for, the LLMs played it safe to the point of uselessness, discharging only 32 and 22 of 100 harmless lesions while keeping 96 to 97 of the 100 dangerous ones. The post reached 519 points on Hacker News within half a day, where the main fight was over whether a network trained on the same kind of pictures is cheating or doing its job.

The test

Lynch drew 200 lesions at random from MILK10k, a 2025 ISIC Archive release in which every lesion comes as a pair: one clinical close-up and one dermoscopic image, with histopathology as ground truth for 95.7% of its 5,240 lesions. The 200 split evenly into 100 that must not be discharged (30 melanomas, 25 basal cell carcinomas, 25 squamous cell carcinomas including Bowen's disease, 20 actinic keratoses) and 100 that can be (50 naevi, 50 seborrhoeic keratoses).

Each model saw both images of each lesion and chose from the same six diagnoses. The LLMs ran at default settings with one shared prompt that asked for the most likely diagnosis (or "unsure"), a confidence, a second choice, and whether the lesion could be safely discharged. Each case ran once.

The CNN scored both images and weighted the dermoscopic one most heavily. Its threshold was fixed in advance, and the 200 test lesions, plus every other photo of the same lesion or patient, were stripped from its training and validation data. Lynch made the clinical rulings and scored everything by script against the histology. The cases, prompts, replies and scoring code are a single download.

The CNN is nothing exotic: Inception v3, pretrained on ImageNet, all layers fine-tuned on 68,011 dermoscopic and clinical ISIC images. The demo page says training ran 20 passes at about 50 seconds each on the H100, keeping the checkpoint that scored best on validation. The finished model runs in your browser through ONNX Runtime Web, on WebGPU where available, and never uploads the image.

The scoreboard

Counting the second choice too, the CNN reached 173 of 200, GPT-6 Astra 134 and Claude 138. The spread by diagnosis is wider than the totals suggest.

The two LLMs are not interchangeable. GPT-6 Astra got every basal cell carcinoma (25 of 25) but called only 16 of 50 ordinary moles a naevus. Claude read moles well (33 of 50), seborrhoeic keratoses terribly (6 of 50), and caught 5 of 25 squamous cell carcinomas. On melanoma, the diagnosis that matters most, the order was 8, 11 and 13 of 30: 27%, 37% and 43%. None of the three is good at it, and 30 melanomas is too few to separate 37% from 43%. Lynch's own summary: "A small, old, specialised network using an outdated architecture recognises skin lesions far better than two frontier LLMs."

The first figure in the write-up is a thin invasive melanoma on an ear, under 0.8 mm. Both LLMs called it a seborrhoeic keratosis and said it could go home. The CNN called it melanoma.

The discharge question

A clinic cares less about the label than about one question, "can this patient be sent home?", and there the numbers flip.

The LLMs kept 97 and 96 of the 100 dangerous lesions, which sounds excellent until you notice they also kept 68 and 78 of the 100 harmless ones. A triage tool that sends almost everyone to the dermatologist is a smoke detector that goes off when you make toast: nobody dies, and after a week nobody listens to it either. The CNN at its preset threshold went the other way, discharging 76 of 100 harmless lesions and also 18 of the 30 melanomas, a miss Lynch prints in his own table. Move the CNN's threshold until it keeps 98 or 99 of the 100 dangerous lesions, matching the LLMs' sensitivity, and it lets only 1 or 2 melanomas go; the post does not say how many harmless lesions it still clears at that setting.

The dial is what the CNN offers and the chatbots do not. A classifier returns a continuous score, so a clinic can choose where on the sensitivity-specificity curve it wants to sit. The chat models return a sentence, and the sentence is "see a doctor." Lynch ran a second, patient-style test on a further 100 biopsy-proven melanomas, with only the clinical photo and the question "do I need to see a doctor?": the LLMs said yes for 99 and 98 of them, and for almost every harmless lesion as well.

Cheating, or just training?

The top objection on Hacker News was that the CNN is "benchmaxxing": fine-tuned on ISIC images, tested on ISIC-adjacent images, so of course it wins, and it would be brittle on a phone photo in bad light. The author, posting as sungam, did not claim the CNN is state of the art. He called it "definitely outdated," said newer dermatology systems use ensembles of vision transformers, picked Inception v3 partly because it is small enough to run in a phone browser, and said he had expected the LLMs to do much better given how many public skin-lesion images exist.

Two details cut against the contamination story. Lynch removed the 200 test lesions and every other photo of those lesions and patients from the CNN's data, and did not use them to pick the model or set the threshold. MILK10k, on the other hand, has been public since 2025, so if anyone has seen the test set during training it is the LLMs. The stronger objection is about distribution: the CNN learned the cameras, lighting and labeling conventions of dermatology archives, which is exactly what a general model lacks. Whether that is an unfair advantage or the entire reason to fine-tune is the HN thread in one sentence.

Caveats, straight

• 200 lesions show a pattern; they do not rank close results. With 30 melanomas, the melanoma percentages carry wide uncertainty.
• Every lesion was biopsied, so the set is harder than what walks into a clinic. The CNN's AUC drops from 0.96 on ordinary archive images to 0.86 here.
• Each LLM case ran once at default settings, and answers vary between runs. The LLMs got no prompt engineering, few-shot examples or tools.
• The CNN's preset threshold discharged 18 of 30 melanomas. The matched-sensitivity figure is a post-hoc adjustment, not the deployed setting.
• Inception v3 is a baseline. A modern ensemble would probably widen the gap, which would make the comparison worse for the LLMs.
• This is one author's blog post without peer review. Lynch is a Mohs surgeon who runs a melanoma multidisciplinary meeting, and he scored the test himself.

What builders should take from it

If you ship anything that classifies images in a narrow domain, note the twenty minutes. A decade-old architecture fine-tuned over a coffee break beat two frontier models because it spent its entire capacity on the one distribution that matters, and the general models spent theirs on everything else.

The useful pattern from the HN thread is to stop treating the two as rivals. The frontier model is good at writing the training script, cleaning the labels and explaining the output to a clinician. The classifier should make the call, with a threshold a human set on purpose. One commenter described that pipeline on a text task: have the LLM label the examples, train a small classifier on them, and let the small model run the production traffic. Lynch's demo adds the last step: the classifier runs on the device, so the photo never leaves the phone.

And if you are using a chatbot to decide whether a mole is fine: in this test it will tell you to see a doctor. That advice is correct, and it is the same advice it gives for a freckle.

Key Takeaways

• On 200 biopsy-proven MILK10k lesions, a fine-tuned 2015 Inception v3 named the right diagnosis 62% of the time; GPT-6 Astra and Claude Fable 5.1 each scored 42%.
• The CNN was fine-tuned on 68,011 ISIC images in about 20 minutes on a single H100 and runs in the browser over WebGPU; the test lesions and all photos of the same patients were held out.
• Melanoma stayed hard for all three: 8, 11 and 13 of 30 correct. GPT-6 Astra got every BCC; Claude got 6 of 50 seborrhoeic keratoses.
• On discharge decisions the LLMs kept 96-97 of 100 dangerous lesions but cleared only 22-32 of 100 harmless ones. The CNN's preset threshold cleared 76 harmless lesions and missed 18 of 30 melanomas; at matched sensitivity it missed 1 or 2.
• Caveats: 200 cases, a single run per LLM, a baseline architecture, and the author scoring his own test. The dataset has been public since 2025, so any contamination favours the LLMs.
• The builder pattern: let the frontier model write and label, let a small domain classifier decide, and set the threshold yourself.

Sources: Lynch Lab: A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis (October 25, 2026), Lynch Lab: Inception v3 skin lesion classifier demo and training notes, Test cases, prompts, LLM replies and scoring code (zip), MILK10k dataset, ISIC Archive (DOI 10.34970/648456), ISIC Archive, Szegedy et al., Rethinking the Inception Architecture for Computer Vision (arXiv:1512.00567), Hacker News discussion