News · 2026-10-26

A specialist skin model beat two chatbots, but its triage threshold missed melanomas

A newly fine-tuned skin-lesion classifier correctly named 125 of 200 biopsy-confirmed lesions, compared with 84 and 85 for two frontier chatbots in a test published by dermatologist Magnus Lynch. Its default triage threshold nevertheless would have discharged 18 of the 30 melanomas. The experiment shows why better diagnostic classification and safer patient triage require separate evidence.

Key facts

• Lynch Lab published the comparison on October 25, 2026, using 200 lesions with paired clinical and dermoscopic images.
• The specialist's first-choice diagnosis was correct in about 62% of cases, versus about 42% for each chatbot.
• The specialist was a newly fine-tuned Inception v3 classifier; the chatbots were identified as GPT-6 Astra and Claude Fable 5.1.
• The primary source is Lynch's comparison, with training details on the classifier page.

The striking part is the age of the design. Lynch describes the architecture as “ten years old.” That does not mean a forgotten model was taken off a shelf and instantly defeated newer systems. Inception v3 dates to 2015, but this classifier had been trained specifically for the job using 68,011 clinical and dermoscopic images. Its specialization is central to the result.

A convolutional neural network learns visual features useful for assigning an image to a category. A general chatbot has a broader job: it interprets images and language across many settings. Comparing them resembles comparing a technician trained on one inspection station with a general troubleshooter. The specialist can be better at the station without being better at every downstream decision.

Each test lesion had an ordinary clinical close-up and a dermoscopic image, which reveals skin structures through a specialized imaging technique. Lynch asked the chatbots to choose among six diagnoses or express uncertainty, give confidence and a second choice, and decide whether the lesion could be discharged. Each chatbot ran once at default settings. The classifier chose among the same six diagnoses and weighted the dermoscopic view more heavily.

The headline advantage was largest on benign lesions. For melanoma itself, the classifier's first diagnosis was correct in 13 of 30 cases; the chatbots correctly named eight and 11. The classifier also trailed both chatbots on actinic keratosis. Those differences matter because an overall accuracy figure mixes several kinds of error with very different consequences.

The discharge result makes the distinction unavoidable. At the classifier's preset research threshold, 18 melanomas would have been discharged. The corresponding chatbot counts were three and four. Lynch reports that matching thresholds reduced the classifier's melanoma discharges to two or one. That demonstrates how much the decision rule matters; it does not retroactively turn the original threshold into a safe one. Our explanation of receiver operating characteristic curves covers this tradeoff between catching dangerous cases and referring more harmless ones.

The dataset also sets a firm boundary on interpretation. Every test lesion had been biopsied, including the benign lesions. A benign mole chosen for removal is already a difficult case, because someone thought it warranted investigation. The experiment deliberately split the sample evenly between cases judged safe and unsafe to discharge. That differs from the mix encountered in ordinary screening or casual phone photographs.

Lynch says the test lesions, other images of the same lesions, and images of the same patients were excluded from training and validation. However, training and testing still drew from the same broader archive ecosystem. The ISIC MILK10k record describes paired images from 5,240 lesions. The classifier page acknowledges limited darker-skin training examples, while the comparison does not report the skin-tone composition of its 200-case test. Holding out patients is valuable, but it does not establish performance across every population or camera setting.

The Hacker News discussion raised the strongest obvious objection: a purpose-trained specialist should have an advantage on its own task, and a single chatbot run leaves uncertainty about prompts and variability. That is a fair limit on the breadth of the result. The useful finding remains that task-specific training can matter more than architecture age on a bounded test.

This is a lab-page experiment, not a peer-reviewed comparison with dermatologists. No dermatologist-reader arm was reported, and exact chatbot interface identifiers were not provided. Lynch's classifier page describes an experimental system rather than a medical device. The supported conclusion is a narrow classification win with a consequential threshold caveat, not a demonstration that any of these systems can independently decide whether a person needs clinical care.

Primary source, verified: read the paper →

Ground Truth. (2026, October 26). A specialist skin model beat two chatbots, but its triage threshold missed melanomas. Ground Truth. https://groundtruth.day/news/specialist-skin-model-beats-chatbots-but-triage-tells-another-story.html

@misc{groundtruth:specialist-skin-model-beats-chatbots-but-triage-tells-another-story,
  title  = {A specialist skin model beat two chatbots, but its triage threshold missed melanomas},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {oct},
  url    = {https://groundtruth.day/news/specialist-skin-model-beats-chatbots-but-triage-tells-another-story.html}
}

Topics: medical-ai · computer-vision · evaluation · specialist-models