← back to journal

Testing my retina model on a clinic it had never seen

Oct 2026 PyTorch medical imaging evaluation 2 min read

My diabetic retinopathy model scored an AUROC of 0.976 on test images from the same clinics it was trained on. That number felt great, and it's also the least useful number I have. A screening tool gets used at new sites, with different cameras and different patients, so the real test is a clinic it has never seen.

My role: a solo project.

The test

I trained on DeepDRiD (clinics in China) and tested on IDRiD, a clinic in India. 257 IDRiD images were held out completely: nothing was trained, calibrated or tuned on them. On those images the model reached an AUROC of 0.938, caught 88% of referable cases (sensitivity 0.882) and correctly cleared 85% of healthy eyes (specificity 0.854).

It passed its own acceptance gate, and the service refuses to load any model that fails it. But 18 referable eyes in that set were still missed, which is why every output is meant to support a human grader, not replace one.

Three mistakes that would have flattered me

Patient leakage. With a naive random split, 142 of 400 patients ended up in both training and test, so the model would have been tested on eyes it had already seen. Splits are now grouped by patient.

Image size. At 224 pixels, the usual default, a tiny lesion shrinks to about 1.3 pixels and can't be told apart from noise. I train at 512.

Calibration. Temperature scaling, the popular fix for overconfident models, made calibration worse here. Isotonic calibration took the error from 0.102 to 0.031.

Ungradable is not healthy

In real screening, 5–20% of photos are unusable, and a blurred photo must never be quietly graded as healthy. A simple sharpness check couldn't tell them apart (AUROC 0.499, a coin flip), so I trained a quality model on human quality ratings (AUROC 0.811), with classical checks as a backstop.

Letting the model say "I'm not sure"

Cases near the decision boundary, with scores between 0.29 and 0.91, fall into an abstention band and go to a human grader instead of getting a confident wrong answer. Above it, the case is referred; the served operating point is 0.67. Every grader verdict is logged, so drift can be spotted after deployment.

What still isn't good enough

The external test set is small, so the true sensitivity could still be below my 0.85 target, and that has to be confirmed on a larger sample from the site that deploys it. The model isn't plug-and-play either: each new site needs its own labelled calibration sample first. It's research software, not a medical device.

What I'd tell anyone building one

Report the external number, not the internal one, and design for the photos and cases your model will get wrong. That's the part a clinic will actually feel.