A Neurosurgery Lab Beat GPT-5 on Brain Scans With No Labels and Under 1,000 GPU-Hours

TL;DR

A lab at the University of Michigan trained NeuroVFM, a foundation model for brain MRI and CT, on 5.24 million raw clinical scans using a self-supervised method called Vol-JEPA, with zero manual labels and zero radiology reports. Published in Nature Medicine, it hit 92.7% AUROC across 82 CT diagnoses and 92.5% across 74 MRI diagnoses, wrote radiology reports that made roughly half the key-finding errors of GPT-5, and cost under 1,000 GPU-hours to pretrain. If you build in a narrow domain, the real takeaway is not about radiology: learning representations from your own messy data pool can beat prompting a giant general model, and it can do it cheap.

The bottleneck was never the model. It was the labels.

The default recipe for domain AI in 2026 is depressingly familiar: take a frontier vision-language model, prompt it, and hope. It underperforms, so you fine-tune, which means you need labeled data, which means someone has to label it. In medicine that someone is a radiologist, and if you have ever tried to get a radiologist to annotate 5 million scans, you already know that is the entire ballgame.

The Michigan team, from the Machine Learning in Neurosurgery Lab, skipped the labels entirely. They call the idea health system learning: point a self-supervised model at the uncurated firehose of scans a hospital already produces during routine care and let it learn anatomy and pathology from the pixels alone. No curation, no human annotations, no radiology-report supervision. The training set, UM-NeuroImages, is 5.24 million volumes from 566,915 studies collected over 20 years at Michigan Medicine, which is to say it is exactly the pile of data every large hospital is already sitting on and not using.

How Vol-JEPA works

Vol-JEPA is a volumetric extension of the JEPA family, the joint-embedding predictive architectures behind I-JEPA for images and V-JEPA for video. The core move is subtle and it is the whole reason this works. Instead of masking part of a scan and reconstructing the missing voxels, the model masks part of the scan and predicts the representation of the missing part in a learned latent space. A teacher encoder (an exponential moving average of the model) supplies the target, and a smooth loss pulls the prediction toward it.

Here is the analogy worth holding onto. Pixel reconstruction is like being asked what is behind a curtain and being forced to paint every thread of the curtain to answer. Latent prediction just describes the idea of what is hidden and throws the threads away. Medical scans are full of threads (scanner noise, texture, irrelevant detail) that a reconstruction model wastes capacity memorizing. JEPA predicts the concept, not the confetti, so the representation it learns is about the anatomy, not the artifacts.

The scoreboard

One frozen encoder, then lightweight heads on top, covers 156 diagnostic tasks. NeuroVFM reached 92.7% average AUROC across 82 CT diagnoses and 92.5% across 74 MRI diagnoses, beating report-supervised baselines (HLIP), voxel-reconstruction baselines (NeuroMAE), and a 3DINO contrastive baseline. Latent prediction won, cleanly, over both of the usual alternatives.

The louder result is the head-to-head with the general models. Bolt a small open language model (Qwen3-14B) onto the encoder in a LLaVA-style setup and NeuroVFM writes radiology reports. On those reports it made roughly half the key-finding errors of GPT-5, with fewer hallucinations and laterality slips, and clinical experts preferred its reports by more than two to one. A model you can train on one department's spare compute beat the frontier at the frontier lab's own game, in this one domain.

The part that should change how you build

Now the number that should stop you: the full Vol-JEPA pretraining run used under 1,000 GPU-hours. A 3DINO-style baseline needed more than seven times that for weaker representations. Under 1,000 GPU-hours would not cover a rounding error in a frontier pretraining budget, and it produced a model that outscores those frontier systems on its home turf.

Generalize the recipe and it stops being a medical story. If you own a deep pool of domain data that nobody has labeled (support tickets, sensor logs, legal filings, satellite tiles, code, transactions), a self-supervised objective can turn that pile into a strong domain encoder without paying anyone to annotate it. The frontier lab's advantage is scale and generality. Your advantage is that your data is yours and it is specific, and a JEPA-style objective is how you cash that in.

It also reframes the "just prompt GPT-5" reflex. General models are extraordinary generalists, and generalists lose to specialists on narrow, high-stakes tasks where the specialist has seen a thousand times more in-domain examples. Prompting is a shortcut, not a moat.

The caveats, straight

Read these before you get excited. This is a single academic health system: one scanner fleet, one patient population, one set of imaging protocols, and generalization to other hospitals is exactly the thing that historically breaks medical models. The reported numbers are the authors' own evaluation, not an external prospective trial.

It is also explicitly not a product. The repo ships with a plain "not a medical device, do not use for clinical decision-making" disclaimer, the model still misses some critical findings in testing, and it has no regulatory clearance. The authors frame it as decision support, not autonomous screening. The weights on Hugging Face are non-commercial (CC-BY-NC-SA) and gated behind an institutional email, while the code is MIT. So the paradigm is the open part, not a plug-and-play diagnostician. Treat this as a proof that the method works, then go apply the method to your own domain.

Key Takeaways

• NeuroVFM learns from raw scans with no labels and no reports, using Vol-JEPA to predict masked regions in latent space instead of reconstructing pixels.
• It hit 92.7% AUROC on 82 CT diagnoses and 92.5% on 74 MRI diagnoses from one frozen encoder plus lightweight heads.
• Its generated reports made roughly half the key-finding errors of GPT-5, and experts preferred them more than two to one.
• The full pretraining run took under 1,000 GPU-hours, more than 7x cheaper than a 3DINO-style baseline.
• The transferable lesson: a self-supervised objective on your own unlabeled domain data can beat prompting a frontier model, and cheaply.
• Caveats are real: single health system, non-commercial gated weights, no regulatory clearance, and it is not a medical device.

Sources: Nature Medicine, arXiv 2511.18640, NeuroVFM on GitHub, MarkTechPost