But radiologists are very busy and this could help many people. Put a strong disclaimer in there. Open it up to subscriptions to everyone. Charge $40 per analysis or something. Integrate some kind of directory or referral service for human medical professionals.
Anyway, I hope some non-profit organizations will see the capabilities of this model and work together to create an open dataset. That might involve recruiting volunteers to sign up before they have injuries. Or maybe just recruiting different medical providers that get waivers and give discounts on the spot. Won't be easy. But will be worth it.
Like, that's the only REAL reason? Not the technological or ethical implications? The dangers in providing people with no real concept of how any of this works the means to evaluate themselves?
On the surface those all sound like additional reasons not to make it available. But they are also great rationalizations for those who want to maintain a monopoly on analysis.
Personally I found all the comparisons to other AI performance bothersome. None of those were specifically trained on diagnostics AFAICT. Comparison against human experts would seem to be the appropriate way to test it. And not people just out of training taking their first test, I assume experts do better over time though I might be wrong on that.
The infantilization of the public in the name of “safety” is offensive and ridiculous. In many countries, you can get the vast majority of medicines at the pharmacy without a prescription. Amazingly, people still pay doctors and don’t just take random medications without consulting medical professionals.
It’s only “necessary” to limit access to medical tools in countries that have perverted the incentive structure of healthcare to the point where, out of desperation, people will try nearly anything to deal with health issues that they desperately need care for but cannot afford.
In countries where healthcare costs are not punitive and are in alignment with the economy, people opt for sane solutions and quality advice because they want to get well and don’t want to harm themselves accidentally.
If developing nations with arguably inferior education systems can responsibly live with open access to medical treatment resources like diagnostic imaging and pharmaceuticals, maybe we should be asking ourselves what is it, exactly, that is perverting the incentives so badly that having ungated access to these lifesaving resources would be dangerous?
Not to mention that in particularly sick patients problems tend to compound one another and exams are often requested to deal with a particular side of the problem, ignoring, perhaps, the major (but already known and diagnosed) problem etc.
Often times factors specific to a hospital play crucial role: eg. in hospitals for rich (but older) patients it may be common to take chest X-rays in a sited position (s.a. not to discomfort the valuable patients...) whereas in poorer hospitals siting position would indicate some kind of a problem (i.e. the patient couldn't stand for whatever reason).
That's not to say that automatic image reading is worthless: radiologists are, perhaps, one of the most overbooked specialists in any hospital, and are getting even more overbooked because other specialists tend to be afraid to diagnose w/o imaging / are over-reliant on imaging. From talking to someone who worked as a clinical radiologist: most images are never red. So, if an automated system could identify images requiring human attention, that'd be already a huge leap.
- X-Ray: $20
- Radiologist Consultation: $200
- Harrison.AI interpretation: $2000 - X-Ray: $15
- Radiologist Consultation: $125
- Harrison.AI interpretation: $20
It's more likely that regardless of disclaimers people will still use it, and at some point someone will decide that that outcome is still the provider's fault, because you can't expect people to not use a service when they're impoverished and scared, can you?
Unfortunately, it's the other way around. The tech sector understands very little about clinical medicine, and therefore spends its time fighting windmills and shouting in the dark at docs.
I'm curious whether this AI model would have been able to detect my issue more competently than the shitty doctor.
I guess the reasoning is that they want to provide „good service“ by giving the patient something to work with directly after the exam and the workload is so high that they couldn’t look at the images so fast. And they accept the risk that some people are getting angry because their exam wasn’t normal in the end.
But on the scale a typical radiology practice operates today, the few patients who don’t have a normal exam don’t matter (the number of normal exams in an outpatient setting is quite high).
I find it highly unethical, but some radiologists are a little bit more ethically relaxed I guess.
What I want to say is that it might be more of a structural/organisational problem than incompetence by the radiologist in your case.
(Disclaimer: I’m a radiologist myself)
Surely your results went to a requesting physician who should have been following up with you? Radiology doctors don’t usually organise follow up care.
Or was the inaccurate result from the requesting physician?
A quick look at the paper in the BMJ shows that the model did not sit the FRCR 2b examination as claimed, but was given a cut down mock up of the rapid reporting part of the examination invented by one of the authors.
https://www.bmj.com/content/bmj/379/bmj-2022-072826.full.pdf
Were the same tests also used here?
However, the actual FRCR 2B Rapids exam question bank is not publicly available and the FRCR is unlikely to agree to release them as this would compromise the integrity of their examination in the future- so the test used are mock examinations, none of which have been provided to the model during training.
"The Fellowship of the Royal College of Radiologists (FRCR) 2B Rapids exam is considered one of the leading and toughest certifications for radiologists. Only 40-59% of human radiologists pass on their first attempt. Radiologists who re-attempt the exam within a year of passing score an average of 50.88 out of 60 (84.8%).
Harrison.rad.1 scored 51.4 out of 60 (85.67%). Other competing models, including OpenAI’s GPT-4o, Microsoft’s LLaVA-Med, Anthropic’s Claude 3.5 Sonnet and Google’s Gemini 1.5 Pro, mostly scored below 30*, which is statistically no better than random guessing."
But if someone is able to connect a network to the raw data outputs from CT or MR machines, one may start seeing these AI's radically outperform humans at a fraction of the cost.
For CT machines, this could also be used to concentrate radiation doses into parts of the body where the uncertainty of the current state is greatest, even in real time.
For instance, if using a CT machine to examine a fracture in a leg bone, one could start out with a very low dosage scan, simply to find the exact location of the bone. Then slightly higher concentrated scan of the bone in the general area, and then an even higher dosage in an area where the fracture is detected, to get a high resolution picture of the damage, and splinters etc.
This could reduce the total dosage the patient is exposed to, or be used to get a higher resolution image of the damaged area than one would otherwise want to collect, or possibly to perform more scans during treatment than is currently considered worth the radiation exposure.
Such machines could also be made multi modal, meaning the same machine could carry both CT, MR, ultrasound sensors (dopler + regular). Possibly even secondary sensors, such as thermal sensors, pressure sensors or even invasive types of sensors.
By fusing all such inputs (+ the medical records, blood sample data etc) for the patient, such a machine may be able to build a more complete picture of a patient's conditions than even the best hospitals can provide today, and a at a fraction of the cost.
Especially for diffuse issues, like back pains where information about bone damage, bloodflow (from the Doppler ultrasound), soft tissue tension/condition etc could be collected simultaneously and matched with the reported symptoms in real time to find location where nerve damage or irritation could occur.
To verify findings (or to exclude such, if more than one possible explanation exists), such an AI could then suggest experiments that would confirm or exclude possibilities, including stimulating certain areas electrically, apply physical pressure or even by inserting some tiny probe to inspect the location directly.
Unfortunately (or fortunately to the medical companies), while this cold lower the cost per treatment, the market for such diagnostics could grow even faster, meaning medical costs (insurance/taxes) might still go up with this.
I still see somewhat of a product gap in this whole area when selling into clinics but that can likely be solved with time.
“AI is a bubble”
We’re still scratching the surface of what’s possible. I’m hugely optimistic about the future, in a way I never was in other hype/tech cycles.
- one of the speakers at a recent health+AI event
I'm wondering what others in healthcare think of this. I've been skeptical about the death of software engineering as a profession (just as spreadsheets increased the number of accountants), but neither of those jobs requires going to medical school for several years.
Radiology remains one of the most competitive and in-demand specialties. In this year's match, only 4 out of ~1200 available radiology residency positions went unfilled. Last year was 0. Only a handful of other specialties have similar rates.
As comparison, 251 out of ~900 pediatric residency slots went unfilled this year. And 636 out of ~5000 family medicine residency slots went unfilled. (These are much higher than previous years.)
However, I do somewhat agree with the speaker's sentiment if for a different reason. Radiologist supply in the US is roughly stable (thanks to the US's strange stranglehold on residency slots), but demand is increasing: the number of scans ordered on a per patient continues to rise, as does the complexity of those scans. I've heard of hospital systems with backlogs that result in patients waiting months for, say, their cancer staging scan. One can hope we find some way to make things more efficient. Maybe AI can help.
You make it sound like the reporting radiologist is given a referral with helpful, legible information on it. That this ever happened doubtful.
https://harrison.ai/news/reimagining-medical-ai-with-the-mos...
I'd interpret it as a foundation model in the radiology domain
NB. In all claims I've seen so far about outperforming radiologist, the common denominator was that people creating these models have mostly never even seen a real radiologist and had no idea how to read the images. Subsequently, the models "worked" due to some kind of luck, where they accidentally (or deliberately) were fed data that made them look good.
This seems generally aligned with AI realities today: it won't necessarily replace whole job functions but it can increase productivity when applied thoughtfully.
How is chatgpt the competion? It’s mostly a text model?
I'd be 2x as productive if I could just speak and it auto filled my template in the correct spots.
And while you’re at it, the current ‘integrations’ between RIS and PACS are so jarring it sets my teeth on edge.
I recently joined [Sonio](https://sonio.ai/platform/), where we work on AI-powered prenatal ultrasound reporting and image management. Arguably, prenatal ultrasounds are some of the more challenging to get right, but we've already deployed our solution in clinics across the US and Europe.
Exciting times indeed!
Prenatal ultrasounds are one of the most rote and straight forward exams to get right.
From their benchmarks it's looking like a great model that beat competition, but I will see the third party tests after they get released to determine the real performance.
"We have proprietary access to extensive medical imaging data that is representative and diverse, enabling superior model training and accuracy. "
Oh, I'd love to see the loicenses on that, :^).
I'd imagine access to the model itself will remain pretty exclusive, but would love to see them adopt a more open approach.
> Filtered for plain radiographs, Harrison.rad.1 achieves 82% accuracy on closed questions, outperforming other generalist and specialist LLM models available to date (Table 1).
The code and methodology used to reach this conclusion will be made available at https://harrison-ai.github.io/radbench/.