Tested against dataset consisting of only multiple choice questions and was only able to achieve 67% accuracy on MedQA (the medical licensure examination.) LLMs alone are not the way forward in this area (and in this instance not generalizable, nor capable of handling necessary edge cases the job demands.)
These are far inferior to professionals and just like what we saw with Watson, hyping them up as job replacements or assistants when they're clearly not up to snuff is not only foolish for the perception of the industry as a whole, but wildly irresponsible.
