What about the patients getting unnecessary treatments? How upset should they be? What about the student expelled for AI plagiarism due to a false reading? These things are unreliable, and despite an infinite amount of caveats there is no way to prevent people from over relying on it. We might as well dunk people in the water to see if they float.
That’s a weird kind of extortion, a demand that we placate a subset of the population to the detriment of others. If a conflict came down to people who understand stats versus those blind to it I would put my money on those who understand stats.