Practical AI did a podcast episode about the dangers of using AI models as a shield to hide behind in justifying your decisions. The episode was titled "Suspicion Machines" and based on the libked article [1], and I think it's worth a read/listen.
[1]: https://www.wired.com/story/welfare-state-algorithms/