Interpreting such results is a job. Disputing such results in a court, or with an arbiter, would be really hard, unless you're a practitioner of the pattern-recognition craft yourself.
Although I do think that the model weights and parameters should in some way be subpoenable so that the model can be 'compelled to testify' if someone has enough legal, technical and fiscal resources to sue the company.
Some of best examples are the juvenile and and family court systems in a state like Ohio. Not only do they not even TRY to do what is fair or right, they have one simple mandate, pay for their own existence. As soon as the system set up to punish children is made to profit off those same children you have a sick and broken system but thats where we are today. So, VERY little chance we are going to convince a company to make less money for the social good when even out children's courts are run like a business.