> With the British Post Office, the issue is whether or not a software system is inscrutable

Not sure what "inscrutable" means in that context. Is it supposed to mean it can't be scrutinized?

A law was passed some years ago that says evidence obtained from a computer system should be accepted as true, unless evidence is provided that opens it to question. That means, in the Post Office case, that postmasters couldn't demand that the Post Office prove that Horizon was working correctly. They had to prove that it was defective, which was difficult; they were kicked out of their shops, and denied access to their own records, including the Horizon terminals they had been using.

Of course if the chatbot cannot what is the point of them - I'll have to get something else to verify anything the chatbot says. Sure the chatbot can say "hello" in 10,000 words or whatever, but it can't do anything useful.