Put gun to persons ahead.
Ask them to do a division.
Then screaming at them "HOW DID YOU DO THAT, TELL ME NOW, OR YOU'RE TOAST".
Even most humans would splutter and not be able to answer.
A human also can't explain how their neurons calculated a division, or anything else, or 'intention', even if a gun is pointed at them.
So, why is that a fault of AI if it can't explain how itself is working? It is not a requirement for AGI.