We have no idea how they work. We know the training making a model and generating things from it, but the thing we want from it being an Oracle or something, we have no idea how that works at the level of the specific knowledge it generates. And when we dig into it doesn't reveal anything very interesting. The premise of put everything digital in a pile and see if makes something like a god is a neat idea, but the god is literally playing dice.