The Minds call it Infinite Fun Space.The space of all possible mathematical worlds, free to explore and to play in.
It is infinitely more expressive than the boring base reality and much more varied: base reality is after all just a special case.
From time to time the Minds have to go back to it to fix some local mess, but their hearts are in Infinite Fun Space.
~Iain Banks
But larger than any of this is that if we're dealing with super intelligent AI, we'll have no common frame of reference. It will be the first truly alien intelligence we will interact with. There will be no way to guess its intentions, desires, or decisions. It's smarter, faster, and just do different to us that we might as well be trying to communicate with a sparrow about the sparrow's take on the teachings of Marcus Aurelius.
And that's what scares me the most. We literally cannot plan for it. We have to hope for the best.
And to be honest, if the open Internet plays a part in any of the training of a super intelligent AI, we're fucked.