It's not. When you're being blinded by sun, you don't suddenly see elephants or spaceships or comic books. You also don't think the world just disappeared - you know to just ignore the visual input.
Also, I'm more worried about temporary, unexpected configuration of things that suddenly make a NN see something weird.