While I think you're right in principle, there's a lot of reason to think the structure of a brain is more complicated than just the connection map of the neural network. There is a lot of complicated behavior inside each neuron that we don't fully understand yet. They aren't just logic gates.
But, of course, that's just physics. It's not magic, so your point stands.