Some of my best code has been written with pair programming but I can see the flip-side. There's no day-dreaming, there's no zoning out, complete engagement all day long.
It's the same mental overhead as with taking care of toddlers; you have to be constantly aware otherwise suffer the consequences.. :)