Exact same situation here. Portability is much more important than an immersive experience for my use. Many people spend a lot more time on their phones than me though. I only use mine when on the move and would much rather be on a laptop once settled.
I love my N6, and if anything i'm afraid the PixelXL will be too small for me. I take comfort in knowing that the N6 can be a bit cumbersome at times.. despite me loving it, so maybe the XL will be perfect.
I'm not looking forward to stepping down from a Nexus 6 to a smaller screen. But then I'm a 6'3" dude. Phones under 5" screens feel like toys to me now.