A lot of people are very binary. Yes, I'd internally grumble a bit if a business trip cut into a weekend for travel. I'd also sometimes check email over a weekend or PTO and resolve something if I could quickly. But I certainly didn't go into the office over weekends (or, indeed, latterly at all). And I never took time off for a doctor's appointment or other personal matters taking less than a full day.

And, to your point, if I did take PTO for vacation, the degree to which I'd be contactable or not was never a factor.