When you're poor and child-less, you can still sleep in on weekends, watch Netflix/game all day/night or whatever other (cheap) hobby you have. It's even possible to do a decent amount of travelling on a very tight budget these days (low-cost airlines, couchsurfing...). All things considered, today is not a bad time to be a poor person in terms of being able to have fun.
I think even for middle class, the main cost is actually the time rather than the monetary cost.
But I do agree that opportunity cost is the right framing.