There are many phone-sized objects, though. It's not so much that you buy pants to fit just your phone, as that you buy pants that are functional according to the current expectations of society on what you should be carrying around with you; and clothiers also take this into account and make most pants with pockets large-enough to fit those same expected accessories.

I mean, I do approach the problem from the other direction as well; one reason I don't buy a phone that's approaching phablet size (other than not being able to hold the things in one hand) is that it wouldn't fit in the pockets of most pants, greatly narrowing the range of pants I could wear (or necessitating an—admittedly probably cheap—tailoring to add deeper pockets.)

But as it stands, I'm not limiting myself much—my phone is only slightly larger than my wallet, so I'm just avoiding clothes that not only wouldn't fit a phone, but wouldn't fit a wallet. There aren't that many pants (for men, at least) that have pockets that small/shallow/tight.