If so, this is a really interesting argument that I've never come across before. I do agree that it wastes company time, but I would think in such small quantities that it had very little impact -- on par with having a 5 minute chat with your office mate about their weekend.
I still have the impression that for many people who do object to this sort of thing, their objection stems from a feeling that being asked out at work is an uncomfortable or even frightening thing, something a person should not have to be subjected to.