I think this hits the nail on the head: who cares?
Who cares what these people (who are judging you for meeting someone in public) think?
They're very unlikely to be anyone meaningful if they're doing that, since being a good conversationalist is a trait that most successful people possess.
Fuck that yeah, do what you want, fuck what people think about who you are or what you're doing. You'll at least give them a good story to tell about that nut who walked up to that guy on the subway and said, "I dig that jacket, wanna get some coffee together?"