I think the difference is found in empathy. Take two cases: One, I feel bad when I see a homeless person because my empathy makes me feel bad, and I want to remove that bad feeling and thus I help them. Getting them 'out of sight' might reduce the bad feeling a little, but any time I think of their being homeless people, I still feel bad for them.

Two, I feel bad because the homeless person is making the place look worse. So I want them to be gone. Once they are out of sight, I no longer feel bad.

I think the two above are more similar that many people treat them because of how selfishness and empathy interact, but I do also think their is still a significant difference because one is based off of empathy and the other is based off of pure selfishness.