Violence does refer to direct physical harm. There's a subset of people, predominantly in the social sciences, that is trying to redefine violence to include things like "economic violence" and "social violence" (e.g. breaking up with an SO you don't like, or not being friends with someone anymore). This is not the average person's understanding of the word, and personally I feel that it drastically washes down the meaning of violence.
If these things constitute violence then lots of violence is completely legal. In fact, you commit violence probably every day when you decide who to be friends with and who not to be friends with, who gets hired, etc.
