> How free you are is then expressed as a graph of all possible actions you may take which are not prohibited by the threat of legitimate violence

It's a good point to think in terms of possible actions you may take. But violence isn't the only thing that can prune that graph.

Say I come across an orchard surrounded by an [unclimbable] fence. I want to eat some fruit in the orchard, but cannot because of the fence. There is no violence I face that prevents my action, and no violence I can level to take the action. Yet my action is prohibited by another, and thus my freedom limited.