My guess is that the signaling servers are centralized. You need those for connection establishment between two peers. Once they 'know' each other, they may aswell talk to each other directly. Routing all calls through central servers would yield an enormous bill every month. On the other hand, it is sometimes necessary as direct communication between peers may not be possible (NAT, firewalling). IIRC, a central server is also required for multi-party calls.