Your personal RSA/ECC SSH keys are there for the initial key exchange, after which a session key is created (cycled after
x bytes /
y seconds).
But if you're handling hundreds or thousands of operations per second (e.g., a web site), then the delays add up.