You potentially send all your key IDs (ie public keys) your agent has. there are command line options to force your ssh not to use your agent in which case, barred any bugs in the ssh client, it’s like browsing to a domain in your browser.
I think it’s more of a privacy leak. E.g., the service could potentially figure out who you are if your public keys are tied to some other service, like GitHub.
If you ever manage to leak a private key it’s easier to track it back to you. There was a recent paper showing the ability to tie rather a lot of keys to the associated developers.
