Explanation: IF you use SSH agent and have ssh options set up, you get a channel thru SSH where you could use your SSH agent on remote host.
Good side: You can then chain authenticate and say use same SSH agent to authorize sudo, hence getting sudo without password, just secured with your private key. Add hardware token to store said key and you're pretty secure.
Bad side: .... so can any other process with right permissions on the system, therefore compromised system can try to impersonate you.
One way to mitigate is is to make sure servers can't talk to eachother to via SSH, if user can access A and B but A can't access B and vice versa the escalation is limited.
Other way is to set agent to ask every time something wants to use the key which half-solves it (attacker would need to time the attack to occur right before "valid" use") but from what I remember it still doesn't show you what is trying to use your key (at least for gpg-agent's ssh agent functionality) so it's kinda not that useful of a feature.