If you get unprpivileged access to a system, and somehow manage to crash sshd, or win a race to bind port 22 when sshd restarts, you can intercept other logins.

If you can bind port 80,you can gets ssl certs via let's encrypt (which could let you intercept not just web, but also smtp/imap etc).

So yes, it can make a difference. Of course - it's better if the user doesn't have access to begin with.

This might be more interesting for classical multi-user servers than "single use" servers that don't allow "regular" users to login via ssh.