Not really, since that would just be kicking the can down the road. Yes, you can have authentication between application servers and HSMs, but ultimately, whatever credential is used for that can be stolen from the application server, and then whoever has it can talk to the HSM.

The real benefit of HSMs is that they can make low-level keys accessible via a high-level interface that restricts the type of operations that can be performed at all. For example, an HSM will never just hand raw keys to application servers, but rather only let them perform various operations (based on their permissions). The higher level and specific these permissions are, the better.