Well, what I mean is that we'd ideally have a situation where there are cheap dedicated hardware modules which contain your private key, so that you never have to trust your private key to a phone or a laptop. A phone can play that role and likely will at some point, but I'd still like something that is dumber than a phone, because I don't want the device itself to be able to harbor malware. Plus, the idea of such an HSM would be that you carry it around and then you can plug it in (or wirelessly connect it) to any system you're using, and all that system would be able to do is ask it to use its private key (which the system wouldn't be able to see) to calculate the response to some authentication query. If you're worried that your terminal might be compromised, you wouldn't really want to connect it to your phone.

Also, Google Authenticator is similar to a really fancy password because it's a symmetric key system. It's nice because you only have to enter in one-time passwords on untrusted devices, but the downside is that both you and the service you're authenticating with has the same key, meaning that if either you or the service you're authenticating with is compromised, the attacker can authenticate as you. With SSL, the service being compromised doesn't actually get the attacker anything except your public key.