I don't own a smartphone, for one, but I could purchase one if it would benefit me. As it is though, I cannot justify the cost and the increased monthly expenses. But for the sake of argument; if I did own a smartphone, I wouldn't want to turn it into a sort of virtual keychain. I can buy three U2F keys for under €60, and have spares. Smartphones are a single point of failure — unless you are in the habit of collecting spare smartphones.

There is also the matter of control. At the moment there are only a handful of smartphones that run an OS that is not under the control of Apple, Microsoft, or Google (Ubuntu smartphones come to mind as an alternative). On the other hand, on a normal computer I can run a full free software stack. To me this seems odd. If I own a piece of hardware, I want to run an OS and applications I can legally modify and tinker with, especially if I use it as a security device. By using tools such as Google Authenticator I get sucked into this walled garden.

The behaviour of hardware tokens such as Yubico's U2F keys on the other hand is completely verifiable — mostly because they are quite simple compared to a smartphone.

(Aside from these personal views on software, I do have high security requirements as well for some services, where a dedicated hardware token is preferable, as you point out.)