I'm failing to see what this has to do with the threat model that port knocking addresses.You said:
I think port knocking is really silly and you shouldn't waste time with it. Disable root logins and password logins in SSH.
I'm pointing out that will lead to less security, not more, for the stated reason.
Suffice it to say that an attacker who has access to your SSH identity file has with virtually total certainty access to your SSH passphrase as well.
How so? By brute for cracking it?