The only threat a passphrase protects against that an encrypted home doesn't is someone walking up to my unlocked laptop (it locks automatically after 5 minutes, and I make a point of locking it when I'm leaving). If you run ssh-agent, you're exposed to this threat anyways.

EDIT: After some thought, passphrases can protect against a poorly executed attack, where just files are copied wholesale, or an accidental leak. So while theoretically you're not protected, it may be sufficient for some practical security. I'll be looking into using ssh-agent in the near future.