pass can encrypt to multiple keys. I use one key for my laptop and one for my phone. If someone gets your gpg key then they can decrypt all your passwords if they also get the repository. It would be prudent to change all your passwords in this case (pass-rotate could help a little with this). If you lose the key, you lose access to all your passwords.