You probably want to encrypt it with a public key for which the raspi has the private key. You could validate the public key with a qr-code on the raspi, or some other pre-arranged scheme.
This is a very similar situation to being able to access the device on the LAN over tcp/ip and needing to pass it a secret.