Unless they've improved it over the last couple of years, Google's voice match is pretty easy to fool: When I played a recording of my friend's voice saying "hey Google" and then completed the sentence myself, the Home Mini thought I was him and let me access his calendar even though we have completely different voices.
Of course this is all academic. Anyone can get into a home via the windows. It's just that half of all Americans live in a home that has a gun, so getting out alive might not be as easy as getting in.