It also triggers all the time accidentally when my watch was nowhere near my mouth. I frequently look at the watch and see a message like “I don’t understand that” with a transcription of something it thinks I said.
“Hey Siri” is also unreliable, mostly because if you hesitate for a few too many milliseconds it thinks you’ve finished your query. I almost exclusively use Siri by holding the power button on my iPhone or Apple Watch.