My experience is the opposite. When I'm not on the go, the likelihood that my AirPods pick up the right device is slim -- like trying to connect to an iPad on a complete different floor vs. the phone I'm trying to make a call on.
When I'm on the go, the only device they can find is my phone, and all works great.