1) People who work remotely are clustered in companies that allow remote working. If you don't work for a company that will let you work remotely then you probably don't work with any remote workers.
2) Remote workers can often choose their location so they cluster in more desirable locations which, statistically speaking, you probably do not live in.
I worked remotely for 2011-2015 within a company that had approximately 1/3 of its workforce working remotely. As I could go where I wanted I spent most of my time overseas. Spent a lot of time in SE Asia. Places like Chiang Mai in Thailand and Bali in Indonesia have no shortage of people working remotely either as an employee, freelancing or starting a business.