The vast majority of people do just that. The small number of people who actually attend the Olympics in person are 1) journalists covering it 2) relatives of the athletes 3) people living near the venue who attend a few events out of curiosity and maybe have discounted tickets because of where they live 4) wealthy people who can afford the high cost of decent tickets with a clear view of the athletes.
The Olympics sound a little miserable to watch in-person because you probably won't have a clear view of a sport you're not all that interested in. And unlike most popular sports where it's an event where you pick a side to cheer for, most Olympic events are individual, so there isn't 90 minutes of cheering for your team, hoping they'll win.
Watching a simple competition like the 100M dash, sure. But watch something worth seeing despite the competition such as the gymnastics, figure-skating, or wrestling perhaps, and I at least can just sit and enjoy the skill.
