You are probably also oblivious to the very real consequences of other people knowing various things or having partial information, etc.
It's stressful because it has consequences. If you don't find it stressful, you probably are failing to connect the dots and see that x outcome is related to y behavior.