But we also look at a lot of other stars in the sky. If every single one (or almost every single one) had a massive 5x mass Oort cloud around it, it would affect the light we see from that star.
Consider that we can currently detect differences in luminosity small enough to tell whether an Earth-size planet is passing between us and the star. A 5x mass Oort cloud would be thousands of times more mass than that. It would have noticeable effect on luminosity.
And, while our sun has an Oort cloud, there are a lot of stars out there that probably don't--too small, too big, too hot, too young, too old, etc.
