The fonts we actually use are interchangeable, and people outside the font bubble won't even notice the differences.
For a more directly relevant example, companies frequently A/B test changes to a UI to see which ones people like better. The specifics of those changes would be pretty marginal if you didn't know what it looked like before (like if you're a new user, you wouldn't notice if the notification was red versus purple, or what the wording in the menu is). Despite this, there are some sites that just "feel" better in a way that you can't really describe.
All of this is a long-winded way of saying that I can't tell if I'm looking at Arial, Helvetica, or this Nebula Sans font unless they were side by side (and even then I'd just be saying they're different, not identifying them by name). But I think the site would feel a lot less modern if it were written in Times New Roman. I think you'd notice if it were too hard to read when small, and I think if it looked "bad," you'd at least subconsciously notice that.
Again, just because you’re unable to notice how exactly you’re being affected does not mean you aren’t. You also don’t notice all the ways you’re affected by advertising, but they still work on you.
Yes, of course not every single subtle change to a font makes a huge impact. Just like a single subtle change to a colour’s hue doesn’t. But pronounced changes do, even when you’re unable to put your finger on it.
If it were a non-fashion criteria, surely we'd be hitting a local maximum on readability.
I don't need or want my everyday use font to "affect" me, or to "make impact" -- that's the branding world, again, and not aligned with readability.
There is no perfect font, just like there is no perfect framework. You pick what suits you or makes sense for your project. Sometimes you don’t understand your requirements until you try to use something.
> I don't need or want my everyday use font to "affect" me, or to "make impact"
And being aware of the details is the best way to avoid that.