A lot of outlets have pointed out that the M1 chip came out in this kind of interim period between Intel and AMD releases, so performance comparisons were a little misleading.
If you compare M1 chips to more recent AMD chips (I'm less clear about Intel), they're more similar. There's differences in power use and singlecore versus multicore attributes but overall they're much more similar than you'd conclude based on when M1 first came out. M1 is still very nice and efficient, but not absurdly better, or maybe even better at all.
There's a lot of issues with memory use (leaks, excessive use) on M1 laptops. There was just a post about this here on HN. The issue is sometimes raised without awareness of it as a general trend, but it keeps coming up over and over. So far I haven't seen any explanation for why it happens or how to avoid it, but it's real. Just Google "macbook memory leaks" and you'll see plenty of discussion. As far as I know, there's been some red herring solutions but no actual resolution.
3-4seconds would drive me insane :O