As a counterpoint, I recently switched from an old iPhone SE to a Pixel 4a and the experience could not have been any better. I find the vanilla Android to be as snappy as iOS for daily tasks if not better.
The problem lies with custom Android ROMs on low spec phones which do hamper the user experience quite a bit
The issue with android phones is knowing which are optimized for speed.
A Samsung Galaxy for example costs almost as much as an equivalent iphone and is significantly less responsive then some motorola phones with stock android
I've switched from an iPhone 7 to an iPhone 11 Pro last year and while the iPhone 7 never struggled as badly the Androids I had before that (where keypresses might take seconds to register at times in the end), it did struggle somewhat to keep up with me at times, whereas the 11 Pro felt (still feels) like a fast desktop computer, in that it never appears like the UI or everyday operations noticeably tax it at all, the UX in that regard is super smooth. I'd expect the SE to fare a bit worse than the 7, since it had an A9 vs. an A10 SoC.
