It's the small 3rd party apps. Stuff like mcdonalds, taco bell, weatherbug, uber eats, etc. The web browser is kinda faster too, but I use firefox on both devices so it's not really a fair comparison on either end. The XS just shows its age in some stuff - I think due to the lack of ram. It'll just randomly hang for a second while it catches up lol. Rare, but it does happen.
The speed thing is balanced out by UI tweaks and things like the Apple Watch that I still use. Unfortunately it was a dumb silly situation where I bought a novelty phone that I actually REALLY enjoy using... and then I don't wanna get rid of my iphone haha.
Edit: Another unfair comparison - the Google Nest app is ludicrously slow on my iPhone.