I have an old iPhone SE. It is small enough that I don't even notice it in my pocket. It does everything I want it to do very well and is actually very responsive.
Also it is about $60 bucks on ebay. I have no compelling reason to switch.
I assume no more iOS updates are coming for the SE. Pretty soon your apps won't upgrade, and won't talk to the app servers, and you can't install new apps.
So you don't have a compelling reason to switch now... but in a year or so you unfortunately will. It's what happened to me with my iPhone 5. :(