That was my thinking as well. I honestly hated the lack of headphone jack.
After getting an iPhone XS from work - replacing my iPhone 5 - I found that I would actually just leave the adapter on my headset. If I ever used the headset for anything else I would take it off but immediately replug it after being done.
My conclusion is that it's been waaaaay less of an issue that I honestly expected it to be.