The problem is that I’d have to unplug that adapter to use my headphones with a device other than my phone, including other Apple devices like my Macbook. I would subsequently loose the adapter, guaranteed.

If the world could agree to use USB-C for everything, that might work, but Apple—who started this mess—is a prime holdout there.