You see, in addition to controlling the cellular radio and all of those details, the bass band processor also does real time noise cancellation and a variety of other call quality functions that you would immediately miss if they were not there.
That processor is actually doing a lot of different things and is difficult to remove from a phone and maintain what most people would consider an acceptable user experience.
As another commenter pointed out - the Pinephone is device attempting to do that.
When they are ready, get one. They'll be amazing no matter which OS you end up with.
I don’t think the current dominance of the big two can end until the hardware and software requirements of making a good phone are much much more accessible to normal developers/engineers than they are today.
I could imagine a family of cellular and Wi-Fi devices that present as Ethernet bridges. They'd offer a configuration interface reminiscent of home routers (go to a magic IP either with a REST API or a browser-controllable menu). This eliminates a lot of the delicate, externally facing configuration options and has the side benefit of eliminating a lot of driver development hassle, especially on low-popularity OSs.
They do do this.