The publishers have voluntarily opted-in to ad exchanges because those exchanges provide value by aggregating demand in a way that would be difficult for all but the largest publishers to do themselves (and the largest publishers do, which is why FB/Google/Snap/Twitter/etc all sell their own ads).

It's not as if anyone is forcing publishers to not deal directly with buyers. The exchanges certainly don't have the market power to do so.