There are plenty of good apps I've sideloaded, but thinking of the majority of the population being relatively tech-illiterate, there's a big difference between a bad actor having to convince someone to change their settings to enable sideloading (despite a big, scary warning) and then convincing them to install an app versus just convincing someone to run/install a random APK. Case in point: a huge percent of the population unknowingly opens and/or installs random bad binaries on Windows every day.
Sideloading (and all that entails) should absolutely be allowed, but the "scary warnings" do a great job preventing a huge portion of people from unknowingly opening a huge threat avenue to some of their most valuable devices/information.