No, they can't do that. On iOS it is the App Store or nothing when it comes to native apps. (Of course, if users jailbreak their phones they have more freedom to install apps, but I assume we're just talking about the standard OS here).

On Android the publisher can bypass the store by distributing their own APK files. Some carriers (glancing towards AT&T) lock this down in some phones, but by default Android is happy to install new apps from anywhere as long as the user agrees to a dialog warning them of the dangers of running random code they download from the net.

This is one of many reasons I'm currently an Android user and self-described "Apple hater".