I don't know about its cryptography specifically, but Stuxnet, for example, is a botnet regarded as a game-changer precisely because it was such a professional hit.
BTW, I'm trying to see if there's any interest in a collaboration on the cryptanalysis of this custom encryption system. Anyone with an interest please ping me. It may turn out to be nothing.
(2) Do you have the source of it? Do you have a non-obfuscated version?
(3) Even if the answer to both questions in (2) is "yes", I still doubt any serious cryptographers would take a look at it. They (mostly) do things to write papers, and you don't get a paper out of "we broke a really weak encryption algorithm in a botnet". The chances that the paper is instead "this botnet has a reasonable encryption algorithm" are so slim, they won't consider it.
(4) That said, there are a number of non-serious cryptographers who would find it an interesting challenge, but if they can't break it, it still doesn't mean it's any good.
(2) No, I haven't heard of anyone getting the source. There are 4.5M PCs with an obfuscated binary, so that should be obtainable. We can de-obfuscate it ourselves (it obviously has to load into memory at some point), or we can also ask other researchers who have already done this. The latter is likely to be successful according to 'who' and 'how serious' we have interested.
(3) You never know and it never hurts to ask. Don't forget, they also often have students looking for projects. Anti-botnet ops is an active research topic in data security.
(4) Yep. I doubt anyone would consider using it for anything else.
There's nothing preventing a scenario where someone with nefarious intent and a large pile of money convinces a skillful cryptographer into building a cipher robust enough be resistant to cryptanalysis.
(I'm not your downvoter by the way)
Still you could easily end up with something much less secure than any of them used independently. For example, read about the "meet-in-the-middle" attack.