DISCLAIMER: I'm not even a rookie at crypto; I'm just trying to make sense of what I'm reading, like you are.

From [1]:

> The cybercriminals replaced RC4 with their own encryption algorithm using XOR swaps and operations. The domain names to which connections are made and the bsh parameter from the cfg.ini file are used as encryption keys.

A bit later:

> The new protocol encryption algorithm for communications between the botnet control center and infected machines ensures that the botnet will run smoothly, while protecting infected computers from network traffic analysis, and blocking attempts of other cybercriminals to take control of the botnet.

So we have authentication of the C&C through encryption ("block other cybercriminals"), and obfuscation of the payload through encryption ("protect from network traffic analysis"). I suppose the bsh parameter is used for auth, and the domain name just to scramble the payload.

[1] http://www.securelist.com/en/analysis/204792180/TDL4_Top_Bot

Maybe the server generates the private key/public key pair, then derives the domain name from the public key (eg, long-random-string-that-comes-from-public-key.dyndns.com)? This derivation process could be the "encryption algorithm" the article refers to?

The client is given the new servername via the P2P network, then derives the public key from the hostname, encrypts using it and communicates with the server which can decode using the private key.

I'm not sure what the max length of a hostname is, which might be a problem.

Another possibility is to put the public key in a TEXT DNS for the server hostname. That could be what they meant maybe?