It did take me quite a while to reverse engineer the encryption. It was mostly done using IDA Pro and SoftIce. The hardest bit was because I had several crypto books, and whichever Microsoft employee implemented their security kept doing what the books said not to do, so I'd never expect to keep encountering the "wrong" way of doing things.
I think the newer RDP versions uses TLS BTW.
But those standards were developed by committee, so the number of ways you could do things was usually the same as the number of committee members. Then their implementation had bugs (eg drawing some parts from bottom to top when the standards said top to bottom), but they didn't know this since they didn't try interoperating with other implementations. The initial "security" was some half assed hack, and yes in later versions they did switch to TLS.
Note that the security is complicated because of domain membership, WAN vs LAN, and a bunch of other related issues.
