Well, obviously something went wrong. But the way I understand it RSA shouldn't work
at all if you used a composite factor, decrypting a message will just give a wrong result.
Unless, by some incredible fluke, they managed to find a carmichael number.