> Isn't the attack surface limited to the methods used in the application though?
No, because the combination of how you put them together can make you vulnerable.
For example, if you are encrypting lots of messages that all have the same header, certain crypto algorithms can be made to leak the key.
Do you know which combinations those are? I sure don't by default.
The fact that an end programmer even has to think about this is the problem.
In any cryptosystem, the application programmer is the person who knows the least about crypto. If he has to make any decision, you can expect he will get it wrong.