I'm not advocating everyone and their mother implement their own crypto. But some software diversity is a good thing. These algorithms aren't quite as scary or fickle as the documentation and existing implementations make them seem. Especially if you stick to good software like DJB's stuff.
For instance, using montgomery/edwards curves instead of weierstrass curves eliminates a lot of the difficulty in writing a constant-time implementation of ECC. And the 25519 implementation comes with a fast, constant-time implementation of a prime field type.
Yet even DJB's stuff can be simplified. You can knock off a good 80% of the scary code at a cost of a mere 10% of performance. To Google or Facebook, that may be unacceptable. But to me, that's entirely worth it. Now you have a tiny library that is easy to understand, and easy to audit.