I'm always reminded of the software optimisation hierarchy, smallest to highest impact:
1. Micro-optimisation, e.g. programming language choice, tightening a loop, cache hits etc.,
2. Change the algorithms,
3. Change the problem.
Sometimes changing the problem makes the software completely go away!