We’ve found that a successful approach to A/B testing is really dependent on the type of company you’re operating and product that you’re offering. Small cosmetic changes to the UI or copy often result in equally small changes to click-thru or conversion rates, and so these A/B tests require relatively greater levels of statistical power in order to achieve significance.
For mega-traffic companies like Google or Amazon, these kinds of tests are worth the cost of testing because a sub-1% lift still contributes substantially to their bottom line.
But for everyone else, ‘shallow’ A/B tests of a button color or call to action will often yield inconclusive results. Here’s an article from the founder of GrooveHQ detailing such an experience: http://www.groovehq.com/blog/failed-ab-tests.
If you’re running a small or medium business – or even a larger one that does not have the scalable testing practices of a tech giant like Amazon in place – testing deeper changes to the product, UI layouts or entire UX workflows are what move the needle. This is what we’re now calling ‘empathic A/B testing’ – where tests are designed with empathy for users.
If you ask the questions: What changes can I make to my product or website that would motivate my users to take the actions I want them to take? What are they looking for? What do they care about? And why? More often than not, I think you’ll find that the answer is not ‘a different button color’
In the end, A/B testing is really a very unsophisticated way of answering the question ‘What works better?’ because you are sending a fixed proportion of your users to a suboptimal variant for the duration of the test. We’ve done a lot of research into better solutions to this problem, and have found that a dynamic approach using a learning algorithm always leads to faster results and higher average conversion rates. You can read more about that here: http://splitforce.com/resources/auto-optimization/
