I do wish they kept better historic purchase data. I assume it's deleted absurdly quickly for compliance?
AB testing is beyond UI. It could be AB testing algorithms for recommendations or search results. When i worked for a CPG company, we would AB test the sample sizes in the boxes. To answer the question what is right amount of product for a sample. We would test shipping to speed, was it worth the upgrade to expedited shipping in terms of LTV to justify the extra cost.
It reminds me of the sentiment I sometimes hear from my teenage children before I explain to them that the world isn't a Bernie Sander/Reddit gotcha soundbite and that reality is complicated.
Or, as anyone who has been on the internet since the 90s will tell you, bang on the damn money.
The downside with hired testers is that they are unlikely to be a statistically representative sample of your target market. I don’t think this is actually a problem for most startups or new products being launched by major players, since a lot of UX issues early in product development will be obvious to anyone who isn’t the original developers, and the number of actual users you have won’t be big enough to serve as a statistically representative sample of your target market anyway.
explaining what your product does is hard when you’re the product expert. A/b testing ad copy helps folks parse what your product does.
If your product is actually good then having more people understand what it is, is also good.
