This is why any good org will make sure to observe all important KPI’s while doing an A/B test. If your “email signup” KPI went to the moon but tanked your “bought shit” metric… you should probably roll back.
It's really easy for this to be noise from false negatives. On an A/A test with five guardrail metrics and a threshold of p>0.05, you'll get a false negative 22.6% of the time.
