(Of course, even better would be a Bayesian decision analysis which takes into account the full posterior distribution, costs, and benefits, allowing one to decide whether the evidence is enough to justify the more expensive box or whether one should extend the experiment further to collect more data.)
* quick, if conversions are 0.25%, what does a 0.1SD mean difference between arms of an A/B test mean? 'Uh...' Exactly.