Marketers often use A/B testing as a tool to compare marketing treatments in a test stage and then deploy the better-performing treatment to the remainder of the consumer population. Whereas these tests have traditionally been analyzed using hypothesis testing, we reframe them as an explicit trade-off between the opportunity cost of the test (where some customers receive a suboptimal treatment) and the potential losses associated with deploying a suboptimal treatment to the remainder of the population. We derive a closed-form expression for the profit-maximizing test size and show that it is substantially smaller than typically recommended for a hypothesis test, particularly when the response is noisy or when the total population is small. The common practice of using small holdout groups can be rationalized by asymmetric priors. The proposed test design achieves nearly the same expected regret as the flexible yet harder-to-implement multi-armed bandit under a wide range of conditions. We demonstrate the benefits of the method in three different marketing contexts—website design, display advertising, and catalog tests—in which we estimate priors from past data. In all three cases, the optimal sample sizes are substantially smaller than for a traditional hypothesis test, resulting in higher profit.

中文翻译：

---

测试和滚动：利润最大化的A / B测试

营销人员经常使用A / B测试作为工具来比较测试阶段的营销方式，然后将性能更好的方式应用于其余的消费者群体。传统上使用假设检验对这些检验进行分析，但我们将其重构为检验的机会成本（某些客户获得次优治疗）与在次要治疗的其余部分中部署次优治疗相关的潜在损失之间的明确权衡。人口。我们推导了最大化利润的检验规模的封闭形式的表达式，并表明它比假设检验通常建议的规模小得多，特别是当响应嘈杂或总人口较少时。通过使用不对称先验可以合理地使用小保留组。拟议的测试设计在各种条件下都可以实现与柔性但难于实施的多臂匪盗几乎相同的预期遗憾。我们在三种不同的营销环境（网站设计，展示广告和目录测试）中演示了该方法的好处，在这些环境中，我们可以根据过去的数据估算先验值。在所有这三种情况下，最佳样本量都比传统假设检验小得多，从而带来了更高的利润。