We propose a new framework for optimal experimentation, which we term the “A/B testing problem.” Our model departs from the existing literature by allowing for fat tails. Our key insight is that the optimal strategy depends on whether most gains accrue from typical innovations or from rare, unpredictable large successes. If the tails of the unobserved distribution of innovation quality are not too fat, the standard approach of using a few high-powered “big” experiments is optimal. However, if the distribution is very fat tailed, a “lean” strategy of trying more ideas, each with possibly smaller sample sizes, is preferred. Our theoretical results, along with an empirical analysis of Microsoft Bing’s EXP platform, suggest that simple changes to business practices could increase innovation productivity.

中文翻译：

---

使用肥尾进行 A/B 测试

我们为优化实验提出了一个新框架，我们称之为“A/B 测试问题”。我们的模型与现有文献不同，允许肥尾。我们的主要观点是，最佳战略取决于大多数收益是来自典型的创新，还是来自罕见的、不可预测的巨大成功。如果未观察到的创新质量分布的尾部不太胖，那么使用一些高功率“大”实验的标准方法是最佳的。但是，如果分布的尾端非常胖，则首选尝试更多想法的“精益”策略，每个想法都可能具有较小的样本量。我们的理论结果以及对 Microsoft Bing 的 EXP 平台的实证分析表明，对业务实践的简单改变可以提高创新生产力。