This paper explores the use of designed experiments in an online environment. Motivated by real-world examples, we model a scenario where the practitioner is given a finite set of units and needs to select a subset of these which are expended toward a one-shot, multi-factor designed experiment. Following this phase, the designer is left with the remaining set of unused units to implement any learnings from the experiments. With this setting, we answer the key design question of how much to experiment, which translates to choosing the number of replicates for a given design. We construct a Bayesian framework that captures the expected cumulative gain across the entire set of units. We derive theoretical results for the optimal number of replicates for all two-level, full and fractional factorial designs with seven factors or fewer. We conduct simulations that serve as validation of the theoretical results, as well as enabling us to explore scenarios and techniques of analysis that are not captured in the theoretical studies. Our overall results indicate that the optimal allocation of units for experimentation varies from 1 to $$20\%$$20% of the total units available, which is mainly governed by the experimental environment and the total number of units. We conclude that experimenting with the optimal number of replicates recommended by our study can lead to a cumulative improvement which is 80–95% greater than the expected cumulative improvement gained when a practitioner chooses the number of replicates randomly.

中文翻译：

---

在线框架下设计实验的最佳重复

本文探讨了设计实验在在线环境中的使用。受现实世界示例的启发，我们模拟了一个场景，在该场景中，从业者被给予一组有限的单位，并需要选择其中的一个子集，用于一次性、多因素设计的实验。在这个阶段之后，设计者剩下一组未使用的单元来实现从实验中学到的任何知识。通过此设置，我们回答了实验数量的关键设计问题，这意味着选择给定设计的重复次数。我们构建了一个贝叶斯框架，可以捕获整个单元集的预期累积增益。我们为所有具有七个或更少因子的两水平、完整和部分因子设计得出最佳重复次数的理论结果。我们进行模拟以验证理论结果，并使我们能够探索理论研究中未捕获的场景和分析技术。我们的总体结果表明，实验单元的最佳分配从可用单元总数的 1 到 $20\%$$20% 不等，这主要取决于实验环境和单元总数。我们得出的结论是，用我们的研究推荐的最佳重复次数进行试验可以导致累积改进，这比从业者随机选择重复次数时获得的预期累积改进大 80-95%。以及使我们能够探索理论研究中未涵盖的场景和分析技术。我们的总体结果表明，实验单元的最佳分配从可用单元总数的 1 到 $20\%$$20% 不等，这主要取决于实验环境和单元总数。我们得出的结论是，用我们的研究推荐的最佳重复次数进行试验可以导致累积改进，这比从业者随机选择重复次数时获得的预期累积改进大 80-95%。以及使我们能够探索理论研究中未涵盖的场景和分析技术。我们的总体结果表明，实验单元的最佳分配从可用单元总数的 1 到 $20\%$$20% 不等，这主要取决于实验环境和单元总数。我们得出的结论是，用我们的研究推荐的最佳重复次数进行试验可以导致累积改进，这比从业者随机选择重复次数时获得的预期累积改进大 80-95%。这主要受实验环境和单元总数的影响。我们得出的结论是，用我们的研究推荐的最佳重复次数进行试验可以导致累积改进，这比从业者随机选择重复次数时获得的预期累积改进大 80-95%。这主要受实验环境和单元总数的影响。我们得出的结论是，用我们的研究推荐的最佳重复次数进行试验可以导致累积改进，这比从业者随机选择重复次数时获得的预期累积改进大 80-95%。