In this paper, we draw attention to a problem that is often overlooked or ignored by companies practicing hypothesis testing (A/B testing) in online environments. We show that conducting experiments on limited inventory that is shared between variants in the experiment can lead to high false positive rates since the core assumption of independence between the groups is violated. We provide a detailed analysis of the problem in a simplified setting whose parameters are informed by realistic scenarios. The setting we consider is a $2$-dimensional random walk in a semi-infinite strip. It is rich enough to take a finite inventory into account, but is at the same time simple enough to allow for a closed form of the false-positive probability. We prove that high false-positive rates can occur, and develop tools that are suitable to help design adequate tests in follow-up work. Our results also show that high false-negative rates may occur. The proofs rely on a functional limit theorem for the $2$-dimensional random walk in a semi-infinite strip.

中文翻译：

---

电子商务中有限库存假设检验的基本问题

在本文中，我们提请注意在线环境中进行假设测试（A/B 测试）的公司经常忽视或忽略的问题。我们表明，在实验中的变体之间共享的有限库存上进行实验会导致高误报率，因为违反了组之间独立性的核心假设。我们在简化的设置中提供了对问题的详细分析，其参数由现实场景提供信息。我们考虑的设置是在半无限长条中的 $2$ 维随机游走。它足够丰富以考虑有限库存，但同时又足够简单以允许假阳性概率的封闭形式。我们证明可能会出现高假阳性率，并开发适合帮助在后续工作中设计适当测试的工具。我们的结果还表明，可能会出现高假阴性率。证明依赖于半无限长条中 $2$ 维随机游走的功能极限定理。