Bid optimization for online advertising from single advertiser's perspective has been thoroughly investigated in both academic research and industrial practice. However, existing work typically assume competitors do not change their bids, i.e., the wining price is fixed, leading to poor performance of the derived solution. Although a few studies use multi-agent reinforcement learning to set up a cooperative game, they still suffer the following drawbacks: (1) They fail to avoid collusion solutions where all the advertisers involved in an auction collude to bid an extremely low price on purpose. (2) Previous works cannot well handle the underlying complex bidding environment, leading to poor model convergence. This problem could be amplified when handling multiple objectives of advertisers which are practical demands but not considered by previous work. In this paper, we propose a novel multi-objective cooperative bid optimization formulation called Multi-Agent Cooperative bidding Games (MACG). MACG sets up a carefully designed multi-objective optimization framework where different objectives of advertisers are incorporated. A global objective to maximize the overall profit of all advertisements is added in order to encourage better cooperation and also to protect self-bidding advertisers. To avoid collusion, we also introduce an extra platform revenue constraint. We analyze the optimal functional form of the bidding formula theoretically and design a policy network accordingly to generate auction-level bids. Then we design an efficient multi-agent evolutionary strategy for model optimization. Offline experiments and online A/B tests conducted on the Taobao platform indicate both single advertiser's objective and global profit have been significantly improved compared to state-of-art methods.

中文翻译：

---

用于电子商务赞助搜索中多目标优化的多代理合作竞标博弈

从单一广告主的角度来看在线广告的出价优化已经在学术研究和工业实践中进行了深入研究。然而，现有的工作通常假设竞争者不会改变他们的出价，即获胜价格是固定的，导致衍生解决方案的性能不佳。虽然有一些研究使用多智能体强化学习来建立合作博弈，但它们仍然存在以下缺点：（1）他们未能避免所有参与拍卖的广告商合谋故意投标极低价格的共谋解决方案. (2) 以前的工作不能很好地处理底层复杂的投标环境，导致模型收敛性差。当处理广告客户的多个目标时，这个问题可能会被放大，这些目标是实际需求但以前的工作没有考虑到。在本文中，我们提出了一种新的多目标合作投标优化公式，称为多代理合作投标游戏（MACG）。MACG 建立了一个精心设计的多目标优化框架，其中包含了广告客户的不同目标。添加了最大化所有广告的整体利润的全局目标，以鼓励更好的合作并保护自我投标的广告商。为了避免勾结，我们还引入了额外的平台收入限制。我们从理论上分析了投标公式的最优函数形式，并相应地设计了一个策略网络来生成拍卖级投标。然后我们为模型优化设计了一种有效的多智能体进化策略。在淘宝平台上进行的离线实验和在线 A/B 测试表明，与最先进的方法相比，单个广告客户的目标和全球利润都得到了显着提高。