Effective exploration is key to a successful search process. The recently proposed negatively correlated search (NCS) tries to achieve this by coordinated parallel exploration, where a set of search processes are driven to be negatively correlated so that different promising areas of the search space can be visited simultaneously. Despite successful applications of NCS, the negatively correlated search behaviors were mostly devised by intuition, while deeper (e.g., mathematical) understanding is missing. In this paper, a more principled NCS, namely NCNES, is presented, showing that the parallel exploration is equivalent to a process of seeking probabilistic models that both lead to solutions of high quality and are distant from previous obtained probabilistic models. Reinforcement learning, for which exploration is of particular importance, are considered for empirical assessment. The proposed NCNES is applied to directly train a deep convolution network with 1.7 million connection weights for playing Atari games. Empirical results show that the significant advantages of NCNES, especially on games with uncertain and delayed rewards, can be highly owed to the effective parallel exploration ability.

有效的探索是成功搜索过程的关键。最近提出的负相关搜索 (NCS) 试图通过协调并行探索来实现这一点，其中一组搜索过程被驱动为负相关，以便可以同时访问搜索空间的不同有希望的区域。尽管 NCS 的应用取得了成功，但负相关的搜索行为大多是由直觉设计的，而缺少更深入的（例如，数学）理解。在本文中，提出了一个更有原则的 NCS，即 NCNES，表明并行探索等效于寻找概率模型的过程，该过程既可以产生高质量的解决方案，又与之前获得的概率模型相距甚远。强化学习，其中探索尤为重要，被考虑用于经验评估。提出的 NCNES 用于直接训练一个具有 170 万个连接权重的深度卷积网络，用于玩 Atari 游戏。实证结果表明，NCNES 的显着优势，尤其是在奖励不确定和延迟的游戏中，很大程度上归功于有效的并行探索能力。