Surprise-based learning allows agents to rapidly adapt to nonstationary stochastic environments characterized by sudden changes. We show that exact Bayesian inference in a hierarchical model gives rise to a surprise-modulated trade-off between forgetting old observations and integrating them with the new ones. The modulation depends on a probability ratio, which we call the Bayes factor surprise, that tests the prior belief against the current belief. We demonstrate that in several existing approximate algorithms, the Bayes Factor Surprise modulates the rate of adaptation to new observations. We derive three novel surprise-based algorithms, one in the family of particle filters, one in the family of variational learning, and one in the family of message passing, that have constant scaling in observation sequence length and particularly simple update dynamics for any distribution in the exponential family. Empirical results show that these surprise-based algorithms estimate parameters better than alternative approximate approaches and reach levels of performance comparable to computationally more expensive algorithms. The Bayes Factor Surprise is related to but different from the Shannon Surprise. In two hypothetical experiments, we make testable predictions for physiological indicators that dissociate the Bayes factor surprise from the Shannon Surprise. The theoretical insight of casting various approaches as surprise-based learning, as well as the proposed online algorithms, may be applied to the analysis of animal and human behavior and to reinforcement learning in nonstationary environments.

中文翻译：

---

在具有贝叶斯因子惊喜的波动环境中学习

基于惊喜的学习使代理能够快速适应以突然变化为特征的非平稳随机环境。我们表明，分层模型中的精确贝叶斯推理会在忘记旧观察和将它们与新观察结合之间产生意外调制的权衡。调制取决于概率比，我们称之为贝叶斯因子意外，它测试先验信念与当前信念。我们证明了在几个现有的近似算法中，贝叶斯因子惊奇调节适应新观察的速率。我们推导出三种新颖的基于惊喜的算法，一种属于粒子过滤器系列，一种属于变分学习系列，一种属于消息传递系列，具有恒定的观测序列长度缩放，并且对于指数族中的任何分布具有特别简单的更新动态。实证结果表明，这些基于惊喜的算法比替代近似方法更好地估计参数，并达到与计算成本更高的算法相当的性能水平。Bayes Factor Surprise 与 Shannon Surprise 相关但又不同。在两个假设实验中，我们对生理指标做出了可测试的预测，这些指标将贝叶斯因子惊喜与香农惊喜分开。将各种方法作为基于惊喜的学习的理论见解，以及所提出的在线算法，可以应用于动物和人类行为的分析以及非平稳环境中的强化学习。