ABSTRACT

Hierarchical temporal memory is an emerging machine learning technology that aims to model the structural and algorithmic properties of the neocortex. It is particularly suitable for learning and predicting sequential data. However, when dealing with long time series or complex sequences, the accuracy is relatively lower than desired. In this paper, a novel hierarchical temporal memory based on recurrent learning unit is proposed, where a feedback mechanism is involved into the model. The original cell is extended with a recurrent unit to capture long temporal dependencies of synaptic connections between neurons. The temporal pooler algorithm is then modified to adapt to the recurrent learning unit, and the supervised gradient information is combined with the Hebbian synaptogenesis learning rule in speeding up the training. The prototype of the proposed hierarchical temporal memory is implemented and extensive experiments are carried out on two public datasets under various settings. Experimental results show that the proposed model obtains an accuracy increase by up to 32% and a perplexity drop by up to 14% on sequence prediction and text generation tasks, respectively, which indicates the hierarchical temporal memory with recurrent feedback outperforms the original model on sequence learning.

摘要

分层时间记忆是一种新兴的机器学习技术，旨在模拟新皮质的结构和算法特性。它特别适合学习和预测序列数据。然而，当处理较长的时间序列或复杂的序列时，精度相对低于预期。本文提出了一种基于循环学习单元的新型分层时间记忆，其中模型中涉及反馈机制。原始细胞通过循环单元进行扩展，以捕获神经元之间突触连接的长期依赖性。然后修改时间池算法以适应循环学习单元，并将监督梯度信息与赫布突触发生学习规则相结合以加速训练。所提出的分层时间记忆的原型已实现，并在不同设置下的两个公共数据集上进行了广泛的实验。实验结果表明，该模型在序列预测和文本生成任务上分别获得了高达 32% 的准确率提升和高达 14% 的困惑度下降，这表明具有循环反馈的分层时间记忆在序列上优于原始模型学习。