Reinforcement learning is a paradigm that can account for how organisms learn to adapt their behavior in complex environments with sparse rewards. To partition an environment into discrete states, implementations in spiking neuronal networks typically rely on input architectures involving place cells or receptive fields specified ad hoc by the researcher. This is problematic as a model for how an organism can learn appropriate behavioral sequences in unknown environments, as it fails to account for the unsupervised and self-organized nature of the required representations. Additionally, this approach presupposes knowledge on the part of the researcher on how the environment should be partitioned and represented and scales poorly with the size or complexity of the environment. To address these issues and gain insights into how the brain generates its own task-relevant mappings, we propose a learning architecture that combines unsupervised learning on the input projections with biologically motivated clustered connectivity within the representation layer. This combination allows input features to be mapped to clusters; thus the network self-organizes to produce clearly distinguishable activity patterns that can serve as the basis for reinforcement learning on the output projections. On the basis of the MNIST and Mountain Car tasks, we show that our proposed model performs better than either a comparable unclustered network or a clustered network with static input projections. We conclude that the combination of unsupervised learning and clustered connectivity provides a generic representational substrate suitable for further computation.

强化学习是一种范例，可以解释有机体如何在稀疏的奖励中学习如何在复杂的环境中适应其行为。为了将环境划分为离散状态，尖峰神经网络中的实现通常依赖于涉及到所指定的位置细胞或感受野的输入体系结构特设由研究员。这对于有机体如何在未知环境中学习适当的行为序列的模型是有问题的，因为它无法解决所需表示形式的无监督和自组织性质。此外，这种方法以研究人员方面的知识为先决条件，即应该如何划分和表示环境，并且随着环境的大小或复杂性而扩展性很差。为了解决这些问题并深入了解大脑如何生成自己的与任务相关的映射，我们提出了一种学习体系结构，该体系结构将对输入投影的无监督学习与具有生物学动机的表示层内的群集连接性相结合。这种组合允许将输入要素映射到群集。因此，网络可以自我组织以产生清晰可辨的活动模式，可以作为加强对输出预测的学习的基础。根据MNIST和Mountain Car的任务，我们证明了我们提出的模型比具有可比性的非聚类网络或具有静态输入投影的聚类网络性能更好。我们得出的结论是，无监督学习和集群连接性的组合提供了适用于进一步计算的通用代表性基础。