Robust modelling of binary decisions in Laplacian Eigenmaps-based Echo State Networks

Abstract

This paper aims to present a framework for supervised binary classification of $n$-Boolean functions through Echo State Networks endowed with Laplacian Eigenmaps for dimensionality reduction. The proposed method is applied both to improve the classification performance when the learnt weights are quantised in view of a digital implementation and as a computational demonstration of the neural reuse theory when parallel outputs are allowed. Our analysis focuses on the effect of various forms of noise (i.e., normal noise, uniform noise and quantisation noise) when all the possible Boolean functions of $n$ input bits are learnt. External disturbances are applied both over the learnt weights and the input features so that we can analyse how resilient the whole architecture is when various forms of parametric noise is injected into the system. Results presented here show that dimensionality reduction allowed by the Laplacian Eigenmaps-based approach improves robustness to these different sources of noise, leading to reduced memory storage requirements while maintaining high classification performance. Our results are compared to those derived from other more common classification techniques in terms of learning performance and computational complexity, also considering a realistic dataset describing a decision making task in a wall-following navigation session with mobile robots.

Introduction

In artificial neural networks theory, different strategies can be followed for solving data classification which is at the basis of any decision-making problem. Even though in literature many efficient algorithms have been introduced, bio-inspired architectures cannot discard that decision making in living beings is not the result of a static processing, but involves a complex spatial–temporal dynamical process from which decisions emerge. Reservoir Computing (RC) aims to be a simple but powerful set of concepts and techniques for defining neural networks that can capture temporal dependencies by both allowing recurrent connections in the hidden layer and preserving internal state between inputs. Echo State Networks (ESNs) constitute a preeminent example of architecture belonging to the RC paradigm (Lukoševičius et al., 2012) and play an important role in plenty of applications, ranging from handwriting recognition (Bunke and Varga, 2007), to online learning control (Jordanou et al., 2019), to Autonomous Surface Vehicles (ASV) cooperation control (Liu et al., 2017). As regards nonlinear dynamical systems modelling and time series prediction, ESNs were successfully applied, for instance, to energy consumption estimation (Hu et al., 2020b) and to wind power generation forecasting adopting deep versions of ESNs (Hu et al., 2020a). Applications of ESNs, able to extract and predict the nonlinear characteristics of time series, and acting concurrently to ARMA models, able to estimate the linear part, were dealt with in Tian et al. (2020). ESNs were also successfully applied to chaotic time series prediction (Tian, 0000), using novel optimisation algorithms for the network parameters. Literature provides a whole and deep list of the ESN advantages, among which we can mention: the simplicity of the training method, real-time processing, global optimality and others. More specifically in relation to our contribution, in Tanaka et al. (2019) it is clearly stated, among the advantages of classical ESNs, the high dimensionality representation: it is possible to map inputs into a high-dimensional space. This property facilitates the separation of originally inseparable inputs in classification tasks and allows reading out spatiotemporal dependencies of inputs in prediction tasks. The main contribution of our paper exploits and enhances this aspects: it consists in designing a ESN-based structure able to provide a large number of decision making tasks in parallel, significantly reducing the network memory requirements, by modifying the weight data representation and, at the same time, without loosing in performances thanks to the intrinsic robustness of the proposed method to quantisation noise. This challenging objective has been obtained adopting the reservoir-based processing paradigm, formalised in terms of an Echo-State-Network, that has been enriched with a series of structural advantages. The first one is to include multiple readout maps on a unique reservoir. In this way in the paper it was demonstrated that a unique ESN reservoir layer is sufficient to allow the representation of all the binary functions with a fixed number of bits. This paradigm is inspired by the concept of neural reuse in neural psychology. In addition, a structural reduction method is applied to the readout maps to obtain a reduced output layer which efficiently compresses the reservoir information, maintaining the same network classification accuracy. Moreover, a weight quantisation strategy applied to the readout maps allows to maintain high performance, saving at the same time, computing resources in view of a hardware implementation. In fact, it is reasonable to imagine digital implementations of ESN-based systems which are requested to classify some input data. However, internal data representation arising from the structure of the network itself is commonly quite redundant and ill-posed (even numerically), due to the absence of general quantitative criteria for the selection of the network size. Moreover, it is often argued that, for many reasons, digital neural networks ought to be preferred for data classification (Kakkar, 0000). By introducing a proper dimensionality reduction algorithm, the obtained network presents lower complexity and can be implemented on digital devices through a quantisation process, leading to discrete integer weights. Our paper shows that if a nonlinear manifold reduction technique, namely Laplacian Eigenmaps (LEs) of the associated graph arising from the network state matrix, is applied, then quantisation requires a lower number of bits for representing all the trained weights than those needed for a double precision storage, allowing a strongly lower memory consumption. This is an interesting step for the overall system optimisation, which is mandatory in case of large neural networks and whenever there are some constraints on the design phase (for example in FPGA-based applications (Bonabi et al., 2014)). The use of manifold reduction has to be seen both as a mathematical tool for data processing and manipulation, conceived for lowering the number of degrees of freedom, and as a simpler representation of the original state space, devoted to the restraint of the so-called curse of dimensionality (Bellman, 2003). This paper exploits this method with particular attention to the problem of manipulating and/or processing $n$-Boolean functions and binary classification, widely discussed in literature (Cagnoni et al., 2005). The network realised in this paper has been trained over all the possible $n$-Boolean functions, supposing that each uniquely identified function corresponds to a specific action/decision/class. In Han and Xu (2018) LEs-based ESNs hyperparameters have been empirically selected; in our paper a constructive method for a data-driven hyperparameter determination targeted to a proper design of an LEs-based ESN has been introduced and the application has been focused to binary decision mappings. In Arena et al. (2019) the strategy has been applied to a liquid state-like classifier to solve two traditional classification tasks. This paper generalises the strategy to the mapping of any arbitrary $n$-Boolean function considered as the archetype of any decision making task.

From a biological perspective, these approaches refer to the so-called neural reuse theory (Anderson, 2010) to indicate those cognitive processes involved when living beings learn new abilities (Anderson, 2016). RC and neural reuse paradigms have already been used jointly to represent neural dynamics in simple brains and lead to the concurrent emergence of multiple behaviours, from motor skill learning to sequence/sub-sequence learning (Patané et al., 2018). Neural reuse theory suggests that complex behaviours arising from brain processing emerge from the activation of smaller computational units that act modularly. Thereby, starting from a very complex nonlinear processing of the signals coming from the environment, even single “read-out” units can be responsible for choosing a complex behaviour or decision. From a computational perspective, the brain seems to adopt a top-down strategy for reducing the complexity of a task. Since reduction transforms a large network into a smaller one preserving the outputs somehow, it is possible to state that reduction applies the same strategy as neural reuse: essentially, reduction implies a reorganisation of the network. This further outlines the generality of the proposed approach which can be successfully extended to other neural network topologies.

The paper is organised as follows: Section 2 reports the basic information about ESNs and LEs algorithm for handling the linear separability problem over a $n$-dimensional binary hypercube. Section 3 reports all the simulation results, the details of the strategy for tuning the LEs algorithm hyperparameters and the noise rejection capability of the overall system. In Section 4 an application of the method is presented referring to a typical example of wall following task in mobile robotics involving a decision making strategy. Some remarks about the proposed method, the design choices and the obtained results are reported in Section 5. Eventually, Section 6 concludes the paper.