Short term memory properties of sensory neural architectures

Abstract

A functional role of the cerebral cortex is to form and hold representations of the sensory world for behavioral purposes. This is achieved by a sheet of neurons, organized in modules called cortical columns, that receives inputs in a peculiar manner, with only a few neurons driven by sensory inputs through thalamic projections, and a vast majority of neurons receiving mainly cortical inputs. How should cortical modules be organized, with respect to sensory inputs, in order for the cortex to efficiently hold sensory representations in memory? To address this question we investigate the memory performance of trees of recurrent networks (TRN) that are composed of recurrent networks, modeling cortical columns, connected with each others through a tree-shaped feed-forward backbone of connections, with sensory stimuli injected at the root of the tree. On these sensory architectures two types of short-term memory (STM) mechanisms can be implemented, STM via transient dynamics on the feed-forward tree, and STM via reverberating activity on the recurrent connectivity inside modules. We derive equations describing the dynamics of such networks, which allow us to thoroughly explore the space of possible architectures and quantify their memory performance. By varying the divergence ratio of the tree, we show that serial architectures, where sensory inputs are successively processed in different modules, are better suited to implement STM via transient dynamics, while parallel architectures, where sensory inputs are simultaneously processed by all modules, are better suited to implement STM via reverberating dynamics.

Appendix

1.1 A.1 Steady-state mean activity profiles for f ≪ 1

We consider a module with mean activity μ = O(f) receiving feed-forward inputs from a module whose mean activity is $g=\frac {f}{s}$. From (15), the fixed point equations relating μ and g is

$$ \mu=gH\left( \frac{\theta-1}{\lambda\sqrt{\alpha\mu}}\right)+(1-g)H\left( \frac{\theta}{\lambda\sqrt{\alpha\mu}}\right) $$

(24)

Using the estimate $H(x) \underset {x \gg 1}{\simeq }\frac {1}{x\sqrt {2\pi }}e^{\frac {-x^{2}}{2}} \simeq e^{\frac {-x^{2}}{2}}$, and $\frac {f|\log f|}{g|\log g|}\simeq \frac {f}{g}$ the fixed point equation can be rewritten

$$ \mu \approx g\left( 1+g^{sx-1}-g^{sx(1-\theta^{-1})^{2}}\right) $$

(25)

with $x=\frac {\theta ^{2}}{2\alpha f|\log f|}$. Comparing the two last terms of (25) allows to understand whether μ increases or decreases compared to g and thus to describe the two regimes of mean activity profiles of Section 4.2. Moreover, for the regime of increasing activity along the path, the transition from μ = O(f) to μ = O(1) arises for s = 1/x, i.e. g = xf. By differentiating (24) as shown in Supplementary materials, this allows to give expressions for the depth L_c at which the transition occurs. For $\theta <\frac {1}{2}$

$$ L_{c} = 2\sqrt{\frac{\pi}{x f^{2(x-1)} |\log f |}} $$

(26)

and for $\theta >\frac {1}{2}$ and α > (2𝜃^− 1 − 𝜃^− 2)α_c the critical depth scale as

$$ 1/L_{c}\propto\frac{1}{(\theta^{-1}-1)\sqrt{x|\log f|}}f^{x(1-\theta^{-1})^{2}}-\frac{1}{\sqrt{x|\log f|}}f^{x-1} $$

(27)

1.2 A.2 Dynamical equations for memory retrieval in a path

In order to describe the retrieval of pattern $\boldsymbol {\xi }^{l_{0},1}$ in a module l₀ receiving feed-forward inputs from module l₀ − 1 (see Section 4.4), we have used the following dynamical equations, whose derivation is detailed in Supplementary materials. $m^{l_{0}}$ (resp. $m^{l_{0}-1}$) is the overlap between the activity in module l₀ (resp. l₀ − 1) and $\boldsymbol {\xi }^{l_{0},1}$.

$$ \begin{array}{@{}rcl@{}} m^{l_{0}}(t+1) &=& m^{l_{0}-1}(t)\{ (1-f)[ I_{t}(1,1) - I_{t}(1,0) ] \\ & & + f [ I_{t}(0,1) - I_{t}(0,0)]\} \\ & & + \mu^{l_{0}-1}(t)\{I_{t}(1,1) - I_{t}(1,0) \\ & & - [I_{t}(0,1) - I_{t}(0,0)]\} \\ & & + I_{t}(1,0) - I_{t}(0,0) \end{array} $$

(28)

and

$$ \begin{array}{@{}rcl@{}} \mu^{l_{0}}(t+1) &=& m^{l_{0}-1}(t) f(1-f)\{I_{t}(1,1) - I_{t}(1,0) \\ & & - [I_{t}(0,1) - I_{t}(0,0)]\} \\ & & + \mu^{l_{0}-1}(t)\{ f[ I_{t}(1,1) - I_{t}(1,0) ] \\ & & + (1-f) [ I_{t}(0,1) - I_{t}(0,0)]\} \\ & & + f I_{t}(1,0 ) + (1-f) I_{t}(0,0) \end{array} $$

(29)

with

$$ I_{t}(a,b) = H\left( \frac{\theta-(a-f)m^{l_{0}}(t)-b}{\sqrt{\alpha\mu^{l_{0}}(t)}}\right) $$

(30)

1.3 A.3 Impact of feed-forward noise on persistent activity

The random sequence of inputs is a form of noise that reduces the capacity for retrieval states. To evaluate the existence of a retrieval state $\boldsymbol {\xi }^{\mu _{0},l_{0}}$ under these conditions, we examine the stability of the module l assuming that it receives random feed-forward inputs from the module l − 1 with a steady state mean activity $\mu _{eq}^{l-1}$. We re-write the equations for retrieval in module l in terms of the order parameters ${f_{0}^{l}}$ and ${f_{1}^{l}}$, which measure the fraction of background neurons (neurons i such that $\xi _{i}^{\mu _{0},l_{0}}=0$) that are active and the fraction of foreground (neurons i such that $\xi _{i}^{\mu _{0},l_{0}}=1$) neurons that are active. These order parameters are related to m^l and μ^l by $m^{l}={f_{1}^{l}} -{f_{0}^{l}}$ and $\mu ^{l}=f*{f_{1}^{l}} +(1-f)*{f_{0}^{l}}$.

$$ \begin{array}{@{}rcl@{}} {f_{1}^{l}}(t+1)&=&\mu_{eq}^{l-1}H\left( \frac{\theta- m^{l}(t)-1}{\sqrt{\alpha\mu^{l}(t)}}\right)+ \\ &&(1-\mu_{eq}^{l-1})H\left( \frac{\theta- m^{l}(t)}{\sqrt{\alpha\mu^{l}(t)}}\right), t\geq1 \\ {f_{0}^{l}}(t+1)&=&\mu_{eq}^{l-1}H\left( \frac{\theta-1}{\sqrt{\alpha\mu^{l}(t)}}\right)+ \\ &&(1-\mu_{eq}^{l-1})H\left( \frac{\theta}{\sqrt{\alpha\mu^{l}(t)}}\right), t\ge1 \end{array} $$

(31)

If we assume that a pattern has been retrieved, i.e. m^l(t = 1) ≃ 1 and $\mu ^{l}(t=1) = f + \mu _{eq}^{l}$. This pattern remains stable if ${f_{1}^{l}}$ remains of order 1 and ${f_{0}^{l}}$ remains of order f, which comes down to have, for $\mu _{eq}^{l-1} = O(f) \ll 1$,

$$ \begin{array}{@{}rcl@{}} H\left( \frac{\theta-1}{\sqrt{\alpha(f + \mu_{eq}^{l})}}\right) = 1-f^{\frac{x(1-\theta^{-1})^{2}}{1+\frac{\mu_{eq}^{l}}{f}}} &=& O(1) \\ H\left( \frac{\theta}{\sqrt{\alpha(f + \mu_{eq}^{l})}}\right) = f^{\frac{x}{1+\frac{\mu_{eq}^{l}}{f}}-1} &\ll& 1 \\ \end{array} $$

(32)

For f ≪ 1 this is satisfied if $x>1+\frac {\mu _{eq}^{l}}{f}$, hence the condition (19) on α.