1932
0
  1. Home
  2. A-Z Publications
  3. Annual Review of Neuroscience
  4. Volume 45, 2022
  5. Article

Annual Review of Neuroscience

Volume 45, 2022

Neuromodulation and Neurophysiology on the Timescale of Learning and Decision-Making

Abstract

Nervous systems evolved to effectively navigate the dynamics of the environment to achieve their goals. One framework used to study this fundamental problem arose in the study of learning and decision-making. In this framework, the demands of effective behavior require slow dynamics—on the scale of seconds to minutes—of networks of neurons. Here, we review the phenomena and mechanisms involved. Using vignettes from a few species and areas of the nervous system, we view neuromodulators as key substrates for temporal scaling of neuronal dynamics.

Keyword(s): dopamine, norepinephrine, reinforcement learning, serotonin
Loading

Article metrics loading...

/content/journals/10.1146/annurev-neuro-092021-125059
2022-07-08
2024-05-01
Loading full text...

Full text loading...

/deliver/fulltext/neuro/45/1/annurev-neuro-092021-125059.html?itemId=/content/journals/10.1146/annurev-neuro-092021-125059&mimeType=html&fmt=ahah

Literature Cited

  1. Aksay E, Baker R, Seung HS, Tank DW. 2003. Correlated discharge among cell pairs within the oculomotor horizontal velocity-to-position integrator. J. Neurosci. 23:3410852–58
     [Google Scholar]
  2. Andrade R. 1991. Cell excitation enhances muscarinic cholinergic responses in rat association cortex. Brain Res 548:1–281–93
     [Google Scholar]
  3. Aponte Y, Atasoy D, Sternson SM. 2011. AGRP neurons are sufficient to orchestrate feeding behavior rapidly and without training. Nat. Neurosci. 14:3351–55
     [Google Scholar]
  4. Armstrong-James M, Fox K, Das-Gupta A 1992. Flow of excitation within rat barrel cortex on striking a single vibrissa. J. Neurophysiol. 68:41345–58
     [Google Scholar]
  5. Arnold DB, Robinson DA. 1997. The oculomotor integrator: testing of a neural network model. Exp. Brain Res. 113:157–74
     [Google Scholar]
  6. Aston-Jones G, Cohen JD. 2005a. An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance. Annu. Rev. Neurosci. 28:403–50
     [Google Scholar]
  7. Aston-Jones G, Cohen JD. 2005b. Adaptive gain and the role of the locus coeruleus-norepinephrine system in optimal performance. J. Comp. Neurol. 493:199–110
     [Google Scholar]
  8. Aston-Jones G, Rajkowski J, Kubiak P. 1997. Conditioned responses of monkey locus coeruleus neurons anticipate acquisition of discriminative behavior in a vigilance task. Neuroscience 80:3697–715
     [Google Scholar]
  9. Athilingam JC, Ben-Shalom R, Keeshen CM, Sohal VS, Bender KJ. 2017. Serotonin enhances excitability and gamma frequency temporal integration in mouse prefrontal fast-spiking interneurons. eLife 6:e31991
     [Google Scholar]
  10. Avery MC, Krichmar JL. 2017. Neuromodulatory systems and their interactions: a review of models, theories, and experiments. Front. Neural Circuits 11:108
     [Google Scholar]
  11. Azmitia EC. 2001. Modern views on an ancient chemical: serotonin effects on cell proliferation, maturation, and apoptosis. Brain Res. Bull. 56:5413–24
     [Google Scholar]
  12. Azmitia EC. 2007. Serotonin and brain: evolution, neuroplasticity, and homeostasis. Int. Rev. Neurobiol. 77:31–56
     [Google Scholar]
  13. Azmitia EC, Segal M. 1978. An autoradiographic analysis of the differential ascending projections of the dorsal and median raphe nuclei in the rat. J. Comp. Neurol. 179:3641–67
     [Google Scholar]
  14. Barack DL, Krakauer JW. 2021. Two views on the cognitive brain. Nat. Rev. Neurosci. 22:6359–71
     [Google Scholar]
  15. Bari A, Theobald DE, Caprioli D, Mar AC, Aidoo-Micah A et al. 2010. Serotonin modulates sensitivity to reward and negative feedback in a probabilistic reversal learning task in rats. Neuropsychopharmacology 35:61290–301
     [Google Scholar]
  16. Bari BA, Grossman CD, Lubin EE, Rajagopalan AE, Cressy JI, Cohen JY. 2019. Stable representations of decision variables for flexible behavior. Neuron 103:5922–33.e7
     [Google Scholar]
  17. Bartolo R, Averbeck BB. 2020. Prefrontal cortex predicts state switches during reversal learning. Neuron 106:61044–54.e4
     [Google Scholar]
  18. Bayer HM, Glimcher PW. 2005. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47:1129–41
     [Google Scholar]
  19. Beani L, Bianchi C, Giacomelli A, Tamberi F. 1978. Noradrenaline inhibition of acetylcholine release from guinea-pig brain. Eur. J. Pharmacol. 48:2179–93
     [Google Scholar]
  20. Behrens TEJ, Woolrich MW, Walton ME, Rushworth MFS. 2007. Learning the value of information in an uncertain world. Nat. Neurosci. 10:91214–21
     [Google Scholar]
  21. Bernacchia A, Seo H, Lee D, Wang X-J. 2011. A reservoir of time constants for memory traces in cortical neurons. Nat. Neurosci. 14:3366–72
     [Google Scholar]
  22. Bertsekas DP, Tsitsiklis JN. 1996. Neuro-Dynamic Programming Belmont, MA: Athena Scientific
  23. Boulougouris V, Robbins TW. 2010. Enhancement of spatial reversal learning by 5-HT2C receptor antagonism is neuroanatomically specific. J. Neurosci. 30:3930–38
     [Google Scholar]
  24. Bouret S, Sara SJ. 2004. Reward expectation, orientation of attention and locus coeruleus-medial frontal cortex interplay during learning. Eur. J. Neurosci. 20:3791–802
     [Google Scholar]
  25. Bouret S, Sara SJ. 2005. Network reset: a simplified overarching theory of locus coeruleus noradrenaline function. Trends Neurosci 28:11574–82
     [Google Scholar]
  26. Branco T, Tozer A, Magnus CJ, Sugino K, Tanaka S et al. 2016. Near-perfect synaptic integration by Nav1.7 in hypothalamic neurons regulates body weight. Cell 165:71749–61
     [Google Scholar]
  27. Briand LA, Gritton H, Howe WM, Young DA, Sarter M. 2007. Modulators in concert for cognition: modulator interactions in the prefrontal cortex. Prog. Neurobiol. 83:269–91
     [Google Scholar]
  28. Brigman JL, Mathur P, Harvey-White J, Izquierdo A, Saksida LM et al. 2010. Pharmacological or genetic inactivation of the serotonin transporter improves reversal learning in mice. Cereb. Cortex 20:81955–63
     [Google Scholar]
  29. Brody CD, Hernández A, Zainos A, Romo R. 2003. Timing and neural encoding of somatosensory parametric working memory in macaque prefrontal cortex. Cereb. Cortex 13:111196–207
     [Google Scholar]
  30. Brunel N, Wang XJ. 2001. Effects of neuromodulation in a cortical network model of object working memory dominated by recurrent inhibition. J. Comput. Neurosci. 11:163–85
     [Google Scholar]
  31. Cai X, Kim S, Lee D 2011. Heterogeneous coding of temporally discounted values in the dorsal and ventral striatum during intertemporal choice. Neuron 69:1170–82
     [Google Scholar]
  32. Cannon CM, Palmiter RD. 2003. Reward without dopamine. J. Neurosci. 23:3410827–31
     [Google Scholar]
  33. Cannon SC, Robinson DA, Shamma S. 1983. A proposed neural network for the integrator of the oculomotor system. Biol. Cybern. 49:2127–36
     [Google Scholar]
  34. Cedernaes J, Huang W, Ramsey KM, Waldeck N, Cheng L et al. 2019. Transcriptional basis for rhythmic control of hunger and metabolism within the AgRP neuron. Cell Metab 29:1078–91
     [Google Scholar]
  35. Charnov EL. 1976. Optimal foraging, the marginal value theorem. Theor. Popul. Biol. 9:129–36
     [Google Scholar]
  36. Chen G, Greengard P, Yan Z 2004. Potentiation of NMDA receptor currents by dopamine D1 receptors in prefrontal cortex. PNAS 101:82596–600
     [Google Scholar]
  37. Clarke HF, Dalley JW, Crofts HS, Robbins TW, Roberts AC. 2004. Cognitive inflexibility after prefrontal serotonin depletion. Science 304:5672878–80
     [Google Scholar]
  38. Clarke HF, Walker SC, Dalley JW, Robbins TW, Roberts AC. 2007. Cognitive inflexibility after prefrontal serotonin depletion is behaviorally and neurochemically specific. Cereb. Cortex 17:118–27
     [Google Scholar]
  39. Cohen JY, Amoroso MW, Uchida N. 2015. Serotonergic neurons signal reward and punishment on multiple timescales. eLife 4:e06346
     [Google Scholar]
  40. Cohen JY, Haesler S, Vong L, Lowell BB, Uchida N. 2012. Neuron-type-specific signals for reward and punishment in the ventral tegmental area. Nature 482:738385–88
     [Google Scholar]
  41. Colwell CS. 2011. Linking neural activity and molecular oscillations in the Scn. Nat. Rev. Neurosci. 12:10553–69
     [Google Scholar]
  42. Cowley MA, Smart JL, Rubinstein M, Cerdán MG, Diano S et al. 2001. Leptin activates anorexigenic POMC neurons through a neural network in the arcuate nucleus. Nature 411:6836480–84
     [Google Scholar]
  43. Curtis CE, Lee D 2010. Beyond working memory: the role of persistent activity in decision making. Trends Cogn. Sci. 14:5216–22
     [Google Scholar]
  44. Daw ND, Kakade S, Dayan P. 2002. Opponent interactions between serotonin and dopamine. Neural Netw 15:4–6603–16
     [Google Scholar]
  45. Daw ND, Niv Y, Dayan P. 2005. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8:121704–11
     [Google Scholar]
  46. Daw ND, O'Doherty JP, Dayan P, Seymour B, Dolan RJ 2006. Cortical substrates for exploratory decisions in humans. Nature 441:7095876–79
     [Google Scholar]
  47. Dayan P, Abbott LF. 2001. Theoretical Neuroscience: Computational and Mathematical Modeling of Neural Systems Cambridge, MA: MIT Press
  48. Dayan P, Huys QJM. 2009. Serotonin in affective control. Annu. Rev. Neurosci. 32:95–126
     [Google Scholar]
  49. Dayan P, Yu AJ. 2006. Phasic norepinephrine: a neural interrupt signal for unexpected events. Netw. Comput. Neural Syst. 17:4335–50
     [Google Scholar]
  50. Decker AL, Duncan K. 2020. Acetylcholine and the complex interdependence of memory and attention. Curr. Opin. Behav. Sci. 32:21–28
     [Google Scholar]
  51. Dembrow NC, Chitwood RA, Johnston D. 2010. Projection-specific neuromodulation of medial prefrontal cortex neurons. J. Neurosci. 30:5016922–37
     [Google Scholar]
  52. Descarries L, Mechawar N. 2000. Ultrastructural evidence for diffuse transmission by monoamine and acetylcholine neurons of the central nervous system. Prog. Brain Res. 125:27–47
     [Google Scholar]
  53. Destexhe A, Rudolph M, Paré D. 2003. The high-conductance state of neocortical neurons in vivo. Nat. Rev. Neurosci. 4:9739–51
     [Google Scholar]
  54. Dickinson A. 1985. Actions and habits: the development of behavioural autonomy. Philos. Trans. R. Soc. B 308:67–78
     [Google Scholar]
  55. Diering GH, Huganir RL. 2018. The AMPA receptor code of synaptic plasticity. Neuron 100:2314–29
     [Google Scholar]
  56. Doya K. 2008. Modulators of decision making. Nat. Neurosci. 11:4410–16
     [Google Scholar]
  57. Doya K, Samejima K, Katagiri K-I, Kawato M. 2002. Multiple model-based reinforcement learning. Neural Comput 14:61347–69
     [Google Scholar]
  58. Dudman JT, Krakauer JW. 2016. The basal ganglia: from motor commands to the control of vigor. Curr. Opin. Neurobiol. 37:158–66
     [Google Scholar]
  59. Durstewitz D, Seamans JK, Sejnowski TJ. 2000. Neurocomputational models of working memory. Nat. Neurosci. 3:1184–91
     [Google Scholar]
  60. Edelman GM, Gally JA. 2001. Degeneracy and complexity in biological systems. PNAS 98:2413763–68
     [Google Scholar]
  61. Egorov AV, Hamam BN, Fransén E, Hasselmo ME, Alonso AA. 2002. Graded persistent activity in entorhinal cortex neurons. Nature 420:6912173–78
     [Google Scholar]
  62. Evarts EV, Tanji J. 1976. Reflex and intended responses in motor cortex pyramidal tract neurons of monkey. J. Neurophysiol. 39:51069–80
     [Google Scholar]
  63. Fischer B, Boch R. 1983. Saccadic eye movements after extremely short reaction times in the monkey. Brain Res 260:121–26
     [Google Scholar]
  64. Foote SL, Morrison JH. 1987. Extrathalamic modulation of cortical function. Annu. Rev. Neurosci. 10:67–95
     [Google Scholar]
  65. Fornal CA, Metzler CW, Gallegos RA, Veasey SC, McCreary AC, Jacobs BL. 1996. WAY-100635, a potent and selective 5-hydroxytryptamine1A antagonist, increases serotonergic neuronal activity in behaving cats: comparison with (S)-WAY-100135. J. Pharmacol. Exp. Ther. 278:2752–62
     [Google Scholar]
  66. Fransén E, Alonso AA, Hasselmo ME. 2002. Simulations of the role of the muscarinic-activated calcium-sensitive nonspecific cation current INCM in entorhinal neuronal activity during delayed matching tasks. J. Neurosci. 22:31081–97
     [Google Scholar]
  67. Fraser DD, MacVicar BA. 1996. Cholinergic-dependent plateau potential in hippocampal CA1 pyramidal neurons. J. Neurosci. 16:134113–28
     [Google Scholar]
  68. Funahashi S, Bruce CJ, Goldman-Rakic PS. 1989. Mnemonic coding of visual space in the monkey's dorsolateral prefrontal cortex. J. Neurophysiol. 61:2331–49
     [Google Scholar]
  69. Furman DJ, Zhang Z, Chatham CH, Good M, Badre D et al. 2021. Augmenting frontal dopamine tone enhances maintenance over gating processes in working memory. J. Cogn. Neurosci. 33:91753–65
     [Google Scholar]
  70. Fuster JM. 1973. Unit activity in prefrontal cortex during delayed-response performance: neuronal correlates of transient memory. J. Neurophysiol. 36:161–78
     [Google Scholar]
  71. Gallistel CR, Gibbon J. 2002. The Symbolic Foundations of Conditioned Behavior New York: Psychol. Press
  72. Gil Z, Connors BW, Amitai Y. 1997. Differential regulation of neocortical synapses by neuromodulators and activity. Neuron 19:3679–86
     [Google Scholar]
  73. Goaillard J-M, Marder E. 2021. Ion channel degeneracy, variability, and covariation in neuron and circuit resilience. Annu. Rev. Neurosci. 44:335–57
     [Google Scholar]
  74. Grossman CD, Bari BA, Cohen JY. 2022. Serotonin neurons modulate learning rate through uncertainty. Curr. Biol. 32:3586–99.e7
     [Google Scholar]
  75. Guitart-Masip M, Beierholm UR, Dolan R, Duzel E, Dayan P. 2011. Vigor in the face of fluctuating rates of reward: an experimental examination. J. Cogn. Neurosci. 23:123933–38
     [Google Scholar]
  76. Guo ZV, Inagaki HK, Daie K, Druckmann S, Gerfen CR, Svoboda K. 2017. Maintenance of persistent activity in a frontal thalamocortical loop. Nature 545:7653181–86
     [Google Scholar]
  77. Haam J, Yakel JL. 2017. Cholinergic modulation of the hippocampal region and memory function. J. Neurochem. 142:Suppl. 2111–21
     [Google Scholar]
  78. Haith AM, Krakauer JW. 2013. Model-based and model-free mechanisms of human motor learning. Adv. Exp. Med. Biol. 782:1–21
     [Google Scholar]
  79. Haj-Dahmane S, Andrade R. 1998. Ionic mechanism of the slow afterdepolarization induced by muscarinic receptor activation in rat prefrontal cortex. J. Neurophysiol. 80:31197–210
     [Google Scholar]
  80. Hangya B, Ranade SP, Lorenc M, Kepecs A. 2015. Central cholinergic neurons are rapidly recruited by reinforcement feedback. Cell 162:51155–68
     [Google Scholar]
  81. Hasselmo ME. 2006. The role of acetylcholine in learning and memory. Curr. Opin. Neurobiol. 16:6710–15
     [Google Scholar]
  82. Hasselmo ME, Wyble BP, Wallenstein GV. 1996. Encoding and retrieval of episodic memories: role of cholinergic and GABAergic modulation in the hippocampus. Hippocampus 6:6693–708
     [Google Scholar]
  83. Hayashi K, Nakao K, Nakamura K. 2015. Appetitive and aversive information coding in the primate dorsal raphé nucleus. J. Neurosci. 35:156195–208
     [Google Scholar]
  84. Hayden BY, Pearson JM, Platt ML. 2011. Neuronal basis of sequential foraging decisions in a patchy environment. Nat. Neurosci. 14:7933–39
     [Google Scholar]
  85. Hervé-Minvielle A, Sara SJ. 1995. Rapid habituation of auditory responses of locus coeruleus cells in anaesthetized and awake rats. Neuroreport 6:101363–68
     [Google Scholar]
  86. Honey CJ, Newman EL, Schapiro AC. 2017. Switching between internal and external modes: a multiscale learning principle. Netw. Neurosci. 1:4339–56
     [Google Scholar]
  87. Houston A, MacNamara J. 1999. Models of Adaptive Behaviour Cambridge, UK: Cambridge Univ. Press
  88. Howard MW, Rizzuto DS, Caplan JB, Madsen JR, Lisman J et al. 2003. Gamma oscillations correlate with working memory load in humans. Cereb. Cortex 13:121369–74
     [Google Scholar]
  89. Hsieh CY, Cruikshank SJ, Metherate R. 2000. Differential modulation of auditory thalamocortical and intracortical synaptic transmission by cholinergic agonist. Brain Res 880:1–251–64
     [Google Scholar]
  90. Hull CL. 1943. Principles of Behavior: An Introduction to Behavior Theory New York: Appleton-Century-Crofts
  91. Hurley LM, Devilbiss DM, Waterhouse BD. 2004. A matter of focus: monoaminergic modulation of stimulus coding in mammalian sensory networks. Curr. Opin. Neurobiol. 14:4488–95
     [Google Scholar]
  92. Iigaya K, Fonseca MS, Murakami M, Mainen ZF, Dayan P. 2018. An effect of serotonergic stimulation on learning rates for rewards apparent after long intertrial intervals. Nat. Commun. 9:12477
     [Google Scholar]
  93. Inagaki HK, Fontolan L, Romani S, Svoboda K. 2019. Discrete attractor dynamics underlies persistent activity in the frontal cortex. Nature 566:7743212–17
     [Google Scholar]
  94. Ishimura K, Takeuchi Y, Fujiwara K, Tominaga M, Yoshioka H, Sawada T. 1988. Quantitative analysis of the distribution of serotonin-immunoreactive cell bodies in the mouse brain. Neurosci. Lett. 91:3265–70
     [Google Scholar]
  95. Iwata J-I, Shima K, Tanji J, Mushiake H. 2013. Neurons in the cingulate motor area signal context-based and outcome-based volitional selection of action. Exp. Brain Res. 229:3407–17
     [Google Scholar]
  96. Jacobs BL, Azmitia EC. 1992. Structure and function of the brain serotonin system. Physiol. Rev. 72:1165–229
     [Google Scholar]
  97. Jacobs BL, Fornal CA. 1991. Activity of brain serotonergic neurons in the behaving animal. Pharmacol. Rev. 43:4563–78
     [Google Scholar]
  98. Kalwani RM, Joshi S, Gold JI. 2014. Phasic activation of individual neurons in the locus ceruleus/subceruleus complex of monkeys reflects rewarded decisions to go but not stop. J. Neurosci. 34:4113656–69
     [Google Scholar]
  99. Kiebel SJ, Daunizeau J, Friston KJ. 2008. A hierarchy of time-scales and the brain. PLOS Comput. Biol. 4:11e1000209
     [Google Scholar]
  100. Kiehn O, Eken T. 1998. Functional role of plateau potentials in vertebrate motor neurons. Curr. Opin. Neurobiol. 8:6746–52
     [Google Scholar]
  101. Kim E, Bari BA, Cohen JY. 2021. Subthreshold basis for reward-predictive persistent activity in mouse prefrontal cortex. Cell Rep 35:5109082
     [Google Scholar]
  102. Kim H, Sul JH, Huh N, Lee D, Jung MW 2009. Role of striatum in updating values of chosen actions. J. Neurosci. 29:4714701–12
     [Google Scholar]
  103. Kim JN, Shadlen MN. 1999. Neural correlates of a decision in the dorsolateral prefrontal cortex of the macaque. Nat. Neurosci. 2:2176–85
     [Google Scholar]
  104. Kimpo RR, Rinaldi JM, Kim CK, Payne HL, Raymond JL. 2014. Gating of neural error signals during motor learning. eLife 3:e02076
     [Google Scholar]
  105. Kimura F, Fukuda M, Tsumoto T. 1999. Acetylcholine suppresses the spread of excitation in the visual cortex revealed by optical recording: possible differential effect depending on the source of input. Eur. J. Neurosci. 11:103597–609
     [Google Scholar]
  106. Kjaerby C, Athilingam J, Robinson SE, Iafrati J, Sohal VS. 2016. Serotonin 1B receptors regulate prefrontal function by gating callosal and hippocampal inputs. Cell Rep 17:112882–90
     [Google Scholar]
  107. Klinkenberg I, Sambeth A, Blokland A. 2011. Acetylcholine and attention. Behav. Brain Res. 221:2430–42
     [Google Scholar]
  108. Kopec CD, Erlich JC, Brunton BW, Deisseroth K, Brody CD. 2015. Cortical and subcortical contributions to short-term memory for orienting movements. Neuron 88:2367–77
     [Google Scholar]
  109. Kovács P, Hernádi I. 2003. Alpha2 antagonist yohimbine suppresses maintained firing of rat prefrontal neurons in vivo. Neuroreport 14:6833–36
     [Google Scholar]
  110. Krashes MJ, Koda S, Ye CP, Rogan SC, Adams AC et al. 2011. Rapid, reversible activation of AgRP neurons drives feeding behavior in mice. J. Clin. Investig. 121:41424–28
     [Google Scholar]
  111. Krashes MJ, Shah BP, Madara JC, Olson DP, Strochlic DE et al. 2014. An excitatory paraventricular nucleus to AgRP neuron circuit that drives hunger. Nature 507:7491238–42
     [Google Scholar]
  112. Krugel LK, Biele G, Mohr PNC, Li S-C, Heekeren HR. 2009. Genetic variation in dopaminergic neuromodulation influences the ability to rapidly and flexibly adapt decisions. PNAS 106:4217951–56
     [Google Scholar]
  113. Kubota K, Niki H 1971. Prefrontal cortical unit activity and delayed alternation performance in monkeys. J. Neurophysiol. 34:3337–47
     [Google Scholar]
  114. Laszlovszky T, Schlingloff D, Hegedüs P, Freund TF, Gulyás A et al. 2020. Distinct synchronization, cortical coupling and behavioral function of two basal forebrain cholinergic neuron types. Nat. Neurosci. 23:8992–1003
     [Google Scholar]
  115. Lau B, Glimcher PW. 2008. Value representations in the primate striatum during matching behavior. Neuron 58:3451–63
     [Google Scholar]
  116. Lee SJ, Lodder B, Chen Y, Patriarchi T, Tian L, Sabatini BL. 2021. Cell-type-specific asynchronous modulation of PKA by dopamine in learning. Nature 590:451–56
     [Google Scholar]
  117. Leon MI, Shadlen MN. 1999. Effect of expected reward magnitude on the response of neurons in the dorsolateral prefrontal cortex of the macaque. Neuron 24:2415–25
     [Google Scholar]
  118. Li J, Daw ND. 2011. Signals in human striatum are appropriate for policy update rather than value prediction. J. Neurosci. 31:145504–11
     [Google Scholar]
  119. Li N, Daie K, Svoboda K, Druckmann S. 2016. Robust neuronal dynamics in premotor cortex during motor planning. Nature 532:7600459–64
     [Google Scholar]
  120. Li Y, Dalphin N, Hyland BI. 2013. Association with reward negatively modulates short latency phasic conditioned responses of dorsal raphe nucleus neurons in freely moving rats. J. Neurosci. 33:115065–78
     [Google Scholar]
  121. Li Y, Zhong W, Wang D, Feng Q, Liu Z et al. 2016. Serotonin neurons in the dorsal raphe nucleus encode reward signals. Nat. Commun. 7:10503
     [Google Scholar]
  122. Lim S, Goldman MS. 2013. Balanced cortical microcircuitry for maintaining information in working memory. Nat. Neurosci. 16:91306–14
     [Google Scholar]
  123. Lipps DB, Galecki AT, Ashton-Miller JA. 2011. On the implications of a sex difference in the reaction times of sprinters at the Beijing Olympics. PLOS ONE 6:10e26141
     [Google Scholar]
  124. Liu T, Kong D, Shah BP, Ye C, Koda S et al. 2012. Fasting activation of AgRP neurons requires NMDA receptors and involves spinogenesis and increased excitatory tone. Neuron 73:511–522
     [Google Scholar]
  125. Liu Z, Zhou J, Li Y, Hu F, Lu Y et al. 2014. Dorsal raphe neurons signal reward through 5-HT and glutamate. Neuron 81:61360–74
     [Google Scholar]
  126. Luce RD. 1986. Response Times: Their Role in Inferring Elementary Mental Organization Oxford, UK: Oxford Univ. Press
  127. Major G, Baker R, Aksay E, Mensh B, Seung HS, Tank DW. 2004. Plasticity and tuning by visual feedback of the stability of a neural integrator. PNAS 101:207739–44
     [Google Scholar]
  128. Mandelblat-Cerf Y, Ramesh RN, Burgess CR, Patella P, Yang Z et al. 2015. Arcuate hypothalamic AgRP and putative POMC neurons show opposite changes in spiking across multiple timescales. eLife 4:e07122
     [Google Scholar]
  129. Marder E, Thirumalai V. 2002. Cellular, synaptic and network effects of neuromodulation. Neural Netw 15:4–6479–93
     [Google Scholar]
  130. Massi B, Donahue CH, Lee D. 2018. Volatility facilitates value updating in the prefrontal cortex. Neuron 99:3598–608.e4
     [Google Scholar]
  131. Matias S, Lottem E, Dugué GP, Mainen ZF. 2017. Activity patterns of serotonin neurons underlying cognitive flexibility. eLife 6:e20552
     [Google Scholar]
  132. Maunsell JH, Gibson JR. 1992. Visual response latencies in striate cortex of the macaque monkey. J. Neurophysiol. 68:41332–44
     [Google Scholar]
  133. McGuire JT, Nassar MR, Gold JI, Kable JW. 2014. Functionally dissociable influences on learning rate in a dynamic environment. Neuron 84:4870–81
     [Google Scholar]
  134. Medina JF, Lisberger SG. 2008. Links from complex spikes to local plasticity and motor learning in the cerebellum of awake-behaving monkeys. Nat. Neurosci. 11:101185–92
     [Google Scholar]
  135. Miconi T, Rawal A, Clune J, Stanley KO. 2020. Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticity. arXiv:2002.10585 [cs.NE]
  136. Milojkovic BA, Radojicic MS, Goldman-Rakic PS, Antic SD 2004. Burst generation in rat pyramidal neurones by regenerative potentials elicited in a restricted part of the basilar dendritic tree. J. Physiol. 558:Pt. 1193–211
     [Google Scholar]
  137. Mitrano DA, Pare J-F, Smith Y, Weinshenker D. 2014. D1-dopamine and α1-adrenergic receptors co-localize in dendrites of the rat prefrontal cortex. Neuroscience 258:90–100
     [Google Scholar]
  138. Miyazaki K, Miyazaki KW, Doya K. 2011. Activation of dorsal raphe serotonin neurons underlies waiting for delayed rewards. J. Neurosci. 31:2469–79
     [Google Scholar]
  139. Moore RY, Bloom FE. 1979. Central catecholamine neuron systems: anatomy and physiology of the norepinephrine and epinephrine systems. Annu. Rev. Neurosci. 2:113–68
     [Google Scholar]
  140. Moore RY, Halaris AE, Jones BE. 1978. Serotonin neurons of the midbrain raphe: ascending projections. J. Comp. Neurol. 180:3417–38
     [Google Scholar]
  141. Munoz DP, Wurtz RH. 1992. Role of the rostral superior colliculus in active visual fixation and execution of express saccades. J. Neurophysiol. 67:41000–2
     [Google Scholar]
  142. Nakamura K, Matsumoto M, Hikosaka O. 2008. Reward-dependent modulation of neuronal activity in the primate dorsal raphe nucleus. J. Neurosci. 28:205331–43
     [Google Scholar]
  143. Nassar MR, Rumsey KM, Wilson RC, Parikh K, Heasly B, Gold JI. 2012. Rational regulation of learning dynamics by pupil-linked arousal systems. Nat. Neurosci. 15:71040–46
     [Google Scholar]
  144. Navarro-Lobato I, Genzel L. 2019. The up and down of sleep: from molecules to electrophysiology. Neurobiol. Learn. Mem. 160:3–10
     [Google Scholar]
  145. Niv Y, Daw ND, Joel D, Dayan P 2007. Tonic dopamine: opportunity costs and the control of response vigor. Psychopharmacology 191:3507–20
     [Google Scholar]
  146. O'Doherty JP, Lee SW, Tadayonnejad R, Cockburn J, Iigaya K, Charpentier CJ. 2021. Why and how the brain weights contributions from a mixture of experts. Neurosci. Biobehav. Rev. 123:14–23
     [Google Scholar]
  147. Opris I, Lebedev M, Nelson RJ. 2011. Motor planning under unpredictable reward: modulations of movement vigor and primate striatum activity. Front. Neurosci. 5:61
     [Google Scholar]
  148. Panigrahi B, Martin KA, Li Y, Graves AR, Vollmer A et al. 2015. Dopamine is required for the neural representation and control of movement vigor. Cell 162:61418–30
     [Google Scholar]
  149. Pasquereau B, Turner RS. 2013. Limited encoding of effort by dopamine neurons in a cost-benefit trade-off task. J. Neurosci. 33:198288–300
     [Google Scholar]
  150. Pressler RT, Strowbridge BW. 2006. Blanes cells mediate persistent feedforward inhibition onto granule cells in the olfactory bulb. Neuron 49:6889–904
     [Google Scholar]
  151. Puig MV, Watakabe A, Ushimaru M, Yamamori T, Kawaguchi Y. 2010. Serotonin modulates fast-spiking interneuron and synchronous activity in the rat prefrontal cortex through 5-HT1A and 5-HT2A receptors. J. Neurosci. 30:62211–22
     [Google Scholar]
  152. Raggio MW, Schreiner CE. 1994. Neuronal responses in cat primary auditory cortex to electrical cochlear stimulation. I. Intensity dependence of firing rate and response latency. J. Neurophysiol. 72:52334–59
     [Google Scholar]
  153. Ranade SP, Mainen ZF. 2009. Transient firing of dorsal raphe neurons encodes diverse and specific sensory, motor, and reward events. J. Neurophysiol. 102:53026–37
     [Google Scholar]
  154. Reppert TR, Servant M, Heitz RP, Schall JD. 2018. Neural mechanisms of speed-accuracy tradeoff of visual search: saccade vigor, the origin of targeting errors, and comparison of the superior colliculus and frontal eye field. J. Neurophysiol. 120:1372–84
     [Google Scholar]
  155. Roemmich RT, Bastian AJ. 2018. Closing the loop: from motor neuroscience to neurorehabilitation. Annu. Rev. Neurosci. 41:415–29
     [Google Scholar]
  156. Roesch MR, Calu DJ, Schoenbaum G. 2007. Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nat. Neurosci. 10:121615–24
     [Google Scholar]
  157. Salamone JD, Correa M. 2002. Motivational views of reinforcement: implications for understanding the behavioral functions of nucleus accumbens dopamine. Behav. Brain Res. 137:1–23–25
     [Google Scholar]
  158. Samejima K, Ueda Y, Doya K, Kimura M. 2005. Representation of action-specific reward values in the striatum. Science 310:57521337–40
     [Google Scholar]
  159. Sara SJ. 2009. The locus coeruleus and noradrenergic modulation of cognition. Nat. Rev. Neurosci. 10:3211–23
     [Google Scholar]
  160. Sara SJ, Segal M. 1991. Plasticity of sensory responses of locus coeruleus neurons in the behaving rat: implications for cognition. Prog. Brain Res. 88:571–85
     [Google Scholar]
  161. Satoh T, Nakai S, Sato T, Kimura M. 2003. Correlated coding of motivation and outcome of decision by dopamine neurons. J. Neurosci. 23:309913–23
     [Google Scholar]
  162. Schall JD, Hanes DP. 1993. Neural basis of saccade target selection in frontal eye field during visual search. Nature 366:6454467–69
     [Google Scholar]
  163. Schultz W, Dayan P, Montague PR. 1997. A neural substrate of prediction and reward. Science 275:53061593–99
     [Google Scholar]
  164. Seamans JK, Durstewitz D, Christie BR, Stevens CF, Sejnowski TJ. 2001. Dopamine D1/D5 receptor modulation of excitatory synaptic inputs to layer V prefrontal cortex neurons. PNAS 98:1301–6
     [Google Scholar]
  165. Seillier L, Lorenz C, Kawaguchi K, Ott T, Nieder A et al. 2017. Serotonin decreases the gain of visual responses in awake macaque V1. J. Neurosci. 37:4711390–405
     [Google Scholar]
  166. Shadmehr R, Ahmed AA. 2020. Vigor: Neuroeconomics of Movement Control Cambridge, MA: MIT Press
  167. Shadmehr R, Reppert TR, Summerside EM, Yoon T, Ahmed AA. 2019. Movement vigor as a reflection of subjective economic utility. Trends Neurosci 42:5323–36
     [Google Scholar]
  168. Shadmehr R, Smith MA, Krakauer JW. 2010. Error correction, sensory prediction, and adaptation in motor control. Annu. Rev. Neurosci. 33:89–108
     [Google Scholar]
  169. Shima K, Tanji J. 1998. Both supplementary and presupplementary motor areas are crucial for the temporal organization of multiple movements. J. Neurophysiol. 80:63247–60
     [Google Scholar]
  170. Shin EJ, Jang Y, Kim S, Kim H, Cai X et al. 2021. Robust and distributed neural representation of action values. eLife 10:e53045
     [Google Scholar]
  171. Smith MA, Ghazizadeh A, Shadmehr R. 2006. Interacting adaptive processes with different timescales underlie short-term motor learning. PLOS Biol 4:6e179
     [Google Scholar]
  172. Soltani A, Murray JD, Seo H, Lee D. 2021. Timescales of cognition in the brain. Curr. Opin. Behav. Sci. 41:30–37
     [Google Scholar]
  173. Stalnaker TA, Calhoon GG, Ogawa M, Roesch MR, Schoenbaum G. 2010. Neural correlates of stimulus-response and response-outcome associations in dorsolateral versus dorsomedial striatum. Front. Int. Neurosci. 4:12
     [Google Scholar]
  174. Stephens DW, Krebs JR. 1987. Foraging Theory Princeton, NJ: Princeton Univ. Press
  175. Stephens EK, Avesar D, Gulledge AT. 2014. Activity-dependent serotonergic excitation of callosal projection neurons in the mouse prefrontal cortex. Front. Neural Circuits 8:97
     [Google Scholar]
  176. Steriade M, Timofeev I, Grenier F. 2001. Natural waking and sleep states: a view from inside neocortical neurons. J. Neurophysiol. 85:51969–85
     [Google Scholar]
  177. Sutton RS, Barto AG. 1998. Reinforcement Learning: An Introduction Cambridge, MA: MIT Press
  178. Sutton RS, Precup D, Singh S. 1999. Between MDPs and semi-MDPs: a framework for temporal abstraction in reinforcement learning. Artif. Intell. 112:181–211
     [Google Scholar]
  179. Tervo DGR, Proskurin M, Manakov M, Kabra M, Vollmer A et al. 2014. Behavioral variability through stochastic choice and its gating by anterior cingulate cortex. Cell 159:121–32
     [Google Scholar]
  180. Thura D, Cisek P. 2017. The basal ganglia do not select reach targets but control the urgency of commitment. Neuron 95:51160–70.e5
     [Google Scholar]
  181. Tononi G. 2004. An information integration theory of consciousness. BMC Neurosci 5:42
     [Google Scholar]
  182. Tsutsui K-I, Grabenhorst F, Kobayashi S, Schultz W. 2016. A dynamic code for economic object valuation in prefrontal cortex neurons. Nat. Commun. 7:12554
     [Google Scholar]
  183. Valero-Aracama MJ, Reboreda A, Arboit A, Sauvage M, Yoshida M. 2021. Noradrenergic suppression of persistent firing in hippocampal CA1 pyramidal cells through cAMP-PKA pathway. eNeuro 8:2ENEURO.0440-20.2020
     [Google Scholar]
  184. Varazzani C, San-Galli A, Gilardeau S, Bouret S. 2015. Noradrenaline and dopamine neurons in the reward/effort trade-off: a direct electrophysiological comparison in behaving monkeys. J. Neurosci. 35:207866–77
     [Google Scholar]
  185. Vecoven N, Ernst D, Wehenkel A, Drion G. 2020. Introducing neuromodulation in deep neural networks to learn adaptive behaviours. PLOS ONE 15:1e0227922
     [Google Scholar]
  186. Vilaró MT, Cortés R, Mengod G, Hoyer D 2020. Distribution of 5-HT receptors in the central nervous system: an update. Handbook of the Behavioral Neurobiology of Serotonin CP Müller, KA Cunningham 121–46 Cambridge, MA: Academic
     [Google Scholar]
  187. Vishwanathan A, Daie K, Ramirez AD, Lichtman JW, Aksay ERF, Seung HS. 2017. Electron microscopic reconstruction of functionally identified cells in a neural integrator. Curr. Biol. 27:142137–47.e3
     [Google Scholar]
  188. Wang AY, Miura K, Uchida N. 2013. The dorsomedial striatum encodes net expected return, critical for energizing performance vigor. Nat. Neurosci. 16:5639–47
     [Google Scholar]
  189. Wang XJ. 1999. Synaptic basis of cortical persistent activity: the importance of NMDA receptors to working memory. J. Neurosci. 19:219587–603
     [Google Scholar]
  190. Wang Y, Markram H, Goodman PH, Berger TK, Ma J, Goldman-Rakic PS. 2006. Heterogeneity in the pyramidal network of the medial prefrontal cortex. Nat. Neurosci. 9:4534–42
     [Google Scholar]
  191. Wang Y, Yin X, Zhang Z, Li J, Zhao W, Guo ZV. 2021. A cortico-basal ganglia-thalamo-cortical channel underlying short-term memory. Neuron 109:3486–99.e7
     [Google Scholar]
  192. Watanabe M. 1996. Reward expectancy in primate prefrontal neurons. Nature 382:6592629–32
     [Google Scholar]
  193. Wolpert DM, Ghahramani Z. 2000. Computational principles of movement neuroscience. Nat. Neurosci. 3:1212–17
     [Google Scholar]
  194. Xiang L, Harel A, Gao HY, Pickering AE, Sara SJ, Wiener SI. 2019. Behavioral correlates of activity of optogenetically identified locus coeruleus noradrenergic neurons in rats performing T-maze tasks. Sci. Rep. 9:11361
     [Google Scholar]
  195. Yang CR, Seamans JK. 1996. Dopamine D1 receptor actions in layers V-VI rat prefrontal cortex neurons in vitro: modulation of dendritic-somatic signal integration. J. Neurosci. 16:51922–35
     [Google Scholar]
  196. Yang H, Bari BA, Cohen JY, O'Connor DH. 2021. Locus coeruleus spiking differently correlates with S1 cortex activity and pupil diameter in a tactile detection task. eLife 10:e64327
     [Google Scholar]
  197. Yoon T, Geary RB, Ahmed AA, Shadmehr R. 2018. Control of movement vigor and decision making during foraging. PNAS 115:44E10476–85
     [Google Scholar]
  198. Yu AJ, Dayan P. 2005. Uncertainty, neuromodulation, and attention. Neuron 46:4681–92
     [Google Scholar]
  199. Záborszky L, Gombkoto P, Varsanyi P, Gielow MR, Poe G et al. 2018. Specific basal forebrain-cortical cholinergic circuits coordinate cognitive operations. J. Neurosci. 38:449446–58
     [Google Scholar]
  200. Zagha E, McCormick DA. 2014. Neural control of brain state. Curr. Opin. Neurobiol. 29:178–86
     [Google Scholar]
  201. Zaghloul KA, Blanco JA, Weidemann CT, McGill K, Jaggi JL et al. 2009. Human substantia nigra neurons encode unexpected financial rewards. Science 323:59201496–99
     [Google Scholar]
  202. Zhang SX, Lutas A, Yang S, Diaz A, Fluhr H et al. 2021. Hypothalamic dopamine neurons motivate mating through persistent cAMP signalling. Nature 597:245–49
     [Google Scholar]
  203. Zhang Z, Matos SC, Jego S, Adamantidis A, Séguéla P. 2013. Norepinephrine drives persistent activity in prefrontal cortex via synergistic α1 and α2 adrenoceptors. PLOS ONE 8:6e66122
     [Google Scholar]
/content/journals/10.1146/annurev-neuro-092021-125059
Loading
Neuromodulation and Neurophysiology on the Timescale of Learning and Decision-Making
Annual Review of Neuroscience 45, 317 (2022); https://doi.org/10.1146/annurev-neuro-092021-125059
/content/journals/10.1146/annurev-neuro-092021-125059
/content/journals/10.1146/annurev-neuro-092021-125059
Loading

Data & Media loading...

  • Article Type: Review Article

Most Cited Most Cited RSS feed

This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error