Dopamine – one signal underlying dynamics of both motivation and learning
Abstract
The layman understanding of dopamine is that of the ‘reward’ neurotransmitter, due to its associations with addiction. However many theories of dopamine function have argued for its role in motivation rather than encoding reward. One initial idea was that different rates of dopamine firing encode distinct types of information related to either motivation or reward. However, more recently evidence shows that the same dopamine-encoded signal is used for both processes. This review looks at the two main theoretical frameworks of dopaminergic function and how they integrate in a novel model.
Introduction
Mesolimbic lesions and pharmacological blockers of dopamine activity impair the ability to obtain rewards, from food to sexual contact and addictive drugs [1, 2]. Despite being well trained to lever press for reward, rats stopped when dopamine release in the nucleus accumbens was blocked, suggesting dopaminergic involvement beyond reward-learning [3]. Two main theories were postulated: learning vs. motivation. In the 1990s, Schultz et al. [4,5] identified a phasic increase of mesolimbic dopaminergic firing in response to unexpected rewards. Their conclusion was that dopamine encoded the difference between expected and actual reward: reward prediction error (RPE), as a learning signal. Berridge et al. [6] challenged the function of dopamine as a driver of learning, proposing it works to motivate learned reward-obtaining behaviours. When mesolimbic dopamine signalling was blocked rats did not work to obtain reward, but still consumed and enjoyed food pellets if directly delivered, suggesting they might lack motivation [6]. When prolonged tonic firing was increased, without altering phasic dopamine firing, reinforcement learning remained unchanged, but motivation to work for reward increased [7]. These results suggested that phasic dopamine release encodes RPE, while tonic firing motivates behaviour. However, it lacked research data, as only tonic fluctuations were manipulated.
One signal, two functions
Hamid et al. [9] disputed the idea that the type of dopamine release (phasic vs. tonic) influences motivation and learning. They contended that dopamine encodes a value of reward availability that can be used both to drive motivation to work and be integrated into RPE. In an adaptive decision-making task, rats were placed in an operant chamber with a five-hole nose poke wall. Rats had to poke a central illuminated hole until they received a Go cue (white noise) which prompted them to move either to a right or left chamber to elicit food delivery. Each of the ports had a probabilistic value of reward (10%, 50% or 90%) and rats were instructed to move, but not which port to choose. Reward delivery was paired with a reward sound cue (Fig. 1). Rats quickly learned to reproduce behaviours that were more likely to yield reward, as their decision-making adapts from random to probabilistic. When the reward rate was high they showed a shorter latency between the time the central hole was illuminated and the nose poke, suggesting they were more motivated to work. Three main stages within the behavioural testing paradigm elicited a sudden increase in dopamine: rats poking their nose through the illuminated central hole (engaging with the task), Go cue being presented and reward cue. Thus, dopamine increased both in energising motivation to work for reward (engaging with task, Go cue) and in learning (reward cue associated with food delivery).
Observing these three different dopamine-dependent time points led Hamid et al. to propose that dopamine encodes a fluctuating ‘value function’. The ‘value function’ monitors distance from reward and increases the closer an animal gets to obtaining reward. If reward is distant, the value will be low and animals will be less likely to engage in work. Once the animals have engaged, they are more likely to remain motivated as reward draws closer. Cues can change this value. A reward cue predicts the reward is very close so there is a significant increase in value. The difference between initial value before the cue and cue-predicted value can be encoded as an RPE. Thus, reward cues will produce high RPEs by abruptly changing the value. Hamid et al. then fed the animal sequence of behaviour into this updated learning model and showed that indeed the changes in ‘value function’ were closely correlated to recorded changes in dopamine levels. Dopamine was proposed to encode fluctuating changes in reward availability in order to motivate behaviour, while sudden changes in reward value, indicated by cues, encoded learning signals in the form of RPEs. In order to confirm the involvement of dopamine in both learning and motivation Hamid et al. used optogenetic excitation and inhibition to increase dopamine firing at different stages of experimental testing. If dopamine release was manipulated during the reward cue, learning was enhanced or abolished, respectively. However, manipulating dopamine release at the beginning of the trials had no effect on subsequent choice behaviour. It did however alter the time it took rats to engage with the task, supporting the idea that mesolimbic dopamine also motivates behaviour.
Discussion
The integrative model proposed by Hamid et al. offers the most likely explanation that mesolimbic dopamine signalling does not solely serve one function. Their work was essential in sparking further research of cellular features of dopamine signalling within this new model of the fluctuating ‘value function’ [9-11]. One issue when researching closely correlated cognitive processes such as learning and motivation is defining clear conceptual parameters. Early-day research used terms such as motivation, ‘wanting’ and drive interchangeably, whereas others made clear distinctions as to what is considered motivation [12, 13]. Investigating both motivation and learning in the same behavioural paradigm allowed for clear distinctions to be made both conceptually and behaviourally. It also highlighted the importance of precisely calibrating animal behavioural assays in order to target specific research question. Even slight variations in designing a decision making task can unveil distinct cognitive processes of motivation or learning.The function of dopamine has posed a complex question for researchers as it is linked to a wide array of pathologies from Parkinson’s disease to addiction. Its involvement in reward and motivation also makes it a key driver of decision making, both on a smaller, individual scale and a wider socio-politico-economic scale. This review shows how multiple cognitive processes employ the same dopamine-encoded flexible signal of reward availability and why different theoretical models of neurotransmitter function are not necessarily mutually exclusive, but potentially compatible. Research discussed in this review has been pivotal in providing a new avenue for understanding and potentially modulating dopamine control of behaviour.
Bibliography
[1] R. Wise and H. Schwartz, “Pimozide attenuates acquisition of lever-pressing for food in rats”, Pharmacology Biochemistry And Behavior, 15(4), 655-656, 1981
[2] C. Spyraki, H. Fibiger and A. Phillips. “Attenuation of heroin reward in rats by disruption of the mesolimbic dopamine system”, Psychopharmacology, 79(2-3), 278-283, 1993
[3] C. Spyraki, G. Nomikos and D. Varonos, “Intravenous cocaine-induced place preference: attenuation by haloperidol”, Behavioural Brain Research, 26(1), 57-62, 1987
[4] W. Schultz and R. Romo, “Dopamine neurons of the monkey midbrain: contingencies of responses to stimuli eliciting immediate behavioral reactions”, Journal Of Neurophysiology, 63(3), 607-624, 1990
[5] W. Schultz, P. Apicella and T. Ljungberg, “Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task”, Journal of neuroscience, 13(3), 900-913, 1993
[6] K. Berridge, I. Venier and T. Robinson, “Taste reactivity analysis of 6-hydroxydopamine-induced aphagia: Implications for arousal and anhedonia hypotheses of dopamine function”, Behavioral Neuroscience, 103(1), 36-45, 1989.
[7] B. Cagniard, J. Beeler, J. Britt, D. McGehee, M. Marinelli and X. Zhuang, “Dopamine Scales Performance in the Absence of New Learning”, Neuron, 51(5), 541-547, 2006
[8] A. Hamid, J. Pettibone, O. Mabrouk, V. Hetrick, R. Schmidt and C. Vander Weel, “Mesolimbic dopamine signals the value of work”, Nature Neuroscience, 19(1), 117-126, 2015
[9] C. Soares-Cunha, B. Coimbra, A. David-Pereira, S. Borges, L. Pinto and P. Costa, “Activation of D2 dopamine receptor-expressing neurons in the nucleus accumbens increases motivation”, Nature Communications, 7(1), 2016
[10] J. Berke, “What does dopamine mean?”, Nature Neuroscience, 21(6), 787-793, 2018
[11] A. Mohebi, J. Pettibone, A. Hamid, J. Wong, L. Vinson, and T. Patriarchi, “Dissociable dopamine dynamics for learning and motivation”, Nature, 570(7759), 65-70, 2019
[12] C. Wyvell and K. Berridge, “Intra-Accumbens Amphetamine Increases the Conditioned Incentive Salience of Sucrose Reward: Enhancement of Reward “Wanting” without Enhanced “Liking” or Response Reinforcement”, The Journal Of Neuroscience, 20(21), 8122-8130, 2000[13] K. Berridge and T. Robinson, “Parsing reward”, Trends In Neurosciences, 26(9), 507-513, 2003
Reviewer’s Report
Access the report(s) of the peers who reviewed this article below: