Figure 5 - uploaded by Suzanna Becker
Content may be subject to copyright.
The internal model after training on different environmental contingencies. Each circle denotes a state of the internal model, presented according to the stimulus/time since onset represented by that state. The connecting lines show the learned strengths of the transition connections between those states. A maximum transition weight of 1 is denoted by a thick line, continuously graded in thickness down to a transition weight of 0 denoted by no line. Only the relevant internal states are shown. The learned reward values, R , are not shown but can be imagined inside each state. They are zero for all states except the reward state itself (i.e., the US). (a)–(c) show the internal model after the three stages of Figure 4c. (d) shows the pattern of activation on the state units of 5c during each stage of the look-ahead process following the onset of CS2. (e) and (f) show the effects of presenting a compound stimulus as CS.
Source publication
The notion of prediction error has established itself at the heart of formal models of animal learning and current hypotheses of dopamine function. Several interpretations of prediction error have been offered, including the model-free reinforcement learning method known as temporal difference learning (TD), and the important Rescorla-Wagner (RW) l...
Contexts in source publication
Context 1
... the early response generates a signal, but the internal model is quickly adapted. Figure 5a shows the internal model constructed by trial 25. The CS is labelled "CS1," and the US is labelled "Reward." ...
Context 2
... the US to CS1, and then from CS1 to CS2 (the earliest reliable predictor of the US). When the interval between CS2 and CS1 is varied, a signal re-appears at the time of CS1. Within the simulation, this re-appearance reflects the internal model's uncertainty regarding the time of arrival of CS1. The original experimental result is summarized in Figures 1f and 1g. Figures 5a-c show the states and the transition connections of the internal model after training on the three different contingencies of figure 4c. Figure 5d shows an example of the propagation of activation through the internal model during look- ahead following presentation of CS2 in trial 80. shows the results of simulating another experiment in ...
Context 3
... 5a-c show the states and the transition connections of the internal model after training on the three different contingencies of figure 4c. Figure 5d shows an example of the propagation of activation through the internal model during look- ahead following presentation of CS2 in trial 80. shows the results of simulating another experiment in which the US only follows the CS on 25% of the trials. This experimental effect is summarized in Figure 1h. ...
Context 4
... overshadowing is not evident in the original electrophysiological data. Figure 5e shows the internal model that is constructed during the conditioning of the compound stimulus, BY. ...
Context 5
... where φ defines the intrinsic associability of each stimulus. Figure 5e shows the internal model after conditioning of a compound stimulus where φ(CS1) = φ(CS2), and Figure 5f shows the model when φ(CS2) = 2 × φ(CS1). As a result, the transition connections from CS2 are twice the strength of those from CS1. ...
Context 6
... where φ defines the intrinsic associability of each stimulus. Figure 5e shows the internal model after conditioning of a compound stimulus where φ(CS1) = φ(CS2), and Figure 5f shows the model when φ(CS2) = 2 × φ(CS1). As a result, the transition connections from CS2 are twice the strength of those from CS1. ...
Similar publications
The activity of primary auditory cortex (A1) neurons is modulated not only by sensory inputs but also by other task-related variables in associative learning. However, it is unclear how A1 neural activity changes dynamically in response to these variables during the learning process of associative memory tasks. Therefore, we developed an associativ...
Phasic activity of midbrain dopamine neurons is currently thought to encapsulate the prediction-error signal described in Sutton and Barto’s (1981) model-free reinforcement learning algorithm. This phasic signal is thought to contain information about the quantitative value of reward, which transfers to the reward-predictive cue after learning. Thi...
Hebbian associative learning is a common form of neuronal adaptation in the brain and is important for many physiological functions such as motor learning, classical conditioning and operant conditioning. Here we show that a Hebbian associative learning synapse is an ideal neuronal substrate for the simultaneous implementation of high-gain adaptive...
In comparison to neutral faces, facial expressions of emotion are known to gain attentional prioritization, mainly demonstrated by means of event-related potentials (ERPs). Recent evidence indicated that such a preferential processing can also be elicited by neutral faces when associated with increased motivational salience via reward. It remains,...
In two appetitive conditioning experiments with rats, we investigated the mechanisms responsible for demonstrations of the superior associability of overshadowed conditioned stimuli (CSs) relative to control CSs. In Experiment 1, we investigated whether previous demonstrations were a consequence of differences in the relationship between the CSs an...
Citations
... Model-based predictions need to be distinguished from model-free predictions employed in reinforcement and associative learning (Smith et al., 2006). To predict (P) its future response (R P ) based on an input stimulus (S), a system can apply a conditional rule-based approach such as, "if S, then R P " which uses predetermined stored values of R P for every value of S. This is an example of model-free predictions. ...
How subjective experience is realized in nervous systems remains one of the great challenges in the natural sciences. An answer to this question should resolve debate about which animals are capable of subjective experience. We contend that subjective experience of sensory stimuli is dependent on the brain’s awareness of its internal neural processing of these stimuli. This premise is supported by empirical evidence demonstrating that disruption to either processing streams or awareness states perturb subjective experience. Given that the brain must predict the nature of sensory stimuli, we reason that conscious awareness is itself dependent on predictions generated by hierarchically organized forward models of the organism’s internal sensory processing. The operation of these forward models requires a specialized neural architecture and hence any nervous system lacking this architecture is unable to subjectively experience sensory stimuli. This approach removes difficulties associated with extrapolations from behavioral and brain homologies typically employed in addressing whether an animal can feel. Using nociception as a model sensation, we show here that the Drosophila brain lacks the required internal neural connectivity to implement the computations required of hierarchical forward models. Consequently, we conclude that Drosophila, and those insects with similar neuroanatomy, do not subjectively experience noxious stimuli and therefore cannot feel pain.
... Ultimately, we would see a spike in the DA system when the squirrel saw an oak tree followed by a lesser spike when it found an acorn and no spike at all if no acorns were found after the climb. The role of a salience system is not to reward success but to predict reward for an organism (Schultz et al. 1997;Berridge and Robinson 1998;Gottfried et al. 2003;Heinz and Schlagenhauf 2010;McClure et al. 2003;Egelman et al. 1998;Smith et al. 2006). ...
... In animals like rodents, this type of behavioral biasing is called incentive salience since it makes potentially rewarding object motivational magnets (Schultz et al. 1997;Robinson 1998, 2003;Braver et al. 1999;Tobler et al. 2005;Kapur 2003;McClure et al. 2003;Smith et al. 2006). Human cognition uses the same dopaminergic salience system at all levels to target cognitive function by selectively enhancing activity in neural circuits processing potentially rewarding information (Braver et al. 1999;Braver and Cohen 2000;Abi-Dargham et al. 2002;Durstewitz and Seamans 2002;Egelman et al. 1998;Goldman-Rakic 1997;Grace 1991). ...
Philosophers discuss Jerome Wakefield's influential view of mental disorder as “harmful dysfunction,” with detailed responses from Wakefield himself.
One of the most pressing theoretical problems of psychiatry is the definition of mental disorder. Jerome Wakefield's proposal that mental disorder is “harmful dysfunction” has been both influential and widely debated; philosophers have been notably skeptical about it. This volume provides the first book-length collection of responses by philosophers to Wakefield's harmful dysfunction analysis (HDA), offering a survey of philosophical critiques as well as extensive and detailed replies by Wakefield himself.
HDA is offered as a definition of mental disorder, but it is also the outcome of a method—conceptual analysis—and contributors first take up HDA's methodology, considering such topics as HDA's influences on the DSM, empirical support for HDA, and clinical practice. They go on to discuss HDA's ultimate goal, the demarcation between normal and abnormal; the dysfunction component of the analysis, addressing issues that include developmental plasticity, autism and neurodiversity, and the science of salience; and the harmful component, examining harmless dysfunction, normal variation, medicalization, and other questions. Wakefield offers substantive responses to each chapter.
The open access edition of this book was made possible by generous funding from Arcadia – a charitable fund of Lisbet Rausing and Peter Baldwin.
Contributors
Rachel Cooper, Andreas De Block, Steeves Demazeux, Leen De Vreese, Luc Faucher, Denis Forest, Justin Garson, Philip Gerrans, Harold Kincaid, Maël Lemoine, Dominic Murphy, Jonathan Sholll, Tim Thornton, Jerome Wakefield, Peter Zachar
... Cette théorie est applicable à une large variété d'apprentissages par essai-erreur (Suttonand Barto, 1981;Smith et al., 2006) dans lesquels le comportement est adapté aussi longtemps que le résultat est différent de celui espéré. Des stimuli aversifs, intenses ou simplement nouveaux sont également capables d'entraîner une stimulation phasique des neurones dopaminergiques (Horvitz, 2000). ...
Les substances addictives détournent les apprentissages par la récompense en augmentant la dopamine (DA) dans le système mésolimbique, en particulier dans le striatum, où elle module durablement la transmission glutamatergique excitatrice et contribue à la mise en place d’altérations comportementales persistantes. L’intégration des signaux dopaminergique et glutamatergique au sein du striatum est réalisée par les neurones moyen épineux du striatum (MSN), qui forment deux populations majoritairement distinctes : les MSN de la « voie directe » exprimant les récepteurs D1 de la DA (D1R-MSN) et ceux de la « voie indirecte » qui expriment les récepteurs D2 de la DA (D2R-MSN). Une hypothèse qui prévaut à l’heure actuelle est que la libération de DA induite par les drogues active les D1R-MSN, ce qui promeut le renforcement, alors qu’elle inhibe les D2R-MSN, diminuant ainsi leurs fonctions « anti-renforcement ». Les travaux du laboratoire ont montré que l’interaction physique (i.e. hétéromérisation) entre le D1R et le récepteur NMDA (NMDAR) du glutamate était nécessaire à la facilitation de la transmission glutamatergique par la DA dans les D1R-MSN. À l'inverse, d'autres travaux ont montré que l'interaction D2R / NMDAR sous-tend l'effet inhibiteur de DA sur la signalisation NMDAR dans les D2R-MSN. Toutefois, la modulation et la fonction de ces hétéromères in vivo dans les réponses induites par la cocaïne sont encore inconnues. À l'aide du test de « proximity ligation assay », nous avons montré que la sensibilisation locomotrice induite par des expositions répétées à la cocaïne est associée à la formation d'hétéromères D1R/NMDAR dans le Noyau Accumbens (NAc) et le Striatum Dorsal, tandis que l'hétéromérisation D2R/GluN2B est majoritairement observée au sein du NAc. Pour identifier les rôles des hétéromères DAR / NMDAR dans les différentes phases des adaptations moléculaires, morphologiques et comportementales induites par la cocaïne in vivo, nous avons conçu une approche virale pour perturber les hétéromères DAR / NMDAR de manière contrôlée dans le temps grâce à un promoteur dépendant de la doxycycline. Nous avons constaté que la perturbation de l'interaction D1R / NMDAR dans le NAc bloque la mise en place des altérations synaptiques induites par la cocaïne dans les D1R-MSN ainsi que le développement de la sensibilisation locomotrice et de la préférence de lieu conditionné par la cocaïne (CPP). A l’inverse, le blocage de l'interaction D2R / NMDAR interfère avec le maintien de la sensibilisation psychomotrice et de la CPP à la cocaïne. L’observation d’un rôle préférentiel des hétéromères D2R/GluN2B dans la maintenance des réponses comportementales à la cocaïne et leur absence d’effet dans le cas d’une récompense naturelle suggèrent que ce sous-type d’hétéromère pourrait être une cible thérapeutique à envisager. J’ai donc mis au point la détection des complexes D2R / NMDAR à partir des tissus striataux humains post-mortem issus d’individus avec un historique de dépendance aux psychostimulants ou des sujets sains. Ceci m’a permis de montrer, qu’en dépit d’une forte baisse de l’expression du D2R, la proportion de D2R formant des hétéromères avec les NMDAR est trois fois supérieure chez les sujets dépendants par rapport au sujets sains. Ce travail renforce donc les évidences en faveur d’un rôle central des interactions entre les systèmes dopaminergique et glutamatergiques dans les réponses aux drogues et identifie les hétéromères DAR / NMDAR comme des cibles moléculaires avec un potentiel thérapeutique non seulement dans la dépendance aux drogues mais également pour les nombreux troubles psychiatriques associés à un déséquilibre entre les transmissions dopaminergiques et du glutamatergiques.
... 14 Therefore, there has been substantial interest in the role of Ca v 1.2 on endophenotypes associated with these disorders. Altered associative learning has been implicated in the pathology of various psychiatric disorders, [15][16][17][18][19][20] particularly in the manifestation of positive (psychotic) symptoms. 21 There is also evidence at a genomic level that these learning processes are implicated in risk for schizophrenia. ...
... Psychosis has been suggested to be a result of aberrant salience-an increased focus on neutral stimuli that leads to altered learning and perception. 19,36,38,57 It has been suggested that positive symptoms in schizophrenia are a result of this fundamental abnormality in learning that leads to neutral or "safe" stimuli being misinterpreted as relevant or important. 38,41 Therefore, within this context, the results presented in this study may be explained as Cacna1c heterozygote rats experiencing aberrant salience; assigning importance to less relevant stimuli within the task leading to abnormal associations being formed. ...
CACNA1C, a gene that encodes an alpha-1 subunit of L-type voltage-gated calcium channels, has been strongly associated with psychiatric disorders including schizophrenia and bipolar disorder. An important objective is to understand how variation in this gene can lead to an increased risk of psychopathology. Altered associative learning has also been implicated in the pathology of psychiatric disorders, particularly in the manifestation of psychotic symptoms. In this study, we utilize auditory-cued fear memory paradigms in order to investigate whether associative learning is altered in rats hemizygous for the Cacna1c gene. Cacna1c hemizygous (Cacna1c+/-) rats and their wild-type littermates were exposed to either delay, trace, or unpaired auditory fear conditioning. All rats received a Context Recall (24 h post-conditioning) and a Cue Recall (48 h post-conditioning) to test their fear responses. In the delay condition, which results in strong conditioning to the cue in wild-type animals, Cacna1c+/- rats showed increased fear responses to the context. In the trace condition, which results in strong conditioning to the context in wild-type animals, Cacna1c+/- rats showed increased fear responses to the cue. Finally, in the unpaired condition, Cacna1c+/- rats showed increased fear responses to both context and cue. These results indicate that Cacna1c heterozygous rats show aberrantly enhanced fear responses to inappropriate cues, consistent with key models of psychosis.
... Yet, this further highlights the selection problem: If actions can be evaluated long before they are needed, which experiences should the brain consider at each moment, and in what order, to set the stage for the most rewarding future decisions? These questions are likely central not just to healthy decisions, but to a variety of disordered including hallucations, craving, and rumination (Huys et al., 2015a;Smith et al., 2006). Addressing them requires a new, more granular theory predicting the patterns in which individual memories should be accessed to compute actions' values. ...
To make decisions, animals must evaluate outcomes of candidate choices by accessing memories of relevant experiences. Yet little is known about which experiences are considered or ignored during deliberation, which ultimately governs choice. Here, we propose a normative theory to predict which memories should be accessed at each moment to optimize future decisions. Using nonlocal “replay” of spatial locations in hippocampus as a window into memory access, we simulate a spatial navigation task where an agent accesses memories of locations sequentially, ordered by utility: how much extra reward would be earned due to the computation enabling better choices. This prioritization balances two desiderata: the need to evaluate imminent choices, vs. the gain from propagating newly encountered information to predecessor states. We show that this theory offers a unifying account of a range of hitherto disconnected findings in the place cell literature such as the balance of forward and reverse replay, biases in the replayed content, and effects of experience. Accordingly, various types of nonlocal events during behavior and rest are re-interpreted as instances of a single choice evaluation operation, unifying seemingly disparate proposed functions of replay including planning, learning and consolidation, and whose dysfunction may underlie pathologies like rumination and craving.
... RL models also offer a useful computational framework for investigations of anticipatory action selection and adaptive behaviour (Killcross and Coutureau, 2003;Matsumoto and Tanaka, 2004;Balleine, 2005;Dolan, 2007;Rushworth and Behrens, 2008), as well as the neuromodulatory contributions to these processes (Yu and Dayan, 2005;Pessiglione et al., 2006;Doya, 2008). Moreover, their non-normative, descriptive nature allows for modelling aberrant modes of learning that occur in disease states such as schizophrenia or depression (Smith et al., 2006;Frank, 2008;Murray et al., 2008;Dayan and Huys, 2009). ...
Successful interaction with the environment requires flexible updating of our beliefs about the world. By learning to estimate the likelihood of future events, it is possible to prepare appropriate actions in advance and execute fast, accurate motor responses. According to theoretical proposals, humans track the variability arising from dynamic environments by computing various forms of uncertainty. Several neuromodulators have been linked to uncertainty signalling but comprehensive empirical characterisation of their roles in perceptual belief updating and motor response modulation has been lacking. This thesis interrogates the contributions of noradrenaline, acetylcholine and dopamine to human learning and action within a unified computational framework of uncertainty. First, I use pharmacological interventions to characterise the impact of noradrenergic, cholinergic and dopaminergic receptor antagonism on individual computations of uncertainty during a probabilistic serial reaction time task. I develop and employ a hierarchical Bayesian model to quantify human learning and action under three forms of uncertainty. I propose that noradrenaline influences learning of uncertain events arising from unexpected changes in the environment, while acetylcholine balances attribution of uncertainty to chance fluctuations within environmental contexts or to gross environmental violations following a contextual switch. In contrast, dopamine supports the use of uncertainty representations to engender fast, adaptive responses. Second, I extend these results by focusing on the effects of natural inter-individual variations in dopaminergic function. Specifically, I employ the same task and model to assess individual learning and action under uncertainty as a function of COMT genotype. Third, I focus on the role of noradrenaline. Uncertainty computations have been linked to changes in pupil diameter, and pupil dilation to noradrenergic neuronal activity in the locus coeruleus. Combining an auditory probabilistic learning task, pharmacological manipulations, pupillometry and computational modelling, I demonstrate that pupil diameter offers an indirect measure of dynamic noradrenergic computations of environmental uncertainty and volatility.
... The last ones could generate not only short but also large time scale predictions from minutes to month or years (Koster-Hale and Saxe, 2013). Typically, in an electrophysiological recording, the PE shows an increase in neural activity after unexpected events and a decrease in predicted ones (Schultz, 2007;Schultz and Dickinson, 2000;Smith et al., 2006). PE signals were found across several regions of the brain and in multiple functions like visual, auditory, somatosensory perception, attention, action, language, cognitive control and motivational value processing (Bubic et al., 2010;Garrison et al., 2013;Koster-Hale and Saxe, 2013;Niv and Schoenbaum, 2008;Ouden et al., 2012;Roesch et al., 2012). ...
... This pattern is typically found in a vast population of neurons from different parts of the brain as shown in Fig. 3 for midbrain dopaminergic (DA) neurons, one of the most studied neurons. Notably, the dopaminergic system is crucial for detection of mismatches and/or novelty (Lisman and Grace, 2005;Schultz et al., 1997;Smith et al., 2006). For example, at the beginning of an appetitive conditioning training, an unexpected reinforcement/reward such as juice, produces a phasic response (positive PE). ...
... Le transfert de l'activité dopaminergique au moment de la présentation du CS, au cours du conditionnement semble avoir un lien avec la surprise de l'animal, ce qui a amené à des interprétations du signal dopaminergique comme un signal de surprise et de nouveauté (Bunzeck et Düzel 2006 ;Smith et al. 2006). Cependant on peut expliquer ce type d'activité plus précisément dans le cadre de l'apprentissage et de l'encodage d'une information de type RPE. ...
... D'autres ont émis l'hypothèse que ce signal reflète le produit de la pertinence du signal et de la surprise associée (Smith et al. 2006). Cependant, ces études ne font qu'apporter un raffinement quant à l'interprétation du signal mais ne remettent pas en question son rôle dans l'apprentissage ni dans l'encodage d'une information de type RPE. ...
In this thesis work, we modelled the role of dopamine in learning and in the processes of action selection through its interaction with the basal ganglia. During the 90’s, the work of Schultz and colleagues has led to major progress in understanding the neural mechanisms underlying the influence of feedback on learning. The activity of dopaminergic neurons exhibited properties of the reward prediction error signal used in so-called Temporal Difference (TD) machine learning algorithms. Thus, DA has been thought to be the neural signal that help us to adapt our behavior. In the first part of my PhD, we analyze the information encoded by dopaminergic neurons recorded during a multi-choice task. In this purpose, we modeled the task and simulated different TD learning algorithms to quantitatively compare their ability to reproduce dopamine neurons activity. Our results show that the information carried out by dopamine neurons is only partly consistent with a reward prediction error and seems to be dissociated from behavioral adaptation.In the second part of my PhD, we study the effect of different levels of dopamine in a biologically plausible model of primates basal ganglia that considers existing connections often neglected in the literature. Indeed, most of current models of basal ganglia assume the existence of two segregated pathway: the direct pathway associated with reward and the indirect pathway associated with punishment. However, anatomical studies in primates revealed that these two pathways are not dissociated. We study the ability of such a model to reproduce beta oscillations observed in Parkinsonian and the differences in reward and punishment sensitivity, with high or low-level of dopamine.
... Particularly intriguingly for the present study, its relevance has been documented with respect to learning (Pessiglione, Seymour, Flandin, Dolan, & Frith, 2006), motivation (Depue & Collins, 1999), risk taking, and attention (Fiorillo, Tobler, & Schultz, 2003). Dopaminergic systems in the brain are involved in neural processes representing reward, salience (Smith, Li, Becker, & Kapur, 2006), and uncertainty (Fiorillo et al., 2003). Latent inhibition (Carson, Peterson, & Higgins, 2003;Kéri, 2011) and reward sensitivity (Bódi et al., 2009;Heilman, Nadeau, & Beversdorf, 2003) are possible links between dopamine and creativity at the level of basic cognitive processes. ...
Creativity can arise in some Parkinson's disease (PD) patients as a side effect of dopaminergic therapy. Schizotypal traits-subclinical traits resembling schizophrenia symptoms-can mediate some effects of dopaminergic drugs on cognition and neural activity. The goal of our study was to relate general intelligence, creative achievement and schizotypal traits to changes in performance on the Just Suppose test (from the Torrance battery), a task designed to measure verbal divergent thinking. Patients with PD and controls were examined at baseline, and at a follow-up session 12 weeks after pharmacotherapy had begun. Patients received dopamine agonist monotherapy (pramipexole and ropinirole). We observed significantly elevated positive schizotypy and impulsivity in PD at follow-up, while divergent thinking scores increased at trend level in the patient group. Linear regression analyses revealed that changes in various aspects of divergent thinking in PD were related to positive and disorganized schizotypy and lifetime creative achievement. The results highlight the relevance of schizotypal traits and lifetime creative achievement to the development of creativity in PD during dopaminergic therapy. The research draws attention to individual differences associated with a side effect of dopaminergic therapy in PD and also contributes to the understanding of the biological aspects of creativity.
... Aberrant salience processing is also involved in the disruption of central information processing, especially when the brain is required to discriminate between relevant and irrelevant stimuli (Smith et al., 2006;Winton-Brown et al., 2014). Under such conditions, increased mesolimbic dopamine activity enhances the salience of irrelevant stimuli, and as a consequence the organism often fails to differentiate between relevant and irrelevant information (Kapur, 2003;Smith et al., 2006). ...
... Aberrant salience processing is also involved in the disruption of central information processing, especially when the brain is required to discriminate between relevant and irrelevant stimuli (Smith et al., 2006;Winton-Brown et al., 2014). Under such conditions, increased mesolimbic dopamine activity enhances the salience of irrelevant stimuli, and as a consequence the organism often fails to differentiate between relevant and irrelevant information (Kapur, 2003;Smith et al., 2006). The essence of this phenomenon can be captured by a behavioral paradigm known as latent inhibition (LI), a model of associative learning in which non-reinforced pre-exposures to a to-be-conditioned stimulus (CS) retard subsequent conditioning between the same CS and the unconditioned stimulus (US) (Lubow and Moore, 1959;Lubow, 2005). ...
... An important implication is that the PYY 3-36 -induced disruption of LI does not simply reflect a general deficit in classical conditioning per se, but rather readily mirrors deficits in salience learning that normally regulate the expression of LI (Weiner, 2003;Lubow, 2005;Young et al., 2005;Nelson et al., 2011). While still hypothetical, these findings are compatible with a neuropsychological model, in which PYY 3-36 can enhance the salience of irrelevant stimuli through neurochemical processes involving increased mesolimbic dopaminergic activity (Kapur, 2003;Smith et al., 2006). ...
