Fig 2 - uploaded by Fernando Bravo
Content may be subject to copyright.
Source publication
This thesis explores the effects of music upon the cognitive processing of visual information. The objective is to address how alterations of specific aspects within the musical structure may influence the interpretation of visual scenarios.
Background is provided from film-sound theory, studies of the expressive capabilities of sound in film, theo...
Context in source publication
Context 1
... the tonal perspective, Paul Hindemith's work is especially noteworthy [15]. According to Hindemith, the overtone series system (see Figure 2) gives a complete proof of the natural basis of tonal relations. In general, as new intervals are intro- duced, the stability decreases and the two tones involved are considered more distant in their relation. ...
Citations
... Things are radically complicated because these parts continuously create dynamic interactions among them. Soundtracks have a pivotal role in building the working narrative of the film; indeed, they have been proved to substantially affect the parasocial interaction of the recipients with the characters on the screen (Ansani et al., 2020;Hoeckner et al., 2011), characters' moral judgments (Steffens, 2020) and personality traits (Ansani et al., 2020;Bravo, 2013), plot anticipations (Shevy, 2007), the perception of the environment (Ansani et al., 2020;Yamasaki et al., 2015), time perception (Ansani et al., 2021), and, most notably to the present of the current study, the subsequent remembering of the scene (Boltz, 2001;Boltz et al., 1991). ...
Several studies have employed music to affect various tasks through mood induction procedures. In this perspective, music's emotional content coherently affects the listeners' mood, which, in turn, affects performance. On the contrary, in film music cognition, schema theory suggests that music adds semantic information that interacts with the viewers' previous knowledge and influences visual information processing. As in this interpretation the viewers' mood is not deeply considered, it is not clear the extent to which music effects are also due to its power of affecting the viewers' mood or rather a mere cognitive priming-like influence. An online experiment (N = 169) on how music biases the recognition memory of a scene was built comparing semantic and emotional effects of soundtracks differing in valence (happy vs scary) during a recognition task. The results show that 1) music affected the viewers' mood coherently with its emotional valence, 2) music led to falsely recognise unseen objects as truly present when coherent with the soundtrack valence; and 3) the effect of music on the biased remembering was not mediated by the viewers' mood, thus suggesting a strong interpretation of the schema theory in film music processing. Finally, a methodological reflection is provided on the issue of the manipulation check in experiments that employ musical stimuli to assess their influence on cognition.
... While these systems perform well in arranging notes and chords, the emotions that they evoke cannot be compared against a tangible reference. Visual perception, on the other hand, can play a significant role in composing and evaluating music [21]- [23]. Hence, we focus on some related work on image to music conversion. ...
We propose a method for generating music from a given image through three stages of translation, from image to caption, caption to lyrics, and lyrics to instrumental music, which forms the content to be combined with a given style. We train our proposed model, which we call BGT (BLIP-GPT2-TeleMelody), on two open-source datasets, one containing over 200,000 labeled images, and another containing more than 175,000 MIDI music files. In contrast with pixel level translation, our system retains the semantics of the input image. We verify our claim through a user study in which participants were asked to match input images with generated music without access to the intermediate caption and lyrics. The results show that, while the matching rate among participants with little music expertise is essentially random, the rate among those with composition experience is significantly high, which strongly indicates that some semantic content of the input image is retained in the generated music. The source code is avaliable at https://github.com/BILLXZY1215/BGT-G2G.
... it should have been created at least two more conditions: one with a solo/anxious piece and another with an orchestral/melancholic one searching for genre or quantity of instruments effects. Undoubtedly, several other finer-grained manipulations could have been performed; a manipulation could have involved the mode of the same melody: major vs. minor, as in Ziv and Goshen (2006), or the degree of dissonance, as in Bravo (2013). Nevertheless, as a first attempt for a multivariate perspective (i.e., a context wherein the attempt was that of eliciting the greatest differences on a wide number of variables), the choice fell on two tracks that were radically dissimilar so to be able to discover also moderate effects with a relatively small sample. ...
... A considerable number of studies have proved that music can bias moral harshness with respect to moral judgement (Ansani et al., 2017(Ansani et al., -2019Seidel & Prinz, 2012Ziv et al., 2012), even in the audiovisual domain (Steffens, 2020). As already mentioned in the section on the soundtrack's functions ( § 4.3), different music pieces have the extraordinary power of shaping our intuitions about the personality traits of the characters (Ansani et al., 2020), their goals, their morality (Steffens, 2020), they can modify the viewers' attitude toward the characters on the screen (Bravo, 2013;Hoeckner et al., 2011). Such music potential might have tangible consequences in a considerable spectrum of domains that need to be investigated thoroughly, for instance, in dating apps and all those similar media environments wherein someone introduces themselves through presentational videos. ...
In the last decades, an increasing number of psychological studies tackled numerous phenomena related to the influence that film music exerts on the perception of audiovisuals. Soundtracks proved to be effective in shaping the viewers' interpretations, attitude toward the characters, plot anticipations, recall of the scenes, and other processes. However, the most recent review on the issue (Herget, 2019) indicates a few criticalities: First, due to the lack of an interconnected research discourse, several of the existent studies stand side by side rather than building on each other. Secondly, in most cases, established instruments were not used for the assessments, thus jeopardizing the external validity. As a third key point, the number of ecologically valid studies is limited and needs to be increased. Lastly, the methodological necessity exists to decrease the complexity of certain experimental paradigms and encourage between-subjects designs that could avoid the risk of automatically drawing attention to musical manipulation. In the present research, four studies are presented on a variety of psychological constructs, processes, and mechanisms that are influenced by film music: impressions of the characters’ personality, plot anticipations, and environment perception (Study 1a-1b); gaze direction, gaze dispersion, and pupillometry (Study 1b); empathy toward the filmed characters (Study 1a-1b-2); affective state attributed to on-screen characters, affective state of the viewers (Study 2); time perception (Study 3); and recall (Study 4).
An effort was made to employ validated measurement tools whenever possible, together with easily digestible and entertaining experimental tasks, without compromising on experimental control and data quality. Aiming at a high ecological validity, all the studies (except Study 1b) implicated an online administration; in doing so, the viewers watched the videos directly from home, through their smartphones, tablets, and laptops, as if they were watching everyday YouTube videos or Netflix series.
The general finding is that the influence of music is truly pervasive; namely, virtually every analyzed variable was affected by it predictably. Thanks to the music’s tonal and expressive cues, the viewers manage to create a network of coherent and interconnected inferences, which end up constituting their interpretation of the scene.
The results are discussed in terms of diverse theoretical frameworks depending on the constructs at hand. Particular attention is lastly given to the future paths that the research could go down to fill a wide number of unsolved theoretical gaps.
In conclusion, a coda about the ethical relevance of such research is provided.
... It is clear now that there is a relationship among visual stimuli, audio stimuli and emotion, and the fact that together they're able to elicit even stronger emotions from the user [1]. Given that, an important direction for this area of research is to find a method that is able to establish a relation between art of a single modality, scene, into art of a different modality, audio. ...
... Each video is originally annotated with valence values ranging from -1 to 1 in regular time intervals. In our work, we consider valence scores below 0.00 to be negative (0) and above that threshold to be a positive emotion (1). The dataset is divided in such a way as to guarantee disjoint train and test datasets, presented in the top and bottom half of the table respectively, so as to prevent overfitting of the model. ...
... where W (1) x is used to obtain the linear combination between f (1) i and c (2) i . ...
This paper presents Scene2Wav, a novel deep convolutional model proposed to handle the task of music generation from emotionally annotated video. This is important because when paired with the appropriate audio, the resulting music video is able to enhance the emotional effect it has on viewers. The challenge lies in transforming the video to audio domain and generating music. Our proposed encoder Scene2Wav uses a convolutional sequence encoder to embed dynamic emotional visual features from low-level features in the colour space, namely Hue, Saturation and Value. The decoder Scene2Wav is a proposed conditional SampleRNN which uses that emotional visual feature embedding as condition to generate novel emotional music. The entire model is fine-tuned in an end-to-end training fashion to generate a music signal evoking the intended emotional response from the listener. By taking into consideration the emotional and generative aspect of it, this work is a significant contribution to the field of Human-Computer Interaction. It is also a stepping stone towards the creation of an AI movie and/or drama director, which is able to automatically generate appropriate music for trailers and movies. Based on experimental results, this model can effectively generate music that is preferred to the user when compared to the baseline model and able to evoke correct emotions.
... In his fascinating work, Bravo (2013) studied the effect of tonal dissonance on interpretations of the emotional content of an animated short film. He hypothesized that in the same film sequence, different levels of tonal dissonance would elicit different interpretations and expectations about the emotional content of the movie scene. ...
... Strictly speaking, we cannot be sure whether such differences could have affected the interpretations; we should have at least created two more conditions: one with a solo/anxious piece and another with an orchestral/melancholic one searching for genre or quantity of instruments effects. Undoubtedly, several other finer-grained manipulations could have been performed; we could have manipulated the mode only of the same melody: major vs. minor, as in Ziv and Goshen (2006), or degree of dissonance, as in Bravo (2013). Nevertheless, as a first attempt, we opted for two tracks that were radically dissimilar so to be able to discover also moderate effects with a relatively small sample. ...
This article presents two studies that deepen the theme of how soundtracks shape our interpretation of audiovisuals. Embracing a multivariate perspective, Study 1 (N = 118) demonstrated, through an online between-subjects experiment, that two different music scores (melancholic vs. anxious) deeply affected the interpretations of an unknown movie scene in terms of empathy felt toward the main character, impressions of his personality, plot anticipations, and perception of the environment of the scene. With the melancholic music, participants felt empathy toward the character, viewing him as more agreeable and introverted, more oriented to memories than to decisions, while perceiving the environment as cozier. An almost opposite pattern emerged with the anxious music. In Study 2 (N = 92), we replicated the experiment in our lab but with the addition of eye-tracking and pupillometric measurements. Results of Study 1 were largely replicated; moreover, we proved that the anxious score, by increasing the participants’ vigilance and state of alert (wider pupil dilation), favored greater attention to minor details, as in the case of another character who was very hard to be noticed (more time spent on his figure). Results highlight the pervasive nature of the influence of music within the process of interpretation of visual scenes.
... A lo largo de los últimos años han sido numerosos los estudios que han avanzado sobre la comprensión acerca de cómo percibe la música el ser humano (Sloboda, 2012;Storr, 2012), como la procesa nuestro cuerpo (Koelsch y Jäncke, 2015;Koelsch, 2015), y que utilidad tiene esto en la musicoterapia (Koelsch, Offermanns y Franzke, 2010), en el arte audiovisual (Bravo, 2012), en eventos sociales (Storr, 2012), entre otros. Las investigaciones acerca de qué factores personales afectan esta percepción parecen estar centradas, en su mayor parte, en el nivel de instrucción musical (Hargraves, 1986), en preferencias musicales (Hargraves, 1986;Pereira, Teixeira, Figueiredo, Xavier, Castro, y Brattico, 2011;Rentfrow y Gosling, 2003), en diferencias culturales (Hargraves, 1896;Storr, 2012; Hodges y Sebald 2015) y en la familiaridad que uno tiene respecto de un género musical (Hargraves, 1986;Pereira y col., 2011). ...
... A lo largo de los últimos años han sido numerosos los estudios que han avanzado sobre la comprensión acerca de cómo percibe la música el ser humano (Sloboda, 2012;Storr, 2012), como la procesa nuestro cuerpo (Koelsch y Jäncke, 2015;Koelsch, 2015), y que utilidad tiene esto en la musicoterapia (Koelsch, Offermanns y Franzke, 2010), en el arte audiovisual (Bravo, 2012), en eventos sociales (Storr, 2012), entre otros. Las investigaciones acerca de qué factores personales afectan esta percepción parecen estar centradas, en su mayor parte, en el nivel de instrucción musical (Hargraves, 1986), en preferencias musicales (Hargraves, 1986;Pereira, Teixeira, Figueiredo, Xavier, Castro, y Brattico, 2011;Rentfrow y Gosling, 2003), en diferencias culturales (Hargraves, 1896;Storr, 2012; Hodges y Sebald 2015) y en la familiaridad que uno tiene respecto de un género musical (Hargraves, 1986;Pereira y col., 2011). ...
Se estudió la relación entre la apercepción musical y los cinco grandes rasgos de personalidad con 44 adultos jóvenes entre 18 y 27 años. En un encuentro grupal, los participantes debían escuchar 4 piezas musicales y narrar un breve relato acerca de qué podrían tratar las canciones. Asimismo debieron marcar que emociones percibían en la música, con que intensidad, que emoción consideraban principal y cuanto les agradaba la canción. Los resultados evidenciaron numerosas correlaciones como por ejemplo: extroversión y una menor apercepción emocional de los estímulos presentados; amabilidad y una menor descripción de ambientes en las narraciones; responsabilidad y una menor apercepción de alegría en las canciones, menor presencia de tachado y mayor elección de grupos como protagonistas en las historias; neuroticismo y una menor apercepción de anticipación, y una mayor elección de la tristeza como la emoción principal transmitida por los estímulos; apertura y una mayor cantidad de referencia a sí mismo en las narraciones. Se espera que el presente estudio contribuya a una mejor comprensión de como las personas perciben la música, que factores influyen en la interpretación de esta percepción, y la utilidad que puede tener un estímulo musical para el psicodiagnóstico.
... The degree of consonance/dissonance has been found to influence liking and the perception of emotional content not only in music (Blood et al., 1999;Koelsch et al., 2006) but also in audiovisual contexts (Cohen, 2001;Boltz, 2001;Bravo, 2012Bravo, , 2014. Connotations ascribed to consonant and dissonant sounds have been frequently employed in film sound to shape the emotional comprehension of visual narratives, mainly through influencing the attribution of mental states (emotions, thoughts or intentions) to the characters depicted onscreen (Cohen, 2001;Boltz, 2001;Bravo, 2012Bravo, , 2014. ...
... The degree of consonance/dissonance has been found to influence liking and the perception of emotional content not only in music (Blood et al., 1999;Koelsch et al., 2006) but also in audiovisual contexts (Cohen, 2001;Boltz, 2001;Bravo, 2012Bravo, , 2014. Connotations ascribed to consonant and dissonant sounds have been frequently employed in film sound to shape the emotional comprehension of visual narratives, mainly through influencing the attribution of mental states (emotions, thoughts or intentions) to the characters depicted onscreen (Cohen, 2001;Boltz, 2001;Bravo, 2012Bravo, , 2014. Notably, to our best knowledge, no empirical work has yet examined the impact of consonance/dissonance on the neural substrates underlying valence inference processes during mental state attribution. ...
We frequently infer others’ intentions based on non-verbal auditory cues. Although the brain underpinnings of social cognition have been extensively studied, no empirical work has yet examined the impact of musical structure manipulation on the neural processing of emotional valence during mental state inferences. We used a novel sound-based theory-of-mind paradigm in which participants categorized stimuli of different sensory dissonance level in terms of positive/negative valence. Whilst consistent with previous studies which propose facilitated encoding of consonances, our results demonstrated that distinct levels of consonance/dissonance elicited differential influences on the right angular gyrus, an area implicated in mental state attribution and attention reorienting processes. Functional and effective connectivity analyses further showed that consonances modulated a specific inhibitory interaction from associative memory to mental state attribution substrates. Following evidence suggesting that individuals with autism may process social affective cues differently, we assessed the relationship between participants’ task performance and self-reported autistic traits in clinically typical adults. Higher scores on the social cognition scales of the AQ were associated with deficits in recognising positive valence in consonant sound cues. These findings are discussed with respect to Bayesian perspectives on autistic perception, which highlight a functional failure to optimize precision in relation to prior beliefs.
... Mapping a systematic relationship between musical structural features and certain aspects of the audiovisual experience might be crucial for potential applications concerned with clinical psychology and affective neuroscience. The present study builds on a previous empirical work by the author [12] that showed strong evidence in support of the effect of tonal dissonance level on interpretations regarding the emotional content of visual information. That experiment stimulated the investigation of tonal and post-tonal interval theory, and the parallel design of interactive multimedia tools to examine these processes in a strictly controlled empirical setting. ...
The ability of music to influence the emotional interpretation of visual contexts has been supported in several psychological studies. However, we still lack a significant body of empirical studies examining the ways in which specific structural characteristics of music may alter the affective processing of visual information. The present study suggests a way to use algorithmically generated music to assess the effect of sensory dissonance on the emotional judgment of a visual scene. This was examined by presenting participants with the same abstract animated film paired with consonant, dissonant and no music. The level of sensory dissonance was controlled in this experiment by employing different interval sets for the two contrasting background music conditions. Immediately after viewing the clip, participants were asked to complete a series of bipolar adjective ratings representing the three connotative dimensions (valence, activity and potency). Results revealed that relative to the control group of no music, consonant background music significantly biased the affective impact by guiding participants toward positive valence ratings. This finding is discussed in terms of interval content theory within the general perspective of post-tonal music theory and David Temperley’s probabilistic framework (model of tonalness).
During the last twenty years, Film Theory has witnessed a growing interest in both the essay Das Unheimliche (Freud, 1919), and in the tendency to substitute the psychoanalytic method for a more operational discipline, as is the case with cognitive psychology. This dissertation aims to discover the reasons for the emergence of these trends that take us back to the early days of cinema. The quoted Freud’s essay and the first book on Film Theory, The Photoplay: A Psychological Study (Münsterberg, 1916), were published almost a hundred years ago, and their authors are respectively the father of psychoanalysis and an eminent German-born American psychologist. Our title establishes a relationship between film psychology and the uncanny phenomenon. To simplify the above, this paper sets out to find out if a film director can deliberately incite the feeling of the uncanny in its viewers, using a series of thematic factors that Freud specified in his essay Das Unheimliche. To this general hypothesis, we have added a second goal, which was raised in the same text: checking if [film] fiction creates new possibilities for the uncanny, which cannot exist in real life. The word “film” is given in brackets as Freud only referred to literature and the medical histories of their patients when he wrote his essay...
