Social Interaction

1. Introduction

Facial movements are one of humans' most salient embodied resources within social interaction. As facial gestures¹ in the sense of "utterance uses of visible action" (Kendon, 2004: 1f), their functions are centered around building intersubjective understanding and affective alignment in co-present interaction, as well as around organizing talk and interaction.

Even before the interactional aspect of interactants' facial movements became an object of study, it was commonsense knowledge that the human face is a central locus where emotions are shown and communicated. However, it was not until the evolutionary approach developed by Darwin (1872) that the moving human face was prominently put into the spotlight of science. Around 100 years after that, emotion psychology—with its most cited representative Paul Ekman—became the dominant field for investigating facial expressions conveying internal emotional states such as anger, surprise, or disgust (e.g., Ekman & Friesen, 1969; Ekman & Oster, 1979; Ekman, 1979). While their work emphasizes the emotion-indexical function, Ekman and colleagues also observe that around two thirds of all facial movements serve as conversational signals (Fridlund et al., 1987: 160–61, Ekman, 1999). The relevance of facial movements for and within social interaction has further been investigated by researchers working in the tradition of sociology and social psychology (e.g., Brunner, 1979; Kraut & Johnston, 1979; Fridlund & Russell, 2006). While those approaches form some kind of counter movement to emotion psychology in that they prioritize the impact of social situations over emotional states for the occurrence and specific manifestation of facial movements, communication theoretical and linguistic studies, on the other hand, emphasize the discourse-related functions and semiotic potentials of facial expressions when co-occurring with speech: As part of gesture-speech ensembles (e.g., Bavelas et al., 2014; Bavelas & Chovil, 2018), they have been found to convey referential content, for example, by pointing at something (see also Enfield, 2001, about lip-pointing) or by enacting imaginable faces (Bavelas et al., 2014: 18–19). Bavelas, Gerwing, and Healing further describe pragmatic-interactive functions of facial gestures, such as recipient-designing at turn-at-talk and signaling the modality of utterances, for example, marking ironic humor (Bavelas et al, 2014: 20; see also Aguert, 2022, from a psycho-linguistic perspective).

In linguistics and experimental phonetics, genuine linguistic functions of facial gestures have been put into focus, showing that they contribute to information structuring (e.g., Beskow et al., 2009; Flecha-Garcia, 2010; Ambrazaitis & House, 2017, for raised eyebrows as visual prosody) and may be indicative of utterance mode (e.g., Nota et al., 2021).

Studies within Multimodality Research/Multimodal Conversation Analysis (CA) (e.g., Stivers & Sidnell, 2005; Mondada, 2016, 2019) shed light on how facial gestures contribute to action formation/action recognition within naturally occurring talk-in-interaction. Some studies investigate one specific sequential environment and find that facial gestures play a prominent role in the multimodal accomplishment of conversational actions; others focus on one specific facial gesture and aim to identify its interactional functions. Since the contributions in this Special Issue (to be presented in Section 2) provide extensive literature reviews of both form- and function-based approaches, we will limit ourselves to presenting a few studies with the aim of broadly introducing the main strands and topics within multimodal CA research on facial gestures.

The seminal study by Goodwin and Goodwin (1986) on doing thinking during word searches provides an example of how the functionality of gaze and other facial movements can be at the forefront of analysis while capturing them in their embeddedness with speech. The authors observe frequent gaze withdrawals during talk perturbations and raise the issue of how a recipient might recognize that the speaker is not distracted but actively involved in a word search. In that respect, they identify cases in which a word search is displayed by means of a facial configuration alone. What Goodwin and Goodwin call thinking face involves pursed lips and eyebrow movements (Goodwin & Goodwin, 1986: 61). The authors thereby take first important steps towards investigating facial gestures with multimodal CA methods. Investigating argumentative activities in which children engage in processes of joint decision-making, Heller (2021) similarly addresses the aspect of displaying cognitive processes as a public practice by using facial gestures. Besides typical thinking postures, the wandering of the eyes and an "imaginative gaze" frequently occur as part of facial thinking displays. In temporal alignment "with lexical, syntactical, and/or morphological markers of epistemic modality, they [are] used to signal a thoughtful and tentative, independent yet cooperative, determined, affirmative or critical stance" (p. 18).

In a study by Kärkkäinen and Keisanen (2012), the raising of both eyebrows is observed as part of embodied practices to accomplish offerings (besides leaning forward and performing instrumental actions). While identifying different multimodal configurations of making offers, this study also hints at the fundamental methodological question of how essential the contribution of facial gestures (as well as the use of other verbal, vocal, or bodily resources) actually is in each case in order to constitute actions in an accountable way. Concerning this matter, Oloff (2018) finds conversational repair to be one sequential context where facial gestures can make a difference between related actions: When comparing different practices of open initiation of repair, she shows that a specific embodied practice involving a lifted eyebrows/head display is treated by participants as referring to troubles in hearing, whereas a freeze display indicates problems in understanding the linguistic format. Similarly focusing on repair initiation, Floyd et al. (2016) mention facial gestures as a potential resource in what the authors refer to as hold, a disruption of the interactional conduct in which "relatively dynamic movements are temporarily and meaningful[ly] held static" (p. 176). With their focus on temporal progression and disruption, Floyd et al. hint at the methodological potential of multimodal CA approaches to trace the exact beginning of facial gestures as well as to capture internal variability regarding their temporal unfolding.

When taking a specific form as the starting point for analysis, studies in multimodal CA analyze either a holistic facial configuration (such as the smile, involving mouth and eye movements) or single out specific "gesticulators" within interlocutors' faces (such as the raising of both eyebrows), which turn out to be salient resources from a participant's perspective. Clift (2021) focuses on the eye roll and describes it as the most salient element of an embodied practice for displaying dissent. She states that the eye roll has an ambivalent status as a conversational object in that it comments in a negative way on the action it targets but at the same time forges alliances with other addressees. Clift concludes that while sometimes not being visible to the producer of the target action, the eyeroll "constitutes a fleeting protest that does not disrupt the progressivity of the sequence" (p. 14).

The research group Anssi Peräkylä, Johanna Ruusuvuori, and Timo Kaukomaa (in particular, Kaukomaa et al., 2013, 2014, and 2015) focuses on different facial gestures—for example, smiles and frowns—and investigates their contribution to the local conduct of talk-in-interaction. They observe that smiles are frequently reciprocated by the recipient "on trust" and serve to establish a joint attitudinal change towards a positive stance regarding the upcoming strings of talk (Kaukomaa et al., 2013). Turn-opening frowns, on the other hand, foreshadow complications like disagreement and signal dispreference (Kaukomaa et al., 2014). Also focusing on the initial use of facial gestures, Groß and Dix (2023) investigate the protrusion of the lips preceding/in beginnings of responsive turns. They show that interlocutors may protrude their lips as an embodied hesitation marker that particularly marks an ongoing validation process regarding the terms of the initial action, thereby projecting a constrained confirmation. The aforementioned studies show that facial gestures may form embodied pre-beginning elements (Schegloff, 1996: 92–93; Ruusuvuori et al., 2021), and, as such, "may critically contribute to the communication of social actions in conversation by providing social action-specific visual information" (Nota et al., 2021: 1017).

While placing an emphasis on interactional processes, multimodal CA research has found its way "back" to the emotional side of the human face. A number of studies (e.g., Goodwin et al., 2012; Ruusuvuori & Peräkylä, 2009; Kaukomaa et al., 2013, 2014, 2015) have identified embodied ways in which emotions are organized as a situated practice involving intonation, (facial) gestures, and body posture. The authors show how interlocutors make available their affective stances in the moment-by-moment unfolding of social encounters and thereby demonstrate the conversational consequentiality of emotional facial expressions within the interlocking local conduct of several interlocutors (see also Ruusuvuori et al., 2021). Goodwin et al. (2012) specifically show how a classic disgust expression is "embedded within ongoing interaction within the lived social world" (p. 24). In that respect, the use of affective facial gestures has been proven to be essential for building intersubjectivity in terms of, on the one hand, shaping the interlocutors' understanding of a turn delivered (e.g., Ruusuvuori & Peräkylä, 2009; Peräkylä & Ruusuvuori, 2012) and on the other hand projecting a specific emotional quality of the upcoming conversational path while not yet producing a verbal turn (Kaukomaa et al., 2015).

Kaukomaa et al. (2015) demonstrate that not only a speaker's but also a recipient's facial expression impinges on the course of the talk, for instance, by shifting the emotional stance towards a proposition being uttered by the current speaker and proposing an alternative reading of what has been said. Analyzing storytelling but similarly focusing on aspects of recipiency and affiliation, Kupetz (2014) mentions different configurations of the face (e.g., the thinking face in the sense of Goodwin & Goodwin (1986), the shocked face with eyes widened) being involved in recipients' displays of empathy in the course of an emotional narrative. She describes facial gestures as a means of demonstrating early in a narrative that the emotional side of the reported events is recognized. While highlighting the frequent succession of response cry plus assessment, Kupetz describes facial gestures as an anticipatory means of recipients demonstrating that they are "with the teller" (p. 11). In a similar vein, Selting (2017) analyzes climaxes of humorous narratives as a locus of the display and management of affectivity, showing that interlocutors anticipate amusing climaxes and reciprocate the teller's facial gestures (such as smile or raised eyebrows) as part of affiliative responses.

Facial gestures may further contribute to the organization of talk and interaction: Focusing on storytellings, Peräkylä and Ruusuvuori (2012) identify the use of facial gestures at the end of narratives as a device for mobilizing adequate forms of recipiency when a listener's response is delayed. The authors discuss such forms of embodied pursuits with regard to their methodological implications, not only in terms of the necessity of including facial gestures more systematically in multimodal interaction analyses, but also regarding the question of how boundaries of turns-at-talk should be captured when taking into account the varying temporal configuration of speech with bodily resources. The authors conclude that a turn can be stretched by means of facial movements that exceed the verbal conduct of interaction.

This Special Issue takes up the emergent topic of facial gestures in talk-in-interaction within multimodal CA, offering a collection of papers by linguists and social scientists who study the social meanings and interpretations of facial gestures. It aims to broaden our understanding of the impact of facial gestures for interlocutors' mutual understanding and to tackle the question of how facial gestures shape local conversational conduct.

2. Overview of this Special Issue

The contributions gathered in this Special Issue introduce a plethora of new empirical developments in the study of facial movements as facial gestures within social interaction using the methodological and theoretical framework of multimodal CA. The papers demonstrate the general conversational significance and multifunctionality of facial gestures by focusing on:

1. different facial gestures, such as raising the eyebrows (Stolle & Pfeiffer, Wang & Li, Dix & Groß, Clift & Rossi, Heller, Schönfelder & Robbins), opening or closing the mouth and moving the lips (Katila), as a stand-alone element, or orchestrated with other verbal, vocal, and embodied resources, within
2. diverse activities and sequential contexts, for example, repair sequences (Stolle & Pfeiffer, Li & Wang), change-of-state moments (Dix & Groß), knowledge queries (Wang & Li), and transition spaces between turns (Clift & Rossi), embedded in
3. naturally occurring mundane as well as institutional talk, conducted in
4. different languages: German, English, Finnish, and Mandarin Chinese. Two of the papers also present
5. methodological developments (Dix on transcribing facial gestures, Katila on adding researchers' interpretations as a tool for studying embodied interaction). Therefore, the Special Issue offers an abundance of new findings on a territory that has seen little research within multimodal CA until now.

Xiaoyung Wang and Xiaoting Li as well as Sarah Stolle and Martin Pfeiffer focus on the use of eyebrows in the context of other-initiation of repair. Wang and Li describe two practices found in Chinese second language teaching. They show how teachers' eyebrow and head movements, either independent from or combined with partial verbal repeats, are designed for different types of problems of understanding. They point out the potential pedagogical value of these practices. Stolle and Pfeiffer focus on facial gestures as standalone repair initiations in everyday conversation between friends. They show how participants can convey information about the location of trouble through eyebrow movements and therefore signal specific types of problems. They also point out how facial movements as independent from verbal aspects of interaction entail the function of minimally disrupting the progressivity of talk, offering the respondent the possibility for self-repair.

Rebecca Clift and Giovanni Rossi investigate eyebrow raises used by the speaker in transition relevance place as a way of pursuing shared understanding of ongoing activity to guarantee progressivity of the conversation. They distinguish two different types of eyebrow raises that are interpreted as either challenging (eyebrows held high) or joking (flashing eyebrows quickly up and down). They refer to the presumption of shared epistemic stance invoked by the eyebrow raises. Carolin Dix and Alexandra Groß, on the other hand, focus on eyebrow raises of the recipients and show how eyebrow raises display a change of their epistemic state, with an eyebrow flash depicting a minor change, a news receipt, whereas holding eyebrows up depicts a more major change of state, showing surprise or astonishment. Vivien Heller, Nora Schönfelder and Denise Robbins also focus on eyebrow shifts accompanied by other subtle changes of the face and body, as well as prosody. They show how eyebrow furrowing combined with other modalities displays either a questioning or a critical stance in adolescents' interactions.

Julia Katila's paper focuses on experiencing face during intimate touch, on the interplay between communicative and felt aspects of face. She suggests that while engaging in intimate touch, participants not only display their inner feeling to their partner but also orient to their felt state. The author takes a stance where she criticizes the inner/outer dichotomy of the affective body versus external expressions of emotion, which is often adopted in multimodal CA. Katila links observable facial expressions to affective or cognitive states, suggesting that the visible focus on internal state is motivated by embodied experience but can also develop to entail a communicative purpose.

Another type of method development is presented by Carolin Dix in her paper on transcribing facial gestures. Dix combines Jeffersonian transcription conventions with the sign inventory of the International SignWriting Alphabet (ISWA). The idea is to create a standardized system of transcribing facial gestures, providing both comparability and adaptability while at the same time reducing interpretative transcriber comments within the transcripts.

On a general level, the papers exhibit closely related yet distinguishable ways of conceptualizing and analyzing facial gestures in interaction. For most of the papers, the movements and positions of the eyelids and the eyebrows are the focal area of face. Furthermore, the papers take a rather holistic and multimodal approach, showing linkages between the facial gestures, prosody, and (in Katila) also tactile and kinetic aspects of interaction.

In this way, the papers offer a starting point for deepening and intensifying research on other facial resources, especially the expressive areas around the mouth, and contributing to the growing interest of researchers within Multimodal Analysis in the complex interplay of verbal, vocal, and visual interactive resources.

In the analytic design of the papers, different aspects of interactional organization are treated as the "context" of the facial gestures. This context can be action—such as repair initiation (as in Stolle & Pfeiffer and Wang & Li)—or, more generally, first position action that invites a response (as in Clift and Rossi). The analytic context for the facial gesture can also be stance display (as in Dix & Groß and Heller) or display of (intimate) social relation (as in Katila). The timing of facial gestures is touched upon in many papers. Alongside the sequential timing (at what point in the unfolding of action sequences the gestures take place), the papers by Dix and Groß and Clift and Rossi address the communicative functions of the internal temporal structure of the gesture (in both cases, raised eyebrows) by scrutinizing their locus and temporal alignment with the emerging turn-at-talk as well as by tracing their intrinsic unfolding within whole multimodal gestalts.

On a more conceptual level, some intriguing questions are touched upon. While most of the papers focus on the communicative functions of facial gestures, Katila raises the question of linkages between communicative actions and lived experience, pointing out that the gestures she examines are linked to subjectively experienced emotions, which can also be perceived and understood by the researcher. In the light of Katila's contributions, facial gestures can be seen as the nexus between a phenomenological approach and an ethnomethodological approach. Such a dialogue between originally rather distant methodologies may open interesting new avenues and developments for the study of meaning-making in social interaction.

In terms of the theoretical contextualization of conversation analyses of facial gestures, it is notable that many papers go beyond the standard CA literature. References to Ekman and Darwin are there alongside the CA classics. However, rather than presenting their findings as contradictory to psychological and biological research on facial expressions, the authors of this Special Issue convey that the multimodal CA approach to facial gestures advances and elaborates the ideas presented in these earlier lines of research. Investigating facial gestures probably requires, or at least makes possible, the conceptual linkages between multimodal CA and other strands of research.

In terms of methods of analyzing facial gestures, the papers use video-based observational methods with detailed transcription and screenshots. While this type of research is well established, dealing with facial movements also widens the field for using new technologies—such as the Openface (Baltrusaitis et al., 2018) and Noldus Face Reader (Lewinski et al., 2014) software—which make possible automatic annotation of changes in facial gestures, even including other modalities, such as the use of space and other embodied interaction. Even if automatic annotation cannot replace the qualitative case-by-case analysis represented by the papers in this collection, it can make the building of collections more effective and offer new possibilities for quantitative elaborations of the qualitive analyses. Furthermore, specific mobile eye-tracking technology is available and has already been used for analyses in authentic environments (e.g., Stukenbrock, 2018). These technological developments may well widen our possibilities for focusing on gestalts consisting of different modalities and help to locate the ways in which these gestalts contribute to building joint activity projects and shared understanding in social interaction.

Acknowledgements

This special issue evolved out of two Workshops hosted at the University of Bayreuth in January 2020 and in October 2021; most of the papers are based on oral presentations given by the participants (see the conference reports by Alt, 2020, and Gäberlein, 2023). We want to express our gratitude and appreciation to all the contributors, to the reviewers of the papers, and the journal editors for their suggestions.

^{1 A note on terminology: Bavelas et al. (2014: 18) offer a useful differentiation which we will adapt to our purposes: While we use facial movement as a neutral umbrella term, facial gesture emphasizes its linguistic and conversational functions. Facial expression is used when it is highlighted that the face reveals the producer's emotions. Facial displays similarly signal underlying emotional states and cognitive processes, but the term stays agnostic about their existence and, rather, focuses on the respective interactional stance. Facial dynamics (such as the eye blink) are facial movements which primarily serve physiological functions. However, the latter as well as facial expressions may be used for conversational purposes and thus may be jointly oriented to by interlocutors. ↩}