Social Interaction

1. VMI in the context of COVID-19 and a “new normal”

The COVID-19 pandemic has led to a huge increase in the use of video-mediated formats in a variety of settings, making it a ubiquitous phenomenon in everyday life. Video-Mediated Interaction (VMI) is generally understood as interaction conducted in and through a specific type of technology (e.g. Skype, Teams, Zoom and the like) that enables synchronous communication via a video link. This enables participants to have mutual access to sound and image in real time, thereby producing a simulacra of “unmediated” face-to-face interaction (Rintel, 2013b; Mlynář et al., 2018). Interacting through video-mediated technologies is not new in itself, but the interactional format has suddenly transformed from a specialised form used primarily in international, geographically dispersed teams, remote healthcare settings and progressive institutions, to an omnipresent, ordinary form of interaction, which is imperative for the accomplishment of interactions that require more modalities than just voice, as in telephone conversations. It is likely that the pervasiveness of working from home and the organisational readiness to use video-mediated forms will continue, resulting in a permanent new normal in which people will interact via video regularly and on a massive scale. As institutions are constituted in and through their communicative practices (Cooren, 2014), research into VMI is thus becoming more important than ever. VMI is used in all kinds of situations, but this special issue focuses specifically on two different kinds of institutional settings: organisations and healthcare environments. In order to provide insights about these settings, the articles in this issue apply an EMCA approach.

2. An EMCA, video-based approach to the study of video-mediated interaction

There are indisputably many ways to study VMI and generate findings. While the dominant strand of research focuses on knowledge production based on surveys, interviews, observations and statistical analysis or more technical issues (e.g. Terplan & Morreale, 2018), the articles in this special issue apply an in-depth methodology based on video recordings of people’s naturally unfolding interactions. The analyses of these recordings were not conducted with the aim of exploring the technical equipment as objects or as software, nor with the aim of providing developers with insights into design processes per se. Likewise, they were not conducted with the aim of quantification, but with the aim of providing deep insights into the sequential organisation of unfolding actions produced by participants as they strive to make sense in situ.

The articles in this issue are based on ethnomethodology and conversation analysis (EMCA). Ethnomethodology grew out of the work of Garfinkel in the 1960s, with a focus on the social organisation, the moral orders, the reflexivity, the accountability and the methods by which people (members of society), go about doing what they are doing (Garfinkel, 1967). The question for Garfinkel was simply: what is the thisness, the haecceity, of any current activity, that is emergingly achieved in and through members of the public’s production of actions (Garfinkel, 2002)? Conversation analysis grew out of ethnomethodology, primarily through early collaborations with Sacks (Garfinkel & Sacks, 1970), but it soon became a more specialised approach to the study of sequential organisations, preference organisation and action formation in talk-in-interaction (Sacks et al., 1974; Schegloff, 2007). Whereas “pure” ethnomethodological studies apply different methods for collecting data (e.g. observations and autoethnographies), researchers working more in line with CA insist on using recordings of naturally organised activities. In the 2000s, small, affordable digital cameras became more prevalent, and thus data collection is now usually conducted using video ethnographic methodology (Heath et al., 2010; B. Due, 2017).

Video-mediated interactions entail methodological challenges related to the fact that the participants’ interaction is fractured – i.e. they are not located in the same physical place. This gives rise to several data-collection issues, including a) setting up cameras at (several) remote, geographically dispersed locations and ensuring they are recording simultaneously; b) using mobile video recorders in combination with screen recordings, and synchronising them; and c) transcribing interactions, given that the sound and the delays may differ significantly between the two or more locations. However, despite these methodological challenges, EMCA methodology based on video recordings of VMI has generated novel findings.

Video recordings of natural interactions enable detailed analysis in situ of the smallest multimodal signs that make up practices and activities. For most publications in this tradition, it is not only the study of talk, but also embodied practices and the use of objects in space that are now usual focus areas (Nevile, 2015; Goodwin, 2018; Mondada, 2019a). Studying video-mediated institutional encounters from an EMCA perspective entails making it evident, through detailed analysis of empirical data, how activities, like for instance being updated about a project in a business meeting or examining a patient for symptoms, are accomplished through the participants’ situated practices, which are produced in and through embodied talk-in-interaction.

To conduct analyses, and to show precisely how the thisness of particular practices is accomplished in situ, the articles in this issue, building on EMCA, provide detailed transcripts embedded in the main text. A key analytical dogma in this tradition is not to make an etic analysis from the researcher’s perspective, aimed at fitting data into theory, but to make claims based on showing through data just how the participants themselves makes sense of what is going on as it is emergingly enacted. This approach therefore refuses to impose general theoretical pre-understandings about the empirical world – or about micro-macro or subject-object dichotomies (Hilbert, 1990) – but insists, on the contrary, that research most be conducted through an emic, bottom-up approach, applying a method of “unmotivated looking” (Psathas, 1995) to secure a solid grounding of empirical claims in the data from the members’ perspective. An overview of EMCA methodology and themes can be found in, among others, Have, 2007; Hutchby, 2008; Llewellyn & Hindmarsh, 2011; Sidnell & Stivers, 2012; and Raymond et al., 2017.

3. Video-mediation in institutional settings

EMCA video-based research has been influential in a number of areas of specific relevance for this special issue, first of all with regards to applications in institutional contexts, or practices or work (Garfinkel et al., 1981) related to Science and Technology Studies (Lynch, 1993) and Human Computer Interaction and Computer Supported Cooperative Work (Suchman, 2007; Dourish & Button, 1998).

VMI is based on a specific form of technology that belongs to the bigger family of Information and Communications Technologies (ICT), i.e. what used to be called telecommunications or, as Hutchby (2001) named it from a CA perspective, “technologies for communication”. These terms more broadly stress the role of unified communication, which involves the integration of different technologies in the pursuit of communicating. ICT refers to any technology that can store, retrieve, manipulate, transmit or receive information electronically, in a digital form. Therefore, by insisting on using the term Video-Mediated Interaction (VMI), our emphasis is not on the technological systems themselves, but on the human interaction that happens to be video-mediated.

According to Mlynář et al. (2018, p. 74), there was a first wave of research on VMI in the early 1990s (e.g. Raudaskoski, 1999), but the technology did not really take off at that time. The arrival of new and more intuitive products led to a resurgence of research interest in the early 2010s. Since then, literature has covered business meetings (e.g. Nielsen, 2019), classroom interaction (e.g. Hjulstad, 2016), courtrooms (e.g. Licoppe, 2015b), health settings (e.g. Stommel et al., 2019), surgery (e.g. Mondada, 2003), public service encounters (e.g. Due et al., 2019) and more experimental settings (e.g. Luff et al., 2003), to name just a few examples. In this introduction, we do not aim to provide a broad overview of the state-of-the-art of VMI research. Comprehensive reviews of the literature have already been provided by Arminen et al. (2016), Mlynář et al. (2018) and Lange (2020). However, we do wish to provide an overview of an evolving and cumulative terminology as it specifically relates to the special characteristics of VMI, as this has not been done before. This follows in the next section. First, a note on the specificities of institutional interaction.

In this special issue, our focus is not on VMI in general, but on practices in institutional encounters. Institutional interaction has special characteristics that differ from those of ordinary interaction. However, precisely what constitutes, for example specialised turn-taking systems or the preference for specific types of action formations, must be shown to be relevant in each case. Contrary to “bucket theory”, in which the context is considered to determine and structure the action, the institutional aspect of institutional interaction is, from an EMCA perspective, continuously enacted by the participants in situ (Heritage & Clayman, 2010). The institutional context is an unfolding project and a product of the participants’ actions, in which each action is both context-shaped and context-renewing (Heritage, 1984).

Drew and Heritage (1992) showed how six domains of interactional phenomena characterise institutional interaction: specific turn-taking organisation; the specific overall structural organisation of the interaction; specific sequence organisation; specific turn design; specific lexical choice; and specific epistemological and other forms of asymmetry. However, precisely what is specific is very different depending on the nature of the interaction, e.g. a business meeting or a healthcare consultation. Nonetheless, it is a key observation that institutional interactions often have a goal orientation that is embedded in the interaction, with regard to e.g. doing the business meeting and doing the consultation – and in accomplishing these goal-oriented activities, specific roles and identities are produced and reproduced in situ (ibid.)

4. Accumulated findings in EMCA studies of VMI: Outline of key terminologies

There are obvious differences between face-to-face (F2F) interaction and video-mediated interaction. However, we do not wish to suggest the need for or primacy of carrying out comparisons between F2F and VMI, as any encounter has its own intrinsic organisation, which it is relevant to study in its own right. In each case, researchers have to show precisely how any video-mediation is oriented to or otherwise made relevant for the unfolding interaction (Dourish et al., 1996).

Nevertheless, over the years, EMCA papers have generated accumulated insights and terminologies specifically related to VMI, and as these can inform theory-building, it is relevant to highlight them. The following sections cover some of these terms.

In this special issue, we research practices that are represented through three connected terms: institutional, video-mediated and interaction. While we previously briefly touched upon how an EMCA approach deals with the notion of institutionality as a social accomplishment, and with interaction as multimodal practices within sequences, we now turn to the final key term: mediation. Video-mediated Interaction (VMI) obviously occurs in and through technology. The technology is a concrete medium, an artefact, and as such it has specific features that enable and restrict interaction in specific ways. Each medium, from the telephone, the internet and various video systems to more advanced VR-systems, mediates the interactional flow in specific ways.

However, in much of the mainstream literature on mediation, there is a tendency to regard mediums and interactions as separable and independent components that affect each other. Within EMCA, this way of theorising is termed a “bucket theory” of context (Arminen, 2005; Heritage & Clayman, 2010), whereby the technological medium has a deterministic role in the interaction. EMCA, on the contrary, requires empirical studies to show how technologies and media can be shown to be both relevant and consequential with respect to the sequential organisation of interaction (Garfinkel & Sacks, 1970; Suchman, 2007). In the most basic forms, all interaction is mediated in some way, e.g. by language, bodies, objects, culture, history, technologies and so on (Gallagher & Zahavi, 2020). There is no unmediated interaction with which the mediated can be contrasted.

Thus, an EMCA approach to Video-Mediated Interaction is not based on the dualistic separation of mediation and non-mediation, but on the dogma that in each case it must be shown how the “mediating technologies” accountably shape the available or observed interaction practices. Specifically, mediation refers to, as described by Arminen et al. (2016), “the way the particular organization and unfolding of activities in definite material settings might constrain or enable or even ‘afford’ the production of particular forms of accountable responses and shape the criteria to assess their relevance. The difference between, say, a face-to-face interaction and a video call is not that one is unmediated and the other mediated; it is rather to be found in the way the production of particular sequences may be accomplished, enabled, constrained, or inhibited” (Arminen et al., 2016, p. 293).

The term affordance, as coined by J. J. Gibson, refers to a fit between abilities and the environment, or objects in the environment, which enables specific activities and constrains others (Gibson, 1979). In the tradition from Hutchby (2001), communicative affordance describes the specific technological shaping of social interaction as it unfolds in situ, and in particular ways enables and restricts how participants perceive what happens in interaction. One of the specificities concerning VMI is that the interactional situation occurs within fractured ecologies, i.e. “fractured from the environment in which it is produced and from the environment in which it is received” (Luff et al., 2003, p. 55). In some cases, participants are seen to fit their interaction into the current affordances of the mediating technology, thereby adjusting to the specificities of the fracturedness (e.g. Luff et al., 2016; this issue: Stommel et al., 2020). In other cases, participants employ novel practices to deal with current limitations and thus change aspects of the setting and/or the technology in order to achieve a current activity (e.g. Hjulstad, 2016; this issue: Bowden & Svahn, 2020; Due & Lange, 2020; see also Due, forth.a).

The opening of the interaction is a key topic in EMCA research on VMI. Opening a conversation is orderly achieved in social interaction, and has been a subject since the very beginning of conversation analysis (Schegloff & Sacks, 1973). Video-mediated calls differ in some respects from the “canonical” organisation of openings in phone conversations (Schegloff, 1968). First, video-mediated openings seem to occasion new kinds of “pre-openings” (Mondada, 2015) involving different kinds of action formation and adjacency pairs than in F2F and telephone conversations (e.g. Licoppe, 2015a). Second, participants in video-mediated communication display an orientation towards the technology and the setup, typically through pre-expansion sequences (Mondada, 2009; Licoppe & Dumoulin, 2010). Comments such as “Can you hear me?” and “Can you see me?” are frequently produced in conjunction with orientations towards the technological equipment and setup, and can be produced before more formalised greetings. Summons may be occasioned by the technological affordances, rather than verbalised accounts (Licoppe, 2012), and the more formal opening phase of the institutional video-mediated meeting may be delayed or even omitted, as more informal pre-openings, involving displays of aural and visual appearance, perform this function (Licoppe and Dumoulin, 2010).

Third, given that the possibility to talk and to see one another is temporally established, unlike phone conversations, both the caller and the call recipients in VMI may speak first. Moreover, when mutual access is established in steps, for instance through preliminary instant messaging, then audio connection, then the video connection, and finally some additional work to produce a “talking heads” configuration (Licoppe & Morel, 2012), openings may unfold as a succession of step-like “appearances”, in which each step marks a recognisable increment in the possibilities for achieving mutual orientation. Participants treat such appearances as noticeable and as sequentially implicative first pair parts, which project some subsequent action, and do so under the form of a responsive move, oriented to the particulars of the appearance, giving rise to protracted opening sequences (Licoppe, 2017) and occasionally multiple greeting sequences (Licoppe, 2020). Such appearances may be used as resources to achieve particular interactional ends, as in the case of large greetings in calls between migrant parents and young children raised by grandparents (Gan et al., 2020).

People always interact within some kind of participation framework or contextual configuration in space (Goodwin, 2007). While participants in VMI are physically separated from each other, each participant is still physically situated within a local environment. Most video-mediating technologies are designed with a small camera on the top of the device, which primarily captures the speaker’s head. Thus, practices have emerged over the years that relate to a talking head configuration (Licoppe & Morel, 2012) as default. This configuration seems to be a strong norm that relates to some of the most basic aspects of human sociality and the importance of the face (Goffman, 1967; Scheflen, 1964; Kendon, 1975; Goodwin, 1979). Even in situations where the technology affords mobility, e.g. when using a mobile phone (Morel & Licoppe, 2010) or in interactions with a telepresence robot (this issue: Nielsen, 2020; Due, forth. a, b), the participants seem to orient to a talking head configuration.

The articulation of visuality and participation frameworks becomes significantly more complex with multi-party video-mediated communication. In these cases, the requirement that the full faces of the speaker and the ratified hearer (if any) should be visible and oriented to the camera must be complemented by other normative considerations in order to facilitate accountable video framings. The participants treat the participants who are visible on screen as potentially relevant in some way to the ongoing interaction (Licoppe, 2017). Video framings of a VMI encounter are accountable as “glosses” of the situation (Sacks and Garfinkel, 1970), which are produced and understood as “visual formulations” of the participation frame. Having participants appear or disappear from screen, either through embodied micromobility of their own, or through distinctive camera work (such as moving the camera, or zooming in and out), may be oriented to as significant actions – in the latter case, as “camera actions” (Licoppe, 2015b).

There are many versions of this configuration, some of which involve other parts of the body being brought into the frame. Just as in face-to-face interaction, participants in VMI use their body as a resource in interaction (Hazel et al., 2014). While most of the body is hidden from the receiver, the arms and upper torso are often available, which enables a range of pointing and depicting practices – both produced by the speaker towards something in their close environment, but also towards their screen, with – what we can gloss as - a “fractured indexicality”. That is, pointing towards something in the remote place, which is practically impossible (Due et al., 2019). This phenomenon of dislocated pointing has also been called the Mona Lisa effect (Rogers et al., 2003; Luff et al., 2016).

Other fractured problems relate to gaze and “eye contact”. In early publications, Heath and Luff noted that multimodal actions were minimised in VMI, and upgrades were required to secure recipiency (Heath & Luff, 1993). One key issue, also identified at an early stage by, e.g. Fornel (1996) and Dourish et al. (1996) and later studied by Nielsen (2014, 2019) on contemporary devices, is the basic problem of achieving eye-to-eye gaze: when one is looking at a person on the screen, the gaze is directed towards the represented person on the screen. However, most cameras capture from a different angle (e.g. from the top of device), and therefore each participant needs to choose whether to look at the camera (seemingly looking at the co-participant) or to look at the person as represented on screen.

What is obvious and distinct about VMI is the fact that participants interact in and through technology and video-mediated representations. However, if and how this is relevant cannot be described beforehand but must be studied in detail. EMCA research has shown how the technology is often made relevant in contexts of repairing sequences, where there is some kind of technical issue to be fixed or at least accounted for in interaction. Technological disruptions (Mlynář et al., 2018, p. 77) are a possibility to which participants are seen to routinely orient through sequences of verification (Mondada, 2007) and by monitoring the functioning of the technology on the screen image (Olbertz-Siitonen, 2015). Specifically, research into delays in transmission suggests that co-presence in communication can be difficult to achieve, which can then disrupt the turn-taking system (Ruhleder & Jordan, 2001). In relation to this, Rintel has focused on how participants can treat interactional disattention or lack of attention as arising from technological causes, thus using not only the affordance of the technology, but also its possible repairables as interactional resources (Rintel, 2013a, 2013c).

In many institutional VMI-mediated encounters, participants feel the need to share access to some artefact or feature of the environment. Besides pointing, which can raise indexical trouble in “fragmented ecologies” (as discussed above in 4.3), one way to accomplish this interactionally is through showing sequences. These are a particular type of “object-centred sequences” in which participants bring some physical object to the (visual) foreground and establish it as a relevant concern in interaction (Tuncer et al., 2020). In F2F settings, because of embodied proximity, the actual, collaborative manipulation of the shown object is often a salient possibility (Kidwell, 2007), as is sequentially organised object transfer in general (Due and Trærupe, 2018; Tuncer and Haddington, 2020). Object-centred sequences in F2F environments often have a multi-sensorial character (see, e.g. Mondada, 2019b; Mortensen & Wagner, 2019). Medical settings offer a case in point, as many encounters involve patients offering their bodies for an examination that is both visual and tactile (see, e.g. Stommel et al., this issue). VMI precludes such a trajectory. In other words, in VMI settings, showings remain “visual”.

In a typical showing sequence in a VMI environment, an object is made relevant as a showable through prefatory sequences that may involve different forms of sequential work, one function of which is to suspend the “talking heads” visual organisation (Tuncer & Licoppe, 2019). This steers the interaction into a phase in which the object is manipulated to ensure joint visual access, in which talk is constrained sequentially and topically to this manipulation, thereby providing emergent slots for topic talk (Licoppe and Morel, 2014). At the same time, this talk, or the absence thereof, may be seen as providing opportunities for further manipulation and consideration. Finally, in the closing phase, the participants strive to display that they have achieved a grasp of the object. In that sense, showing sequences also function as “‘relational bids” (Licoppe, 2017; Tuncer et al., 2019).

VMI-mediated institutional settings – and medical settings in particular – provide opportunities for the initiation of showing sequences that enact some part of the body as a showable, either by presenting it to the camera or moving the camera to it – see, e.g. the showing of a foot in this issue (Due and Lange, 2020). However because the body can only be made available visually, and eludes tactile manipulation, showing sequences may be rarer in F2F medical encounters than in VMI ones, as in post-surgery consultations (Stommel et al., 2019; Stommel et al., this issue).

5. Articles in this issue

In this special issue, we present seven articles that deal with video-mediated interaction in two types of institutional settings: organisational and healthcare. We include papers from different healthcare settings: Jessica Pedersen Belisle Hansen presents findings from video-mediated interpretations in hospital encounters. Ann Merrit Rikke Nielsen presents findings from video-mediated consultations via a telepresence robot in a residential rehabilitation facility. Sakari Ilomäki and Johanna Ruusuvuori present findings from video-mediated tele-homecare for older adults. Wyke Stommel, Christian Licoppe and Martijn Stommel present findings from video-mediated post-surgery consultations, and Brian L. Due and Simon B. Lange present findings from video-mediated physiotherapy consultations. Organisational encounters make up the second type of setting, and this special issue includes papers based on data from organisational settings: Helen Melander presents findings from studies of video-mediated homework support, while Tuire Oittinen presents findings from video-mediated business meetings, focusing on aspects of noticings.

Jessica Hansen: Invisible participants in a visual ecology: Visual space as a resource for organising video-mediated interpreting in hospital encounters

This paper explores multilingual hospital encounters in which medical professionals and patients do not speak the same language and interpreting is facilitated through video technology. The participants use video technology to create an interactional space for interpreting.

Helen Melander and Johanna Svahn: Collaborative work on an online platform in video-mediated homework support

This study concerns the interactional work involved in the accomplishment of video-mediated homework support, and is based on a single case analysis of an instructional encounter between a tutor and an upper secondary student working together on mathematical assignments.

Ann Merrit Rikke Nielsen: Co-constructing the video consultation-competent patient

This paper explores consultations between a geriatric patient in a residential rehabilitation facility, his local caregivers, his relative, and his GP, who is present via telepresence robot. The analysis focuses on the patient’s interaction with the telepresence robot.

Sakari Ilomäki and Johanna Ruusuvuori: From appearings to disengagements: Openings and closings in video-mediated tele-homecare encounters

This article examines openings and closings in video-mediated tele-homecare for older adults in Finland. It demonstrates how participants organise these boundaries sequentially and multimodally, how visual appearing and disengaging are of key importance in these processes, and how openings and closings mirror each other in this institutional setting.

Wyke Stommel, Christian Licoppe, Martijn Stommel: “Hard to assess in this manner”: An “ineffective” showing sequence in post-surgery video consultation

In this article, one case of an emergent showing sequence in a video-mediated post-surgery consultation is examined to track its sequential organisation, which develops towards an ultimately inadequate showing. The case comes from a set of post-surgery consultations with patients who had undergone tumour resection (abdominal surgery) two weeks earlier.

Brian L. Due and Simon B. Lange: Body part highlighting: Exploring two types of embodied practices in two sub-types of showing sequences in video-mediated consultations

This paper describes physiotherapy consultations in Denmark, in which two types of practices are employed for successful history-taking through body-part showings.

Tuire Oittinen: Noticing-prefaced recoveries of the interactional space in a video-mediated business meeting

This study investigates the sequential and temporal organisation of recoveries of the interactional space in business meetings. It focuses on moments in which either an auditory or a visual barrier emerges, and the participants’ orientation to these troubles through intensified bodily-visual displays (embodied noticings).