Social Interaction. Video-Based Studies of Human Sociality.

2023 Vol. 6, Issue 1

ISBN: 2446-3620

DOI: 10.7146/si.v6i1.137382

Social Interaction

Video-Based Studies of Human Sociality


Reporting Mobile Device-Mediated Text to Manage Action and Agency in Co-Present Conversation


Jessica S. Robles1, Stephen M. DiDomenico2, Joshua Raclaw2 & Jack B. Joyce3

1Loughborough University
2West Chester University
3University of Oxford

Abstract

The paper considers the role of agency in human interaction with mobile devices. We use multimodal conversation analysis to trace how mobile screen content is reproduced as locally relevant for updating information for co-present interlocutors. While informing-centered actions supported by mobile devices may sometimes have the character of an agentic intrusion into the local interaction, we show that the organization of device-accessed information and its meaningfulness is nonetheless positioned in relation to how device-supported updates are animated into social action by human participants. This research contributes to understanding how device-related content is sequentially incorporated into face-to-face interaction.

Keywords: conversation analysis, mobiles, informings, updating, reported speech, agency, affordances

1. Introduction

Technology is a ubiquitous resource in everyday life, with our day-to-day interactions increasingly involving device-supported activities (Ling, 2012). These uses have renewed longstanding debates about the potential ontological status of digital artifacts (e.g., mobile phones, tablets, laptop computers), as participants may increasingly attribute human-like qualities to technological objects and tools (Katz, Halpern, & Crocker, 2015; Krummheuer, 2015; Mays, 2021). This raises the question of what the status is of new mobile and digital technologies for co-present naturally-occurring conversation (DiDomenico, Raclaw, & Robles, 2020; Robles, DiDomenico, & Raclaw, 2021; Mantere, 2022a, 2022b; Oloff 2021a, 2021b). In what ways do devices (and the interaction-centered functions they afford) make themselves available to accomplish situated, practical activities?

Recent work by language and communication researchers has looked to highlight the ways technologies for communication can bring both affordances and constraints to interaction (Robles, DiDomenico, & Raclaw, 2021; Licoppe, 2020; Mantere, 2022b). For instance, previous scholarship has shown how participants can read from or show their mobile phone screens to co-present others in order to share epistemic access for doing assessments (Raclaw, Robles, & DiDomenico, 2016). This mobile phone-supported, public display of knowledge in the form of direct reported speech or images allows participants to share the same referent of talk simultaneously. It is also well-documented that participants routinely interact with their mobile phones under the constraints of the normative face-to-face turn-taking system and its speakership and recipient roles (DiDomenico & Boase, 2013; DiDomenico, Raclaw, & Robles, 2020). However, mobile phones belong to a class of technologically-enabled objects that hold unique communicative capacities beyond material ones. In view of this, to what extent can mobiles phones (and other portable communication devices) be understood as having their own ontological status independent from participants' own? What implications does this have for the role of human agency, or the influence of technology on human lives and societies?

The current paper explores this conceptual question in the context of participants' explicitly importing aspects of their mobile phone usage, specifically the linguistic (or linguistically reconstructable) content that appears via device screens, into their social conduct. We examine how participants treat mobile devices as sources of live information by using newly-accessed material as epistemic resources for producing locally relevant updates addressed to their co-present interlocutors. While informing-centered actions supported by mobile devices may sometimes have the character of an agentic intrusion into the local interaction, we show that the organization of device-accessed information and its meaningfulness is nonetheless positioned in relation to how device-supported updates are animated into social action by human participants. In the next section, we review and integrate themes from several bodies of literature that stretch across Communication Studies, Linguistics, and Conversation Analysis.

2. Situating Affordances, Reported Speech, and the Contingencies of Updating

Affordance has become a leading conceptual framework in social science scholarship, seeking to strike a compromise between the social and technical aspects of contemporary communication technologies. The initial notion of affordance was first articulated by Gibson (1977, 1986) in the field of ecological psychology; however, its conceptual innovation is most apparent against a backdrop of the paradigm of technological determinism, a view that overemphasizes the degree to which technology molds and generates society and behavior (or "technology shapes society," see Ling, 2008), diminishing the role of human agency. Related ideologies towards technology as being detrimental to society have since been debated in public and scholarly circles (e.g., Beck, 2016; Hampton, 2016; Hampton & Wellman, 2020; Ling, 2012; Turkle, 2012, 2015). In contrast to deterministic views, Gibson's core premise was that objects hold intrinsic properties that are shaped by a users' perception of the objects' utility as well as broader cues in the environment. More recent extensions and applications have included communicative affordance (e.g., boyd, 2010; Evans, Pearce, Vitak, Treem, 2017; Meisner & Ledbetter, 2022; Schrock, 2015), where new possibilities for communication emerge when participants' goals are framed by the capacities (perceived and actual) of communication technologies.

In this sense, adopting a communicative affordance perspective towards technology use acknowledges the materiality of a technology without neglecting issues of social construction and human agency (boyd, 2010; Nagy & Neff, 2015). For the sake of the current investigation, an emphasis on affordances for communication highlights mobile use in situ, specifically in how it "frames the practices through which technologies come to be involved in the weave of ordinary conduct" (Hutchby, 2001, p. 450). Scholars of language and social interaction have documented these conversational practices by analyzing how participants orient to and bring into conversation their devices and the content accessible therein. For example, participants can keep their phones ready to hand (or even in sight) in order to monitor them for incoming notifications, manipulate them to look at the time, check for new text messages, or display/access information to relay to co-present interlocutors (e.g., DiDomenico, Raclaw, & Robles, 2020; Arminen, 2010; Brown et al., 2018; Porcheron, Fischer, & Sharples, 2016).

A routine concern in social interaction involves the continual monitoring and management of the distribution of knowledge and information among interlocutors, and mobile phones offer a unique, ubiquitous, and portable resource for negotiating this important dimension of everyday life. Scholarship on social epistemics examines how participants orient to presumptions of their own and others' relevant knowledge, or their epistemic status, through features of interaction such as turn design (Heritage, 2012). Participants continually display what they know when they enact their epistemic stance, which positions them as more knowledgeable (K+ position) or less knowledgeable (K- position) than their interlocutors (Heritage, 2012). As new information is provided in interaction to which all parties do not previously have equal access, the ongoing coordination of action and intersubjectivity must be explicitly renewed (Heritage, 1984).

In light of these ongoing epistemic concerns in encounters, participants may provide information to their interlocutors through the vehicle of different social actions such as announcements (e.g., Terasaki, 2004), storytellings (e.g., Gardner & Mushin, 2013), and confirmations (e.g., Seuren & Huiskes, 2017). One additional form of informing activity that we examine in the current article is updating, when speakers design their informing in a way that presumes some prior familiarity (or epistemic status) on the part of their recipients (Searles, 2019; DiDomenico, 2015). Participants, of course, may additionally use various material and communicative resources such as the mobile phone if these are relevant to updating their interlocutors.

The content accessed through mobile devices can be used in the service of many informing actions, especially as sources of "tellables," and evidence for judgments (Raclaw, Robles, & DiDomenico, 2016). Text accessed on the mobile devices (including screen-based "speech") can be the basis for these reports, as when someone paraphrases or purports to read directly off a mobile's screen text messages from themselves or non-present parties. These reports are similar to other representations of prior communication, or reported speech, and are routinely deployed in activities such as stories, troubles talk, and so forth (Holt, 1996; Terraschke, 2013). Further empirical attention has been given to the way people recontextualize prior discourse in assessing and enacting prior utterances (Fox & Robles, 2010), framing investigative public hearings (Unuabonah, 2018), and in sign languages (Hermann & Steinbach, 2012). In all of these instances, people quote the prior talk of themselves or others, expressions, and sometimes even "thoughts" that were not directly communicated (or not "accurately" accessible) (Buchstaller, 2009).

The ubiquity of mobile phones provides a unique resource for accessing quotables from the recorded text and other media available on a mobile device (Brown et al., 2018). Such quotables may be presented as a more direct or verbatim representation (rather than an indirect or paraphrased representation). Despite some mobile phone content being text-based, reports may be delivered in ways that enact an affective stance toward the reported content in the same way that verbal or embodied reported speech is reconstructed through, for example, performatively conveying the message through intonation and rate of speech (Günthner, 1999; Yao & Scheepers, 2011). That this occurs even though the message was a text and neither spoken nor heard aloud demonstrates how the voice is positioned as a semiotic resource for indicating to recipients how to understand the quoted content.

Our own investigation builds on these lines of previous research to extend what is known about situated agency and people's leverage of the affordances of mobile communication devices while they otherwise engage in co-present interaction. We examine how alleged content accessed from the devices (represented primarily through text, though pictures or emojis may be included) is made relevant and actionable. For example, someone may pull out their phone because of its relevance to the details of an in-progress storytelling they are producing (e.g., to look up information or read a text message aloud to recipients). Our central focus is the verbal practices through which participants introduce apparently directly read device-mediated content for informing actions. The content in question may take a variety of forms: it may be text messages participants have received or are (or soon will be) composing to send to non-present others, or it may be information obtained through an app on their device (such as the current time of day) or from a website or platform (such as search engine results about the weather). In the next section, we discuss our data and methods for investigating these issues.

3. Data and Methods

The data examined for this study included video-recordings of naturally-occurring conversations from three corpora that were recorded in the United States between 2005–2016 (approximately 50 hours). All recordings were conducted under the full approval of institutional review processes. These recordings were captured naturalistically: participants were recorded in a range of locations that were more or less where they would ordinarily be, and they were not instructed to do anything (nor avoid doing anything) with their co-located communication devices, which included mobile phones in most cases but occasionally also laptops. For the current study, we used 12 relevant video recordings in which participants had mobile devices present during co-present interaction, and we extracted 18 instances from across these videos where participants produced their talk as being read aloud from the screen of a mobile phone or laptop. This dataset did not include participants simply recalling device-mediated messages, but rather required that gaze direction, suspension of co-present activity, orientation to the device as the source of content, and prosodic delivery could be interpretable as "being read aloud." These cases were each transcribed using Jeffersonian (1984) transcription conventions, and in some cases Mondada's (2018, 2022) notations have been added to capture features of embodied action (Appendix A). We use Conversation Analysis (Schegloff, 2007) to examine how orientations to mobile screens' content were sequentially organized into the ongoing conduct and what social actions were accomplished in doing so.

Our collection consisted of 18 cases in total where participants orient to a device (usually a mobile phone), voice content they claim is accessed through the device, and manage bodily resources such as gaze to present their voicing as based on reading screen-accessed content. For this paper, we focus specifically on the analysis of five extracts that best illustrate the systematic, embodied organization of talk and device usage. Our findings suggest that device-mediated text is read to support the enactment of informings that are routinely treated by recipients as updates to previously known topics or information. Additionally, these updates are shown to be consequential for arranging future joint activities. We are not arguing for the veracity of actual, direct reading of mobile-based text messages, given that we do not have screen-capture data of any of the devices depicted in this paper. Instead, we focus analytic attention on participants' demonstrable linguistic and embodied orientations to their devices in ways that legibly position them as reading and reporting either to a partial degree (such as paraphrasing) or in full (verbatim).

4. Analysis

The participants in our data routinely orient to their devices as tools for accessing content that is treated as worth sharing with co-present interlocutors. One reason for why participants report on this content (text based or otherwise) as a part of informing actions is because it is treated as practically relevant to the ongoing unfolding of an in-progress activity. Such informings may be oriented to as inviting assessments, implicating co-participants in formulating a response to the text message that was reported aloud, and producing announcements that make relevant the next phase of the current activity at hand. Thus, material accessible on a mobile device is leveraged by participants to act, both in the moment and over the course of coordinated projected actions beyond the situated interaction. In this way, mobile phones can become enrolled as a resource participants use in their capacity to act in concert, as a form of agency (Enfield, 2013).

The current analysis examines device-supported informings that are positioned as new, confirmed, or revised information in the unfolding sequential conduct. In the following subsections, we first examine mobile phone-supported informings that occur when made relevant by the ongoing face-to-face talk. Then we examine mobile-supported informings that are accomplished as though the mobile content was suddenly received.

4.1 Mobile-based reporting as occasioned by prior co-present talk

One robust finding of our investigation involves the verbal and embodied practices that converge with apparent readings from a mobile phone. Sequentially, these practices routinely emerge as participants access, report, and segue out of reading text from a screen. Moreover, a subset of our collection consists of apparent readings that are either elicited indirectly or directly solicited from an interlocutor. All of these cases can generally be characterized as look-ups because they are oriented to brief moments of using a device to check information that is relevant either to the activity currently underway or what will be a future activity among several interlocutors.1

The first subsection focuses on the reporting of mobile-supported information being occasioned by an interlocutor's prior talk. This type of access to one's mobile phone highlights its affordance of portability (Schrock, 2015), with the mobile phone being easy enough to carry on one's body and transport to any place in the world (Ito, Okabe, & Matsuda, 2005). Interactionally speaking, this amounts to participants having the potential to open up lines of communication with non-proximal parties (e.g., by simply reaching into their pocket). In some cases, in this portion of our collection, the participant who produces the action that makes mobile phone use relevant may not themselves have access to their own device as a relevant resource. In this context, mobile phone use can be seen as occasioned by a co-participants' actions. In Extract 1, a group of friends (Ann, Bonnie, Cam, and Demi) are discussing an event they are planning to attend the next day, specifically matters related to the time of the event and the forms of transit they will use to get to the event. In the course of this discussion, another participant initiates a new topic related to the projected weather forecast for the day.

Extract 1
  Open in a separate window

This case demonstrates how any interactant who is able to check their mobile phone (and directly read from it) may do so when prompted by a co-participant's action (in this case, a question) in the co-present encounter. At the beginning of the extract (lines 1–5), the participants are asserting their preferred weather situations for the next day's event, which involves a hope for a lack of rain (indirectly implying that they hope it will be sunny). Both speakers turn their gaze to the nearby window at line 1, establishing a shared orientation to the weather as a point of concern (Figure 1). This topicalization of the weather makes the general confirmation of the details of the day's projected weather relevant. In line 8, Bonnie's report of "it didn't say" particularly suggests there may have been an earlier check on the weather forecast.

Ann's pronominal use of "it" in line 10 ("It was supposed to be?") projects a further elaboration of this earlier forecast. During the production of this turn, she concurrently retrieves her phone and interacts with its screen (line 11; Figure 2). During the trajectory of this embodied activity (lines 11–12), Ann goes on to produce a more detailed report related to the next week's weather ("...supposed to rain"). While not entirely clear from the recording (or without screen captures of the phone), the timing of Ann's active use of her phone at this precise point makes it a likely relevant resource for confirming her apparent remembering of weather-related information. Following this, Demi and Cam offer their verbal agreement with the report of rain (lines 14 and 15) while Ann continues her visible orientation towards her phone's screen, presumably as part of her ongoing search for the latest weather information.

Next, just as Demi begins to articulate her own knowledge of the specific days forecasted for rain (line 15), Ann takes the floor in overlap to correct her interlocutor's reported forecast (possibly attending to the ambiguity of which day's weather they are speculating about). Still directing her gaze at her phone (presumably still attempting to access weather-related resources; Figure 3), she then offers further details related to her previously reported forecast. In line 16, her "I think" construction works to mitigate certainty about the current day's weather, potentially because she is basing it solely on her own recollection of previously accessed weather updates. However, just as she begins saying what is projectable as "cloudy," she abandons this formulation mid-production to initiate self-repair. She goes on to produce a change of state token,"oh" (Heritage, 1984), displaying an orientation to this as new information about the weather, and then issues a further "no"-prefaced forecast. This rejects the previously predicted course of the day and updates the locally available weather information for the group's planning for the day.

It is unclear precisely what was abandoned in Ann's talk at line 16 (for instance, it could be that today was supposed to be cloudy but turns out the cloudiness is temporary). Yet Ann's choice to produce a mobile-verified confirmation of the weather, with accuracy attuned to the hour at which clouds might dissipate, bears relevance to the future event they have been discussing. Elsewhere in our collection, we have further instances of "weather checks" where mobile phone use is sequentially occasioned and unfolds in a similar manner. For instance, weather becomes topicalized relative to a future event and, in one case, specifically what to wear and the coordination of matching outfits; a prediction or assertion of the weather is made, and in response the other participant takes this prompt to confirm the weather situation.

Our next example offers a further illustration of the phenomenon of participants reading directly read from their devices. In both of these initial cases, this occurs after some local interactional contingency conversation has made some piece of device-accessible information relevant, during which the device is retrievable but not in active use. Here, the participants using a mobile device are those who initiate an offer of mobile-supported information to their interlocutor(s). This provides further in situ evidence of another affordance of the mobile phone, multimediality (Schrock, 2015), which refers to the wide range of communicable sources and information that can be embedded in mobile media (e.g., text messages from friends, web searches, weather apps, notifications, and so forth). Throughout our collection, the way participants orient to this affordance can be framed in terms of the epistemic access it can provide co-interlocutors (see Raclaw, Robles, & DiDomenico, 2016). Thus, there are many complex ways in which mobile devices are introduced into co-present interaction for the purpose of looking up or checking something that comes up in the course of the co-present activity, specifically in relation to the activity of planning.

Extract 2 occurs in the course of an interaction among three college friends (Susan, Krista, and Aly) as they are revisiting an earlier query regarding a non-present roommate (Jana). At the start of this segment, they are talking about whether Jana is returning or staying elsewhere that night. Though the topic has been raised before, it has not been confirmed, nor has it previously been made clear why it matters. As emerges here, the implication of the roommate being absent is that one of the women (Aly) can borrow her bed for the night and stay over. Where the extract begins, they have briefly discussed the times of their morning classes, and Susan announces that this means Aly can borrow the roommate Jana's bed, to which Aly asks why, occasioning Susan's access to and use of her mobile phone.

Extract 2
  Open in a separate window

Though "she's not coming back tonight" is a potentially sufficient explanation of the availability of the bed, Krista (KRI) challenges Susan's assertion on epistemic grounds in line 2 ("how do you know?"). This, too, is linked to the previous conversation, as Krista was the person who several minutes before shared a text message that suggested Jana was returning to their location after all—therefore, the source of Susan's certainty about her assertion at line 1 is unclear. Next, Susan downgrades the prior description "she's not coming back tonight" to "I don't think she is" in line 7, attending to Krista's query and disattending to Aly's offer to take the couch (line 3). Susan produces a further, latched turn as she begins to get her phone out of her pocket while simultaneously introducing her mobile phone into the interaction. Susan's temporal formulation ("right now") and downward gaze direction (towards her pocket where her phone is stored) projects her possible use of some type of communicative device and, given the prior sequential context, compose a message addressed to Jana. This developing text message is thus a necessary "check" on the information Krista has queried and is relevant to Aly's sleeping arrangements later that evening. Further, this "check" should therefore be done "right now" to comply with Krista's request for confirmation as well as to establish the near-future sleeping location of Aly.

However, it is not at "right now" that the phone is physically accessible and engaged with. The description of actions across lines 8–14 are glosses of several successive embodied movements which, while not fully visible in the video, appear to indicate that Susan's phone is in her pocket. Thus, the larger course of action of preparing to text-message a non-present person is built out of a larger trajectory realized through a combination of embodied and verbal actions (see Figures 4–6)—announcing the phone's relevance, getting the phone out of the pocket, opening the phone, and preparing it for use. Each of these multiple and layered preparatory actions, action trajectories, and phases of completions are what some scholars refer to as "microsequentiality" (e.g., Deppermann & Streeck, 2018; De Stefani & Mondada, 2018).

At line 21, in time with the completion of Aly's laughter, Susan appears to read aloud the message she is currently engrossed in composing to Jana: her thumbs are moving in a typing motion, her gaze continues to be directed at her phone screen, and she produces the question she had previously proposed (to her co-present interlocutors) to Jana in an elongated prosody that increases the timing of each word. Because saying words aloud ordinarily takes less time than typing multiple individual letters into a device, this enacts an apparent dictation of the message she is composing though she is speaking aloud the words as she is typing them, similar to providing a "running commentary" on one's own behavior, but doing so through an enactment that makes available the content of what would otherwise be inaccessible to those co-present (c.f., Mortensen, 2013). This also puts her potential engagement with Aly on hold while the message is being typed, accounts for her use of the phone, shows that she is following through on what she said she would do, and also provides an oral record of the precise question being addressed to Jana. Additionally, this suggests that if Jana responds while they are all still in the shared presence of one another she may be accountable for providing this update on the phone-mediated exchange.

The third case of this subsection showcases a similar sequential unfolding, yet this time with the phone user themselves being the person who makes relevant the mobile-supported action. In Extract 3, several friends (Monica, Bri, and Eve) are in someone's room sitting on a bed. Monica's turn in line 1 formulates a prediction that a mutual friend (Jim) will not be going ahead with having a social event at his house later that evening. At the start of the segment, Bri expresses some uncertainty about the status of Jim's gathering while reporting that her mobile phone is placed nearby (out of reach). Monica, who is positioned within reach of Bri's phone, treats this as a request for physical assistance and picks up Bri's phone in order to hand it to her.

Extract 3
  Open in a separate window

Monica looks at the screen of Bri's phone just before announcing in line 12 "oh Jim Drams texted you." Monica passes the phone to Bri as Eve asks Bri a question, delaying Bri's possible inclusion of whatever content Monica alerted her to on the phone. Once this insertion seems to come to a close, Bri reads out the message (lines 21–22). This confirms that Monica's prediction that Jim would not be doing anything is incorrect, whereas Bri's supposition that he might is correct, since Jim's question asks for a confirmation as to whether either Bri or all of them are "still" coming over.

In this excerpt, the reading is once again delayed minutely to be introduced at a sequentially appropriate moment but is still done "as news" for the participants. It is highly relevant news that concerns them all, as the subsequent talk deals with matters related to getting ready to go to the party (what to wear, how drunk to be), and, as a key preparatory element to getting ready for the party, the women spend several turns formulating candidate responses to Jim's text message. Each candidate response is done and redone in various ways showing an orientation to turn-design in the context of text messaging and the various pragmatic inferences that might be drawn by Jim upon receiving the message. This allows them to participate in Bri's conversation with the non-co-present Jim, but also allows Monica and Eve to have a separate conversation with each other. This is similar to the conversation in excerpts 1–2 in which two participants talk to each other while the third is texting but who also talk around each other's verbal contributions and intermittently re-engage with each other despite having parallel primary involvements.

These examples show how self- or other-initiated actions make relevant the introduction of a mobile phone-on the way to introducing content from the phone and while managing dual engagements-through use of gaze, touch, bodily movement, manipulation of devices, and shifts in voice. These practices allow participants to import relevant information for future activities into the current interaction in relatively non-disruptive ways. These informings are provided as confirmations and updates regarding information that may be known or assumed but that may also need revisions or extensions given their implications for projectable ongoing or future activities. In these cases, the use of the device and introduction of the content are similar regardless of whether the content read is from the device's clock app, a weather website, or a friend's text message. They are all treated as updates to prior information with future consequences.

4.2 Apparent Noticings of Mobile Communications Followed by Reporting

Participants may also incorporate device-supported readings in a more spontaneous fashion that is presumably occasioned by the receipt of new text messages in real time. Asynchronous, text-based forms of communication (e.g., text messages, emails, other messaging applications) are an affordance of mobile phones, making it possible for the progressivity of an ongoing text-based conversation to be halted or intermittently delayed at different points while one person or the other has not responded to a text-based action or while one person or the other is unaware of an incoming responding message (DiDomenico & Boase, 2013; DiDomenico, Raclaw, & Robles, 2020). Because of the frequent gaps in turns that this provides for, local participants with an accessible mobile may become spontaneously aware of an incoming message, either because of an incoming notification, or because they happen upon the visible record of a previous notification as they look at their phone.

In the extracts that follow (consistent with others in the collection), we highlight how participants, upon already being engaged in using their mobile phones, report device-accessed information as though noticed or received just now. In doing so, this neatly allows such information to be introduced as an interjection without a need to "prepare" the informing sequence to make an informing relevant to the ongoing co-present interaction. These read-as-noticed updates emerge across a range of interactional circumstances that all involve participants 1) reading information as though it has just been received or just been seen, which is now 2) treated as new and relevant news to those co-present. These examples are the most like news announcements of the "this just in" variety, as participants manage ongoing situations that require updates as new information is deemed relevant.

Extract 4 below is a continuation of Extract 2, involving a text message being composed in (and seemingly read aloud throughout) the course of confirming roommate sleeping arrangements for the night. To recap, the participants (Susan, Krista and Ally) have been discussing whether a non-present roommate (Jana) returning on that evening would mean their guest, Aly, will have to sleep on the couch. In the further segment appearing below, Krista announces, as though having just noticed it, the receipt of a new text message suggesting Jana would in fact (contrary to everyone's expectation) be returning to the house that evening. Though no bed is mentioned at this time, this information is treated as relevant to the present parties when Krista seemingly reads the text message aloud to her interlocutors.

Extract 4
  Open in a separate window

In line 2, Krista projects the relevance of her phone by showing it to Susan, interjecting into Susan's initiation of a new turn. This attributes a relative urgency to the message, suggesting that it is of sufficient relevance that it should be shared as soon as possible. The relevant portion of the content on her phone screen (Jana's text "just two more days" in line 3) is what Krista reads at first, which is the line that she has interpreted as a contradiction to the previously shared and assumed knowledge that Jana would not be back at their shared accommodation. Susan makes this assumption explicit at line 6, and Krista then reads both the initial message she had composed to Jana and re-reads Jana's message in response. This is received in lines 20–22 as "bad news": Susan provides a negative assessment (line 20) followed by a brief silence and Krista's formulation of negative affect (confusion), which could serve as a complaint (line 22).

Around 15 minutes later, this apparent change of state is queried again as Susan once more asserts that Jana will not return, while Krista challenges Susan's assertion (likely because Krista had received this contradictory information). That exchange prompts Susan to text Jana to confirm the situation. A few minutes after Susan has sent the message, she apparently receives a response, which is depicted in Extract 5 below.

Extract 5
  Open in a separate window

In this segment, Susan has been regularly monitoring her phone to assess how much time they have been recording (lines 1–4, presumably to infer how much time remains until they complete the course assignment the recording helps fulfill). At the next available moment in line 8, Susan announces a new piece of information as a direct report of a message from Jana: "um I don't think so" (lines 8–9). This is introduced with "Jana said" and there is no lead-in to producing this information: Susan already has the phone in hand and her gaze has been directed at it for the purpose of checking the time. Whether Jana's message actually arrived just then or was actually seen in the process of checking the time and was introduced at the next available moment is not confirmable in the data but shows that the message is produced at the "right moment" in the first slot at which it could reasonably be shared. That the response is shared and then immediately thereafter the message from several minutes earlier in the interaction to which it responds is repeated ("when I said 'are you coming home tonight'" lines 9–10) emphasizes the importance of the response over the initial question, as the response is what is relevant to sleeping arrangements discussed previously.

Susan then prepares the phone for texting and again appears to read while composing a responding text message (lines 12–14; Figure 7). This allows Susan to account for what she is doing and make explicit her confirmation for present and non-present interlocutors, but also makes her message available for revisions, as taken up by Krista in lines 18–19 while proposing an alternative or additional text message that Susan might send in response. Krista's suggestion reformulates the "just wondering" confirmation-action produced by Susan as more of a repair initiation that seeks to correct a previous error or misunderstanding: "wait what I thought you said you were going home." Like the "um" in Jana's message, the "wait what" in Krista's proposed message incorporates turn-initial objects that indicate some form of uncertainty or choice relevance and an element of surprise (c.f., Beňuš, Gravano, & Hirschberg, 2011). Yet Susan does not orient to any aspect of these features, and given Susan's disattending Krista proceeds to engage with her own mobile phone (lines 21–23; Figure 8).

Krista begins touching the phone before accounting for a reason for doing so ("didn't she say she was going home," lines 23–24), which allows for her to then check the initial message in which Krista first obtained this impression). However, Krista picks up her phone in line 15 and starts looking at it, making it available as a source of evidence for her earlier confusion. Aly orients to this projectable use of Krista's phone as evidential in line 25, which alongside physical orientation to Krista and gazing toward Krista's phone in lines 26 and 30 (Figure 9) positions Krista's phone as a source of relevant information in the investigation of the previous misunderstanding. In line 29, Krista does a pre-telling, introducing upcoming reported speech with "she said," and then after a two-second silence to find and read the message, she repairs the initiation of the reading with turn-initial "OH" which projects new information before re-doing the reading.

After this point in the extract, it becomes more difficult to assess precisely what is said by Jana and when (if at all) the co-participants are reconstructing text messages faithfully or not. However, the significance of this exchange for the current analysis lies in the way the apparently verbatim-read messages are introduced to manage what is known (and precisely what needs to be known) among the participants of the in-person encounter. Apparently-noticed messages are possible because they have been prepared beforehand (Susan had read aloud the message she had sent to Jana and thus they could expect a response) and/or because a device is physically available, which might at any point produce information that has either already been prepared in this way or could be treated as newly-relevant. Thus, the manner in which the mobile phone affords asynchronous, text-based communication simultaneously allows participants to negotiate the inclusion of content from the device (and intermittently reorient to it as relevant along the way). Participants treat these messages and their details as relevant updates that project changes in the trajectory of their activities—in this case, the question of where Aly will sleep that night. This constitutes important interactional work in the practical updating of mundane social activities (in this case, friends attempting to coordinate with each other) as well as proposing activities to take place in the future.

Other examples we have of apparently-just-noticed information involve picking up a phone and apparently seeing a message, possibly in response to a notification in one instance, and apparently in response to an audible chime in another. In these cases, the message is read and this invites involvement from the others present: one involves a participant who is prompted to check her own phone for a similar message or, alternatively, coordinate a response in collaboration with the other participants. "Noticed" cases in our data relate to text messages rather than other forms of screen text, probably because (at least at the time the data were collected) it is more common to receive new information from a person in an ongoing text conversation than, say, via a phone notification (this is possible, but we have no instances where this occurs during our recordings). Regardless, the messages and responses to them are oriented to as important information for upcoming social engagements, and reading the messages establishes shared information and invites collaboration from co-participants.

5. Mobile-Mediated Updating as a Social Practice

A central goal of this paper is to examine participants' use and leveraging of device affordances in order to import mobile-mediated screen content into their conversational actions.

Stepping back from our corpus and analysis, we return to a central question about agency: Was it the human participant or the phone that initiated these lines of activity? Broadly speaking, an affordances view holds that the affordances and constraints that technologies provide are embedded in the user's perception within a particular environment. In this respect, we can say with certainty that the participants utilizing phones in our collection have greater agentive status than that of the objects in terms of mobilizing action. Yet at a more granular level, vis-à-vis our analysis of two distinct types of mobile-supported phenomena, the question of agency becomes more complex. In the first set of cases involving sequentially occasioned mobile reports, the participants' use of the mobile is (seemingly, however directly or indirectly) made relevant by some aspect of the emerging social conduct (for instance, being directly prompted by an interlocutor's question). These instances also support theorizing of the mobile phone's potential agency as being wholly dependent on the way the participants make it relevant (i.e., by virtue of the emerging sequential context).

The second set of cases involving apparently noticed text messages, however, begin with the individual's own monitoring of their mobile's operations (in some cases, completely independent of their own manipulations of the device) and then proceed to demonstrate how the messages are worked into the social fabric of the interaction. Here, the ontological status of the mobile relative to its user bears closer scrutiny. In these cases, it could indeed be argued that the mobile-using participants initially "respond to" the notification or other display-related feature of their mobile. In this sense, the phone itself could be oriented to as "interrupting" an in-progress interaction the user is engaged in, possibly without any of the user's own manipulations of the phone itself (e.g., if the phone's screen lights up and displays an alert of a new text message, regardless of whether the user is actively using the device). As has been documented by other scholars, this has the potential to result in greater distraction from, for instance, co-present conversations (e.g., Abeele, Hendrickson, Pollmann, & Ling, 2019). At the same time, it must be made clear that participants themselves (at least, in these cases) are ultimately responsible for how something observed on their device's screen (in many cases, privately) can be made meaningful to others. Put differently, mobile phones (including their communicative affordances) are not actionable without humans constructing them as such.2

It is also worth highlighting our systematic observations on updating as a mundane activity. Across our collection and the cases analyzed here, these actions were overwhelmingly done as informing actions. Several of the extracts analyzed feature one particular variation of informing in the sense of updating recipients on previously known topics or details. While prior work has examined updating in contexts related to health and illness (Beach, 2001; DiDomenico, 2015) and family socialization (Searles, 2019; cf. deSouza, 2021), the current study details its mundane enactment in talk among friends and intimates. It is possible that this activity may have been more prevalent due to how recurrent the theme of making arrangements for future activities (e.g., Asmuß & Oshima, 2012; Ekberg, 2011) was across our corpus. Our analysis of the extracts in this report has also provided insight into the ways mobile technology can become a fundamental, epistemically-oriented resource in the management of social arrangements and, at a broader level, the sequential unfolding of social relations across modalities (Aronsson & Cekaite, 2011; Goodwin, 2006). In addition to the ability to look up information relevant to proposed plans or accessing previous communications with others, the communicative affordances of the mobile device also allow participants to compose and receive communications from non-present parties in real time to confirm details with relevant parties across space (in-person and proximally distant) and time (with the asynchronous nature of these forms of communication).

6. Conclusion

This paper explored the impact of device-use in co-present interaction and its implications for the status of agency in the context of participants' quoting content from the screens of their devices. As mobile phones and other portable devices have become routine sources of ubiquitously-accessible information in everyday life, we highlight the intersection of the communicative affordances of such devices and, most of all, the interactional practices by which they become implicated in the texture of everyday social encounters. Our analysis underscores the way participants can report aloud others' and their own text messages to co-present interlocutors the contingencies of local, face-to-face talk. In doing so, we argue, they leverage technologically-afforded, asynchronous semiotic resources for their own agentive ends. While some accounts might suggest that mobile devices, perhaps by virtue of their mere presence, "interrupt" or otherwise detract from how conversation is managed or from its quality, our data points to the crucial ways in which participants utilizing technologies (regardless of what sounds or other notifications are emitted from a device) are still ultimately responsible for the interactional work of formulating and leveraging device-related content in a way that is fitted to the local sequential context. As such, the organization of device-accessed information and its meaningfulness is, like other aspects of our interactive environments and ecologies, embedded into and animated by participants for social action and situates the locus of agency in service of human social conduct.

References

Abeele, M. M. V., Hendrickson, A. T., Pollmann, M. M., & Ling, R. (2019). Phubbing behavior in conversations and its relation to perceived conversation intimacy and distraction: An exploratory observation study. Computers in Human Behavior, 100, 35-47.

Aronsson, K., & Cekaite, A. (2011). Activity contracts and directives in everyday family politics. Discourse & Society, 22(2), 137-154. https://doi.org/10.1177/0957926510392124

Arminen, I. (2010). New reasons for mobile communication: Intensification of time-space geography in the mobile era. In Ling, R., & Campbell, S.W. (Eds.), The Reconstruction of space and time: Mobile communication practices (pp. 89–107). Transaction Publishers.

Asmuß, B., & Oshima, S. (2012). Negotiation of entitlement in proposal sequences. Discourse Studies, 14(1), 67-86. https://doi.org/10.1177/1461445611427215

Beach, W. A. (2001). Stability and ambiguity: Managing uncertain moments when updating news about mom's cancer. Text & Talk, 21(1-2), 221-250. https://doi.org/10.1515/text.1.21.1-2.221

Beck, J. (2016, June 14). Ignoring people for phones is the new normal: A study looks at how phone snubbing—"phubbing"—becomes socially acceptable. The Atlantic. Retrieved from https://www.theatlantic.com/technology/archive/2016/06/ignoring-people-for-phones-isthe-new-normal-phubbing-study/486845/

Beňuš, Š., Gravano, A., & Hirschberg, J. (2011). Pragmatic aspects of temporal accommodation in turn-taking. Journal of Pragmatics, 43(12), 3001-3027. https://doi.org/10.1016/j.pragma.2011.05.011

Boyd, d. (2010). Social network sites as networked publics: Affordances, dynamics, and implications. In Z. Papacharissi (Ed.), Networked self: Identity, community, and culture on social network sites (pp. 39–58). Routledge.

Brown, B., O'hara, K., McGregor, M., & Mcmillan, D. (2018). Text in talk: Lightweight messages in co-present interaction. ACM Transactions on Computer-Human Interaction (TOCHI), 24(6), 1-25. https://doi.org/10.1145/3152419

Buchstaller, I. (2009). The quantitative analysis of morphosyntactic variation: Constructing and quantifying the denominator. Language and Linguistics Compass, 3(3), 1010-1033. https://doi.org/10.1111/j.1749-818X.2009.00142.x

Deppermann, A., & Streeck, J. (2018). The body in interaction. In A. Deppermann & J. Streeck (Eds.), Time in Embodied Interaction: Synchronicity and sequentiality of multimodal resources (pp. 1-29). John Benjamins.

deSouza, D. K. (2021). Everyday updates: How parents ask about their young children's lived experiences. Journal of Social and Personal Relationships, 38(4), 1172-1193.

De Stefani, E., & Mondada, L. (2018). Encounters in public space: How acquainted versus unacquainted persons establish social and spatial arrangements. Research on Language and Social Interaction, 51(3). https://doi.org/10.1080/08351813.2018.1485230

DiDomenico, S. (2015). Help seeking in action: Managing interaction and mental health on a crisis helpline. Dissertation. Rutgers University.

DiDomenico, S. & Boase, J. (2013). Bringing mobiles into the conversation: Applying a conversation analytic approach to the study of mobile phones in co-present interaction. In D. Tannen & A. Trester (Eds.), Discourse 2.0: Language and new media (pp. 119—132). Georgetown University Press.

DiDomenico, S. M., Raclaw, J., & Robles, J. S. (2020). Attending to the mobile text summons: Managing multiple communicative activities across physically co-present and technologically mediated interpersonal interactions. Communication Research, 47(5), 669-700.

Ekberg, S. (2011). Making arrangements: Remote proposal sequences and attendant structural phenomena in social interaction (Doctoral dissertation, The University of Adelaide).

Enfield, N. J. (2013). Relationship thinking: Agency, enchrony, and human sociality. Oxford University Press.

Evans, S. K., Pearce, K. E., Vitak, J., & Treem, J. W. (2017). Explicating affordances: A conceptual framework for understanding affordances in communication research. Journal of Computer-Mediated Communication, 22(1), 35-52. https://doi.org/10.1111/jcc4.12180

Fox, B. A., & Robles, J. (2010). It's like mmm: Enactments with it's like. Discourse Studies, 12(6), 715-738.

Gardner, R., & Mushin, I. (2013). Teachers telling: Informings in an early years classroom. Australian Journal of Communication, 40(2), 63.

Gibson, J. J. (1977). The theory of affordances. In R. Shaw, & J. Bransford (Eds.), Perceiving, acting and knowing: Toward an ecological psychology (pp. 67-82). Erlbaum.

Gibson, J. J. (1986). The Ecological Approach to Visual Perception. Erlbaum.

Günthner, S. (1999). Polyphony and the 'layering of voices' in reported dialogues: An analysis of the use of prosodic devices in everyday reported speech. Journal of pragmatics, 31(5), 685-708. https://doi.org/10.1016/S0378-2166(98)00093-9

Goodwin, M. H. (2006). Participation, affect, and trajectory in family directive/response sequences. Text & Talk—An Interdisciplinary Journal of Language, Discourse Communication Studies, 26(4–5), 515–543. https://doi:10.1515/TEXT.2006.021

Hampton, K. N. (2016). Persistent and pervasive community: New communication technologies and the future of community. American Behavioral Scientist, 60(1), 101-124.

Hampton, K., & Wellman, B. (2020). All the lonely people? The continuing lament about the loss of community. In L. Lievrouw & B. Loader (Eds.), Handbook of digital media and communication (pp. 281-296). Routledge.

Heritage, J. (1984). A change of state token and aspects of its sequential placement. In Heritage, J., & Atkinson, J. M. (Eds.), Structures of social action: Studies in Conversation Analysis (pp. 299-345). Cambridge University Press.

Heritage, J. (2012). Epistemics in action: Action formation and territories of knowledge. Research on Language & Social Interaction, 45(1), 1-29.

Hermann, A. & Steinbach, M. (2012). Quotation in Sign Languages: A Visible Context Shift. In I, Van Alphen, & I. Buchstaller (Eds.), Quotatives: Cross-Linguistic and Cross-disciplinary Perspectives (pp. 203-228). John Benjamins.

Holt, E. (1996). Reporting on talk: The use of direct reported speech in conversation. Research on Language and Social Interaction, 29(3), 219–245. https://doi.org/10.1207/s15327973rlsi2903_2

Hutchby, I. (2001). Technologies, texts and affordances. Sociology, 35(2), 441–456. https://doi.org/10.1177/S0038038501000219

Ito, M., Okabe, D., & Matsuda, M. (2005). Personal, portable, pedestrian: Mobile phones in Japanese life. Boston Review.

Jefferson, G. (1984) Transcription Notation. In J. Atkinson, & J. Heritage (Eds.), Structures of Social Interaction (pp. ix-xvi). Cambridge University Press.

Katz, J.E., Halpern, D., & Crocker, T. (2015). In the company of robots: Views of acceptability of robots in social settings. In Social Robots from a Human Perspective, Springer International Publishing, pp. 25–38.

Krummheuer, A. (2015). Technical Agency in Practice: The enactment of artifacts as conversation partners, actants and opponents. PsychNology Journal, 13(2).

Licoppe, C. (2020). Mobile phones in action. The Oxford Handbook of Mobile Communication and Society, 95.

Ling, R. (2008). New tech, new ties: How mobile communication is reshaping social cohesion. MIT press.

Ling, R. (2012). Taken for grantedness: The embedding of mobile communication into society. MIT press.

Mantere, E. (2022a). Smartphone Moves: How Changes in Embodied Configuration with One's Smartphone Adjust Conversational Engagement. Social Sciences, 11(5), 219. https://doi.org/10.3390/socsci11050219

Mantere, E. (2022b). Smartphone Situation: Personal Smartphone Use During Face-To-Face Encounters. Dissertation. Tampere University, Finland

Mays, K. K. (2021). Humanizing robots? The influence of appearance and status on social perceptions of robots. Unpublished doctoral dissertation, Boston University.

Meisner, C., & Ledbetter, A. M. (2022). Participatory branding on social media: The affordances of live streaming for creative labor. New Media & Society, 24(5), 1179-1195. https://doi.org/10.1177%2F1461444820972392

Mondada, L. (2018). Multiple Temporalities of Language and Body in Interaction: Challenges for Transcribing Multimodality. Research on Language and Social Interaction, 51(1), 85-106. https://doi.org/10.1080/08351813.2018.1413878

Mondada, L. (2022). Multimodal transcription. https://www.lorenzamondada.net/multimodal-transcription

Mortensen, K. (2013). "Writing Aloud: Some Interactional Functions of the Public Display of Emergent Writing." In Proceedings of the Participatory Innovation Conference, edited by H. Melkas and J. Buur, 119–125. PIN-C, Lahti, Finland.

Nagy, P., & Neff, G. (2015). Imagined Affordance: Reconstructing a Keyword for Communication Theory. Social Media + Society, 1(2), 1-9. https://doi.org/10.1177%2F2056305115603385

Oloff, F. (2021). 'Show' imperatives in smartphone-based showing sequences in Czech and German, Gesprächsforschung - Online-Zeitschrift zur verbalen Interaktion, vol. 2021, no. 22, pp. 691-724.

Oloff, F. (2021). New technologies—new social conduct? A sequential and multimodal approach to smartphone use in face-to-face interaction, Bulletin suisse de linguistique appliquée, vol. 1, no. Special, pp. 13-34

Porcheron, M., Fischer, J. E., & Sharples, S. (2016). Using Mobiles Phones in Pub Talk. 19th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW '16). https://doi.org/10.1145/2818048.2820014

Raclaw, J., Robles, J. S., & DiDomencio, S. M. (2016). Providing Epistemic Support for Assessments Through Mobile-Supported Sharing Activities. Research on Language and Social Interaction, 49(4), 362-379. https://doi.org/10.1080/08351813.2016.1199089

Robles, J. S., DiDomenico, S.M., & Raclaw, J. (2021). Using objects and technologies in the immediate environment as resources for managing affect displays in troubles talk. In J.S. Robles & A. Weatherall (Eds.), How emotions are made in talk (pp. 101—128). John Benjamins Press.

Schegloff, E. A. (2007). Sequence organization in interaction: A primer in conversation analysis I (Vol. 1). Cambridge University Press.

Searles, D. (2019). Positioning updates as relevant: An analysis of child-initiated updating in American and Canadian families. Research on Children and Social Interaction, 3(1-2), 144-167.

Seuren, L. M., & Huiskes, M. (2017). Confirmation or elaboration: What do yes/no declaratives want?. Research on Language and Social Interaction, 50(2), 188-205. https://doi.org/10.1080/08351813.2017.1301307

Shrock, A. R. (2015). Communicative Affordances of Mobile Media: Portability, Availability, Locatability, and Multimediality. International Journal of Communication, 9, 1229-1246.

Terasaki, A. K. (2004). Pre-announcement sequences in conversation. In Lerner, G. H. (Ed.), Conversation analysis: Studies from the first generation (pp. 171-223). John Benjamins.

Terraschke, A. (2013). A classification system for describing quotative content. Journal of pragmatics, 47(1), 59-74. https://doi.org/10.1016/j.pragma.2012.11.015

Turkle, S. (2012). Alone together: Why we expect more from technology and less from each other. Basic Books.

Turkle, S. (2015). Reclaiming conversation: The power of talk in a digital age. Penguin.

Unuabonah, F. O. (2018). Direct quotations in Nigerian investigative public hearings. Text & Talk, 38(4), 503-524. https://doi.org/10.1515/text-2018-0012

Yao, B., & Scheepers, C. (2011). Contextual modulation of reading rate for direct versus indirect speech quotations. Cognition, 121(3), 447-453. https://doi.org/10.1016/j.cognition.2011.08.007


1 It is also possible these cases could be considered confirmations, where the information is proposed as potentially known to all parties, yet requires some further establishing prior to the next phase of a relevant ongoing or future activity.

2 This further reinforces that, as Nagy and Neff (2015) acknowledge, affordances in communication scholarship are typically framed for the purposes of articulating precisely "what users and their sociality get from a technology" (p. 2).