When a Robot Comes to Life

The Interactional Achievement of Agency as a Transient Phenomenon





human-robot interaction, social robotics, agency, participation, sequence organization, autonomy


Conceptualizing agency is a long-standing theoretical concern. Taking an ethnomethodological and conversation analytic perspective, we explore agency as the oriented to capacity to produce situationally and sequentially relevant action. Drawing on video recordings of families interacting with the Cozmo toy robot, we present a multimodal analysis of a single episode featuring a variety of rapidly interchanging forms of robotic (non-)agency. We demonstrate how agency is ongoingly constituted in situated interaction between humans and a robot. Describing different ways in which the robot’s statuses as either an agent or an object are interactionally embodied into being, we distinguish “autonomous” agency, hybrid agency, ascribed agency, potential agency and non-agency.


Alač, M. (2016). Social robots: Things or agents? AI & SOCIETY, 31(4), 519–535. https://doi.org/10.1007/s00146-015-0631-6

Alač, M., Movellan, J., & Tanaka, F. (2011). When a robot is social: Spatial arrangements and multimodal semiotic engagement in the practice of social robotics. Social Studies of Science, 41(6), 893–926. https://doi.org/10.1177/0306312711420565

Antaki, C., & Crompton, R. J. (2015). Conversational practices promoting a discourse of agency for adults with intellectual disabilities. Discourse & Society, 26(6), 645–661. https://doi.org/10.1177/0957926515592774

Asaro, P. M. (2000). Transforming society by transforming technology: the science and politics of participatory design. Accounting, Management and Information Technologies, 10(4), 257–290. https://doi.org/https://doi.org/10.1016/S0959-8022(00)00004-7

Auer, P., Bauer, A., & Hörmeyer, I. (2020). How Can the ‘Autonomous Speaker’ Survive in Atypical Interaction? The Case of Anarthria and Aphasia. In R. Wilkinson, J. P. Rae, & G. Rasmussen (Eds.), Atypical Interaction: The Impact of Communicative Impairments within Everyday Talk (pp. 373–408). Cham: Springer International Publishing. https://doi.org/10.1007/978-3-030-28799-3_13

Axelsson, A., Buschmeier, H., & Skantze, G. (2022). Modeling Feedback in Interaction With Conversational Agents—A Review. Frontiers in Computer Science, 4. https://doi.org/10.3389/fcomp.2022.744574

Bekey, G. A. (2005). Autonomous Robots: From Biological Inspiration to Implementation and Control. MIT Press.

Bohus, D., & Horvitz, E. (2011). Multiparty Turn Taking in Situated Dialog: Study, Lessons, and Directions. In Proceedings of the SIGDIAL 2011 Conference (pp. 98–109). USA: Association for Computational Linguistics.

Boudouraki, A., Fischer, J. E., Reeves, S., & Rintel, S. (2021). “I can’t get round” Recruiting assistance in mobile robotic telepresence. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW3), 1–21. https://doi.org/10.1145/3432947

Bradshaw, J. M., Hoffman, R. R., Johnson, M., & Woods, D. D. (2013). The Seven Deadly Myths of “Autonomous Systems.” IEEE Intelligent Systems, 28(3), 54–61. https://doi.org/10.1109/MIS.2013.70

Braitenberg, V. (1986). Vehicles. Experiments in Synthetic Psychology. MIT Press.

Bratman, M.E., 1987, Intention, Plans, and Practical Reason, Cambridge, MA: Harvard University Press.

Breazeal, C. (2002). Designing Sociable Robots. Cambridge, MA: MIT Press.

Broth, M., Cromdal, J., & Levin, L. (2019). Telling the other’s side: Formulating others’ mental states in driver training. Language & Communication, 65, 7–21. https://doi.org/10.1016/j.langcom.2018.04.007

Button, G., & Sharrock, W. W. (2010). The structure problem in the context of structure and agency controversies. In A. Denis & P. J. Martin (Eds.), Human Agents and Social Structures (pp. 17–33). Manchester: Manchester University Press.

Bødker, S., Dindler, C., Iversen, O. S., & Smith, R. C. (2021). Participatory Design. Synthesis Lectures on Human-Centered Informatics, 14(5), i–143. https://doi.org/10.2200/S01136ED1V01Y202110HCI052

Clark, H. H., & Fischer, K. (2022). Social robots as depictions of social agents. Behavioral and Brain Sciences, 1–33. https://doi.org/10.1017/S0140525X22000668

Clayman, S. E. (2013). Agency in response: The role of prefatory address terms. Journal of Pragmatics, 57, 290–302. https://doi.org/10.1016/j.pragma.2012.12.001

Coleridge, S. T. (1817/2004). Biographia Literaria. https://www.gutenberg.org/files/6081/6081-h/6081-h.htm

Coulter, J. (1979). The social construction of mind: Studies in ethnomethodology and linguistic philosophy. New Jersey: Bowman & Littlefield.

Curl, T. S. (2006). Offers of assistance: Constraints on syntactic design. Journal of Pragmatics, 38(8), 1257–1280. https://doi.org/10.1016/j.pragma.2005.09.004

Davidson, D., 1980, Essays on Actions and Events, Oxford: Clarendon Press.

Defense Science Board. (2012). The role of autonomy in DoD Systems (Task Force Report No. ADA566864). Department of Defense United States of America. https://dsb.cto.mil/reports/2010s/AutonomyReport.pdf

Demuth, C. (2021). Managing accountability of children’s bodily conduct: Embodied discursive practices in preschool. In S. Wiggins & K. O. Cromdal (Eds.), Discursive Psychology and Embodiment: Beyond Subject-Object Binaries (pp. 81–111). Cham: Palgrave Macmillan. https://doi.org/http://link.springer.com/10.1007/978-3-030-53709-8_4

Dennett, D. C. (1987). The Intentional Stance. Cambridge, MA: MIT Press.

Due, B. L. (2021a). RoboDoc: Semiotic resources for achieving face-to-screenface formation with a telepresence robot. Semiotica, 2021(238), 253–278. https://doi.org/10.1515/sem-2018-0148

Due, B. L. (2021b). Distributed Perception: Co‐Operation between Sense‐Able, Actionable, and Accountable Semiotic Agents. Symbolic Interaction, 44(1), 134–162. https://doi.org/10.1002/symb.538

Edwards, D. (1997). Discourse and Cognition. London: Sage Publications. Retrieved from http://www.uk.sagepub.com/books/9780803976979

Enfield, N. J. (2013). Relationship Thinking: Agency, Enchrony, and Human Sociality. USA: OUP.

Enfield, N. J. (2014). Human agency and the infrastructure for requests. In P. Drew & E. Couper-Kuhlen (Eds.), Requesting in Social Interaction (pp. 35–54). Amsterdam: John Benjamins Publishing Company.

Fischer, K. (2011). Interpersonal variation in understanding robots as social actors. In 2011 6th ACM/IEEE International Conference on Human-Robot Interaction (HRI) (pp. 53–60). https://doi.org/10.1145/1957656.1957672

Fischer, K. (2021). Tracking Anthropomorphizing Behavior in Human-Robot Interaction. ACM Transactions on Human-Robot Interaction, 11(1). https://doi.org/10.1145/3442677

Forlizzi, J., & DiSalvo, C. (2006). Service Robots in the Domestic Environment: A Study of the Roomba Vacuum in the Home. In Proceedings of the 1st ACM SIGCHI/SIGART Conference on Human-Robot Interaction (pp. 258–265). New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/1121241.1121286

Garfinkel, H. (1967). Studies in Ethnomethodology. Englewood Cliffs NJ: Prentice-Hall.

Garfinkel, H. (1991). Respecification: evidence for locally produced, naturally accountable phenomena of order*, logic, reason, meaning, method, etc. in and as of the essential haecceity of immortal ordinary society (I): an announcement of studies. In G. Button (Ed.), Ethnomethodology and the Human Sciences (pp. 10–19). Cambridge, U.K.: Cambridge University Press.

Goffman, E. (1981). Forms of Talk. Philadelphia: University of Pennsylvania Press.

Goodwin, C. (2004). A Competent Speaker Who Can't Speak: The Social Life of Aphasia. Journal of Linguistic Anthropology, 14(2), 151-170. https://doi.org/https://doi.org/10.1525/jlin.2004.14.2.151

Goodwin, C. (2018). Co-Operative Action. Cambridge: Cambridge University Press.

Greenbaum, J., & Kyng, M. (Eds.). (1991). Design at Work: Cooperative Design of Computer Systems. CRC Press.

Harjunpää, K. (2021). Brokering co-participants’ volition in request and offer sequences. In A. P. Jan Lindström Ritva Laury & M.-L. Sorjonen (Eds.), Intersubjectivity in Action: Studies in language and social interaction (pp. 135–159). Amsterdam: John Benjamins. Retrieved from https://www.jbe-platform.com/content/books/9789027259035-pbns.326.07har

Hedayati, H., Szafir, D., & Andrist, S. (2019). Recognizing F-Formations in the Open World. In 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI) (pp. 558–559). https://doi.org/10.1109/HRI.2019.8673233

Heritage, J. (1984). Garfinkel and Ethnomethodology. Cambridge [Cambridgeshire]; New York, N.Y: Polity Press.

Heritage, J., & Raymond, G. (2012). Navigating epistemic landscapes: acquiesence, agency and resistance in responses to polar questions. In J. P. de Ruiter (Ed.), Questions: Formal, Functional and Interactional Perspectives (pp. 179–192). Cambridge, U.K.: Cambridge University Press. https://doi.org/10.1017/CBO9781139045414.013

Hirschauer, S. (1991). The manufacture of bodies in surgery. Social Studies of Science, 21(2), 279–319.

Hollan, J., Hutchins, E., & Kirsh, D. (2000). Distributed cognition. ACM Transactions on Computer-Human Interaction, 7(2), 174–196. https://doi.org/10.1145/353485.353487

Hollnagel, E., & Woods, D. (2005). Joint Cognitive Systems: Foundations of Cognitive Systems Engineering. Boca Raton, FL: CRC Press.

International Organization for Standardization. (2021). Robotics — Vocabulary (ISO Standard No. 8373). Retrieved from https://www.iso.org/obp/ui/#iso:std:iso:8373:ed-3:v1:en

Jakonen, T., & Jauni, H. (2021). Mediated learning materials: visibility checks in telepresence robot mediated classroom interaction. Classroom Discourse, 12(1–2), 121–145. https://doi.org/10.1080/19463014.2020.1808496

Jung, M. F. (2017). Affective grounding in human-robot interaction. In Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction (pp. 263–273). New York, NY, USA: ACM. https://doi.org/10.1145/2909824.3020224

Keevallik, L. (2017). Negotiating deontic rights in second position. In L. R. M.-L. Sorjonen & E. Couper-Kuhlen (Eds.), Imperative Turns at Talk: The Design of Directives in Action (pp. 271–295). Amsterdam: John Benjamins. https://doi.org/10.1075/slsi.30.09kee

Klowait, N., & Erofeeva, М. A. (2021). The Rise of Interactional Multimodality in Human-Computer Interaction. The Monitoring of Public Opinion: Economic & Social Changes, 1(1), 46–70. https://doi.org/10.14515/monitoring.2021.1.1793

Krummheuer, A. (2015a). Technical agency in practice: The enactment of artefacts as conversation partners, actants and opponents. PsychNology Journal, 13(2–3), 179–202.

Krummheuer, A. (2015b). Performing an action one cannot do: Participation, scaffolding and embodied interaction. Journal of Interactional Research in Communication Disorders, 6(2), 187–210. https://doi.org/http://dx.doi.org/10.1558/jircd.v6i2.26986

Laurier, E., Maze, R., & Lundin, J. (2006). Putting the Dog Back in the Park: Animal and Human Mind-in-Action. Mind, Culture, and Activity, 13(1), 2–24. https://doi.org/10.1207/s15327884mca1301_2

Latour, B., 2005. Reassembling the Social: An Introduction to Actor-Network-Theory. Oxford: Oxford UP.

Licoppe, C. (2021). The politics of visuality and talk in French courtroom proceedings with video links and remote participants. Journal of Pragmatics, 178, 363–377. https://doi.org/10.1016/j.pragma.2021.03.023

Linell, P. (2009). Rethinking Language, Mind, and World Dialogically. Charlotte, NC: Information Age Publishing.

Lipp, B. (2019). Interfacing RobotCare – On the Techno-Politics of Innovation. Technische Universität München, Munich, Germany. Retrieved from https://mediatum.ub.tum.de/doc/1472757/1472757.pdf

Miksell, L. (2010). Repetitional responses in frontotemporal dementia discourse: Asserting agency or demonstrating confusion? Discourse Studies, 12(4), 465–500. https://doi.org/10.1177/1461445610370127

Mondada, L. (2019). Contemporary issues in conversation analysis: Embodiment and materiality, multimodality and multisensoriality in social interaction. Journal of Pragmatics, 145, 47–62. https://doi.org/10.1016/j.pragma.2019.01.016

Nass, C., & Moon, Y. (2000). Machines and Mindlessness: Social Responses to Computers. Journal of Social Issues, 56(1), 81–103. https://doi.org/10.1111/0022-4537.00153

Otsuka, K., Yamato, J., Takemae, Y., & Murase, H. (2006). Conversation Scene Analysis with Dynamic Bayesian Network Basedon Visual Head Tracking. In 2006 IEEE International Conference on Multimedia and Expo (pp. 949–952). IEEE. https://doi.org/10.1109/ICME.2006.262677

Parenti, L., Marchesi, S., Belkaid, M., & Wykowska, A. (2021). Exposure to Robotic Virtual Agent Affects Adoption of Intentional Stance. In Proceedings of the 9th International Conference on Human-Agent Interaction (pp. 348–353). New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/3472307.3484667

Pelikan, H. R. M., Broth, M., & Keevallik, L. (2020). “Are You Sad, Cozmo?”: How Humans Make Sense of a Home Robot’s Emotion Displays. In Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction (pp. 461–470). New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/3319502.3374814

Peräkylä, A. (2002). Agency and authority: extended responses to diagnostic statements in primary care encounters. Research on Language and Social Interaction, 35(2), 219–247. https://doi.org/10.1207/S15327973RLSI3502_5

Pitsch, K. (2016). Limits and Opportunities for Mathematizing Communicational Conduct for Social Robotics in the Real World? Toward Enabling a Robot to Make Use of the Human’s Competences. AI & Society, 31(4), 587–593. https://doi.org/10.1007/s00146-015-0629-0

Porcheron, M., Fischer, J. E., & Reeves, S. (2020). Pulling back the curtain on the Wizards of Oz. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW3). https://doi.org/10.1145/3432942

Reeves, B., & Nass, C. (1996). The Media Equation. How People Treat Computers, Television, and New Media Like Real People and Places. Cambridge: Cambridge University Press.

Sacks, H. (1992). Lectures on conversation. (G. Jefferson, Ed.). Oxford: Blackwell.

Schegloff, E. A. (2007). Sequence Organization in Interaction. Sequence Organization in Interaction: A Primer in Conversation Analysis I. Cambridge: Cambridge University Press.

Schlosser, M. (2019). Agency. In E. N. Zalta (Ed.), The Stanford Encyclopedia of Philosophy (Winter 2019 Edition). Metaphysics Research Lab, Stanford University. Retrieved from https://plato.stanford.edu/archives/win2019/entries/agency/

Sheridan, T. B., & Verplank, W. L. (1978). Human and Computer Control of Undersea Teleoperators. Cambridge, MA. Retrieved from https://apps.dtic.mil/sti/citations/ADA057655

Skantze, G. (2021). Turn-taking in Conversational Systems and Human-Robot Interaction: A Review. Computer Speech & Language, 67, 101178. https://doi.org/https://doi.org/10.1016/j.csl.2020.101178

Sterponi, L. (2003). Account episodes in family discourse: the making of morality in everyday interaction. Discourse Studies, 5(1), 79–100. https://doi.org/10.1177/14614456030050010401

Suchman, L. A. (1987). Plans and Situated Actions: The Problem of Human-Machine Communication. Cambridge: Cambridge University Press.

Suchman, L. A. (2007). Human-machine reconfigurations: Plans and situated actions, 2nd ed. New York, NY, US: Cambridge University Press.

Thellman, S., & Ziemke, T. (2021). The Perceptual Belief Problem: Why Explainability Is a Tough Challenge in Social Robotics. J. Hum.-Robot Interact., 10(3).

Traum, D. R. (1994). A Computational Theory of Grounding in Natural Language Conversation. Rochester University, Rochester, NY, USA. Retrieved from https://apps.dtic.mil/sti/citations/ADA289894

Waring, H. Z. (2011). Learner initiatives and learning opportunities. Classroom Discourse, 2(2), 201–218. https://doi.org/10.1080/19463014.2011.614053

Warnicke, C., & Broth, M. (forthcoming). Making Two Actions at Once: How interpreters address different parties simultaneously in the Swedish Video Relay Service. Translation and Interpreting Studies.

Weatherall, A. (2020). Constituting Agency in the Delivery of Telephone-Mediated Victim Support. Qualitative Research in Psychology, 17(3), 396–412. https://doi.org/10.1080/14780887.2020.1725951

Weatherall, A., & Keevallik, L. (2016). When claims of understanding are less than affiliative. Research on Language and Social Interaction, 49(3), 167–182. https://doi.org/10.1080/08351813.2016.1196544

Wilson, G., & Shpall, S. (2012). Action. In Edward N. Zalta (Ed.), The Stanford Encyclopedia of Philosophy (Winter 2016 Edition). Retrieved from https://plato.stanford.edu/archives/win2016/entries/action




How to Cite

Pelikan, H., Broth, M., & Keevallik, L. (2022). When a Robot Comes to Life: The Interactional Achievement of Agency as a Transient Phenomenon. Social Interaction. Video-Based Studies of Human Sociality, 5(3). https://doi.org/10.7146/si.v5i3.129915